Bend a piece of wire with two offsets so you have 3 straight runs that are close to parallel.Start with a piece of wire like a coat hanger or if you have a long peg board hooh it is even better.
The offset should be so that it comes close to the DP table when you chuck it up.
Raise the table so that it just clears a credit card or what ever you choose to use as a "gauge"
Turn the chuck by hand and see if you have the same clearance all the way around the table.
If not adjust the table so it does.
The only other thing I can think of is a dull bit.
Do you get the same results regardless of the bit you use?
Speed should not be a factor.
Piece of stock cut on a bias will also cause a bit to "drift"
One more thought.
If you are using a vice but your stock isn't square that would also cause your exit hole to be off center.
As Dario pointed out more info is needed but the above information should rule out the obvious.
One more idea.
Start with bigger blanks.In 2 inches it should be close enough for hand grenades.[
]