Brian,
I don't know if this is your problem or not, but stay away from brad point drill bits when drilling on the lathe. I love them on the drill press, but on the lathe, just a little bit of slop in the jacobs chuck or the tail stock puts the point of the bit off center, which is hard to correct for with the brad point. Using a standard bit (preferably a split bit) allows the bit to settle in the true center of the blank, as long as you start the hole slowly. After learning this lesson the hard way, I now do all my drilling on the lathe and get much better results than I did on the drill press. Now, as long as the blank is square (or at least has parallel sides), no more off center drilling.