The situation you have encountered is very common.
The bottom of the tailstock has a rectangular protrusion that rides between the bedways of the lathe. The protrusion is very slightly narrower than the spacing of the bedways, which means that it can wobble just a bit. And this can lead to centering problems when drilling - problems that get worse as the gap between headstock and tailstock increases (due to the size blank you are drilling, the length of the drill bit, or how far you have extended the tailstock ram).
There are two things that you need to do. First, you need to make sure that your tailstock lines up with your headstock in the vertical direction. To do this, put a cone center in your tail stock, and a spur center in your headstock, and move the tailstock up to the point where the points almost touch. While there can be some mismatch in the horizontal plane that changes as you wiggle the tailstock, the points should be aligned vertically. If not, you may need to loosen the bolts holding the headstock to the bedways, and insert some shim material under the headstock to bring the points into alignment. This is something that you need to do once, but that you may not need to revisit on a regular basis.
Next, you have to be very careful (and disciplined) about drilling.
1. Mount the blank in the headstock chuck, and square off the face before starting to drill.
2. With the blank rotating, use a skew to cut a dimple in the exact center of the blank.
3. Mount the drill bit in the jacobs chuck in the tailstock, and slide the tailstock up to the point where the drill bit is centered in that dimple BEFORE LOCKING DOWN THE TAILSTOCK. That assures that the tailstock is exactly centered to the axis of rotation of the blank.
4. Finally, advance the tailstock ram to drill the hole SLOWLY, backing the bit out frequently to clear the swarf building up in the flutes of the bit.
