My suspicion is that there is some hysteresis in your tailstock,
This is a fairly common issue with less expensive lathes. What frequently happens is that the extension on the bottom of the tailstock casting that aligns the tailstock to the bedways of the lathe is very slightly narrower than the spacing between the ways. As a result, the tailstock can rotate very slightly about a vertical axis - perhaps only a degree or two. But that means that the alignment between the headstock and the tailstock in the horizontal plane can vary depending on where the tailstock is in that very narrow angle of rotation.
If you are drilling, and the headstock and tailstock axes are not bang-on aligned, the drill bit will enter the piece at a very slight angle. If you are drilling very slowly with a fairly robust bit, you will get a hole that is very slightly conical rather than a pure cylinder - an error so small that in most cases you won't notice it. But if you are using a small bit (which will be more flexible), you may see visible wobble in the bit. And in extreme cases (such as drilling a deep, small hole when redressing BIC pens), the bit can wander off axis ending up with a hole through the side of the blank (DAMHIKT).
I suppose it might be possible for a good machinist to add shims on the bottom of the tailstock to correct this problem, but I'm not a good machinist. Instead, I've found that making sure that the tailstock jacobs chuck is firmly seated in the taper in the ram, and also making sure that the bit is centered in a center dimple in the workpiece before locking down the tailstock, will essentially eliminate the problem.