The first thing you need to do is check the alignment of your lathe - does the tailstock line up with the headstock?
But even if the lathe is aligned properly, it is possible for the tailstock to not be centered exactly on the axis of the rotating blank. When that happens, there can be visible wobble in the drill bit, the hole will be elliptical, and if the blank is thin and you are using a very small drill bit, you can end up with the hole going through the side of the blank. DAMHIKT.
On my lathe, there is a very slight amount of backlash in the mounting of the tailstock - the spacing between the bedways is a fraction of a milimeter wider than the protrusion on the bottom of the tailstock, which means that the tailstock can rotate a fraction of a degree until it is locked down. And if the tailstock can rotate slightly, there can be a noticeable shift in the tip of a drill bit - and the longer the bit (and the greater the extension of the tailstock ram), the greater that shift will be. I've compared notes with others who own the same lathe, and they report the same characteristic; I suspect this is typically of less expensive midi-lathes.
The solution is to first flatten the end of the blank, and then use the tip of a skew to cut a dimple in the exact center as the blank is rotating. Then, bring up the tailstock and center the bit in that dimple before locking down the tailstock. That will assure that the bit is exactly on-center with the axis of rotation of the blank.