As long as the wood is tight in the chuck the wood will have a center point. (or else headstock bearings are out but that's your new lathe) You can prove this with a pencil or use the live center just barely denting the wood. Take a skew and make a cone in the center, stop the lathe, is there a small hole from the skew? It's not the chuck or the headstock if there is a small point where the skew went in. The live center could be off from a misaligned tailstock though.
So it's either your bit or your tailstock. The biggest cause for bit drift is non concentric cutting surfaces on the two wings. If your bit is not HSS, then one pass, wobbling, through a 5" blank is enough to ruin it. Try several bits, if they all give the same result the answer is the tailstock shaft is not running on center or if on center then at an angle through the workpiece.
Another test is pick one of the bits and rather than crank it through the blank, crank it about a half inch and push it the rest of the way, at a low speed, with a loose tailstock. Hang onto the chuck when you pull it back out. I have a mini Rikon too and this is what I have to do to get a round hole. I also step drill it with 3 drill although some say not to do that. It works for me though. I get very tight 7mm holes. For large holes I cheat, I use a boring bar with a carbide cutter.
When you first see a wobble, take out the chuck and put in the live center. Lock down the tailstock, loosen the chuck, advance the tailstock forcing the round hole to center on the live center with the blank against the back of the chuck, and tighten the chuck forcing the chuck to center on the live center. Pull the center out slightly and turn the chuck, does a gap go all the way around with the chuck? If so redo till it's centered, now your bit will usually go straight through.
I suspect bobleibo may be right about those chucks even though I think for it to do this would probably have it come apart.