Lots of good advice here. Let me add a few more thoughts:
1. Pen blanks usually have the grain parallel to the long dimension. The grain pattern in the wood reflects the difference in the rates of growth in the summer (rapid growth) and winter (slow growth) seasons back when the blank was actually a tree. If the grain pattern is strong and distinct, that often also means that there is a big difference in the hardness of the wood between summer and winter growth. It's easier for the drill bit to penetrate softer wood. Taken together, all of that means that its entirely too easy for the drill bit to flex enough to follow the softer grain in the wood rather than maintain a straight line, especially if the bit is advanced into the blank too fast. To avoid this problem, regardless of whether drilling on a drill press or a lathe, advance the bit very slowly. The bit has to cut a hole before the bit can advance into it - if you force the bit, it will deviate and follow the softer portions of the grain.
2. When drilling on a drill press, the bit is rotating against a stationary blank. When drilling on the lathe, the bit is stationary and the blank is rotating. In either case, the relative speed of rotation is important. Too slowly, and the bit will cut slowly, and since you have to cut a hole in the blank before you can advance the bit into that hole, the entire drilling process will be very slow. Too fast, and you will generate heat from the unavoidable friction caused by the relative movement between the bit and the blank, and too much heat can cause the blank to crack. As others have noted, the sweet spot for rotational speed is in the range of 400-800 r/min.
3. Allowing swarf to build up in the flutes of the bit increases friction and heat. So periodically back the bit out of the hole and clear the swarf out of the flutes. How periodically? I generally don't try to advance the bit more than about 1/2" before backing out to clear swarf.
4. You can reduce friction by lubricating the bit with BLO or wax. However, if you are drilling a pen blank where you intend to glue a tube in the hole, putting oil or wax into the hole could complicate gluing.
5. When drilling on the lathe, the jacobs chuck is held in the tailstock ram by friction, and when you extract it to clear swarf, you can break the seal which will cause the bit and chuck to loosen and start spinning. It is generally recommended that you grip the chuck with your left hand when drilling to keep this from happening. I would add that it is helpful to wear a glove on your left hand - if the chuck does come loose and start spinning, the knurled outer shell of the chuck will act like sandpaper on a bare hand - DAMHIKT. I use a kevlar carver's glove for this purpose.