Just curious - isn't there a bit of danger to your hand when you do this? If the Morse taper is seated I don't see much of an issue. But if it isn't, won't the chuck spin if the bit binds? And if you have that "death grip" on it..... I do have a chuck but don't use it for drilling pen blanks.
1. You don't keep a deathgrip on it ... just firm pressure to hold the chuck (and bit) steady, with light pressure towards the tailstock just to keep the taper seated securely.
If the drill chuck starts to spin, simply release the chuck and let it spin in your hand (very loose hold) ... the options you have at this point are:
* Release the chuck and see what happens (dumbest move ever...) Don't forget to keep your insurance up to date.
* Grab the chuck firmly to see if the drill comes loose (scary idea, but it DOES work ... if it doesn't work, then the grab will force the headstock to a halt along with your motor or belt .... or your hand will spin around the chuck, which is the scary and dangerous part. You CAN damage your lathe motor if you do this a lot.) Follow this up by re-seating the chuck.
* While still keeping the chuck spinning centered on the lathe, reach over with your OTHER hand and hit your emergency stop (safest option) .... follow this up by re-seating the chuck.
2. Drill chucks, unlike 3-jaw scrolling or other large body chucks, are streamlined attachments. They should have nothing sharp or knobby sticking out of their sides to catch/grab your hands and fingers. They should be quite safe to hold steady.
I routinely still hand hold a plastic bodied keyless drill chuck (taken off of a keyless chuck drill) while drilling freehand ... I only really do this anymore for items that don't need to be precisely drilled. For instance, I'll chuck up a large diameter bit with a tape flag on it for depth gauge, and drill my depth in a small bowl or cup blank. I don't need to be precisely centered, though with my blank spinning it's actually pretty difficult to get things really off center... things tend to want to find the centerpoint of the spindle. The chuck body is smooth black plastic, not overly heavy, and easy to use.