It does matter, but only to a degree - depending on the individual material. Every material has its own characteristics including different kinds of wood. Most people don't know that and most people are just lucky most of the time. You (anyone) can speed drill a 3/8" hole through a 2x4 without a problem, but drill a 3/8" hold in a 3/4 square by 3 inch long hole in the same wood . . . and without the proper support, you will get a blowout like you had. A pen blank cannot take the same pressures that the same wood in a 2x4 can!
I prefer and do all my drilling on the lathe.Many factors could have come into play. I will list them. The blank was not secured to a base that is flat . You can not use those type jigs and have the blank suspended in mid air. Needs to be resting flat on the bottom. The speed of the drill press must be slow enough to match the material you are drilling which in this case around 500rpm. tops. The drill press quill must not be wobbly or out of round. You need to secure the jig so it does not move. You need to drill in steps and not all at once with a bit that size. You do need lubricant of some sort. You need to clear the swarf from the hole often. You need to keep the bit cool. I use DNA. Has worked well for me for many years. The jig needs to hold the blank straight and steady and not too much crushing pressure. That blanks gets thinner the bigger the bit and the remaining material becomes weaker.
I too prefer to drill on the lathe. Better control and more accurate. Good luck.