It does matter, but only to a degree - depending on the individual material. Every material has its own characteristics including different kinds of wood. Most people don't know that and most people are just lucky most of the time. You (anyone) can speed drill a 3/8" hole through a 2x4 without a problem, but drill a 3/8" hold in a 3/4 square by 3 inch long hole in the same wood . . . and without the proper support, you will get a blowout like you had. A pen blank cannot take the same pressures that the same wood in a 2x4 can!