The problem that speed presents in drilling is that as the swarf builds up in the flutes of the bit, it rubs on the inside of the hole, and the friction produces heat that can cause damage to the blank. Blanks can crack or even blow out if moisture trapped in the wood is converted to steam by the excessive heat.
But rotational speed is not the only concern. In my experience, the rate that the bit advances into the blank is a far greater concern. If you try to advance the bit too fast, you take the risk that it will wander off-axis. This is especially true in timbers with a pronounced grain pattern reflecting the seasonal growth cycles in the tree - summer growth produces softer wood, while winter growth produces harder wood, and if the bit is forced too rapidly into the timber, it will attempt to keep to the softer wood, deflecting off the boundary between summer and winter growth.
So my experience is that it is better to drill at a higher rotational speed, and periodically back the bit out of the wood to clear the swarf from the flutes of the bit, while advancing the bit into the wood at a fairly slow speed. That way, the sharp leading edge of the bit can cut a hole in the timber that isn't biased by the summer/winter grain boundary, and then the bit can move into that hole. I keep an small brass or steel wire brush on the end of my lathe that I use to clear swarf from the flutes of bits while drilling.
The other key word here is 'sharp' - drilling is always easier with a sharp bit. Many people use mechanical sharpeners (Drill Doctor's and the like), but I find that a diamond paddle can very easily and quickly tune up the cutting surfaces on drill bits. The secret is to study the geometry of the bit to learn where the cutting edge is, and what you need to hone to restore that edge.
The actual ideal rotational speed depends on the type and size of the bit, the hardness of the timber, and whether you are drilling face grain or end gain. In general, I have found that the rotational speed has to increase as the bit increases in diameter, when switching from twist drills to forstner bits, as the wood becomes harder, and when drilling end grain. And with increased rotational speed comes more frequent stops to clear swarf. And unless the intention is to glue something into the hole, using a little lubricant can be helpful - some people suggest BLO, but I find that having an old cancle or block of canning parafin next to the lathe for use in lubricating the bit is very helpful, especially when drilling end grain. Ultimately, drilling is an art that you learn through experience and by making mistakes.
