I'm with Dan, I wait usually at least 24 hours after I'm done with applying CA. Once I get the CA on enough to sand smooth with 800 grit paper, it sits for a day. then I go back to MM pads and then buffing. I find that if I finish the CA right after I'm done applying it, it looks great when I'm done, but will dull a bit and often form slight texture, especially if the wood has voids and such.
I also wait overnight at least after polishing before I assemble the pen, simply because I'm not at 100% success rate with CA, so if I happen to have sanded or buffed through the CA at some point, it will be nice and shiny off the lathe, but the next morning there will be a dull spot where there is no CA. Much easier to fix this before you assemble...
CA may dry to the touch very quickly, but how long it takes to be completely hardened and stable is affected by the total thickness it is applied, temperature/humidity, time, brand, viscosity, and several other factors.