Before applying CA, I make a final pass along the blank with my rounded skew laying flat on it's side like a scraper, and just "kiss" the blanks for a few passes, creating a very fine dust along the skew cutting edge. No sanding necessary. Apply the CA, then on to the MM & polish.
I sand every material to 12,000 before applying any finish. Am I wasting time? MAYBE! When I selll a pen ( also made msuical instruments, at one time), I know that I have made it as well as I can possibl make it. The wasted time is made up with good sleep.