Yep, acetone absolutely...!
Use a good paper towel, the ones that don't leave "fluff" stuff behind. Wiping pen barrels just before applying the CA finish, and due to the small amount of wood left, make sure you don't soak the wood too much, otherwise it can compromise if not destroy the glue you have holding the tube in place.
Acetone being very fast drying liquid, should be used with the lathe running and the acetone in the paper towel should just wet the wood (pen barrel) as you spread it. Continuing rubbing the paper towel on the blanch, (friction) will evaporate the acetone before it has time to affect the tube glue...!
Good luck.
Cheers
George