Quit fighting it. There is an easier way if you spend a little time preparing; once learned it is easy every time with holly, metal spacers, and other colored wood too.
Look at these:
Solder added to the 30 pieces of silver blank
www.penturners.org
Elegant Beauty TI Gold Bloodwood, BOW, Holly core solder CA finish
www.penturners.org
Those silver dots smear like pencil lead with ANY sanding. The key is as Ken mentioned above. If you have a carbide scraper, or HSS scraper, spend a few minutes sharpening the HSS and then a few minutes honing it. Hone it well. The above pen did not have a piece of sandpaper touch the wood and metal parts. With the HSS scraper or carbide (fresh new carbide) turn at about 3000 rpm and touch and let it graze by. When you get very close to the size you want, turn the carbide to a new side or freshly hone the HSS scraper and do your final touches. Smooth as a Baby's behind. Don't play around with the CA. Add three or four GOOD layers of CA before you decide to sand the CA. You do not want to sand through the CA to the wood or it will be visible once you re-coat it with CA. Well, some people don't pay attention to it but to a good eye and good critic, it will be noticeable.
As to bushings, I use TBC. With TBC, there is a thickness difference between the centers and the blank. So you won't be touching the centers as you turn it. You can use the bushings to get it near to size but take them off, put it between the centers (TBC) and use calipers to determine the final sizing with using the Carbide or HSS scraper. IF you are proficient with the skew (I am not) you can do the same with it. Just make sure you have honed the HSS scraper before final turns.
For your information, I have worked with holly and bloodwood for about 12 years. I love those contrasting natural colors. I have tried stabilizing holly but after considerable tries, I finally came to the conclusion that Holly does not need stabilizing. Bloodwood can stain holly if acetone is used to clean a segmented blank. DAMHIKT.
In case you are trying to figure out how it is going to look:
This is a view of the metal and wood segments from a pen made in late '08 and used almost daily since then. Do segments move, shrink, grow? Here is a photo of my one of my pens and experience.
www.penturners.org
I "Think" that was before CA was applied and see how clean the divisions are of dust. It works.
Last, and another subject altogether: Scrapers (HSS and carbide inserts) don't do as well with soft wood unless they are stabilized. On soft woods, skews will do better. On hard woods, scrapers will do fine. (Some people use a skew like a scraper, on its side. In that case the skew IS a scraper.)