Nancy,
Basically on a mandrel, Bushings are usually considered as "consumables" because the occasional hit they take from the chisel edge, not to mention the very thing you are talking about - sand paper. Even going from the center outward, you will wear the bushings down over 10 to 30 pens - to the point that correctly sizing the blank by the bushing will be impossible. If you are proficient and meticulous you might be able to postpone this scenario.
For sizing, with calipers - measure the fittings of the pen (nib, center band, clip end) mark them down and turn to size without the bushings. Learning to do this will result in a much more consistent fit than turning to size with bushings. Even worn bushings can be used if you size by measuring with calipers.
I wrote the above to suggest the mandrel-less method but you don't like it. I have big hands but haven't bumped them on the dead center or live center yet.
There is one other way to avoid bushing dust, but it is a little controversial because a few people don't think it can't be done. Proficiency with a skew or even with a very very sharp gouge with small bites produces clean cuts to the point that sanding is not needed. But a good bit of experience is usually necessary to accomplish this. I made one pen (and subsequently several more) in which ANY sanding would produce dust smearing that was impossible to remove without drastic measures. I used this method and it worked wonderfully.