Yeah, when you're down to the slimline bushings, you've got a millimeter left all the way round before you hit brass... that's not a lot of holding power against a lathe chisel. Switch to sandpaper at that point and you'll have less problems, unless you're working with a segmented blank with contrasting colors that like to bleed.
As for changing the size of a bushing .... you can do that easily by just sanding the bushings themselves with a few plain brass tubes on the mandrel ... (one in front of the bushing pair you want to alter, and one tube behind, with another bushing that you won't alter after that to tighten up for your tailstock end).
The idea is, that you just want to lightly sand some of the outer surface of your bushings away so that it's a tiny bit smaller than the standard size. Sandpaper will do this nicely, while your lathe chisels won't really want to do the job.
Don't forget to go all the way up with your micromesh pads to polish the heck out of those bushings, and then apply a few coats of paste wax for good measure, in case you forget to swap them out for vinyl or delrin bushings right before you do your CA finishing.