I'm a new turner, but I had similar issues with my CA finishes adhering to the metal bushings. Caused the finish to crack when taking it off, not even an Exacto worked right, and I quickly noticed my bushing were no longer the right size, they had grown.
As mentioned, I got an HDPE rod from amazon, 3/4" x 24" I believe. I cut off 3/4" segments, center drilled them to 1/4" to accept the mandrel. I then turned them into a tapered bushing by leaving one side large, and tapering the "spindle" down to flush with the mandrel. Next time, I'm going to mount them on a 1/4" wooden dowel rod to turn them, as I left gouges in my pen mandrel.
When you have the blanks turned and sanded to the desired shape, remove the blanks and metal bushings from the mandrel, and remount the blanks using the HDPE bushings on each end of the blanks. These flush bushings will leave a noticeable ledge/gap at the end of your pen blanks, perfect for the CA to wrap around. As they are tapered, they will automatically fit into and center any size pen tube.
The CA then will slide right off the bushings (usually, sometimes I get 2 adjacent bushings stuck together through a physical, not chemical, bond, they crack apart easily enough), leaving some scrub CA finish near the tube, but not on the edge of the finished blank.
I have also started to take the blanks back to the barrel trimmers to get rid of the excess CA finish, then remount the blanks on the HDPE for the micro mesh sanding and polishing step. This allows the lower grit mesh to rub off any CA burrs left on the ends. It turns out to be a bit more work, but the finishes are much better than before, and the fit of the pens is near perfect every time, as the blanks are flush with the tube again.
Guru's, please feel free comment on that, as I'm brand new, and it's what I've found to work for me, but insight is always appreciated.
HDPE Rod:
https://www.amazon.com/gp/product/B0013HO2XK/ref=oh_aui_detailpage_o06_s00?ie=UTF8&psc=1
It's about the same price as 2 sets of commercially available bushings, but you get a few dozens sets out of it. And heck, we're turners here, this type of project should be our bread and butter.