I had the same problem when I first started. I believe it was caused by 3 things. First, I wasn't doing a very good job of sanding/MM near the ends becuase I was trying to not sand the bushings. Second, I was not doing a good job putting the finish on near the pens becuase I was trying to avoid coating the bushings. Third, once my fish was on the pens I was not doing a good job sanding/polishing near the ends. Yep trying to avoid those bushings again.
I now sand and finish between centers without any bushings. The problem went away instantly. BTW I had the problem with both Friction Polish and CA finishes.