I had similar problems with CA when I was using accelerator and was in too much of a rush to let the accelerator work and evaporate. I can't say that I'm correct, but my theory was that after putting on one coat of CA and then using accelerator on it, when there was still a small amount of accelerator on the pen and I hit it with the next coat of CA, the CA cured before it had a chance to bond with the previous layer, hence you have what amounts to 2 thin sheets of CA right next to each other, rather than a thicker bonded pair. Some of the CA around this area will have bonded, so it stays there and is just fine until some stress comes along (like trying to separate the bushings, or even pressing the pen parts together) and shifts the parts that are not bonded, taking some of the parts around it that did bond with is, and starts separating the layers...
I stopped using accelerator because of this, because I'm too impatient to be sure that it has all evaporated before moving on....
Now if you aren't using accelerator, you can ignore all the above...And all this is assuming that under the layer that has come off is more CA, If the CA is chipping off right down to the wood, then there are other issues at work...