Agree with above comments 100%. When I work with those materials I typically do a "test fit" when getting ready to assemble. After checking for "junk" in the tube as Mike Broberg says, I will scrape off any big stuff with a small blade or an Exacto-style knife. I'm looking for a fit that is stiffer than a "slide in". Then I'll take some 220 (or 400) and wrap it around a rod and sand the interior till I get a fit where I can force it in but pull it out, too.
If at any point the component slides in easily, I stop doing anything that will enlarge the ID and proceed as below.
Then I will take a wee bit of thin CA and put it on the inside of the tube - NOT ON THE COMPONENT - and complete the assembly. Doing this prevents the CA from getting on the outside of the blank or the component.
Note that I said a WEE bit of CA. Takes very little to coat the mating surfaces and give a super good bond. You don't want too much CA on the inside as it will kill your twist or click. Never tried Loctite as JT says above. I think I might try that one.
It's a dance but doesn't take too much practice.
By the way, excellent finish on that pen and great pairing of the blank to the pen.