Definitely avoid CA glue. Ed from Exotic Blanks warned me about this, and it was after that that I noticed on a few pens where I'd glued parts into tubes with CA, the platings had a few discolored areas. Not overtly noticable, but its apparently a potential consequence of using CA.
Aside from Epoxy, you might try Red Loctite is another potential option.