I've had that problem also. It's probably not the lathe itself. Many of us here are pretty picky about our work. Most pen turners out there probably turn multiple barrels just as you are doing here, and are happy enough with the results. Also, most of them are slightly, if not grossly, non-concentric. You will see the wood does not meet up with the kit parts as shown here. (I just grabbed a random pic from the internet of what I feel is a poorly made pen.) You can see the difference at the union of the barrel to the tip. The upper part of the union is smaller (overturned) than the tip piece, but the bottom looks even. There are other reasons this can happen, but the mandrel is the most likely cause.
View attachment 366464
To re-iterate earlier comments, that mandrel was meant to be used with a 60° live center. You will see a dimple in the end of the mandrel meant to meet up with this piece. You will find this much more stable, but do not over-tighten it or it will bend the mandrel. Once the mandrel is bent, there's really nothing that will give you an evenly carved barrel when using it. The hole in the mandrel saver is generally a little bigger than the mandrel, allowing it room to develop that wobble even if it's not bent. Also, as Dave mentioned, the threaded part is a bit narrower, but I find the fit to be slightly loose even with the mandrel properly inserted into the saver.
View attachment 366462
Shortening the mandrel by doing one barrel at a time will limit the wobble some. That's just a simple rule of physics.
Many of us turn "between centers" to take any mandrel problems completely out of the equation. If you search "TBC", you will find plenty of discussion on that. Below is my result using the TBC method. Many are able to achieve this using a mandrel, but I didn't want to spend money on a really good mandrel ($100-150ish) when TBC was an easy option for me. Even TBC will be a problem IF you don't have the ends of the barrel properly squared (trimmed).
View attachment 366463
As for it feeling rough, that could be dull tools or just not sanding down to a low enough grit. Those come with practice of both sharpening and sanding/finishing. Just keep turning and you will get this part fine tuned.
With all of that being said, don't overlook the simple thing that there may be some foreign matter in the morse taper. Make sure to clean that out, both inside the lathe shaft and on the mandrel. Even a small drop of glue will cause a problem there. I know this from my own experience.