What kind of live center are you using? Is it a 60° live center? If you don't know about this, you should read up on it. It sounds like your cause.
1. Most pen lathes are WOOD lathes. Wood lathes come with a live center made for WOOD. The mandrel is METAL. When a live center that is made for Wood, is used in a mandrel, the point of the Wood live center does not match the Metal cup of the mandrel, it balls up into a tiny ball or bends and the mandrel wobbles.
2. You need a 60° live center.
3, If it is not that, you are probably pulling your tail stock up too tight.
4. OR the tail stock in not locked down,
5. OR the tail stock and head stock are not in alignment.
From your description, I would go with # 1 as my first choice.