I think I see the problem. You are probably using an older pen mandrel without an MT2 end. Look at the two centers above. There are two different angles used on the centers. The one on the right has a 60° center and is correct for the dimple in the end of the mandrel. The point on the left is used for WOOD, not metal pen mandrels. Metal pen mandrels REQUIRE a 60° center on both ends. Most mandrels come with a MT2 on one end and require a 60° dimple on the other. IF that is the case, you should not be using the wood point on the left side.
The point on the LEFT will roll around inside the dimple of a mandrel because it (the point) is too narrow and has space to roll around inside the dimple. You need a 60° DRIVE center to match the 60° live center of the tail stock.
What ever the problem is - there should not be the disparity of a 60° on one end and wood drive point on the other end.