Are you using a mandrel? Have you checked the bushings with a caliper to see if the hole is off center?
Tail stock just a smidgen too tight on the mandrel; bushings with a .003 off center hole, a bent mandrel with runout of .003 on one end. Using the original tail stock center instead of a 60° live center can cause this. Sanding with just a smidgen too much pressure while using a mandrel will do the same. There are other circumstances that cause this too. And the result will be one side in one spot will be sanded more than the rest of the blank.
And I have just mentioned the mechanical-techincal aspects. Not the ones mentioned above of moisture, oil, CA etc.
From my experiences in reading here - more times than not, the problem is that there is not enough of a buildup of CA before sanding. A slight out of center or out of round can still be overcome and not noticed if the CA build up is enough that you are not sanding through. Still if you are having high spots that are sanded though to the wood, I would look for the cause.