What kind of 'damage' are you seeing?
I suggest looking very closely to determine exactly how the pen mill is affecting the tube. Is the 'damage' uniform around the entire circumference of the tube, or is it confined to one side? Is the 'damage' in the form of scoring marks inside the tube? Or is the 'damage' on the end of the tube?
A pen mill is intended to remove surplus material from the end of the blank, leaving an area around the end of the tube that is perpendicular to the tube. In doing that, the mill may also remove a small amount of brass from the end of the tube. You should stop using the mill as soon as you see bright metal at the end of the tube indicating that the mill has reached metal. Removing a small amount of brass is OK, but you don't want to remove so much that the actually shorten the tube. And note that this means that you will see some brass shavings in the material removed by the mill - that is also normal.
However, a mill has a second purpose - it is very easy to get some glue inside the tube, and the mill is also intended to scour out the inside of the tube. The diameter of the mill shank should match the inside diameter of the tube very closely, so if you have some dried glue inside the tube, the mill will remove it, but without significantly removing any brass from the inside of the tube. However, if you find that the 'damage' inside the tube is not uniform around the circumference of the tube, but rather is concentrated on one side, that suggests that the end-mill cutting edge of the mill isn't exactly perpendicular with the shank.
Without seeing the 'damage', there are four things that I can think of that could be happening:
1. What you think is 'damage' is actually the edge on the mill shaft peeling away dried glue from the inside of the tube - ie, a total false alarm. In this case, the scoring inside the tube will be around the entire circumference.
2. What you think is 'damage' is the mill machining a bit off the end of the tube. Again, this is normal (as long as you don't get carried away and make the tube too short).
3. I have the PSI mill, with both 7mm and 10mm shanks. The hole in the mill is slightly larger than the shank, so when it is locked in place with the grub screw, the axis of the end mill is very slightly displaced from the axis of the shaft - parallel but displaced. But I could imagine that a small amount of crud inside the hole in the mill could cause the end mill to be tilted very slightly, resulting in the mill and shaft not being exactly perpendicular. This could result in scoring inside the brass tube that appears on one side, but not around the entire circumference.
4. The third possibility is a stretch, but I will mention it anyway. At one point, I bought a length of brass tube at a hardware store that was nominally the same diameter as the tube furnished with pen kits, but I found that the wall thickness was actually very slightly greater than the thickness of the walls of the tubes in pen kits. As a result, while the OD was the same, the ID of the tube from the hardware store was very slightly less than the ID of the tube in the pen kit. Pen kits are generally made in Asia, while the tube I bought at the hardware store was made in the US. That showed up as a mismatch between the tube and my pen mill - the mill actually wanted to remove a bit more brass from inside the ends of the tube.