Fangar,
I agree, a vise may induce uneven trimmings. However, my solution is to still use a barrel trimmer, but to place it rather loosely in my vise (just tight enough to allow it to keep from rotating in the vise. I use a waste block underneath it to keep the blank above the height of the vise. This allows the barrel trimmer to follow the tube and, if necessary, knock the blank out of alignment. I also mark the blank before I drill it, and consistently put the blanks in the vise in the same orientation every time, regardless of what I'm doing to them (drilling, trimming, etc.).