Yours is a rather common problem and the most common reasons are 2: The wood wasn't as dry as you thought it was, or the wood expanded from heat while drilling and the hole shrank as it cooled.
The drill sizes that are available for some of the larger pens don't allow any room for shrinkage, and there isn't enough wood in the barrel to allow drilling a 1/32" larger hole. Some of them are so tight that using a file to fit the brass tubes is almost a requirement under the best of conditions. The El Grande can be this way if it is made to the kit directions.
Sometimes folks wet the inside of the hole before using a Polyurethane glue, and find that is enough moisture to swell the wood to where the tubes won't fit. Letting the wood dry again will usually solve this one, and not wetting the inside of the hole will keep it from happening. This is how I discovered that Poly Glue will dry perfectly well without adding moisture. The onlt difference is that it takes a little longer.
And, sometimes there is enough fuzz inside the hole from the drilling that the tubes don't fit. A wood rasp easily solves this one.
You can put the blanks in the Microwave and get them hot, and then glue in the tubes before they cool down. The problem is that there may be enough stress on the wood for it to crack when it cools and shrinks around the brass tube. I would try this on a couple blanks before doing it a bunch of them.
You could also use a press-fit of the brass tube into the wood. I have done this and then drizzled some thin CA around the joint to lock them in. The only problem is, again, that there are stresses placed on the wood that can cause cracking later.
The only "safe" solutions that you have now have already been given in other's replies. You can work a sharp drill bit through the holes (preferably by hand) and one that is 1/64" larger would really solve the problem. Wrapping sandpaper around a dowel might be as quick, and easier. Or, you can do it with a wood rasp or a file.
If you are a member of the North Florida Woodturners, you will know Austin Tatum, and I recommend that you contact him. Austin is the best problem solver I know of, and he is the best resource you have in Tallahassee.
A couple tips for next time. Make sure the wood is dry, and don't let it get hot.
It is sometimes better if the wood is "over dry" to below equilibrium when making these larger pens with their tighter fits. The benefit of it being "too dry" is that the wood will expand after it is drilled and the hole will become larger. You can dry the blanks in the oven or use several 1/2 power cycles in the Microwave.
Always drill a 1/4" diameter pilot hole through the blanks before drying to get the internal moisture out. This will also act as a pilot drill for the final drilling.
Make sure the drill is sharp (a new drill is not always sharp), and <b>do not let the wood get hot</b>.