You are drilling on the lathe ... that is a good practice in order to get a hole which is accurately aligned to the blank axis.
However .... the accuracy of the resulting hole can depend on the nature of your lathe chuck, as well as the starting shape of your pen blank + material.
For best results, the pen blank should be turned round before drilling on the lathe.
The pen jaws you can buy to fit onto a lathe headstock are not particularly accurate ... after all, we are talking about an error of only a few thou.
I think one should not expect too much from those jaws.
In your case, I would recommend touching up the tight part of the hole with a round file ( rat-tail file ).
Another help would be a variable-diameter, adjustable reamer such as these which I buy from Busy Bee Tools in Canada.
Surely Grizzly or HF has them in the US.
I use these reamers a lot for adjusting the drilled hole for tight-fitting tubes. . But take care with these not to enlarge too much.
B3125 7pc Adjustable Blade Reamer Set The blades have hollow ground cutting edges for minimal friction and the ends are center drilled. Size: 1/4" - 9/32" to 7/16" - 15/32"
www.busybeetools.com
Bear in mind another couple of facts. . First, the actual size of a drill bit ( jobber size drills ) is usually just a shade ( 0.002" or so ) under specified size.
Why that is so, I do not know, but I have measured many brand new drill bits and found that to be the case.
Second, after you remove the drill bit from the drilled hole the hole collapses in size by a few thou due to the elastic properties of all materials.
This "elastic collapse" is in addition to any "thermal collapse" due to the heating of the material from the friction of drilling.
So, the bottom line is that drilling is inherently an inaccurate process and the inaccuracies can vary with the choice of materials and the details of your drilling process. . The inaccuracies are not very much, but they are enough to affect the fit of the brass tube .
As an example, consider making a Sierra pen kit. .
The recommended drill size is 27/64" diameter = 0.422" ( not counting the undersizing of the drill bit ). . Your brass tube measures at 0.408".
So already, the difference in those two numbers is only 0.014" .... at the most. . That would be your MAXIMUM clearance between tube and hole.
If an elastic shrinkage of the hole by only 1/2 % occurs, that can reduce your hole by another 0.002".
Allow the thermal shrinkage of about the same amount.
So due to those 3 effects, the hole is down from the MAX clearance of 0.014" by about 0.006" to about 0.008" not considering anything else.
I am just trying to give you a rough idea of SOME of the reasons for an undersize hole.
Another thing you mention is a slight tapering of the hole .... tighter towards the center of the blank, looser towards the drill starting point.
At the drill starting point, the accumulated time that the drill has been in contact with the blank is longer than near the blank's mid point.
So you see that the "drilling action" has been going on longer near the blank end ... hence removing a little more material.
Add to all this, the slight wandering of the drill bit as it enters the material at the end of the blank, just as John T. has said above.
This slight wandering will be worse if the end of the blank is not exactly square with the blank axis.
I realize that none of this is 100% quantitative, but I have confidence in the general ideas.
And what you have described is totally consistent with my own experiences.
I hope this helps a little in the understanding. . Of course, I stand to be corrected if I am in error on anything.