I always sand the brass,holding the brass on a nail punch.I believe the punches were almost perfect for me,made a purpose built brass holder and now use that.Mostly do them in batches of a hundred at a time. A guy I know makes high quality pens and has never sanded a brass sleeve,never had one come apart,another never glues the timber on buys his drills custom by the hundred,presses the timber on the blanks.Horses for courses.
The sanding task is a bit mindless like churning butter,milking cows.polishing shoes,all for a reason.For me it is essential,just another part of to be sure to be sure,must be the Irish in me.
Peter.