For those who really prize their unfinished pens, they will clean them and wax them and then wipe the wax off so as not to leave residue on hands or shirt pocket. But as others said, hand oils are not good for the long run.
As an example, look at kitchen cabinets and unless someone cleans them almost daily, dirty hand wear show up after months or a couple of years. Same for corners where there is heavy traffic, and the dirty corners are at hand level.
Fine furniture does not go through the kind of handling that pens do and if so, they are cleaned and waxed on a regular basis. There is a well sought after patina that builds up due to hand touches on wood but these woods are still cleaned and waxed regularly, or the patina will turn dirty looking. Men in general avoid cleaning wood like hired help or mom's used to do, except for those that understand the whole process and are meticulous at it!
