My approach is similar to Chris's. I bought the HF set. Then when I want/need a certain bit that's not int the kit, I almost always go to Victor Machinery because they have almost any size I need in both metric and SAE. They have longer length bits also. I've bought 2 colt bits (3/8, and 27/64 not from Victor Machinery). And I even bought a reamer (15/32) because the Jr Gent 1 cap material gets really thin. It was overkill though.
But the thing is, once you get settled in, you'll pick your favorite kits (or the kits that sell), and most of your pens will come from those kits. And yours will probably differ from mine. Mainly because there are just so many kits out there, and nobody makes them all.
That being said, here's what I think would be on most peoples list.
7 mm or J - Slimlines
25/64 & 15/32 for Jr. Gent 1 and it's variations.
3/8 (several kits use this size. I use it mostly for long clicks)
10.5mm & 12.5mm - Jr Gent II and it's variations.
10mm - Cigar
8mm or O - larger styled euro's and others.
Of course, there are more. But I think this would give you a good start. But the HF set will have almost everything you'll need, or at least something close enough. Then fill in where you need to, and when you need a replacement, get better quality.