I can give you some insight in this since I went down the exact same road.
I too started with a drill press (Harbor Freights smallest model no less), and the lathe attachment. Your back will thank you when you get the lathe. Turning on a DP is bearable and I thought nothing of it until I switched to a lathe. I'm very proud to say that I made my first 23 pens on the drill press. I'm even happier that I no longer use that method (I was only going to do one at first, and don't know how this all happened.

).
My first lathe was the smaller Harbor Freight lathe you are eyeballing. There are many who do like that lathe. It's variable speed is a plus (I had two of them go out on me before I upgraded), and it's easy to move around (since I used my office as my shop, I had to routinely rotate machinery on and off the desk as there wasn't enough room, and it was my office:redface

. After trying a friends Delta, and having to take back the second mini (the service plan is worth it), I decided to upgrade to the bigger bench top model. There is a significant difference in the horsepower, and you can notice it. The more powerful one will take less time to turn, and due to weight resist vibration. Changing belts can be a hassle (especially when your used to VS), but the power trade off makes it worth it.
Funny thing is, as I was buying the first one, a gentleman fellow shopper told me the same story I'm telling you (minus the drill press lathe attachment).
The weight and size of the bigger one do make a difference. Moving the little one around was no big deal, and I truly did like the portability of it, while the big one is big and heavy.
I used the Inside Track Club to get their cheap imitation workmate, and mount a 4' 2x12 on it to support the lathe. It's a surprisingly sturdy arrangement. But, did I say that it's big and heavy?