It's nice to have the drill press level, but the only thing that really is important is to have the drill bit at 90 degrees to the base of the table in both planes. To get this relationship, you will need to check with an accurate square, such as an engineers square, or a Starrett combo square. Yoor drill press probably has a table that will tilt from side to side, check to see that is perpendicular to the bit. To ensure the table is true to the bit front to back, the table must be locked after adjusting the table in the up and down direction. If it's not locked tight, it will tend to tilt down, which will affect your drilling accuracy.
As for technique, take very tiny bites with the bit as you are starting. a heavy hand will force the bit to wander as it enters the wood. I make about three or four initial very light "pecks" to get the bit started accurately, then proceed from there with a light touch, taking only a quarter to half inch at a time, clearing the chips in between each advance.
Other possible sources of your problem could be that your improvised vise may not be true, thus it may not hold the blank perpendiular to the bit. If your backer board has any warp or cup, that could have the same effect.
I tried the wooden clamp at first, realized that it was not very accurate, and buit this for very little cost:
http://www.penturners.org/forum/topic.asp?TOPIC_ID=31715
Dan