back in my machinist days we used to use something called a watts drill to make square holes. we even had hex hole drills also. you had a square guide you fastened to the end of the material. the drill, for the square had three cutting edges. operated similar to a wenkle engine. i believe the hex drill had 5 sides but not sure. that was a long time ago, about 20 years. if you go to this web site you can look at one of them and watch a video.
http://www.ohgizmo.com/2009/10/07/square-holes-drilled-with-a-watts-drill-blows-my-mind/
check it out. if i remember correctly it worked pretty well.