An guy in our local wood shop that sales our pen supplies told me to watch squeezing the blank to tight in the vise because some material will compress and once drilled expend to make a larger hole. Don't know if it is true but makes some since.
I had this issue some time back... but their was a combination of things that were going wrong to create the perfect storm. I had my drill speed too fast, not clearing waste, and too much of a squeeze on it.
And here's why it was the perfect storm... going too high of RPM and not clearing the bit generated heat, heat made the acrylic more pliable, more pliable made for a larger oblong hole around halfway through the blank to the bottom of the hole.
I have solved this issue by using just enough squeeze to hold the blank, slowing down the drill and "dynamically" clearing the bit...I have an air nozzle directed down the drill bit while I am drilling. It keeps it clear and cool and is just one smooth motion.... oh ya, keep your bits sharp, wear safety glasses and be prepared for the mess!!