I'm guessing that the real issue is likely a combination of the larger hole being drilled, and how the remaining material is supported. As you drill the hole, there is stress being put on the material, trying to twist it in the direction of drilling by the drill bit. At the same time, most blank vises or lathe chucks hold 2 corners of the blanks, effectively leaving 2 corners unsupported, but more importantly, relatively long side profiles of the blank also unsupported, and if the vice/chuck is tightened on 2 of the corners, you are squeezing those sides. Look at the blank from the end after you drill a hole and notice the thin part in the center of the sides of the blank where the hole comes closest to the edge. Think of what stress squeezing two opposite corners causes along those sides and especially at that very thin part. It can quite easily cause the blank to distort a bit, making your nice round hole a bit oval. Add that to the stress mentioned from the drill bit trying to pull the blank in the direction it is turning, and with a hard, yet brittle material like tru-stone, these many stresses can cause problems...
In the end, everyone's general drilling suggestions are key, the VERY slow drilling speed is a HUGE help in reducing the largest source of stress. VERY sharp bits is another big help, and advancing the bit VERY slowly goes another step. But one thing that can be often overlooked, especially when drilling larger holes, is not to crank down the vice/chuck too tight, once the hole is drilled, that is going to squeeze the remaining material. It is important to have the blank held firmly but not squeezed too much. The vice ONLY needs to keep the blank from turning or sliding, anything else is adding to the circus of stress...