Don't know if you are doing it this way or not, but drill through the web in the base of the case and out through the primer pocket FROM THE INSIDE, IE, slide the drill bit in through the neck of the cartridge and then drill out the back. Reason is that the flash hole that the flame from the primer enters the casing from is an integral part of the casing and is usually dead centered, which helps center the bit. If you try to drill into the spent primer from the back, the dent from the firing pen is usually off center and the primer itself is slightly domed, and not always level. this makes it difficult for the drill to start it's cut centered.
If you are using a drill bit larger than the ID of the casing mouth, try to find a way to remove the spent primer, then drill from the back. Not as good as from the inside, but better than trying to start the drill on the primer itself.