Received this reply from CDC re my query. Gives me some comfort.
CDC Public Inquiry [inquiry@cdc.gov]
In general, there's very little risk associated with deer antlers or dust generated by cutting or sanding them.
However, that being said, there are couple of exceptions to note:
1) If the individual is removing antlers from a deer harvested in a Chronic Wasting Disease endemic region, they should take reasonable precautions to avoid or limit contact with brain material when cutting through the skull plate.
2) If the deer came from a region where anthrax outbreaks among cattle and wildlife are common, and the deer was found dead (as opposed to harvested during a hunting season), the carcass should not be moved and antlers should not be removed or salvaged.