Perhaps I am going out on a limb here but I discovered if you WAX the inserts then it's simply wipe away the junk when your done, no buildup.
Seems also there is more buildup on the outside that contacts the chisel/sandpaper than anything and the index point on the centers gets the most abuse it seems (those 2 areas)
If you could drive a 1909 Hudson into an ultra sonic cleaner filled with Krud Kutter, not only would it look like new, the Hudson Motor Company would likely still be in business!
One night after a few too many Coors Lights, I spilled a gallon of krud Kutter all over myself . NOT ONLY was my wife "hitting on me", but the neighborhood women brought me bacon and junk
.