There is a specialty tool called a buff rake that is designed just for this, and guess what? It works! Imagine that. Most places sell it for $15 plus. I got just the metal part from
www.jantzsupply.com years ago, put a wooden handle on it, and still use it frequently. They still have it for under $7 You can eat a buff down with it as far as you want to go. It's incredibly messy though, and the fluff sticks to you and drives you nuts, especially if you have a beard.
If a buff is contaminated too deeply, it's best just to get a new one. I got like a dozen really nice quality flannel ones a few years ago off ebay for like $2.50 each plus shipping, and still have 3 or 4 new ones left. They are consumables folks, just like sandpaper. They do last longer, but not forever. If you buy in bulk when you find them reasonably priced, you can afford to get rid of them when they start to get too dirty to clean, or too small to use from cleaning.
Just checked, price seems to have gone up a bit, or the bargains are gone at the moment, but a little watching will pay off I'm sure.