Have you cleaned up the ends of the blanks before you assembled them ? If not then that's your problem . When you press in the parts the uneven lacquer on the ends of the blank are getting crushed and causing the chips in the ends .
Use a sanding mill to clean them up . You can buy one through a couple of the members or make your own .
Take your barrel trimmer and remove the cutterhead from the pilot shaft and reinstall it upside down . Cut a disk of sticky backed sandpaper (I use 220 grit) the same size as your cutterhead and use a paper punch to make the center hole in the disk , and stick the disk to the flat side of the cutterhead . Use it the same way you use your barrel trimmer .