I used an old plugin by Matt666 that removes all materials that have been applied to the Group/Component as a whole rather than to the faces.
https://sketchucation.com/pluginstore?pln=Remove_CG_mat