This is one of those things that can make you throw a computer monitor if you don't know about it. Let us say you have a Category and it has a material override (for this example I am overriding a category within the "Object Styles" dialogue). Or, there is an existing category with an material override you want to remove.
Where the heck is the "Remove Material" button.... Look closer....
Well, I'll be damned... There it is!!
Now, take a look around some other overrides like the material override for phases... Same little button!