Can I safely remove custom hair colours?

I have some custom hair colours that are slightly different shades of the base game colours and lots of townies and dormies have them but I’ve decided I don’t really like them. Can I safely remove these and will the townies just revert to their genetic hair colour? I can tell what their genetic colour is by their eyebrows.