Are you trying to custom code the Colors in your CSS? If so, you can still do this without using the Customizer….I do it all the time by just editing the style.css file directly. If you don’t use the Customizer/Color setting then it shouldn’t affect anything.
Let me know if you’re referring to something different.