Agreed. I've made a not-altogether-good reputation for myself by walking out of a number of bands over the last 10 years, either because our 'vision' didn't align or because they proved to be just not good enough (IMHO, of course).
Some of my current bandmates are surprised at the way I do this, perhaps even think I'm a bit of a loose cannon.
Me? I've never lost a moment's sleep about it. I walked away from a very well-paid job four years early, not because I don't like money but because I was well aware that those four years were (at that moment) the best remaining years of my life.
The same equation still applies. I'll be 69 next month (no, I can't believe it either) and I have no interest in allowing anyone or anything to waste the best remaining years of my life right now.
Use them or lose them.