With only one exception, who was/is a total dick, in the same 35 years as the OP I've played with very good drummers who have been very normal people.
The good thing for me is I don't play with beginners, guys who have little experience or anyone playing as a hobby. All the guys I know have been around the block, know what to do and how to do it and most are making a living out of playing drums.
The rule is that you work on recommendation and anyone who doesn't match up doesn't get asked.
On the other hand I've know far more flaky guitarists and singers!