I wear clothes that fit (for the most part, my jeans are a bit tight at the waist to be fair) but in all honesty, I have more important things to worry about than my appearance.
There's something about seeing bands go on stage all dressed in clothes that they aren't 100% comfortable in and dressed the exact same as the other members that I find a little embarrassing and off-putting. I only ever go see bands that I like now so it may be a part of that but I think if the music is good then why should I give a toss whether the guitarists jeans match the singers t-shirt?