I hear you. Listen, I'm a jazz fan, along with metal, dub, funk, Britpop (😉) etc etc... a vast array of different genres.... I'm a music fan, like most of us are. I love fusion.... Mahavishnu Orchestra, Weather Report, Jamiroquai, (early) Level 42 etc... some amazing stuff there.
I also love art and poetry. However, there are, for example some works of art and poetry that make me work too hard to understand. I want to understand, but no matter how hard I work, I don't get it. There's nothing for me, intellectually, to grab on to. It makes me feel like I'm too stupid to get it, and gets my back up.
That's how I feel about this music.
In my opinion, art should make you feel good and make you want to go back for more. For me, this art makes me feel like a Cy Twombly painting. Confused and irritated.