Composer/musician here. I think one of the reasons, as other comments have pointed out, that even designers sometimes (often?) suck at UX is because UX and the constraints of Capitalism don't mix, much in the same way that art in general and Capitalism don't mix.

UX is generalized optimization at the intersection of multiple specific needs. The software has to work, the appearance/disappearance of features needs to make sense within a workflow or many workflows (but probably without too heavy-handedly forcing a specific workflow). Features need to be discoverable, the UI needs to be information-dense without being cluttered. The application of the application needs to feel inseparable from the app's design. It's a tough problem, and unfortunately it takes a lot of time, trial and error, iteration, and refinement -- all things that Capitalism tends to not like, especially w/r/t things that don't clearly demonstrate a contribution to the bottom line.

If engineering is applied science, then UX is a form of applied art, just as music-as-a-job is. "Art" in this case meaning "something that requires a novel approach for each separate problem and can't really have too many rules-of-thumb or best practices applied to it without strangling the thing in the process."

The science -> engineering conversion, while fraught with its own problems, works because best practices are more readily able to be applied to logical problems. Optimization in this case is identical to Capitalism's goal -- to make more money in less time while spending less money -- so any and all optimizations in the realm of engineering tend to be met with open arms and a corresponding fatter bottom line.

Capitalism -- or rather, those that tend to run Capitalism -- expect that everything is like this. But applied art does not follow the same rules as applied science, one reason being that what works today may not work tomorrow. Styles change, novelty (something highly optimization-resistant) tends to sell. Every app needs a bespoke approach in order for its function to be optimally eXperienced by the largest audience of users possible. UX being a kind of meta-optimization means that, like composing and producing music, you're often discovering the thing in the sea of not-that-things that is the act of creation, sort of like sculpting. Every time I try to optimize the music itself or the process of making it (which is notably separate from the tools that I use to make it -- these should very definitely be optimized), I end up not reaching my goal of making good, impactful, novel music. Instead, I get what always happens when you try to optimize art: generic, uninspired shit.

Whereas every time I allow the process to be what it by definition is -- obtuse, messy, non-linear -- and stumble around in the dark, slowly discovering the "solution" (analogous to "the right approach" or something like that, in the case of music), I end up with a whole that is greater than the sum of its parts, full of thematic and conceptual connective tissue that is as invisible as it is essential.

Unfortunately, deadlines tend to disagree with the ivory tower approach.

This same phenomenon is my theory as to why so many apps break the "prime directives" of UX as outlined above -- they're somehow not information dense while still cluttered, seemingly essential features are non-obvious or non-discoverable, one way of using an app becomes preferred by the app's own internal logic (so, just in the same way that "the medium is the message," "the app is the medium" -- something that can really hamstring creative software for me), etc. On the design front, this tends to result in wasted space in the name of "minimalism," unimpactful and unopinionated color/type choices, and so on. The Windows 10 effect, in a nutshell.

You can't optimize design and experience like you can a supply chain or a pipeline. You just can't.

“I didn't have time to write a short letter, so I wrote a long one instead.” - Mark Twain