I have a a very similar background as the OP.

Also seen/worked on projects that are a mess and devs there built hacks upon hacks to keep everything up.

I also felt like I am stupid (besides studying software engineering and working in the web business for 15 years) or I miss something. But then I also had the privilege to work with very skilled people and get a different perspective.

Here comes my argument:

I argue that most projects that went to be a mess have been set up by people that had to work with incomplete information or insufficient time to conquer the stated problem. So I project the question "is-technology-the-problem?" to "is-the-team-the-problem?".

In my experience much bad influence - that create messy projects - comes from the lack or the way of communication between technical designers (who say whats possible), developers (who say how its possible) and other stakeholders (who normally "just want something"). I think of a situation here, where I had to work with a sales guy who proudly sold "rapid development" and "agile iterations" without understanding the technical or management perspective of true rapid development.

When it comes to making design decisions (which WILL influence the "messiness") the web is full of different approaches, technologies and so called "best practices". If you're a designer of a "will-become-big"-project it is difficult to make those decisions based on this variety of solutions (which as the OP stated almost everytime suck after some months). The fact that every framework presents itself as the best and most versatile does not help at all.

My best practice to go out and talk to people about my problem and not ask the internet. Its others people experience you only can access in a discussion, because normally you don't and cannot know what information you lack and therefore ask superficial questions on the web and thus get superficial solutions and have to conquer previously unexpected problems in long-term.

Another argument I want to make:

If you were a carpenter and would have to build the interior of an opera you would not go to the web, google for a month and then think you have exhausted all available resources and now you are ready to build that opera interior.
The ubiquity of "quick solutions" make developers believe there are no long-term-effects of such decisions and thus most such small decisions never get discussed.
If you have worked in teams in other branches (building digital arts, building a house boat, stuff like that, where you MUST have a team) you get the point of what I am saying. Its only computer science where people stop talking because they think everything is clear to everybody.

About your question about orientation:

I suggest getting aware of the high level concepts of full stack web development and truly understand the concerns every layer tries to attack. Web-Development - like every other software engineering discipline - is all about Devide&Conquer.
N-Tier development (one approach to D&C) is nothing new, but the semantics of web development are new. A good designer has to speak those new semantics fluently. It also helps getting a good feeling about what actually is a "best practice" and what is just sold as such. There simply is no universality to web programming as there is none to other software engineering disciplines.

I gained a lot of confidence in knowing about the classification of problems and how you would generally combine them instead of building a big hashmap of problem -> web-tech-framework (This also reminded me of some complexity courses I took in my studies). Giving clear semantics to different parts of your application and discussing those layers using natural language also reduces your teams vulnerability to total chaos and prepares for unexpected changes of requirements.

Further I learned from experience that its often the time-constraint that promotes messy code that never gets cleaned up because the project somewhat dies a slow death, which in my eyes is just a reminder that web-tech is something new to the society as a whole.

Finally:

Thus I developed the following perspective: I see myself as a moderator of language and problem-awareness. I often argue in front of customers that if you describe the project as a map of the problem domain there are A LOT of blank areas and its our teams effort that will unveil/debunk/uncover those blank areas. I also point out that there are a lot of semi-blank-areas on our project map. I for example know how agile development looks like that does NOT generate a hack-upon-hack architecture but the others don't. On the other hand I have no knowledge whatsoever about interiors of operas (to reuse the example from above). Its that explicit merge of information that prevents a projects structure going astray.