I admit I was never in this exact situation, so I'm not sure if what i have to offer can help, but I can relate to the feeling of "something being outside of my capability".

What I don't understand from your post is what your end goal is? Why do you want to learn low-level stuff like building compilers or operating systems? Is it for work? Desire to improve yourself? What can help you better depends on the answer to this question.

I can offer a couple of pointers I hope they can help:

1. I was frustrated by my lack of understating of low-level concepts and how it prevented me from understanding the more complex pattern. The thing that clicked from me was building a small CPU and related hardware (timer, VGA frame-buffer, etc...), I don't remember which site/guide I follow, but today ben eater videos or nand2tetris seems good choices.

I also read a lot about game console programming (GameBoy color in particular), a lot of that low-level concepts were really a lot simpler to grasp with the visual demo to demonstrate. I'm also constantly surprised how much concepts like interrupt, how a mmu works, etc... are useful even when writing high-level code like I'm doing now.

2. Stop considering Databases, compilers, and operating systems as a single unit to learn. They are made of a lot of smaller and simpler components that together form a system. And while when are together seem very complex, often when divided into smaller part became a lot easier to learn even if the total "complexity" remains the same.
Take compiler, for example, there is code generation, optimization, and syntax parsing. Instead of trying to build or understand a compiler start by learning the basics of parsing, for example by building a simple chatbot (one of the first things I wrote was an IRC bot), evolve it to understand more complex queries (for example SQL).

Related to this: One thing I could not understand was how big relational databases, like MySQL, worked, a lot of the literature on the topic seem to have a perspective that didn't help me (like the math behind it). The ah-ha moment to me was with sqlite and seeing that at the base was a simple key-value store, and everything else was built on top of this. One takeaway, for me, was that when you feel that you don't understand something, maybe you are asking the wrong question! in my case I keep searching "how relational databases work" instead of what I was really interested in "how relational databases are implemented".

3. I would not start with a thing like "the dragon book", from what I see is an extremely complete book about "everything you need to know about compiler" (hasn't read it, so correct me if I'm wrong). Book like that can seem intimidating and you risk of reading a lot of material without really learning and I feel more like an "information dump". Search for smaller book or course that are an "introduction" to compiler instead of the "everything you need to know", they may not teach you everything, but you will have a better understating of "core" concepts that allow you to tackle more complex topics a lot easier.

Side note: in the specific case of compiler search for a book/course that focuses on building an actual small language! The book my professor gave to me only had a disjointed example for each topic. Something like "crafting interpreter" with a precise goal is MUCH easier to understand while teaching the same concept.

Hope this can help, or at least a starting point for a discussion.

edit: formatting