I'm not telling you you're wrong; simply stating a fact. You are looking at it the wrong way.

Projects/solutions often don't make sense unless you look at them from the perspective of the implementers; and like it or not, there is nothing wrong with the reversal of client and server in reference to X. The rendering to be done requires that your particular local hardware be controlled and orchestrated by X aware software. It makes no sense for the remote machine to in any way be burdened with the implementation details of every prospective display set up it could potentially have to talk to. Therefore, the better approach is to define an abstract API from which the remote machine can just speak "X" and leave the details up to the server on the other end to display. Thus, the reversal of Client and Server makes sense. The Server translates your local interactions and display state into "X" protocol messages while the remote client tracks and translates the specifics of the raw display interactions into the implementation details specific to how what is displayed on your local machine interacts with the system present on the remote machine.

You're getting confused by Client and Server meaning something in particular, when that isn't really the case. X isn't doing just one thing. The X Server provides the service allowing the client to draw on your hardware. The remote Client is also a Server in that it provides an interface through which actions that take place through your local system are translated into useful functionality on the remote system; however that is a fundamentally different perspective from which to view what X does than through the lens of being a rendering mechanism alone. X does both things. As a rendering technology, the "X client" is remote, the "X server" is local. As a binding between I/O events and program functionality, you can absolutely view the "X server" running on your machine as a client behaviorwise of the server that is the "X client" on the remote machine; but it is still a server in the sense of being aware what your specific hardware setup is, and how to handle it, because the "X client" on the remote is blissfully unaware, and is only there to translate what your "X server" reports to it into meaningful state changes on the remote host. Client/Server is not an atomic concept in computing. You can be both depending on how you are looking at the system. It's a bit like a duck-rabbit.

Is it complicated? Yes. Why? Because something that powerful is pretty much guaranteed to be so, and because we try to apply terminology that is generally used in a much more straightforward manner elsewhere to a system that taken as a whole adopts a bidirectional client/server relationship with regards to different subsets of the functionality that the software is intended to enable; namely, to allow graphical interfaces to be utilized between machines with wildly different characteristics, without burdening the "source" machine (the remote machine) with the responsibility of driving our (the local machine's) hardware.

Again, There is nothing wrong with the approach except that yes, at first blush, it feels unpleasant to wrap your head around terminology-wise because the conventional way the terms are used exists in both directions in the same software. I don't expect most to grok X the first time. You just have to dig in and get uncomfortably familiar with the hell that is the rat's nest of abstractions that makes remote graphical computing possible.