Yeah, that's true. Wayland is, of course, just a protocol (and some C libraries.) You can always implement Wayland compositing under a different display server or compositor, and the best example of that I am aware of is Arcan FE with its Wayland bridge.

Even core functions of the Wayland server that are in the same process as the compositor may be split out to common libraries, and of course there are several compositor server libraries for different programming languages that provide the vast majority of the base functionality of a display server implementing the Wayland protocol, notably wlroots, Qt's own Wayland libraries, and Smithay.

Wayland compositors generally also inherit large parts of what used to be the X server. Over the past couple of decades, the Linux graphics stack has massively expanded and refactored. X used to manage it all: modesetting, buffers, contexts; the graphics drivers were essentially XFree86/X.org drivers. Today, almost all of that has moved to the kernel and Mesa, in the process giving very powerful new primitives as well. It definitely came with a lot of pain, but it improved both what X.org was and what Wayland can become in the future.

Likewise for the input stack: Linux evdev expanded and improved a lot and unified the format of event data from devices, but of course drivers were still needed to handle smoothing out the differences between devices. libinput is probably not everyone's favorite, but I'd say it does a pretty damn good job considering how broad its device support is.

Pipewire is not strictly needed, but it is the defacto way to do screen capture on Wayland thanks to desktop portals. I think this is mostly because of GNOME/Mutter. The Mutter project is very cognizant about features and dependencies being put into the Mutter process itself, and seem to favor moving things out of the compositor process. Their approach to doing this in general seems to be by moving things into DBus servers, like the Desktop Portals system.

Personally, I have some very mixed feelings about this situation, as could probably be seen based on my past interactions. DBus servers can be an awkward fit for some of this functionality; desktop portals are nominally executed as systemd user services, which as far as I know are typically per-user and not per-session, which would make e.g. nested Wayland sessions trickier.

It would, of course, be possible for the Wayland protocols to handle things like this without it explicitly being "in-process", which I personally would prefer; I would prefer if the API doesn't have to map to the underlying structure of the desktop environment. I can definitely see how this is an additional burden and additional point of failure though, so I can't really claim to not understand it at all.

It also has allowed Wayland, for better or worse, to side-step the issue of authorization. Instead, it can be handled using DBus, polkit, etc. Wayland is nominally a capabilities-based system, but with the caveat that it doesn't actually have a way to authorize or authenticate anything. As far as I know, there's no protocol to e.g. "ask permission", you just either get an object or you don't.

All in all, the practical architecture of Wayland clients is... unique. I think it's pretty clear to anyone that it's a bit of a mess. Honestly, though, I am pretty happy for the groundwork that enables Wayland, as I think it is starting to bear fruit on things where there's just no way incremental improvements were going to get there. Definitely hope that some of the mess can be cleaned up in the future so that users (even power users) don't have to worry about understanding this architecture to do practical day-to-day work.