> A Wayland-style compositor, on the other hand, seems to be a much higher barrier to entry. ... I don't recall ever seeing "You have to use TWM because AfterStep won't work with your Trident 9440 video card" back in 1998.

All in all, the basics of Wayland are a pretty tight package. https://wayland-book.com/ goes through the pieces, and it's not a super thick read. The system of passing around surfaces is comprehensible, tight, makes sense, and there is very little fluff or barriers here, imo.

Wayland has a common core, but absolutely I'd grant that the various protocols do indeed make it a much less tightly coupled thing, with different compositors having different sets of protocols they support. So yes, some apps that require advanced capabilities run much better in some compositors than others; the compositor choice matters. Sometimes there are multiple competing protocols for the same feature-sets, but usually/historically, wayland-protocols hammers stuff out reasonably quickly & most of this is a matter of time.

Still, this is often easier than the past, where apps would have to each test for extensions & have various fast/regular/fallback codepaths depending on available extensions; not necessarily a hindrance to the window-manager, but a bundle of complexity for everyone else trying to use X11 adequately. The Wayland common primitives, on the other hand, are fairly universally performant & well chosen.

Returning to complexity for window-manager/compositor, the situation is not unlike X11 itself, where yes, a simple window manager (or compositor) is possible to spin up relatively quickly, but where there is a sea of different standards to implement to do a good job. Window manager hints, extended window manager hints, and a plethora of other standards existed around X11 that were up to the window-manager to tackle, and implementing each of those took a lot of time too, if you wanted good support for all apps. Different Wayland compositors also have different support for different protocols, and those are a bit deeper rooted capabilities, less superficial than many of the X11 hints (which, if ignored, were less likely to impede use), but the idea is the same: real support to really be decent took work in X11, and it takes work in Wayland to implement a good suite of protocols here too.

Where I disagree highly is calling out the hardware here. Wayland is closely tied to kernel fundamentals; any reasonably supported video card will perform adequately under any competent/non-specialist compositor today. (Certainly some compositors could demand higher standards, such as some of the experimental compositors requiring Vulkan, but generally compositors have very similar, very common requirements.)

> I wonder if it would have made more sense to go with a paired approach-- a single master compositor implementation, with the complicated and more hardware-sensitive stuff involved, and a pluggable window manager that spoke to it.

I like where we are, where there are various toolkits/libraries for accelerated implementing. Wlroots, which underpins chiefly Sway (the i3 replacement), has given rise to a variety of other compositors, spanning the gamut from quick/fast/experimental to rich/deep/powerful. libwayland still defines some core ideas, if not compositing implementations, to speed development somewhat. Weston is still available as a reference compositor, although yes it's designed (more or less) to be forked & enhanced, not built to be preserved & built (extensibly) on top of. Wlroots & other alternative toolkits fill this need, & provide a diversity of ideas for how we might get going. Projects like Greenfield, the HTML5 compositor (https://github.com/udevbe/greenfield) demonstrate the diversity we get from not having a single common core technology, are possible because of this belief in protocol & standards over implementations, eased though implementations might be from promoting something like Weston to the one-and-only implementation.

> The whole "nVidia works, but only with the GNOME compositor" sort of stuff reads as a sign that there's way too much involved in there.

We can't look at a anti-plays-well-with-others entity like Nvidia to assess what is/isn't a good idea. Nvidia spent nearly a decade stomping their feet & demanding only their way was ok. The fact that Embedded OpenGL itself, what the rock their obstinacy was built around (EGLStreams), is quite heavily on the way out further should stress how foolish & self-centered this vendor has been. This discompatibility indicates nothing, is no sign, except an indicator of what kind of a company Nvidia is/was (one that obstructed any implementations of well known & common kernel constructs).