I'm an enduser and I expect things to "just work". That seems to match a lot with Red Hat's Linux distros (i.e. RHEL): "Just use it, it's a defacto standard so most third party software is tested on it, don't worry about underlying components as Red Hat has made sane choices for you."

That's the theory (and Red Hat's marketing). I wish I could just be using CentOS/RHEL and not worry about the choices Red Hat makes. But the shit is piling up (sorry for my harsh words, no offence intended).

So, let's containerise something simple on top of the "centos" container image, let's say postfix. Oh, postfix on centos needs systemd for logging. But systemd in a container is a nightmare. You can get systemd to work in a container if the host system is using containerd and you pass certain special files from the host to the container. So just use Podman instead of Docker, right? Podman has functionality to make systemd in a container work. OK, I can run my container on my fat workstation (a Centos system with Podman). But is it still portable? Can I just run it in Github Actions, AWS ECS, a Windows machine with Docker Desktop? Nope. It's not portable.

Let's put the systemd rant aside (I really like the systemd CLI UX, but the architecture (dbus, etc) seems to only serve one usecase properly (Desktop systems) and seems to be heavily wrong for the container usecase).

Let's rant about the architectural choices Red Hat is making for containers.
Before Red Hat started to get involved there was the open source, (now) cross-platform containerd that a lot of tools are building on top. It consumes the low level runtime runc and both provides an API for Kubernetes (CRI) and additional things. It is highly pluggable and therefore the "only container runtime you'll ever need". And it's purely community governed (CNCF).

What is Red Hat doing? Are they building their container tools on top of containerd? Nope. They create their own low level container runtime (crun), their own mid level container runtime (cri-o). But cri-o only covers what's needed for Kubernetes. So to be able to build container images, etc (the "working with containers on a single machine" usecase) additional tools are needed (podman, buildah, skopeo) and they have to implement the missing functionality themselves. So Red Hat's Open Shift (Kubernetes distribution) builds on an entirely different stack than Podman.

Fragmentation everywhere. Was that really necessary?

Why should I care as an enduser? Well, I can inspect and debug all tools that use containerd the same way (i.e. using the "ctr" tool). I can relatively easily even write my own tools that use containerd (via it's grpc api) and can access the resources (containers, images, etc) that these tools are managing to solve my special custom needs.

For the Red Hat stuff everything is different. I could work with cri-o using crictl and write custom tools using cri interface. For Podman I would need to use Podman's API. (Which has annoyingly entirely changed between v1 and v2).

ok, enough rant (I could go on with how hard it is to get an up to date version of podman on a supported RHEL system even with app streams, that UBI images don't provide podman/buildah/skopeo, etc etc etc).

Please don't mistake me: Podman, cri-o, crun, open shift are all amazing technologies. And it's great that they are mostly open source via upstream projects. But I wish this whole fragmentation hadn't happened. For me as an enduser it's nightmare.