> I’ve been a SW dev for 10+ years now and know very little about WP

WP has become a gigantic ecosystem with its own sub-ecosystems and specializations that pertain to those ecosystems at this point. Seeing it as a 'Linux ecosystem, but as a web app' would be a better way to understand the landscape. Therefore, you should definitely try to learn more about WP because:

> But it has always seemed crazy to me how everyone in the “WP space” seems to be making changes directly in prod o_O

...even starting from the need to understand that, there are a lot of things to understand. People make changes directly to prod because making WP high available is very difficult due to the HA-enabled infra that is out today not supporting stateful workloads well, hence its difficult to have dev, staging, prod deployments etc. Especially because a change in plugin code or setting (an update, changing a setting etc) may trigger database changes and one wont work without the other.

And when you end up setting up one, it comes out way too expensive for any individual or small business to afford paying for it. As opposed to a managed, non-HA WP hosting provider allowing you to host a WP site that handles million or more visitors every month for $45 as opposed to having to pay $200-300 for a HA setup so that they wont make changes in production and they will have copies of their site running. Which, they can also do by buying another $45/month hosting and duplicating their site behind an IP-gating etc by the way.

WP site owners use plugins to make happen what they want to make happen, and a lot of these plugins need to be able to modify files in the filesystem to be able to work their magic. This ranges from being able to resize thumbnails to creating sitemaps to more complicated ecommerce application needs. Some need to modify .htaccess in order for a major feature to work. Others may need to modify the config file to add something etc.

And at that moment the whole containerization, docker, deployments, multiple environments thing hit the wall: If all the plugins are built into the container, they wont be able to modify the filesystem, especially their own files when needed. Which is a game breaker for any average WP site owner.

You could use an NFS to mount the wp-content folder, and that would fix a LOT of your issues. However, cheap, reliable managed NFS offerings are extremely scarce. And what's there is expensive. So you will end up having to rack up the price to a point where most people wont want to pay.

Even when you solve that, you still have problems - you are now building the WP core into the containers, and therefore are able to deploy automatic WP version updates and also having improved performance, great. But what will happen when you deploy a WP version update that breaks various important plugins because they are not compatible with that version yet? Boom. Upset users. A broken plugin and feature is most of the time is as bad as your entire site being broken.

You could prevent users from installing plugins and you could offer a selection of plugins that the users can use. Assuming that you solved the NFS problem. Then you could avoid version conflicts, compatibility issues. But then many users wont use your service because they cant install the plugins they want. You could have a market like that and a lot of web agencies do that apparently, but they are for their clients with specific needs who wouldnt need anything more freeform and complicated like most of the WP users do.

...

The best fit on the market for solving all of these is Google Kubernetes Engine + Google Filestore + Google Cloud SQL. (people complain about AWS nfs offering having spotty performance - bad for WP). But Google egress charges literally kill any chances of Google having any foothold in the WP ecosystem because with their outrageous egress pricing, a small WP site owner that hosts a middling WP site for $25/month would have to pay another $15/month for egress traffic costs on average, and that makes google double or triple more expensive for out-of-the-box WP compatible web hosts right from the start. And that's a variable price too - no WP site owner would want to pay a surprise $100 in a random month because their site has had more traffic suddenly. Standard WP hosts can provide everything from cpu/db to traffic costs even for a million visitors for $50/month fixed.

So anyone who builds any WP SaaS or consumer grade hosting at Google infra would have to start with 3-4 times more expensive compared to all other offerings when they add their own profit margin.

If you end up choosing to not allow users install plugins and themes, using WP multisite could be much better for your setup - it acts as a singular WP installation that can host an infinite amount of WP sites inside itself, with various plugin bundles activated/deactivated for different packages. You can build this into a container, and if you don't have any plugin that needs to use the filesystem, you can even deploy it in K8 in front of a managed DB and with the media uploads sent to some S3 compatible storage or something. (That requires custom code).

Also - nginx is not a good choice for WP. A large swath of WP plugins require .htaccess to be present and accessible for adding custom rules for their features, and this makes Apache the better candidate for such a situation. Apache with event mpm and php-fpm behind it is as performant as nginx these days, so you can just slap Apache and not ever have to deal with the rewrite rules that your customers will be asking you to add to nginx rewrites.