Providing software for users to host themselves is quite difficult. We do it at Pachyderm. Our software is basically data version control where you can incrementally process data as it's added (or your code changes). This is much more business-y than the use cases you list, but I can describe why it's tough even when your customers are compute platform teams.

Pachyderm started off as a self-hosted offering, but we did try doing a fully-managed hosted solution, because it would be super easy for users to get started. You write some automation to make a perfect install in your preferred cloud environment, and then users click a button and have a perfectly configured, monitored, and backed up environment. If there's a problem, customer support can go play with the underlying resources (always with explicit user consent, of course; we were super strict about this), and that means that users get a solution instead of a fun debugging problem. Less surface for them to configure, less chance for configuration problems. Ultimately, this turned out to be unappealing to users, for many of the reasons that you list. People want to own their own data and not have to trust even the nicest and most competent people with it. For that reason, we doubled down on the self-hosted option.

There are many pain points with self-hosting. From the start, it's difficult to offer something like a free trial. We'll give you the software for free (hey, it's open source), but you're on your own to find a computer to run it on. You'll need to setup DNS. You'll need to get a TLS certificate. For our software, you'll need to run Kubernetes. Before you can even see what value we offer, you have to do a lot of grunt work. What this means is pretty long sales cycles (while we assist you in navigating all those requests through your organization), and a dropoff among people that just want to have their problem solved today. (You can see this yourself. I bet you'll click any link on HN that sounds interesting. But if I just said "download and run this .exe", you'd probably be quite scared, and it would take some convincing. And rightfully so! That's the difference between SaaS and self-hosted, at the earliest possible phase, "do I even want this thing they make?")

From a technical standpoint, the amount of work you have to do increases dramatically. We have to write code to talk to every variant of object storage. We have to provide configuration documentation and advice for 3+ cloud providers. (For example, we used to depend on etcd pretty heavily. etcd's performance is very heavily dependent on disk iops, and the standard storage classes in your average cloud provider's are woefully inadequate. So we have to get people's data from the inadequate storage to one that is fast enough, and that's just a lot of work.) We have to support customers' internal security initiatives. If you thought keeping up with changes to one cloud provider for one instance of your application was a lot of tedium, multiply that by 3 cloud providers, by ARM and AMD64, by 6 ingress controllers, by 5 Kubernetes versions, by Local SSD, networked SSD, spinning rust, by corporate-mandated HTTP proxies, etc. It is a ton of extra work to make software that fits the customer's environment over just making your own environment that is perfect for your software.

Users dictate the upgrade cycle, not software engineers. I have worked on teams where every commit to master just gets deployed to production within 5 minutes. It's awesome. All you can do when you self-host is to make available another version; if users want it, they will upgrade it. Upgrades are always risky, especially when you wait a long time. So every release has to provide value worth the risk, or the same bug from 2 years ago will be reported again and again. Because of the risk, we have to maintain stable and testing release channels. Customers that want to try a new feature in their dev environment can get a nightly build with the latest features, but we still have to maintain a stable branch for customers that are happy with the featureset and just need security and bug fixes. We have to do work to lower their ops risk. This is honestly twice as much work as just pushing master to prod every so often. Backports cleanly apply most of the time, but when they don't, you're just implementing the same fix against slightly different code. It's work. (I'm a "if the tests pass, there are no known bugs in the code" kind of guy, so I've never been worried about deploying master to production. Obviously, some care in your software engineering is required and this doesn't work for many people, but it does work for me with a very good track record of uptime. None of that matters when you self-host, nobody would simply trust my assurance that the new version is good to upgrade to.)

Even ignoring all of that, it's quite the change in how you as a software engineer operate. You can't reach out and touch the system, ever. You ship a build artifact, suggest that the customer install it, and wait for their feedback. If you need certain information to show health/non-health, you have to make that understandable to the operator (who is often different from the user, who has other health concerns). If you need to be able to debug something, you need to add code to your app to do everything that you would think of doing if you were debugging a live environment, and make the results understandable to the user.

The tooling around this is pretty much the wild west. I recently investigated using Bazel to build our software, because we have three parts that are pretty much completely coupled to each other bidirectionally; Go, Typescript, and Python. There just isn't tooling around all of our requirements (we support both AMD64 and ARM deployments, as customers like those cheap ARM instances and developers like M1s without binfmt_misc hacks), because big tech companies that make these tools have the liberty to dictate their requirements. As a result, we spend a lot of time hacking together tools that meet our customers' requirements, but as a startup, don't have the resources to build something perfect, so developer experience honestly isn't as nice as, say, writing an internal app at Google.

Finally, the hiring situation is complicated. Pretty much nobody is building software that users deploy themselves, so it takes a certain rare type of developer to be productive in this environment. Your code has to show its work; operators that didn't write the code will be running it in production, and your software has to lead them to a solution to their problems. (Even things like "my job ran out of memory" are hard to detect and convey to users and operators.) You can't just push a fix you think will work to prod to see if it fixes things; you have to reproduce the customer issue in your own environment and fully test it before your code goes into a release. It's slow going, and care is of the utmost importance. Code review has to be thorough; you need to understand your customers environment and ensure that a fix for one doesn't break some other. (Yes, we have tests for this sort of thing, but great care is the last line of defense against regressions.) It's not just software engineers; marketing types always want to run A/B tests, and customers don't want to be subjected to this sort of thing. We have to make the right decision in advance, we can't give 10% of users an experiment and see if they rage-unsubscribe over the change.

All in all, it's a very different world. If you feel like your job isn't hard enough just running one instance of your software in an environment you control, selling your software for self-hosting is the job for you. It's hard. It's a slog. But, it's great for your users.