> last I knew, a lot of IBM mainframe stuff was running on emulators. Albeit on highly available hardware.

That's rather different from what I've heard: IBM refuses to allow any remotely recent versions of their mainframe operating systems to run in production on anything except their own physical hardware. A lot of hobbyists run ancient OS versions under Hercules – like MVS 3.8J, which has bugfixes up to circa 1985, but feature-wise is stuck in the early-to-mid 1970s – which is legally allowed since IBM intentionally released those super-old versions into the public domain. But, I doubt anybody is still using OS versions that old in production – maybe there's a handful of sites still running 1990s versions, but all of those IBM won't let you legally run on Hercules. Basically the only emulator they allow is their own super-expensive one (zPDT/etc), and only for development use. Hercules can run newer versions fine, but not legally. [0]

So, while almost every other mainframe vendor has moved (or is moving) to a purely software-emulated solution, IBM insists on physical hardware – because they are trying to keep their high-margin mainframe hardware business alive for as long as possible. Eventually, as the install base continues to gradually decline, they'll likely terminate the physical hardware line, and pivot to software emulation as other mainframe vendors have. However, (just like all the other remaining mainframe vendors do), they'll only license the OS to run on their own proprietary emulator, not open source ones like Hercules.

One option some sites have gone with is "application-level" emulation aka mainframe rehosting software – don't emulate the actual mainframe hardware to run the real OS, instead recompile the app using a COBOL (or PL/I or whatever) compiler for Unix/Linux/Windows, combined with an emulator which just emulates the mainframe OS calls (and calls to middleware products such as CICS). This approach is taken by products such as NTT/Dell's UniKix and Oracle's Tuxedo ART. The downside is you need to recompile your app on the new platform, and the emulator only supports a subset of the OS/middleware calls. If you are lucky, and your app is pretty vanilla, it may all just work. If your app relies on obscure or newer OS/middleware features, you may have to rewrite significant portions of it. Also, it doesn't address large parts of the skills gap – you still need to know COBOL and mainframe APIs, but now you also need to understand what subset of those mainframe APIs is actually implemented by the rehosting package – although at least OS-level admin is now mainstream and the exotic stuff is purely application/middleware level

[0] With some super-rare exceptions: e.g. IBM used to allow people to run their operating systems on FSI's FLEX-ES, mostly just for development, but including some low-end production use. From what I've heard, they stopped allowing this for new customers 15+ years ago, but sites who licensed it for low-end prod use back then are allowed to keep on running it, and there are still a small number who do. I've also heard that those customers are stuck on really old mainframe OS versions – although late 1990s/early 2000s rather than the 1970s/1980s of the public domain MVS 3.8J – because IBM refuses to license newer versions to run on FLEX-ES