> You can't do things like share a page cache and dedupe inodes and dentries AFAIK with MicroVM

I work on several large Kubernetes clusters each running diverse jobs and this would be a huge BENEFIT. The Linux virtual memory (VM) subsystem regularly locks under heavy contention (e.g. one process performing a ridiculous amount of local, buffered I/O, dirtying a ton of pages, while another process struggles to page back in its executable code), leaving random processes frozen for seconds or minutes. (Default frozen lock failsafe triggers after 120s, but when things are really bad I've seen processes frozen for much longer.) It's a persistent source of alerts and complaints from people. It was much worse last year (2019) owing to some kernel regressions, but it's beginning to creep back up as the clusters see more production use so I'm not looking forward to 2021.

Kubernetes pods rarely need to share filesystems between themselves--at least not for heavy usage--and if they do it's usually via NFS or something similar. So MicroVMs really only leave you with the fixed cost of duplicating kernel data structures, which given the amount of memory on most nodes isn't that much. And there's performance potential here precisely because processes in different VMs aren't contending for as many of the same locks, etc. The tradeoffs very much mirror those of multi-process vs multi-threaded and monolithic vs microservice architectures.

Realistically, though, I can see some performance degradations due to the indirection needed for accessing local storage, except where you can make use of hardware passthrough. But I'll take a few percentage losses in runtime costs over frozen processes.

Linux just isn't as robust with these heavy, multi-tenant environments. That's what operating systems like Solaris excelled at. It's kind of ironic, but not really because that's typically how these things play out, unfortunately :( People are attracted by the ease and low-cost of running popular Linux-based solutions, but then try to bend them to run the kinds of workloads and architectures the older systems were designed and optimized for. MicroVM architectures are better suited to Linux' strong points.

And that's all before we ever consider security. User namespaces are just a band-aid over the problem of Linux security. The real issue is the endless parade of kernel exploits. That's where seccomp comes in. But expecting SREs or even developers to properly seccomp-jail all the myriad containers deployed--homegrown and especially third-party--is not realistic. The whole conceit of containers is to lower the barrier of entry to writing and deploying applications at scale while pretending we can still protect people from themselves. If people couldn't figure out how to run multi-tenant before containers (that is, leverage traditional APIs, process model, and networking stack), how could we ever expect them to do any better with even more complex infrastructure? We already tried all that with SELinux. seccomp is better than SELinux because it puts the application developer in control, who is best positioned to know when, where, and how privileges are needed (assuming they know at all). Moving that work back to, effectively, the system administrator isn't going to turn out any better than SELinux did, or present-day container security for that matter.