As with many other concepts, you can create your own virtual machine or whathaveyou (in the sense of "C virtual machine" or "JVM", not a full stand-alone VM necessarily) that can implement whatever you like, but every time you have to reach outside of that, you end up back in the old world, and that limits your ability to truly enforce whatever new security thing you want to enforce. e.g., you can wrap all access to external executables in your code all you like, but once you allow access to bash or Python or something, your new capabilities-based security system is now gone and you're back in the world of Unix.

In a true top-to-bottom capabilities based system, that would not be true. You could hand out access to a shell along with a certain set of capabilities and be much less concerned about what will happen in that shell. Hypothetically, implemented correctly, with CPU & RAM-based capabilities, you could safely hand out shell access to random people on the internet and be sure they won't do anything wrong. That is not the case today, if you just have your own little VM world.

If you try to extend your VM, you find yourself re-implementing more and more of the world. Creating even a simple GUI, for instance, is a huuuuuge undertaking for a very small group of interested people. The baseline of expected functionality in a new language environment is going up every year. It's hard to get over the initial hump.

On the flip side, if you do it from the OS level, and you truly implement this new model and don't have a generic "give up and just go back to UNIX/Windows/etc" callout, you have the problem that while you have an OS, you can't do anything in it, no matter how much better it may theoretically be. You can't get anyone to work in it, because there's no value there. See Plan 9 for an example of how this goes down.

Getting something like a capabilities-based OS going is basically "rebooting the computing world" because it's not just a matter of getting "an OS", it's a matter of getting an OS, a windowing environment with some usable GUI, a browser capable of browsing the "real" web and not just some 1995-esque subset, some kind of terminal, a whackload of libraries for programmers to use, a whole bunch of apps I'm not even thinking of here, and more before anyone will even give you the time of day in the real world, and without making it out to the "real world" all your security is pointless. It's very difficult to create a business plan where that makes any sort of sense; no matter how glorious the payoff may be in 20-30 years, simple time-based discounting of value makes it very difficult to be rational to commit the vast amounts of effort it would take today to start getting there. Those research projects are great, and I value them, and wouldn't ask them to stop, but they are far, far less than 1% of what would be required to make them truly useful. Nobody has crossed that point in the last 20 years, even without trying to bring a novel security mechanism to everybody. The closest thing to a new OS we've gotten are the mobile OSes which are still fundamentally just competently-managed UNIX systems under the hood. What capabilities appear to be there are still just bashed on top of the UNIX permissions model, not the true, granular capabilities of those research projects.