People understand Erlang scaling wrong. Erlang’s distribution model, and its apps like Mnesia, were never designed for horizontal shared-nothing scaling. They were designed for SOA: taking the components or operational roles of a system (e.g. “master” vs “hot standby”) and codifying them into distinct Erlang nodes (which can be developed as separate POSIX processes on the same machine, before being moved to prod to live on separate machines.)

People try to scale Erlang systems by trying to get one distribution set to have thousands of nodes; and then taking that thousands-of-nodes cluster, calling that the “master” cluster, and then having another thousands-of-nodes cluster and calling that the “hot standby” cluster. But that’s precisely backwards.

Think like Ericsson. You’re designing a switch. You have two independent problems:

1. How do you make your switch Highly Available / fault-tolerant?

2. How do you handle higher load than one switch could possibly handle (for either hardware or software reasons)?

To solve problem 1, you create nodes, with roles (e.g. a master and hot standby), where there are an exact number of pre-defined nodes, each with exact static roles. These nodes have defined fault-tolerance relationships. Databases like Mnesia that replicate from one node to another. Applications that have defined chain-of-command such that it’s clear which node becomes the leader if the first leader fails. Etc. You design your system so that some of these nodes can go down, or netsplit, without the system itself going down/crashing. And then you deploy these nodes to separate machines, so that hardware failures will end up being treated the same as node failures.

To solve problem 2, you stamp out copies of this entire system of nodes, and add more infrastructure external to it to route between them.

In this model, a system of nodes — a distribution-set — is the same type of thing as a Kubernetes Pod. It’s several static processes, one of each role, that inter-depend, and are “wired” to one-another; and then get managed together / treated as one workload. The big difference is that, in contrast to K8s pods that all must get scheduled to one host together, Erlang distribution-sets usually get “scheduled” across several machines. But they still have one identity. They’re a single abstract machine that happens to consist of workloads running on multiple physical machines; where those workloads going down is a designed-for and tolerated aspect of the abstract machine they compose.

Everything that happens inside that single abstract machine, is the purview of Erlang and Erlang’s stdlib. Application failover, Mnesia, the thin-client code_server: they’re all designed to make a single abstract machine out of a defined set of Erlang nodes where each node has a static role.

Everything that happens outside of the abstract machine, is the purview of user code. Want to horizontally scale the abstract machine? Don’t use the distribution protocol; it’s not tuned for that. Use a userspace library. Have an application that does abstract-machine-to-abstract-machine peering. Make it the role of one or more of your nodes to run that application. Make it fault-tolerant, too.

Under this model, I hope you can see that statements like this are mostly incoherent:

> Erlang's not the slowest language, but it's not very fast, and using something 5-10x faster can "scale" nicely too by needing that many fewer systems in the first place.

...because an Erlang system architecture that builds one Highly Available abstract machine out of five-to-ten nodes, cannot be swapped out for a system architecture composed of one node, no matter the language it’s written in.

The other system, to implement the same fault-tolerance strategies, will need exactly the same number of nodes, because each node performs one or more roles, and serves as a VM-level failure-kernel / bulkhead for HA within those roles.

The non-Erlang-implemented system will just have a harder time implementing such strategies, because it’ll mostly have to use userspace libraries/frameworks — rather than VM-level abstractions — to do so. (Though you can get close with languages that have basic syntax that compiles down to framework code in other languages, e.g. if there were a hypothetical “Erleans” that compiled down to C# + .NET Orleans framework calls.)

——————

Of course, none of this matters if you don’t have the problem that Erlang solves. If you don’t need a stateful abstract machine that keeps itself online and computing over that state at all costs with fancy multi-node footwork, then you really don’t need any of Erlang’s distribution stuff.

But there are still, today, problems that do put you in that position. Stateful packet switching is still a thing. Game servers are another good example. App-layer cache servers (Redis, memcached, etc.) where your architecture couldn’t survive the thundering-herd that would happen if you needed to re-warm the cache from a cold start. Etc.

In those contexts, do you really want to try to jury-rig a solution using e.g. Hazelcast + IPVS failover? Better to just use Erlang. Because it’s still ahead of its time, for this problem domain.