What is a Distributed Systems Engineer?
Tags: distributedsystem, distributedsystemsengineer, datastructures, career
Published: 2019-11-28 16:36:20

In the last few months at [Ably](https://www.ably.io/) we’ve spoken with hundreds of candidates for our [Lead Distributed Systems Engineer](https://jobs.ably.io/o/lead-distributed-systems-engineer) and [Distributed Systems Engineering](https://jobs.ably.io/o/distributed-systems-engineer-2) roles. We’ve been surprised by how varied each candidate’s knowledge has been. It got us wondering if the challenge in finding the right people is that there is no clear definition of what skills are required to excel in this role.

Give that we've been working on our [distributed realtime messaging platform](https://www.ably.io/), [global cloud network](https://www.ably.io/network), and [realtime APIs](https://www.ably.io/solutions/realtime-apis) for [over six years](https://www.ably.io/about#what), we think we're qualified enough to take a stab at defining what a distributed systems engineer needs.

If you want to [become a distributed systems engineer](https://jobs.ably.io/o/distributed-systems-engineer-2), believe you are one, or want to recruit one for your team then here’s Ably's opinionated guide on the concepts you should have a thorough understanding of.

**The concepts a Distributed Systems Engineer needs to know**

1. [Microservices or SoA is not a distributed system](https://www.ably.io/blog/what-is-a-distributed-systems-engineer/#microservicesorSoAisnotadistributed-system)
2. [Understanding hash rings is a pre-requisite](https://www.ably.io/blog/what-is-a-distributed-systems-engineer/#understandinghashringsisaprerequisite)
3. [Gossip protocols and consensus algorithms underpin everything](https://www.ably.io/blog/what-is-a-distributed-systems-engineer/#gossipprotocolsandconsensusalgorithmsunderpineverything)
4. [Eventually consistent data types and read/write consistencies](https://www.ably.io/blog/what-is-a-distributed-systems-engineer/#eventuallyconsistentdatatypesandreadandwriteconsistencies)
5. [Deep understanding of network protocols](https://www.ably.io/blog/what-is-a-distributed-systems-engineer/#deepunderstandingofnetworkprotocols)

**Microservices or SoA is not a distributed system**

Here's an example of a simplistic design of a service based architecture with horizontal scalability:

![A simplistic service based architecture that scales horizontally](https://ik.imagekit.io/ably/ghost/medium/max/1600/1*bKLMQfAFb69PbaTTH5ISIg.png?tr=w-1520)

There’s not much “distributed” about this system. There are multiple hosts and network interconnections but they are tightly coupled. And their network interactions are reliable, have low-latency, and are predictable. Genuinely distributed, in our view, means:

- Systems where nodes are distributed globally
- Network interactions are unpredictable and can create partitions
- Nonetheless those nodes work together to create a predictable outcome

Distributed systems, at scale, involve state being distributed and re-balanced across the system, reacting as nodes are added and removed, and they do this in spite of the unpredictability that is inherent in a global system.

**Understanding hash rings is a pre-requisite**

If you think a hash ring has something to do with a criminal cannabis organization, then that’s certainly amusing, but unfortunately means you’re missing knowledge of [a common pattern used for distributed systems](https://www.ably.io/blog/implementing-efficient-consistent-hashing/).

![Hash Ring example. Credit: Mathias Meyer from Travis CI](https://ik.imagekit.io/ably/ghost/medium/max/1600/1*ZdJ1oIK10rkdKl08nhfY1w.png?tr=w-1520)

If the above doesn’t look familiar, then I recommend you start by diving into how popular distributed systems work, all of which rely on the ideas behind a consistent hash ring. See:

- [Cassandra](https://en.wikipedia.org/wiki/Apache_Cassandra)
- [Riak](https://en.wikipedia.org/wiki/Riak)
- [Dynamo](https://en.wikipedia.org/wiki/Dynamo_%28storage_system%29)
- [Couchbase Server](https://en.wikipedia.org/wiki/Couchbase)

**Gossip protocols and consensus algorithms underpin everything**

Large distributed systems usually have to track changes in cluster topology in response to network partitions, failures, and scaling events. Various protocols exist to ensure that this can happen, with varying levels of consistency and complexity. This needs to be dynamic and real time because:

- Nodes come and go in elastic systems
- Failures need to be detected quickly
- Load and state need to be rebalanced in real time

With a stateful system like Ably, state also needs to be moved in real time between new and old nodes whilst providing [continuity](https://www.ably.io/blog/10-hardest-aspects-of-realtime-messaging/) throughout.

If you have never worked with Gossip or consensus algorithms, then I recommend you read up on:

- [Gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol)
- [Paxos protocol](https://en.wikipedia.org/wiki/Paxos_%28computer_science%29)
- [Raft consensus algorithm](https://raft.github.io/)
- Popular consensus backed systems like [etcd](https://github.com/coreos/etcd) and [Zookeeper](https://zookeeper.apache.org/), and gossip backed systems like [Serf](https://www.google.co.uk/search?q=serf+gossip&oq=serf+gossip&aqs=chrome..69i57.2442j0j1&sourceid=chrome&ie=UTF-8)

**Eventually consistent data types and read and write consistencies**

Generally in a distributed system, locks are impractical to implement and impossible to scale. As a result, trade-offs need to be made between the consistency and availability of data. In many cases, availability can be prioritised and consistency guarantees weakened to eventual consistency, with data structures such as CRDTs.

If you’re not familiar with CRDT or Operational Transform, the concepts of variable consistencies for queries or writes to data in a distributed data store, then you’ve got some reading to do:

- [Operational Transform](https://en.wikipedia.org/wiki/Operational_transformation) — originally implemented by [Google](https://www.ably.io/blog/google-polling-like-its-the-90s) in their Wave product and now in Google Docs. It has uses in collaboration apps, but OTs are complex and not widely implemented.
- [Conflict-free Replicated Data Types](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) or CRDT provides an eventually consistent result so long as the data types available are used. Used by [Riak distributed database](http://basho.com/products/) and [presence in Phoenix](https://dockyard.com/blog/2016/03/25/what-makes-phoenix-presence-special-sneak-peek).
- [Consistency levels for both read and writes](http://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_config_consistency_c.html) in distributed databases like Cassandra.

**Deep understanding of network protocols**

In a distributed system, you’ll almost certainly be working within all layers of the networking stack. Ably relies extensively on various higher level protocols such as HTTP, [WebSockets](https://www.ably.io/concepts/websockets), [gRPC](https://grpc.io/), and TCP sockets. But without a deep understanding of those protocols and the full stack of protocols they rely on all the way down to the OS itself, you’ll likely struggle to solve problems in a distributed system when things go wrong.

Take for example the following request or [WebSocket](https://www.ably.io/concepts/websockets) connection which involves all of the following. At each layer you should be confident in your understanding and ability to debug problems at a packet or frame level:

- [DNS](https://www.ably.io/blog/practical-strategies-for-dns-failure/) protocol and UDP for address lookup
- File descriptors (on *nix) and buffers used for connections, NAT tables, conntrack tables etc.
- IP to route packets between hosts
- TCP to establish a connection
- TLS handshakes, termination and certificate authentication
- HTTP/1.1 or more recently 2.0 used extensively by gRPC
- [WebSocket](https://www.ably.io/concepts/websockets) upgrades over HTTP

**And that’s not all…**

From our perspective of operating a truly [global and distributed system](https://www.ably.io/network), a working understanding of the specific concepts described above is what we expect from a distributed systems engineer. Before that you need to also be a solid systems engineer. This requires you to have fundamentals in place such as programming languages, general design patterns, version control, infrastructure management, and continuous integration and deployment systems.

**Further reading**

- [An introduction to distributed systems by Kyle Kingsbury](https://github.com/aphyr/distsys-class)
- [Hidden scaling issues of distributed systems](https://www.ably.io/blog/hidden-scaling-issues-of-distributed-systems-real-world/)
- [How Ably efficiently implemented consistent hashing](https://www.ably.io/blog/implementing-efficient-consistent-hashing/)
- [Implementing connection state and stream continuity over a distributed realtime messaging system](https://www.ably.io/blog/10-hardest-aspects-of-realtime-messaging/)
- [Building a distributed rate limiter that scales horizontally](https://www.ably.io/blog/distributed-rate-limiting-scale-your-platform/)


**About the author**

I’m Matt, technical CEO and co-founder of [Ably](https://www.ably.io/). I'm very interested in realtime problems, realtime engineering, and where the industry is headed. That’s the reason I co-founded Ably, which provides cloud infrastructure and APIs to help developers simplify complex realtime engineering. Organizations build with Ably because we make it easy to [power and scale realtime features in apps](https://www.ably.io/solutions/realtime-apis), or [distribute  data streams to third-party developers](https://www.ably.io/solutions/api-management) as realtime APIs.

**We're hiring across our engineering and commercial teams, including Distributed Systems Engineering roles** so [check out our job board](https://www.ably.io/careers). Not looking for a role but know someone who is? Refer an engineer that we employ and we'll send you $3k as a thanks. [One email = $3k](https://gist.github.com/mattheworiordan/44354fca2a92dfe991e72f1a43c8578b).

And if you’re interested in chatting to me about realtime problems, distributed systems, or this article, please do reach out to me @mattheworiordan or @ablyrealtime on Twitter.