Building a simple Distributed System with Go & Consul
Tags: go, dockercompose, distributedsystems, consul
Published: 2019-04-26 17:42:09

---
title: Building a simple Distributed System with Go & Consul
published: true
tags: golang,docker-compose,distributed-systems,consul
canonical_url: https://medium.com/@didil/building-a-simple-distributed-system-with-go-consul-39b08ffc5d2c
---

Having worked with a few Distributed Systems over the years such as Apache Kafka and Ethereum nodes I was wondering what it would take to build a minimal one. In this article we’ll be building a micro “kafka-like” distributed logging app with super simple features, using Go and Hashicorp Consul. You can follow along using this [github repository](https://github.com/didil/go-consul-distributed-loggers).

![](https://cdn-images-1.medium.com/max/743/1*z-KdPJS01RJP5vmu4cfOLg.png)<figcaption>Go + Consul: A solid combination for building Distributed Systems</figcaption>

#### What is Consul ?

Looking at the definition in the intro section of the official [website](https://www.consul.io/intro/index.html):

> Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality

That might sounds a bit scary, but don’t leave just yet. The main Consul feature we’ll use in this article is the **Key/Value store** , which will support our simple [**Leader Election**](https://www.consul.io/docs/guides/leader-election.html) ** ** system **.**

#### System Architecture

The diagram below shows the system’s architecture :

![](https://cdn-images-1.medium.com/max/870/1*JLf894B4Pc7SU-8evfMd7A.png)<figcaption>Distributed Loggers App Architecture</figcaption>

The system is composed of 3 parts:

- **consul** : Consul instance that provides support for leader election and service discovery.
- **distributed-logger** : The Distributed Logger nodes expose a REST API that logs received messages to Stdout. Only the cluster leader accepts messages at any given time. A new node takes over in case of leader node failure.
- **producer** : The producer queries Consul periodically to determine the distributed-logger leader and sends it a numbered message.

#### Demo

You can run the demo using [docker-compose](https://docs.docker.com/compose/) by pulling the [repo](https://github.com/didil/go-consul-distributed-loggers) and running:

```
$ docker-compose up -d --scale distributed-logger=3
```

What happens then is:

- The Consul instance is started.
- 3 distributed-logger instances are started and register a [Session](https://www.consul.io/docs/internals/sessions.html) with Consul.
- A distributed-logger wins the election (in this case just because it’s faster in acquiring the lock from Consul) and becomes the Leader.
- The producer comes online, then polls Consul for the leader every 5 seconds and sends it a message.
- The leader receives the messages and logs it to Stdout
- If the leader dies or is killed via SIGTERM or SIGINT, a new distributed-logger node takes over as the leader and starts receiving messages from the producer

You can see an example below where Node 3 is the leader, then Node 2, then Node 1:

![](https://cdn-images-1.medium.com/max/1024/1*f_CwLC37BRPKyg7GeTbVgg.png)<figcaption>Distributed Loggers demo</figcaption>

#### Show me the code

The full code is available in the [repo](https://github.com/didil/go-consul-distributed-loggers), but let’s look at a couple of interesting aspects of the app:

- To acquire leader status, each distributed-logger node runs a goroutine that tries periodically to acquire a lock in Consul:

{% gist https://gist.github.com/didil/b4a470224140b51128567c9ed7d339b3 %}

- When signaled with SIGINT or SIGTERM, the distributed-logger nodes destroy their Consul Session, effectively releasing the lock and losing Leader status if the node was the Leader:

{% gist https://gist.github.com/didil/d3cbff8f8b226218a24647317d83e7bf %}

- The producer runs a loop to discover the Leader node and send messages:

{% gist https://gist.github.com/didil/b69de98fbe033d5f5e576b8aad2183b7 %}

#### Conclusion

Our little Proof of Concept is very far from being production ready. Distributed Systems is a very vast field and we are missing many aspects like: Replication, High Availability and solid Fault Tolerance.

That being said, I hope that you have enjoyed this example and that it will allow you to start diving into this complex topic and hopefully start building your own Distributed Systems ! Let me know of you have any questions or remarks.