The Overlay Network Driver | Networking in Docker #7
Tags: docker, devops, cloud, networking
Published: 2022-10-12 13:21:48

> Learn how the Overlay driver simplifies multi-host networking on Docker, how to use it, possible use cases and limitations

---

Link to video: https://www.youtube.com/watch?v=gJVcKVdYhpI

---


What is the `overlay` network driver in Docker and how does it power docker swarm and multi-host container networking?

This blog will try to answer that (and more) as simply as possible.

# Introduction

This blog is the seventh one in a series for Docker Networking.

- [blog #1 - **Why is Networking important in Docker?**](https://www.techmormo.com/posts/docker-networking-1-why-is-networking-important/)
- [blog #2 - **Basics of Networking**](https://www.techmormo.com/posts/docker-networking-2-basics-of-networking/)
- [blog #3 - **Docker Network Drivers Overview**](https://techmormo.com/posts/docker-networking-3-drivers-overview/)
- [blog #4 - **None Network Driver**](https://www.techmormo.com/posts/docker-networking-4-none-driver/)
- [blog #5 - **Host Network Driver**](https://techmormo.com/posts/docker-networking-5-host-driver/)
- [blog #6 - **Bridge Network Driver**](https://techmormo.com/posts/docker-networking-6-bridge-driver/)

If you are looking to learn more about the basics of Docker, I’ll recommend checking out the [Docker Made Easy](https://www.techmormo.com/posts/docker-made-easy-1-what-is-docker/) series.

Here’s the agenda for this blog:

- What is the `overlay` network driver?
- How to use it?
- When to use it? - possible use cases
- What are its limitations?

Here are some quick reminders:

- A **Docker host** is a physical or virtual machine that runs the Docker daemon.
- **Docker Network drivers** enable us to easily use multiple types of networks for containers and hide the complexity required for network implementations.

Alright, so...

# What is the `overlay` driver?

> The `overlay` network driver creates a distributed network that can span multiple docker hosts.


![overlay driver](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5rthf6rfqdx91e878kdi.png)

Overlay networks were designed to be used with _docker swarm services_.

Docker swarm is a container orchestration tool (like Kubernetes) that allows us to manage multiple containers on multiple hosts.

To deploy an application (ie. docker image) on the swarm, we create a swarm service, which can run 1 or more containers from the image.


![swarm service example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bigjubgbu0gurj0gc3lk.png)

They will make more sense when we actually try them out in a moment.

When a docker host initiates or joins a swarm, 2 new networks are created:

- a default `overlay` network called `ingress` which handles the control and data traffic related to swarm services.
- and a virtual `bridge` network called `docker_gwbridge` that connects overlay networks to the individual Docker daemon’s physical network.

Like custom bridge networks we saw in the last tutorial, custom or user-defined overlay networks can be created using the `docker network create` command (using the `--driver overlay` option).

Let’s directly see overlay drivers in action…

# How to use the `overlay` driver?

I encourage you to follow along to get the most out of this tutorial.

### **Initialize the swarm**

**Provision hosts with network connectivity**

To see the full potential of overlay networks, we need multiple hosts. Let’s go with 3.

The simplest solution is to use a Virtual Machine tool like [VirtualBox](https://www.virtualbox.org/) or [UTM](https://mac.getutm.app/) (on Mac) to create multiple VMs.

For multiple machines to communicate together as a swarm using overlay, docker requires the following ports to be open between the hosts:

- TCP port 2377 for cluster management communications
- TCP and UDP port 7946 for communication among nodes
- UDP port 4789 for overlay network traffic

If you’re running VMs locally, make sure the VMs can reach each other over the network (ie. select ”Internal Network” on VirtualBox, or “Shared Network” on UTM).

If you’re running VMs on the cloud or in a strict networked environment, make sure to update your firewall rules / security groups to account for the open port requirements.

Before moving forward, ensure connectivity between your hosts with something like `ping`.

I have created 3 Debain Linux VMs named node1, node2 and node3.

**Install Docker**

Follow the official docker instructions to install Docker Server on all your hosts.

[https://docs.docker.com/engine/install/](https://docs.docker.com/engine/install/)

**Create the swarm and join the nodes**

On the first node, we will run the following:

```bash
docker swarm init
```

This will initialize the swarm with the first node as the manager and the only member for now.

It should also print the command that will let us add other nodes into the swarm as workers, like the following:

```bash
docker swarm join --token SWMTKN-1-07ti6ugkt6hia44ljws48eik9t4v0806515jkodf3zn1pb7uod-15c75qxan0guvqmcswt4blz2l 192.168.64.4:2377
```

We will copy the full `docker swarm join` command and run it on each of our remaining nodes.

When all are successfully joined, we can check the nodes by running the following on the manager node:

```bash
docker node ls
```

We can also check that the 2 new networks were created with:

```bash
docker network ls
```

Awesome, we now have a docker swarm cluster with an overlay network!

### The default overlay network - `ingress` and routing mesh

Let us start off by creating a service named `whoami` using the `containous/whoami` image:

```bash
docker service create \
	--name whoami \
	-p 8000:80 \
	containous/whoami
```

This image runs a tiny webserver on port 80 inside the container, and simply returns some OS and HTTP information.

The `-p 8000:80` option tells docker to map port 80 inside the container to port 8000 on the host.

_But on which host?_

Check which host (NODE) the container is running on using:

```bash
docker service ps whoami
```

Let’s say the container is running on node2.

How can we confirm that the `ingress` network is being used?

Switch to the node2 terminal and get the container ID:

```bash
CONTAINER_ID=$(docker ps --filter name=whoami.1 --format '{{.ID}}')
```

Then `inspect` its network:

```bash
docker inspect $CONTAINER_ID --format='{{json .NetworkSettings.Networks}}' | jq .
```

You should see a json object named `ingress`, confirming our hypothesis.

Since the service has been published on port 8000, if we send a request with curl from node2, like:

```bash
curl 127.0.0.1:8000
```

We will get something like this:

```bash
Hostname: fb772a20940c
IP: 127.0.0.1
IP: 10.0.0.28
IP: 172.18.0.3
RemoteAddr: 10.0.0.3:59882
GET / HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/7.64.0
Accept: */*
```

Not surprising.

But the cool thing is, if you try to run the same curl command on the other hosts, you will still get the same response (except for `RemoteAddr`), even though those nodes don’t have the container running in them. Try it out!

Why is that?

All swarm nodes participate in an ingress **routing mesh**. The routing mesh enables each node in the swarm to accept connections on published ports for any service running in the swarm, even if there’s no container running on the node.


![1 replica ingress](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cpc7ch1ech6727phzxy8.png)

This means, whenever a docker swarm service publishes a port (with `-p`/`--publish`), that service is reachable on that port across all swarm nodes. The overlay network is responsible for routing the traffic to the correct container on the correct node.

But what if there are more than 1 containers?

Let’s find out!

Running multiple containers of a service is as simple as:

```bash
docker service scale whoami=3
```

In a few seconds, we should have 3 replicas of the whoami container running.

Check which host (NODE) the containers are running on using:

```bash
docker service ps whoami
```

We should see that all 3 nodes are assigned a container.


![3 replica ingress](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xgpl2mlpbiunqta2e2m1.png)

Docker swarm is smart enough to distribute the container workload among the available nodes.

If we now curl again on any of the nodes with:

```bash
curl 127.0.0.1:8000
```

We should see that the Hostnames and IPs are also changing.

This is because, containers running under a swarm service on the overlay network are automatically load-balanced!

Awesome.

Now, what if we want to attach a standalone container (not a swarm service) on the overlay network, for example to communicate across hosts?

If we try to do the following:

```bash
docker run -d --rm \
	--network ingress \
	--name app1 \
	nginx:alpine
```

This will raise an error like “PermissionDenied desc = network ingress not manually attachable.”

The default overlay network (`ingress`) doesn’t allow attaching standalone containers. For that, we can use…

### **User-defined Overlay Networks**

At first, we have to create a network with the overlay driver option:

```bash
docker network create \
	--driver overlay \
	--attachable \
	my-overlay
```

By using the `--attachable` flag, docker will allow us to attach standalone containers to it.

If we try to run a standalone container again but this time using our custom overlay network `my-overlay`:

```bash
docker run -d --rm \
	--network my-overlay \
	--name app1 \
	nginx:alpine
```

it should work.

Since app1 is lonely, let’s run a service with 3 replicas (containers) on `my-overlay` as well:

```bash
docker service create \
	--name web \
	--network my-overlay \
	--replicas 3 \
	nginx:alpine
```

NOTE: we have not published any ports, but this image runs [nginx](https://www.nginx.com/) server on port 80 inside the container.

Now, `web` service containers can communicate with `app1` container and vice versa!

Really?

Try to reach `web` service from `app1` simply by using the name of the service with curl:

```bash
docker exec app1 curl web
```

It should print out the `index.html` page of nginx.

To test the other way around ie. communication from `web` service to `app1`, first, get the container id of a `web` service container.

```bash
CONTAINER_ID=$(docker ps --filter name=web --format '{{.ID}}' | head -n 1)
```

Then use exec again to curl `app1`:

```bash
docker exec $CONTAINER_ID curl app1
```

See the nginx index page?

Congratulations! You now know how to use overlay networks! 🎉

Of course, there’s a lot more to it but we have covered the essentials. Feel free to learn more from [the official docs](https://docs.docker.com/network/overlay).

### **Clean up**

Let’s clean up our mess.

Remove `app1`:

```bash
docker rm -f app1
```

Then remove the services from the manager node:

```bash
docker service rm whoami web
```

After the services are down, remove the custom overlay:

```bash
docker network rm my-overlay
```

Then, remove each of the worker nodes from the swarm (by running on each worker node):

```bash
docker swarm leave
```

Finally, make the manager leave the swarm:

```bash
docker swarm leave --force
```

Well done! 👏

# When to use the `overlay` driver? - possible use cases

- If you have multiple nodes to deploy your containerized workloads, but you have a small team with limited containerization experience.
- If you need a simple container orchestrator like docker swarm, but do not have the time/resources to invest in more complex orchestrators like Kubernetes.

# **Limitations of the `overlay` driver**

1. **`overlay` driver is slower than `host` driver**: If you need the highest network performance and your workload is running on a single host, or you are okay with setting up the OS level routing by yourself, then the `host` driver would be more suitable than `overlay`. This is because `host` driver directly uses the host’s network, instead of having to setup separate interfaces, and port-mapping configurations.
2. **Docker (swarm) does not natively support autoscaling**: autoscaling is not supported out of the box as you need to provision new infrastructure/servers which is out of the scope of docker. There are 3rd party tools/scripts that can help, but if you really need the ability to autoscale, you should consider something like Kubernetes or AWS ECS.

# Conclusion

In this blog, we learnt about the `overlay` network driver in docker - what it is, how to use it, and some possible use cases and limitations.

By connecting multiple docker daemons in a swarm, the `overlay` driver simplifies multi-host networking. But its drawbacks are to be kept in mind, especially when using it in production.

I hope I could make things clearer for you, be it just a tiny bit.

In the next blog, we will summarize what we learnt in this docker networking series.

Thanks for making it so far! 👏

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!