Supercharge Your Container Networking: Seamless Host Communication with VxLAN and Docker
Tags: docker, devops, linux, cloud
Published: 2023-07-15 05:19:42

In this hands-on demo, we will explore how to set up communication between two hosts using Virtual Extensible LAN (VxLAN) and Docker. The goal is to create a VxLAN overlay network tunnel between the hosts' containers, allowing them to communicate with each other. Let's dive into the steps involved in this process.

## What are we going to cover in this hands-on demo?

- **Creating two VMs and installing Docker:** We will use two virtual machines (VMs) and install Docker to run the containers.

- **Creating separate subnets and assigning static IP addresses:** To simplify the setup, we'll create separate subnets for each VM and assign static IP addresses.

- **Creating a VxLAN bridge:** We'll utilize the Linux "ip link vxlan" feature to create a VxLAN bridge.

- **Binding the VxLAN to the Docker bridge:** We'll bind the VxLAN to the Docker bridge to establish the tunnel.

- **Verifying communication between containers:** Finally, we'll test the communication between containers on different hosts.

**Let's get started!**
To facilitate effective communication between hosts, we need to deploy two VMs using any hypervisor or virtualization technology. It is crucial to ensure that both VMs are connected to the same network.

**Step 1: Creating the VMs**

We begin by creating two [Lubuntu VMs](https://help.ubuntu.com/community/Installation/MinimalCD) using [UTM](https://mac.getutm.app/) / [Multipass](https://ubuntu.com/server/docs/virtualization-multipass). These VMs will serve as the hosts for our containers.


**Step 2: Installing necessary tools and Docker**

In this step, we will install several necessary tools and Docker on our Ubuntu VMs. Here's a brief overview of the tools we are going to install:

- `net-tools`: This package includes important network management tools such as `ifconfig`, `netstat`, `route`, etc.
- `iputils-ping`: This package provides the `ping` command, which is used to test the reachability of a network host.
- `bridge-utils`: This package provides utilities for configuring Ethernet bridging on Linux.

To install these tools and Docker, execute the following commands:

```bash
apt-get update
apt-get install net-tools
apt-get install iputils-ping
apt-get install bridge-utils
apt-get install -y docker.io
```

Additionally, if the IP address of one of the VMs is the same as the other, you'll need to modify it to avoid conflicts. For example, you can change the IP address of one of the VMs to **192.168.64.7/24** using the following commands:

```bash
ifconfig eth0 192.168.64.7/24
route add default gw 192.168.64.1 eth0
```

To configure DNS resolution, open the resolv.conf file:
```
nano /etc/resolv.conf
```

Delete the existing content and add the following lines: 
```bash
nameserver 8.8.8.8
nameserver 8.8.4.4
```

![Image 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/trdnewwtoi3v3f5yjcig.png)

Now, let's proceed to set up the containers and their communication.

**Step 3: Running Docker Containers on the VxLAN Network**
Now for host1(amicable-hyena),

```bash
# create a separate docker bridge network 
docker network create --subnet 172.18.0.0/16 vxlan-net

# list all networks in docker
docker network ls
# The output should include the newly created vxlan-net network.
NETWORK ID     NAME        DRIVER    SCOPE
53be5ce8e682   bridge      bridge    local
757eb3a14b73   host        host      local
c1c0e4f01fa6   none        null      local
08bdd2dc3a82   vxlan-net   bridge    local

# Check interfaces
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 36:db:7c:3f:11:58 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.9/24 brd 192.168.64.255 scope global dynamic enp0s2
       valid_lft 85395sec preferred_lft 85395sec
    inet6 fde4:cfbc:2cca:cd78:34db:7cff:fe3f:1158/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591923sec preferred_lft 604723sec
    inet6 fe80::34db:7cff:fe3f:1158/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:aa:59:e4:e3 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-08bdd2dc3a82: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:4b:42:67:6b brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-08bdd2dc3a82
       valid_lft forever preferred_lft forever
```

For Host 2(affluent-toad),
```bash
docker network create --subnet 172.18.0.0/16 vxlan-net

docker network ls
cdc5ceed0de2   bridge      bridge    local
11ed966e63df   host        host      local
5ea8856ad30c   none        null      local
55316818ea9f   vxlan-net   bridge    local

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether f2:5f:8d:d3:9f:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.10/24 brd 192.168.64.255 scope global dynamic enp0s2
       valid_lft 85276sec preferred_lft 85276sec
    inet6 fde4:cfbc:2cca:cd78:f05f:8dff:fed3:9fc9/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591908sec preferred_lft 604708sec
    inet6 fe80::f05f:8dff:fed3:9fc9/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:b3:cb:9a:b9 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-55316818ea9f: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:2c:83:cd:df brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-55316818ea9f
       valid_lft forever preferred_lft forever
```


**Step 4: Run docker container**
Let's run docker container on top of newly created docker bridge network and try to ping docker bridge

For Host 1,
```bash
# running alpine container with "sleep 3000" and a static ip
docker run -d --net vxlan-net --ip 172.18.0.11 alpine sleep 3000

# check the container running or not
docker ps
CONTAINER ID   IMAGE     COMMAND        CREATED              STATUS              PORTS     NAMES
d2b96eddea7a   alpine    "sleep 3000"   About a minute ago   Up About a minute             funny_jepsen

# check the IPAddress to make sure that the ip assigned properly
docker inspect d2 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.18.0.11",

# ping the docker bridge ip to see whether the traffic can pass
ping 172.18.0.1 -c 2
PING 172.18.0.1 (172.18.0.1) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_seq=1 ttl=64 time=0.166 ms
64 bytes from 172.18.0.1: icmp_seq=2 ttl=64 time=0.050 ms

--- 172.18.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.050/0.108/0.166/0.058 ms
```

For Host 2,
```bash
docker run -d --net vxlan-net --ip 172.18.0.12 alpine sleep 3000

docker ps
CONTAINER ID   IMAGE     COMMAND        CREATED         STATUS         PORTS     NAMES
9d2e17598b8b   alpine    "sleep 3000"   3 seconds ago   Up 2 seconds             eloquent_black

docker inspect 9d | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.18.0.12",

ping 172.18.0.1 -c 2

PING 172.18.0.1 (172.18.0.1) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_seq=1 ttl=64 time=0.193 ms
64 bytes from 172.18.0.1: icmp_seq=2 ttl=64 time=0.067 ms

--- 172.18.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1017ms
rtt min/avg/max/mdev = 0.067/0.130/0.193/0.063 ms
```

**Step 5: Testing Communication**
Now, let's access one of the running containers and test communication between hosts. Note that containers within the same host should communicate, but container-to-container communication between hosts will fail at this stage because there is no tunnel or anything to carry the traffic.

```bash
docker exec -it d2 sh

apk update
apk add net-tools
apk add iputils-ping

ping 172.18.0.12 -c 2
ping: -c: Try again

ping -c 2 192.168.64.10
PING 192.168.64.10 (192.168.64.10) 56(84) bytes of data.
64 bytes from 192.168.64.10: icmp_seq=1 ttl=63 time=3.93 ms
64 bytes from 192.168.64.10: icmp_seq=2 ttl=63 time=0.967 ms

--- 192.168.64.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.967/2.447/3.928/1.480 ms
```

**Step 6: Creating the VxLAN Tunnel**
To establish communication between the containers, we need to create a VxLAN tunnel and attach it to the Docker bridge. Make sure the VNI (Virtual Network Identifier) ID is the same for both hosts.

For Host 1,

```bash
# check the bridges list on the hosts
brctl show

bridge name	bridge id		STP enabled	interfaces
br-08bdd2dc3a82		8000.02424b42676b	no
docker0		8000.0242aa59e4e3	no

# create a vxlan
# 'vxlan-demo' is the name of the interface, type should be vxlan
# VNI ID is 100
# dstport should be 4789 which a udp standard port for vxlan communication
# 192.168.64.10  is the ip of another host
ip link add vxlan-demo type vxlan id 100 remote 192.168.64.10 dstport 4789 dev enp0s2

# check interface list if the vxlan interface created
ip a | grep vxla

7: vxlan-demo: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000

# make the interface up
ip link set vxlan-demo up

# now attach the newly created vxlan interface to the docker bridge we created
brctl addif br-08bdd2dc3a82 vxlan-demo

# check the route to ensure everything is okay. here '172.18.0.0' part is our concern part.
route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.64.1    0.0.0.0         UG    100    0        0 enp0s2
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-08bdd2dc3a82
192.168.64.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s2
192.168.64.1    0.0.0.0         255.255.255.255 UH    100    0        0 enp0s2
```

For Host 2,

```bash
brctl show

bridge name	bridge id		STP enabled	interfaces
br-55316818ea9f		8000.02422c83cddf	no
docker0		8000.0242b3cb9ab9

ip link add vxlan-demo type vxlan id 100 remote 192.168.64.9 dstport 4789 dev enp0s2

ip a | grep vxlan

7: vxlan-demo: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000

ip link set vxlan-demo up
brctl addif br-55316818ea9f vxlan-demo

route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.64.1    0.0.0.0         UG    100    0        0 enp0s2
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-55316818ea9f
192.168.64.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s2
192.168.64.1    0.0.0.0         255.255.255.255 UH    100    0        0 enp0s2
```

**Step 7: Testing Communication between Containers**
Now that the VxLAN overlay network tunnel has been created, let's test the communication between the containers on different hosts.

```bash
docker exec -it a9 bash
ping 192.168.64.10 -c 2
```

You can find result of ping from below images. [Note: Re ran docker container in both host as previous container already ran for 3000 seconds] 

![Image 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/33l8m5pr7pidcv9pa9sw.png)

You should see successful ping responses, indicating that communication between the containers on different hosts is now established.

Congratulations! You have successfully set up communication between hosts using VxLAN and Docker.

**Reference:**
The demonstration code and steps mentioned in this blog post were adapted from the following GitHub repository: [vxlan-docker-hands-on](https://github.com/faysalmehedi/vxlan-docker-hands-on)

Feel free to explore the repository for more in-depth details and examples.

Thank you for reading this blog post, and I hope you found it informative and useful.