Docker ARP Analysis Guide
Tags: docker, networking, ubuntu, multipass
Published: 2024-01-16 21:49:16

## Table of Contents

- [Introduction](#introduction)
- [Requirements](#requirements)
  - [Install Homebrew & Multipass](#install-homebrew-and-multipass)
  - [Download an Ubuntu Server from Multipass](#download-an-ubuntu-server-from-multipass)
  - [Go to the Shell of the Ubuntu Server](#go-to-the-shell-of-the-ubuntu-server)
  - [Install Docker](#install-docker)
  - [Mount a Local Folder to an Ubuntu Directory](#mount-a-local-folder-to-an-ubuntu-directory)
  - [Verify Ubuntu Server Installation](#verify-ubuntu-server-installation)
  - [Install Wireshark](#install-wireshark)
- [Architecture Deployment Guide](#architecture-deployment-guide)
  - [Enter into the Shell of the Ubuntu Server Again](#enter-into-the-shell-of-the-ubuntu-server-again)
  - [Create a New Network](#create-a-new-network)
    - [Verify Network](#verify-network)
  - [Pull Alpine Image](#pull-alpine-image)
  - [Listen for ARP Packets in each Bridge](#listen-for-arp-packets-in-each-bridge)
  - [Create 2 Containers in the Default Bridge, Also Connect Them to the Custom Bridge](#create-2-containers-in-the-default-bridge-also-connect-them-to-the-custom-bridge)
  - [Send Pings to the Internet From the First Interface](#send-pings-to-the-internet-from-the-first-interface)
  - [Send Pings to the Internet From the Second Interface](#send-pings-to-the-internet-from-the-second-interface)
  - [View Mac Addresses of Each Bridge Interface](#view-mac-addresses-of-each-bridge-interface)
  - [View Mac Addresses of Each Container Interface](#view-mac-addresses-of-each-container-interface)
  - [End ARP Packet Capture 1](#end-arp-packet-capture-1)
  - [End ARP Packet Capture 2](#end-arp-packet-capture-2)
  - [Move the Files to the Local Directory](#move-the-files-to-the-local-directory)
- [View Packet Captures](#view-packet-captures)
  - [Docker0 Bridge Interface](#docker0-bridge-interface)
  - [Docker1 Bridge Interface](#docker1-bridge-interface)
- [Clean-Up](#clean-up)
  - [Stop and Remove the Container](#stop-and-remove-the-container)
  - [Delete Custom Bridge](#delete-custom-bridge)
  - [Verify Network List](#verify-network-list)
  - [Stop Ubuntu Server](#stop-ubuntu-server)
- [Conclusion](#conclusion)
- [Resources](#resources)

> Read this as [technical documentation](http://docker-network.readthedocs.io/) instead.

## Introduction
I've been immersing myself in CI/CD pipeline studies lately, and now that I've gotten a good grasp of it, I can finally dedicate my full attention to understanding container technologies. To start, I decided to focus on networking, which is the stack I'm most familiar with.
In this simple how-to guide, I'll walk you through observing the Address Resolution Protocol (ARP) in action within a Docker environment.
An Address Resolution Protocol(ARP) allows a container to learn the MAC address of another device (in this lab, it’s the bridge and other containers) dynamically.
Without an ARP, a ping between containers, external networks, or even a web request between containers and other devices fails.

## Requirements
I recently made the switch to a Mac and found using [Multipass](https://multipass.run/) from Canonical simpler than spinning up a VM with tools like Virtualbox. However, when it comes to container networking labs, non-Linux systems are not recommended. While I downloaded Docker Desktop for Mac, I had trouble seeing all the Docker network interfaces.
For Windows users, I will advise you to use WSL2 instead, it’s easier to deploy and manage compared to having to use a virtual machine like Virtualbox.
Keep in mind that the Ubuntu installation requirement is only for Mac and Linux users. This guide will use three terminal tabs throughout.

### Install Homebrew and Multipass
Download [Homebrew](https://brew.sh/)
```bash
# Terminal-1 Mac/Linux
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

Download Multipass
```bash
# Terminal-1 Mac/Linux
brew install multipass
```

### Download an Ubuntu Server from Multipass

This Ubuntu server will be customized to run the same specifications as an AWS EC2-T2 Micro instance type.

- 1 CPU
- 1 GB of RAM
- 8 GB of disk
- version 22.04

Simple and fast, right?

```bash
# Terminal-1 Mac/Linux
multipass launch jammy --name=ubuntu --cpus=1 --disk=8G --memory=1G
```

`jammy` is the [image name](https://multipass.run/docs/create-an-instance#heading--create-an-instance-with-a-specific-image) for Ubuntu server version 22.04.

### Go to the Shell of the Ubuntu Server

```bash
# Terminal-1 Mac/Linux
multipass shell ubuntu
```

### Install Docker

Download [docker](https://docs.docker.com/engine/install/ubuntu/) on the Ubuntu server.

```bash
# Terminal-1 Ubuntu
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Close this Ubuntu shell.
# Sometimes you might need to use the exit command severally
# to successfully exit the shell.
exit
```

### Mount a Local Folder to an Ubuntu Directory

Mount the `Documents/` folder in your Mac, or other machine to the `mnt/` directory in the Ubuntu server running inside Multipass.

```bash
# Terminal-1 Mac/Linux
multipass mount /Users/apple/Documents /mnt/
```

### Verify Ubuntu Server Installation

```bash
apple@Charles-MBP ~ % multipass info ubuntu
Name:           ubuntu
State:          Running
Snapshots:      0
IPv4:           192.168.64.4
                172.17.0.1
Release:        Ubuntu 22.04.3 LTS
Image hash:     9dxa2awl28c8 (Ubuntu 22.04 LTS)
CPU(s):         1
Load:           0.00 0.00 0.00
Disk usage:     2.3GiB out of 7.7GiB
Memory usage:   201.4MiB out of 951.6MiB
Mounts:         /Users/apple/Documents => /mnt
                    UID map: 501:default
                    GID map: 20:default
```

### Install Wireshark

Choose your preferred machine and [download](https://www.wireshark.org/download.html).

## Architecture Deployment Guide

I have a simple architecture that deploys two docker containers in two different subnets. In docker, you can attach one container to several subnets. This is achieved by a new interface being created and assigned to that subnet.

![architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/unu933ml2k86emcydhj7.png)

![table](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ibv48bc3mf5838htqser.png)

### Enter Into the Shell of the Ubuntu Server Again

```bash
# Terminal-1 Mac/Linux
multipass shell ubuntu
```

### Create a New Network

I noticed that in docker, you only specify a subnet and mask. This makes sense because if you are deploying this on AWS, a VPC will be defined already, all you need to do is create a new subnet mask for your containerized environment.

```bash
# Terminal-1 Ubuntu
docker network create \
-o com.docker.network.bridge.name=docker1 \
--subnet=172.18.0.0/24 \
--gateway=172.18.0.253 \
custom_bridge
```

### Verify Network

```bash
# Terminal-1 Ubuntu

# This is a simulated command and output
ubuntu@ubuntu:~$ docker network ls
NETWORK ID     NAME            DRIVER    SCOPE
6e5ffkvms8c3   bridge          bridge    local
**d3b0029faed7   custom_bridge   bridge    local**
e9f55dsdf605   host            host      local
e9708nj5179a   none            null      local
```

### Pull Alpine Image

I love using Alpine Linux because it’s lightweight.

```bash
# Terminal-1 Ubuntu
docker pull alpine
```

### Open a New Terminal

Execute this command to open a new tab. `"⌘ + T"`

Then enter the Ubuntu shell

```bash
# Terminal-2 Mac
multipass shell ubuntu
```

### Listen for ARP Packets in Each Bridge

Now that the Ubuntu shell has been initialized, execute the below command to capture all packets.

#### docker0

```bash
# Terminal-2 Ubuntu
sudo tcpdump -i docker0 -w capture_docker_0.pcap
```

Open a third terminal tab `"⌘ + T"`

#### docker1

Execute another command to listen for all packets in the `docker1` bridge interface.

```bash
# Terminal-3 Ubuntu
sudo tcpdump -i docker1 -w capture_docker_1.pcap
```

### Create 2 Containers in the Default Bridge, Also Connect Them to the Custom Bridge

```bash
# Terminal-2 Ubuntu

# Create containers in the default bridge
docker run -itd \
--name=alpine1 \
--ip=172.17.0.2 \
alpine

docker run -itd \
--name=alpine2 \
--ip=172.17.0.4 \
alpine

# Connect new interfaces in the containers to another network.
docker network connect \
--ip=172.18.0.3 \
custom_bridge alpine1

docker network connect \
--ip=172.18.0.5 \
custom_bridge alpine2
```

### Send Pings to the Internet From the First Interface

```bash
# Ping google.com four times in each container from 'bridge'
docker exec -it alpine1 ping -I 172.17.0.2 -c 2 google.com
docker exec -it alpine2 ping -I 172.17.0.4 -c 2 google.com
```

### Send Pings to the Internet From the Second Interface

```bash
# Ping google.com four times in each container from 'bridge'
docker exec -it alpine1 ping -I 172.18.0.3 -c 2 google.com
docker exec -it alpine2 ping -I 172.18.0.5 -c 2 google.com
```

### View MAC Addresses of Each Bridge Interface

> The Organizationally Unique Identifier (OUI) of all Docker network adapters is **02:42**. So expect all docker container MAC addresses to begin with that.

```bash
# Terminal-1 Ubuntu
ip --brief link | grep -E 'docker0|docker1' | awk '{print $1, $3}'
```

Output

```bash
docker0 02:42:28:a8:cb:f5
docker1 02:42:7c:61:6d:f0
```

### View Mac Addresses of Each Container Interface

#### bridge

```bash
# Terminal-1 Ubuntu
docker network inspect bridge --format '{{range .Containers}}{{.Name}}: {{.MacAddress}}{{"\n"}}{{end}}'
```

Output-1

```bash
alpine1: 02:42:ac:11:00:02
alpine2: 02:42:ac:11:00:03
```

#### custom_bridge

```bash
# Terminal-2 Ubuntu
docker network inspect custom_bridge --format '{{range .Containers}}{{.Name}}: {{.MacAddress}}{{"\n"}}{{end}}'
```

Output-2

```bash
alpine1: 02:42:ac:12:00:03
alpine2: 02:42:ac:12:00:05
```

### End ARP Packet Capture 1

```bash
# Terminal-2 Ubuntu
"control + c"

```

### End ARP Packet Capture 2

```bash
# Terminal-3 Ubuntu
"control + c"

```

### Move the Files to the Local Directory

The captured packet files will be moved to the local directory so they can be viewed and analyzed using Wireshark.

```bash
# Terminal-1, 2, or 3 Ubuntu
mv capture_docker_0.pcap /mnt
mv capture_docker_1.pcap /mnt
```

### View Packet Captures
#### Docker0 Bridge Interface

Now I located and opened the packet capture.

![docker0](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mrg7ypzkl8lnsldzojpg.png)

We noticed something interesting: The default MAC addresses of `alpine1: 02:42:ac:11:00:02`, and `alpine2: 02:42:ac:11:00:03`, were requesting the MAC address for the bridge's interface, `docker0: 02:42:28:a8:cb:f5`, to send an Ethernet frame to `google.com`. The bridge's interface was used as the next hop.

We also see that the DNS name server lookup for `google.com` could be possible only after an ARP reply from `docker0: 02:42:28:a8:cb:f5`.  

#### Docker1 Bridge Interface

![docker1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iixiinskrgfsjv17klix.png)

The same also applies here. the custom MAC addresses of `alpine1: 02:42:ac:12:00:03`, and 
`alpine2: 02:42:ac:12:00:05` were requesting the mac-address for the bridge’s interface `docker1: 02:42:7c:61:6d:f0` so it can send an ethernet frame destined to `google.com`. The bridge’s interface is used as the next hop. 

> Go to, [View Mac Addresses of Each Bridge Interface](#view-mac-addresses-of-each-bridge-interface) and [View Mac Addresses of Each Container Interface](#view-mac-addresses-of-each-container-interface) to confirm the mac-addresses of each device when analyzing the packet capture.

## Clean-Up

### Stop and Remove the Container

```bash
docker stop alpine1 alpine2
docker rm alpine1 alpine2
```

### Delete Custom Bridge

```bash
docker network rm custom_bridge
```

### Verify Network List

```bash
root@ubuntu:/home/ubuntu# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
2eabde4e866c   bridge    bridge    local
e9f54b21c605   host      host      local
e9708605179a   none      null      local
```

### Stop Ubuntu Server

```bash
multipass stop ubuntu
```

## Conclusion

Now, I understand that everyone can't stop talking about Kubernetes, but a lot of senior engineers have advised that it'd be best to learn docker before picking kubernetes up. Though I've played with Kubernetes severally, I struggled. However, each new day I keep spending with docker makes understanding kubernetes a piece of cake. This guide serves as a strong foundation for analyzing ARP packets in containers.

## Resources

I enjoyed these two articles from Hank Preston, a principal engineer at Cisco. The last one is from me.

- [Exploring Default Docker Networking Part 1](https://blogs.cisco.com/learning/exploring-default-docker-networking-part-1#:~:text=With%20those%20basics%20covered%2C%20let%27s,that%20uses%20the%20bridge%20driver.)
- [Exploring Default Docker Networking Part 2](https://blogs.cisco.com/learning/exploring-default-docker-networking-part-2)
- [A Routers Intimacy With MAC Addresses](https://charlesuneze.substack.com/p/a-routers-intimacy-with-mac-addresses) (I wrote this a few years ago. It discusses ARP and ICMP in a Cisco environment.)