Tracing pod to pod network traffic in Kubernetes
Tags: kubernetes, devops
Published: 2023-05-09 06:42:25

_How does Pod to Pod communication work in Kubernetes?_

_How does the traffic reach the pod?_

**In this article, you will dive into how low-level networking works in Kubernetes.**

Let's start by focusing on the pod and node networking.

When you deploy a Pod, the following things happen:

1. The pod gets its own network namespace.
1. An IP address is assigned.
1. Any containers in the pod share the same networking namespace and can see each other on localhost.

![Network namespaces in a Kubernetes node](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-2_ljm7yb.png)

**A pod must first have access to the node's root namespace to reach other pods.**

This is achieved using a virtual eth pair connecting the 2 namespaces: pod and root.

The bridge allows traffic to flow between virtual pairs and traverse through the common root namespace.

![Bridge that connects all containers in the node](https://res.cloudinary.com/learnk8s/image/upload/v1682303466/threads/intranode-3_l2gzvp.png)

_So what happens when Pod-A wants to send a message to Pod-B?_

Since the destination isn't one of the containers in the namespace Pod-A sends out a packet to its default interface `eth0`.

This interface is tied to the veth pair and packets are forwarded to the root namespace.

![Tracing the flow: starting from the container](https://res.cloudinary.com/learnk8s/image/upload/v1682303466/threads/intranode-4_rrd81s.png)

The ethernet bridge, acting as a virtual switch, has to somehow resolve the destination pod IP (Pod-B) to its MAC address.

![The packet reaches the cni0 bridge](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-5_do5st6.png)

**The ARP protocol comes to the rescue.**

When the frame reaches the bridge, an ARP broadcast is sent to all connected devices.

The bridge shouts "Who has Pod-B IP address?"

![ARP queries](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-6_s8kcgz.png)

A reply is received with the interface's MAC address that connects Pod-B, which is stored in the bridge ARP cache (lookup table).

![ARP reply](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-7_wmsmpu.png)

Once the IP and MAC address mapping is stored, the bridge looks up in the table and forwards the packet to the correct endpoint.

The packet reaches Pod-B veth in the root namespace, and from there, it quickly reaches the eth0 interface inside the Pod-B namespace.

![The packet reaches the other pod](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-8_widdzl.png)

With this, the communication between Pod-A and Pod-B has been successful.

An additional hop is required for pods to communicate across different nodes, as the packets have to travel through the node network to reach their destination.

![Tracing pod traffic across nodes](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-9_iciy58.png)

This is the "plain" networking version.

_How does this change when you install a CNI plugin that uses an overlay network?_

Let's take Flannel as an example.

**Flannel installs a new interface between the node's `eth0` and the container bridge `cni0`.**

All traffic flowing through this interface is encapsulated (e.g. VXLAN, Wireguard, etc.).

![The Flannel interface encapsulates the traffic](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-10_btire7.png)

The new packets don't have pods' IP addresses as source and destination, but nodes' IPs.

So the wrapper packet will exit from the node and travel to the destination node.

![Packets are encpsulated using different backends](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-11_lxkvs4.png)

Once on the other side, the `flannel.1` interface unwraps the packet and lets the original pod-to-pod packet reach its destination.

![The packet is unwrapped and forwarded to the interface](https://res.cloudinary.com/learnk8s/image/upload/v1682303467/threads/intranode-12_lwj7cw.png)

_How does Flannel know where all the Pods are located and their IP addresses?_

On each node, the Flannel daemon syncs the IP addresses allocations in a distributed database.

Other instances can query this database to decide where to send those packets.

![The flannel deaemons sync IP addresses to a (distributed) database](https://res.cloudinary.com/learnk8s/image/upload/v1682303468/threads/intranode-13_bqjynv.png)

Here are a few links if you want to learn more:

- https://learnk8s.io/kubernetes-network-packets
- https://zhuanlan.zhihu.com/p/340747753
- https://blog.laputa.io/kubernetes-flannel-networking-6a1cb1f8ec7c
- https://www.sobyte.net/post/2022-07/k8s-flannel/

And finally, if you've enjoyed this thread, you might also like:

- The Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training
- This collection of past threads https://twitter.com/danielepolencic/status/1298543151901155330
- The Kubernetes newsletter I publish every week https://learnk8s.io/learn-kubernetes-weekly