Kubernetes: Service, load balancing, kube-proxy, and iptables
Tags: linux, aws, kubernetes, todayilearned
Published: 2020-11-01 15:29:42

---
title: Kubernetes: Service, load balancing, kube-proxy, and iptables
published: true
date: 2020-11-01 06:23:01 UTC
tags: linux,aws,kubernetes,todayilearned
canonical_url: https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/
---

![](https://cdn-images-1.medium.com/max/730/1*QBuagr7tdmL-Gso3D70cPg.png)

One day I wondered — how is load balancing between pods working in Kubernetes?

I.e. — we have an external Load Balancer. Then a Service. And behind it — Pods.

What happens when we are receiving a network packet from the world, and we have a few pods — how the traffic will be distributed between them?

- [kube-proxy](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#kubeproxy)
- [User space proxy mode](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#User_space_proxy_mode)
- [iptables proxy mode](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#iptables_proxy_mode)
- [IPVS proxy mode](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#IPVS_proxy_mode)
- [kube-proxy config](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#kubeproxy_config)
- [Kubernetes Pod load-balancing](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#Kubernetes_Pod_loadbalancing)
- [AWS LoadBalancer traffic modes](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#AWS_LoadBalancer_traffic_modes)
- [kube-proxy and iptables](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#kubeproxy_and_iptables)
- [iptables rules](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/#iptablesrules)

### kube-proxy

So. the rouning rules between pods between a Service and its Pods are controlled by the `kube-proxy` service that can be working in one of the three following modes - **_user space proxy mode_** , **_iptables proxy mode_** _,_ and **_IPVS proxy mode_**.

#### User space proxy mode

Links:

- [User space proxy mode](https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace)
- [Implementation: proxy via userspace socket](https://arthurchiao.art/blog/cracking-k8s-node-proxy/#ch_4)

A deprecated mode, previously was the default.

When using this mode, `kube-proxy` watch for changes in a cluster and for each new Service will open a TCP port on a WorkerNode.

Then, iptables on this WorkerNode begin routing traffic from this port to the `kube-proxy` service which is acting as a proxy service using the [round-robin](https://rtfm.co.ua/nginx-http-proksi-load-balancing-bufery-i-keshirovanie/#nginx_upstream_chg) approach, i.e. by sending traffic to a next pod in its backend's list. During this, `kube-proxy` can try to send a package to another pod if the first one didn't respond.

![](https://cdn-images-1.medium.com/max/966/0*m2dqKXTsMlz0-70y.png)

#### iptables proxy mode

Links:

- [iptables proxy mode](https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables)
- [Implementation: proxy via iptables](https://arthurchiao.art/blog/cracking-k8s-node-proxy/#ch_5)

Our case, which we will investigate in this post. Currently, is the default one.

When using this mode, `kube-proxy` watch for changes in a cluster and for each new Service will open a TCP port on a WorkerNode.

Then, the `iptables` on this WorkerNode sends traffic from this port to a Kubernetes Service which is actually a chain in the `iptables` rules, and via this chain, traffic goes directly to pods which are a backend for this Service. During this, a targeting pod is selected randomly.

This mode is less expensive for system resources as all necessary operations are performed in the kernel by the [`netfilter`](https://www.netfilter.org/) module. Also, this mode works faster and is more reliable because there is no a "middle-ware" - the `kube-proxy` itself.

But if the first pod where a packet was sent did not respond — then a connection fails, while in the _user space proxy mode_ `kube-proxy` will try to send it to another pod.

This is why is so important to properly configure [Readiness Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/), so Kubernetes will not send traffic to pods that are not ready to accept it.

Furthermore, this mode is more complicated for debugging, because in the _user space proxy mode_ the `kube-proxy` will write its logs to the `/var/log/kube-proxy`, while with the `netfilter` you have to go to debug into the kernel itself.

![](https://cdn-images-1.medium.com/max/976/0*sT8v61ppVjmj2YE2.png)

#### IPVS proxy mode

Links:

- [IPVS proxy mode](https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs)
- [Implementation: proxy via IPVS](https://arthurchiao.art/blog/cracking-k8s-node-proxy/#ch_6)
- [IPVS-Based In-Cluster Load Balancing Deep Dive](https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/)

And the most recent mode. It uses the [`netlink`](https://man7.org/linux/man-pages/man7/netlink.7.html) kernel module and creates new IPVS rules for new Kubernetes Services.

The main is the diversity of the load-balaning modes:

- `rr`: round-robin
- `lc`: least connection (smallest number of open connections)
- `dh`: destination hashing
- `sh`: source hashing
- `sed`: shortest expected delay
- `nq`: never queue

![](https://cdn-images-1.medium.com/max/976/0*m5soTJ9RFwWMZACY.png)

#### `kube-proxy` config

Let’s go to check which mode is used in our case, in the AWS Lastic Kubernetes Service cluster.

Find `kube-proxy` pods:

```
$ kubectl -n kube-system get pod -l k8s-app=kube-proxy -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-proxy-4prtt 1/1 Running 1 158d 10.3.42.245 ip-10–3–42–245.us-east-2.compute.internal <none> <none>
kube-proxy-5b7pd 1/1 Running 0 60d 10.3.58.133 ip-10–3–58–133.us-east-2.compute.internal <none> <none>
…
kube-proxy-tgl52 1/1 Running 0 59d 10.3.59.159 ip-10–3–59–159.us-east-2.compute.internal <none> <none>
```

On every WorkerNode of the cluster, we have a dedicated `kube-proxy` instance with the _kube-proxy-config_ ConfigMap attached:

```
$ kubectl -n kube-system get pod kube-proxy-4prtt -o yaml
apiVersion: v1
kind: Pod
…
spec:
…
containers:
…
volumeMounts:
…
- mountPath: /var/lib/kube-proxy-config/
name: config
…
volumes:
…
- configMap:
defaultMode: 420
name: kube-proxy-config
name: config
```

Check this ConfigMap content:

```
$ kubectl -n kube-system get cm kube-proxy-config -o yaml
apiVersion: v1
data:
config: |-
…
mode: “iptables”
…
```

Now, when we are more familiar with the `kube-proxy` modes - let's go deeper to see how it works and what `iptables` is doing here.

### Kubernetes Pod load-balancing

For our journey, let’s take a real application with an Ingress (AWS Application Load Balancer, ALB) which sends traffic to a Kubernetes Service:

```
$ kubectl -n eks-dev-1-appname-ns get ingress appname-backend-ingress -o yaml
…
- backend:
serviceName: appname-backend-svc
servicePort: 80
…
```

Check the Service itself:

```
$ kubectl -n eks-dev-1-appname-ns get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
appname-backend-svc NodePort 172.20.249.22 <none> 80:31103/TCP 63d
```

Here we have the NodePorttype - it's listening to a TCP port on a WorkerNode.

The ALB _e172ad3e-eksdev1appname-abac_ ALB routes the traffic from clients to the _e172ad3e-4caa286edf23ff7e06d_ AWS TargetGroup:

![](https://cdn-images-1.medium.com/max/1024/0*kTwik3EAEbQOH0WH.png)

ЕС2 in this TargetGroup are listening to the _31103_ port which we saw in the Service details above:

![](https://cdn-images-1.medium.com/max/1024/0*2rcyvDMhAhtsEIcD.png)

#### AWS LoadBalancer traffic modes

[Documentation>>>](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html).

A side note: AWS ALB supports two modes for traffic — IP, and Instance Mode.

- **_Instance mode_** : the default mode, requires a Kubernetes Service to have the NodePort type, and routes traffic to a TCP port of a WorkerNode
- **_IP mode_** : with this mode targets for an ALB are Kubernetes Pods directly instead of the Kubernetes Worker Nodes.

Now, we need to have an access to one of these nodes — connect to a Bastion host, and then to one of the WorkerNode:

```
ubuntu@ip-10–3–29–14:~$ ssh ec2-user@10.3.49.200 -i .ssh/bttrm-eks-nodegroup-us-east-2.pem
Last login: Thu May 28 06:25:27 2020 from ip-10–3–29–32.us-east-2.compute.internal
__|__ |_ )
_| ( / Amazon Linux 2 AMI
___|\___ |___|
39 package(s) needed for security, out of 64 available
Run “sudo yum update” to apply all updates.
[ec2-user@ip-10–3–49–200 ~]$ sudo -s
[root@ip-10–3–49–200 ec2-user]#
```

#### `kube-proxy` and `iptables`

So, a packet from a client came to the WorkerNode.

On this node, the `kube-proxy` service is binding on the port allocated so no one another service will use it, and also it creates a set of `iptables` rules:

```
[root@ip-10–3–49–200 ec2-user]# netstat -anp | grep 31103
tcp6 0 0 :::31103 :::* LISTEN 4287/kube-proxy
```

The packet comes to the _31107_ port, where it’s started following by the iptables filters.

#### `iptables` rules

Links:

- [Traversing of tables and chains](https://www.frozentux.net/iptables-tutorial/chunkyhtml/c962.html)
- [Packet flow in Netfilter and General Networking](https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg)
- [A Deep Dive into Iptables and Netfilter Architecture](https://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture)

The Linux kernel accepts this packet and sends it to the `PREROUTING` chain of the `nat` table:

![](https://cdn-images-1.medium.com/max/807/0*NtoF1xYa0rR0Rpjf.png)

See the [describing kube-proxy iptables rules](https://twitter.com/thockin/status/1191766983735296000).

Check rules in the `nat` table and its `PREROUTING` chain:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L PREROUTING | column -t
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all — anywhere anywhere /* kubernetes service portals */
```

Here we have a target to the following chain - the `KUBE-SERVICES`, which have the next chain - `KUBE-NODEPORTS` as the last rule, which captures packets for a Service with the NodePort type:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L KUBE-SERVICES -n | column -t
…
KUBE-NODEPORTS all — 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in th
is chain */ ADDRTYPE match dst-type LOCAL
…
```

Check rules in this chain:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L KUBE-NODEPORTS -n | column -t | grep 31103
KUBE-MARK-MASQ tcp — 0.0.0.0/0 0.0.0.0/0 /* eks-dev-1-appname-ns/appnamed-backend-svc: */ tcp dpt:31103
KUBE-SVC-TII5GQPKXWC65SRC tcp — 0.0.0.0/0 0.0.0.0/0 /* eks-dev-1-appname-ns/appname-backend-svc: */ tcp dpt:31103
```

And here it is intercepting packets for the `dpt:31103` (_destination port 31103_) and they are sent to the next chain - `KUBE-SVC-TII5GQPKXWC65SRC`, check it now:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L KUBE-SVC-TII5GQPKXWC65SRC | column -t
Chain KUBE-SVC-TII5GQPKXWC65SRC (2 references)
target prot opt source destination
KUBE-SEP-N36I6W2ULZA2XU52 all — anywhere anywhere statistic mode random probability 0.50000000000
KUBE-SEP-4NFMB5GS6KDP7RHJ all — anywhere anywhere
```

Here we can see the next two chains where is the “routing magic” happens — the packet randomly will be sent to one of these chains, each has 0.5 from 1.0 “weight”  —  `statistic mode random probability 0.5`, as per the official Kubernetes [documentation](https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables):

> By default, kube-proxy in iptables mode chooses a backend at random.

See also [Turning IPTables into a TCP load balancer for fun and profit](https://scalingo.com/blog/iptables).

Check those chains.

The first one:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L KUBE-SEP-N36I6W2ULZA2XU52 -n | column -t
Chain KUBE-SEP-N36I6W2ULZA2XU52 (1 references)
target prot opt source destination
KUBE-MARK-MASQ all — 10.3.34.219 0.0.0.0/0
DNAT tcp — 0.0.0.0/0 0.0.0.0/0 tcp to:10.3.34.219:3001
```

And the second one:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L KUBE-SEP-4NFMB5GS6KDP7RHJ -n | column -t
Chain KUBE-SEP-4NFMB5GS6KDP7RHJ (1 references)
target prot opt source destination
KUBE-MARK-MASQ all — 10.3.57.124 0.0.0.0/0
DNAT tcp — 0.0.0.0/0 0.0.0.0/0 tcp to:10.3.57.124:3001
```

And here we can see that the `DNAT` (Destination NAT) chain is sending the packet to an IP and _3001_ port, which is actually our `ContainerPort` - check the Deployment:

```
$ kubectl -n eks-dev-1-appname-ns get deploy appname-backend -o json | jq ‘.spec.template.spec.containers[].ports[].containerPort’
3001
```

And go to see our pods IPs.

Find the pods:

```
$ kubectl -n eks-dev-1-appname-ns get pod
NAME READY STATUS RESTARTS AGE
appname-backend-768ddf9f54–2nrp5 1/1 Running 0 3d
appname-backend-768ddf9f54-pm9bh 1/1 Running 0 3d
```

And IP of the first pod:

```
$ kubectl -n eks-dev-1-appname-ns get pod appname-backend-768ddf9f54–2nrp5 — template={{.status.podIP}}
10.3.34.219
```

And the second one:

```
$ kubectl -n eks-dev-1-appname-ns get pod appname-backend-768ddf9f54-pm9bh — template={{.status.podIP}}
10.3.57.124
```

Isn’t it great? :-) So simple — and so great.

Now, let’s try to scale our Deployment to see how the `KUBE-SVC-TII5GQPKXWC65SRC` will be changed to reflect the scaling.

Find the Deployment:

```
$ kubectl -n eks-dev-1-appname-ns get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
appname-backend 2/2 2 2 64d
```

Scale it from two to three pods:

```
$ kubectl -n eks-dev-1-appname-ns scale deploy appname-backend — replicas=3
deployment.extensions/appname-backend scaled
```

Check the iptables rules:

```
[root@ip-10–3–49–200 ec2-user]# iptables -t nat -L KUBE-SVC-TII5GQPKXWC65SRC | column -t
Chain KUBE-SVC-TII5GQPKXWC65SRC (2 references)
target prot opt source destination
KUBE-SEP-N36I6W2ULZA2XU52 all — anywhere anywhere statistic mode random probability 0.33332999982
KUBE-SEP-HDIQCDRXRXBGRR55 all — anywhere anywhere statistic mode random probability 0.50000000000
KUBE-SEP-4NFMB5GS6KDP7RHJ all — anywhere anywhere
```

Now we can see that our `KUBE-SVC-TII5GQPKXWC65SRC` got three rules: the first one with the 0.33332999982 random, as there are two more rules after, then the second one with the 0.5 weight, and the last one - without rules at all.

Check the [iptables statistics module](http://ipset.netfilter.org/iptables-extensions.man.html#lbCD).

Actually, “_That’s all folks!_” ©

### Useful links

- [Kubernetes: ClusterIP vs NodePort vs LoadBalancer, Services и Ingress — обзор, примеры](https://rtfm.co.ua/kubernetes-clusterip-vs-nodeport-vs-loadbalancer-services-i-ingress-obzor-primery/)
- [Kubernetes Networking Demystified: A Brief Guide](https://www.stackrox.com/post/2020/01/kubernetes-networking-demystified/)
- [A Deep Dive into Iptables and Netfilter Architecture](https://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture)
- [Traversing of tables and chains](https://www.frozentux.net/iptables-tutorial/chunkyhtml/c962.html)
- [Turning IPTables into a TCP load balancer for fun and profit](https://scalingo.com/blog/iptables)
- [Cracking kubernetes node proxy (aka kube-proxy)](https://arthurchiao.art/blog/cracking-k8s-node-proxy/)
- [A minimal IPVS Load Balancer demo](https://medium.com/@benmeier_/a-quick-minimal-ipvs-load-balancer-demo-d5cc42d0deb4)
- [How to enable IPVS mode on AWS EKS?](https://medium.com/@selfieblue/how-to-enable-ipvs-mode-on-aws-eks-7159ec676965)
- [Load-Balancing in Kubernetes](https://rancher.com/load-balancing-in-kubernetes)

_Originally published at_ [_RTFM: Linux, DevOps and system administration_](https://rtfm.co.ua/en/kubernetes-service-load-balancing-kube-proxy-and-iptables/)_._

* * *