Learn why you can't ping a Kubernetes service
Tags: kubernetes, devops
Published: 2023-01-23 12:05:39

_TL;DR: in this article, you will learn how `ClusterIP` services and kube-proxy work in Kubernetes._

**Have you ever tried to ping a Service IP address in Kubernetes?**

You might have noticed that it doesn't work.

![Pinging Kubernetes services doesn't work](https://res.cloudinary.com/uasabi/image/upload/v1674194605/threads/kube-proxy-no-ping.gif)

Unless it just works.

_Confusing, I know — let me explain._

**Kubernetes Services exist only in etcd.**

There's no process listening on the IP address and port of the Service.

Try to execute `netstat -ntlp` in a node — there's nothing.

![Kubernetes services don't listen to incoming connections](https://res.cloudinary.com/uasabi/image/upload/v1674194605/threads/kube-proxy-netstat.gif)

_How do they work, then?_

Consider a cluster with three Nodes.

The red pod issues a request to the brown service using the IP `172.17.0.1`.

![A cluster with three pods and a ClusterIP service](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-2.png)

But Services don't exist, and [their IP address is only virtual.](https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ip-addressing-mechanism)

_How does the traffic reach one of the pods?_

![If Kubernetes services don't load balance connections, how do they work?](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-3.png)

Kubernetes uses a very clever trick.

Before the request exits from the node, **it is intercepted by iptables rules.**

![The traffic is intercepted by iptables rules when it exits the node](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-4.png)

The iptables rules know that the Service doesn't exist and proceed to replace the IP address of the Service with one of the IP addresses of the Pods belonging to that Service.

![iptables rules intercept and rewrite the traffic to one of the pods](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-5.png)

The destination is a pod IP address, and since [Kubernetes guarantees that any pod can talk to any other pod in the cluster,](https://github.com/kubernetes/design-proposals-archive/blob/main/network/networking.md) the traffic can flow to the brown pod.

![In Kubernetes, any pod can talk to any pod.](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-6.png)

![The packets reach the pod](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-7.png)

_Who is configuring those iptables rules?_

**It's kube-proxy that collects endpoints from the control plane** and maps service IP addresses to pod IPs (it also load balances the connections).

Kube-proxy is a DaemonSet that listens to changes to the Kubernetes API.

_Let's have a look at how it works._

![A kubernetes cluster with three nodes and the state of kube-proxy and control-plane.](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-8.png)

Let's observe what happens when you create a ClusterIP service.

![A new service of type ClusterIP is created](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-9.png)

A fixed virtual IP address is allocated in the control plane, and a companion Endpoint object is created.

The endpoint contains a list of IP addresses and ports where the traffic should be forwarded.

![The control plane updates its state with the new Service and Endpoint object](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-10.png)

**Kube-proxy subscribes to changes to the control plane.**

For every endpoint addition, deletion or update, it is notified.

There's a new Service (and Endpoint object) in this case.

![Kube-proxy is notified of the change and updates its internal state](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-11.png)

**Kube-proxy updates its node with a new list of iptables rules.**

Since there's a kube-proxy for every node in the cluster, each of them will go through the same process.

In the end, the service is "ready".

![Kube-proxy writes the iptables rules on its node](https://res.cloudinary.com/uasabi/image/upload/v1674194603/threads/kube-proxy-clusterip-12.png)

This explains how Service doesn't exist and how kube-proxy sets up load balancing rules on every node but doesn't answer why (sometimes) you can't ping a Service.

The answer is simple: **there's no rule in iptables for ICMP traffic.**

So iptables skips the packets.

But since the Service IP is virtual (it doesn't exist anywhere but etcd), the traffic is not intercepted and goes nowhere.

_So why does it work on my cluster?_

iptables is not the only mechanism to implement a ClusterIP service.

Other options include technologies such as [IPVS](https://projectcalico.docs.tigera.io/networking/use-ipvs) and [eBPF](https://archive.fosdem.org/2020/schedule/event/replacing_iptables_with_ebpf/attachments/slides/3622/export/events/attachments/replacing_iptables_with_ebpf/slides/3622/Cilium_FOSDEM_2020.pdf), which might behave differently (depending on which product you use).

_This is just the tip of the iceberg._

[You can find a more extensive explanation (that includes things like conntrack and Linux namespaces) here.](https://learnk8s.io/kubernetes-network-packets)

And finally, if you've enjoyed this thread, you might also like the Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330