What happens when you create a Pod in Kubernetes
Tags: kubernetes, devops
Published: 2023-05-30 17:54:17

What happens when you create a Pod in Kubernetes?

A surprisingly simple task reveals a complicated workflow that touches several components in the cluster.

Let's start with the obvious: **kubectl sends the YAML definition to the API server.**

In this step, kubectl:

- Discovers the API endpoints using OpenAPI (Swagger).
- Negotiates the resource version.
- Validates the YAML.
- Issues the request.

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-2_rx9jaj.png)

When the request reaches the API, it goes through the following:

- **Authentication & authorization.**
- **Admission controllers.**

In the last step, it's finally stored in etcd.

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-3_mdzyec.png)

After this, the pod is added to the scheduler queue.

**The scheduler filters and scores the nodes to find the best one.**

And it finally binds the pod to the node.

The binding is written in etcd.

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-4_xiwltw.png)

At this point, the pod exists only in etcd as a record.

**The infrastructure hasn't created any containers yet.**

Here's where the kubelet takes over.

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-5_gylrqt.png)

The kubelet pulls the Pod definition and proceeds to delegate:

1. Network creation to the CNI (e.g. Cilium).
1. Container creation to the CRI (e.g. containerd).
1. Storage creation to the CSI (e.g. OpenEBS).

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-6_xcwqod.png)

Among other things, the Kubelet will execute the Pod's probes and, when the Pod is running, report its IP address to the control plane.

**That IP and the containers' ports are stored as endpoints in etcd.**

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-7_gy4hqr.png)

_Wait… endpoint what?_

In Kubernetes:

- endpoint is a 10.0.0.2:3000 (IP:port) pair.
- Endpoint is a collection of endpoints (a list of IP:port pairs).

For every Service in the cluster, **Kubernetes creates an Endpoint object with endpoints.**

_Confusing, isn't it?_

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661220/threads/create-pod-8_mzr9n8.png)

The endpoints (IP:port) are used by:

- kube-proxy to set iptables rules.
- CoreDNS to update the DNS entries.
- Ingress controllers to set up downstreams.
- Service meshes.
- And more operators.

As soon as an endpoint is added, the components are notified.

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661221/threads/create-pod-9_yzij65.png)

When the endpoint (IP:port) is propagated, you can finally start using the Pod!

_What happens when you delete a Pod?_

The exact process but in reverse.

This is annoying because there are few opportunities for race conditions.

The correct sequence is:

1. App stops accepting connections.
1. Controllers (kube-proxy, ingress, etc.) to remove the endpoint.
1. App to drain existing connection.
1. App to shut down.

![](https://res.cloudinary.com/learnk8s/image/upload/v1675661221/threads/create-pod-10_uuogt7.png)

If you want to learn more about the graceful shutdown in Kubernetes, you can find my article here https://learnk8s.io/graceful-shutdown

And finally, if you've enjoyed this thread, you might also like the Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330