Kubernetes tutorial - Static Pods Explained
Tags: kubernetes
Published: 2022-05-22 06:57:22

<!-- wp:paragraph -->
<p>Are you ready for some Kubernetes fun? You know a few minutes on a Kubernetes tutorial wouldn't be a waste.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Alright, let's start with a Kubernetes buzzword - <strong>Pods.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://kubernetes.io/docs/concepts/workloads/pods/">Pods</a> are the most granular resource you can manage in a Kubernetes cluster by using Kubernetes control plane. Pods can contain one or more containers, where one will deploy applications. You may also mount storage to your pods. And the pods have their own ephemeral IPs. And more. But let's not get lost in the details of a Pod definition. Our goal is to understand what Static Pods are.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://raw.githubusercontent.com/kubernetes/community/master/icons/png/resources/labeled/pod-256.png" alt=""/></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>First, let's take a look at a simple example of creating a pod using <strong>kubectl</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/home/ubuntu/pods# cat test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test-pod
  name: test-pod
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: test-pod

root:/home/ubuntu/pods# kubectl apply -f test-pod.yaml
pod/test-pod created

root:/home/ubuntu/pods# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          20s</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Nothing fancy going on here - a pod resource was created using a simple yaml file which has the pod specifications.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But what is not obvious is that <strong>kubectl</strong> has interacted with Kubernetes API Server (also referred to as<strong> Kube-apiserver</strong>) which in turn communicates with a <strong>kube-scheduler </strong>component to find the best node to create the pod resource on and then instructs a <strong>kubelet</strong> agent sitting on that particular node to create the pod. Then all the task details are stored in an <strong>etcd</strong> database. This is a typical workflow that happens mostly.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>By the way, a couple of ways to validate me ;)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph {"fontSize":"normal"} -->
<p class="has-normal-font-size"><a href="https://www.youtube.com/c/ExecuteonCommand/videos">Subscribe to my YouTube channel</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph {"fontSize":"normal"} -->
<p class="has-normal-font-size"><a href="https://www.linkedin.com/in/bharathramdba/">Connect with me on LinkedIn</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>The components like kube-apiserver, kube-scheduler, kube-controller-manager, and etcd database together form the control plane of a Kubernetes cluster. </strong></p>
<!-- /wp:paragraph -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://d33wubrfki0l68.cloudfront.net/2475489eaf20163ec0f54ddc1d92aa8d4c87c96b/e7c81/images/docs/components-of-kubernetes.svg" alt="Components of Kubernetes"/><figcaption>Source: <a href="https://kubernetes.io/docs/concepts/overview/components/">Kubernetes Components | Kubernetes</a></figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>While the (normal) pods are created in response to the instruction from kube-apiserver by kubelet, kubelet itself can create and manage pods on its own. The type of pods that are managed by a kubelet is called <strong>static pods</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Kubelet Service</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To understand further, we will start by looking at how a kubelet service is configured. Then we will see how to get a kubelet agent to create a static pod.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>My test/lab Kubernetes cluster runs on Ubuntu operating system. I believe any linux distribution should look pretty similar to what we are going to see here. Just to be clear, we are observing one of my worker nodes, where Kubernetes resources are commonly created. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/home/ubuntu# ps -ef|grep kubelet
root       820     1  2 May21 ?        00:12:57 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=node1 --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>/var/lib/kubelet/config.yaml</strong> file is particular important because it contains a line that will help us to create static pods. More specifically, a key-value pair. I am referring to <strong>staticPodPath</strong> highlighted below.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/home/ubuntu# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
<strong>staticPodPath: /etc/kubernetes/manifests</strong>
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>One way to create a static pod</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We can simply create a pod yaml file, which has the static pod definition, in the directory '<strong>/etc/kubernetes/manifests</strong>'. Kubelet will create the static pod shortly. Let's try that.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code># list the pods prior to creating a pod yaml file
root:/etc/kubernetes/manifests# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
test-pod           1/1     Running   0          21m

# this is the file I created in the directory we found before.
root:/etc/kubernetes/manifests# cat static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: static-pod
  name: static-pod
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: static-pod

# static pod gets created. kubelet appends the node name to the pod name. Interesting.
root:/etc/kubernetes/manifests# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
static-pod-node1   1/1     Running   0          26s
test-pod           1/1     Running   0          21m</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>What if we removed the yaml file in the manifests directory?</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/etc/kubernetes/manifests# rm static-pod.yaml

root:/etc/kubernetes/manifests# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
static-pod-node1   1/1     Running   0          8m51s
test-pod           1/1     Running   0          30m

root:/etc/kubernetes/manifests# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          30m</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Well, the static pod gets removed automatically.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Another way to create a static pod</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Alternatively, we can create the pod yaml file in a different directory as shown below.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:<strong>/etc/kubelet.d</strong># cat static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: static-pod
  name: static-pod
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: static-pod</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>If we observe the status of kubelet service using a systemd command, we will see an additional configuration file being used from a <strong>Drop-In directory. I think of this directory as a place that contains the override configuration of particularly kubelet service.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root@ip-172-31-27-42:/etc/kubelet.d# <strong>systemctl status kubelet.service</strong>
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
<strong>  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf</strong>
   Active: active (running) since Sat 2022-05-21 19:02:44 UTC; 10h ago
     Docs: https://kubernetes.io/docs/home/</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let's dig this file - <strong>/etc/systemd/system/kubelet.service.d/10-kubeadm.conf</strong></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/etc/kubelet.d# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
&#91;Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-<strong>/etc/default/kubelet</strong>
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let's add an extra argument to "<strong>/etc/default/kubelet</strong>" file such that kubelet will be started with a new manifest path. And restart kubelet service. You will see 'static-pod-node1' gets created.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>#Do this on a worker node
#Add new pod manifest path
root:/etc/kubelet.d# cat <strong>/etc/default/kubelet</strong>
<strong>KUBELET_EXTRA_ARGS="--pod-manifest-path=/etc/kubelet.d/"</strong>

#list pods
root:/etc/kubelet.d# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          47m

#restart kubelet service
root:/etc/kubelet.d# <strong>systemctl restart kubelet.service</strong>

root:/etc/kubelet.d# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
<strong>static-pod-node1   1/1     Running   0          14s</strong>
test-pod           1/1     Running   0          52m

#Observe kubelet was restarted with the new argument you added.
root:/etc/kubelet.d# ps -ef|grep kubelet
root     20820     1 13 06:13 ?        00:00:00 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=node1 --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2 <strong>--pod-manifest-path=/etc/kubelet.d/</strong></code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>By the way, do you know the control plane components we looked at before run as pods as well in a namespace called kube-system.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/home/ubuntu# kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS         AGE
etcd-master                               1/1     Running   8 (5d11h ago)    53d
kube-apiserver-master                     1/1     Running   8 (5d11h ago)    53d
kube-controller-manager-master            1/1     Running   8 (5d11h ago)    53d
kube-scheduler-master                     1/1     Running   8 (5d11h ago)    53d</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>At this point we understand the normal pods are created by kubelet as per the instruction from kube-apiserver. But then who told kubelet to create kube-apiserver pod? Obviously it would not have been kube-apiserver, because it didn't exist.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Well, kube-apiserver and other control plane components' pods are indeed created as static pods by kubelet. Because during the installation, the YAML files of these components are created in /etc/kubernetes/manifests directory. Now you know why we need static pods. :)</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>root:/home/ubuntu/pods# ls -l /etc/kubernetes/manifests/
total 16
-rw------- 1 root root 2209 Mar 29 14:49 etcd.yaml
-rw------- 1 root root 3854 Mar 29 14:49 kube-apiserver.yaml
-rw------- 1 root root 3251 Mar 29 14:49 kube-controller-manager.yaml
-rw------- 1 root root 1435 Mar 29 14:49 kube-scheduler.yaml</code></pre>
<!-- /wp:code -->