Kubernetes: Pods and WorkerNodes — control the placement of the Pods on the Nodes
Tags: kubernetes, devops, todayilearned
Published: 2023-09-03 10:34:14

---
title: Kubernetes: Pods and WorkerNodes — control the placement of the Pods on the Nodes
published: true
date: 2023-08-19 08:45:37 UTC
tags: kubernetes,devops,todayilearned
canonical_url: https://rtfm.co.ua/en/kubernetes-pods-and-workernodes-control-the-placement-of-the-pods-on-the-nodes/
---

### Kubernetes: Pods and WorkerNodes — control the placement of the Pods on the Nodes

![](https://cdn-images-1.medium.com/max/1024/1*wOMtZQ4cb_gxIZ1HkiKOfw.png)

Kubernetes allows very flexible control over how its Pods will be located on servers, i.e. WorkerNodes.

This can be useful if you need to run a pod on a specific node configuration, for example — a WorkerNode must have a GPU, or an SSD instead of an HDD. Another example is when you need to place individual Pods next to each other to reduce their communication latency, or to reduce cross Availability-zone traffic (see [AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity](https://rtfm.co.ua/en/aws-grafana-loki-interzone-traffic-in-aws-and-kubernetes-nodeaffinity/)).

And, of course, this is important for building a High Availability and Fault Tolerance architecture, when you need to divide pods into individual nodes or Availability Zones.

We have four main approaches to control how Kubernetes Pods are hosted on WorkerNodes:

- configure Nodes in such a way that they will accept only individual Pods that meet the criteria specified on the node
- `taints` and `tolerations`: on the Node we set the _taint_, for which Pods must have the appropriate _toleration_ to run on this node
- configure the Pod itself in such a way that it will select only individual Nodes that meet the criteria specified in the Pod
- for this, we can use `nodeName` – only a Node with the specified name is selected
- or `nodeSelector` to select Nodes with corresponding labels and their values
- or `nodeAffinity` and `nodeAntiAffinity` - the rules by which Kubernetes Scheduler will choose a Node to launch the Pod depending on the parameters of this Node
- configure the Pod itself so that it will select a Node based on how other Pods are running
- for this, we can use `podAffinity` and `podAntiAffinity` - the rules by which Kubernetes Scheduler will choose a Node to launch the Pod depending on the other Pods on this Node
- and a separate topic — Pod Topology Spread Constraints, i.e. the rules for placing Pods by failure domains — regions, Availability zones, or nodes

### kubectl explain

Just a tip: you can always read the relevant documentation for any parameter or resource using `kubectl explain`:

```
$ kubectl explain pod
KIND: Pod
VERSION: v1
DESCRIPTION:
Pod is a collection of containers that can run on a host. This resource is
created by clients and scheduled onto hosts.
…
```

Or:

```
$ kubectl explain Pod.spec.nodeName
KIND: Pod
VERSION: v1
FIELD: nodeName <string>
DESCRIPTION:
NodeName is a request to schedule this pod onto a specific node. If it is
non-empty, the scheduler simply schedules this pod onto that node, assuming
that it fits resource requirements.
```

### Node Taints and Pods Tolerations

So, the first option is to set restrictions on the Node on what Pods can be run on it using [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).

Here a `taint` “repels” Pods that do not have a corresponding `toleration` to that Node, and a `toleration` “pulls” a Pod to a specific Node that has a corresponding one `taint`.

For example, we can create a Node on which only Pods with some critical services such as controllers will be launched.

To do so, specify a `tain` with the `effect: NoSchedule` - that is, prohibit the creation of new Pods on this Node:

```
$ kubectl taint nodes ip-10–0–3–133.ec2.internal critical-addons=true:NoSchedule
node/ip-10–0–3–133.ec2.internal tainted
```

Next, create a Pod with a `toleration` with the key `"critical-addons"`:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  tolerations:
    - key: "critical-addons"
      operator: "Exists"
      effect: "NoSchedule"
```

Deploy, and check Pods on that Node:

```
$ kubectl get pod --all-namespaces -o wide --field-selector spec.nodeName=ip-10–0–3–133.ec2.internal
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default my-pod 1/1 Running 0 2m11s 10.0.3.39 ip-10–0–3–133.ec2.internal <none> <none>
dev-monitoring-ns atlas-victoriametrics-loki-logs-zxd9m 2/2 Running 0 10m 10.0.3.8 ip-10–0–3–133.ec2.internal <none> <none>
…
```

But where does Loki come from? Because while the Taint was set, the Scheduler managed to move a Loki’s Pod to this Node.

To prevent this, add a key `NoExecute` to the Tain - then the scheduler will perform [Pod eviction](https://sysdig.com/blog/kubernetes-pod-evicted/) to move already running Pods from this Node to other Nodes:

```
$ kubectl taint nodes ip-10–0–3–133.ec2.internal critical-addons=true:NoExecute
```

Check `taints` now:

```
$ kubectl get node ip-10–0–3–133.ec2.internal -o json | jq '.spec.taints'
[
{
“effect”: “NoExecute”,
“key”: “critical-addons”,
“value”: “true”
},
{
“effect”: “NoSchedule”,
“key”: “critical-addons”,
“value”: “true”
}
]
```

For our Pod add the second one `toleration`, otherwise it will be evicted from this Node too:

```
...
  tolerations:
    - key: "critical-addons"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "critical-addons"
      operator: "Exists"
      effect: "NoExecute"
```

Deploy and check Pods on this Node again:

```
$ kubectl get pod --all-namespaces -o wide --field-selector spec.nodeName=ip-10–0–3–133.ec2.internal
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default my-pod 1/1 Running 0 3s 10.0.3.246 ip-10–0–3–133.ec2.internal <none> <none>
kube-system aws-node-jrsjz 1/1 Running 0 16m 10.0.3.133 ip-10–0–3–133.ec2.internal <none> <none>
kube-system csi-secrets-store-secrets-store-csi-driver-cctbj 3/3 Running 0 16m 10.0.3.144 ip-10–0–3–133.ec2.internal <none> <none>
kube-system ebs-csi-node-46fts 3/3 Running 0 16m 10.0.3.187 ip-10–0–3–133.ec2.internal <none> <none>
kube-system kube-proxy-6ztqs 1/1 Running 0 16m 10.0.3.133 ip-10–0–3–133.ec2.internal <none> <none>
```

Now, on this Node, we have only our Pod, and Pods from DaemonSets which by default should run on all Nodes and have the corresponding `tolerations`, see [How Daemon Pods are scheduled](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#how-daemon-pods-are-scheduled).

In addition to the `Exists` that only checks for the presence of a specified label, it is possible to check the value of this label.

To do so, use `Equal` in the operator, and add a required value:

```
...
  tolerations:
    - key: "critical-addons"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "critical-addons"
      operator: "Equal"
      value: "true"
      effect: "NoExecute"
```

To delete a `tain` - add a minus at the end:

```
$ kubectl taint nodes ip-10–0–3–133.ec2.internal critical-addons=true:NoSchedule-
node/ip-10–0–3–133.ec2.internal untainted

$ kubectl taint nodes ip-10–0–3–133.ec2.internal critical-addons=true:NoExecute-
node/ip-10–0–3–133.ec2.internal untainted
```

### Choosing a Node by a Pod: `nodeName`, `nodeSelector`, and `nodeAffinity`

Another approach is when we configure a Pod in such a way that “it” chooses which Node to run on.

For this we have `nodeName`, `nodeSelector`, `nodeAffinity` and `nodeAntiAffinity`. See [Assign Pods to Nodes](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/).

#### `nodeName`

The most straightforward way. Has precedence over all others:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  nodeName: ip-10-0-3-133.ec2.internal
```

#### `nodeSelector`

With the `nodeSelector` we can choose Nodes which has a corresponding labels.

Add a label to the Node:

```
$ kubectl label nodes ip-10–0–3–133.ec2.internal service=monitoring
node/ip-10–0–3–133.ec2.internal labeled
```

Check it:

```
$ kubectl get node ip-10–0–3–133.ec2.internal -o json | jq '.metadata.labels'
{
…
“kubernetes.io/hostname”: “ip-10–0–3–133.ec2.internal”,
“kubernetes.io/os”: “linux”,
“node.kubernetes.io/instance-type”: “t3.medium”,
“service”: “monitoring”,
…
```

In the Pod’s manifest set the `nodeSelector`:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  nodeSelector:
    service: monitoring
```

If several labels are assigned in the Pod’s `nodeSelector`, then the corresponding Node must have all these labels in order for this Pod to run on it.

#### `nodeAffinity` and `nodeAntiAffinity`

`nodeAffinity` and `nodeAntiAffinity` operate in the same way as the `nodeSelector`, but have more flexible capabilities.

For example, you can set hard or soft launch limits — for a soft limit, the scheduler will try to launch a Pod on the corresponding Node, and if it cannot, it will launch it on another. Accordingly, if you set a hard limit and the scheduler cannot start the Pod on the selected Node, the Pod will remain in Pending status.

The hard limit is set in the field `.spec.affinity.nodeAffinity` with the `requiredDuringSchedulingIgnoredDuringExecution`, and the soft limit is set with the `preferredDuringSchedulingIgnoredDuringExecution`.

For example, we can launch a Pod in AvailabilityZone _us-east-1a_ or _us-east-1b_ using node-label `topology.kubernetes.io/zone`:

```
$ kubectl get node ip-10–0–3–133.ec2.internal -o json | jq '.metadata.labels'
{
…
“topology.kubernetes.io/region”: “us-east-1”,
“topology.kubernetes.io/zone”: “us-east-1b”
}
```

Set a hard-limit:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-east-1a
            - us-east-1b
```

Or a soft limit. For example, with a non-existent label:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: non-exist-node-label
            operator: In
            values:
            - non-exist-value
```

In this case, the Pod will still be launched on whichever Node is most available.

You can also combine conditions:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-east-1a
            - us-east-1b
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: non-exist-node-label
            operator: In
            values:
            - non-exist-value
```

When using several conditions in the `requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms`, the first one that coincided with the Node's label will be selected.

When using several conditions in the `matchExpressions` field they all must match.

In the [operator](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#operators) you can use operators In, NotIn, Exists, DoesNotExist, Gt (greater than) and Lt (less than).

#### soft-limit and the weight

In the `preferredDuringSchedulingIgnoredDuringExecution` you can set a weight of the condition setting a value from 1 to 100.

In this case, if all other conditions coincide, the scheduler will select a Node with the largest condition weight:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-east-1a
      - weight: 100
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-east-1b
```

This Pod will be launched on a Node in _the us-east-1b_ zone:

```
$ kubectl get pod my-pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-pod 1/1 Running 0 3s 10.0.3.245 ip-10–0–3–133.ec2.internal <none> <none>
```

And the zone of this Node:

```
$ kubectl get node ip-10–0–3–133.ec2.internal -o json | jq -r '.metadata.labels."topology.kubernetes.io/zone"'
```

us-east-1b

### `podAffinity` and `podAntiAffinity`

Similar to selecting a Node using hard and soft limits, you can adjust Pod Affinity depending on what labels Pods already running on the Node will have. See [Inter-pod affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity).

For example, Grafana Loki has three Pods — Read, Write, and Backend.

We want to run the Read and Backend in the same AvailabilityZone to avoid cross-AZ traffic, but at the same time, we want them not to run on those Nodes where there are Write Pods.

Loki Pods have labels corresponding to a component — `app.kubernetes.io/component=read`, `app.kubernetes.io/component=backend`, and `app.kubernetes.io/component=write`.

So, for the Read Pod, we can set a `podAffinity` to Pods with the label `app.kubernetes.io/component=backend`, and `podAntiAffinity` to Pods with a label `app.kubernetes.io/component=read`:

```
...
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                - backend
            topologyKey: "topology.kubernetes.io/zone"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                - write
            topologyKey: "kubernetes.io/hostname"
...
```

Here in the `podAffinity.topologyKey` we set that we want to place Pods using the `topology.kubernetes.io/zone` domain - that is, `topology.kubernetes.io/zone` for Read Pods must match the Backend Pods.

And in the `podAntiAffinity.topologyKey` we set the `kubernetes.io/hostname`, that is, do not place on WorkerNodes, where there are Pods with the label `app.kubernetes.io/component=write`.

Let’s deploy and check where there is a Write Pod:

```
$ kubectl -n dev-monitoring-ns get pod loki-write-0 -o json | jq '.spec.nodeName'
“ip-10–0–3–53.ec2.internal”
```

And AvailabilityZone of this Node:

```
$ kubectl -n dev-monitoring-ns get node ip-10–0–3–53.ec2.internal -o json | jq -r '.metadata.labels."topology.kubernetes.io/zone"'
us-east-1b
```

Check where the Backend Pod is placed:

```
$ kubectl -n dev-monitoring-ns get pod loki-backend-0 -o json | jq '.spec.nodeName'
“ip-10–0–2–220.ec2.internal”
```

And its zone:

```
$ kubectl -n dev-monitoring-ns get node ip-10–0–2–220.ec2.internal -o json | jq -r '.metadata.labels."topology.kubernetes.io/zone"'
us-east-1a
```

And now, a Read Pod:

```
$ kubectl -n dev-monitoring-ns get pod loki-read-698567cdb-wxgj5 -o json | jq '.spec.nodeName'
“ip-10–0–2–173.ec2.internal”
```

The Node is different from the Write or Backend Nodes, but:

```
$ kubectl -n dev-monitoring-ns get node ip-10–0–2–173.ec2.internal -o json | jq -r '.metadata.labels."topology.kubernetes.io/zone"'
us-east-1a
```

The same AvailabilityZone as in the Backend Pod.

### Pod Topology Spread Constraints

We can configure Kubernetes Scheduler in such a way that it distributes Pods by “domains”, that is, by nodes, regions, or Availability Zones. See [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/).

For this, we can set the necessary config in the field `spec.topologySpreadConstraints`, which describes exactly how pods will be created.

For example, we have 5 WorkerNodes in two AvailabilityZones.

We want to run 5 Pods and for fault tolerance we want each Pod to be on a separate Node.

Then our config for a Deployment can look like this:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: nginx:latest
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: my-app
```

Here:

- `maxSkew`: the maximum difference in the number of pods in one domain (`topologyKey`)
- plays a role only if `whenUnsatisfiable=DoNotSchedule`, when `whenUnsatisfiable=ScheduleAnyway` then a Pod will be created regardless of the conditions
- `whenUnsatisfiable`: can have value `DoNotSchedule` - do not allow Pods to be created, or `ScheduleAnyway`
- `topologyKey`: a WorkerNode label, by which the domain will be selected, that is, by which label we group the Nodes on which the placement of Pods is calculated
- `labelSelector`: what Pods to take into account when placing new Pods (for example, if Pods are from different Deployments, but should be placed in the same way - then in both Deployments we configure `topologySpreadConstraints` with mutual ones `labelSelecto`r)

In addition, you can set the `nodeAffinityPolicy` parameters and/or `nodeTaintsPolicy` with the `Honor` ​​or `Ignore` values to configure if `nodeAffinity` or `nodeTaints` of a Pod must be taken into account during calculating the placement of a Pod.

Let’s deploy and check the Nodes of these Pods:

```
$ kk get pod -o json | jq '.items[].spec.nodeName'
“ip-10–0–3–53.ec2.internal”
“ip-10–0–3–22.ec2.internal”
“ip-10–0–2–220.ec2.internal”
“ip-10–0–2–173.ec2.internal”
“ip-10–0–3–133.ec2.internal”
```

All are placed on separate Nodes.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/kubernetes-pods-and-workernodes-control-the-placement-of-the-pods-on-the-nodes/)_._

* * *