Kubernetes Labels: Expert Guide with 10 Best Practices
Tags: kubernetes, cloud, devops, guide
Published: 2022-11-29 17:21:14

<!-- wp:paragraph -->
<p>With Kubernetes labels, DevOps teams can troubleshoot issues faster, apply configuration changes en masse, and respond quickly to issues. Labels also give crucial insights into your costs, boosting your monitoring, allocation, and management capabilities. Following best practices when using labels helps you realize <strong>tremendous benefits from infrastructure visibility and efficient operations</strong>.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here's everything you need to know about Kubernetes labels - what they are, how they work, when to use them, and the <strong>10 best practices to follow to build a solid labeling strategy</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-what-are-kubernetes-labels">What are Kubernetes labels?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kubernetes labels are key-value string pairs that link identifying metadata to Kubernetes objects. Kubernetes provides teams with integrated support for using labels to retrieve and filter the data from the Kubernetes API and carry out bulk operations on the selected objects. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Many teams use Kubernetes labels to provide DevOps with information about the ownership of a node, a pod, or other Kubernetes objects for easier tracking and operational decision-making.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>When creating a new label, you must comply with the restrictions Kubernetes places on the length and allowed values. A label value must:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>contain 63 characters or less (a label's value can also be empty),</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>start and end with an alphanumeric character (unless it’s empty),</li>
<!-- /wp:list-item -->

<!-- wp:list-item -->
<li>only include dashes (-), underscores (_), dots (.), and alphanumerics.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>You can find the labels a Kubernetes object has by using <code>kubectl</code>. For example, to get all labels for a pod named <code>pod1</code>, you can run:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pod1 -o json | jq .metadata.labels</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To create a label, you can specify them in your <a href="https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/" target="_blank" rel="noreferrer noopener">configuration file</a> spec’s <code>metadata.labels</code> object. Let's consider the <code>pod.yaml</code> file that describes a single pod:</p>
<!-- /wp:paragraph -->

<!-- wp:code {"textColor":"cast-gray-050"} -->
<pre class="wp-block-code has-cast-gray-050-color has-text-color"><code>apiVersion: v1
kind: Pod
metadata:
&nbsp;name: nginx
&nbsp;labels:
&nbsp;&nbsp;&nbsp;environment: dev
&nbsp;&nbsp;&nbsp;critical: "true"
spec:
&nbsp;containers:
&nbsp;&nbsp;&nbsp;- image: nginx
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;name: nginx
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;resources:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;requests:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cpu: 500m</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Note that the value of the <code>critical</code> label is <code>“true”</code> and not <code>true</code>. That is because labels, as well as their values, must be strings.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s apply the configuration file:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl apply -f pod.yaml
pod/nginx created</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>You can now apply or overwrite a label directly on an already existing Kubernetes object using <code>kubectl</code>. First, get all the labels that the pod has:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pod nginx -o json | jq .metadata.labels

{
&nbsp;&nbsp;"critical": "true",
&nbsp;&nbsp;"environment": "dev"
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now, to change the <code>environment</code> label’s value and add a new key-value label pair <code>deprecated=true</code>, we execute the following command:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl label pod nginx environment=prod --overwrite
pod/nginx labeled

&gt; kubectl label pod nginx deprecated=true
pod/nginx labeled</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Keep in mind that updating a label’s value is not allowed unless you explicitly overwrite it with the <code>–overwrite</code> flag. The resulting labels are as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pod nginx -o json | jq .metadata.labels

{
&nbsp;&nbsp;"deprecated": "true",
&nbsp;&nbsp;"critical": "true",
&nbsp;&nbsp;"environment": "prod"
}</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2 id="h-kubernetes-labels-vs-annotations">Kubernetes labels vs. annotations</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kubernetes offers two tactics for connecting metadata with objects: labels and annotations.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Annotations are key-value pairs that connect non-identifying metadata with objects. For instance, an annotation could contain logging or monitoring information for a given resource.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The main difference between labels and annotations is that annotations are not used to filter, group, or operate over the Kubernetes resource. Instead, you can use them to access additional information about it.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For example, the annotations for the node where the previously deployed pod has been scheduled are as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get node demo-node -o json | jq .metadata.annotations

{
&nbsp;&nbsp;"kubeadm.alpha.kubernetes.io/cri-socket": "unix:///var/run/cri-dockerd.sock",
&nbsp;&nbsp;"node.alpha.kubernetes.io/ttl": "0",
&nbsp;&nbsp;"volumes.kubernetes.io/controller-managed-attach-detach": "true"
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Those annotations do not provide any information about the node’s characteristics. Instead, they offer some data on how the node works.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-when-to-use-kubernetes-labels">When to use Kubernetes labels?</h2>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3 id="h-group-resources-for-object-queries">Group resources for object queries</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you add the same label key-value pair to multiple resources, other people can easily query for all of them. For example, a DevOps engineer discovers that a development environment is unavailable. At this point, they can quickly check the status of all pods including the label <code>environment:dev</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here’s an example command:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pods -l 'environment=dev'

NAME    READY   STATUS              RESTARTS    AGE
nginx   0/1     CrashLoopBackOff    1           5m</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This lets the team instantly see the affected pods and resolve the issue much faster than going through all the resources and picking just the ones in the <code>dev</code> environment.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In a complex case with many different deployments, finding the right <code>dev</code> pods would take the DevOps engineer ages if the engineering team didn’t add the <code>environment:dev</code> label to the resources. The DevOps engineer would have to use a generic <code>kubectl get pods</code> command and then comb through the output using a tool like <code>grep</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-perform-bulk-operations">Perform bulk operations</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Another use case of Kubernetes labeling is to carry out bulk operations based on the resource labels.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Suppose that an engineer removes all staging environments every night to <a href="https://cast.ai/blog/kubernetes-cost-monitoring-the-best-way-to-cut-your-cloud-bill-in-half/" target="_blank" rel="noreferrer noopener">reduce cloud costs</a>. By using Kubernetes labels, they can easily automate this task.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For instance, here’s a command that deletes all objects labeled <code>environment:local</code>, <code>environment:dev</code> or <code>environment:staging</code>:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl delete deployment,services,statefulsets -l 'environment in (local,dev,staging)'</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3 id="h-schedule-pods-based-on-node-labels">Schedule pods based on node labels</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The hidden gem of Kubernetes labels is that they are heavily used in Kubernetes itself for scheduling pods to appropriate nodes. By using labels, you can have more control over the resources you create by making Kubernetes schedule specific deployments onto specific nodes.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s see how this works in practice:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get nodes

NAME				STATUS	ROLES	AGE	VERSION
gke-node-1fe68171	Ready	&lt;none&gt;	1d	v1.22.12-gke.2300
gke-node-3cdf3d2b	Ready	&lt;none&gt;	3d	v1.22.12-gke.2300
gke-node-5f7b4cf1	Ready	&lt;none&gt;	5d	v1.22.12-gke.500

&gt; kubectl get nodes -l ‘critical=true’
No resources found</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Currently, no nodes with the label <code>critical:true</code> exist.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s try to create a pod that has to be scheduled on a node with the label <code>critical:true</code> using a node selector. Here is a <code>pod.yaml</code> configuration file for that:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
   environment: prod
spec:
 nodeSelector:
   critical: "true"
 containers:
   - image: nginx
     name: nginx
     resources:
       requests:
         cpu: 500m</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now let's apply it and check what happens:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl apply -f pod.yaml
pod/nginx created

&gt; kubectl get pod nginx

NAME	READY		STATUS	RESTARTS	AGE
nginx	0/1			Pending	0			1m

&gt; kubectl get events --field-selector involvedObject.name=nginx

LAST SEEN	TYPE	REASON				OBJECT		MESSAGE
46s			Warning	FailedScheduling	pod/nginx	0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Note that the pod cannot get scheduled on any of the nodes since none of them has the required label.&nbsp; Now, let’s label one of the nodes with the required label:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl label node gke-node-5f7b4cf1 critical=true
node/gke-node-5f7b4cf1 labeled

&gt; kubectl get nodes -l 'critical=true'

NAME				STATUS	ROLES	AGE	VERSION
gke-node-5f7b4cf1	Ready	&lt;none&gt;	5h	v1.22.12-gke.500</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><br>And now, let’s check the pod:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pod nginx

NAME	READY	STATUS	RESTARTS	AGE
nginx	1/1		Running	0			3m31s</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The pod has been successfully scheduled to the node. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Keep in mind that if multiple labels are specified in the node selector, they all must be satisfied by a node in order for the pod to get scheduled on it.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-10-best-practices-for-kubernetes-labels">10 best practices for Kubernetes labels</h2>
<!-- /wp:heading -->

<!-- wp:heading {"level":3} -->
<h3 id="h-1-make-use-of-the-labels-recommended-by-kubernetes">1. Make use of the labels recommended by Kubernetes</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kubernetes provides a <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels" target="_blank" rel="noreferrer noopener">list of recommended labels</a> for grouping objects. For example, Kubernetes recommends using <code>app.kubernetes.io/name</code> and <code>app.kubernetes.io/instance</code> to represent the application’s name and instance, respectively. Just drop the prefix “app.kubernetes.io” and add your company’s subdomain to customize the labels.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-2-pay-attention-to-correct-syntax">2. Pay attention to correct syntax</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To create a Kubernetes label key-value pair, you need to use the following syntax: <code>&lt;prefix&gt;/&lt;name&gt;</code>. Let’s dive into the details:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>&lt;prefix&gt;</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The prefix is optional; if you choose to use it, it needs to be a valid DNS subdomain (such as "cast.ai") and have no more than 253 characters in total. Prefixes come in handy for tools and commands that aren't private to users. They are also helpful because they let teams use multiple labels that would otherwise conflict (think of the ones in third-party packages).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note that the <code>kubernetes.io/</code> and <code>k8s.io</code> prefixes are reserved for Kubernetes core components.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>&lt;name&gt;</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This part refers to the arbitrary property name of the label. Teams can use the name “environment” with label values such as “production” or “testing” for clarity.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>A name must meet the same requirements as the label value but it can’t be empty. Hence, the name needs to have 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics in between.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-3-standartize-label-naming-conventions">3. Standartize label naming conventions</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Multiple teams using Kubernetes need to follow the same labeling conventions. Otherwise, all the labeling effort will bring you no value.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It’s a good practice to have your development pipeline carry out static code analysis on resource configuration files to ensure that all the required labels are there. If you fail to apply labels properly, automated processes may get broken - and any monitoring solutions you use may send you false-positive alerts.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-4-avoid-unnecessary-changes-to-labels">4. Avoid unnecessary changes to labels</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Labels in Kubernetes are used to identify and select resources for scheduling, deployment, and administration purposes. As a result, modifying a resource's label can have far-reaching and unforeseen implications.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For instance, if you switch a group of pods' "app" label from "frontend" to "backend," Kubernetes can reschedule those pods onto nodes that aren't set up to run the "backend" app. The pods can crash; as a result, making them unavailable.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It's crucial only to modify labels when it is absolutely essential and carefully evaluate the ramifications of any changes before making them to avoid these kinds of issues.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-5-use-label-selection-options">5. Use label-selection options</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Teams can select labeled objects based on equality and sets.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Selections based on equality allow you to retrieve objects with labels equal or not equal to the specified value (or values). Diving down into syntax, = and == both represent equality, while != represents inequality. It’s possible to add multiple labels separated by commas (all conditions need to match here). For example, if you execute the following command:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pods -l ‘environment=dev,release=daily’</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>it will return all the pods that have labels <code>environment:dev</code> AND <code>release:daily</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>On the other hand, selections based on sets allow finding resources with multiple values at once. Sets are similar to the <code>IN</code> keyword in SQL. For example, the following command:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>&gt; kubectl get pods -l ‘environment in (prod,dev)’</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>will find all the pods that contain the label <code>environment=prod</code> OR <code>environment=dev</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-6-don-t-store-application-level-semantics-in-labels">6. Don’t store application-level semantics in labels</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Kubernetes labels may come together with an object’s metadata, but they’re not supposed to serve as a data store for applications. Given that Kubernetes resources are often used for a short period of time and aren’t tightly associated with applications, labels quickly become unsynchronized and, therefore, useless.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-7-don-t-store-sensitive-information-in-labels">7. Don’t store sensitive information in labels</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If someone gains access to your Kubernetes cluster while you store passwords or API credentials, or other sensitive data in labels, they will be able to see it in plain text. This is a significant security risk and may have negative effects like identity theft or data breaches.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It is advisable to preserve sensitive information in secrets rather than labels. Secrets are encrypted, and only the pods that require them may decrypt them. By doing this, even if someone manages to access your Kubernetes cluster, they won't be able to view the private data kept in secrets.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-8-add-labels-to-pod-templates">8. Add labels to pod templates</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Add essential labels to pod templates that are part of workload resources. That way, Kubernetes controllers can consistently create pods with the states you’ve specified.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The goal should not be to create as many labels as possible but to create labels that bring value to your team. Start small and create a list of labels to be part of the template. For example, you can start by identifying the resource owners, the environment the resource is running in, and the release.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-9-automate-your-labeling-practice">9. Automate your labeling practice</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Automation can save you plenty of time, and labeling is no exception to that. If you have a continuous integration/continuous delivery (CI/CD) pipeline set up, you can easily automate some labels for cross-cutting concerns.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>It’s smart to automatically attach labels with CD tooling since it guarantees consistency and makes engineers more productive. It’s also a good practice to have CI jobs enforce proper labeling by making a build fail and sending a notification to the responsible team if a label is missing.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3 id="h-10-use-labels-for-cost-monitoring">10. Use labels for cost monitoring</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Labels are very helpful for gaining a better understanding of your Kubernetes cloud costs. Cost monitoring, allocation, and management all rely on a proper labeling strategy.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If multiple tenants share resources in a single cluster, you need to use relevant labels to create a cost allocation report. This is how you can determine which team, service, or application generated specific costs, which helps greatly when investigating an unexpected cost spike.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2 id="h-use-this-free-monitoring-tool-to-track-your-costs-by-labels">Use this free monitoring tool to track your costs by labels</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>CAST AI provides a cost monitoring tool that allows you to stay updated on the costs of any of your workloads. The costs can be filtered by any label that exists on any of your workloads, making it easy to track cloud costs per team, service, or any other label that you use. The option to group workloads by label is coming soon.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>See the difference good labeling and cost monitoring can make by connecting your cluster to CAST AI’s free cost monitoring solution.</p>
<!-- /wp:paragraph -->