Isolating Kubernetes pods for debugging
Tags: kubernetes
Published: 2023-01-09 12:19:07

This article introduces a technique that helps you with debugging running Pods in production.

By changing labels, you can detach Pods from the Service (no traffic), and you troubleshoot them live.

![Isolating pods for troubleshooting](https://res.cloudinary.com/uasabi/image/upload/v1673242188/threads/quarantine.gif)

_Here's how it works._

Imagine you have a Deployment with three replicas.

Each Pod has an `app=hello` label.

A Service routes the traffic to your Pods using the selector `app=hello`

![A deployment with three replicas and a service](https://res.cloudinary.com/uasabi/image/upload/v1673242148/threads/troubleshooting-2.png)

If you want to isolate a Pod you can overwrite the existing label with: `kubectl label pod <pod-name> app=debug --overwrite`

**Two things happen next.**

![Changing the label to a Pod in a deployment](https://res.cloudinary.com/uasabi/image/upload/v1673242148/threads/troubleshooting-3.png)

First, the Service stops routing traffic to the Pod because the Service's selector doesn't match the label.

![The service stops routing traffic to the pod](https://res.cloudinary.com/uasabi/image/upload/v1673242148/threads/troubleshooting-4.png)

Then, the ReplicaSet notices that there are only two replicas, but you asked for 3.

The ReplicaSet creates a new Pod.

![The ReplicaSet creates a new Pod](https://res.cloudinary.com/uasabi/image/upload/v1673242148/threads/troubleshooting-5.png)

You end up with 4 pods:

- 3 Pods running production traffic.
- A single isolated pod that used to receive traffic, but doesn't anymore.

**The pod is still running and you can inspect its state.**

_What tool should you use?_

![](https://res.cloudinary.com/uasabi/image/upload/v1673242148/threads/troubleshooting-6.png)

For simple tasks, you could use:

- `kubectl exec`, to attach to the container.
- `kubectl port-forward`, to tunnel the traffic.
- `kubectl debug` to run a sidecar container alongside the existing one.

_But there are more options._

[Inspector gadget](https://github.com/inspektor-gadget/inspektor-gadget) is a tool designed to introspect and debug Kubernetes applications using eBPF.

If you use VS Code, you might like the [official Kubernetes extension.](https://code.visualstudio.com/docs/azure/kubernetes)

[In this tutorial,](https://heka-ai.medium.com/introduction-to-debugging-locally-and-live-on-kubernetes-8c8ecd3acbaa) you'll learn how to use it to attach the debugger to the application in the Pod.

If you feel at home in the command line, you should check out [netshoot — a Docker + Kubernetes network trouble-shooting swiss-army container.](https://github.com/nicolaka/netshoot)

Which, of course, you can combine it with `kubectl debug`.

If you need more help to troubleshoot your deployments in Kubernetes, you should [check out my flowchart:](https://learnk8s.io/troubleshooting-deployments)

[![Troubleshooting Kubernetes deployments flowchart](https://res.cloudinary.com/uasabi/image/upload/v1673242790/threads/troubleshooting-8.png)](https://learnk8s.io/troubleshooting-deployments)

And finally, if you've enjoyed this thread, you might also like the Kubernetes workshops that we run at Learnk8s https://learnk8s.io/training or this collection of past Twitter threads https://twitter.com/danielepolencic/status/1298543151901155330