kubectl: show me the logs
Tags: kubernetes, devops, docker
Published: 2020-10-08 04:43:32

When I started using Kubernetes, and consequently `kubectl`, figuring out how to quickly view my pod's logs was always a pain point. Why was the verb/noun usage not the same? How do I know if there's more than one container running in the pod? How do I filter any of this except with CLI magic like `grep` or `awk`? In this post, I'm hoping to shed some light on the multitude of ways you can grab logs locally from your Kubernetes cluster.

If you don't have a Kubernetes cluster accessible, feel free to use any of these:

- https://kind.sigs.k8s.io/
- https://kubernetes.io/docs/tasks/tools/install-minikube/
- https://github.com/rancher/k3d

---

### kubectl logs

First up is the simplest of all the logging options: `kubectl logs`

Here is some yaml below that will generate the date and time every second. If you have only a single container in the pod, you can simply run `kubectl logs echo-date` to see all of the output.

```
apiVersion: v1
kind: Pod
metadata:
  name: print-date
  namespace: default
spec:
  containers:
    - name: main
      image: alpine
      args:
        - /bin/sh
        - -c
        - >
          while true;
          do
            date
            sleep 1
          done
```

Now, what happens when you add a second image into the pod? Let's apply the same yaml with a second image added into the pod.

```
apiVersion: v1
kind: Pod
metadata:
  name: print-dates
  namespace: default
spec:
  containers:
    - name: main
      image: alpine
      args:
        - /bin/sh
        - -c
        - >
          while true;
          do
            date
            sleep 1
          done
    - name: sidecar
      image: alpine
      args:
        - /bin/sh
        - -c
        - >
          while true;
          do
            date
            sleep 1
          done
```

When there are multiple containers running inside a single pod, you must pick the container. In order to see how many containers are in the pod, you can run `kubectl get pods`. Below, the `READY` field indicates how many containers are up and running in the pod.

```
NAME          READY   STATUS    RESTARTS   AGE
print-date    1/1     Running   0          5m25s
print-dates   2/2     Running   0          5m56s
```

Now, you can get the logs for either container: `kubectl logs print-dates sidecar`. And in case you're working with a pod whose containers' names you don't know, here is a fancy command utilizing json parsing: `kubectl get pod print-dates -o json | jq .spec.containers[].name`

---

Alright, so we've got some basic knowledge on how to see logs, 
but I'd really like to filter these logs down some and see them streaming live to the terminal. For now, let's switch back to the single container pod in order to make the commands a bit easier. 

First up, let's follow the logs live: `kubectl logs print-date --follow` 

Next, let's filter based on the timestamp: `kubectl logs print-date --since=10s`

I want to follow this info without all the previous logs drowning my terminal in rows of data though: `kubectl logs print-date --tail=1 --follow`. 

So, it's pretty evident `kubectl logs` provides a lot of extensibility on how to view container logs, but let's take this further and look at some community built options.

---

### stern

[stern](https://github.com/wercker/stern) is a popular open source option for viewing pod logs by name, namespace, time, and many other options. You can think of it as a `kubectl logs` aggregator. It's a great solution when trying to debug a solution that may span multiple pods across a namespace. 

To install on Mac: `brew install stern`

Because we have two pods populating logs in the default namespace, let's start with viewing both of these at the same time: `stern -n default print-`. By adding `print-`, we are limiting the logs to only pods that are prepended with that pod name. One of the great things about `stern` is that is classifies each pod's logs as a different color allowing easy distinction between the two. This is, perhaps, the best feature I've come across when debugging.

While this output is not color coded in the markdown below, it is worth noting that the pod and container are added into the logs:

```
print-date main Thu Oct  8 04:06:52 UTC 2020
print-dates main Thu Oct  8 04:06:52 UTC 2020
print-dates sidecar Thu Oct  8 04:06:53 UTC 2020
print-date main Thu Oct  8 04:06:53 UTC 2020
```

`stern --help` provides tons of really good options for filtering pods, so I won't go too in depth with examples. However, I'd like to show an example above for a similar solution of printing since a certain time. Except we'll be printing output for multiple containers now. 

- Print by namespace: `stern -n default .`
- Print all pods: `stern .`
- Print the last 15 seconds for both pods' logs: `stern -t --since 15s print-date`


Definitely checked out the [README](https://github.com/wercker/stern/blob/master/README.md) for additional examples as well. 

--- 

### kubectl plugins

For the final iteration, let's take a look at [kubectl plugins](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/), which are basically just binaries prepended with `kubectl-`. [Krew](https://github.com/kubernetes-sigs/krew/) is a package manager that allows easy installation of these. We can use `kubectl tail` to accomplish much of the same logging capabilities as `stern`.

While this can be installed by `kubectl krew install tail`, the GitHub repo resides [here](https://github.com/boz/kail).

In order to compare to `stern`, here are basically the same examples:

- Print by namespace: `kubectl tail --ns default`
- Print all pods: `kubectl tail`
- Print the last 15 seconds for both pods' logs: `kubectl tail --pod print-date --pod print-dates --since 15s
`

Make sure to check out the flags with `kubectl tail --help` to see all potential options!

---

### wrap up

While none of this was exactly groundbreaking information, I hope it created some muscle memory of using `kubectl logs` and showed open source solutions that are a great alternative as you become a more advanced k8s user. Happy programming!

p.s. -- it's pronounced `kube`-`see`-`tee`-`el` :p