How to debug an Argo Workflow
Tags: kubernetes
Published: 2021-03-01 14:38:31

Argo Workflows is a tool for running a series (or a graph) of containers on Kubernetes, tying them together into a workflow.

It's a relatively young project, so things like error messages and documentation still need some work.

In the meantime, here are some steps you can take when your Workflow doesn't behave as expected.

# Inspect the Workflow with Argo CLI

For simple issues, the Argo CLI will include a short description in the `MESSAGE` column of its `argo get` output.

```
$ argo get workflow-template-dag-diamond 
Name:                workflow-template-dag-diamond
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Conditions:          
 Completed           True
Created:             Mon Mar 01 08:51:26 -0500 (7 minutes ago)
Started:             Mon Mar 01 08:51:26 -0500 (7 minutes ago)
Finished:            Mon Mar 01 08:51:36 -0500 (7 minutes ago)
Duration:            10 seconds
Progress:            1/1
ResourcesDuration:   5s*(1 cpu),5s*(100Mi memory)

STEP                              TEMPLATE                                               PODNAME                                   DURATION  MESSAGE
 ✔ workflow-template-dag-diamond  diamond                                                                                                      
 └─✔ A                            workflow-template-whalesay-template/whalesay-template  workflow-template-dag-diamond-2997968480  6s
```

If there's a message that's just not _quite_ detailed enough to figure out the problem, copy the `PODNAME` of the failed step, and skip to the section about [using kubectl to describe the Pod](#use-kubectl-to-describe-the-pod).

If there's no useful message, try describing the Workflow.

# Use kubectl to describe the Workflow

Sometimes there's a problem with the _whole Workflow_ that doesn't fit nicely in `argo get`'s `MESSAGE` column.

`kubectl describe workflow` will print a lot more details about the Workflow, including a list of Events. The Events often contain details about what went wrong.

```
$ kubectl describe workflow workflow-template-dag-diamond
Name:         workflow-template-dag-diamond

...

Events:
  Type    Reason                 Age    From                 Message
  ----    ------                 ----   ----                 -------
  Normal  WorkflowRunning        8m42s  workflow-controller  Workflow Running
  Normal  WorkflowNodeSucceeded  8m32s  workflow-controller  Succeeded node workflow-template-dag-diamond.A
  Normal  WorkflowNodeSucceeded  8m32s  workflow-controller  Succeeded node workflow-template-dag-diamond
  Normal  WorkflowSucceeded      8m32s  workflow-controller  Workflow completed

```

# Use kubectl to describe the Pod

If a Pod fails, there are a number of places which may hold clues. One is the Events associated with the Pods.

Pod names may be unpredictable (they often have random suffixes like `-93750129`), so use `argo get` to get the name of the suspect Pod.

Then use `kubectl describe po` to see the Pod details, including the Events.

```
$ kubectl describe po workflow-template-dag-diamond-2997968480
Name:         workflow-template-dag-diamond-2997968480

...

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  36s                default-scheduler  Successfully assigned default/workflow-template-dag-diamond-2997968480 to docker-desktop
  Normal   Pulled     35s                kubelet            Container image "argoproj/argoexec:v2.12.9" already present on machine
  Normal   Created    35s                kubelet            Created container wait
  Normal   Started    35s                kubelet            Started container wait
  Warning  Failed     31s                kubelet            Failed to pull image "docker/whalesay": rpc error: code = Unknown desc = Error response from daemon: Head https://registry-1.docker.io/v2/docker/whalesay/manifests/latest: x509: certificate is valid for auth.docker.io, not registry-1.docker.io
  Normal   Pulling    17s (x2 over 35s)  kubelet            Pulling image "docker/whalesay"
  Warning  Failed     16s (x2 over 31s)  kubelet            Error: ErrImagePull
  Warning  Failed     16s                kubelet            Failed to pull image "docker/whalesay": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: x509: certificate is valid for auth.docker.io, not registry-1.docker.io
  Normal   BackOff    5s (x2 over 30s)   kubelet            Back-off pulling image "docker/whalesay"
  Warning  Failed     5s (x2 over 30s)   kubelet            Error: ImagePullBackOff
```

This Workflow failed because a proxy issue is preventing pulls from Docker Hub.

Sometimes the problem doesn't show up in the Events, because the failure is _inside_ one of the step's containers.

# Use kubectl to read the Pod logs

Pods run as part of Argo Workflows have two or three containers: `wait`, `main`, and sometimes `init`.

The `wait` sidecar is injected by Argo to keep an eye on the `main` container (your code) and communicate with the Argo Workflow controller (another Pod) about the step's progress.

The `main` container is the one you set up when you defined the Workflow in yaml. (Look for the `image`, `command`, `args`, and `source` items to see part of this Pod's configuration.)

The `init` container, if present, is also injected by Argo. It does things like pulling artifacts into the Pod.

To read the logs, use `kubectl logs`. For example:

```
$ kubectl logs workflow-template-dag-diamond-2997968480 init
error: container init is not valid for pod workflow-template-dag-diamond-2997968480
C02D507EMD6P:test_workflows ekmed$ kubectl logs workflow-template-dag-diamond-2997968480 wait
time="2021-03-01T14:09:18.339Z" level=info msg="Starting Workflow Executor" version=v2.12.9
time="2021-03-01T14:09:18.346Z" level=info msg="Creating a docker executor"
time="2021-03-01T14:09:18.346Z" level=info msg="Executor (version: v2.12.9, build_date: 2021-02-16T22:51:48Z) initialized (pod: default/workflow-template-dag-diamond-2997968480) with template:\n{\"name\":\"whalesay-template\",\"arguments\":{},\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"A\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"resources\":{}}}"
time="2021-03-01T14:09:18.346Z" level=info msg="Waiting on main container"
time="2021-03-01T14:14:17.998Z" level=info msg="Alloc=4699 TotalAlloc=14633 Sys=70080 NumGC=6 Goroutines=7"
```

The logs from `init` and `wait` may be a bit difficult to read, because they come from Argo. The logs for `main` will be from your configured `image`, so they'll probably be more familiar.

# Use kubectl to read the Workflow controller logs

Argo comes with a Pod called the "Workflow controller" to sort of usher a Workflow through the process of running all its steps.

If all the other debugging techniques fail, the Workflow controller logs may hold helpful information.

First, find the Pod name. If you used the default Argo installation command, the Pod will be in the `argo` namespace.

```
$ kubectl get po -n argo
NAME                                  READY   STATUS    RESTARTS   AGE
argo-server-6bb488c6c8-ff88g          1/1     Running   0          40m
workflow-controller-57db6b46f-7qfr9   1/1     Running   0          40m
$ kubectl logs workflow-controller-57db6b46f-7qfr9 -n argo

... lots of stuff here ...
```

# Ask for help

If none of these solves your problem, [ask a question on StackOverflow](https://stackoverflow.com/questions/tagged/argo-workflows), [start a discussion on GitHub](https://github.com/argoproj/argo-workflows/discussions/new), or [ask in the Argo Slack](https://argoproj.github.io/community/join-slack).

These are just a few of my go-to tools. If I'm missing anything, please comment!