Enabling KEDA debugging
Tags: keda, go, kubernetes
Published: 2021-02-03 04:57:54

If you are contributing the [KEDA](https://github.com/kedacore/keda), you might be wondering how we can debug it. I'd like to share the tips for debugging KEDA.

We have several ways to do it. 

# Remote Debugging
Remote debugging is not very recommended since it is relatively slow, however, sometimes, it requires. KEDA has two servers. one is `keda-operator` that has reconcile loop. It ask the infromation to the `keda-metrics-apiserver` that is only reside in the k8s cluster. In case you want to debug `keda-metrics-apiserver` we need the remote debugging.  

## Create a Dockerfile
Create a Dockerfile that enalbes us to debug. It is just adding delv and  build with `gcflags "-N -l"`.  then start the execution file with wrapping with `dlv` command with exposing port. 

```dockerfile
ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]
```

The parameters means following. `--` means passing arguments to the executable (in this case, keda-adapter)

```bash
$ dlv --help
Delve is a source level debugger for Go programs.

Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.

The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.

Pass flags to the program you are debugging using `--`, for example:

`dlv exec ./hello -- server --config conf/config.toml`

Usage:
  dlv [command]

Available Commands:
  attach      Attach to running process and begin debugging.
  connect     Connect to a headless debug server.
  core        Examine a core dump.
  dap         [EXPERIMENTAL] Starts a TCP server communicating via Debug Adaptor Protocol (DAP).
  debug       Compile and begin debugging main package in current directory, or the package specified.
  exec        Execute a precompiled binary, and begin a debug session.
  help        Help about any command
  run         Deprecated command. Use 'debug' instead.
  test        Compile test binary and begin debugging program.
  trace       Compile and begin tracing program.
  version     Prints version.

Flags:
      --accept-multiclient   Allows a headless server to accept multiple client connections.
      --api-version int      Selects API version when headless. New clients should use v2. Can be reset via RPCServer.SetApiVersion. See Documentation/api/json-rpc/README.md. (default 1)
      --backend string       Backend selection (see 'dlv help backend'). (default "default")
      --build-flags string   Build flags, to be passed to the compiler.
      --check-go-version     Checks that the version of Go in use is compatible with Delve. (default true)
      --headless             Run debug server only, in headless mode.
      --init string          Init file, executed by the terminal client.
  -l, --listen string        Debugging server listen address. (default "127.0.0.1:0")
      --log                  Enable debugging server logging.
      --log-dest string      Writes logs to the specified file or file descriptor (see 'dlv help log').
      --log-output string    Comma separated list of components that should produce debug output (see 'dlv help log')
      --only-same-user       Only connections from the same user that started this instance of Delve are allowed to connect. (default true)
      --wd string            Working directory for running the program. (default ".")

Additional help topics:
  dlv backend Help about the --backend flag.
  dlv log     Help about logging flags.

Use "dlv [command] --help" for more information about a command.
```



_Docker.remote_

```dockerfile
FROM golang:1.15 as builder

WORKDIR /workspace

RUN go get -u github.com/go-delve/delve/cmd/dlv

# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

COPY Makefile Makefile

# Copy the go source
COPY hack/ hack/
COPY version/ version/
COPY main.go main.go
COPY adapter/ adapter/
COPY api/ api/
COPY controllers/ controllers/
COPY pkg/ pkg/

RUN mkdir -p /apiserver.local.config/certificates && chmod -R 777 /apiserver.local.config

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -gcflags "-N -l" -o keda-adapter adapter/main.go

FROM ubuntu:18.04

WORKDIR /
COPY --from=builder /apiserver.local.config  /apiserver.local.config
COPY --from=builder /workspace /workspace
COPY --from=builder /go/bin/dlv /dlv

ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]
```
Then build and publish to your dockerhub account. In my case, 

```bash
$ docker login
$ docker build -f Docker.remote -t tsuyoshiushio/adapter:1
$ docker push tsuyoshiushio/adapter:1
```

## Update the deployment
We can see the two deployment is available on the k8s cluster.

```bash
ushio@DESKTOP-KIUTRHV:~/Code/keda$ kubectl get deployment -n keda
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
keda-metrics-apiserver   1/1     1            1           6h2m
keda-operator            1/1     1            1           6h2m
```

let's edit the keda-metrics-apiserver. 

```bash
$ kubectl edit deployment keda-metrics-apiserver -n keda
```

let's change the image part. 

```bash
    spec:
      containers:
      - args:
        - /usr/local/bin/keda-adapter
        - --secure-port=6443
        - --logtostderr=true
        - --v=0
        env:
        - name: WATCH_NAMESPACE
        image: docker.io/tsuyoshiushio/adapter:3    // <- Change
        imagePullPolicy: Always
```

then port forward it. 

```bash
$ kubectl get pods -n keda
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-6d5689599f-884x9   1/1     Running   0          68s
keda-operator-6b546dc696-cxdbq            1/1     Running   0          5h49m
$ kubectl port-forward keda-metrics-apiserver-6d5689599f-884x9  -n keda 4040:4040
Forwarding from 127.0.0.1:4040 -> 4040
Forwarding from [::1]:4040 -> 4040
```

For VS Code, you can configure `launch.json` settings to attach remote server. 

```json
        {
            "name": "Remote debug",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/workspace",
            "port": 4040,
            "host": "127.0.0.1"
        }
```

However, it is sometimes, hard to wait very slow response. I recommend to go with Local debugging and adding log to see the remote container. 

# Local Debugging

Set the `launch.json` with VSCode. Just start the `main.go` on the root of the KEDA repo. We can not use keda-metrics-apiserver since it reside in the cluster. However, we can do the local debugging for the keda-operator. 

```json
        {
            "name": "Connect to server",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/main.go",
            "env": {
                "WATCH_NAMESPACE":""
            }
        },
```

# Build and publish KEDA to your kubernets cluster

`make undeploy` will remove the keda resources. `make deploy` deploy the keda resources to the cluster when you in development. 

```bash
$ kubectl get all -n keda
NAME                                          READY   STATUS    RESTARTS   AGE
pod/keda-metrics-apiserver-6dff8c4f7f-8qc9v   1/1     Running   0          30s
pod/keda-operator-6b546dc696-pf5ck            1/1     Running   0          29s

NAME                             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/keda-metrics-apiserver   ClusterIP   10.0.16.80   <none>        443/TCP,80/TCP   31s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/keda-metrics-apiserver   1/1     1            1           30s
deployment.apps/keda-operator            1/1     1            1           29s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/keda-metrics-apiserver-6dff8c4f7f   1         1         1       31s
replicaset.apps/keda-operator-6b546dc696            1         1         1       30s
```