Enabling Memory Profiling on KEDA v2
Tags: keda
Published: 2020-10-20 18:57:43

I have a minor memory leak issue to solve. So that I'm wondering how can we enable profiler on KEDA. KEDA uses [Operator SDK](https://sdk.operatorframework.io/docs/). 

Go lang has a good profiler. 

* [Getting started with Go CPU and memory profiling](https://flaviocopes.com/golang-profiling/)

* [KEDA](https://github.com/kedacore/keda)

It is very simple to use it. Just add this line for memory profiling.

# Configuration and Issue

According to the documentation, the configuration will be like this.

```bash
go get github.com/pkg/profile
```

then add following line to the `main.go`

```go
import (
    //...
	"github.com/pkg/profile"
)
   :
defer profile.Start(profile.MemProfile).Stop()
```

Run the KEDA on my local machine. However, it doesn't work. The root cause is, we need to stop the operator sdk with ctr+c. In this case, `defer` method doesn't execute `Stop()` method. 

```
ushio@DESKTOP-KIUTRHV:~/Code/keda$ make run ARGS="--zap-log-level=debug"
/home/ushio/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go run \
-ldflags "-X=github.com/kedacore/keda/version.GitCommit=d0788b943ecf94f7f7c5b1a09f777cc3af6b746e -X=github.com/kedacore/keda/version.Version=v2" \
./main.go --zap-log-level=debug
2020/10/20 09:19:49 profile: memory profiling enabled (rate 4096), /tmp/profile174831365/mem.pprof
                   :
^C2020/10/20 09:20:02 profile: caught interrupt, stopping profiles
Exit 0 for profiling
make: *** [Makefile:96: run] Interrupt

$ ls -l /tmp/profile287657069/mem.pprof
-rw-r--r-- 1 ushio ushio 0 Oct 20 09:18 /tmp/profile287657069/mem.pprof
```

# Solution

If you want to handle the SIG interrupt, you can use `signal.Notify` method for handling the signal. 

```go
func SetupCloseHandler() {
	c := make(chan os.Signal)
	signal.Notify(c, os.Interrupt, syscall.SIGTERM)
	go func() {
		<-c
		fmt.Println("\r- Ctrl+C pressed in Terminal")
		DeleteFiles()
		os.Exit(0)
	}()
}
```

I read the [main.go](https://github.com/kedacore/keda/blob/v2/main.go#L118-L121) they are also having the signal handling part. Let's change it to call `profile.Stop()` method.

```go
func main() {
	// Memory profiling
	f := profile.Start(profile.MemProfile, profile.NoShutdownHook)
    :
	//	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
	//		setupLog.Error(err, "problem running manager")
	//		os.Exit(1) // Profiling test
	//	}
	fmt.Println("-----SetupCloseHandler")
	setupSignalHandler := func() (stopCh <-chan struct{}) {
		stop := make(chan struct{})
		c := make(chan os.Signal, 2)
		signal.Notify(c, os.Interrupt, syscall.SIGTERM)
		go func() {
			<-c
			fmt.Println("Exit 0 for profiling")
			f.Stop()
			os.Exit(1) // second signal. Exit directly.
		}()

		return stop
	}

	if err := mgr.Start(setupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}
}
```

Run it. Works!

```bash
$ ls -l /tmp/profile388737932/mem.pprof
-rw-r--r-- 1 ushio ushio 47888 Oct 20 09:31 /tmp/profile388737932/mem.pprof
```

# Viewer

The next step will be the view the analysis report. 
It looks requires [graphviz](https://www.graphviz.org/download/). Install it.

```bash
$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf
failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH
$ sudo apt install graphviz
```

Works! 

```bash
$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/b8vwd9biqalyjpghud88.png)

I don't know if there is better way to enable profiling using Operator SDK. Once I know it, I'll update this blog.

# Reference
* [Handle Ctrl+C (Signal Interrupt) Close in the Terminal](https://golangcode.com/handle-ctrl-c-exit-in-terminal/)