Diagnostics on Kubernetes: Obtaining a Memory Dump
Tags: kubernetes, linux, dotnet
Published: 2022-01-10 19:14:38

If like me you found that there is a slight memory leak in your application running on Kubernetes and you are searching non-stop on how to collect a memory dump, it can be hard to find exactly how to do this on a .NET application running on Linux-based containers.

There are a handful of articles with useful instructions and tips, however none take into account whether your Docker images are [hardened](https://techcommunity.microsoft.com/t5/azure-developer-community-blog/hardening-an-asp-net-container-running-on-kubernetes/ba-p/2542224). So below, I've broken down the process into four steps which explain just how exactly to do this:

1. Installing the **dotnet-gcdump** tool
2. Running the tool on your application
3. Collecting the results
4. Analysing the results

## Pre-requisites
Just some important things to note before we get into the good stuff:

- Node OS: Linux
- Local machine OS: Windows
- Alpine Linux-based containers
- .NET 6 SDK & Runtime

The steps below are tailored for this particular setup. They may differ in some way depending on what you yourself are running; for example, the **dotnet-gcdump** tool is only available in .NET Core 3.1 and above, so be sure to find the details of your setup before attempting this.

## Installing the dotnet-gcdump tool
Like I mentioned earlier, if your containers are _hardened_ (i.e. your containers are not running as root) you will not be able to directly install this tool onto your running container.

What you need to do instead is add the instructions to your Dockerfile to install the tool onto your container for it then to be available when your container is running:

```
FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine3.14 as build
...
RUN dotnet tool install --tool-path /tools dotnet-gcdump
...
```

In the above example; I am referencing the .NET 6 Alpine SDK base image, from which I am running the `dotnet tool install` command and installing the dotnet-gcdump tool to the `/tools` folder in my container.

## Running the tool
First things first is to check that your tool has successfully installed onto your container. To do this, boot up an interactive terminal session on one of your pods by running the `kubectl exec` command with the necessary arguments. From here, change to the `tools` directory and run the `ls` command to list the files within this folder, and you should see the dotnet-gcdump tool within.

![Booting up a terminal session on a pod](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w2i3eiwyof96znwulamo.png)

While you are in this folder, you will want to run the `dotnet-gcdump ps` command to find the process ID of your running application (if you don't already know what it is). Make a note of this as it will be needed for the next step.

![Running dotnet-gcdump ps command](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k8sypjh0154ra42d0mgk.png)

Now we have everything we need to run the `dotnet-gcdump collect` command. You may find that your application does not have write permissions to the `/tools` folder if you try to run this command without declaring an output file location. If this is the case, declare the output file location to the temporary directory (`/tmp` in linux) which should allow your application to write to.

You will also need to give your file a name. In the below example, I have called the file `memory-dump-test`.

![Running dotnet-gcdump collect command](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6vs9m30w301layjzpf9w.png)

The process should take no longer than a few seconds, and when succeeded, you should see the message `Finished writing XXXX bytes.`

To confirm, change directories to your `/tmp` file and run the `ls` command again to find that your `.gcdump` file has successfully been generated.

![Generated gcdump file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b01dnxrbs6soc5vldxxy.png)

## Collecting the results
Now you have generated your memory dump, the next step is porting this file over from your pod to your local machine so you can read the results. In my example below, I used the `kubectl cp` command. Before executing this command, be sure that you have exited from the terminal session in the previous section!

![Running the kubectl cp command](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4bv75a4km7et2yz08ew4.png)

This command can be fiddly due to needing to match the file path syntax with that of the corresponding operating system (Linux uses '/' while Windows uses '\') as you can see in the screenshot.

There is no success message when this command has completed, but you may or may not see a `tar` message referring to leading '/' from member names, however if you check the output file you declared, you should see that your `gcdump` file has successfully ported over.

## Analysing the results
The resulting `gcdump` file can be opened in Visual Studio.

![gcdump file in Visual Studio](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6nofnaoagcpteun1tst8.png)

This file gives you the count of a specific object, the size in bytes, and inclusive size (which is the size of all objects that are referenced by the type). Visual Studio also gives you the option to select two memory dumps and compare the difference between them.

I won't go into too much detail about these files, as Microsoft themselves have many great, bitesize videos and documents on how to interpret these results, such as [this](https://www.youtube.com/watch?v=SHGeE_PFA4s&ab_channel=dotNET) one.

## Conclusion
And that's all there is too it! The solution is fairly trivial in retrospect, however trying to piece together the separate steps can be a bit of a nightmare especially when security measures are in place. Hopefully you can get to the bottom of your memory issues in peace!