Monitor Your Containers with Csysdig
Tags: docker, linux
Published: 2020-05-12 18:05:20

This is the second part of our series about Sysdig. Whereas our previous blog post focused on capturing data with the `sysdig` command-line utility, this tutorial will show how you can use Csysdig, the ncurses based GUI that ships with Sysdig. It's worth mentioning that Csysdig goes beyond what most monitoring tools provide in terms of functionalities, by giving you the ability to **perform various actions** directly from its interface. As an example, you can start a shell inside of a container, display logs, kill a container and many more.

#Prerequisites#

As a prerequisite, we assume you followed our [first tutorial on Sysdig](https://appfleet.com/blog/sysdig-what-it-is-and-how-to-use-it/).  If so, both Sysdig and Csysdig should be installed on your system. Also, a capture file called `monitoring-wordpress.scap` should already exist.

#Command-line Options#

If you enter the `csysdig` command with the `-h` option, the list of command-line options is displayed as in the following output:

```
csysdig --help
```

```
csysdig version 0.26.4
Usage: csysdig [options] [filter]

Options:
 -A, --print-ascii  When emitting JSON, only print the text portion of data buffers, and echo
                    end-of-lines. This is useful to only display human-readable
                    data.
 -B<bpf_probe>, --bpf=<bpf_probe>
                    Enable live capture using the specified BPF probe instead of the kernel module.
                    The BPF probe can also be specified via the environment variable
                    SYSDIG_BPF_PROBE. If <bpf_probe> is left empty, sysdig will
                    try to load one from the sysdig-probe-loader script.
 --cri <path>       Path to CRI socket for container metadata
                    Use the specified socket to fetch data from a CRI-compatible runtime

 --cri-timeout <timeout_ms>
                    Wait at most <timeout_ms> millisecond
```

*Note that the above output was truncated for brevity*.

Let's look at some of the most important flags:
* `-r`. You can analyze an existing trace file by entering the `csysdig` command followed by the `-r` flag and the name of the capture file:

```
csysdig -r monitoring-wordpress.scap
```

This will display the default view that lists the processes for the period in which the events were captured:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/27xm69m101nw4uf3pd0b.png)

* `-d`. If you don't use the `-r` flag, then you can run Csysdig with the `-d` flag to specify the delay between updates, expressed in milliseconds.
* `-pcontainer`. Configures Csysdig to display data in a container-friendly format:

```
csysdig -r monitoring-wordpress.scap -pcontainer
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/x47ddhx6k2168vror3xu.png)

☞ For the scope of this tutorial, you'll use the capture file created in the previous tutorial - `monitoring-wordpress.scap`.

#What is a View?#

At first glance, the Csysdig interface looks similar to the one provided by tools such as `htop`. However, Csysdig is based on the concept of **views**, which offers a larger array of functionalities and makes it easily customizable.

> Under the hood, **a view is a Lua script** that makes use of the Sysdig processing engine to determine how metrics are collected, processed, and displayed.

The views are usually placed in the `/usr/share/sysdig/chisels` and `~/.chisels` directories.

Enter the following commands to display the list of views:

```
ls  /usr/share/sysdig/chisels
```

```
COPYING                memcachelog.lua        topcontainers_cpu.lua    v_directories.lua             v_procs_cpu.lua
ansiterminal.lua       netlower.lua           topcontainers_error.lua  v_docker_events.lua           v_procs_errors.lua
around.lua             netstat.lua            topcontainers_file.lua   v_errors.lua                  v_procs_fd_usage.lu
```

*Note that the above output was truncated for brevity.*

You can also access the list of views from inside of Csysdig, by pressing the `F2` key:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/clr71x1f9fvdfnu352h9.png)

You will see the list of views and a detailed description of the highlighted view:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/52liasazgyrz6wbjaxb2.png)

Use the arrow keys to navigate to a different view:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/knb6xqct5cp93woowyfg.png)

☞ When navigating through various Csysdig screens, keep in mind the following principles:

* `F2 ` **switches** to a different view
* `Enter` **drills down** into the active selection
* `Backspace` navigates to the **previous** screen

#The Containers View#

Continuing with our example, we selected the `Containers` view and applied it with the `Enter` key:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xtl2qzn9i30ujhdugcda.png)

This provides a brief overview of the three containers running on your machine. Next, you can follow these steps to get in-depth information:

1.Navigate to the first container in the list and press the `Enter`key. You will be presented with the list of **processes** running in this container. At the top of the screen, Sysdig shows the list of active filters. This means you can retrieve the same information stringing these filters in the command-line.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8whqd2nm1wuix1n0fnn1.png)

2.With Csysdig, you can dig even further. Select a process from the list and then press the `Enter` key again. This will display the list of **threads** for the selected process. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/u47yqcilm8ct9oq705ia.png)

3.From here, you have two options:
**Option A: Press the Enter key**
**A.1** You will be presented with the **list of files** that were accessed by this thread during the period in which you captured events:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/elwjzfayfyjjsvgwwioh.png)

As you can see, the list of filters is getting pretty long. Imagine doing this without a GUI😉.
**A.2** Csysdig allows you to retrieve even more detailed information. Select the `.htaccess` file and press the F5 key. This shows the **inputs and the outputs for the selected file.**

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/44j96egg9lk5e8yfn4eu.png)

**Option B: Press the `F5` key**
* **B.1** This shows the I/O activity of the selected process.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ol0bi4g5zcg5p2ph03ma.png)

* **B.2** If you press the `F2` key, Csysdig allows you to select the format in which the data is formatted:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ua4wxkdfgf81w3ga7j4w.png)

* **B.3** Select `Printable ASCII`. Then, you can use the arrow keys to navigate through this view:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/g9x9b40fewtbhw2ejqcy.png)

The screenshot above shows the activity of the selected process. As you can see, it accepted an incoming connection initiated by *ApacheBench*, read from the `.htaccess` file and so on.

☞ Similarly to the `Containers` view, you can access any other view.

#The Actions Panel#

As mentioned, Csysdig lets you perform actions directly from its interface. In this section, we'll look at some examples of you can use Csysdig's actions panel.

The `sysdig` container is pretty barebones and, out of the box, it lacks the utilities needed to make the most of the actions panel. For simplicity's sake, we'll analyze the capture file on the host.

1.On the host, list the active containers with:

```
docker ps
```

```
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
67c012a9e92b        wordpress:latest    "docker-entrypoint.s…"   24 hours ago        Up 24 hours         0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1
f81ef16a5043        mysql:5.7           "docker-entrypoint.s…"   24 hours ago        Up 24 hours         3306/tcp, 33060/tcp    wordpress-sysdig_db_1
0cd95a6d2e4f        sysdig/sysdig       "/docker-entrypoint.…"   24 hours ago        Up 24 hours                                sysdig
```

2.Copy the trace file to the host by running the `docker cp` command followed by the source and the destination path. The source path is formed by concatenating the `ID` of the container and the path inside of the consider, separated by :. In our example, the source path is `0cd95a6d2e4f:/monitoring-wordpress.scap` but yours will be different.

```
docker cp 0cd95a6d2e4f:/monitoring-wordpress.scap .
```

Now that you've copied the trace file to the host, we'll walk you through two of the most useful action panels.

###The Action Panel for Containers###

1.On the host, start Csysdig with the following command:

```
csysdig -r monitoring-wordpress.scap
```

2.Navigate to the `Containers` view and then press the `F8` key. On the left side, you will see the list of available actions:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9r9vb0j5ge2qoocdu3ru.png)

3.As an example, let's start a shell inside of the `wordpress` container by pressing the `b` key.
4.At this point, you can perform whatever actions you want:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/gp5467eedk0qpvfjpm7f.png)

5.When you're done, enter the `exit` command, and you will be taken back to the GUI.
6.From here, you can press the `l` key to display Docker logs:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o94cjf72wxc3f11jg7ji.png)

Similarly, you can invoke the rest of the commands.

###The Action Panel for Processes###

1.In the main view, highlight a process and then press the `F8` key. This will display a different set of actions:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4gmkp63p3gy5bq9w8vce.png)

From here, you can choose whatever action you want to perform by pressing the corresponding hotkey.

###Configure Csysdig###

In this section, we'll show how you can customize Csysdig to your preferences. The following example will:

* Change the hotkey that triggers the `docker pause` command.
* Add the `docker ps` command to the list of available commands and assign a hotkey to it.

1.On the host, open (**as root**) the `/usr/share/sysdig/chisels/v_containers.lua` file. As an example, if your editor of choice is `vi`, then you should enter the following command:

```
sudo vi /usr/share/sysdig/chisels/v_containers.lua
```

2.Set the hotkey for the `docker pause` command to `p`:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8jo6zwc5rch2i7ysiazx.png)

3.At the bottom of the file, add a new command by pasting the following snippet into the `actions` object:

```
{
	hotkey = "o",
	command = "docker ps",
	description = "docker ps"
},
```

With these updates, your `/usr/share/sysdig/chisels/v_containers.lua` should look similar to the following:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4e4hbri5tmzvtkfwydpq.png)

4.Save the file. If you're using `vi`, type `:wq!` and then press the `Enter` key. 

> An in-depth explanation of how you can write more complex views and actions is outside of the scope of this tutorial.  Refer to the [Csysdig View Format Reference](https://github.com/draios/sysdig/wiki/csysdig-View-Format-Reference) page to further your knowledge.

5.Start Csysdig and then navigate to the `Containers` view. From there, press the `F8` key to display the list of actions:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/nb94drfu4g7927yufxz8.png)

Note that the list of actions has been updated!🤓

6.Press the `p` key to pause the container. You'll see the following confirmation message:

```
67c012a9e92b
Command finished. Press ENTER to return to csysdig.
```

In the output above, the `id` of our container is `67c012a9e92b`. Yours will be different.

7. You can use the new command you've just added to check if everything went well. Type the `o` key, and you'll see something similar to this output:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/54wse1uh4ad34q97eez1.png)

Hooray, your container has been paused!

This concludes the second part of our series about Sysdig. In part three, we'll focus on container troubleshooting and security investigation with Sysdig Inspect.