Troubleshooting Containers with Sysdig Inspect
Tags: docker, devops
Published: 2020-06-04 09:23:24

This is the third part of our series about Sysdig. The [first tutorial](https://appfleet.com/blog/sysdig-what-it-is-and-how-to-use-it/) walked you through the process of capturing and analyzing data with the Sysdig command-line utility. The [second tutorial](https://appfleet.com/blog/monitor-your-containers-with-csysdig/) showed how to use Csysdig, the ncurses based GUI. Now, we'll focus on explaining how you can troubleshoot your containers with **Sysdig Inspect**.

#Prerequisites#

Before starting this tutorial, you should follow the first part of the series. We assume that Docker Compose and Sysdig are installed on your system. Also, you should have a running WordPress installation. Enter the following command to verify that everything is properly set up:

```
docker ps
```

```
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
f2a5bf941eba        wordpress:latest    "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1
95c44dd63ba9        mysql:5.7           "docker-entrypoint.s…"   About a minute ago   Up About a minute   3306/tcp, 33060/tcp    wordpress-sysdig_db_1
8405c2c87f59        sysdig/sysdig       "/docker-entrypoint.…"   About a minute ago   Up About a minute                          sysdig
```

Make sure the following three Docker containers are running on your system:

* WordPress
* MySQL
* Sysdig

#Install Sysdig Inspect#

There are two ways in which you can install Sysdig inspect:

* Download the pre-built binary for your operating system from the project's [GitHub repository](https://github.com/draios/sysdig-inspect/releases).
* Run it as a Docker container.

For this tutorial, you will install Sysdig Inspect inside of a Docker container.

In this section, you'll install Sysdig Inspect and make it so that the trace files captured with `sysdig` can be accessed from the Sysdig Inspect container.

1.In a new terminal window (**on the host**), run the following command to create a directory called `captures`:

```
mkdir captures
```

2.You can install Sysdig Inspect in a container with the following `docker run` command:

```
docker run -d -v /home/vagrant/captures/:/captures -p8080:3000 sysdig/sysdig-inspect:latest
```

```
Unable to find image 'sysdig/sysdig-inspect:latest' locally
latest: Pulling from sysdig/sysdig-inspect
f661dfa5eb7d: Pull complete 
572ec58ac70f: Pull complete 
c1d2c84e7140: Pull complete 
1ebe1ad7056b: Pull complete 
e5e1416d36d0: Pull complete 
9b379815df57: Pull complete 
9034f4cfeb96: Pull complete 
91638da20eef: Pull complete 
a4105e889626: Pull complete 
b7359900512b: Pull complete 
2daad7eb7645: Pull complete 
d0fb8981639d: Pull complete 
ed8bc4c05912: Pull complete 
29d1d8034bab: Pull complete 
fead5f9a4603: Pull complete 
Digest: sha256:0c6270a09cfb28f2c248761f4783d0bc53d3dfd9b83d27f03b98039d5366cf45
Status: Downloaded newer image for sysdig/sysdig-inspect:latest
ee88efbc56205164a269102d266509ce5a677d024071d29774b12cd3414a7a69
```

A few things to note about the above command:

* The `-v` flag specifies the files on the host that Sysdig Inspect can access. In this example, the files placed inside of the `/home/vagrant/captures/` directory will be available in the `captures` folder in the container running Sysdig Inspect.
* The `-d` flag starts the container in detached mode
* The `-p` flag publishes the `3000` port to the `8000` port on the host.

3.At this point you can inspect the status of  the Sysdig Inspect container with:

```
docker ps
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qmdplsm2sljtgy3v8s77.png)

Note that the status is currently set to `starting`. Wait a bit until it changes to `healthy`:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/c8cx96b52s0evgy44ogi.png)

4.Now you can point your browser to http://localhost:8080:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xche96y5p92824enla85.png)

#Capture Data#

As you can see in the screenshot above, you can specify the path to a capture file taken with Sysdig. In this section, you'll create a new trace file, simulate an issue by deleting a file, and then load the data into Csysdig Inspect for inspection.

1.In a new terminal window (**on the host**), display the list of your active containers:

```
docker ps
```

```
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
7da3fee501db        sysdig/sysdig-inspect:latest   "/docker-entrypoint.…"   2 minutes ago       Up 8 minutes (healthy)   0.0.0.0:8080->3000/tcp   agitated_leakey
f2a5bf941eba        wordpress:latest    "docker-entrypoint.s…"   8 minutes ago       Up 8 minutes        0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1
95c44dd63ba9        mysql:5.7           "docker-entrypoint.s…"   8 minutes ago       Up 8 minutes        3306/tcp, 33060/tcp    wordpress-sysdig_db_1
8405c2c87f59        sysdig/sysdig       "/docker-entrypoint.…"   9 minutes ago       Up 9 minutes                               sysdig
```

2.Start a shell inside of the `wordpress` container by entering the following `docker exec` command and passing the identifier of the container:

```
docker exec -ti f2a5bf941eba /bin/bash
```

> ☞ In the above example, the identifier of our container is `f2a5bf941eba` but yours will be different.

3.Similarly, start a shell inside of the `csysdig` container with:

```
docker exec -ti 8405c2c87f59 /bin/bash
```

4.Initiate a new capture by entering the following command inside of the `sysdig` container:

```
sysdig -w monitoring-wordpress-error.scap
```

5.Open a new terminal window (**on the host**), and use ApacheBench to make 10000 requests with a maximum of 100 requests running concurrently:

```
ab -n 10000 -c 100 http://localhost:8000/
```

6.While ApacheBench is running, move back to the `wordpress` container and delete the `index.php` file:

```
rm -rf index.php
```

7.When the benchmark is finished, move to the `sysdig` container and stop capturing data by entering "CTRL+C".

8.If everything went well, a file named monitoring-wordpress-error.scap should have been created:

```
ls -la monitoring-wordpress-error.scap
```

```
-rw-r--r--. 1 root root 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
```

9.Switch back to the host, and use the `docker cp` command to copy the capture file to the `captures` directory:

```
docker cp 8405c2c87f59:/monitoring-wordpress-error.scap captures/
```

10.Verify whether the file has been successfully copied to the host with:

```
ls -la captures/
```

```
total 143356
drwxrwxr-x. 2 vagrant vagrant        45 Jan 16 21:02 .
drwx------. 8 vagrant vagrant       208 Jan 16 18:27 ..
-rw-r--r--. 1 vagrant vagrant 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
```

11.Point your browser to http://localhost:8080 and type the path to the trace file (that is `/captures/monitoring-wordpress-error.scap`) in the "capture file path" text area:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hskg8ea88jis57dpt2tf.png)

12.Press the `Enter` key and then wait a bit while Sysdig Inspect loads the trace file:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/u6btdu76xx69xce49hqt.png)

13.When everything is ready, you will see the overview page:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pb6kw7p2beopik0qb1rf.png)

#How to Navigate the Interface#

In this section, we'll explain the basic concepts behind Sysdig's Inspect interface. Once the trace file is loaded, you'll be presented with a screen similar to the one below (the annotations are ours):

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/twukabxbto304jv2k3kc.png)

There's a lot of information packed in here. Let's break this down:

**(A)** At the top of the screen, Sysdig Inspect shows the name of the currently open trace file.

**(B)** The drill-down section shows the list of active filters. Select the "Containers" view and then navigate to the "Processes view":

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/wrlrgvnql6c0esg35hwv.png)

☞ You can customize what data is displayed by manually editing the "Sysdig Filter" textbox.

**(C)** Similar to how Csysdig works, Sysdig Inspect lets you filter the data using several built-in views.

**(D)** The main window contains several tiles. Each tile shows the values of a particular metric and its evolution in the period of time in which the capture was taken. Tiles are grouped together to make navigation easier.

**(E)** The timeline allows you to use two sliders to narrow down the period during which the data is analyzed. Continuing our previous example in which we selected the "Container" view and then "Navigated" to the process view, we used the sliders to narrow down the period of time for which the data is displayed:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/h2n4r9zg74xg5s75svye.png)

We've covered the basics of using the interface! In the next section, we'll show how you can use Sysdig Inspect to troubleshoot your containers.

#Put It to the Test#

Remember that, in the "Capture Data" section, we've purposely introduced an error. Now, we'll make use of that to showcase how you can use Sysdig Inspect to analyze data and troubleshoot your containers.

1.First, navigate to the overview page:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/h8u6pvo1tgdyprwcai70.png)

2.From the "Views" section, select "Errors". You'll see a high number of `ENOENT` (which stands for **E**rror **NO** **ENT**ry) errors:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ysvwyl2uc216du9r25kn.png)

3.Highlight the `ENOENT` line by clicking on it. This will display three round buttons:

* Drill-Down
* I/O Streams
* Syscalls

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/kjav0p6g4o4qy4a9q4tm.png)

4.Select "Drill-down' and Sysdig Inspect will show the list of processes that raised this error:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8n7cq4yj5gb97psff8kb.png)

5.Pick the one from top of the list. Then, select "Syscalls". Since we've deleted the `index.php` file a bit later in the capture, you must scroll down a few lines until you see lots of error messages similar to the ones below:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5acl8do88zle0k3t2bn3.png)

☞ Note that the Sysdig Filter checkbox has been automatically updated.

Now, this sheds some light, but it doesn't tell you what has happened.

6.Let's clear all the filters and navigate to the default view by selecting  "Overview":

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9iu13maw592vblzyyb2d.png)

7.See the "Security" section. It shows that two commands were executed, and one file was deleted.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/v5ofcnn4z3h9o85hhtse.png)

8.To get more details, select the `Deleted Files` tile:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/2i6plw7wird073psb423.png)

Note that the `Drill-down` button is now displayed. Also, the moment when the file was deleted is shown at the bottom of the page.

9.Select the `Drill-down` button, and Sysdig Inspect will show that the `index.php` file was deleted:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/lqm4h4gv0afb9alsq7cr.png)

10.If you select the  `Executed commands -> Drill-down` tile, then you'll see 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/us66msuda1k4dtm965vt.png)

As you can see, Sysdig Inspect displays a host of useful information you can rely on to troubleshoot your containers.

In this example, it wasn't overly difficult to figure out what caused the issue, but there may be cases in which you'll have to dig a bit deeper. The knowledge you acquired is a complete tool belt that'll help you get through in any situation that requires troubleshooting your container-based applications.

Thanks for reading!