An Investigative Debugging Toolbox
Tags: resources, debugging, investigativedebuggi
Published: 2020-11-08 18:23:23

---
title: An Investigative Debugging Toolbox
published: true
date: 2020-05-24 01:29:50 UTC
tags: Resources,debugging,investigativedebuggi
canonical_url: https://avdi.codes/an-investigative-debugging-toolbox/
---

Last week I ran an experimental [workshop on _Investigative Debugging_](https://avdi.codes/investigative-debugging-workshop/)_._ It was a lot of fun and the attendees seemed to get a lot out of it. Afterwards I sent out some extra reference material, including this Linux-oriented list of investigative debugging tools!

The focus here is on “surveillance tools” – utilities that give you observability into exactly what programs are _really_ doing at runtime. It’s far from complete, but it represents a ton of research and I hope you find it helpful!

| Tool | Description |
| --- | --- |
| [BCC](https://github.com/iovisor/bcc) tools | A collection of over 70 specialized tracing and performance profiling tools built on BPF. |
| `bpftrace` | A Linux tool analogous to DTrace, built on BPF. `bpftrace` enables users to compose “questions” at the command-line, using a concise [AWK](https://en.wikipedia.org/wiki/AWK)-inspired syntax, for which they would once have had to use (or create) a dedicated tracing tool. It is often possible to recreate BCC tools using bpftrace commands. But when a dedicated BCC tool exists for a given query, using it may be easier than hand-rolling a `bpftrace` command. |
| Docker | A technology for isolating Linux-based services into _containers_, each with their own isolated filesystem, process table, and virtual network. Containers are useful from an investigative debugging standpoint because they enable us to isolate and observe how a process interacts with its environment. |
| `docker diff` | Get a comprehensive list of all files changed within a Docker container. |
| `ftrace` | A low-level, “kind of janky” (Julia Evans’ words) interface to Linux kprobe and uprobe tracing. |
| `HTTP_PROXY`, `http_proxy`, `HTTPS_PROXY` | These environment variables are _sometimes_ respected by programs as a way to configure a SOCKS proxy. |
| `LD_PRELOAD` | An environment variable that can be used to force a program to link to arbitrary libraries before running. This can be used to substitute user-provided functions that override default behavior with e.g. tracing instrumentation. |
| `ldd` | List all the libraries that an ELF executable links to. |
| `lsof` | List open filehandles, including network connections. |
| [LTTng](https://lttng.org/) | A linux tracing framework. I believe this is now one of the inputs to BPF? |
| `ltrace` | Trace library calls. |
| `mitmproxy` | An interactive HTTP proxy server that can log and inspect requests and responses. Can be configured as either a SOCKS proxy or a transparent proxy. |
| `nm` | List all the _names_ (e.g. function names) in an ELF executable/library. |
| [Postman](https://www.postman.com/) | A workspace for experimenting with and documenting HTTP calls, which can also act as a `mitmproxy`-like request inspector for other processes. |
| `readelf` | A general tool for inspecting and dumping information about ELF executables and libraries. |
| `strace` | Trace system calls. |
| `strings` | Dump all the static strings found in an ELF executable or library. |
| `tplist` | A BCC tool for listing all USDT tracepoints available in a process. |
| `uflow` | A BCC tool for tracing method calls and returns in high-level languages like Python, Ruby, and Java. May be installed somewhere weird and not in the path, like `/usr/sbin/lib/uflow` |
| [WireShark](https://www.wireshark.org/) | The preeminent tool for tapping into and dissecting IP network traffic at the packet level. |