I realized that Address Sanitizer recording the stacktrace of an allocation and printing it upon a memory error is a very limited form of introspection, almost like a subset of what rr can capture. It would be cool to take an ASAN-enabled binary and ask the sanitizer who created a particular memory allocation, to trace causality across an unfamiliar codebase and learn the code structure.

> Feel free to email us at inquiries@pernos.co and we can maybe work something out.

Tell me if my email got spam-filtered or not.

> We could do those things but we don't. Before implementing features like that we'd want to understand what the underlying problem is that you're trying to solve --- maybe there's a more direct way.

The reason I wanted SQL-like queries is because I was debugging a complex bug (PipeWire setup for low latencies, using ALSA to output through a USB audio interface, gets stuck in XRUN during startup), and wanted to find all calls that PipeWire made to libasound.so.2, to build a self-contained test case reproducing the bug. But I lost interest in the exact userspace API calls once I started tracing execution in the kernel, and I lost interest in the bug altogether once I discovered that USB audio latencies on Linux are awful even on the "low-latency" modes of PipeWire/jack2.

I do think this is useful in more situations though. For example, GTK3 and GTK4 talk to the X server in different ways, causing GTK4 apps to have incorrect positioning/title bars on KWin and XFCE. It would be cool to take a rr dump, then log all calls into Xlib/xcb in an automated fashion (I never did find the difference, the GTK4 maintainers closed my bug report and didn't investigate, and I don't know enough about X11/Xorg or kwin/xfce to diagnose from the symptoms and my gdb poking). Or perhaps log all interaction between a volume control and libpulse, and so on.

----

Why do I like KCacheGrind-style call graphs? Partly because it's the closest automated visualization to my preferred technique of a hierarchical/tree execution trace. The way I learn/debug, refactor, and design codebases is by tracing control flow as bullets in a "notes" Google Doc. I have a screenshot at https://drive.google.com/uc?id=1KLmGsQS7c9pKan6byh0oZ7VYXSB6....

I generally use tracing when I want to see what high-level operations affect particular low-level state, or to find the effects of a function call, or to see how a program enters an specific/erroneous state. I usually read type definitions and record them in a "types" section, then hand-trace the execution of relevant functions and record the control flow tree (with or without the values of variables) as nested bullet points. Crucially, I turn function calls into nested blocks (just like local scopes), and write down which branches are taken in which situations. The resulting tree (assuming no loops) can be read from top to bottom as a straight-line representation of what the CPU (or C abstract machine) is doing, where causes always come before effects. Whereas in the raw source code, functions are out of order or even in separate files, and virtual calls are impossible to statically resolve without nonlocal information, obscuring control flow to readers.

I personally find this approach very effective for learning code, understanding bugs, and tracing the effects of refactors and bugfixes. Though I find it time-consuming to build control flow bullets while reading/learning existing code, and faster to write bullets before writing new code. And I don't know if others find this technique as useful as I do.

----

I don't know if there's a better way for me or others to understand code, but I haven't found any yet. UML is a joke(?). Doc generators (like Rustdoc and Doxygen) are built to teach APIs and abstract away implementations, not to teach implementations. Sourcetrail is a (discontinued) code browsing/reading tool. In my experience, it functioned like IDE "go to definition" but with fancier graph-based presentation, and didn't help me understand code better than an IDE did. Maybe it was because it didn't show multiple levels of function calls on a single screen, or didn't let me annotate code with values and branches taken when running the code, or it didn't produce "notes" summarizing what the code does in a particular situation, which I can read without having the source on hand.

I like how in Google Docs, I can use highlighting to color-code types/variables in different expressions as having the same value. I dislike Google Docs's slow loading and bloat, deep bullet indentations, bullet nesting limit, and lack of code blocks. I dislike how Markdown interprets asterisks and underscores in code as italics (in-band coding ambiguity) unless you spam backslashes (impairing plaintext readability) or backticks, lacks color-coding, and lacks cloud sync. Other than those issues, Markor is a pretty good WYSIWYG-ish Markdown editor (I find it awkward to cross-reference in a dual-pane editor, and editing Markdown in a monospace code editor is "good enough" if you're fine with monospace and no images).

----

Unfortunately this technique doesn't extend well to asynchronous code, where one function "schedules" another function to run, in an event loop in the same or another process. Though I feel asynchronous code intrinsically makes it difficult to understand what causes what (I often give up debugging when I see a process in poll() waiting for "who knows what"). I usually struggle at first, then once I figure out, I hand-wave "this causes this to happen" in my notes (in the above pic, core_method_marshal_sync in one process "invokes" core_method_demarshal_sync in another).

(I wonder if you can more easily trace poll() causailty, by running a process in strace and looking at what created the file descriptor that poll() is blocking on. But I haven't had the chance to practice this much. Maybe I'll try on the Firefox extension "copy multiple tabs" notification hang.)