Repository: pytorch
Issue: Symbolicate crashes from releases
Labels: triaged

## 🚀 Feature

PyTorch releases are built in “Release” mode (go figure), which means they do not include debugging information. Some symbols (those that we export via TORCH_API) are present in the release libraries, but that’s it. This makes it hard to track down user bugs since they often come from releases and devs have to reproduce it locally or walk users through generating a backtrace.

## Pitch

This stuff all only applies to Linux/GCC for now for simplicity's sake, but could be expanded to all platforms in the future. Also it's very similar to [Chromium's crash reporting](https://chromium.googlesource.com/breakpad/breakpad/+/master/docs/getting_started_with_breakpad.md).

### Crash dumps

Linux automatically records certain process failures into a core dump file, which is a splat of the process' memory into a file along with some other information. GDB can load core dumps so developers can diagnose crashes after they happen. The core files can potentially be huge (depending on how much memory allocation the process has done):

```bash
# Enable infinite core dump size
$ sudo ulimit 0
# Send core dumps to /tmp/cores
$ mkdir -p /tmp/cores
$ sudo sysctl kernel.core_pattern=/tmp/cores/core.%e.%p.%h.%t

$ python
>>> import torch
...

$ kill --signal SIGSEGV <python process PID>
$ ls -lh /tmp/cores
-rw------- 1 user users 699M Apr 19 11:17 core.1053640
```

700 MB is a little more than GitHub will allow attached to an issue and is cumbersome to send over other channels like email. Minidumps solve most of these problems by being "mini"-er than core dumps (i.e. they exclude memory far away from `rsp` at the time of the crash). The code for this is already landed (but relies on [`libbreakpad`](https://github.com/google/breakpad) to be installed on the system):


```bash
$ cat <<EOF > test.py
import torch

torch.utils._crash_handler.enable_minidump_collection()

@torch.jit.script
def fn() -> int:
    return '2'
EOF

$ python test.py
Wrote minidump to /tmp/pytorch_crashes/646c522c-7710-4639-f1678895-2c15f7c3.dmp                                               
Traceback (most recent call last):                                                                                            
  File "/data/users/davidriazati/pytorch/test.py", line 6, in <module>                                                        
    def fn() -> int:                                                                                                          
  File "/data/users/davidriazati/pytorch/torch/jit/_script.py", line 1047, in script                                          
    fn = torch._C._jit_script_compile(
....

$ ls -lh /tmp/pytorch_crashes
976K Apr 19 13:06 646c522c-7710-4639-f1678895-2c15f7c3.dmp

# minidumps compress pretty well too
$ tar -czvf out.tar.gz /tmp/pytorch_crashes/646c522c-7710-4639-f1678895-2c15f7c3.dmp
$ ls -lh
41K Apr 19 13:07 crash.tar.gz
```

Users can then attach minidumps to their bug reports, then developers can convert them to core files via Chromium's minidump tools and debug as usual. But these are still missing rich debug info / symbols since they're not there in Release mode.


### Debug Information

We build in `RelWithDebInfo` mode (or something similar via `-gsplit-dwarf`) and use the ye olde technology of [Debug Fission](https://gcc.gnu.org/wiki/DebugFission) to split out the debug information into a separate binary. This way the debug info can be associated with a particular release while still keeping the release binary size small:

```bash
cp build/lib/libtorch_cpu.so build/lib/libtorch_cpu.so.dbg
strip --only-keep-debug build/lib/libtorch_cpu.so.dbg
strip --debug build/lib/libtorch_cpu.so
cd build/lib

# This will tell 'gdb' to look in a series of known directories (cwd, /usr/local/debug, etc.)
# for the 'libtorch_cpu.so.dbg' file and read that for symbols. This also adds
# a CRC32 of the library into libtorch_cpu.so to ensure that the right library
# is used.
objcopy --add-gnu-debuglink=libtorch_cpu.so.dbg libtorch_cpu.so
```

We would then host these debug symbols at download.pytorch.org/whl/debug named by their CRC32's, so then we could have an API in PyTorch that reads libtorch_cpu.so's `.gnu_debuglink` section, fetches the correct debug information from pytorch.org, then installs it so `gdb` will transparently symbolicate crashes.

This will also address #11982

## Alternatives

Do nothing, keep repro-ing issues manually and being sad.