Repository: go
Issue: cmd/link: generate external DWARF debuginfo archives directly
Labels: Proposal,Proposal-Accepted,compiler/runtime

TL;DR: Add an `-external-debuginfo` command line argument to `go tool link` that causes the `link` command to write [a `.debug` archive](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html) containing symbol tables and DWARF debuginfo for the executable being linked.

`-w` and/or `-s` will continue to have their current effects on the output executable. These flags will have no effect on the archive produced by `-external-debuginfo`, which will always include the full symbol table and DWARF debuginfo.

This could be implemented by using the functionality of elfutils `eu-strip` in `cmd/link` / `pkg/tool/link` via `libelf`.

For Windows targets a `.pdb` file could be produced, or external debuginfo support for Windows targets could be initially omitted if it's too hard to do at the same time.

Rationale
---

Kubernetes project binaries and container images are currently built and distributed with the `-w -s` linker flags, so DWARF debuginfo is omitted then the executable is stripped of all symbols.

This saves disk space by producing smaller binaries and container images, though it has no memory consumption or performance benefit at executable runtime.

The cost is that [debugging such executables is extremely difficult](https://github.com/kubernetes/kubernetes/issues/73986) - only numeric addresses are available, with no symbols for functions, variables, etc. If a problem only occurs in a production or near-production system and you want to inspect the running component, you will see backtraces like:

```
(gdb) bt
#0  0x000000000046fc63 in ?? ()
#1  0x0000000000432fc6 in ?? ()
#2  0x00000000048e2110 in ?? ()
#3  0x0000000000000080 in ?? ()
#4  0x0000000000000000 in ?? ()
```

In traditional gcc or llvm based C/C++ build tool chains a balance between executable size and debuggability is maintained using [external debuginfo archives](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html). These are supported by `gcc`, `lldb` and anything using `elfutils` or `libgdb` for symbol loading. Delve supports loading of external symbols from `/usr/lib/debug/.build_id` and `/usr/lib/debug/{path}` etc, like `gdb` and other tools.

Neither GNU `ld` and LLVM `ld.lld` provide a command line argument for writing external debuginfo archives. Instead the release process for most binaries includes a separate pass to split the original executable with debuginfo into separate stripped executable and debuginfo archive files using `objcopy --only-keep-debug`, `eu-strip` and `objcopy --add-gnu-debuglink`.

The golang toolchain tries to be an integrated single-stop shop without external linker dependencies etc, so golang projects including the official k8s distribution don't tend to use this workflow. They're release-size conscious so they just don't provide any debuginfo. They just build stripped binaries for release. This is convenient but makes it much more difficult to diagnose problems.

`go/link` should instead support directly generating an external debuginfo archive ready for upload to a [debuginfod](https://sourceware.org/elfutils/Debuginfod.html) symbol server, inclusion in a container image's `/usr/lib/debug` and `/usr/lib/debug/.build_id` trees, and/or distribution in release tarballs.

Note that [`delve` already supports debuginfod](https://github.com/go-delve/delve/blob/8c392d2fdf9c550bac1ad26b72da96790f086abe/pkg/proc/debuginfod/debuginfod.go). If `go/link` provided an easy way to generate external symbol archives, the k8s project or an interested 3rd party could host a public debuginfod with debuginfo for all future release binaries for use in `gdb`, `delve`, `lldb`, etc.

For example in gdb, one might add `https://debuginfod.k8s.io/` to the `DEBUGINFOD_URLS` env-var, add `set debuginfod enabled` to `.gdbinit`, and have symbols automatically downloaded on demand when debugging k8s release binaries. Even over a remote `gdbserver`. A similar approach is possible with Delve.

Symbols can also be downloaded on-demand from a debuginfod with `eu-unstrip` then used as normal local detached symbol archives, and `eu-make-debug-archive` can be used to make a full archive of a system or container's libraries and debuginfo for remote debugging use. But the symbols must be available for that to be possible.

Interim workaround
---

Golang binaries can be built with debuginfo and symbols (neither `-s` nor `-w` `LDFLAGS` should be used) then split into separate debuginfo with `eu-strip` for linux targets.

This requires each build script to implement features that require dependencies that aren't part of the main go toolchain. So it's unlikely to see wide adoption until and unless the golang toolchain itself provides a standard way to do it like it has with `-s` for stripped executables.

But it's not overly complicated:

```
mv mybin mybin.tmp
eu-strip -o mybin -f mybin.debug mybin.tmp
rm mybin.tmp
```

will yield a stripped `mybin` with embedded external debuginfo link and a detached debuginfo `mybin.debug`. If `mybin.debug` is placed in the appropriate location in `/usr/lib/debug`, `/usr/lib/debug/.build_id`, and/or on a debuginfod server, tools like `dlv` and `gdb` will automatically find its symbols when attaching to it.