Repository: go
Issue: proposal: x/debug: make the `core` and `gocore` packages public
Labels: Proposal

Authors: @mknyszek @prattmic @aclements 

## Purpose

The purpose of this proposal is to help fill in an important gap in Go debugging by providing a foundation for Go heap analysis. The primary use-cases for heap analysis (vs. heap profiles) are to pinpoint memory leak issues and debug out-of-memory errors by providing an in-depth look at the memory graph. Other use-cases exist as well, mostly related to the large amount of detail available in typical heap analysis software.

Go currently has two heap analysis options: analyzing core files for Go processes (via `golang.org/x/debug/cmd/viewcore`) or analyzing heap dumps (produced by `runtime/debug.WriteHeapDump`). As of today, both of these options are not easily available to users. The former has bit-rotted, while the latter lacks working tooling (and so may not actually work either).

Some companies using Go have expressed interest in building their own heap analysis tooling, while members of the Go community have contributed time and effort to updating `viewcore`. The utility of such functionality is lauded in the Java world, especially when it's available for servers over a connection (e.g. `net/http/pprof`).

## Plan

We propose the following plan for improving the state of Go heap analysis, with the core API change here being to make the `golang.org/x/debug/internal/core` and `golang.org/x/debug/internal/gocore` packages public.

Before that can happen, however, we also need to fix them and prevent them from breaking again by:

- [ ] Updating `gocore` and `core` to work with Go at tip.
- [ ] Ensure that `gocore` and `core` are well-tested, and tested at tip.
    - [ ] Running the full test suite for a core dump created from a Go binary built at tip.

After that, I propose making `gocore` and `core` externally-visible APIs, allowing Go developers to build on top of it with some fairly minor changes to the API that I would like to hash out. To identify the full scope of the API changes and get feedback on the design, we plan to first fork the API into `x/exp` before moving them back to `x/debug`. To let `viewcore` take advantage of improvements in `x/exp`, and as a data point in trialing the API, `x/debug` will temporarily depend on `x/exp` in the process.

Once the API moves to its final home in `x/debug`, I would like to lastly propose

- [ ] Setting up the `golang.org/x/debug` builder to run against the main Go repository.

in order to identify future breakage of these APIs as early as possible in the development of the Go toolchain.

These small steps would bring back heap analysis to the Go ecosystem and will ensure it continues to work.

Since one goal of this package is to support third-party tooling, we need a compatibility story. Simultaneously, these packages rely on and poke at Go internals that are subject to change with each release. Exposing these internals to the Go 1 compatibility promise would likely be too large of maintenance burden to carry forever across all Go versions. As a compromise, I would like to propose the following policy, which will be posted in the package's documentation:
- We guarantee compile-time compatibility: upgrading `x/debug` versions will never break code at compile-time.
- We guarantee support for analyzing core files created from Go programs built using one of the last two major Go releases.
- Support for core files created by Go programs built with older Go versions will be best-effort.
    - In the interest of user-friendliness we will make an effort to making it keep older Go versions working, but if package maintenance becomes untenable we reserve the right to remove support.
    - If a core file is too old to work with an `x/debug` version, it will be reported by the gocore package.

`viewcore`, though minimalistic, provides a proof-of-concept for the `gocore` library that we hope will motivate Go developers to build fancier tooling for heap analysis. Documentation will be critical to making this tooling more widely known ("how to debug a memory leak with `viewcore`").

## Potential future work

I want to also mention some ideas for future work to gauge interest, even though they're not relevant to the proposal directly.

1. Stub out `runtime/debug.WriteHeapDump` to reduce eliminate some technical debt.
2. Provide a way to construct a core file at runtime and expose it through `net/http` (perhaps via `WriteHeapDump`).
3. Reduce dependence on DWARF information for type-casting regions.

The goal of (1) and (2) is to replace WriteHeapDump with a way to produce a process snapshot that works with existing tooling (i.e. gocore and viewcore). Constructing our own core file isn't terribly difficult, provides us with a stable binary format specification so we don't have to go through the trouble of defining our own, and is a format already automatically generated by Linux, so we have fewer formats to support.

The goal of (3) would be to improve the debugging experience with core dumps from stripped binaries. This might not need any additional API, but should be considered up-front.