Repository: go
Issue: runtime/pprof,net/http/pprof: improve delta profiles efficiency and correctness
Labels: Performance,NeedsInvestigation

### Proposal Details

# The issue
In Golang, allocation, mutex and block profiles are cumulative. They only grow over time and show allocations/blocks that happened since the beginning of the running program.
Not only values grow, but the size of the profile itself grows as well. It could grow up to megabytes in size for long-running processes.

In many cases, it's more useful to see the differences between two points in time.
You can use delta profile from net/http/pprof package.
Using the delta profile requires passing seconds argument to the pprof endpoint query.

```
go tool pprof http://localhost:6060/debug/pprof/heap?seconds=30
```

What this does:
1. Dump profile `p0`
2. Sleep
3. Dump profile `p1`
4. Decompress and parse protobuf `p0`
5. Decompress and parse protobuf `p1`
6. Subtract `p0` from `p1`
7. Serialize protobuf and compress the result

The resulting profile is *usually* much smaller (`p0` may be megabytes, while the compressed result is usually tens of kilobytes).

There are number of issues with this approach:

1. Heap profile contains both allocation values and in-use values. In-use values are not cumulative. In-use values are _corrupted_ by the subtraction.
  **Note:** It can be fixed if `net/http/pprof` package would use `p0.ScaleN([]float64{-1,-1,0,0})`, instead of `p0.Scale(-1)` for memory profiles  - that would subtract allocation values and zero out in-use values in `p0`.
2. It requires dumping two big profiles.
3. It produces a lot of allocations putting pressure on GC.


## DataDog's fastdelta

DataDog's [fastdelta profiler](https://github.com/DataDog/dd-trace-go/blob/30e1406c2cb62af749df03d559853e1d1de0e3bf/profiler/internal/fastdelta/fd.go#L75) uses another approach. 

It improves the runtime/pprof approach by keeping a copy of the previous profile and subtracting the current profile from it.
The fastdelta profiler uses a custom protobuf pprof parser that doesn't allocate as much memory.
This approach is much more efficient, faster, and produces less presure on GC. It also doesn't require using two profiles. 
However, the fastdelta profiler still parses huge profiles up to megabytes, just to discard most of it.

## Grafana's godeltaprof

[godeltaprof](https://github.com/grafana/pyroscope-go/tree/main/godeltaprof) does a similar job but slightly differently.

Delta computation happens before serializing any pprof files using `runtime.MemprofileRecord` and `BlockProfileRecord`.
This way, huge profiles don't need to be parsed. The delta is computed on raw records, all zeros are rejected, and results are serialized and compressed.

The source code for godeltaprof is based (forked) on the original [runtime/pprof package](https://github.com/golang/go/tree/master/src/runtime/pprof).
godeltaprof is modified to include delta computation before serialization and to expose the new endpoints.
godeltaprof relies on a bunch of golang runtime internal functions, specifically `runtime_FrameStartLine`, `runtime_FrameSymbolName`,   `runtime_expandFinalInlineFrame` and `runtime_cyclesPerSecond` [link](https://github.com/grafana/pyroscope-go/blob/main/godeltaprof/internal/pprof/stub_go23.go)  and potentially even [more](https://github.com/golang/go/issues/67941#issuecomment-2162125504) internal functions . Relying on internal functions becomes harder and more dangerous due to https://github.com/golang/go/issues/67401.


# Proposal

We propose to  to allow efficient delta memory, mutex, block profiles collection, both in `runtime/pprof` for push-based integrations and `net/http/pprof` for scraping integrations.

The key points for improvements:
- It should not require dumping two profiles
- It should not require gzip decompressing and pprof parsing
- Delta memory profile should be correct. (Either the inuse values are not corrupted, or `alloc_*` and `inuse_*` values are put into separate profiles)

The specifics of API and implementation details are left to be determined during discussion of the issue after we see an agreement we need to address the issue and this is something that could be accepted into golang runtime