Repository: go
Issue: runtime: ill-fated GC outcome in spiky/bursty scenarios
Labels: GarbageCollector,NeedsInvestigation,compiler/runtime

## Motivation

The Go GC machinery leaves the door open to several ill-fated scenarios, that have been reported in other issues. The one closest to what I'm describing here (and providing a reproduction for) is https://github.com/golang/go/issues/10064.

Imagine the Go program has 64GiB available memory. The last garbage collection resulted in an live set of 54GiB. With the default GOGC=100 value, the pacer will schedule a collection once another 54GiB have been allocated, or in the next 2 minutes, whichever happens first.

If there's a rapid spike of heap allocation due to program mode change (e.g. a database compaction, which is how my team discovered this), such that it amounts to more than 10GiB, an OOM panic will occur.

And that appears to be reasonable behaviour, if those 10GiB are effectively retained / reachable / in scope.

However, what's not reasonable is that the same will occur **even if 9.90GiB of those 10GiB have been released / become unreachable**. For example, the program underwent 99 iterations of this logic, in under 2 minutes from the last GC:

1. allocate a 100MiB slice, populate it.
2. flush it to disk.
3. syntactically the slice goes out of scope (becomes unreachable).

The next iteration (100th) will cause the go runtime to expand the heap beyond its available memory, and that will cause an OOM panic. Instead, I would've expected the runtime to detect the impending OOM, and instead choose to run a forced GC.

The above scenario is greatly simplifying things, of course.

## Reproduction harness

I built a reproduction harness here: https://github.com/raulk/trampoline/.

This program creates a cgroup and enforces the memory limit indicated by the -limit parameter (default: 32MiB). The cgroup's swap memory value is set to the same value, to prevent the program from using any swap. (IMPORTANT: make sure the right cgroup options are enabled to enforce this caging; check README for more info).

The program will then allocate a byte slice of size 90% of the configured limit (+ slice overhead). This will simulate a spike in heap usage, and will very likely induce GC at around 30MiB (with the default limit value).

Of course, the exact numbers are dependent on many conditions, and thus non-deterministic. Could be less or more in your setup, and you may need to tweak the limit parameter.

Given the default value of GOGC=100, the GC pacer will schedule to run when the allocated heap amounts to 2x of the live set at GC mark phase end. In my setup, this clocks in at 60MiB. Of course, that's beyond our 32MiB limit.

Next, the program releases the 90% byte slab, and allocates the remaining 10%. With the default limit value, it releases 30198988 bytes to allocate 3355443 bytes (obviating slice headers).

At that point, the program has enough unused heap space that it could reclaim and assign to the new allocation. But unfortunately, GC is scheduled too far out, and the Go runtime does not run GC as a last resource before going above its limit. Therefore, instead of reusing vacant, resident memory, it decides to expand the heap and goes beyond its cgroup limit, thus triggering the OOM killer.

The gist here is that the Go runtime had 9x times (roughly) as much memory free as it needed to allocate, but it was not capable of reclaiming it in time.

## Discussion & ideas

* I'm not sure if it's at all possible to implement a reactive GC trigger, i.e. trap a failure in malloc and then force a GC run before trying to allocate again. I don't think it is, because most OS will use virtual memory with overcommit modes, so the malloc won't fail. I guess it's not possible either to trap the SIGKILL sent by the OOM killer; it induces immediate program termination.
* All JVM GC algorithms call GC reactively, just like I would've expected Go to do. But the JVM also has a configurable maximum heap size, which Go doesn't (see https://github.com/golang/go/issues/9849) That's probably what makes it possible to implement such a mechanism.
* For large heaps, I think the go runtime should automatically regulate the GC percentage, reducing it as it the heap increases in size. The doubling effect of GOGC=100 (default) is ineffective with large heap sizes, as it allows too much garbage creation before a GC is triggered. Maybe also regulate the forced GCs frequency accordingly, to make it adaptable.

## Workaround

We'll probably end up setting up a memory watchdog, initialized with a user-configured memory limit (à la JVM -Xmx). As the heap grows, we'll probably reduce GOGC dynamically by calling `debug.SetGCPercent`. As the heap approaches the limit, we'll trigger GC more aggressively and manually.

## Related issues

https://github.com/golang/go/issues/42430 (gc pacer problems meta-issue)
https://github.com/golang/go/issues/14735 (throttling heap growth)
https://github.com/golang/go/issues/16843 (mechanism for monitoring heap size)
https://github.com/golang/go/issues/10064 (GC behavior in non-steady mode)
https://github.com/golang/go/issues/9849 (make max heap size configurable)