Repository: go
Issue: runtime: large address space footprint on 32-bit linux
Labels: NeedsInvestigation,compiler/runtime

<!-- Please answer these questions before submitting your issue. Thanks! -->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.13.4 linux/arm
</pre>

### Summary

Running a simple golang program on linux/arm (or other 32-bit linux arch) shows ~800MB of the virtual address space is reserved up front:

```golang
package main

import (
	"fmt"
	"os"
	"os/exec"
)

func main() {
	cmd := exec.Command("pmap", "-x", fmt.Sprint(os.Getpid()))
	cmd.Stdout = os.Stdout
	cmd.Run()
}
```

```
8627:   /tmp/testpmap
Address   Kbytes     RSS   Dirty Mode  Mapping
00010000     624     444     444 r-x-- testpmap
000b0000     704     176     176 r---- testpmap
00160000      72      36      36 rw--- testpmap
00172000      68      28      28 rw---   [ anon ]
01400000    4096     196     196 rw---   [ anon ]
01800000  524288       0       0 -----   [ anon ]
66cb4000     644      52      52 rw---   [ anon ]
66d55000  264060       0       0 -----   [ anon ]
76f34000     256      32      32 rw---   [ anon ]
76f74000       4       0       0 r-x--   [ anon ]
7ee2c000     132      12      12 rw---   [ stack ]
ffff0000       4       0       0 r-x--   [ anon ]
-------- ------- ------- -------
total kB  794952     976     976
```

---

The Linux kernel can be compiled with one of three different vmsplit modes: [VMSPLIT_3G, VMSPLIT_2G, VMSPLIT_1G](http://thinkiii.blogspot.com/2014/02/arm32-linux-kernel-virtual-address-space.html)

![](http://1.bp.blogspot.com/-6l3o55sxVDQ/Uv8nW6TqGMI/AAAAAAAAK7c/iy1EeB0tb90/s1600/source.png)

---

I have a golang program that makes heavy use of memory mapped files, and runs on ARM appliances where the vendor has compiled their kernel with VMSPLIT_2G. Since the golang runtime reserves 40% of the available address space up-front, my program is limited in the amount of files it can `mmap` for its own purposes. When the address space is exhausted, bad things happen including panics when trying to spawn threads:

```
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0xb68bd6 m=2 sigcode=4294967290
```

---

The golang runtime documents its memory mappings in malloc.go. The ~800MB reserved is made up of two large reservations: the first is ~258MB and the second is 512MB.

https://github.com/golang/go/blob/a23f9afd9899160b525dbc10d01045d9a3f072a0/src/runtime/malloc.go#L541-L545

https://github.com/golang/go/blob/a23f9afd9899160b525dbc10d01045d9a3f072a0/src/runtime/malloc.go#L551-L552

---

The 512MB initial reservation can be tweaked with a patch to `arenaSizes`:

https://github.com/golang/go/blob/a23f9afd9899160b525dbc10d01045d9a3f072a0/src/runtime/malloc.go#L586-L592

However, it is not clear to me how I can reduce the ~258MB reservation. The code comment states "We could reserve a smaller amount of space up front if this is problem.", so I'm looking for some guidance on how to do this.

Since userspace on 32-bit linux can access a maximum of 3GB of memory (when the kernel is compiled with `VMSPLIT_3G` [which is the default]), it seems like at the very least the 258MB reservation could be reduced by 1/4 to ~198MB.

cc @aclements