Repository: go
Issue: runtime: allocation performance worse on two socket server
Labels: NeedsInvestigation,compiler/runtime

<!--
Please answer these questions before submitting your issue. Thanks!
For questions please use one of our forums: https://github.com/golang/go/wiki/Questions
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
1.16.4
</pre>

### Does this issue reproduce with the latest release?
Did not try on latest release


### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env

GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build394973932=/tmp/go-build -gno-record-gcc-switches"
</pre></details>

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
-->
I ran go based application ([TiDB](https://github.com/pingcap/tidb) v5.1, which is compiled by go 1.16.4) on 2 socket server (24c per socket), and compared its performance with on 1 socket (I used numactl to bind application to a single socket).

### What did you expect to see?
Application has better performance on 2 socket comparing with on 1 socket. As 2 sockets have double number of cpu cores.

### What did you see instead?
I found application has worse performance on 2 socket comparing with on a single socket. More specifically, application has 93% performance on 2 socket comparing with on 1 socket.

I used perf top to check hotspots of application. The top hot function is runtime.HeapBitsSetType. The function took 6.45% overhead on 1 socket while took 17.25% overhead on 2 sockets. 

Then I used perf c2c to check if the performance degeneration was caused by cache sharing issue. With the tool, I found runtime.heapBitsSetType is the major source of HITM. And major source of store  came from runtime.(*mspan).sweep (79.16%), runtime.(*mheap).relcaim (13.34%) and runtime.sweepone(7.48%). More detail data as below.

----- HITM ------  -- Store REfs --    -------- CL ------        Functions
RmtHitm  LclHitm   L1 Hit               Off  Node  PA Cnt
2.03%       0.38%     79.16%             0x0        1       1          runtime.(*mspan).sweep
0.86%       0.07%     7.48%               0x30      1       1          runtime.sweepone
0.18%       0.03%     13.34%             0x30      1       1          runtime.(*mheap).relcaim
89.98%     92.52%   0.00%               0x38      1       1          runtime.heapBitsSetType

Then I located the lines of code in above functions:
For runtime.heapBitsSetType, the related line of code is: "ha := mheap_.arenas[arena.l1()][arena.l2()]" (in heapBitsForAddr func of mbitmap.go)
For runtime.(*mspan).sweep, the related line of code is: "atomic.Xadd64(&mheap_.pagesSwept, int64(s.npages))" (in sweep func of mgcsweep.go)
For runtime.sweepone, the related line of code is "atomic.Xadduintptr(&mheap_.reclaimCredit, npages)" (in sweepone func of mgcsweep.go)
For runtime.(*mheap).relcaim, the related line of code is "if atomic.Casuintptr(&h.reclaimCredit, credit, credit-take) {" (in reclaim func of mheap.go)

According to above code, I located the relatived variables defined in mheap struct:pagesSwept, reclaimCredit, arenas. And suppose pagesSwept is the start of a cacheline at 0x00, then reclaimCredit will locate at 0x30, arenas will locate at 0x38. 
It will exactly match output of perf c2c tool (CL off). 

0x00        pagesSwept         uint64  
0x08        pagesSweptBasis    uint64  
0x10        sweepHeapLiveBasis uint64  
0x18        sweepPagesPerByte  float64 
0x20        scavengeGoal uint64
0x28        reclaimIndex uint64
0x30        reclaimCredit uintptr
0x38        arenas [1 << arenaL1Bits]*[1 << arenaL2Bits]*heapArena

According to above analysis, I think the performance issue on 2 socket server can be solved by either adding padding before arenas or changing order of the related variable definition in mheap struct to ensure they locate at different cachelines.