Repository: go
Issue: runtime: reducing preemption in suspendG when G is running large nosplit functions
Labels: Performance,NeedsInvestigation,compiler/runtime

<!--
Please answer these questions before submitting your issue. Thanks!
For questions please use one of our forums: https://github.com/golang/go/wiki/Questions
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
tip version, on arm64
</pre>

### Does this issue reproduce with the latest release?
Yes.


### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env

</pre></details>
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/xiaji01/.cache/go-build"
GOENV="/home/xiaji01/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/xiaji01/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/xiaji01/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/xiaji01/src/go.gc"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/xiaji01/src/go.gc/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/xiaji01/src/go.gc/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build944660995=/tmp/go-build -gno-record-gcc-switches"

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
-->

live-lock problems are spotted on arm64 in a couple of micro benchmarks from text/tabwriter (there should be more similar cases). 
If a goroutine is running 'nosplit' functions which are time consuming, like
      bulkBarrierPreWriteSrcOnly
      memmove
called in growslice when the slice is a large one, and its background worker tries to suspend it to do stack scanning, the G under preemption may hardly make progress, especially on arm64 machines which seem to experience poor signal handling performance.

pprof data of BenchmarkTable/1x100000/new on an arm64 machine:
```
(pprof) top
Showing nodes accounting for 12.46s, 95.70% of 13.02s total
Dropped 118 nodes (cum <= 0.07s)
Showing top 10 nodes out of 40
      flat  flat%   sum%        cum   cum%
     5.41s 41.55% 41.55%      5.41s 41.55%  runtime.futex
     3.22s 24.73% 66.28%      3.59s 27.57%  runtime.nanotime (inline)
     0.99s  7.60% 73.89%      0.99s  7.60%  runtime.tgkill
     0.84s  6.45% 80.34%      0.84s  6.45%  runtime.osyield
     0.77s  5.91% 86.25%      0.77s  5.91%  runtime.epollwait
     0.37s  2.84% 89.09%      0.37s  2.84%  runtime.nanotime1
     0.33s  2.53% 91.63%      6.38s 49.00%  runtime.suspendG
     0.30s  2.30% 93.93%      0.30s  2.30%  runtime.getpid
     0.14s  1.08% 95.01%      0.14s  1.08%  runtime.procyield
     0.09s  0.69% 95.70%      1.38s 10.60%  runtime.preemptM (inline)
``` 

I'm thinking to introduce a flag into G to indicate whether it's running a time-consuming nosplit function and let suspendG yield to avoid the live-lock, the flag is set manually for known functions, I tried it for bulkBarrierPreWriteSrcOnly and memmove in runtime.growslice and runtime.makeslicecopy when the slice is longer than a threshold (select 4K for now), the text/tabwriter package witnessed significant improvement on arm64:

arm64-1 perf:
```
name                        old time/op    new time/op    delta
Table/1x10/new-224            6.59µs ± 8%    6.67µs ± 7%     ~     (p=0.841 n=5+5)
Table/1x10/reuse-224          1.94µs ± 1%    1.94µs ± 1%     ~     (p=0.802 n=5+5)
Table/1x1000/new-224           494µs ±13%     381µs ± 3%  -22.85%  (p=0.008 n=5+5)
Table/1x1000/reuse-224         185µs ± 0%     185µs ± 0%   -0.37%  (p=0.016 n=5+4)
Table/1x100000/new-224         2.73s ±73%     0.04s ± 2%  -98.46%  (p=0.008 n=5+5)
Table/1x100000/reuse-224       4.25s ±76%     0.02s ± 1%  -99.55%  (p=0.008 n=5+5)
Table/10x10/new-224           20.7µs ± 6%    20.1µs ± 5%     ~     (p=0.421 n=5+5)
Table/10x10/reuse-224         8.89µs ± 0%    8.91µs ± 0%   +0.21%  (p=0.032 n=5+5)
Table/10x1000/new-224         1.70ms ± 7%    1.59ms ± 6%     ~     (p=0.151 n=5+5)
Table/10x1000/reuse-224        908µs ± 0%     902µs ± 0%   -0.59%  (p=0.032 n=5+5)
Table/10x100000/new-224        2.34s ±62%     0.14s ± 2%  -94.21%  (p=0.008 n=5+5)
Table/10x100000/reuse-224      911ms ±72%     103ms ± 2%  -88.66%  (p=0.008 n=5+5)
Table/100x10/new-224           143µs ±15%     138µs ± 5%     ~     (p=0.841 n=5+5)
Table/100x10/reuse-224        78.1µs ± 0%    78.1µs ± 0%     ~     (p=0.841 n=5+5)
Table/100x1000/new-224        13.1ms ± 6%    11.9ms ± 5%   -9.16%  (p=0.008 n=5+5)
Table/100x1000/reuse-224      8.13ms ± 0%    8.15ms ± 1%     ~     (p=1.000 n=5+5)
Table/100x100000/new-224       1.29s ±17%     1.31s ±10%     ~     (p=0.310 n=5+5)
Table/100x100000/reuse-224     1.23s ± 2%     1.26s ± 5%     ~     (p=0.286 n=4+5)
```
arm64-2 perf:
```
name                       old time/op    new time/op    delta
Table/1x10/new-64            4.33µs ± 1%    5.40µs ± 4%  +24.92%  (p=0.004 n=6+5)
Table/1x10/reuse-64          1.77µs ± 0%    1.77µs ± 0%   -0.36%  (p=0.024 n=6+6)
Table/1x1000/new-64           318µs ±10%     379µs ±10%  +19.04%  (p=0.008 n=5+5)
Table/1x1000/reuse-64         172µs ± 0%     171µs ± 0%     ~     (p=0.537 n=6+5)
Table/1x100000/new-64         4.26s ±88%     0.05s ± 4%  -98.77%  (p=0.002 n=6+6)
Table/1x100000/reuse-64       4.79s ±73%     0.02s ± 2%  -99.61%  (p=0.004 n=6+5)
Table/10x10/new-64           14.6µs ± 3%    16.4µs ± 9%  +12.51%  (p=0.004 n=5+6)
Table/10x10/reuse-64         8.76µs ± 0%    8.79µs ± 0%     ~     (p=0.329 n=5+6)
Table/10x1000/new-64         1.18ms ± 3%    1.32ms ± 4%  +12.12%  (p=0.002 n=6+6)
Table/10x1000/reuse-64        890µs ± 0%     897µs ± 0%   +0.84%  (p=0.002 n=6+6)
Table/10x100000/new-64        1.43s ±36%     0.16s ± 3%  -88.56%  (p=0.004 n=5+6)
Table/10x100000/reuse-64      375ms ±55%     138ms ±11%  -63.31%  (p=0.004 n=5+6)
Table/100x10/new-64           103µs ± 0%     105µs ± 4%     ~     (p=0.429 n=5+6)
Table/100x10/reuse-64        79.6µs ± 0%    80.0µs ± 1%     ~     (p=0.329 n=5+6)
Table/100x1000/new-64        10.2ms ± 3%    10.5ms ± 1%     ~     (p=0.052 n=6+5)
Table/100x1000/reuse-64      8.75ms ± 0%    9.23ms ± 6%     ~     (p=0.126 n=5+6)
Table/100x100000/new-64       1.42s ± 6%     1.46s ±11%     ~     (p=0.394 n=6+6)
Table/100x100000/reuse-64     1.40s ± 0%     1.45s ± 6%     ~     (p=0.690 n=5+5)
Pyramid/10-64                15.3µs ±45%    14.5µs ±20%     ~     (p=0.662 n=6+5)
Pyramid/100-64               1.16ms ±13%    0.88ms ±11%  -23.90%  (p=0.002 n=6+6)
Pyramid/1000-64              71.6ms ± 8%    77.8ms ±12%     ~     (p=0.056 n=5+5)
Ragged/10-64                 13.2µs ±17%    14.0µs ± 2%     ~     (p=0.792 n=6+5)
Ragged/100-64                 104µs ± 3%     123µs ± 2%  +18.21%  (p=0.008 n=5+5)
Ragged/1000-64               1.12ms ±12%    1.35ms ± 9%  +20.55%  (p=0.002 n=6+6)
Code-64                      3.51µs ± 1%    3.80µs ± 3%   +8.46%  (p=0.004 n=5+6)
```

x86 perf:
```
name                       old time/op    new time/op    delta
Table/1x10/new-32            5.36µs ± 2%    5.58µs ± 3%   +4.03%  (p=0.004 n=6+6)
Table/1x10/reuse-32          1.50µs ± 8%    1.48µs ± 0%     ~     (p=0.433 n=6+5)
Table/1x1000/new-32           318µs ± 2%     355µs ± 2%  +11.43%  (p=0.004 n=5+6)
Table/1x1000/reuse-32         127µs ± 0%     134µs ± 1%   +5.36%  (p=0.004 n=6+5)
Table/1x100000/new-32        49.9ms ± 3%    48.1ms ± 3%   -3.57%  (p=0.026 n=6+6)
Table/1x100000/reuse-32      13.8ms ± 1%    15.2ms ± 1%   +9.87%  (p=0.002 n=6+6)
Table/10x10/new-32           17.4µs ± 3%    17.6µs ± 2%     ~     (p=0.310 n=6+6)
Table/10x10/reuse-32         7.20µs ± 1%    7.21µs ± 0%     ~     (p=0.429 n=5+6)
Table/10x1000/new-32         1.38ms ± 1%    1.43ms ± 2%   +4.16%  (p=0.002 n=6+6)
Table/10x1000/reuse-32        687µs ± 1%     693µs ± 0%     ~     (p=0.052 n=6+5)
Table/10x100000/new-32        131ms ± 2%     133ms ± 5%     ~     (p=0.699 n=6+6)
Table/10x100000/reuse-32     89.2ms ± 2%    90.3ms ± 2%     ~     (p=0.177 n=5+6)
Table/100x10/new-32           122µs ± 1%     122µs ± 1%     ~     (p=0.632 n=6+5)
Table/100x10/reuse-32        62.3µs ± 0%    62.4µs ± 0%     ~     (p=0.429 n=5+6)
Table/100x1000/new-32        12.2ms ± 4%    12.0ms ± 3%     ~     (p=0.180 n=6+6)
Table/100x1000/reuse-32      6.29ms ± 0%    6.32ms ± 0%   +0.54%  (p=0.015 n=6+6)
Table/100x100000/new-32       1.01s ± 2%     1.00s ± 4%     ~     (p=0.429 n=6+5)
Table/100x100000/reuse-32     972ms ±10%     962ms ±18%     ~     (p=1.000 n=6+6)
Pyramid/10-32                14.3µs ± 4%    14.8µs ± 6%     ~     (p=0.240 n=6+6)
Pyramid/100-32                833µs ± 2%     841µs ± 2%     ~     (p=0.394 n=6+6)
Pyramid/1000-32              53.5ms ± 1%    56.2ms ± 3%   +4.98%  (p=0.004 n=5+6)
Ragged/10-32                 15.1µs ± 0%    14.8µs ± 1%   -1.75%  (p=0.008 n=5+5)
Ragged/100-32                 127µs ± 4%     130µs ± 2%     ~     (p=0.180 n=6+6)
Ragged/1000-32               1.27ms ± 5%    1.33ms ± 3%   +3.96%  (p=0.026 n=6+6)
Code-32                      3.75µs ± 1%    3.81µs ± 2%   +1.67%  (p=0.048 n=5+6)
```

The potential overhead is acquiring the current G in the two slice functions, which seem to be tiny.
I'm working on benchmarking more packages and evaluate its impact to x86 platform.

Any comment is highly appreciated.

### What did you expect to see?
Better performance.


### What did you see instead?