Repository: go
Issue: runtime: FreeBSD nanotime performance regression with different CPU setups
Labels: Performance,OS-FreeBSD,NeedsInvestigation,compiler/runtime

### The setup
<pre>
$ go version
go version go1.17.6 freebsd/amd64
$ freebsd-version
12.2-RELEASE
</pre>

1.17.6 is currently the latest release.

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="freebsd"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1526234500=/tmp/go-build -gno-record-gcc-switches"
</pre></details>

### The case
I've found that the performance of the scheduler on FreeBSD highly depends on the CPU setup I have.  While it is OK for a one-CPU VirtualBox instance running on my Mac, it decreases a lot either for more-than-one-CPU VirtualBox or for any-number-of-CPUs DigitalOcean instances (as far as I know DigitalOcean uses KVM as a hypervisor for their instances).

This benchmark can be used as a minimal reproducer
```
$ cat scheduler_test.go
```
<pre>
package whatever

import (
	"testing"
)

func BenchmarkScheduler(b *testing.B) {
	ch := make(chan struct{})

	go func() {
		for range ch {}
	}()

	for i := 0; i < b.N; i++ {
		ch <- struct{}{}
	}
}
</pre>
```
go test -bench=. scheduler_test.go
```

While this benchmark runs on a single-CPU VirtualBox instance it performs quite convenient.
```
$ sysctl hw.ncpu
hw.ncpu: 1

$ go test -bench=. scheduler_test.go
goos: freebsd
goarch: amd64
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkScheduler 	 7658449	       152.8 ns/op
PASS
ok  	command-line-arguments	1.331s
```

However if the benchmark was run on more-than-one-CPU VirtualBox or on a DigitalOcean instance with any number of CPUs the significant decrease in performance emerges.

```
$ sysctl hw.ncpu
hw.ncpu: 2

$ go test -bench=. scheduler_test.go
goos: freebsd
goarch: amd64
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkScheduler-2   	 1108549	      1163 ns/op
PASS
ok  	command-line-arguments	2.213s
```
```
$ sysctl hw.ncpu
hw.ncpu: 1

$ go test -bench=. scheduler_test.go

goos: freebsd
goarch: amd64
cpu: DO-Premium-Intel
BenchmarkScheduler 	  706951	      1949 ns/op
PASS
ok  	command-line-arguments	1.405s
```
Profiling shows that the scheduler code for FreeBSD relies highly on time machinery. And `runtime.nanotime()` performance highly affects the scheduler.
<details><summary>Execution flow for 1CPU VirtualBox</summary>
<img width="226" alt="1CPU VirtualBox" src="https://user-images.githubusercontent.com/2266882/151974371-1d6375c1-dbaa-4594-b895-3bbd21c7e4e8.png">
</details>

<details><summary>Execution flow for 2CPUs VirtualBox</summary>
<img width="308" alt="2CPUs VirtualBox" src="https://user-images.githubusercontent.com/2266882/151974749-53018dc8-23fd-4511-bd71-bc28d39f8315.png">
</details>

<details><summary>Execution flow for 1CPU DigitalOcean</summary>
<img width="241" alt="1CPU DigitalOcean" src="https://user-images.githubusercontent.com/2266882/151975456-f3befefa-7ef2-4b3b-bd25-476cafe46f6b.png">
</details>

I've also made benchmarks for the `time.Now()` function in the same setups. The results of them correlates with scheduler ones.