Repository: go
Issue: runtime/cgo: immediately handoff P before returning to C host program
Labels: NeedsFix,compiler/runtime

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.19.3 linux/amd64
</pre>

### Does this issue reproduce with the latest release?
Yes

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/yongjie.yyj/.cache/go-build"
GOENV="/home/yongjie.yyj/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/yongjie.yyj/gopath/pkg/mod"
GONOPROXY="*.alipay-inc.com,*.alibaba-inc.com,*.alipay.com"
GONOSUMDB="*.alipay-inc.com,*.alibaba-inc.com,*.alipay.com"
GOOS="linux"
GOPATH="/home/yongjie.yyj/gopath"
GOPRIVATE="*.alipay-inc.com,*.alibaba-inc.com,*.alipay.com"
GOPROXY="https://goproxy.cn"
GOROOT="/home/yongjie.yyj/go1.19.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/yongjie.yyj/go1.19.3/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1750707572=/tmp/go-build -gno-record-gcc-switches"

</pre></details>

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on go.dev/play is best.
-->

Recently we're building our go program as dynamic linking lib(.so) and run it on a C-embedded program using cgo, and we found that there is still room for optimization.

As shown in the following demo, under the condition of limited P's resource, there's some delay between the cgo returns and the background goroutine being scheduled.

```go
// foo.go
package main

import "C"
import (
	"fmt"
	"sync/atomic"
	"time"
)

var ch = make(chan int64, 1)
var t, n int64

func init() {
	go func() { // background goroutine
		for {
			start := <-ch
			now := time.Now().UnixNano()
			atomic.AddInt64(&t, now-start) // from cgo returns to background goroutine being scheduled
			atomic.AddInt64(&n, 1)
		}
	}()
}

//export foo
func foo() {
	ch <- time.Now().UnixNano() 
        // cgo returns
}

//export report
func report() {
	fmt.Println(atomic.LoadInt64(&t) / atomic.LoadInt64(&n), "ns")
}

func main() {}

```

```c
// main.cc
#include <unistd.h>

#include "libfoo.h"

int main() {
	for(int i = 0; i < 10000; i++) {
		usleep(1000); // do somethings...
                foo();   // cgo call
	}
	sleep(1);
	report();
}
```

run
```
> go build -buildmode=c-shared -o libfoo.so foo.go     // build go as dynamic link lib

> gcc main.cc -lfoo -L./ -lpthread -o main -g        // build C host program

> GOMAXPROCS=1 LD_LIBRARY_PATH=./ ./main         // run demo under limited P's resource
```

The above demo indicates that there's some schedule delay between the cgo returns and the background goroutine being scheduled. After going through runtime code, we found that when the cgo returns, `reentersyscall` changes P's status to `_Psyscall` and left it waiting until sysmon retake, which leading to sub-optimized performance.

If we try to handoff p immediately after cgo returns, as shown in the related pr, we can observe much better cgo performance.
```
$ GOMAXPROCS=1 LD_LIBRARY_PATH=./ ./main-before
14214 ns

$ GOMAXPROCS=1 LD_LIBRARY_PATH=./ ./main-after
7163 ns
```

Therefore, this issue and the related pr request changes that the runtime could handoff p immediately before cgo returns to the C host program for better performance. However, how to determine whether it's returning to C host program or it's just a normal syscall (should not handoff p) is still a question. A possible way is to add compiler directive such as `//go:handoffp` on the exported go function? 

### What did you expect to see?
the background goroutine should be scheduled as soon as possible


### What did you see instead?
there is some delay between cgo return and the background goroutine being scheduled, leading to sub-optimized performance.