Repository: go
Issue: runtime: difficult to find memory leak that is a result lots of defers in a hot loop
Labels: NeedsInvestigation,compiler/runtime

<!--
Please answer these questions before submitting your issue. Thanks!
For questions please use one of our forums: https://github.com/golang/go/wiki/Questions
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.15.3 linux/amd64
</pre>

Cross compiled for GOOS=linux GOARCH=arm GOARM=7.

### Does this issue reproduce with the latest release?
yes


### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE="auto"
GOARCH="arm"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY="gitlab.corp.wabtec.com"
GONOSUMDB="gitlab.corp.wabtec.com"
GOOS="linux"
GOPATH="/go"
GOPRIVATE="gitlab.corp.wabtec.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
GOARM="7"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD="/workspaces/rm-edge-hst/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build052064669=/tmp/go-build -gno-record-gcc-switches"
</pre></details>

### What did you do?
I tried to use pprof to debug a memory leak in a long running program that was showing growth in live objects (`MemStats.Mallocs - MemStats.Frees`) over time.
<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
-->



### What did you expect to see?
pprof to reveal the problematic code that was causing the leak.


### What did you see instead?
Setting `sample_index=inuse_objects` and running top showed lots of objects coming from `runtime.systemstack`. pprof was not giving useful info regarding what is actually running on the stack and causing the leak of live objects. `web` and `peek` showed that `runtime.mstart` was calling `runtime.systemstack` suggesting a possible leak of threads.
![image (2)](https://user-images.githubusercontent.com/11697749/97973141-8d34bc00-1dbd-11eb-82e3-3a0d1e8a5595.png)

However this was not actually relevant, I think due to [this line in asm_arm.s](https://golang.org/src/runtime/asm_arm.s?#L338) that states "make it look like mstart called systemstack on g0, to stop traceback".

After lots of manual debugging, the culprit was found to be a bug in my code where a `defer` was being called in a hot and long running loop, rather than being called once before the loop.

e.g.
With bug - http://play.golang.org/p/0AzEwnbplyq
Bug fixed - http://play.golang.org/p/VbTEFgYXmMI

This was causing lots of calls to `runtime.newdefer` which was [appending to the defer pool in a systemstack block](https://golang.org/src/runtime/panic.go?h=systemstack#L396):
```go
systemstack(func() {
    lock(&sched.deferlock)
    for len(pp.deferpool[sc]) < cap(pp.deferpool[sc])/2 && sched.deferpool[sc] != nil {
        d := sched.deferpool[sc]
	sched.deferpool[sc] = d.link
	d.link = nil
	pp.deferpool[sc] = append(pp.deferpool[sc], d)
    }
    unlock(&sched.deferlock)
})
```
I'm not sure what could be done to fix this, but it would have been nice if pprof was able to point me to the offending code more directly.