Repository: go
Issue: runtime: post-mortem analysis of Go 1.13 runtime/mgcsweep.go fatal error ("error: non in-use span in unswept list")
Labels: Documentation,Thinking,NeedsInvestigation


### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.13.3 linux/amd64


</pre>

### Does this issue reproduce with the latest release?

No


### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB="no"
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build603283197=/tmp/go-build -gno-record-gcc-switches"


</pre></details>

### What did you do?

We have a machine running Docker containers in production, and Docker Engine reported an error that the main process had exited without any operation being performed. The error message is as follows:

![error](https://user-images.githubusercontent.com/8202490/221112533-2bc77bab-8cb6-4a4a-ae5f-86dc392413b2.png)



From the error message, it seems that it may be related to the memory cleanup operation of the runtime. We looked at the relevant code as follows:

![code](https://user-images.githubusercontent.com/8202490/221111222-17dc13d6-7f0e-41bb-aabe-4e9cd147dcc7.png)


After the error occurred, the entire dockerd service crashed and dumped the core. We made some guesses based on the error log, but we have not yet come to a conclusion.

To summarize the reproduction conditions:

    When the span is taken out of heap.sweepSpan, its status is mSpanFree.
    The sweepgen of the span equals heap.sweepgen - 1, indicating that the span is being swept.

We have some guesses about this:

    It is possible that when sweepone is executed concurrently, different threads take the same span from heap.sweepSpan.
    It is possible that a span failed to be swept and was later inserted back into heap.sweepSpan with an incorrect state.
    It is possible that the span returned by alloc_m was modified by another thread after being added to heap.sweepgen.

However, all of these guesses were ruled out later when we reviewed the code, and because we are unable to reproduce this bug, we have made no progress so far. We would be very grateful if anyone could help us with the analysis. If more information is needed, we can provide it at any time.

### What did you expect to see?

No GC panic error in dockerd.


### What did you see instead?

A GC panic error in dockerd, as described above. If possible, please help us analyze this issue. The Go version used is quite old. If this issue has been resolved in a higher version, we would like to know how it was resolved and whether any relevant code or patches can be provided.

[full-stack.log](https://github.com/golang/go/files/10821466/full-stack.log)