Repository: go
Issue: runtime: some runtime throws print all goroutine stack dumps by default
Labels: NeedsInvestigation,compiler/runtime

I'm going to be using `go version devel go1.17-d19a53338f Wed Jun 30 02:00:49 2021 +0000 linux/amd64` as an example here, but this also reproduces on 1.16.5.

From https://golang.org/pkg/runtime/, emphasis mine:

> The GOTRACEBACK variable controls the amount of output generated when a Go program fails due to an unrecovered panic or an unexpected runtime condition. **By default, a failure prints a stack trace for the current goroutine**, eliding functions internal to the run-time system, and then exits with exit code 2.

So it seems like, by default, a panic should only print the stack trace for the goroutine that panicked, and nothing else.

However, that doesn't seem correct in some cases: https://play.golang.org/p/zWw2TUS4YzZ

```
$ go run -trimpath f.go
fatal error: concurrent map writes

goroutine 7 [running]:
runtime.throw({0x467980, 0x0})
	runtime/panic.go:1198 +0x71 fp=0xc00003cf78 sp=0xc00003cf48 pc=0x42cb91
runtime.mapassign_fast64(0x45f100, 0xc000072060, 0x2)
	runtime/map_fast64.go:176 +0x2b4 fp=0xc00003cfb0 sp=0xc00003cf78 pc=0x40dcb4
main.main.func3()
	./f.go:12 +0x2e fp=0xc00003cfe0 sp=0xc00003cfb0 pc=0x45814e
runtime.goexit()
	runtime/asm_amd64.s:1581 +0x1 fp=0xc00003cfe8 sp=0xc00003cfe0 pc=0x453dc1
created by main.main
	./f.go:10 +0x85

goroutine 1 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:193 +0x12e
main.main()
	./f.go:21 +0xd1

goroutine 5 [sleep]:
time.Sleep(0xdf8475800)
	runtime/time.go:193 +0x12e
main.main.func1()
	./f.go:6 +0x25
created by main.main
	./f.go:6 +0x26

goroutine 6 [sleep]:
time.Sleep(0xdf8475800)
	runtime/time.go:193 +0x12e
main.main.func2()
	./f.go:7 +0x25
created by main.main
	./f.go:7 +0x34

goroutine 8 [runnable]:
main.main.func4()
	./f.go:17 +0x2e
created by main.main
	./f.go:15 +0xc7
exit status 2
```

As per the docs, I would expect to just see one goroutine stack dump - the "running" one that hit the panic via `runtime.mapassign_fast64`, but I get all of them.

This seems to be on purpose in the code:

https://github.com/golang/go/blob/d19a53338fa6272b4fe9c39d66812a79e1464cd2/src/runtime/panic.go#L1385-L1399

This is problematic for two reasons:

A: It's confusing. This had me puzzled when looking at a Go process crash output along with the runtime docs.

B: It can lead to large amounts of panic output, depending on what kind of panic one runs into. At work, someone was very surprised that a panic dumped ~50MiB of goroutine stacks, while most other panics dumped just one goroutine stack. That actually caused infra issues, because it appears our log ingestion didn't like that sudden dump of output very much, and we never found out before given that most panics don't behave that way.

I see three possible resolutions to this problem:

1) The code is working as intended. We should update the docs to clarify the nuance that the default does print all goroutine stack traces in some cases, and what those cases are.

2) We should amend the implementation to match the docs. Concurrent map writes should result in a panic that only prints one stack trace, i.e. the "running" goroutine from above.

3) Something in between, such as "print the stack traces for all goroutines that are involved in writes to the map in question". I can imagine that the current behavior exists for debugging purposes: if you get concurrent map writes, you might want to know what those two or more goroutines were doing, not just one of them. However, you presumably don't need *all* goroutines for that; just the ones involved in the race.

cc @aclements @randall77 @mknyszek @prattmic as per https://dev.golang.org/owners