Repository: go
Issue: testing: annotate output text type
Labels: Proposal,Proposal-Accepted

Currently the `go test -json` provides no way to disambiguate output produced by `(*testing.T).Error{,f}` and `(*testing.T).Fatal{,f}` from other test logging. This fact makes it difficult for CI systems to identify what part of the output is most pertinent for constructing summaries of test failures.

Consider a test that failed, producing a lot of log messages as a result. Reporting the full log of the test can often be distracting and overly specific, resulting in many CI systems (such as LUCI) not constructing test failure summaries for Go tests at all.

Heuristics such as taking the last line logged before a test fails work often, but not often enough to fully solve the problem. Enough tests call `(*testing.T).Error{,f}` and fail later (after more log output) that this heuristic is unreliable. This works even less well when considering tests that crash or panic: that last line is often going to be some random line in the stack trace.

I propose adding a boolean field to "output" actions in the JSON produced by `cmd/test2json` called "Error". This field would be "true" for any "output" actions that were produced via `(*testing.T).Error{,f}` and `(*testing.T).Fatal{,f}`. This field would be also be "true" for the first few lines of a crash (but notably *not* the stack trace itself).

For example:
```
{"Time":"2023-09-19T15:19:12.447816654Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"--- FAIL: TestHeap (0.00s)\n","Error":false}
{"Time":"2023-09-19T15:19:12.449904543Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"panic: ahhh [recovered]\n","Error":true}
{"Time":"2023-09-19T15:19:12.449923813Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"\tpanic: ahhh\n","Error":true}
{"Time":"2023-09-19T15:19:12.449928103Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"\n","Error":false}
{"Time":"2023-09-19T15:19:12.449932593Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"goroutine 6 [running]:\n","Error":false}
{"Time":"2023-09-19T15:19:12.449936283Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"testing.tRunner.func1.2({0x51ad20, 0x577910})\n","Error":false}
{"Time":"2023-09-19T15:19:12.449940423Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"\t/usr/local/google/home/mknyszek/work/go-1/src/testing/testing.go:1545 +0x24a\n","Error":false}
{"Time":"2023-09-19T15:19:12.449953903Z","Action":"output","Package":"golang.org/x/exp/trace","Test":"TestHeap","Output":"testing.tRunner.func1()\n","Error":false}
```

Which lines of a crash get marked would be entirely up to `cmd/test2json`. Because it lives in the standard library, it can reasonably be coupled (with tests) to the crash output that the Go runtime produces.

Testing binaries' `-test.v=test2json` output would need to be adjusted such that`cmd/test2json` could interpret errors correctly. For this, I propose the use of a new marker character (I am not picky as to which) at the beginning of a frame (delineated currently by ^V (0x16)).