Repository: go
Issue: proposal: testing: store test artifacts
Labels: Proposal,LibraryProposal

This is an offshoot of #43936.

Some tests produce output files which the user may want to examine. For example, a test might produce files which are compared to some reference. If the comparison fails, the user will want to examine the generated files. Or a test might produce a packet capture or similar trace which can be used to debug failures.

We call these output files "test artifacts".

This is a proposal to add support for storing test artifacts to the testing package.

We add a new method to testing.TB:

```
package testing

// OutputDir returns a directory for the test to store output files in.
// When the -outputdir flag is provided, this directory will be located
// under that directory. Otherwise, OutputDir returns a temporary directory
// which is removed after the test completes.
//
// Each test or subtest has a unique artifact directory.
// Repeated calls to OutputDir in the same test or subtest return the same directory.
// Subtest outputs are not located under the parent test's output directory.
func (t *testing.T) OutputDir() string
```

The -outputdir flag already exists, and is currently used to specify a location to put output files from profiling. We're adding an additional meaning to it here: It's now where all your saved test outputs go.

When -outputdir is specified, the first call to OutputDir in a test or subtest will emit a line to the test output consisting of "=== ARTIFACTS ", the test name, and the test artifact directory, separated by spaces:

```
=== ARTIFACTS TestName/subtest_name /path/to/root/artifact/dir/TestName__subtest_name
```

When -json is specified, this will appear as an entry with an Action of "artifacts", the usual Time, Package, and Test keys, and a "Path" key containing the artifact directory:

```
{"Time":"2025-01-15T13:39:27.75235-08:00","Action":"artifacts","Package":"path","Test":"TestName","Path":"/path/to/root/artifact/dir/TestName"}
```

That's the proposal.

A few points on the design:

* I'm reusing the existing -outputdir flag, on the theory that output files from profiling are just another test artifact. If we don't like that reuse, then we could add a new -artifactdir flag and rename TB.OutputDir to TB.ArtifactDir for consistency.
 
* The test output uses the word "ARTIFACTS" because the JSON already has "output" events.

* TB.OutputDir returns a directory, same as TB.TempDir. This seems simpler than asking the testing package to copy files around.

* TB.OutputDir returns a directory even if we aren't saving artifacts so test behavior doesn't change depending on the presence or absence of the -outputdir flag.

In simple interactive use, users can pass -outputdir to store test artifacts when debugging a failing test.

Test frameworks that collect artifacts can arrange to pass -outputdir to the tests they run and collect any artifacts after the fact.

As a concrete use case, within Google our testing infrastructure sets an environment variable to the location of a directory. Tests can write files into this directory, and those files will be stored and associated with the test run. If we implement this proposal, we can arrange for the test infrastructure to also pass this directory as an -outputdir flag, and any test using TB.OutputDir will automatically use the right location.