Repository: go
Issue: cmd/go: add fips140 module selection mechanism
Labels: Proposal,Proposal-Accepted

> [!IMPORTANT]
> Nov 20, 2024: The latest version of the proposal is [here](https://github.com/golang/go/issues/70200#issuecomment-2490017956).

_Update_, Nov 11 2024: An updated version is at https://github.com/golang/go/issues/70200#issuecomment-2468562595. The change is to stop presenting the choice as anything like a module, and instead to use a GOFIPS140 environment variable, analogous to GOOS, GOARCH, CGO\_ENABLED, and so on.

- - - 

We are working toward getting Go crypto FIPS validated, to be able to remove BoringCrypto. The relevant code is going to be `crypto/internal/fips/...`. People are going to need to be able to select between a few different crypto/internal/fips trees when using any given Go distribution. We propose to treat the crypto/internal/fips tree as its own pseudo-module, with a new go build flag, `-fips140`, to choose the version. 

In general, any build needs to choose between the actual latest source in $GOROOT/src/crypto/internal/fips and an earlier source snapshot stored in module zip form in $GOROOT/lib/fips140. (Earlier snapshots will have been through more validation processes and may be necessary for certain government-related use.)

The crypto/internal/fips module will use these version numbers:

 - the version in the actual Go 1.X.Y source tree will be called v1.X.Y.
 - snapshots taken ahead of Go 1.X.Y will be called v1.X.Y-fips.N for N starting at 1 and incrementing.

Note that the versions order correctly: v1.24.0-fips.1 < v1.24.0. The idea is that toward the end of the 1.24.0 cycle, a snapshot would be taken and sent to a lab for validation, so that by the time 1.24.0 is released, "v1.24.0-fips.1" will be an "in-process" FIPS module, meaning a lab has validated it but NIST has not yet signed off. (When NIST signs off, it becomes a "certified" module.)

The `-fips140` flag can take one of the following possible values:

 - `-fips140=off` (the default) means to use the source in `$GOROOT/src/crypto/internal/fips` and set the default GODEBUG for compiled binaries to [`fips140=off`](https://go.dev/issue/70123). All other flag settings set the default GODEBUG to `fips140=on`.
 - `-fips140=latest` means to use the source in `$GOROOT/src/crypto/internal/fips` (and default binaries to `fips140=on`, which I will stop saying).
 - `-fips140=v1.X.Y-fips.N` means to use the snapshot in `$GOROOT/lib/fips140/v1.X.Y-fips.N.zip`.
 - `-fips140=inprocess` means to use the version listed in `$GOROOT/lib/fips140/inprocess.txt` (it will be `v1.X.Y-fips.N` for some X, Y, N).
 - `-fips140=certified` means to use the version listed in `$GOROOT/lib/fips140/certified.txt` (it too will be `v1.X.Y-fips.N` for some X, Y, N).

Note that the definitions of `inprocess` and `certified` are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of -fips140=certified in an older Go release.)

crypto/internal/fips will be like a module in the sense that:

 - When you run go list, packages in crypto/internal/fips/... will be reported as coming from a module named crypto/internal/fips.
 - crypto/internal/fips has a set of versions, module zips, and checksums.
 - Earlier snapshots maintained in $GOROOT/lib/fips140 will use the module zip format.
 - When using an earlier snapshot, the go command unpacks the module zip into the module cache and then uses that source in the build.
 - Programs that use go list or x/tools/go/packages will see the module information for crypto/internal/fips and will see source coming from the module cache when using earlier snapshots.
 - crypto/internal/fips will be listed in the `go version -m` output with its own version, in all builds using those packages. This will enable tools like govulncheck to report vulnerabilities accurately.

crypto/internal/fips will be special, or unlike a module, in these ways:

 - The module zips will only ever be loaded from $GOROOT/lib/fips140, never from the module proxy.
 - The checksums for the zips will only ever be loaded from $GOROOT/lib/fips140/fips140.sum, not from the checksum database.
 - crypto/internal/fips will not be listed in go.mod or participate in module version selection; the `-fips140` build flag is the only way to choose the version.
 - The name "crypto/internal/fips" does not start with a domain name element, which violates golang.org/x/mod/module.CheckPath ("missing dot in first path element"). 
 - In -mod=vendor builds, fips code will come from the fips places ($GOROOT or else an unpacked zip), never from the vendor directory.

The specialness seems unavoidable. We could go all the way and make fips completely different and not look anything like a module, but it is necessary to be able to say what version of fips a given program uses, and modules are the vocabulary we use for talking about versions. So it makes sense to reuse that vocabulary. In general the special cases seem like they will not cause much trouble. In particular, programs that use x/tools/go/packages or go list to find information about packages in a build will see a Dir set to a directory in the module cache when the alternate fips locations are being used. Since they already handle seeing other packages in the module cache, they should keep working without any changes.

The only special case that might cause trouble is if clients of module.CheckPath try to check "crypto/internal/fips". If that turns out to be a problem, we can update module.CheckPath to allow crypto/internal/fips as a special case. In my prototype of this functionality, I skipped the calls to module.CheckPath in the few places where it mattered, avoiding any changes to x/mod.

I have a prototype of this code working. It needs some cleanup and tests, but the necessary changes are quite small:

```
 src/cmd/go/internal/cfg/cfg.go               |   1 +
 src/cmd/go/internal/load/pkg.go              |   5 +-
 src/cmd/go/internal/modfetch/cache.go        |  11 +-
 src/cmd/go/internal/modfetch/codehost/git.go |   2 +
 src/cmd/go/internal/modfetch/coderepo.go     |  14 +-
 src/cmd/go/internal/modfetch/fetch.go        |   2 +-
 src/cmd/go/internal/modfetch/fips.go         | 239 +++++++++++++++++++++++++++
 src/cmd/go/internal/modfetch/repo.go         |  11 +-
 src/cmd/go/internal/modfetch/sumdb.go        |   4 +
 src/cmd/go/internal/modload/build.go         |  10 +-
 src/cmd/go/internal/modload/import.go        |  10 +-
 src/cmd/go/internal/modload/init.go          |   8 +
 src/cmd/go/internal/modload/load.go          |   2 +-
 src/cmd/go/internal/modload/query.go         |   2 +-
 src/cmd/go/internal/work/build.go            |   7 +-
 15 files changed, 306 insertions(+), 22 deletions(-)
```

In summary, the proposal is to establish the crypto/internal/fips pseudo-module with the version scheme and properties described above, and to add the `-fips140` go build flag. (Note that “build flags” apply not just to `go build` but to most other go commands, including `go install`, `go test`, and `go list`.)