Repository: kubernetes
Issue: [PSA] Potential pitfalls with dependency bumps 
Labels: sig/architecture,area/code-organization,triage/accepted

_Note: this issue is only meant to document the situation that currently exists because discussions in slack are ephemeral and hard to discover._

## TL;DR

https://github.com/golang/go/issues/65573 is a proposal accepted and targeting Go 1.23 that will address the issue being described here! 

In the meantime, please exercise extra caution in the following scenarios:
- Backporting dependency changes - please check what the `go` directives are in these dependencies before we backport them.
- Upgrading `go` directives in community owned, single branch dependencies such as k/utils, k/gengo, k/kube-openapi etc. 

## Context

go1.21 introduced 2 immensely helpful features: 

1. Default `GODEBUG` settings are now taken based on the `go` directive in `go.mod`s (ref: https://go.dev/doc/godebug). This is particularly helpful to maintain compatibility behaviour when Go makes breaking(-but-compatible) changes as part of its minor releases. This is also especially important since Kubernetes now bumps minor versions of Go on its release branches: https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/3744-stay-on-supported-go-versions 
2. The `go` directive in `go.mod` now specifies the _minimum_ go version required to build your own code. We couldn't do this before, and that meant older go toolchains were allowed to compile code that was meant for newer go versions, even though it would lead to breaking changes: https://go.dev/blog/toolchain 

## The Problem

While (1) is highly desirable, (2) interferes with its efficacy of being able to default `GODEBUG` settings, illustrated below.

The currently supported release branches of Kubernetes have their versions of Go bumped to go1.21.x (same as `master`). What this means is code is tested, built and released using go1.21.x:
- [Ref for K8s 1.27](https://github.com/kubernetes/kubernetes/blob/b9e2ad67ad146db566be5a6db140d47e52c8adb2/build/dependencies.yaml#L97-L98)
- [Ref for K8s 1.28](https://github.com/kubernetes/kubernetes/blob/c8dcb00be9961ec36d141d2e4103f85f92bcf291/build/dependencies.yaml#L116-L117)
- [Ref for K8s 1.29](https://github.com/kubernetes/kubernetes/blob/4b8e819355d791d96b7e9d9efe4cbafae2311c88/build/dependencies.yaml#L120-L121)

However, the `go` directive in `go.mod` is as follows for the currently supported Kubernetes versions:
- [K8s 1.27](https://github.com/kubernetes/kubernetes/blob/b9e2ad67ad146db566be5a6db140d47e52c8adb2/go.mod#L9): `go 1.20`
- [K8s 1.28](https://github.com/kubernetes/kubernetes/blob/c8dcb00be9961ec36d141d2e4103f85f92bcf291/go.mod#L9): `go 1.20`
- [K8s 1.29](https://github.com/kubernetes/kubernetes/blob/4b8e819355d791d96b7e9d9efe4cbafae2311c88/go.mod#L9): `go 1.21`

Essentially, we leave the `go` directive in the `go.mod`s to indicate the Go version with which a Kubernetes minor was first shipped. Because of go1.21's `GODEBUG` defaulting, we get to preserve that behaviour for our users even though we now continue to _build_ with higher versions of Go (to stay well within the support window of Go). 

However, [many of the fixes](https://github.com/kubernetes/release/issues/2815#issuecomment-1373891562) needed to bump minor versions of Go on release branches involves backporting dependency bumps, for example:

If dependency `D` has a `go` directive in its `go.mod` of `go 1.21`, was bumped at `master` to fix a bug and we need to backport this to our release branches that has a `go` directive of `go 1.20`, the bump will essentially force the `go` directive in our release branches to `go 1.21` because of (2) - the `go` directive required is going to be the `max(all dep. go directives)`. This is not desirable since we now also loose the compatibility behaviour of go1.20 that our users would expect.

## What Do We Do?

https://github.com/golang/go/issues/65573 is a proposal that is being worked on in the Go community to address this dichotomy between (1) and (2) and will hopefully land in go1.23.

In the meantime, we need to exercise extra caution in the following scenarios:
- Backporting dependency changes - please check what the `go` directives are in these dependencies before we backport them.
- Upgrading `go` directives in community owned, single branch dependencies such as k/utils, k/gengo, k/kube-openapi etc. If we bump the `go` directive here and later need to absorb a fix in one of these dependencies into k/k via backporting, we will run into the same issue.
  - I bit the bullet on this one already: https://github.com/kubernetes/utils/pull/304, thanks to @liggitt for catching that error!

There is also work happening to safeguard accidental bumps:
- https://github.com/kubernetes/kube-openapi/issues/463
- https://github.com/kubernetes/gengo/issues/264
- https://github.com/kubernetes/utils/issues/305

/sig architecture
/area code-organization
/triage accepted
/cc @kubernetes/dep-approvers