Repository: kubernetes
Issue: [PSA] SHA-1 signature support fully going away in go1.24
Labels: sig/auth,priority/important-longterm,sig/architecture,triage/accepted

_Note: this issue is only meant to document the current situation, provide historical context and elucidate any operational aid that exists for this particular situation._

## What's Happening?

From the go1.23rc1 release notes (https://tip.golang.org/doc/go1.23):

> The x509sha1 GODEBUG setting will be removed in the next Go major release (Go 1.24). This will mean that crypto/x509 will no longer support verifying signatures on certificates that use SHA-1 based signature algorithms.

The decision to stop support for SHA-1 by default was made in go1.18: https://golang.org/doc/go1.18#sha1 

The decision to fully remove support for this in go1.24 is because of the extended Go compatibility work to keep `GODEBUG`s around for a min. of 2 years: https://go.dev/blog/compat 

## Issue In The Context of Kubernetes 

While it is established that SHA-1 isn't safe anymore, there might still be cases where user-provided certificates are signed using SHA-1 based algorithms through private authorities. This can happen in aggregated API Servers or while using webhooks.

If you have relied on SHA-1 based certs in the past, and assuming you are on a version of Kubernetes that uses a version of Go greater than 1.18, the way you would have continued support for this would be to set the env variable `GODEBUG=x509sha1=1` at kube-apiserver startup time.

Once the `x509sha1` GODEBUG goes away in go1.24, your cluster will **not** continue to work as expected. 

Past tracking issues:
- https://github.com/kubernetes/kubernetes/issues/108910
- https://github.com/kubernetes/kubernetes/issues/108940

## Observability Around SHA-1 Certs

As part of absorbing the go1.18 release, we've added in some observability to help detect the use of these certs and also where they are being used.

### Detecting If SHA-1 Certs Are Being Used

The following metrics are exposed by the kube-apiserver that can be monitored (added in https://github.com/kubernetes/kubernetes/pull/109024):
```
apiserver_kube_aggregator_x509_insecure_sha1_total
apiserver_webhooks_x509_insecure_sha1_total
```

You should also be able to query this using a client like kubectl:
```
❯ kubectl get --raw '/metrics' \
 | prom2json \
 | jq '.[] | select(.name=="apiserver_kube_aggregator_x509_insecure_sha1_total")'
```

Furthermore, since go1.21, the Go runtime also exports metrics around usage of individual GODEBUGs and Kubernetes by default exposes all of these metrics, if you'd like to verify the number of times the `crypto/x509` package processed a cert with SHA-1 signatures because `GODEBUG=x509sha1=1` was enabled, you can do the following:
```
❯ kubectl get --raw '/metrics' \
 | prom2json \
 | jq '.[] | select(.name=="go_godebug_non_default_behavior_x509sha1_events_total")'
```

### Detecting Where SHA-1 Certs Are Being Used

If you see non-zero usage of SHA-1 certs, you can also get information about the specific hosts that are serving these certificates via (added in https://github.com/kubernetes/kubernetes/pull/109107):
- kube-apiserver log (log lines containing `invalid-cert.kubernetes.io`)
- kube-apiserver audit log: https://kubernetes.io/docs/reference/labels-annotations-taints/audit-annotations/#insecure-sha1-invalid-cert-kubernetes-io-hostname

## Approximate and Highly Tentative Timelines

Go major versions are typically released every August and February. go1.23 is slated for August of 2024 and by extrapolation, go1.24 should come out in Feb of 2025.

Typically, `HEAD` will start using go1.24 ~immediately once released, and ~3 months post  the release of go1.24, all active release branches will also move to go1.24 (please see [KEP 3744](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/3744-stay-on-supported-go-versions) for more information).

This means that by *May/June of 2025*, **all** supported Kubernetes versions will fully start rejecting SHA-1 based certs, with no possibility of override. Given that timeline, please start rotating those certs in case you rely on them!

/sig architecture auth
/triage accpeted
/priority important-longterm 
/cc @dims @liggitt