Repository: kubernetes
Issue: Improve performance of Service account token creation code path at scale
Labels: sig/scalability,kind/feature,sig/auth,needs-triage

### What would you like to be added?


Currently ServiceAccount token creation code path is issuing GET calls directly to etcd [here](https://github.com/kubernetes/kubernetes/blob/6fdacf04117cef54a0babd0945e8ef87d0f9461d/pkg/registry/core/serviceaccount/storage/token.go#L99-L100 to validate if ServiceAccount exists before a token gets created instead of leveraging APIServer cache and then falling back to etcd.

Ask is to leverage APIServer cache for ServiceAccount GET calls during validation phase of Service Account token creation and fall back to etcd if/when cache bails out.

### Why is this needed?

At scale, i.e; at higher pod churn i.e around 600-800 QPS, I was able to see this is becoming a bottleneck for pod-start-up SLOs.

I have seen cases where it took `O(seconds)` for just for kubelet token creation because of the latencies from etcd. 

See below audit log snippet:

example 1  log entry:

```
[annotations.apiserver.latency.k8s.io/apf-queue-wait](http://annotations.apiserver.latency.k8s.io/apf-queue-wait) : 26.072µs
annotations.apiserver.latency.k8s.io/etcd : 3.950256834s

annotations.apiserver.latency.k8s.io/response-write : 1.393µs

annotations.apiserver.latency.k8s.io/serialize-response-object : 5.418656ms

annotations.apiserver.latency.k8s.io/total : 3.996358921s
```
example 2 log entry: 
```
annotations.apiserver.latency.k8s.io/apf-queue-wait  : 14.56416759s

annotations.apiserver.latency.k8s.io/etcd  : 33.853028992s

annotations.apiserver.latency.k8s.io/response-write  : 789ns

annotations.apiserver.latency.k8s.io/serialize-response-object  :95.365µs

annotations.apiserver.latency.k8s.io/total   : 48.457248047s
```


example 2 log entry:

I patched APIServer to add some  more detailed logs:


```
  I0114 04:05:22.398438 11 token.go:84] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna Token creation started for name: default
  2025-01-14T04:05:22.000Z

  I0114 04:05:22.398447 11 token.go:92] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna Namespace: test-d7w9e9-1
  2025-01-14T04:05:22.000Z

  I0114 04:05:22.398451 11 token.go:104] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna Looking up ServiceAccount: default
  2025-01-14T04:05:22.000Z

  I0114 04:05:23.433142 11 token.go:113] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna ServiceAccount lookup completed in 1.034684445s
  2025-01-14T04:05:23.000Z

  I0114 04:05:23.437082 11 token.go:164] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna PodObjectRef Get call in 3.87446ms
  2025-01-14T04:05:23.000Z

  I0114 04:05:23.437153 11 token.go:178] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna Node ObjectRef for Pod Get call from cache in 3.945812ms
  2025-01-14T04:05:23.000Z

  I0114 04:05:23.437163 11 token.go:226] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna BoundObjectRef processed in 3.95784ms
  2025-01-14T04:05:23.000Z

  I0114 04:05:23.437176 11 token.go:255] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna jwt token claims processed 1.038734165s
  2025-01-14T04:05:23.000Z

  I0114 04:05:23.440265 11 token.go:266] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna Token generated in 3.077087ms
  2025-01-14T04:05:23.000Z
........

  I0114 04:05:23.440291 11 token.go:277] [RequestID: de602c51-8531-49e3-9ff3-94373cda648b] Hakuna Token creation completed in 1.04184648s for JTI id 666734ee-f6a0-4bf8-bf56-50eb23c20161
  2025-01-14T04:05:23.000Z
```

You can see that in both log entries just the ServiceAccount lookup from etcd took `O(seconds)`.




While I understand there could be staleness of cache may be in the O(100ms), I'm assuming that shouldn't be a huge concern at the cost of performance gain because of this at scale (I can post numbers later if needed with leveraging cache and w/o leveraging cache at 800 QPS).
Also, my understanding  is token is anyways invalidated when associated object i.e secret/pod/node is deleted, so I don't see a security risk here as much relying on cache for validating service-account existence. Would like to here from community what they think ?

I also see that we do similar thing for ServiceAccountTokenPodNode info for Pods logic already [here](https://github.com/kubernetes/kubernetes/blob/6fdacf04117cef54a0babd0945e8ef87d0f9461d/pkg/registry/core/serviceaccount/storage/token.go#L162)