Repository: kubernetes
Issue: API Server package does per-request cert verification
Labels: priority/backlog,sig/scalability,kind/feature,sig/auth,lifecycle/frozen

<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:
The API Server's secure serving settings today [only requests the client cert](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/secure_serving.go#L64), and its [x509 Authenticator and Verifier implementations verify the cert signing chain of every request](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/authentication/request/x509/x509.go#L116), even after mTLS has already taken place upon connection handshake.

As an improvement, if the secure serving, Authenticator, and Verifier implementations were changed so that the API Server's TLS config verified the cert upon initial connection (VerifyClientCertIfGiven instead of RequestClientCert), and the auth libraries used the VerifiedChains of the negotiated connection, then all requests streamed through that connection would not be doing cert verification anymore. Any project using the generic API Server package also has to change its TLS config options to VerifyClientCertIfGiven, so it would be a breaking change (for example, the [Kubelet has to change here](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go#L1026)).

To preserve backwards compatibility we could implement a new Authenticator/Verifier implementation that's feature gated in the generic API Server options.

**Why is this needed**:
TL;DR this will save an enormous amount of CPU that's being wasted on extra auth for every request.

Per request auth isn’t necessary as it’s not going to change during the establishment of the connection. If we are worried about the certificate expiring, then we can timebox the length of the connection (as most keepalive TLS connections do) and cause the client to reconnect forcing a re-authorization. If we are worried about long term polling style connections potentially expiring their certs (or being revoked), the best solution is again, timebox and cause them to reconnect. 

We can do all sorts of verification if we really want to on each request (though, I would argue, just like other systems, once you’ve authorized yourself to them, you have that entitlement for whatever period you’re connected) without doing the full EC or RSA math required to validate the signatures of the certificates. That math won’t change.

**Additional Info**:
This mainly bit us because our clusters were using 384bit ECDSA root certs, which does not have a constant time implementation in Go's standard crypto library: https://github.com/golang/go/issues/11499

The effect was that API Servers effectively run out of CPU once scaled to several hundred thousand requests per minute, because of the per-request mTLS auth logic.

Once switched to per-connection auth, the issue went away. Here's a graph showing CPU usage on the API Servers before and after the patch:

<img width="1708" alt="Screen Shot 2020-03-18 at 10 10 04 AM" src="https://user-images.githubusercontent.com/8000471/80413203-f1614b00-8883-11ea-907c-f21d71959307.png">

Even though it seems other Kubernetes users aren't experiencing reliability issues with the extra auth in their clusters because they're using cheaper certs, I'd imagine that many organizations using Kubernetes would want the reduced CPU from this relatively simple change. 

I haven't run scale tests using different certs, but I'd be interested if anyone has also internally switched to only per-connection mTLS and seen the performance improvements!

I'm creating the issue to kick off discussion around if I'm missing something here as to why this is the connection behavior today, and also to propose some solutions. Feedback is super welcome, thanks!