Repository: kubernetes
Issue: Monitoring gaps in apiserver extension mechanisms
Labels: sig/scalability,sig/api-machinery,kind/feature,sig/auth,help wanted,sig/instrumentation,triage/accepted

### What would you like to be added?

Following up from this discussion - https://github.com/kubernetes/kubernetes/pull/116420#issuecomment-1500602135

There are different components that fall in the critical path for the k8s API today (apiserver (core), authn/authz webhooks, mutating/validating/conversion webhooks, etcd, extension apiservers - maybe more I'm missing). While some of those do, bunch of them don't seem to have metrics tracking request/error counts and latency metrics. Here's what I found so far (will update as we learn more):

- [Authentication webhook](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/endpoints/filters/authentication.go#L47)
  - [x] request/error counts ([xref](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/endpoints/filters/metrics.go#L53))
  - [x] request latencies ([xref](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/endpoints/filters/metrics.go#L62))
- [Authorization webhook](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/endpoints/filters/authorization.go#L45)
  - [x] request/error counts ([#117211](https://github.com/kubernetes/kubernetes/pull/117211))
  - [x] request latencies ([#117211](https://github.com/kubernetes/kubernetes/pull/117211))
- [Admission webhooks](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go#L186) (mutating/validating)
  - [x] request/error counts ([xref](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/admission/metrics/metrics.go#L201-L219))
  - [x] request latencies ([xref](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/admission/metrics/metrics.go#L175))
- [CRD Conversion webhook](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/webhook_converter.go#L216)
  - [x] request/error counts ([#118292](https://github.com/kubernetes/kubernetes/pull/118292))
  - [x] request latencies ([#118292](https://github.com/kubernetes/kubernetes/pull/118292))
- [Etcd](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go#L139)
  - [x] request/error counts ([#117222](https://github.com/kubernetes/kubernetes/pull/117222))
  - [x] request latencies ([xref](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go#L36))
- [Extension apiserver](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go#L113)
  - [ ] request/error counts
  - [ ] request latencies

Finally, wrt the apiserver itself, we measure request/error counts and these flavors of latency metrics today:
- [x] [e2e latency](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L91) (capturing customer experience)
- [x] [SLI-based latency](https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L120) (measuring cloud-provider QoS) 
- [ ] ~We discussed adding a third flavor of latency metrics that measures only the apiserver "core" latency (not including any external callback/webhook mechanisms) [here](https://github.com/kubernetes/kubernetes/pull/116420#issuecomment-1499652447).~
We don't have consensus that benefits of doing so outweigh the additional metric churn. We can revisit later as TBD

### Why is this needed?

Metrics at component/dependency level allow us to:
- track each component's performance in isolation
- set internal (non-customer-facing) SLOs for teams owning those components
- narrow down the root-cause for API errors/latencies easily

/sig api-machinery
/sig auth
/sig scalability
/kind feature
/help