Repository: kubernetes
Issue: controllers: Check if informers are synced on `/healthz`/`/readyz`?
Labels: kind/bug,sig/scheduling,sig/apps,lifecycle/stale,needs-triage

### What happened?

Right now the kube-apiserver has a readyz check if the informers are synced: 

```
% kubectl get --raw='/readyz/informer-sync'
ok
```

The corresponding source code:
https://github.com/kubernetes/kubernetes/blob/948afe5ca072329a73c8e79ed5938717a5cb3d21/staging/src/k8s.io/apiserver/pkg/server/healthz/healthz.go#L95-L122
https://github.com/kubernetes/kubernetes/blob/948afe5ca072329a73c8e79ed5938717a5cb3d21/staging/src/k8s.io/apiserver/pkg/server/config.go#L897-L901

However, there is no such healthz/readyz check for controllers (kube-controller-manager, kube-scheduler, etc.).

I was wondering whether it would make sense to add such checks if informers are synced. Informers can get out of sync. If we bind this condition to the `/healthz` endpoint and then the `/healthz` endpoint to the liveness probe, then kubelet can restart the Pod if the informer is not synced.

I was wondering while working with informers should the `HasSynced` be checked only on startup or should it be checked periodically?

I am not an expert in the topic. Feel free to comment if the proposal makes sense or not.

kube-controller-manager's `/healthz` endpoint returns status for the various controllers:
```
$ curl -k https://localhost:10257/healthz?verbose
[+]leaderElection ok
[+]serviceaccount-token-controller ok
[+]cronjob-controller ok
[+]certificatesigningrequest-signing-controller ok
[+]node-lifecycle-controller ok
[+]endpointslice-controller ok
[+]daemonset-controller ok
[+]statefulset-controller ok
[+]bootstrap-signer-controller ok
[+]persistentvolumeclaim-protection-controller ok
[+]persistentvolume-protection-controller ok
[+]garbage-collector-controller ok
[+]job-controller ok
[+]deployment-controller ok
[+]node-ipam-controller ok
[+]persistentvolume-attach-detach-controller ok
[+]validatingadmissionpolicy-status-controller ok
[+]endpoints-controller ok
[+]certificatesigningrequest-cleaner-controller ok
[+]persistentvolume-binder-controller ok
[+]ttl-after-finished-controller ok
[+]ephemeral-volume-controller ok
[+]resourcequota-controller ok
[+]namespace-controller ok
[+]replicaset-controller ok
[+]ttl-controller ok
[+]legacy-serviceaccount-token-cleaner-controller ok
[+]endpointslice-mirroring-controller ok
[+]pod-garbage-collector-controller ok
[+]disruption-controller ok
[+]token-cleaner-controller ok
[+]persistentvolume-expander-controller ok
[+]replicationcontroller-controller ok
[+]serviceaccount-controller ok
[+]horizontal-pod-autoscaler-controller ok
[+]certificatesigningrequest-approving-controller ok
[+]clusterrole-aggregation-controller ok
[+]root-ca-certificate-publisher-controller ok
[+]taint-eviction-controller ok
healthz check passed
```

However, I am not sure if there is a meaningful check behind it. 
According to https://github.com/kubernetes/kubernetes/blob/948afe5ca072329a73c8e79ed5938717a5cb3d21/cmd/kube-controller-manager/app/controllermanager.go#L779-L795, `controllerhealthz.NamedPingChecker` is being used as I don't see implementation of `controller.HealthCheckable` in the source code. It always returns `nil` (reports no issues; does not perform a meaningful check).

### What did you expect to happen?

Issues with the underlying informers to be reflected in `/healthz`/`/readyz` endpoints for controller components like kube-controller-manager, kube-proxy, etc.

### How can we reproduce it (as minimally and precisely as possible)?

1. Start kube-controller-manager with low qps and burst settings to reproduce informers out of sync issues.

```
--kube-api-qps=1
--kube-api-burst=1
```

2. Create a Deployment and scale it to 100 replicas

3. Make sure the kube-controller-manager logs are full with client-side throttling errors
```
I1021 13:24:04.441128       1 request.go:700] Waited for 25.992938012s due to client-side throttling, not priority and fairness, request: POST:https://kube-apiserver/api/v1/namespaces/default/pods
I1021 13:24:05.441182       1 request.go:700] Waited for 26.992997013s due to client-side throttling, not priority and fairness, request: POST:https://kube-apiserver/api/v1/namespaces/default/pods
```

and the issue is not reflected in the `/healthz`/`/readyz` endpoint.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Server Version: v1.31.1
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>