Repository: kubernetes
Issue: [umbrella issue]: poor visibility/observability of the kubelet (cpu,memory,device,topology) managers
Labels: sig/node,kind/feature,priority/important-longterm,triage/accepted

### What happened?

The kubelet {cpu, memory, device, topology} manager, collectively known as resource managers, give little to none feedback to their consumers. Once the kubelet is configured with the relevant settings (e.g. cpuManagerPolicy=static or topologyManagerPolicy=restricted, say) and once the workload requests exclusive resources provided by the managers, the implicit contract is that if the pod is admitted, then all the resources are provisioned according to the spec. Similarly, if the kubelet cannot allocate the resources as requested (and as configured) the pod will fail admission with a TopologyAffinityError or similar generic error.

The problem is twofold:
- there's no **explicit** feedback about resource allocation. For example, the system explicitly acknowledges a pod got its image pulled succesfully with a event, yet it does not for resource allocation, even though both are arguably prerequisites to have the pod running, and both could be theoretically inferred from the fact the pod started. The point of this example is to highlight the lack of explicit feedback about allocation.
- on error, there's no way to learn what gone wrong except in most generic term (pod failed with TopologyAffinityError). The only option is to look at kubelet logs, but the kubelet verbosity has to be >= 4 anyway.

### What did you expect to happen?

On success, the system should somehow explcitely acknowledge the resource classes being allocated, for example "CPUs" and that NUMA alignment took place.

On admission failure, the system should give indication of what caused admission failure, like unable to allocate enough memory on the selected NUMA zone, or missing device affine to the selected NUMA zone. But any more information that is presented now is welcome.

In both cases we probably don't want or need to expose the details of the concrete resources, at least not initially.

### How can we reproduce it (as minimally and precisely as possible)?

1. succesfully run a pod requiring a device, or exclusive access to CPUs, or NUMA alignment. Observe lack of feedback.
2. try to run a pod requiring a device or exclusive access to CPUs, or NUMA alignment. Make it fail admission (e.g. ask for more CPUs than existing on the machine's NUMA zone). Observe the error is a generic "TopologyAffinityError" with lack of indication about the resource failed to be allocated

### Anything else we need to know?

The observability, or lack thereof, of the resource management subsystem was also a [recurring theme in the related Production Readiness Reviews](https://kubernetes.slack.com/archives/CPNHUMN74/p1696357853594279), culminating in a [blocking comment](https://kubernetes.slack.com/archives/CPNHUMN74/p1696518756979339?thread_ts=1696357853.594279&cid=CPNHUMN74) for memory manager GA graduation.

### Kubernetes version

<details>
any since 1.18 (cpumanager introduced, the first of resource managers)
</details>


### Cloud provider

<details>
not relevant
</details>


### OS version

<details>
not relevant
</details>


### Install tools

<details>
not relevant
</details>


### Container runtime (CRI) and version (if applicable)

<details>
any
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
not relevant
</details>