Repository: kubernetes
Issue: /sys/fs/cgroup/cpuset.cpus Not Found In Privileged Pod from Kubernetes 1.24 onwards with Cgroups V2
Labels: kind/bug,sig/node,needs-triage

### What happened?

In Kubernetes version 1.24 and onwards (Testing on K8s 1.23 does not show the error), I am seeing an issue whereby the file `/sys/fs/cgroup/cpuset.cpus` (used by my application to read what CPUs have been assigned to the container by the K8s CPU Manager) disappears when the container is privileged. With a non-privileged container, the file `/sys/fs/cgroup/cpuset.cpus` is accessible. This error is occuring with Cgroups V2 (I have not tested with Cgroups V1). It looks to me that in newer versions of Kubernetes (1.24 and onwards) the host directory  `/sys/fs/cgroup/` is being mounted into privileged containers thus overwriting the containers own Cgroup fs. If this is expected, it would be good to know what way an application in a privileged container should retrieve it's assigned CPUs now that it no longer has access to it's own individual `/sys/fs/cgroup/` directory.

### What did you expect to happen?

In a K8s privileged container, I would expect it should have access to its own individual `/sys/fs/cgroup/cpuset.cpus`

### How can we reproduce it (as minimally and precisely as possible)?

On the following setup, perform the proceeding steps:
**Kubernetes Version:** 1.26.2 (Note this can be replicated on 1.24 and onwards)
**Cgroup driver:** systemd
**Containerd Version:** 1.6.20
**OS Version:** Ubuntu 22.04.2
**Kernel:** 5.15.0-76-generic

Start off with a very simple pod spec (`test.yaml`):
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-0
spec:
  nodeSelector:
    kubernetes.io/hostname: "node1"
  containers:
    - name: pod-0
      image: ubuntu:22.04
      command: [ "/bin/sh" , "-c", "tail -f /dev/null" ]
      resources:
        requests:
          cpu: 4
          memory: 2Gi
        limits:
          cpu: 4
          memory: 2Gi
```
Start the pod and exec into it:
```bash
kubectl apply -f test.yaml 
kubectl exec -it test-0 -- /bin/bash
```
Once in the container, run the following:
```bash
cat /sys/fs/cgroup/cpuset.cpus
```
The output is similar to: `44,46,100,102` (i.e the cpus allocated to my pod are printed as expected using cgroupv2 filesystem)

Next I remove the pod (`kubectl delete pod test-0`) and create a new pod with the following spec (Note the new privileged securityContext):
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-0
spec:
  nodeSelector:
    kubernetes.io/hostname: "node1"
  containers:
    - name: pod-0
      image: ubuntu:22.04
      securityContext:
        allowPrivilegeEscalation: true
        privileged: true
      command: [ "/bin/sh" , "-c", "tail -f /dev/null" ]
      resources:
        requests:
          cpu: 4
          memory: 2Gi
        limits:
          cpu: 4
          memory: 2Gi
```
Same as before, I start the pod and exec into it:
```bash
kubectl apply -f test.yaml 
kubectl exec -it test-0 -- /bin/bash
```
Again, once in the container, I run the following:
```bash
cat /sys/fs/cgroup/cpuset.cpus
```
But this time I get the following error:
```
cat: /sys/fs/cgroup/cpuset.cpus: No such file or directory
```
Thus this shows that adding privileged mode to the container removes the container cgroupv2 entry `/sys/fs/cgroup/cpuset.cpus`


### Anything else we need to know?

_No response_

### Kubernetes version

<details>

Kubernetes Version: 1.26.2

</details>


### Cloud provider

<details>
Bare Metal
</details>


### OS version

<details>

**OS Version:** Ubuntu 22.04.2
**Kernel:** 5.15.0-76-generic

</details>


### Install tools

<details>
kubeadm
</details>


### Container runtime (CRI) and version (if applicable)

<details>
**CRI**: Containerd
**Version:** 1.6.20
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>