Repository: kubernetes
Issue: cpu allocation for static policy should not only limited to Guaranteed Pod.
Labels: sig/node,kind/feature,triage/accepted

### What happened?

i am running a pod which has two containers, the first is the main container(called container1) which has gpu resources, the second is a sidecar container(called container2) with no gpu which just do normal workload, besides both of the  request/limits are equal.

To achieve optimal performance, I have enabled both the CPU Manager and Topology Manager with options below
```
topologyManagerPolicy: restricted
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  align-by-socket: "true"
  distribute-cpus-across-numa: "true"
```

when i applied the yaml and the pod become running, both containers are bound to CPU cores, result is below
```
{
  "policyName": "static",
  "defaultCpuSet": "0-7,14-15,22-23,26-63",
  "entries": {
    "2825b50a-bc6f-44c5-b04c-e6b8b68d8dfe": {
      "container1": "8-13,16-21",
      "container2": "24-25"
    }
  },
```
I don't expect the second container  also bind to CPUs, as it occupies CPU resources on a new NUMA node, resulting in fewer CPU resources corresponding to the unused GPU card on that NUMA. 

if i make the second container request and limits different, the qosClass of the pod becomes burstable,  both of the container will use shared pool resources.
```
{
  "policyName": "static",
  "defaultCpuSet": "0-63",
  "checksum": 1058907510
}
```

### What did you expect to happen?

the main container(limits==requests) bind cpus but the sidecar container(limits != requests) should not,  but use the shared pool resources,
we should not rely on 'guarantee pod' as a condition but rather on 'containers' since resource allocation is done at the granularity of containers.

### How can we reproduce it (as minimally and precisely as possible)?

Below is the yaml file, you just change the resource of the second container with two conditions, request and limit resources are equal and not equal, then check cpu allocation result.
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: guarantee
spec:
  replicas: 1
  selector:
    matchLabels:
      gpu: baichuan
  template:
    metadata:
      labels:
        gpu: baichuan
    spec:
      containers:
      - command:
        - sh
        - -c
        - while true;do sleep 1s;done
        name: container1
        image: ubuntu:20.04
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: "12"
            memory: 5Gi
            nvidia.com/gpu: "2"
          requests:
            cpu: "12"
            memory: 5Gi
            nvidia.com/gpu: "2"
      - command:
        name: container2
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: "2"
            memory: 2Gi
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>