Repository: kubernetes
Issue: Pod Failed with TopologyAffinityError because of init-container CPU NUMA Topology BUG
Labels: kind/bug,sig/node,priority/important-longterm,triage/accepted

### What happened?

My Node is 250Gi memory and 64 Core CPU, which has two NUMA node, both is 125Gi and 32 Core CPU, topology policy is single-numa-node.
Create 2 pod,  with init-container 1 Core CPU and 1Gi memory, app container 26 Core and 32Gi memory.
The first pod create success.
The second pod will failed with TopologyAffinityError.

Result from `lscpu`:
```
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63
```

In `/var/lib/kubelet/cpu_manager_state` file: 
```
{
  "policyName": "static",
  "defaultCpuSet": "1,3,15-31,47-63",
  "entries": {
    "69aa6124-4e0e-4c66-94ea-feae7df25b68": {  # the second Pod, init-container use NUMA 0
      "init-container": "35"
    },
    "e49ac32c-22a3-4c8c-8be3-721bd1fd9eaa": {   # the first Pod, use NUMA 0
      "centos": "0,2,4-14,32,34,36-46",
      "init-container": "33"
    }
  },
  "checksum": 3748757820
}
```



### What did you expect to happen?

Both two pods can be created successfully.
First pod use NUMA 0;
The sceond pod init-container can use either NUMA 0 or NUMA 1, but app container use NUMA 1;

### How can we reproduce it (as minimally and precisely as possible)?

1. Create one pod, which use CPU, for example 10 of 32(Numa Node) CPU of the node
2. Create another pod, which need init-container 1Core CPU, and app container need most of NUMA Node CPU,  like 25 Core (greater than the left of the first NUMA)
3.  The second pod will failed with TopologyAffinityError.

### Anything else we need to know?

I think the bug is in `generateCPUTopologyHints ` function:  https://github.com/kubernetes/kubernetes/blob/7ea3d0245a63fbbba698f1cb939831fe8143db3e/pkg/kubelet/cm/cpumanager/policy_static.go#L623 
1. when allocate init-container CPU, CPUManager will record as cpuToReuse for pod. https://github.com/kubernetes/kubernetes/blob/7ea3d0245a63fbbba698f1cb939831fe8143db3e/pkg/kubelet/cm/cpumanager/policy_static.go#L345  
2. before call `generateCPUTopologyHints`, get cpuToReuse of the pod, as reusable https://github.com/kubernetes/kubernetes/blob/7ea3d0245a63fbbba698f1cb939831fe8143db3e/pkg/kubelet/cm/cpumanager/policy_static.go#L587
3. when caculate app container CPU, CPUManager will filter the NUMA Node which is not equal to reusableCPU's NUMA Node
4. Even if NUMA 1 is the only one that fits CPU resource acquirement, it still passed in calculating TopologyHint; NUMA 0 fits with reusableCPU, but CPU left is not enough. 

### Kubernetes version

<details>

```console
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T11:47:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T11:42:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>