Repository: kubernetes
Issue: kubelet unbalanced affinity pod in different numa node
Labels: kind/bug,sig/node,triage/accepted

### What happened?

in current kubelet below algorithm, the pods/containers shall be affinity to different numa node unbalanced. 

for example: if there is 6 pods with integer cpu container configured, most of pods will be scheduled into numa0 if numa0 has enough available cpus (4 pods in numa0 and 2 pod in numa1). 

then if turboboost is enabled with OS control or static max performance, the cpu frequency will be different in callload running. for example cpus in numa0 maybe 2500 but cpus in numa1 maybe 2900 since workload is different in two numa nodes. 

in this environment, even the incoming traffic to those pods which are scheduled in numa0 and numa1 is same (round robin), the cpu usage in those pods are different. that will cause issue which traffic load can not be increased to expected values since 4 pods in numa0 is already with high cpu usage (call maybe starting drop)  but 2 pods in numa1 with low cpu usage.

       // Algorithm: topology-aware best-fit
        // 1. Acquire whole NUMA nodes and sockets, if available and the container
        //    requires at least a NUMA node or socket's-worth of CPUs. If NUMA
        //    Nodes map to 1 or more sockets, pull from NUMA nodes first.
        //    Otherwise pull from sockets first.
        acc.numaOrSocketsFirst.takeFullFirstLevel()
        if acc.isSatisfied() {
                return acc.result, nil
        }
        acc.numaOrSocketsFirst.takeFullSecondLevel()
        if acc.isSatisfied() {
                return acc.result, nil
        }

        // 2. Acquire whole cores, if available and the container requires at least
        //    a core's-worth of CPUs.
        acc.takeFullCores()
        if acc.isSatisfied() {
                return acc.result, nil
        }

        // 3. Acquire single threads, preferring to fill partially-allocated cores
        //    on the same sockets as the whole cores we have already taken in this
        //    allocation.
        acc.takeRemainingCPUs()
        if acc.isSatisfied() {
                return acc.result, nil
        }


### What did you expect to happen?

affinity pod/container balanced to different numa instead of filling numa0 first and then numa1

### How can we reproduce it (as minimally and precisely as possible)?

1. turboboost enabled
2. cpuManagePolicy as static 
3. Pod with GQoS and Container with integer cpu request/limit

### Anything else we need to know?

_No response_

### Kubernetes version

test version is 1.27


### Cloud provider

any cloud provider


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>