Repository: kubernetes
Issue: align by socket not work when pod using multiple gpu card
Labels: kind/bug,sig/node,triage/accepted

### What happened?

I am running an AI application and expect to achieve the highest performance by using the Topology Manager to establish NUMA affinity binding between CPU and GPU.

so i restart kubelet with  such options
```
kubeReserved:
  cpu: "1"
  memory: "2Gi"
topologyManagerPolicy: restricted
topologyManagerPolicyOptions:
  prefer-closest-numa-nodes: "true"
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  align-by-socket: "true"
  distribute-cpus-across-numa: "true"
featureGates:
  CPUManagerPolicyAlphaOptions: true
  TopologyManagerPolicyOptions: true
  TopologyManagerPolicyAlphaOptions: true
```
then I use multiple GPU cards, 
if the number of GPU cards are more than one socket, the allocated cpu and gpu are not aligned by socket.

### What did you expect to happen?

The CPU cores requested by the pod should be from the NUMA node where the GPUs are located.

### How can we reproduce it (as minimally and precisely as possible)?

create pod with more than one nvidia gpu cards, like two gpu cards

1. create pod with 5 nvidia gpu cards
there is one gpu card on every numa and one socket consists of 4 numa node,  so i request 5 cards thus gpus have to cross socket.
```
       resources:
          limits:
            cpu: "40"
            memory: 4Gi
            nvidia.com/gpu: 5
```
3. waiting pod running
```
# kubectl get pod
NAME                                  READY   STATUS    RESTARTS   AGE
guarantee-gpu5-40c-5c665ff85c-f67vr   1/1     Running   0          8m48s
```
4. check gpu and cpu core numa topology
```
{
  "policyName": "static",
  "defaultCpuSet": "0-7,48-63",
  "entries": {
    "8a77ef80-6ff1-4ca5-82a8-f45c2d82ae87": {
      "nginx-gpu5-40c": "8-47"
    }
  },
  "checksum": 2742015222
}

{
  "Data": {
    "PodDeviceEntries": [
      {
        "PodUID": "8a77ef80-6ff1-4ca5-82a8-f45c2d82ae87",
        "ContainerName": "nginx-gpu5-40c",
        "ResourceName": "nvidia.com/gpu",
        "DeviceIDs": {
          "1": [
            "GPU-92d67a0e-9b17-c7fb-45c0-f9232a956872"
          ],
          "4": [
            "GPU-c29f034e-0614-387a-225c-1367a2a67f34"
          ],
          "5": [
            "GPU-7c4bf20f-e22d-6221-223d-c15655a5f703"
          ],
          "6": [
            "GPU-480ed0ed-c229-844f-7206-837c483eaa27"
          ],
          "7": [
            "GPU-02d33038-442c-5562-ef6a-bfb3ca27fb2d"
          ]
        },
        "AllocResp": "CucBChZOVklESUFfVklTSUJMRV9ERVZJQ0VTEswBR1BVLTAyZDMzMDM4LTQ0MmMtNTU2Mi1lZjZhLWJmYjNjYTI3ZmIyZCxHUFUtNDgwZWQwZWQtYzIyOS04NDRmLTcyMDYtODM3YzQ4M2VhYTI3LEdQVS03YzRiZjIwZi1lMjJkLTYyMjEtMjIzZC1jMTU2NTVhNWY3MDMsR1BVLWMyOWYwMzRlLTA2MTQtMzg3YS0yMjVjLTEzNjdhMmE2N2YzNCxHUFUtOTJkNjdhMGUtOWIxNy1jN2ZiLTQ1YzAtZjkyMzJhOTU2ODcy"
      }
    ],
    "RegisteredDevices": {
      "nvidia.com/gpu": [
        "GPU-480ed0ed-c229-844f-7206-837c483eaa27",
        "GPU-7c4bf20f-e22d-6221-223d-c15655a5f703",
        "GPU-c29f034e-0614-387a-225c-1367a2a67f34",
        "GPU-903d67d4-ef71-0285-c620-d8cd7dd4b99a",
        "GPU-e2a16c50-b350-94c2-4acf-735f38745c20",
        "GPU-92d67a0e-9b17-c7fb-45c0-f9232a956872",
        "GPU-ba72128f-59d8-4b2f-7968-c0b0029dd19f",
        "GPU-02d33038-442c-5562-ef6a-bfb3ca27fb2d"
      ]
    }
  },
  "Checksum": 2746309433
}
```

```
# lscpu | grep 'NUMA node'
NUMA node(s):                       8
NUMA node0 CPU(s):                  0-7
NUMA node1 CPU(s):                  8-15
NUMA node2 CPU(s):                  16-23
NUMA node3 CPU(s):                  24-31
NUMA node4 CPU(s):                  32-39
NUMA node5 CPU(s):                  40-47
NUMA node6 CPU(s):                  48-55
NUMA node7 CPU(s):                  56-63

# lscpu | grep -i Socket
Core(s) per socket:              32
Socket(s):                       2

# nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	24-31	3		N/A
GPU1	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS	16-23	2		N/A
GPU2	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	8-15	1		N/A
GPU3	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	0-7	0		N/A
GPU4	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	56-63	7		N/A
GPU5	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	48-55	6		N/A
GPU6	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	40-47	5		N/A
GPU7	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	32-39	4		N/A
```
5. we can see the  allocated gpus  are on numa node1,4,5,6,7 while cpus are on numa node1,2,3,4,5,  they are not aligned.


### Anything else we need to know?

_No response_

### Kubernetes version

```console
# kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

# uname -a
Linux k8s-work1 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux


```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

### Related plugins (CNI, CSI, ...) and versions (if applicable)