Repository: kubernetes
Issue: NUMA-aware memory manager and Topology Manager policy of "restricted" results in TopologyAffinityError when it shouldn't
Labels: priority/backlog,sig/node,kind/feature,triage/accepted

### What happened?

Running K8s 1.29.2, with the kubelet NUMA-aware memory manager policy set to "Static" and the Topology Manager policy set to "restricted".

1. The /var/lib/kubelet/memory_manager_state file shows:

```
[sysadmin@controller-0 pods(keystone_admin)]$ sudo cat /var/lib/kubelet/memory_manager_state
Password:
{"policyName":"Static","machineState":{"0":{"numberOfAssignments":0,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99759243264,"systemReserved":10538188800,"allocatable":78483636224,"reserved":0,"free":78483636224}},"cells":[0]},"1":{"numberOfAssignments":0,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99283886080,"systemReserved":1101004800,"allocatable":87445463040,"reserved":0,"free":87445463040}},"cells":[1]}},"checksum":2552710201}[sysadmin@controller-0 pods(keystone_admin)]$ 
```

2. Created a pod kube-mgrr-1, then checked pod status and memory_manager_state file content

```
[sysadmin@controller-0 pods(keystone_admin)]$ cat test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-mgrr-1
  labels:
    app.starlingx.io/component: 'application'
spec:
  containers:
  - image: "gcr.io/kubernetes-e2e-test-images/resource-consumer:1.4"
    imagePullPolicy: IfNotPresent
    name: kube-mgrr-1
    resources:
      limits:
        memory: 256Mi
        cpu: '200m'
      requests:
        memory: 256Mi
        cpu: '200m'
  nodeSelector:
    kubernetes.io/hostname: controller-0

[sysadmin@controller-0 pods(keystone_admin)]$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
kube-mgrr-1   1/1     Running   0          7s

[sysadmin@controller-0 pods(keystone_admin)]$ sudo cat /var/lib/kubelet/memory_manager_state
{"policyName":"Static","machineState":{"0":{"numberOfAssignments":1,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99759243264,"systemReserved":10538188800,"allocatable":78483636224,"reserved":268435456,"free":78215200768}},"cells":[0]},"1":{"numberOfAssignments":0,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99283886080,"systemReserved":1101004800,"allocatable":87445463040,"reserved":0,"free":87445463040}},"cells":[1]}},"entries":{"84accdfa-8263-4672-9bfe-2ffd93427666":{"kube-mgrr-1":[{"numaAffinity":[0],"type":"memory","size":268435456}]}},"checksum":989528933}[sysadmin@controller-0 pods(keystone_admin)]$ 
```

3.  Created a new pod which exceeds memory of NUMA node 0 and check if the pod is pinned to both NUMA nodes.

```
[sysadmin@controller-0 pods(keystone_admin)]$ cat test-pod_2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-mgrr-2
  labels:
    app.starlingx.io/component: 'application'
spec:
  containers:
  - image: "gcr.io/kubernetes-e2e-test-images/resource-consumer:1.4"
    imagePullPolicy: IfNotPresent
    name: kube-mgrr-2
    resources:
      limits:
        memory: 85Gi
        cpu: '200m'
      requests:
        memory: 85Gi
        cpu: '200m'
  nodeSelector:
    kubernetes.io/hostname: controller-0

[sysadmin@controller-0 pods(keystone_admin)]$ kubectl apply -f test-pod_2.yaml 
pod/kube-mgrr-2 created 

[sysadmin@controller-0 pods(keystone_admin)]$ kubectl get pods NAME          READY   STATUS    RESTARTS   AGE kube-mgrr-1   1/1     Running   0          109s kube-mgrr-2   1/1     Running   0          3s 

[sysadmin@controller-0 pods(keystone_admin)]$ sudo cat /var/lib/kubelet/memory_manager_state {"policyName":"Static","machineState":{"0":{"numberOfAssignments":2,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99759243264,"systemReserved":10538188800,"allocatable":78483636224,"reserved":78483636224,"free":0}},"cells":[0,1]},"1":{"numberOfAssignments":1,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99283886080,"systemReserved":1101004800,"allocatable":87445463040,"reserved":13052854272,"free":74392608768}},"cells":[0,1]}},"entries":{"84accdfa-8263-4672-9bfe-2ffd93427666":{"kube-mgrr-1":[{"numaAffinity":[0],"type":"memory","size":268435456}]},"8e7262b9-c5ae-4b2e-a0b6-5dba8c4b6d64":{"kube-mgrr-2":[{"numaAffinity":[0,1],"type":"memory","size":91268055040}]}},"checksum":3041831511}
```

4. Delete kube-mgrr-1 pod

```
[sysadmin@controller-0 pods(keystone_admin)]$ kubectl delete -f test-pod.yaml
pod "kube-mgrr-1" deleted
[sysadmin@controller-0 pods(keystone_admin)]$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
kube-mgrr-2   1/1     Running   0          49s
```
 
5. Add kube-mgrr-1 back . Pod status is TopologyAffinityError

```
[sysadmin@controller-0 pods(keystone_admin)]$ kubectl get pods
NAME          READY   STATUS                  RESTARTS   AGE
kube-mgrr-1   0/1     TopologyAffinityError   0          21m
kube-mgrr-2   1/1     Running                 0          22m
[sysadmin@controller-0 pods(keystone_admin)]$
 
[sysadmin@controller-0 pods(keystone_admin)]$ sudo cat /var/lib/kubelet/memory_manager_state
Password:
{"policyName":"Static","machineState":{"0":{"numberOfAssignments":1,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99759243264,"systemReserved":10538188800,"allocatable":78483636224,"reserved":78215200768,"free":268435456}},"cells":[0,1]},"1":{"numberOfAssignments":1,"memoryMap":{"hugepages-1Gi":{"total":10737418240,"systemReserved":0,"allocatable":10737418240,"reserved":0,"free":10737418240},"hugepages-2Mi":{"total":0,"systemReserved":0,"allocatable":0,"reserved":0,"free":0},"memory":{"total":99283886080,"systemReserved":1101004800,"allocatable":87445463040,"reserved":13052854272,"free":74392608768}},"cells":[0,1]}},"entries":{"5131157f-ca8a-4824-8fda-a57a4ceab719":{"kube-mgrr-2":[{"numaAffinity":[0,1],"type":"memory","size":91268055040}]}},"checksum":3733538882}
```


Note: if the topology manager policy is set to "best-effort" the request to respawn the Pod succeeds.






### What did you expect to happen?

When re-creating the pod that was killed, it should be able to start up again since there is enough memory available on a single node to satisfy the memory and CPU requests.

### How can we reproduce it (as minimally and precisely as possible)?

Set the memory manager policy to "Static" and topology manager policy to "restricted".   On a two-NUMA-node worker node create a smallish Pod in the Guaranteed QoS class that easily fits on one NUMA node worth of memory.  Create a second Pod in the Guaranteed QoS class with a big enough memory request that it cannot fit on one NUMA node.  Delete the first Pod.  After the memory freed up by the Pod deletion shows as available, attempt to recreate the first Pod.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
sysadmin@controller-0:~$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2
```

</details>


### Cloud provider

<details>
Wind River Cloud Platform
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
sysadmin@controller-0:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux controller-0 6.6.0-1-rt-amd64 #1 SMP PREEMPT_RT StarlingX Debian 6.6.52-1.stx.95 (2024-10-31) x86_64 GNU/Linux

```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>