Repository: kubernetes
Issue: NUMA-aware memory manager and Topology Manager policy of "restricted" results in UnexpectedAdmissionError
Labels: kind/bug,sig/node,priority/important-longterm,triage/accepted

### What happened?

While trying to reproduce https://github.com/kubernetes/kubernetes/issues/128669 I spun up a VM to test 1.31.2 via minikube and I think I might have uncovered a new and different bug.

I had 8GB of allocatable memory on each of two NUMA nodes. Key kubelet args were:

-cpu-manager-policy=static --kube-reserved=memory=1Gi --memory-manager-policy=Static --reserved-cpus=0,4 --reserved-memory=0:memory=1Gi;1:memory=1Gi --system-reserved=memory=1Gi --topology-manager-policy=restricted

I was able to create the first pod with 1cpu and 256Mi of memory, but when I tried to create the second pod with 1cpu and 9Gi of memory (to force it to allocate memory from both NUMA nodes) it errored out unexpectedly:

cfriesen@debian:~$ minikube kubectl -- get pod kube-mgrr-2 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"kube-mgrr-2","namespace":"default"},"spec":{"containers":[{"image":"gcr.io/kubernetes-e2e-test-images/resource-consumer:1.4","imagePullPolicy":"IfNotPresent","name":"kube-mgrr-2","resources":{"limits":{"cpu":1,"memory":"9Gi"}}}]}}
  creationTimestamp: "2024-11-08T19:34:55Z"
  name: kube-mgrr-2
  namespace: default
  resourceVersion: "1028"
  uid: 2e769357-b700-49fa-96dc-2416a0379cb9
spec:
  containers:
  - image: gcr.io/kubernetes-e2e-test-images/resource-consumer:1.4
    imagePullPolicy: IfNotPresent
    name: kube-mgrr-2
    resources:
      limits:
        cpu: "1"
        memory: 9Gi
      requests:
        cpu: "1"
        memory: 9Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-27rcb
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: minikube
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-27rcb
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  message: 'Pod was rejected: Allocate failed due to [memorymanager] failed to find
    NUMA nodes to extend the current topology hint, which is unexpected'
  phase: Failed
  reason: UnexpectedAdmissionError
  startTime: "2024-11-08T19:34:55Z"


Kubelet logs were:

Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.632981 2326 scope_container.go:75] "TopologyHints" hints={} pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633153 2326 policy_static.go:541] "TopologyHints generated" pod="default/kube-mgrr-2" containerName="kube-mgrr-2" cpuHints=[{"NUMANodeAffinity":1,"Preferred":true},{"NUMANodeAffinity":2,"Preferred":true},{"NUMANodeAffinity":3,"Preferred":false}]
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633187 2326 scope_container.go:75] "TopologyHints" hints={"cpu":[{"NUMANodeAffinity":1,"Preferred":true},{"NUMANodeAffinity":2,"Preferred":true},{"NUMANodeAffinity":3,"Preferred":false}]} pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633254 2326 scope_container.go:75] "TopologyHints" hints={} pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633276 2326 policy.go:71] "Hint Provider has no preference for NUMA affinity with any resource"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633289 2326 policy.go:71] "Hint Provider has no preference for NUMA affinity with any resource"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633304 2326 scope_container.go:83] "ContainerTopologyHint" bestHint={"NUMANodeAffinity":1,"Preferred":true}
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633330 2326 scope_container.go:50] "Best TopologyHint" bestHint={"NUMANodeAffinity":1,"Preferred":true} pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633342 2326 scope_container.go:56] "Topology Affinity" bestHint={"NUMANodeAffinity":1,"Preferred":true} pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633391 2326 policy_static.go:303] "Static policy: Allocate" pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633407 2326 policy_static.go:352] "Topology Affinity" pod="default/kube-mgrr-2" containerName="kube-mgrr-2" affinity={"NUMANodeAffinity":1,"Preferred":true}
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633421 2326 policy_static.go:392] "AllocateCPUs" numCPUs=1 socket="01"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.633496 2326 state_mem.go:88] "Updated default CPUSet" cpuSet="0,3-7"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.635778 2326 policy_static.go:424] "AllocateCPUs" result="2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.635830 2326 state_mem.go:80] "Updated desired CPUSet" podUID="2e769357-b700-49fa-96dc-2416a0379cb9" containerName="kube-mgrr-2" cpuSet="2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.638325 2326 policy_static.go:106] "Allocate" pod="default/kube-mgrr-2" containerName="kube-mgrr-2"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.638367 2326 policy_static.go:123] "Got topology affinity" pod="default/kube-mgrr-2" podUID="2e769357-b700-49fa-96dc-2416a0379cb9" containerName="kube-mgrr-2" hint={"NUMANodeAffinity":1,"Preferred":true}
Nov 08 19:34:55 minikube kubelet[2326]: E1108 19:34:55.638418 2326 memory_manager.go:257] "Allocate error" err="[memorymanager] failed to find NUMA nodes to extend the current topology hint"
Nov 08 19:34:55 minikube kubelet[2326]: I1108 19:34:55.638450 2326 kubelet.go:2306] "Pod admission denied" podUID="2e769357-b700-49fa-96dc-2416a0379cb9" pod="default/kube-mgrr-2" reason="UnexpectedAdmissionError" message="Allocate failed due to [memorymanager] failed to find NUMA nodes to extend the current topology hint, which is unexpected"


I modified the second pod to request '200m' worth of CPU rather than a whole CPU, and the pod started up as expected.


### What did you expect to happen?

The pod should have started up with one exclusive CPU and memory from both NUMA nodes.

### How can we reproduce it (as minimally and precisely as possible)?

Set the memory manager policy to "Static" and topology manager policy to "restricted". On a two-NUMA-node worker node create a smallish Pod with a single exclusive CPU (request/limit both 1 cpu) that easily fits on one NUMA node worth of memory. Create a second Pod in the Guaranteed QoS class with a big enough memory request that it cannot fit on one NUMA node, with cpu request/limit both set to 1 cpu. 

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
cfriesen@debian:~$ minikube kubectl -- version
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.2

```

</details>


### Cloud provider

<details>
n/a
</details>


### OS version

<details>

```console
# On Linux:
cfriesen@debian:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
cfriesen@debian:~$ uname -a
Linux debian 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux


```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>