Repository: kubernetes
Issue: ResourceQuota computed used resources for extended resources is wrong
Labels: kind/bug,sig/api-machinery,help wanted,triage/accepted

### What happened?

When using the `ResourceQuotas` admission controller for extended resources (`nvidia.com/gpu`) the reported used resources are inconsistently and wrong.

### What did you expect to happen?

The reported used extended resources should respect the current status in the Namespace.

### How can we reproduce it (as minimally and precisely as possible)?

Apply the following ResourceQuota

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: capsule-skg00000-2
spec:
  hard:
    requests.nvidia.com/gpu: "10"
```

Check the RQ is applied correctly and reported the expected status:

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ResourceQuota","metadata":{"annotations":{},"name":"capsule-skg00000-2","namespace":"skg00000-gpu"},"spec":{"hard":{"requests.nvidia.com/gpu":"10"}}}
  creationTimestamp: "2024-05-26T17:00:28Z"
  labels:
    capsule.clastix.io/managed-by: skg00000
  name: capsule-skg00000-2
  namespace: skg00000-gpu
  resourceVersion: "8225530"
  uid: 3badf2c2-9959-4882-9315-7ffff127e08e
spec:
  hard:
    requests.nvidia.com/gpu: "10"
status:
  hard:
    requests.nvidia.com/gpu: "10"
  used:
    requests.nvidia.com/gpu: "0"
```

Create a Deployment with a template consuming the extended resource (GPU).

```yaml
spec:        
  automountServiceAccountToken: false   
  containers:
  - args:    
    - sleep 3600      
    command: 
    - /bin/bash       
    - -c     
    - --     
    image: nvidia/cuda:11.0.3-base-ubuntu20.04   
    imagePullPolicy: Always    
    name: nvidia      
    resources:        
      limits:
        cpu: "1"      
        memory: 1Gi   
        nvidia.com/gpu: "1"    
      requests:       
        cpu: "1"      
        memory: 1Gi   
        nvidia.com/gpu: "1"
```

After the Pod is deployed, the ResourceQuota reports the wrong amount of used resources:

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ResourceQuota","metadata":{"annotations":{},"name":"capsule-skg00000-2","namespace":"skg00000-gpu"},"spec":{"hard":{"requests.nvidia.com/gpu":"10"}}}
  creationTimestamp: "2024-05-26T17:00:28Z"
  labels:
    capsule.clastix.io/managed-by: skg00000
  name: capsule-skg00000-2
  namespace: skg00000-gpu
  resourceVersion: "8225807"
  uid: 3badf2c2-9959-4882-9315-7ffff127e08e
spec:
  hard:
    requests.nvidia.com/gpu: "10"
status:
  hard:
    requests.nvidia.com/gpu: "10"
  used:
    requests.nvidia.com/gpu: "2"
```

The same applies when performing scale out of the Deployment, with inconsistent reporting.

```
$: kubectl scale deployment gpu-workload --replicas=2
$ RQ reports 4 used instances
$: kubectl scale deployment gpu-workload --replicas=3
$ RQ reports 6 used instances
$: kubectl scale deployment gpu-workload --replicas=4
$ RQ reports 8 used instances
```

Upon scaling down, the used instances are still inconsistent from time to time.

```
$: kubectl scale deployment gpu-workload --replicas=3
$ RQ reports 3 used instances
$: kubectl scale deployment gpu-workload --replicas=2
$ RQ reports 3 used instances
$: kubectl scale deployment gpu-workload --replicas=1
$ RQ reports 1 used instances
$: kubectl scale deployment gpu-workload --replicas=0
$ RQ reports 0 used instances
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
$ uname -a
Linux REDACTED 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>
kubeadm
</details>


### Container runtime (CRI) and version (if applicable)

<details>
N.R.
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
N.R.
</details>