Repository: kubernetes
Issue: ValidatingAdmissionPolicy: auditAnnotations are included in the audit event always
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

When an audit annotation is defined in a `Validating Admission Policy` (VAP), this annotation is added to the api-server audit event always.

### What did you expect to happen?

The audit annotation is only included to the audit event in case any of the VAP validations expressions evaluates to false, in oder words, in case the VAP is violated.

> The documentation is kind of confusing in that regard, because [here](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/#audit-annotations) it says: _In this example the annotation will only be included if the `spec.replicas` of the Deployment is more than 50, otherwise the CEL expression evaluates to null and the annotation will not be included_. But if in fact `spec.replicas` is more than 50, it means that the VAP is met, because the CEL expression evaluates to true.

### How can we reproduce it (as minimally and precisely as possible)?

Create a VAP and binding that prevents creating pods in `default` namespace, like this:

> **Edited:** I had a typo in the  `spec.auditAnnotations[0].key`, correct value is validation_failed, and not validation_failure as I initially posted.

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "prevent-default-namespace"
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["*"]
      apiVersions: ["*"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["pods"]
  validations:
    - expression: "object.metadata.namespace != 'default'"
      message: "Pods should not be created in the default namespace"
  auditAnnotations:
    - key: "validation_failed"
      valueExpression: "'true'"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "prevent-default-namespace"
spec:
  policyName: "prevent-default-namespace"
  validationActions: [Warn, Audit]
```

This policy expression will validate to `false` for all pods created in `default` namespace and to `true` for all pods created in any other pod.

Now create two pods, one of them in `default` namespace:
```bash
❯ kubectl run my-pod-10 --image=curlimages/curl:8.6.0 --command -- sleep 1000
Warning: Validation failed for ValidatingAdmissionPolicy 'prevent-default-namespace' with binding 'prevent-default-namespace': Pods should not be created in the default namespace. 
pod/my-pod-10 created

❯ kubectl run my-pod-20 -n testing --image=curlimages/curl:8.6.0 --command -- sleep 1000
pod/my-pod-20 created
```

If we check the api-server audit event corresponding to the creation of each pod, we can see that the audit annotation `"prevent-default-namespace/validation_failed":"true"` has been set in both cases:

```bash
❯ talosctl read /var/log/audit/kube/kube-apiserver.log | grep "my-pod-10" | grep "validation_failed"
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7703d71a-00e8-4b81-9fb2-d08fbd136fc5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/pods?fieldManager=kubectl-run","verb":"create","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["192.168.65.1"],"userAgent":"kubectl/v1.28.0 (darwin/arm64) kubernetes/855e7c4","objectRef":{"resource":"pods","namespace":"default","name":"my-pod-10","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-15T15:14:40.823489Z","stageTimestamp":"2024-06-15T15:14:40.830217Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}","mutation.webhook.admission.k8s.io/round_0_index_4":"{\"configuration\":\"opentelemetry-operator-mutation\",\"webhook\":\"mpod.kb.io\",\"mutated\":false}","pod-security.kubernetes.io/enforce-policy":"privileged:latest","prevent-default-namespace/validation_failed":"true","validation.policy.admission.k8s.io/validation_failure":"[{\"message\":\"Pods should not be created in the default namespace\",\"policy\":\"prevent-default-namespace\",\"binding\":\"prevent-default-namespace\",\"expressionIndex\":0,\"validationActions\":[\"Warn\",\"Audit\"]}]"}}

❯ talosctl read /var/log/audit/kube/kube-apiserver.log | grep "my-pod-20" | grep "validation_failed"
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7ba57b1d-6cbd-489d-bef1-709f337b9eda","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/testing/pods?fieldManager=kubectl-run","verb":"create","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["192.168.65.1"],"userAgent":"kubectl/v1.28.0 (darwin/arm64) kubernetes/855e7c4","objectRef":{"resource":"pods","namespace":"testing","name":"my-pod-20","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-15T15:15:49.607569Z","stageTimestamp":"2024-06-15T15:15:49.614769Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}","mutation.webhook.admission.k8s.io/round_0_index_4":"{\"configuration\":\"opentelemetry-operator-mutation\",\"webhook\":\"mpod.kb.io\",\"mutated\":false}","pod-security.kubernetes.io/enforce-policy":"privileged:latest","prevent-default-namespace/validation_failed":"true"}}
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

I'm using in this case Talos in Docker, but it should not be relevant in this case:

```console
❯ talosctl read /etc/os-release
NAME="Talos"
ID=talos
VERSION_ID=v1.7.2
PRETTY_NAME="Talos (v1.7.2)"
HOME_URL="https://www.talos.dev/"
BUG_REPORT_URL="https://github.com/siderolabs/talos/issues"
VENDOR_NAME="Sidero Labs"
VENDOR_URL="https://www.siderolabs.com/"

```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>