Repository: kubernetes
Issue: Cronjobs occasionally run much later than scheduled
Labels: kind/bug,sig/apps,needs-triage

### What happened?

CronJobs will occasionally fail to create their Jobs at the scheduled time, with delays ranging from 3 minutes to over 50 minutes. In the most extreme cases we've observed delays over 60 minutes. 

From the kube-controller-manager logs, if I'm understanding correctly (which i may not), it appears that CronJobs rely exclusively on the reconciliation process to trigger Job creation. If no reconciliation occurs around the scheduled time, the Job is not created until a subsequent reconciliation cycle is triggered.

Logs consistently show the following pattern (verbosity set to 8), even for CronJobs that execute on time, this job ran 5 minutes later than scheduled:
```console
I1120 11:05:32.562672       1 cronjob_controllerv2.go:526] "No unmet start times" logger="cronjob-controller" cronjob="camel-test/mkt-cloud-segments-servicecloud-v2-simu-buidar-pp" 
I1120 11:05:32.562702       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" logger="cronjob-controller" cronjob="camel-test/mkt-cloud-segments-servicecloud-v2-simu-buidar-pp" requeueAfter="23h54m27.538988861s"
```
Ocasionally this pattern can also be observed in the kube-controller-manager logs:
```console
E1120 13:33:05.831607       1 cronjob_controllerv2.go:168] error syncing CronJobController golang/datafeeds-download-datafeeds, requeuing: Operation cannot be fulfilled on cronjobs.batch "datafeeds-download-datafeeds": the object has been modified; please apply your changes to the latest version and try again
```

Performing manual actions, such as editing the CronJob object or deleting an old Job, forces an immediate reconciliation and triggers the creation of the pending Job.

Another thing we observed is that the issue only affects cronjobs that run on a daily basis (these are distributed at different times) but those that run frequently, like every 15 minutes, never get delayed.

Things we checked:
1. Control-plane clocks verified: All clocks are synchronized using NTP. No drift observed.
2. Increased logging verbosity: Enabled v=8 for kube-controller-manager
3. The number of cronjobs does not seem to be too much for the size of the cluster and resource utilization

We don't know what could be causing this behavior and we don't know how to continue debugging it further.

### What did you expect to happen?

CronJobs should create their corresponding Jobs at the scheduled time.

### How can we reproduce it (as minimally and precisely as possible)?

These are the details of the basic current cluster configuration, however I do not know if the issue could be replicated in another similar cluster.

- Cluster Size: 8 worker nodes, 3 control-plane nodes
- CronJob API Version: batch/v1
- Number of CronJobs: ~100 scheduled at various intervals, distributed in multiple namespaces

Resource Utilization:
- Control-plane nodes have sufficient CPU and memory resources (not near limits).
- The kube-controller-manager pod does not have resource limits set.

### Anything else we need to know?

Keda operator is installed and we have 3 deployments that are managed by Keda, logs regarding these objects are frequent:
```console
11:46:17.755947 horizontal.go:841] "Proposing desired replicas" logger="horizontal-pod-autoscaler-controller" desiredReplicas=0 metric="external metric s0-cron-Europe-Madrid-07xx1-5-021xx1-5(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: cron-activemq-dev,},MatchExpressions:[]LabelSelectorRequirement{},})" timestamp="2024-11-19 11:46:17 +0100 CET" scaleTarget="Deployment/activemq-test/activemq-dev"
11:46:17.760043 horizontal.go:841] "Proposing desired replicas" logger="horizontal-pod-autoscaler-controller" desiredReplicas=1 metric="external metric s0-cron-Europe-Madrid-0000xxx-5523xxx(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: cron-integracions-erpex,},MatchExpressions:[]LabelSelectorRequirement{},})" timestamp="2024-11-19 11:46:17 +0100 CET" scaleTarget="Deployment/camel/integracions-erpex"
11:46:31.385874 graph_builder.go:681] "GraphBuilder process object" logger="garbage-collector-controller" apiVersion="keda.sh/v1alpha1" kind="ScaledObject" object="camel/cron-integracions-erpex" uid="d9d7f22a-0eef-4c21-8c23-7823fc6e1f60" eventType="update" virtual=false
11:46:31.900165 graph_builder.go:681] "GraphBuilder process object" logger="garbage-collector-controller" apiVersion="keda.sh/v1alpha1" kind="ScaledObject" object="activemq-test/cron-activemq-dev" uid="eef2ed20-c9ef-40a2-bee4-3c0fcc71dd5a" eventType="update" virtual=false
11:46:32.782034 horizontal.go:841] "Proposing desired replicas" logger="horizontal-pod-autoscaler-controller" desiredReplicas=0 metric="external metric s0-cron-Europe-Madrid-07xx1-5-021xx1-5(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: cron-activemq-dev,},MatchExpressions:[]LabelSelectorRequirement{},})" timestamp="2024-11-19 11:46:32 +0100 CET" scaleTarget="Deployment/activemq-test/activemq-dev"
11:46:32.787861 horizontal.go:841] "Proposing desired replicas" logger="horizontal-pod-autoscaler-controller" desiredReplicas=1 metric="external metric s0-cron-Europe-Madrid-0000xxx-5523xxx(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: cron-integracions-erpex,},MatchExpressions:[]LabelSelectorRequirement{},})" timestamp="2024-11-19 11:46:32 +0100 CET" scaleTarget="Deployment/camel/integracions-erpex"
```

There is a custom operator that observes events on several custom resources.

We've observed this issue through multiple kubernetes versions, no upgrade has fixed it.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.30.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6
```
</details>


### Cloud provider

<details>
On premise
</details>


### OS version

<details>

```console
# On Linux:
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux k8scp02 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```
</details>


### Install tools

<details>
kubeadm
</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd 1.7.12-0ubuntu2~22.04.1
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
vwmare-csi v3.3.1

cilium v1.16.1
</details>