Repository: kubernetes
Issue: Unexpected Job Creation After CronJob Schedule Update
Labels: kind/bug,sig/apps,triage/accepted

### What happened?

I have a CronJob running on an EKS cluster (1.28) with an initial schedule of `40 23 * * *`.
1. At `23:40 UTC`, a job was successfully completed, taking few seconds. 
2. At around `23:55 UTC`,  I changed the CronJob schedule to `50 23 * * *`. 
3. At `03:31:03 UTC`, an unexpected job was created.

I was able to reproduce the issue on another EKS cluster (1.28). Since I can't increase the verbosity level in EKS to gather more detailed logs from the controller manager, I decided to reproduce the issue on a local cluster created via Kind. I managed to reproduce the problem on both 1.28 and 1.31.

When testing on local cluster(1.28), I discovered a more interesting scenario. I had two CronJobs `cronjob-test-2` and `cronjob-test-3`
- cronjob-test-2: `49 9 * * *` -> `51 9 * * *`, with the change occurring shortly after `9:51 UTC`
- cronjob-test-3: `35 4 * * *` -> `38 4 * * *`, with the change occurring shortly after `4:38 UTC`

Surprisingly, both CronJobs created an unexpected job at the same time, `22:50:22 UTC`. Additionally, `cronjob-test-2` even missed an expected job
```
NAME                      COMPLETIONS   DURATION   AGE
cronjob-test-2-28816429   1/1           8s         38h    # a job was missed after this one
cronjob-test-2-28817871   1/1           6s         110m   # 22:50:22 UTC
cronjob-test-3-28817555   1/1           5s         20h
cronjob-test-3-28817558   1/1           4s         110m   # 22:50:22 UTC
``` 
<details>
  <summary>Cronjob Controller Logs</summary>
Following is the logs from cronjob_controllerv2 and job_controller, and the verbosity was set as 4.

```
I1015 09:45:19.354428       1 job_controller.go:226] "Starting job controller"
I1015 09:45:19.507521       1 cronjob_controllerv2.go:139] "Starting cronjob controller v2"
I1015 09:47:47.961434       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:47:47.961466       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="1m12.13858604s"
I1015 09:49:00.115477       1 cronjob_controllerv2.go:633] "Created Job" job="default/cronjob-test-2-28816429" cronjob="default/cronjob-test-2"
I1015 09:49:00.115531       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:00.115632       1 job_controller.go:1545] "Too few pods running" key="default/cronjob-test-2-28816429" need=1 creating=1
I1015 09:49:00.119191       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m59.999440367s"
I1015 09:49:00.119243       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:49:00.119255       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m59.980770867s"
I1015 09:49:00.122067       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:00.123710       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:00.124323       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:00.124356       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28816429" elapsed="8.80725ms"
I1015 09:49:00.124368       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:49:00.124392       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m59.975645909s"
I1015 09:49:00.127551       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:01.122770       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28816429" elapsed="373.625µs"
I1015 09:49:06.780723       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:07.781300       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28816429" elapsed="113.042µs"
I1015 09:49:07.843924       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:08.784246       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:08.849759       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:08.849794       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:49:08.849808       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m51.250220655s"
I1015 09:49:08.852371       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:08.854308       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28816429" elapsed="9.805583ms"
I1015 09:49:08.854309       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28816429"
I1015 09:49:08.854353       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:49:08.856452       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m51.245670446s"
I1015 09:49:08.856481       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:49:08.858155       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m51.243529696s"
I1015 09:49:08.858215       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1015 09:49:08.858229       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="23h59m51.241795738s"
I1015 09:49:09.850763       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28816429" elapsed="88.25µs"
I1016 04:33:45.800541       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:33:45.800579       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="1m14.299487083s"
I1016 04:35:00.107918       1 cronjob_controllerv2.go:633] "Created Job" job="default/cronjob-test-3-28817555" cronjob="default/cronjob-test-3"
I1016 04:35:00.108011       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:00.108159       1 job_controller.go:1545] "Too few pods running" key="default/cronjob-test-3-28817555" need=1 creating=1
I1016 04:35:00.109944       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m59.999111701s"
I1016 04:35:00.109981       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:00.109991       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m59.990031618s"
I1016 04:35:00.110014       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:00.110039       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m59.989994201s"
I1016 04:35:00.113206       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:00.115032       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:00.115046       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817555" elapsed="7.007833ms"
I1016 04:35:00.115074       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:00.115103       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m59.984938868s"
I1016 04:35:00.115152       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:00.118528       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:01.115214       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817555" elapsed="191.917µs"
I1016 04:35:03.142287       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:04.142862       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817555" elapsed="230.125µs"
I1016 04:35:04.210513       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:05.149604       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:05.218654       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:05.218724       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:05.218770       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m54.881306115s"
I1016 04:35:05.221914       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:05.223483       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817555"
I1016 04:35:05.223533       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817555" elapsed="11.988792ms"
I1016 04:35:05.223535       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:05.225526       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m54.876482823s"
I1016 04:35:05.225572       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:05.227006       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m54.874438157s"
I1016 04:35:05.227037       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 04:35:05.227051       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="23h59m54.87297224s"
I1016 04:35:06.220824       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817555" elapsed="90.458µs"
I1016 22:50:22.080092       1 cronjob_controllerv2.go:633] "Created Job" job="default/cronjob-test-3-28817558" cronjob="default/cronjob-test-3"
I1016 22:50:22.080204       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:22.080190       1 cronjob_controllerv2.go:633] "Created Job" job="default/cronjob-test-2-28817871" cronjob="default/cronjob-test-2"
I1016 22:50:22.080302       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:22.080310       1 job_controller.go:1545] "Too few pods running" key="default/cronjob-test-3-28817558" need=1 creating=1
I1016 22:50:22.080342       1 job_controller.go:1545] "Too few pods running" key="default/cronjob-test-2-28817871" need=1 creating=1
I1016 22:50:22.082913       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m38.030291112s"
I1016 22:50:22.082947       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 22:50:22.082983       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m38.017065028s"
I1016 22:50:22.083007       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m38.030373028s"
I1016 22:50:22.083050       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1016 22:50:22.083069       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m38.016961653s"
I1016 22:50:22.086221       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:22.086290       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:22.087934       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:22.087979       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 22:50:22.088001       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m38.012036153s"
I1016 22:50:22.088076       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817558" elapsed="7.858958ms"
I1016 22:50:22.088471       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:22.088499       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1016 22:50:22.088512       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m38.011512362s"
I1016 22:50:22.088654       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28817871" elapsed="8.343708ms"
I1016 22:50:22.088727       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:22.088770       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:22.092991       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:22.095171       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:23.087955       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817558" elapsed="118.291µs"
I1016 22:50:23.087976       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28817871" elapsed="154.625µs"
I1016 22:50:24.887122       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:25.888042       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817558" elapsed="201.375µs"
I1016 22:50:25.929334       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:26.891926       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:26.894910       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:26.932416       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:26.932463       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 22:50:26.932503       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m33.167550999s"
I1016 22:50:26.934834       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:26.936763       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-3-28817558"
I1016 22:50:26.936769       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817558" elapsed="6.805541ms"
I1016 22:50:26.936798       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 22:50:26.938716       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m33.163216499s"
I1016 22:50:26.938804       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 22:50:26.940126       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m33.161207957s"
I1016 22:50:26.940153       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-3"
I1016 22:50:26.940166       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-3" requeueAfter="5h47m33.15985679s"
I1016 22:50:27.895393       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28817871" elapsed="355.792µs"
I1016 22:50:27.932640       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-3-28817558" elapsed="31.875µs"
I1016 22:50:27.958622       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:28.906520       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:28.963964       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:28.964008       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1016 22:50:28.964022       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m31.136008123s"
I1016 22:50:28.966974       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:28.968936       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28817871" elapsed="9.14725ms"
I1016 22:50:28.968962       1 job_controller.go:562] "enqueueing job" key="default/cronjob-test-2-28817871"
I1016 22:50:28.968997       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1016 22:50:28.972178       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m31.131016956s"
I1016 22:50:28.972209       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1016 22:50:28.973757       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m31.127801331s"
I1016 22:50:28.973821       1 cronjob_controllerv2.go:526] "No unmet start times" cronjob="default/cronjob-test-2"
I1016 22:50:28.973837       1 cronjob_controllerv2.go:219] "Re-queuing cronjob" cronjob="default/cronjob-test-2" requeueAfter="11h0m31.126191373s"
I1016 22:50:29.964871       1 job_controller.go:717] "Finished syncing job" key="default/cronjob-test-2-28817871" elapsed="122.875µs"
```
</details>

### What did you expect to happen?

I expected the CronJobs to execute only according to the updated schedules and not trigger any additional or unscheduled jobs.

### How can we reproduce it (as minimally and precisely as possible)?

1. Create a CronJob with the schedule to run at time `A` every day. 
2. Allow the job to complete at time `A`.
2. At a later time `C`, update the schedule to `B`. `A < B < C`, `A` is the earliest time. 
3. Wait for several hours. Based on my tests, an unexpected job will be triggered, sometimes even more than 24 hours after `C`.  When this happens after 24 hours, the expected job that should be created at time `C` on the next day is replaced by the unexpected one.

```
kubectl create cronjob cronjob-test --image=busybox --schedule="A.minute A.hour * * *" -- date

# wait until job created at time A, run it at time C
kubectl patch cronjob cronjob-test -p '{"spec": {"schedule": "B.minute B.hour * * *"}}'
```

Unfortunately, I haven't been able to narrow down the creation time of the unexpected job, but this sequence reliably reproduces the issue.

### Anything else we need to know?

Timezone: 
```
cat /etc/localtime
TZif2UTCTZif2UTC
UTC0
```

Based on my observation:
- **All the unexpected jobs were triggered at the exact same time**
For example, I had three CronJobs with different schedules. After rescheduling them to a later time, a few hours later, the controller manager created all three unexpected jobs simultaneously, even though neither the original nor the new schedules of these CronJobs were the same.
- **The issue only occurs when the reschedule creates an unmet job schedule.** For example, updating from `49 9 * * *` to `48 9 * * *` works fine, but doing the reverse (moving from an earlier to a later time) causes the problem.

This is similar to the issue mentioned [here](https://github.com/kubernetes/kubernetes/issues/123220#), though in my case, the unexpected jobs can be created either on the same day or the next day.

A big thanks to @soltysh  for helping me troubleshoot this issue.


### Kubernetes version

<details>

I managed to reproduce the same issue across three Kubernetes clusters with the following versions

```
eks: v1.28.13-eks-a737599
kind: v1.28.12 and v1.31.0
```

</details>


### Cloud provider

<details>
EKS and Kind on local macOS
</details>