Kubernetes Cronjob Best Practices
Tags: kubernetes, devops
Published: 2020-11-25 21:56:30

This post is to document the best practices I have realized over last few months of using kubernetes **cronjob** for scheduled workloads.

Like most developers out there, I started writing my first kubernetes cronjob yaml files following [the official guide example](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) that does not specify a lot of useful parameters. However, it took me quite some time and unexpected execution behavior of kubernetes cronjobs to realize the importance of _consciously_ specifying all parameters and not rely blindly on default values.

Here are such useful parameters that I think we should always define for a kubernetes cronjob specification (the emphasis is on always defining these parameters and not necessarily on all their values I am showing below) -


1. concurrencyPolicy
  * Set `concurrencyPolicy: Forbid` if you want to restrict another cron execution when a previous one is still running.

2. suspend
  * Set `suspend: true` if you want to temporarily disable cronjob executions for some time to troubleshoot any issue without having to delete the cronjob itself.

3. successfulJobsHistoryLimit
  * Set `successfulJobsHistoryLimit: 2` to retain last 2 successful cronjob execution history.

4. failedJobsHistoryLimit
  * Set `failedJobsHistoryLimit: 3` to retain last 3 failed cronjob execution history. Default value is 1 which I think may not be enough to troubleshoot failures.\ 

Although following parameters belong to kubernetes `Job` which is the unit of `Cronjob`, but I think are equally important when configuring a cronjob -


1. parallelism
  * Even though this defaults to 1, set it explicitly in your code to help yourself understand your code weeks/months after.
  * Set `parallelism: 1` to restrict only 1 job to run in parallel at any point of time.

2. completions
  * Set `completions: 1` when you want the job to be marked as successful.
  * This makes more sense when, say, you have 3 pods running in parallel and want to mark the job as successful when all 3 pods successfully complete execution.

3. restartPolicy
  * Set `restartPolicy: OnFailure` at the pod specification level to let kubernetes restart the pod container when your workload process exits with failure. 
  * Use `backoffLimit` parameter along with this to protect from back to back restarting behavior during consistent failures.

4. backoffLimit
  * Jobs can fail for variety of reasons like a process failing within pod or a failure at kubernetes controller layer, that can cause automatic retries of job executions.
  * Set `backoffLimit: 3` to ensure that jobs are not retried unlimited times but only 3 times after which the job (i.e. cronjob execution in our article) will fail and halt.

5. activeDeadlineSeconds
  * If there is a possibility of your workload process potentially consuming longer than expected (like when interacting with external dependent service), then you must set a time limit up to which kubernetes will allow the cron execution and kill it if exceeded.
  * Set `activeDeadlineSeconds: 60` (example value) to let kubernetes kill the job execution if it takes longer than 60 seconds.

I hope this helps someone and feel free to share your related experiences.