Running scheduled jobs / tasks using GitLab CI/CD
Tags: docker, gitlab, automation, cicd
Published: 2021-03-14 11:40:29

# TL;DR
I use GitLab's CI/CD free feature to automatically run code in the cloud.

---

# Background
Every now and then I come across the need to automate a task (written in code) to be performed periodically, so that it'll run by itself, without my intervention.
Such tasks can be submitting an online form, scraping data from a website, running some script that updates something else online, etc.
I obviously don't want this code to run on my computer, as it is not on and ready all the time, nor do I want to setup and plug in a Raspberry Pi device or alike just for that.

Now, running it periodically sounded like a very simple and straight-forward task, but it turned out to be a bit tricky! After some research ([not too much](https://lmgtfy.app/?q=free+cron+scheduler), I admit), I was not able to find a free and easy-to-use tool / service that would allow me to have some piece of code run periodically in the cloud. I did not want to pull out the bigger guns such as [AWS](https://aws.amazon.com/lambda/) or [GCP](https://cloud.google.com/functions) just in order to setup a tiny serverless function either.

---

# Solution
My solution was to turn to [GitLab's awesome CI/CD-as-a-service feature](https://docs.gitlab.com/ce/ci/) - which I know very well already and have used in many other personal / work project.
This feature, in a nutshell, allows you to define a `.yml` file with your CI/CD pipeline configuration in a repo, and have set it to run in any automatic schedule (or with a trigger). A pipeline is composed of jobs, which run any script you need in any container.

So in that case, all we need to do is:
1. Create a project in GitLab.
1. Code the task that we want to perform (in any language / framework).
1. Set up the GitLab's project's CI/CD so that it will run the correct job with your desired settings (schedule, webhook, manually, etc.).
1. Move on with your (easier) life.

## Create a GitLab project
1. Sign up if you haven't already, and go to https://gitlab.com/projects/new#blank_project in order to create a new project.
1. Fill in the relevant details and create the project.

![Creating a new project on gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u3pdi1mqeiq403sifb28.png) 

## Add the task itself
For this example, let's create this `run-me.js` file:

```js
console.log('Hi there, I ran and made your life easier!', new Date());
```

You can either clone the repo that you've just created and commit-push the task's file from your computer, or easily enough add it straight into gitlab.com using their nice "Web IDE":

![Creating a new file straight from gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n9owhbeofq9wrod7k6w3.png)

![Entering the task's code on gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/je7ixm1xn4jocgosp0rb.png)
 
Commit the changes, and now you have the task's code in the repo - this code will be available inside the container that will run the pipeline.

## Set up CI/CD to run your code as you wish
The same way you did in the previous step, add another file called `.gitlab-ci.yml`.
GitLab offer some useful templates, but in our case, we'll just go ahead with a blank file, and in it add your definition for container image, and what to run in it (including the task you wrote):

```yml
image: node:alpine
run-task:
 only:
   - schedules
 script:
   - node ./run-me.js
```

![Creating a new `.gitlab-ci.yml` file on gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ezo9uq0d2wsmi4r1k6xm.png)
  
### Notes:
* Here I chose `node:alpine` as the container. Why? `node` because my code is in Node.js, and `:alpine` because it's light and fast (quicker to launch). Why not? You may not work with Node.js, and in [Alpine Linux](https://alpinelinux.org/) you might need to install additional tools / programs that do not come out-of-the-box.
* The `only:` part makes sure that our job only runs in scheduled runs, and not in any push to `master` (which is the default).
* Whatever is under `script:` are commands that you run in a `bash` (in Alpine it's `sh`), already inside the directory of the cloned repo. This is already a script by itself, so if you're a `bash` expert, go ahead and do what you need directly there.

For testing, you can go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipelines/new and (provided that you commit to `master`) start a new pipeline manually.

After running, you can go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipelines, and see a new line with the pipeline that has started:

![Running pipeline on gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4txt0t7fzq2pl40rui5s.png)
 

The pipeline will fetch and run the `node:alpine` image, and then execute our code from `run-me.js` inside it 💪.

![Log of a pipeline's job on gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dxxiere70wzya037mihk.png)

This is already awesome, as you can now run any code inside a container in the cloud!

In order to schedule it, go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipeline_schedules/new, and create a new schedule for this repo's pipeline.
In this example, let's say we want it to be Sunday to Friday at 7:30 AM, which means setting a custom interval pattern as `30 7 * * 0–5`. If you need help with cron syntax, check https://crontab.guru.

![Pipeline schedule configuration on gitlab.com](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iyixi4mro3ot7fogpup6.png)
 

Save the pipeline schedule and you can expect it to run surely and independently ☁️.

---

# Notes
* [GitLab's free plan](https://about.gitlab.com/pricing/) includes 400 CI/CD minutes per month, which should be more than enough for simple tasks running on a lean container (such as Alpine). For example the code in this example takes 22 seconds to run, which means it has ~1,200 runs per month before exceeding the 400 minutes quota. If you need more, you can either upgrade your plan, or [setup your own CI/CD runner](https://docs.gitlab.com/runner/install/index.html) anywhere you want.
* You may want to expose the failure / success of your code run by exiting with a code. So for example, if your code exits with anything that's not 0, it will be pronounced a failure, and from there you can monitor and even get notified about.
* Suspending the schedule can be easily done by deactivating the pipeline's schedule.
* GitLab CI/CD supports any publicly available docker image, and uses [Docker Hub](https://hub.docker.com/) by default. So either use one from there (choose carefully), or even [publish your own](https://docs.docker.com/docker-hub/publish/publish/).
* You can have more than one task in one repo - you'll need to play around with different branches, or [configure your .gitlab-ci.yml properly](https://docs.gitlab.com/ee/ci/yaml/).
* Refer to [GitLab CI/CD's extensive documentation](https://docs.gitlab.com/ee/ci/) - you can achieve really awesome things with it (especially for bigger serious projects).