Schedule jobs with systemd timers, a cron alternative
Tags: linux, devops, tutorial
Published: 2020-10-22 21:48:00

For any sysadmin, devops engineer, or general Linux enthusiast, automating the annoying/boring/difficult stuff is crucial. And task scheduling plays a key role in automation.

For scheduling jobs, the old standby is [cron](https://wiki.archlinux.org/index.php/cron). A central file (the crontab) contains the list of jobs, execution commands, and timings. Provided you can [master the schedule expressions](https://crontab.guru/), cron is a robust and elegant solution.

For Linux sysadmins there is an alternative that provides tighter integration with [systemd], intuitively named [systemd timers][systemd-timer].

> While unlikely, you may use a Linux distribution (or BSD or other Unix-like system) that does not have systemd. If you use [\*](https://www.freebsd.org/)[B](https://www.openbsd.org/)[S](https://www.netbsd.org/)[D](https://ghostbsd.org/)[,](https://www.dragonflybsd.org/) [Alpine Linux](https://alpinelinux.org/), [Gentoo](https://www.gentoo.org/), [Knoppix](http://knopper.net/knoppix/index-en.html), [Void](https://voidlinux.org/), [Tiny Core](http://tinycorelinux.net/), [Devuan](https://devuan.org/), [Artix Linux](https://artixlinux.org/), and others using a different init system other than systemd, then this article may be just a curiosity. Read on, or simply enjoy the cron you have.

## Choosing systemd timers instead of cron

If cron works, why use systemd timers? I do not believe this is a question about superiority. Both work well, and have pros and cons.

I turn to systemd timers in the following cases:

- systemd is already available (in other words, it is there so why not use it instead of installing another package)
- Time zone handling is desired (to respect daylight savings, or simply to set times as something other than UTC)
- Logging should be well integrated and accessible with `journalctl`
- Testing of the job by itself is wanted, without waiting for the schedule

On the other hand, cron may win out if you want straightforward email notifications, and you and your team are highly familiar with the tool already.

The [ArchWiki also lists some excellent benefits and caveats](https://wiki.archlinux.org/index.php/Systemd/Timers#As_a_cron_replacement) of using systemd timers over cron.

## The service

For a systemd timer, there are two files that need to be created:

1. The service that will be started
2. The timer that schedules the service

I find the nomenclature of _service_ a little confusing here. But it should be pointed out that a systemd service does not need to be a long running one. In this case, a "oneshot" service will do nicely. Even though it exits immediately, it is still called a service.

A simple example, to be installed as `/etc/systemd/system/motd-weather.service`

```ini
[Unit]
Description=Update message of the day with current weather

[Service]
ExecStart=/usr/bin/curl -o /etc/motd http://wttr.in/?1Fq
Type=oneshot
```

A few notes:

- The naming of the files is important if you favor the efficient/terse format here. As long as the service and the timer have the same name, except for the extension, they will automatically be able to find each other, without needing to explicitly reference the filenames. The extension for the service should be `.service` and the extension for the timer should be `.timer`.
- You can make `Description` whatever you would like
- `ExecStart` should be assigned to the appropriate command. Full paths to the executable are usually necessary, which is why we specify `/usr/bin/curl` here, not just `curl`.
- Assuming that the command executes then finishes, I set `Type=oneshot`. This signals to systemd that the service is not to be considered "dead" just because it finishes.

Before we install the timer, it is possible to test the service.

```console
sudo systemctl start motd-weather.service
```

Did the above change `/etc/motd`?

Excellent.

## The timer

Timers with systemd are a two-file system. We did one part, the service, and now we make a matching timer for scheduling the service.

A timer file associated with the example service above:

```ini
[Unit]
Description=Download weather to motd nightly

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target
```

If the service above is in `/etc/systemd/system/motd-weather.service`, then this file should be `/etc/systemd/system/motd-weather.timer`. Notice that the only difference is the extension: `.service` vs. `.timer` while the stem `motd-weather` is kept the same for auto-discovery.

A few notes:

- You can make `Description` whatever you would like
- For simplicity, `OnCalendar` is using `daily` to run the service at midnight. See below for more flexibility
- What if the server is shutdown or disconnected for maintenance or server/network failure? It would be a crying shame if, when the server comes back online, the message of the day has _yesterday's_ weather! So we use `Persistent=true` so that the service is triggered on next boot if it was supposed to have run in the interim offline period. Leave this line out if this is not desired, as the default is false.
- Assuming we have a bunch of timers running with `OnCalendar=daily`, we are at risk of a dogpile of services running at midnight and affecting system performance. We could change `daily` to a specific time, of course. In this instance, though, I set `RandomizedDelaySec` to 3600. Don't see the 3600 number? That is because [systemd time span abbreviations][systemd-spans] allow us to denote 3600 seconds as `1h` for obvious reasons. The end result is that systemd will randomly choose a launch time within 1 hour of midnight. If we do the same with other `daily` timers, there will be harmony and balance and we will therefore sleep better at night.
- In the `[Install]` section, we let systemd know that [the system `timers.target`](https://www.freedesktop.org/software/systemd/man/systemd.special#timers.target) `Wants` this timer. That way, upon reboot, when the `timers.target` starts, it will bring this and other associated timers online as well. That doesn't mean the associated services are triggered; rather, it just means that the timers are activated at boot. Fun fact: the `timers.target` [also works in user scoped systemd timers](https://www.freedesktop.org/software/systemd/man/systemd.special#Special%20User%20Units).

## Enable and start the timer

Assuming we have tested the service and it works as it should, we are ready to start the timer. To do so:

```console
sudo systemctl enable --now motd-weather.timer
```

Notice that we enable and start the _timer_, and the timer then calls the service when scheduled. We do not start the service directly.

If it installed correctly, you should see it and some scheduling information when listing the systemd timers:

```console
systemctl list-timers
```

If the list is long, you can filter with wildcards:

```console
systemctl list-timers motd*
```

## Calendar expressions

One component of systemd timers worth some ongoing study (or at least a browser bookmark) is what `OnCalendar` is set to: [the calendar expression][systemd-calendar].

A few examples that may help introduce the syntax:

- UTC midnight on the first day of every year: `*-01-01 00:00:00 UTC`
- Midnight in your timezone on the first day of every year: `*-01-01 00:00:00` (this could also be written `yearly`)
- 8am daily on the U.S. East Coast: `*-*-* 08:00:00 America/New_York`
- Yeah, you can leave the off the seconds: `*-*-* 08:00 America/New_York`
- Just weekdays at 2am: `Mon..Fri *-*-* 02:00 America/New_York`
- Every Sunday at 10pm: `Sun *-*-* 22:00 America/New_York`

As seen above, `*` is used to mean "every." Sometimes, to remind myself of the format, I run `systemd-analyze timestamp now` to see the normalized format for this current second, then start substituting `*` in the right places, changing the date, time, and timezone as appropriate. As soon as you start substituting with `*`, `systemd-analyze timestamp` no longer works; instead, use `systemd-analyze calendar` (see below).

You may also use these shorthand expressions: `minutely`, `hourly`, `daily`, `monthly`, `weekly`, `yearly`, `quarterly`, or `semiannually`.

> To see a list of possible timezones, try `timedatectl list-timezones`

### [`systemd-analyze calendar`][systemd-analyze] is your friend

You can test any of the above [using `systemd-analyze calendar`][systemd-analyze]. For instance, I want my service to run every Monday, Wednesday, and Friday at 11pm UTC, but not in December. Did I get it right? Let's check the validity of the syntax.

```console
systemd-analyze calendar "Mon,Wed,Fri *-1..11-* 23:00 UTC"
```

Eureka! It checked out OK. I can be happy, but I can also learn from the normalized form, and tweak the above to be `Mon,Wed,Fri *-01..11-* 23:00:00 UTC` instead.

One very useful sanity check: `systemd-analyze calendar` can show several of the next iterations, just to reassure you and help you think through when things will happen. For instance, I want the service to launch the first and third Wednesday of every month. That takes a bit more complexity than some other examples, so I want to be sure I have it right. The following will show me a year's worth (24 occurrences) of such events.

```console
systemd-analyze calendar --iterations=24 "Wed *-*-1..07,15..21 02:00"
```

After painstakingly looking at all 24 while perusing a desk calendar, everything checks out OK. Oh, but wait, I wanted to see the _calendar_ year, not just a year from now. For that, we can use the `--base-time` option, and pick January 1 of the desired year. How about 2026:

```console
systemd-analyze calendar --base-time="2026-01-01" --iterations=24 "Wed *-*-1..07,15..21 02:00"
```

Once your calendar expression checks out OK, set `OnCalendar=` to your chosen expression, in the timer file ending in `.timer`

## Countdown timers

A systemd timer does not have to have a single or repeated calendar event. In other words, there are options other than `OnCalendar=`.

These include:

- `OnActiveSec=` triggers service the specified time after the timer is activated
- `OnBootSec=` and `OnStartupSec=` are roughly equivalent, and I would tend to use `OnStartupSec=` for its flexibility. `OnBootSec=` refers to time since system boot, and `OnStartupSec=` refers to time since service manager startup. As these are very close, I favor the latter as it also applies to user-scoped services that may only be started after login.
- `OnUnitActiveSec=` and `OnUnitInactiveSec=` are interesting. They trigger the service the specified time after the service was last activated or deactivated, respectively.

Again, you may wish to use [systemd time span abbreviations][systemd-spans], so you can give the time in hours or days, if seconds seems to lack readability.

## User service manager

A systemd timer and service do not need to be installed in `/etc/systemd/system/` and therefore run at the system level. Instead, they can be installed per user, and run within the user service manager, usually launched upon login. [The ArchWiki has a great article about systemd user units](https://wiki.archlinux.org/index.php/Systemd/User), and the [official systemd unit guide](https://www.freedesktop.org/software/systemd/man/systemd.unit) is a good reference.

The timer and service above will work fine when installed in `~/.config/systemd/user/`, but of course the service would be unable to write to `/etc/motd`. Something like `ExecStart=/usr/bin/curl -o %E/motd http://wttr.in/?1Fq` would work better, but would also require something like `cat ~/.config/motd` at the end of your `.bashrc` or other shell script executed at login. Like that `%E` to refer to `$XDG_CONFIG_HOME` (usually `~/.config`)? See [the list of variables available in systemd unit files](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers).

The one big caveat with user services: they don't necessarily run at boot. Instead, they run at login. There is a nice workaround, though. If you want your user services and timers to run at boot, not just login, you can make a particular user ["linger"](https://www.freedesktop.org/software/systemd/man/loginctl.html#enable-linger%20USER%E2%80%A6). Then things work even when the user has not explicitly logged in. To do this:

```console
sudo loginctl enable-linger my_username
```

And substitute `my_username` with the username you want to linger after reboot.

## The docs

You may enjoy reading systemd's documentation regarding timers and units:

- [systemd]
- [systemd timers][systemd-timer]
- [systemd units](https://www.freedesktop.org/software/systemd/man/systemd.unit)
- [systemd time][systemd-time]
- [systemd-analyze]

Please feel free to post ideas, advice, and questions in the comments!

[systemd]: https://www.freedesktop.org/wiki/Software/systemd/
[systemd-timer]: https://www.freedesktop.org/software/systemd/man/systemd.timer
[systemd-time]: https://www.freedesktop.org/software/systemd/man/systemd.time
[systemd-timestamp]: https://www.freedesktop.org/software/systemd/man/systemd.time#Parsing%20Timestamps
[systemd-calendar]: https://www.freedesktop.org/software/systemd/man/systemd.time#Calendar%20Events
[systemd-spans]: https://www.freedesktop.org/software/systemd/man/systemd.time#Parsing%20Time%20Spans
[systemd-analyze]: https://www.freedesktop.org/software/systemd/man/systemd-analyze