Systemd And Timing Issues
Tags: linux, annoyances, systemd, legacyinit
Published: 2018-08-17 02:39:57

---
title: Systemd And Timing Issues
published: true
tags: linux,annoyances,systemd,legacy-init
canonical_url: https://thjones2.blogspot.com/2018/08/systemd-and-timing-issues.html
---

Unlike a lot of Linux people, I'm not a knee-jerk hater of `systemd`. My "salaried UNIX" background, up through 2008, was primarily with OSes like Solaris and AIX. With Solaris, in particular, I was used to `sytemd`-type init-systems due to SMF.  
  
That said, making the switch from RHEL and CentOS 6 to RHEL and CentOS 7 hasn't been without its issues. The change from `upstart` to `systemd` is a lot more dramatic than from `SysV-init` to `upstart`.  
  
Much of the pain with `systemd` comes with COTS software originally written to work on EL6. Some vendors really only due fairly cursory testing before saying something is EL7 compatible. Many &mdash; especially earlier in the EL 7 lifecycle &mdash; didn't bother creating `systemd` services at all. They simply relied on `systemd-sysv-generator` utility to do the dirty work for them.  
  
While the `systemd-sysv-generator` utility does a fairly decent job, one of the places it can fall down is if the legacy-init script (files hosted in `/etc/rc.d/init.d`) is actually a symbolic link to someplace else in the filesystem. Even then, it's not super much a problem if "someplace else" is still within the "`/`" filesystem. However, if your SOPs include "segregate OS and application onto different filesystems", then "someplace else" can very much be a problem &mdash; when "someplace else" is on a different filesystem from "`/`".  
  
Recently, I was asked to automate the installation of some COTS software with the "it works on EL6 so it ought to work on EL7" type of "compatibility". Not only did the software not come with `systemd` service files, its legacy-init files linked out to software installed in `/opt`. Our shop's SOPs are of the "applications on their own filesystems" variety. Thus, the `/opt/<APPLICATION>` directory is actually its own filesystem hosted on its own storage device. After doing the installation, I'd reboot the system. ...And when the system came back, even though there was a boot script in `/etc/rc.d/init.d`, the service wasn't starting. Poring over the logs, I eventually found:

 ```
 systemd-sysv-generator[NNN]: stat() failed on /etc/rc.d/init.d/<script\_name>
 No such file or directory
 ```

This struck me odd given that the link and its destination very much _did_ exist.  
  
Turns out, `systemd` invokes the `systemd-sysv-generator` utility very early in the system-initialization proces. It invokes it _so_ early, in fact, that the /opt/<APPLICATION>  filesystem has yet to be mounted when it runs. Thus, _when it's looking to do the conversion_ the file the sym-link points to actually does _not_ yet exist.  
  
My first thought was, "screw it: I'll just write a `systemd` service file for the stupid application." Unfortunately, the application's starter was kind of a rats nest of suck and fail; complete and utter lossage. Trying to invoke it from directly via a `systemd` service definition resulted in the application's packaged controller-process not knowing where to find a stupid number of its sub-components. Brittle. So, I searched for other alternatives...  
  
Eventually, my searches led me to both the nugget about when `systemd` invokes the `systemd-sysv-generator` utility and [how to overcome](https://access.redhat.com/solutions/3094591) the "sym-link to a yet-to-be-mounted-filesystem" problem. Under `systemd`-enabled systems, there's a new-with-`systemd` mount-option you can place in `/etc/fstab` &mdash; `x-initrd.mount`. You also need to make sure that your filesystem's `fs_passno` is set to `0` ...and if your filesystem lives on an LVM2 volume, you need to update your GRUB2 config to ensure that the LVM gets onlined prior to `systemd` invoking the `systemd-sysv-generator` utility. Fugly.  
  
At any rate, once I implemented this fix, the `systemd-sysv-generator` utility became happy with the sym-linked legacy-init script ...And my vendor's crappy application was happy to restart on reboot.  
  
Given that I'm deploying on AWS, I was able to accommodate setting these fstab options by doing:  

 ````
 mounts:
   - ["/dev/nvme1n1", "/opt/<application> , "auto", "defaults,x-initrd.mount", "0", "0"]
 ```

Within my `cloud-init` declaration-block. This should work in any context that allows you to use `cloud-init`.

I wish I could say that this was the worst problem I've run into with this particular application. But, really, this application is an all around steaming pile of technology.