Following slippery processes
Tags: go, linux, systemsprogramming, showdev
Published: 2021-01-07 13:17:32

---
title: Following slippery processes
published: true
description: Some processes do fork and we may lose them. ¿How can we follow and wait for them to finish?
tags:
  - go
  - linux
  - systemsprogramming
  - showdev
series: Writing init with go
---

Hey, if you followed the previous parts about writing a PID 1 (and a service launcher) with Go you may expect this.
The approach of the created `service-launcher` is to follow a **single** process, so if it forks and main process exits, the service fails/succeeds with code from main process instead of following the forked one.

To follow those processes (that can spawn several processes) we should tweak the `service-launcher` to follow a variable number of them.

Instead of doing this and thus, making a monolith around `service-launcher` I have decided to create a helper to launch and follow processes that fork.

> Important idea: What do we call "main" process?
> We call main process to the process spawned by `service-launcher`. That process will be monitored and will decide if a service is running or exited (failed/succeeded).

# Default behavior
From `fork(2)`:
> `fork()` creates a new process by duplicating the calling process. The new process is referred to as the child process. The calling process is referred to as the parent process.

So if we call a `fork`-ing process and do pstree, there will be 2 identical processes, one as parent and the other as child.

Take the following C code:
```c
# include <unistd.h>
# include <sys/types.h>
# include <sys/wait.h>

int main() {
    pid_t pid=fork();
    switch(pid){
        case -1:
            return -1;
        case 0:
            // Forked process. Sleep and exit
            sleep(20);
            return 0;
    }

    wait(NULL); // Wait for its child

    return 0;
}
```

If we run `htop` or `pstree` while running that code we get the following tree:
![Process tree, 2 fork executables](https://dev-to-uploads.s3.amazonaws.com/i/pd55dpccmx8e7kmh8nf6.png)

Ok, that's right, but, what if the parent process exits before child? To achieve this, remove the `wait` call from C code. In sake of brevity I won't show it, but that "forked" process bubbles trying to find the next CHILD_SUBREAPER. The PID 1 acts as child reaper implicitly, so if none of the processes on tree has that role, PID 1 will become the parent of our orphaned process. We lost it.

This is the default behavior, and the process that launched `./fork` cannot follow the `fork(2)`-ed process.

# Having control over them
So we want a program able to follow **all** its descendent processes.

Thanks Linux for `cgroups(7)`. We'll be using the version 2 (unified hierarchy).

## How do `cgroups2` work
First, you need `cgroup2` mounted on `/sys/fs/cgroup`. `service-launcher` does mount this filesystem by default on init.

Now, simply cd to `/sys/fs/cgroup` and `mkdir`. `cd` into that dir and surprise! All the enabled controllers are present just after you did `mkdir`. Add your current shell PID to the cgroup `echo $$ >> cgroup.procs`. You can also `cat` that file to see all PIDs inside.

Some rules that apply to "adding processes to cgroups":
* If the added process already has child processes, child processes are not moved inside the cgroup, only the specified PID.
* Processes and their threads are added into the cgroup.
* Child processes created by a process **inside** the cgroup, are also added automatically to the cgroup (and their PID to `cgroup.procs`)

If we take this rules we can make a program that:
* Creates a new cgroup with an specified name
* Executes another process inheriting an fd ( with options `O_APPEND|O_WRONLY`) to the `cgroup.procs` file. Arguments are to be passed to the real executable process. This child process does:
  * Append it's PID to the fd 3 (first fd inherited)
  * Closes the fd
  * Calls `execve(2)` (`golang.org/x/sys/unix.Exec` in Go) with arguments from cmdline.
* Child process is followed until it `fork(2)`s and exits.
* Now their children (from `cgroup.procs`) are watched.
* Exits with status from last process in cgroup
* Removes cgroup before exiting

How do we "watch". First thing I thought is to create a ticker and `unix.Wait4` with `WNOHANG` but then came across a section on `cgroups(7)` talking about the `cgroup.events` file. So we could use `inotify(7)` to watch them for changes and react to process changes inside the cgroup.

Ok, this works, but what about the process tree? It still holds the same default behavior explained previously.

## Changing the default behavior
We simply call `prctl(2)` with `PR_SET_CHILD_SUBREAPER` to become a subreaper on the current tree so orphaned child processes are attached to us.

But with great power comes great responsibility. If this process becomes a subreaper, it must `wait(2)` child processes when appended. To achieve this we listen to our favourite signal, `SIGCHLD`, wait processes and check if cgroup still contains at least 1 process. If cgroup is empty, our exit code will be the code from last process inside the cgroup.

# Code & Run

* Main process (supervisor). [`cmd/run-in-cgroup`](https://gitlab.com/mrvik/go-pid1/-/tree/master/cmd/run-in-cgroup)
* Subprocess (executor). [`cmd/exec-in-cgroup`](https://gitlab.com/mrvik/go-pid1/-/blob/master/cmd/exec-into-cgroup/main.go)

Let's run `dhclient` on `eth0`. This process forks and previously the service succeeded when main process exited, leaving the forked process untracked.

We changed the service file as follows:

```yaml
---
name: dhclient@eth0
description: DHCP client for eth0
exec: run-in-cgroup
arguments:
  - -name=dhclient-eth0
  - --
  - dhclient
  - eth0
```

The result looks like this:
![pstree with forked dhclient](https://dev-to-uploads.s3.amazonaws.com/i/5qtib25zf7zite1hlymi.png)

So `service-launcher` detects dhclient as running because the monitor process is alive and trackable. Also `dhclient` is a child of its monitor process so the tree is clean and there are no lost things floating around.

Do you have any thoughts here? Feedback? Can we make this better? Write a comment or feel free to create an issue.

Help is needed with the name, I don't feel good calling this thing `go-pid1` (and my lack of creativity is notable, I thought `gp1` 😅).

On next article I'll be making it work in a complete system with Void + SDDM + KDE (or something simpler, but KDE should work).

See you next time.