The Walking Dead (but with processes)
Tags: linux, systems, init, kernel
Published: 2017-06-13 16:47:10

---
title: The Walking Dead (but with processes)
published: true
description: A deep dive into zombies, init, and what can go wrong.
tags: linux, systems, init, kernel
cover_image: 
---

## What's a zombie?

![](http://turnoff.us/image/en/zombies.png)

When interviewing system engineers a common answer for "what is a zombie process?" is  "A process which is dead and doesn't have a parent" or something like "you can kill it with `kill -9 <pid>`". There seems to be a misunderstanding of zombie processes. Let's delve into what exactly a zombie is, and how they *should* be handled by Linux.

A zombie process is a process which has `exit()`'d and whose parent has not called `wait()/waitpid()` [syscall](https://linux.die.net/man/2/waitpid) against its process id (PID). Meaning, the process exited and left a status code for the parent to read, but the parent has yet to read it from the process table. Every process that terminates is briefly a zombie, they become an issue when they stick around too long.

Some key points:

- They cannot be "killed" - SIGKILL, SIGTERM are ineffective as the process is already dead.
- They are not orphan processes.
- They can be removed via orphaning where init adopts the process (by killing the parent).
- They utilize no resources and simply occupy a process table entry.
- Every process, upon termination, is a zombie until their parent process calls `wait()`.

This little snippet demonstrates how a zombie can be created:

```c
int main(void)
{
    pid_t pid;
    int status;

    if ((pid = fork()) < 0) {
        // Failed to fork
        exit(1);
    }
    // Child process starts execution from here, child pid == 0
    if (pid == 0)
        // All the child does is exit
        exit(0);
    
    // Parent continues from here. A zombie roams for 100 seconds..brains..
    sleep(100);

    // After 100 seconds, the parent removes the zombie by calling wait()
    pid = wait(&status);
   
    return 0;
}
```

A zombie shows up as `<defunct>` in the output of `ps`, when running the above snippet.

```
129274  \_ ./zombie 
129275    \_ [zombie] <defunct> 
```

This means the parent (PID 129274) hasn't called `wait()/waitpid()` on PID 129275. Until it calls wait, 129275 will eat up a process table entry. Since they take up no resources, they typically aren't an issue. 

The problem emerges when too many exist and you run out of PIDs, meaning no more new processes (ex: can't `ssh` to a server that can't spawn you a shell..)

You can see the max PIDs with the following command. 

```
# cat /proc/sys/kernel/pid_max
131072
```

####It's a good idea to monitor process count on your machines. If you have a process leaking zombies that is long-running it will eventually fill the process table which could take down your service, prevent remote-access, etc.

A way you can remove a zombie is by killing the parent of the zombie, which would then re-parent the process to init (PID 1) who *should* periodically call `wait()` on its child processes.

Zombies usually appear if a program has a flaw where it doesn't call `wait()` on its children. what if the process got stuck in some infinite loop and never called wait()? what if that process was named init? 

## When init doesn't do its job...

A while ago I came across a number of servers that had thousands of zombie processes. My team had simply been rebooting these boxes to clear them up, but I became curious after noticing something odd: 

####All the zombie processes had init as a parent!

This means that processes were exiting, and init (as their parent), was never calling `wait()`. Additionally, if you were to kill a zombies parent to make init the parent, init would do nothing to help you.
 
init, as one of its core functions, should be calling `wait()` on its child processes to clear out any zombies, so what was happening?

```
% sudo strace -p 1 -r -s 500
Process 1 attached - interrupt to quit
     0.000000 write(8, "init: serial-ttyS main process ended, respawning\r\n", 50) = ? ERESTARTSYS (To be restarted)
   273.565117 --- SIGCHLD (Child exited) @ 0 (0) ---
     0.000062 write(4, "\0", 1)         = 1
     0.000066 rt_sigreturn(0x4)         = 1
     0.000046 write(8, "init: serial-ttyS main process ended, respawning\r\n", 50) = ? ERESTARTSYS (To be restarted)
     0.003033 --- SIGCHLD (Child exited) @ 0 (0) ---
     0.000026 write(4, "\0", 1)         = 1
     0.000048 rt_sigreturn(0x4)         = 1
     0.000039 write(8, "init: serial-ttyS main process ended, respawning\r\n", 50) = ? ERESTARTSYS (To be restarted)
     7.825845 --- SIGCHLD (Child exited) @ 0 (0) ---
```

stracing showed init stuck in a loop trying to write to a tty device, and each write getting `ERESTARTSYS` back (meaning: please re-attempt that write). Init had no mechanism to handle that error, so it gets stuck in an infinite loop.

As for why it was getting `ERESTARTSYS`, the tty it was writing to in this case was a serial tty. grepping the serial tty driver code for `ERESTARTSYS` found:

```c
/**
 *	tty_send_xchar	-	send priority character
 *
 *	Send a high priority character to the tty even if stopped
 *
 *	Locking: none for xchar method, write ordering for write method.
 */

int tty_send_xchar(struct tty_struct *tty, char ch)
{
	int	was_stopped = tty->stopped;

	if (tty->ops->send_xchar) {
		tty->ops->send_xchar(tty, ch);
		return 0;
	}

	if (tty_write_lock(tty, 0) < 0)
		return -ERESTARTSYS;

	if (was_stopped)
		start_tty(tty);
	tty->ops->write(tty, &ch, 1);
	if (was_stopped)
		stop_tty(tty);
	tty_write_unlock(tty);
	return 0;
}
```

Writing to a serial tty takes a lock in the form of `atomic_write_lock`. Searching for info regarding this lock I found a [bug](http://bit.ly/2rOkSAE): 

```
>>  Possible unsafe locking scenario:
>>
>>        CPU0                    CPU1
>>        ----                    ----
>>   lock(&tty->termios_rwsem);
>>                                lock(&tty->atomic_write_lock);
>>                                lock(&tty->termios_rwsem);
>>   lock(&tty->atomic_write_lock);
```

Wrapping all this back together found a new bug due to a different bug. 

If the serial tty gets into a deadlock, any logging that init tries to perform against it will get init stuck into a loop of `write -> ERESTARTSYS -> write` ...

If init stays in this state long enough, without the tty resetting, then zombies will pile up and you'll run out of PIDs.

## How should init handle child processes?

Looking at the sysvinit source in src/init.c we see a couple different ways init handles it child processes. Here we see init establishing a signal handler for `SIGCHLD`

```c
SETSIG(sa, SIGCHLD,  chld_handler, SA_RESTART);

void chld_handler(int sig)
        CHILD           *ch;
        int             pid, st;
        int             saved_errno = errno;

        /*
         *      Find out which process(es) this was (were)
         */
        while((pid = waitpid(-1, &st, WNOHANG)) != 0) {
                if (errno == ECHILD) break;
                for( ch = family; ch; ch = ch->next )
                        if ( ch->pid == pid && (ch->flags & RUNNING) ) {
                                INITDBG(L_VB,
                                        "chld_handler: marked %d as zombie",
                                        ch->pid);
                                ADDSET(got_signals, SIGCHLD);
                                ch->exstat = st;
                                ch->flags |= ZOMBIE;
                                if (ch->new) {
                                        ch->new->exstat = st;
                                        ch->new->flags |= ZOMBIE;
                                }
                                break;
                        }
                if (ch == NULL) {
                        INITDBG(L_VB, "chld_handler: unknown child %d exited.",
                                pid);
                }
        }
```

Examining `chld_handler`, this will execute if init gets a `SIGCHLD` signal which is sent to a parent when a child of theirs dies. Init handles zombies in a push model here by calling `waitpid()` and flagging the process as a zombie for later cleanup

`ch->flags |= ZOMBIE;`

Earlier, I said when init gets stuck in this state that it would not be able to reap any of its own children who exit, but it has a signal handler for this? so what's going on?

The `initlog()` function blocks all signals while logging:

```
/*
 *      Re-establish connection with syslogd every time.
 *      Block signals while talking to syslog.
 */
sigfillset(&nmask);
sigprocmask(SIG_BLOCK, &nmask, &omask);
openlog("init", 0, LOG_DAEMON);
syslog(LOG_INFO, "%s", buf);
closelog();
sigprocmask(SIG_SETMASK, &omask, NULL);
```

So when we call initlog() anywhere in the main loop, and call `syslog(LOG_INFO, "%s", buf);` we hit our earlier bug. `syslog()` is respecting `ERESTARTSYS` from `write()` so we get stuck in here and we block all signals (including `SIGCHLD`).

Outside of the `chld_handler`'s mechanism for reaping zombies, the rest of the main loop handles a variable called `family` which stores all the child processes of init. It loops over these looking for processes to kill and/or reap from the process table if they have died.