UNIX-like processes and the imaginary disco ball

May 2018

Earlier this week I found myself discussing UNIX process states with my colleague Michael. When it came to orphaned processes and daemonizing I vaguely remembered that I had to fork() twice to get a daemon process. That statement was immediately challenged by Michael. Rightfully so, because forking twice is not the key action here. The missing piece was to obtain a new session between the first and the second forking. This ensures a properly daemonized process can not obtain a terminal again. 🤯

But let’s slow down a little. What is all this forking and sessions that I am talking about? Time for a quick refresher in operating system internals.

That is a wonderful chance to write some good ol’ C code! Why C? Because it is the right language for the job. Most operating system kernels are written in C (and assembler). Furthermore, according to my other colleague Robert (†), the moment you start writing C “the light dims, an imaginary disco ball lowers from the ceiling, and an encouraging atmosphere is created”. I could not have described the coding C feeling better!

I prepared a couple of Docker images to make it easier to follow the article. Please find the source code at Github in the process-fun repository. Small, ready to run images are available on Dockerhub in the (surprise!) process-fun repository. Run the images on your local Docker-enabled machine as you like.

A Natural Process State

Processes don’t exist just for fun. They are there to get work done, play music, mine crypto coins, train a neural net, or send spam emails. A process’ natural state is running or runnable. That means everything is more or less in order and the Kernel may grant some computing time to the process. Writing a program that does something meaningful is hard. Writing a program that does just something is much easier. Let’s stick with easy. 😉

#include <time.h>

int main(int argc, char *argv[]) {
    // run for 10 seconds
    time_t end = time(0) + 10;

    // do something
    volatile int i;
    while (time(0) < end) {
        i++;
    }

    // exit ok
    return 0;
}

This program just runs for ten seconds, incrementing an integer to make sure some computing time is wasted. Let’s run it and see what the state of the corresponding process is:

$ docker run danrl/process-fun:running
✂️
Starting process-fun... done!
Process list:
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
    7     1     1     1 R    process-fun
    9     1     1     1 R    ps

We find the process state in the STAT column. Little surprise here, it is R which stands for running or runnable. Boring!

Just in case you are wondering: The tool used to generate the process list is ps but with a custom output format.

Sleeping And Waiting

A wise person once told me that the best things in life are worth waiting for. I don’t know if that holds true for the following piece of code. 🤔

#include <unistd.h>

int main(int argc, char *argv[]) {
    // wait
    sleep(10);

    // exit ok
    return 0;
}

This program just sleeps for ten seconds. Let’s run it and see what the state of the corresponding process is:

$ docker run danrl/process-fun:waiting
✂️
Starting process-fun... done!
Process list:
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
    6     1     1     1 S    process-fun
    8     1     1     1 R    ps

Again, expected result, the process is in the interruptible sleep state which is indicated by S. Still pretty boring, right? Let’s awake the undead, that should be more fun! 🧟‍♀️🧟‍♂️

Bad Parenting

Once ready and started, we expect a process to be either doing something, waiting for something, being stopped (e.g. for debugging), or terminated. But there is another state: The defunct or zombie state. This happens when a child process, that was forked off from a parent process, has been terminated but the parent process has not yet collected the return state. The return state of a child can be collected using the wait() call (we call that reaping). However, a parent process may be busy doing something else or just decided not to reap. In that case, the child process, although not alive, still has a process control block maintained by the Kernel. The process is neither dead nor alive. A true zombie!

Here are a few lines of code that create a zombie process:

#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char *argv[]) {
    pid_t pid = fork();
    if (pid == 0) {
        // child exits immediately
        return 0;
    }
    // parent sleeping, not eager to call wait()
    sleep(10);

    // yeah, maybe now...
    wait(NULL);

    // exit ok
    return 0;
}

This program forks and immediately exists the child process. The parent process is lazy, it sleeps for a couple of seconds before it bothers to reap the child. During that timeframe, in between the child’s death and the parent’s call to wait() we run ps:

$ docker run danrl/process-fun:zombie
✂️
Starting process-fun... done!
Process list:
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
    7     1     1     1 S    process-fun
    9     7     1     1 Z    process-fun <defunct>
   10     1     1     1 R    ps

The child process is in the defunct or zombie state. In the PID column, we can see the unique process identifier (PID). The PPID column shows us the parent’s process identifier (PPID) respectively. The output of ps confirms that process number 9 is a zombie child of process number 7. Once process number 7 calls wait() or exits, the zombie child will be removed from the process list.

The Forgotten Child

Although not waiting for a child process may be considered bad parenting, it is not the worst that can happen to a process. What happens if we forked off a child process and then the parent process terminates but the child is still out there?

Here is the code for that:

#include <unistd.h>
#include <sys/types.h>

int main(int argc, char *argv[]) {
    pid_t pid = fork();
    if (pid == 0) {
        // child being lazy
        sleep(30);
        return 0;
    }
    // parent exiting without waiting for the child
    sleep(5);
    return 0;
}

After about five seconds the parent process exits. The child process becomes orphaned, meaning it has no valid parent process identifier anymore. Orphaned processes become foster children of the process with the PID 1 which is often the init process. In our container, the PID 1 process is the state.sh shell script (the CMD configured in the Dockerfile).

$ docker run danrl/process-fun:orphaned
✂️
Starting process-fun... done!
Process list (before orphaned):
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
    6     1     1     1 S    process-fun
    8     6     1     1 S    process-fun
    9     1     1     1 R    ps

The parent process has PID 6 and the child process is identified by PID 8. The child’s parent is, therefore, the process with PID 6 (see column PPID). Once the parent process terminates the situation changes:

Process list (after orphaned):
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
    8     1     1     1 S    process-fun
   11     1     1     1 R    ps

Now the process with PID 6 is gone and the child process is assigned to a new parent process. The PPID now reads 1.

Summoning The Daemon

In the previous example, the child process was successfully detached from the parent process. But it was still part of the same (terminal) session. This means the process could theoretically attach to the terminal session again. We call a process a daemon process when it can not attach to a terminal session. This means it has to migrate to a new session to be fully detached from the parent process and the parent process’ session.

Let’s have a look how we can make that happen:

#include <unistd.h>
#include <sys/types.h>

int main(int argc, char *argv[]) {
    pid_t pid = fork();
    if (pid == 0) {
        // child 1 migrates to a new session
        setsid();
        pid_t pid2 = fork();
        if (pid2 == 0) {
            // child 2 (daemon) being lazy
            sleep(30);
            return 0;
        }
        // child 1 exiting ok
        return 0;
    }
    // parent exiting without waiting for the child
    sleep(5);
    return 0;
}

First, we fork off a child process and exit the parent. This orphans the child process and its new parent will be the init process. In the child, we request to be assigned to a new session. By doing so, we become the session leader. We don’t want a daemon to be a session leader, because we do not want a daemon to be able to attach to a terminal. The second child (the child of the first child) will finally belong to the new session which is different from the (terminal) session we originally ran the parent process from.

$ docker run danrl/process-fun:daemonized
✂️
Starting process-fun... done!
Process list (before daemonizing):
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
    7     1     1     1 S    process-fun
    9     7     9     9 Zs   process-fun <defunct>
   10     1     9     9 S    process-fun
   11     1     1     1 R    ps

Here we see all three processes at once:

PID 7: The parent process.
PID 9: The first child. It is a zombie process since the parent is still running. It is the session leader of the new session. This is indicated by the lowercase s in the STAT column.
PID 10: The second child and the new daemonized process.

After waiting for the parent and the first child to terminate, the process table looks like this:

Process list (after daemonizing):
  PID  PPID  PGID  SESS STAT COMMAND
    1     0     1     1 Ss   state.sh
   10     1     9     9 S    process-fun
   13     1     1     1 R    ps

The process with PID 10 is now a true daemon. 😈

And this is why we fork twice. Mystery solved!

A Natural Process State

Sleeping And Waiting

Bad Parenting

The Forgotten Child

Summoning The Daemon

Further Reading