Wednesday, April 6, 2016

linux process number shenanigans

How do you check how many processes are present on your Linux system? Will ps aux do the trick? Is this really the thing you would want to check? Let's see.

A process is a program being executed. It has several properties, like security credentials (uid, gid, labels etc.), address space and of course a PID.
Each process has at least one thread. Threads are what is executing the code, and so threads assigned to one process share a lot of properties (the program being executed, the address space etc.).

On FreeBSD there is a struct proc with all relevant properties. Then there is struct thread and the process has the list of threads. Pretty straightforward.

Historically threads were implemented as processes just sharing address space and the like. Basics of this model survived in Linux to this very day. It does not have a separate 'process object', everything is stuffed into the thread.

This means that Linux threads belonging to one process have separate PIDs, just as if they were completely separate processes. They do share a 'thread group' which is how their relationship is maintained. There is also a designated thread acting as the 'thread group leader'.

As such, things are blurred a little bit and any mentions of 'threads' or 'processes' have to be examined closely. Let's examine few things which happen in practice.

The kernel provides a special sysctl kernel.threads-max. Unsurprisingly it's an upper limit for the number of threads we can have in the system. We also get kernel.pid_max, which is not the upper limit of processes, although it may act like one in some cases. There is no explicit limit of processes.

Since each process has to have at lest one thread, creation of a new process adds a thread. So we are not going to have more processes than threads-max. Further, each thread has to have a PID, so we are not going to have more than pid_max threads either. But if we are already limited, what's the use for pid_max? It is used to provide the range of valid PIDs, so that it will take more time to reuse them (e.g. with threads limited to 32k and pid_max bumped from 32k to 64k the kernel has another set of pids before it wraps).

How about RLIMIT_NPROC (ulimit -u)? It limits threads, which has a side effect of limiting processes.

With this in mind, let's go back to the initial question.

How many processes are present? Well, turns out you typically don't want to ask this question as it has little relevance. Instead you want to check how many threads are present. The information is provided in /proc/loadavg file as the divider in the second to last field. That is, if the file contains "4.37 3.98 3.55 9/762 23384", the number of threads is 762.

But let's say we really want to know how many processes are present.

The primary source of information is the proc filesystem mounted on /proc.
Listing the content typically reveals plenty of directories with the name being a number. Each entry is either a thread group leader (so it can serve as a representation of a process) or a kernel thread. How to spot kernel threads? This one is fortunately quite easy - everything with a parent pid of 2 and the pid 2 itself.