How traditional resource limits are handled

 For the purpose of this document we will define the following:

- resource limits as mentioned earlier will be referred to as rlimits

- we will have an unprivileged user joe

You most likely have set rlimits at some point, either by editing

/etc/security/limits.conf (or some other file), playing with ulimit etc.

But how and when such limits are meaningful?

You may have heard about RLIMIT_NPROC, which is supposed to limit the amount of processes given user is allowed to have. Yet, it is possible you will configure this limit and the user in question will have twice as many processes. Not only that, he may still be able to spawn more. What is going on here?

Statements which follow are true enough and sufficient to understand points I'm trying to make.

Process actions are subject to rlimits.

Resource limit is a property of a process, not a user.

Processes are owned by <uid>:<gid>, which are just some numerical values.

If there is a resource limit covering more than one process, it does so by looking at the uid.

Creation of a new process is accomplished with clone() systemcall. Provided

with appropriate arguments it will create a copy of the executing process.

There are some differences (pid, parent pid) but pretty much all the rest is

identical. This includes rlimits.

Now let's say we have our custom program running as root and we want to extend it so that it runs stuff as an unprivileged user.

In order to have a process running with given uid:gid, we need to use setuid() and setgid() systemcalls whose names are self explanatory (strictly speaking one can also run a suid/sgid binary, but that's irrelevant to the subject in question).

Let me reiterate: creating a process owned by given user looks as follows:

clone();

setgid();

setuid();

So... which one of these applies security limits as defined in /etc/security/limits.conf?

That's right, NONE.

This new process owned by joe has the same rlimits its parent does.

Rlimits from limits.conf etc. can be applied by additional code (typically a PAM module).

The point is, this has to be done separately and prior to {u,g}id change.

sshd, su and so on do this stuff.

With that in mind let's consider some problems.

Let's assume joe has 80 running processes and all of them have applied nproc limit to 100.

1. An administrator modified limits.conf:

joe hard nproc  50

1.1 Is it possible for joe to spawn a new process?

Yes, currently running processes have 100 as their limit and only 80 are running.

Thus rlimits are not in the way here.

1.2 Is it possible for joe to log in over ssh?

No. That would-be joe's process would have the limit set to 50, but there are

already 80 processes running, thus it would error out.

1.3. A custom daemon is running, it spawns 200 processes owned by joe, but does not apply his resource limits. Consider processes from 1.1. Can any of them clone()?

No. There are 280 joe's processes running, but ones from 1.1 have the limit set to 100.