Fun fact: transient ETXTBSY when trying to open a script on Linux

When you run given binary, the kernel marks it as being executed which is then used to disallow opening it for writing. Similarly, when a file is opened for writing, it is marked as such and execve fails.

The error is ETXTBSY or Text file busy.

Situation with scripts is a little bit more involved. Doing "sh script.sh" will typically result in execve of /bin/sh, i.e. the kernel does not really know nor care what script.sh is.

Let's consider a file with the following content being passed to execve:
#!/bin/sh

echo meh
exit 0

A special handler recognizes #! and proceeds to change the executed binary to /bin/sh.

However, once execution gets going, you can open the file for writing no problem.

This poses two questions:
- will the execution fail if the script is opened for writing?
- will opening the file for writing ever fail because the script is being executed?

Let's experiment. The following program will help:

#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int
main(int argc, char **argv)
{
        int fd;

        if (argc != 2)
                return 1;

        for (;;) {
                fd = open(argv[1], O_WRONLY);
                if (fd != -1) {
                        close(fd);
                        continue;
                }
                perror("open");
        }
       
        return 1;
}

As you can see the program just repeatedly tries to open the file for writing.
We will run the script ("script.sh") in one terminal, while running the program in another. That is:

shell1$ ./write script.sh

shell2$ while true; do ./script.sh; done

And this gives....

shell1$ ./write script.sh
open: Text file busy
open: Text file busy
open: Text file busy
open: Text file busy
open: Text file busy
open: Text file busy
[snip]

shell2$ while true; do ./script.sh; done
zsh: text file busy: ./script.sh
zsh: text file busy: ./script.sh
meh
meh
zsh: text file busy: ./script.sh
meh
[snip]

So we see 2 failure modes:
- sometimes we fail to execve because the file is opened for writing
- sometimes we fail to open for writing because the file is being executed

The second condition is transient - the file is unmarked as the kernel proceeds to look up /bin/sh instead of the script.

A side observation is that if you have a file which is executable by others, they may interfere with your attempts to write it by repeatedly calling execve.h

can strace affect traced processes?

Please read "when strace fails to obtain syscall" first if you don't know how strace works.

Can strace/truss/whatever usage change the result of running a program?

Using ptrace introduces quite a lot of overhead (i.e. it slows things down). But besides that, can strace usage result in changing what the process ends up doing?

Yes it can and the key lies in waking up threads in an interruptible sleep. Note that you can count yourself unlucky if you encounter any of things below. strace is perfectly fine to use most of the time.

Depending on the code which put the thread to sleep, the thread will:
- do something without going back to the syscall boundary
- go back to the boundary and restart the syscall
- exit the kernel with EINTR

All of these are invasive in principle, but most of the time harmless.

One funny example of EINTR interferring with troubleshooting was few years back, when nginx would just hang around doing nothing when it was supposed to shutdown. Attaching to it revealed it was blocked in epoll_wait, which now returned with EINTR and that prompted nginx to exit.

Let's see a demo for the second case - automagic restart.

We are going to create a named pipe. The thread opening it for reading will block until it gets opened for writing (writers are blocked in the same manner).

shell1$ mkfifo fifo
shell1$ cat > fifo

shell2$ cat fifo

Stuff typed into the first cat should be read by the second cat.

Now consider:

shell1$ mkfifo fifo
shell1$ cat > fifo

shell2$ rm fifo 
shell2$ mkfifo fifo
shell2$ cat fifo

"cat > fifo" results in the shell trying to open the file, before it can proceed to execve cat. It got blocked:
[<ffffffff81226480>] pipe_wait+0x70/0xc0
[<ffffffff81226501>] wait_for_partner+0x31/0x70
[<ffffffff81226f60>] fifo_open+0x1b0/0x310
[<ffffffff8121bbff>] do_dentry_open+0x1ff/0x2f0
[<ffffffff8121d096>] vfs_open+0x56/0x60
[<ffffffff8122c0c4>] path_openat+0x1e4/0x12a0
[<ffffffff8122e34a>] do_filp_open+0x8a/0x100
[<ffffffff8121d45a>] do_sys_open+0x13a/0x230
[<ffffffff8121d56e>] SyS_open+0x1e/0x20
[<ffffffff817795ae>] entry_SYSCALL_64_fastpath+0x12/0x71
[<ffffffffffffffff>] 0xffffffffffffffff

But the name got unlinked and the other cat is waiting on a different pipe.

What if we strace? The sleep is interruptible and open is going to be restarted. And indeed:

open("fifo", O_WRONLY|O_CREAT|O_NOCTTY|O_TRUNC, 0666) = 3

Since it went all the way back to the boundary, the name "fifo" got looked up again and the thread proceeded to open the new pipe. And indeed, messages get relayed:

shell1$ cat > fifo
test

shell2$ cat fifo 
test

So, stracing blindly can sometimes give interesting results.

There are other methods which are not invasive in this manner, including systemtap, dtrace or ktrace (BSD-specific).

kernel game #1

Syscalls accepting file descriptors fit the following scheme:

1 2 3 4 5 6 7 8 9 10 11 12 13

int meh_syscall(int fd) { struct file *fp; int error; fp = getfile(fd); if (fp == NULL) return (EBADF); error = do_meh(fp); putfile(fp); return (error); }

That is, passed fd is used to obtain a pointer to  struct file, which is then used to perform the actual operation.

getfile will increase reference counter on struct file, while putfile will decrease it. If the new value is 0, there is nobody else using the file and it can be freed.

This is important if there are multiple threads in the process. If the counter was not maintained, and one thread closed the file while another one just got the pointer, there would be a bug.

However, if there is only one thread, what's the point of maintaining the counter? There is nobody to close the file from under us.

This optimisation is in fact in use on Linux. But their equivalent of getfile passes an information whether a reference was obtained, which is then used by putfile to check if it has to free it.

Why do they bother with that?

What would be wrong with the following approach: check if there is only 1 thread. If so, nobody can close the file from under us, therefore there is no need take the reference. Also this syscall does not create new threads. Then, after do_meh, we check the thread count again to see if we have to putfile. In other words:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

int meh_syscall(int fd) { struct file *fp; int error; if (curproc->threads > 1) fp = getfile(fd); else /* * just get the pointer, * do not modify the reference * counter */ fp = getfile_noref(fd); if (fp == NULL) return (EBADF); error = do_meh(fp); if (curproc->threads > 1) putfile(fp); return (error); }

So, what's the bug?

While it is true no new threads can appear, it very well may be that we will have 2 threads before we call do_meh and the other thread will exit before we return. Then the thread count check prior to putfile would see only one thread and the code would fail to release the previously obtained reference due to spotting the other thread.

a primitive to read data from userspace

As was outlined previously, a special primitive is needed to access userspace data safely. There are several highly specialized variants in both Linux and FreeBSD kernels, but they all work based on the same principle. Example below is taken from FreeBSD since Linux equivalent is way more convoluted.

Let's reiterate, consider:

int val;

val = *some_userspace_pointer;
printf("%d\n", val);

If some_userspace_pointer e.g. contains garbage, a page fault is going to occur. The page fault handler will conclude the fault cannot be satisified. But there is no way to tell this code about this issue - it only reads the value and assumes it succeeded.

What's needed is a function which will be able to actually detect the condition and return an error to the caller. With such a primitive in place the code becomes:

int val, error;
error = copyin(some_userspace_pointer, &val, sizeof(val));
if (error != 0)
        return error;
printf("%d\n", val);

A super slow variant would lock the address space, ensure relevant mappings are fine and only then do the read. That's a lot of of work completely unnecessary in the common case.

Instead, the standard approach is to have a way to tell the page fault handler where to jump if the page fault cannot be serviced. The place is supposed to clean up after failed copy and go back to the original caller.

In pseudo-code it would look like this:

int
copyin(void *from, void *to, size_t len)
{
       
        set_fault_handler(copyin_fault);
        if (len == 0)
                goto done_copyin;
        if (!fits_userspace(from, len))
                goto copyin_fault;
        memcpy(to, from, len);
done_copyin:
        set_fault_handler(0);
        return 0;
copyin_fault:
        set_fault_handler(0);
        return EFAULT;
}    

Let's take a look at an actual implementation with straightforward assembly (copyin(9) from the FreeBSD tree):

/*
 * copyin(from_user, to_kernel, len) - MP SAFE
 *        %rdi,      %rsi,      %rdx
 */
ENTRY(copyin)
        PUSH_FRAME_POINTER
        movq    PCPU(CURPCB),%rax
        movq    $copyin_fault,PCB_ONFAULT(%rax)

The handler is first set...

        testq   %rdx,%rdx                       /* anything to do? */
        jz      done_copyin

        /*
         * make sure address is valid
         */
        movq    %rdi,%rax
        addq    %rdx,%rax
        jc      copyin_fault
        movq    $VM_MAXUSER_ADDRESS,%rcx
        cmpq    %rcx,%rax
        ja      copyin_fault

... the range is then validated ...

        xchgq   %rdi,%rsi
        movq    %rdx,%rcx
        movb    %cl,%al
        shrq    $3,%rcx                         /* copy longword-wise */
        cld
        rep
        movsq
        movb    %al,%cl
        andb    $7,%cl                          /* copy remaining bytes */
        rep
        movsb

 ... and finally the copy actually done.  In an event of a page fault which cannot be satisified, the kernel will go to copyin_fault label which will unset the handler and return an error effectively cleaning up after the function. The target buffer may now contain partially copied data, but that's an acceptable state - if the syscall failed, buffer content is not specified. Finally, if a page fault could be serviced without an issue (e.g. a page was swapped in) or there were no page faults, copying finishes and the code falls below to unset the handler and return 0.

done_copyin:
        xorl    %eax,%eax
        movq    PCPU(CURPCB),%rdx
        movq    %rax,PCB_ONFAULT(%rdx)
        POP_FRAME_POINTER
        ret

        ALIGN_TEXT
copyin_fault:
        movq    PCPU(CURPCB),%rdx
        movq    $0,PCB_ONFAULT(%rdx)
        movq    $EFAULT,%rax
        POP_FRAME_POINTER
        ret
END(copyin)

the kernel vs userspace arguments

Plenty of syscalls (e.g. open(2)) write to or read from userspace memory using dedicated primitives and maintain a local copy. Why not just deal with it like with regular kernel memory? As outlined in one of previous posts, mere access should work.

Passed address may belong to kernelspace, so it has to be validated. But let's say we already did that.

Consider a toy syscall:

1 2 3 4 5 6 7 8 9 10 11 12 13

int sys_meh(const char *name, int value) { if (!is_root()) { if (strcmp(name, "special") == 0) return -EPERM; } spin_lock(&meh_lock); meh_modify(name, value); spin_unlock(&meh_unlock); return 0; }

Here we accept a name and a value, but only root is allowed to modify the object identified as special.

First access is at line 5. What if the passed address is garbage? The read will trigger a page fault and with no way to communicate the problem to strcmp, the kernel is forced to oops/panic.

So let's say the address is not garbage.

The name is read twice: by sys_meh itself and later by meh_modify. Or in other words, the code relies on the value not changing. Is the expectation met? No. For instance there can be a second thread which will try to modify the string after strcmp is done, but before meh_modify is called. This would in effect circumvent the protection we had in place.

Here the situation is even worse. By the time the code reaches meh_modify, the kernel could have decided to evict the page backing the string. On access a page fault will occur and the kernel will try to bring it in. But it took a spinlock, which means it is illegal to service a page fault due to deadlock potential.

In situations like this the standard way is to store relevant data in a temporary buffer.

This causes serious trouble when various security-oriented syscall wrappers were implemented. For instance, code trying to restrict file access by monitoring filenames had the exact same bug visible with sys_meh above (but it could be also circumvented in myriad of other ways, including symlinks). Interested parties are invited to read Exploiting Concurrency Vulnerabilities in System Call Wrappers.

the kernel vs NULL pointer dereference

FreeBSD, Linux and plenty of other kernels deny userspace requests to mmap pages at address 0 as a rudimentary hardening measure. This guarantees the kernel catches its own "NULL pointer deferences" and in turn lets it panic/oops. This changes a guaranteed privilege escalation vector into a local denial of service.

Let's see what exactly is going on here.

We will focus on amd64, but conceptually this is also true for i386 and likely several other architectures which have the address space shared between the kernel and userspace. If said space is disjoint, description below does not apply.

The address space looks roughly like this:

+---------------+ 0xffffffffffffefff    
| the kernel    |               
|(in some areas)|               
+---------------+ 0xffff800000000000               
|address space  |               
|    hole       |               
+---------------+ 0x800000000000
| userspace     |               
|               |               
+---------------+ 0x0  

The hole covers addresses which cannot be accessed on this architecture. Outlined userspace and kernel placement is the de facto standard. Note that both are mapped in the same address space. Spaces can be split in principle, but are not due to performance reasons.

These addresses are virtual. Actual physical memory pages may or may not be backing them up. The size of a page varies, it can be either 4KB, 2MB or 1GB.

Let's say an address 0xc0ffee belongs to an area mmapped with read and write permissions, and backed by a physical page at this very moment. When a thread enters the kernel (to e.g. execute a system call), the in-kernel code will be able to read and write said memory without any special measures.

Userspace can request arbitrary addresses with calls to mmap(2). Normally the kernel will provide whatever address it wants, but this can be changed by passing MAP_FIXED flag. As such, userspace can request to map a page at address 0.

Without pedantry void *p = NULL; will mean that consists of zeroes.

To sum this up:

p = mmap(NULL, 4096, PROT_WRITE|PROT_READ, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0);
p->val = 8;

Provided the kernel grants the request, this will effectively dereference a NULL pointer.

But most importantly, should the kernel try to access such an address itself, it will now succeed. Why would it do that? Of course due to a bug. NULL is often the default  value of pointers in various structures, so e.g. code which forgets to NULL check a field which can legitimately be NULL would be susceptible. There are plenty of real-world bugs which manifest themselves like this.

How to use this to escalate privileges? Depends on the bug, let's take the most blatant issue: a pointer to a function is NULL, but the code calls it. Userspace could mmap the page (with execute permissions) at 0 and fill it with whatever code it wants. When the bug is encountered, the kernel unknowingly starts executing the code planted by userspace.

Here is an example: CVE-2009-2692.txt Linux NULL pointer dereference due to incorrect proto_ops initializations.

While mappings at 0 are denied, there are other less frequent bugs which can result in the kernel unknowingly accessing userspace memory. A general solution with dedicated CPU support consists of SMAP (Supervisor Mode Access Prevention) and SMEP (Supervisor Mode Execution Prevention), but note these technologies are relatively new (read: your machines likely don' have them). Finally, a software-based implementation was provided with grsec.

when strace fails to obtain syscall information

strace(1) (or truss(1) on BSDs) is a system call tracer. You may have seen threads waiting for various operations in the kernel which were successfully reported (e.g. open). Yet sometimes you attach to the target process and don't get any output. The boring answer is that no threads in the process are executing any syscalls and as a result there is nothing to report. But what if we can tell for sure at least one thread is executing a syscall or at least called one and is now blocked?

Let's see how strace works in the first place. The kernel provides a special interface: ptrace(2). It can be used to observe various actions of the target process and interact with it. In particular, it can be told to stop the target process on syscall entry and exit. Once it is stopped, the tracer can read the state and determine what syscall is being called and what arguments were provided. The key here is that the target process has to reach this code.

So how does strace manage to properly report a thread waiting for open? [1] The thread in such a state is in an interruptible sleep. It is woken up, goes all the way back to kernel<->userspace boundary where it executes ptrace bits and proceeds to re-executes the syscall.

For what threads will strace fail to obtain syscall information? Definitely ones blocked in an uninterruptible sleep as they cannot be woken up like that, and in effect can't go back to let the tracer do its thing. The other possibility is a thread actively executing code in the kernel - it does not sleep and there is no mechanism to tell it to go back to the boundary.

What to do for such threads? In most (not all!) cases it is possible to read kernel backtrace (/proc/<tid>/stack) and try to work out stuff from there.

As a final remark, not all threads entering the kernel are executing syscalls. A typical example is a page fault or floating point exception, none of which are reported by strace.

[1] Of course there is no guarantee that all open operations will be interruptible, but a popular example of waiting for the writer when opening a fifo is.