execve and process-image replacement | Linux System Programming

QUESTION

Does execve create a new process, or change the current one?

execve does not create a new PID. The calling task receives a new mm, code, stack, and credentials and continues under the same PID. The largest difference from an ordinary function is that successful execution never returns to the old code.

Not all state disappears. Fds without close-on-exec, the current working directory, umask, some signal-mask state, and resource limits remain, while caught signal dispositions and address mappings are reset.

STRUCTURE

Structure diagram

Figure 1. What is replaced and retained under the same PID

before execve

PID 4120
multiple threads
old text/data/heap/stack
fd 0,1,2,7
caught signal handlers

retain/recompute

retain PID·cwd·umask
apply FD_CLOEXEC
recompute credentials
retain resource limits

after execve

PID 4120
one calling thread
new PT_LOAD + stack
fd 0,1,2
caught handler reset

execve does not add a new process. It preserves parts of process identity and replaces the userspace image according to the new ELF.

CALL PATH

Call path

Figure 2. From userspace code to observable results

parent state prepare argv/env/fds

execve validate binary

point of no return discard old mm

ELF loader new stack/mm

entry same PID, new image

View the transition in terms of process identity and process image. The PID remains, but every userspace pointer and the thread configuration now belongs to the new image.

Figure 3. Major points along the kernel-internal path

do_execveat_common filename and argv

bprm_execve binary handler

begin_new_exec replace old image

setup_new_exec personality/name

start_thread entry and stack

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

SOURCE COORDINATES

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

File	Function / structure	What to inspect
fs/exec.c	do_execveat_common(), bprm_execve()	Copy argv/envp and invoke the binary handler
fs/exec.c	begin_new_exec(), setup_new_exec()	The irreversible region after the old mm is discarded
fs/binfmt_elf.c	load_elf_binary()	Create ELF segments and the initial stack

COMPLETE PROGRAM

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 exec_replace.c -o exec_replace

01#define _GNU_SOURCE
02#include <errno.h>
03#include <fcntl.h>
04#include <stdio.h>
05#include <stdlib.h>
06#include <unistd.h>
07
08extern char **environ;
09
10int main(void)
11{
12    int fd = open("exec.log", O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
13    if (fd < 0) {
14        perror("open");
15        return 1;
16    }
17    dprintf(fd, "before exec pid=%ld\n", (long)getpid());
18
19    char *const argv[] = { "sh", "-c",
20        "printf 'after exec pid=%s\n' $$; ls -l /proc/$$/fd", NULL };
21    execve("/bin/sh", argv, environ);
22
23    int saved = errno;
24    dprintf(STDERR_FILENO, "execve failed: %d\n", saved);
25    close(fd);
26    return 1;
27}

CODE NOTES

Code notes

Source line 12O_CLOEXEC

Combines open and FD_CLOEXEC in one syscall, eliminating the race in which another thread forks or execs between them and leaks the fd.

Source line 17dprintf(fd

The data written to the file before exec remains, but O_CLOEXEC makes the slot appear closed in the new image's fd table.

Source line 19char *const argv[]

By convention, argv[0] is the program name, and the array must end with a NULL pointer. The kernel limits both the strings and the number of pointers.

Source line 21execve("/bin/sh"

On success, the following line is never executed. A new userspace flow starts at /bin/sh's ELF entry.

Source line 23int saved = errno

Saves errno immediately after failure, preserving the original reason before dprintf or close can change it.

DETAILS

Detailed behavior

Design fd inheritance as an allowlist

By default, fds survive exec. Use the CLOEXEC variant of every creation syscall, then place only explicitly passed fds at standard or agreed numbers with dup2/dup3 during child setup.

When cleaning up a large fd table, close_range() has fewer races than walking /proc/self/fd.

Credential changes occur together with binary metadata

The setuid/setgid bits, file capabilities, no_new_privs, a nosuid mount, and ptrace state all participate in computing the new effective credentials. Clearing environment variables alone does not complete a privilege boundary.

In secure-execution mode, the dynamic linker restricts environment variables such as LD_PRELOAD.

Separate the fallible region from the unrecoverable region

The kernel validates as much of the binary as possible before discarding the old mm. Even so, a fatal error after replacement begins cannot return to the original image and may terminate the process with a signal.

In production, the parent commonly checks whether child exec succeeded with a CLOEXEC pipe. On success the pipe's write end closes automatically; on failure the child writes errno.

OBJECTS

Objects and lifetimes

Object	Creation and release	Values to inspect
`linux_binprm`	Exists from exec preparation until the binary handler completes	file, cred, argc/envc
`fd table`	Remains attached to the process, but CLOEXEC slots close at exec commit	close_on_exec bitmap
`mm_struct`	The mm for the new image is installed and the old mm reference is released	VMA, arg_start/end, exe_file

FAILURE PATH

Failure conditions and common misconceptions

Observed symptom	Likely causes	How to verify
execve returns ENOENT	The binary or PT_INTERP is absent	Check readelf -l and the pathname
EACCES	directory search permission, noexec mount, file mode	Inspect namei -l and findmnt options
The child retains an unintended fd	Missing CLOEXEC or an open/fcntl race	Inspect /proc/PID/fd and strace -f

LAB

Verify it yourself

After execution, verify that exec.log is absent from the fd list in /proc/<new-shell-PID>/fd.
Remove O_CLOEXEC and compare how the same fd is inherited by the shell.
Create a dedicated pipe and implement a handshake in which the child sends the errno from a failed exec to the parent.

Run./exec_replace

Tracestrace -f -e trace=execve,close ./exec_replace

PRIMARY REFERENCES

Primary sources

execve(2)close_range(2)credentials(7)