mprotect, madvise, mlock | Linux System Programming

QUESTION

Are keeping a page resident and changing its access permissions the same operation?

mprotect changes VMA/PTE permissions and turns CPU access violations into faults. madvise supplies the kernel with policy hints for reclaim, readahead, huge pages, dumps, and other behavior. mlock is closer to a resource reservation that keeps pages unevictable.

All three APIs take an address range, but their guarantees and failure causes differ. Protection changes induce TLB invalidation, mlock is constrained by RLIMIT_MEMLOCK and may induce page faults, and some madvise operations are always best-effort.

STRUCTURE

Structure diagram

Figure 1. Different page policies inside one mapping

base+4K+8K+12K+16K

R-X · executable code

R-- · immutable table

RW- · locked secret

RW- · MADV_DONTNEED

mprotect changes access bits, madvise changes kernel policy, and mlock changes reclaimability.

Permission, reclaim advice, and residency locking are independent properties. One VMA can split into ranges with different policies.

CALL PATH

Call path

Figure 2. From userspace code to observable results

mmap RW anonymous range

touch page fault/allocate

mprotect read-only PTE

madvise reclaim policy

mlock pin resident set

Even for the same range, protection bits, reclaimability, and current residency are separate state. Inspect VmFlags, Locked, and Rss independently in /proc/smaps.

Figure 3. Major points along the kernel-internal path

do_mprotect_pkey VMA split/merge

change_protection PTE permission

do_madvise behavior dispatch

mlock_fixup VM_LOCKED

TLB flush update CPU translations

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

SOURCE COORDINATES

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

File	Function / structure	What to inspect
mm/mprotect.c	do_mprotect_pkey(), mprotect_fixup()	Change VMA permissions and split/merge VMAs
mm/madvise.c	do_madvise(), madvise_vma_behavior()	Per-advice handling and page-range operations
mm/mlock.c	do_mlock(), mlock_fixup()	Locked accounting and page population

COMPLETE PROGRAM

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 page_policy.c -o page_policy

01#define _DEFAULT_SOURCE
02#include <stdio.h>
03#include <sys/mman.h>
04#include <unistd.h>
05
06int main(void)
07{
08    long page = sysconf(_SC_PAGESIZE);
09    unsigned char *area = mmap(NULL, (size_t)page, PROT_READ | PROT_WRITE,
10                               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
11    if (area == MAP_FAILED)
12        return 1;
13    area[0] = 7;
14
15    if (madvise(area, (size_t)page, MADV_DONTDUMP) < 0)
16        return 1;
17    if (mlock(area, (size_t)page) < 0)
18        perror("mlock");
19    if (mprotect(area, (size_t)page, PROT_READ) < 0)
20        return 1;
21
22    printf("value=%u; mapping is now read-only\n", area[0]);
23    munlock(area, (size_t)page);
24    return munmap(area, (size_t)page) != 0;
25}

CODE NOTES

Code notes

Source line 13area[0] = 7

Triggers a write fault before mprotect so the page actually exists. The same store causes SIGSEGV after the range becomes read-only.

Source line 15MADV_DONTDUMP

Marks this VMA for exclusion from a core dump. For secrets, examine swap, logging, and child inheritance in addition to dump policy.

Source line 17mlock(area

Requests that pages remain resident. The call can fail because of privileges or RLIMIT_MEMLOCK, so the return value must be handled.

Source line 9PROT_READ

Removes write permission from the VMA and PTEs. If another alias mapping is writable, the object as a whole is not immutable.

Source line 23munlock(area

Explicitly drops locked accounting before unmapping. munmap also removes the lock on its range.

DETAILS

Detailed behavior

mprotect is used for W^X and JIT protocols

A JIT uses a publish protocol that generates code in RW pages and then changes them to RX. Minimize pages that are simultaneously writable and executable, and obey architecture-specific instruction-cache synchronization requirements.

RELRO is another case in which mprotect makes a relocated range read-only after relocations complete.

Each madvise command has different semantics

MADV_DONTNEED can discard the contents of private anonymous pages so that the next access sees zero-fill; its page-cache meaning differs for a file mapping. MADV_FREE is a lazy discard whose old contents may remain visible until memory pressure.

Do not infer one common immediate effect from the advice name. Check the man-page conditions for the specific mapping type.

mlock reduces latency by fixing system cost

Locked pages cannot be reclaimed, so excessive locking raises memory pressure for other workloads. Measure and limit the stack, code, and buffer ranges needed on the real-time path.

Because large stack growth after mlockall(MCL_FUTURE) can hit the resource limit, define a startup-prefault and failure policy.

OBJECTS

Objects and lifetimes

Object	Creation and release	Values to inspect
`VMA protection`	Updated over the mprotect range and retained for the lifetime of any split VMAs	VM_READ/WRITE/EXEC
`PTE permission`	Reflects VMA policy in hardware bits and changes together with a TLB flush	writable, executable
`locked_vm accounting`	Incremented in the mm by mlock and decremented by munlock/unmap	RLIMIT_MEMLOCK, VmLck

FAILURE PATH

Failure conditions and common misconceptions

Observed symptom	Likely causes	How to verify
mprotect ENOMEM	A VMA split exhausts map count or metadata	Check vm.max_map_count and range alignment
mlock EPERM/ENOMEM	A capability or memlock limit	ulimit -l, CapEff, VmLck
Data changes through another path despite being read-only here	A writable alias or writer to shared backing	Inspect every mapping and fd

LAB

Verify it yourself

After mprotect, add a handler that writes area[0] and inspects siginfo.si_addr and SEGV_ACCERR.
In /proc/self/smaps, observe how dd/lo in VmFlags and the Locked value change before and after advice/locking.
After MADV_DONTNEED, inspect the anonymous-page value and minor-fault count.

Run./page_policy

Tracestrace -e trace=mmap,mprotect,madvise,mlock,munlock,munmap ./page_policy

PRIMARY REFERENCES

Primary sources

mprotect(2)madvise(2)mlock(2)