Verifying execution paths with /proc, strace, and perf

QUESTION

Where should time that is invisible to strace be examined?

strace uses ptrace/seccomp to show syscall entry/exit and signals, but cannot directly separate userspace computation from internal kernel-function intervals. /proc exposes fd, maps, status, and sched information at a point in time, while perf observes CPU and kernel events through counters, sampling, and tracepoints.

Do not infer a cause from one tool's output. Align syscall wall time, off-CPU wait, page faults, scheduler delay, and userspace CPU samples on the same request timeline.

STRUCTURE

Structure diagram

Figure 1. Tools and visibility by observation question

questiontoolobserved objectnot visible which syscall is slow?strace -ttTentry/return·errnoCPU code between syscallswhere did the fd leak?/proc/PID/fdcurrent fd tablepast lifetime after closewhy is RSS large?smaps_rollupmapping/accountingallocation call stackwhere is the CPU hot path?perf recordsampled IP/call graphevery exact call countwhy is a runnable task delayed?perf sched/tracepointscheduler eventapplication predicate

Each tool has a different unit of observation. A syscall trace cannot explain a userspace CPU hot path, and an RSS snapshot cannot explain page-fault latency.

CALL PATH

Call path

Figure 2. From userspace code to observable results

symptom latency/CPU/RSS/fd

strace syscalls and errno

/proc object snapshot

perf counter/sample/trace

source compare branches and lifetimes

Choose the observation question first. Match the object and tool: the fd table for an fd leak, fault counters for page-fault latency, and sampling for a CPU hot path.

Figure 3. Major points along the kernel-internal path

syscall trace entry/exit timestamps

procfs show query task/mm/files

perf_event PMU/software event

tracepoint stable event fields

analysis correlation ID/timebase

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

SOURCE COORDINATES

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

File	Function / structure	What to inspect
fs/proc/base.c	proc_pid_status(), proc_pid_fd_operations	Inspect which task information per-PID status/fd files expose
kernel/events/core.c	perf_event_open(), perf_event_alloc()	perf event context and fd lifetime
kernel/trace/trace_events.c	event_trace_add_tracer()	Enable tracepoint events and connect the ring buffer

COMPLETE PROGRAM

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 observe_me.c -o observe_me

01#define _DEFAULT_SOURCE
02#include <fcntl.h>
03#include <stdio.h>
04#include <stdlib.h>
05#include <sys/mman.h>
06#include <unistd.h>
07
08int main(void)
09{
10    long page = sysconf(_SC_PAGESIZE);
11    size_t length = 32UL * 1024 * 1024;
12    unsigned char *area = mmap(NULL, length, PROT_READ | PROT_WRITE,
13                               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
14    if (area == MAP_FAILED)
15        return 1;
16    for (size_t offset = 0; offset < length; offset += (size_t)page)
17        area[offset] = (unsigned char)(offset / (size_t)page);
18
19    int fd = open("/dev/null", O_WRONLY | O_CLOEXEC);
20    if (fd < 0)
21        return 1;
22    dprintf(fd, "pid=%ld pages=%zu\n", (long)getpid(), length / (size_t)page);
23    close(fd);
24    printf("pid=%ld checksum=%u\n", (long)getpid(), area[length - page]);
25    munmap(area, length);
26    return 0;
27}

CODE NOTES

Code notes

Source line 10sysconf(_SC_PAGESIZE)

Matches the measurement loop to the actual base-page size so the fault count can be compared with access count.

Source line 12mmap(NULL, length

strace shows one fast mmap syscall, but the actual page cost occurs as minor faults in the following write loop.

Source line 17area[offset] =

This is where page-faults/minor-faults in perf stat increase and the userspace loop consumes CPU time.

Source line 19open("/dev/null"

An fd briefly appears in /proc/self/fd, and its openat/close lifetime can be checked in strace.

Source line 25munmap(area

Explicitly ends the mapping lifetime. Even though process exit would clean it up, unmap timing matters to RSS/VA in a repeating long-lived process.

DETAILS

Detailed behavior

strace separates time inside and outside syscalls

-T reports elapsed time from syscall entry to return, but ptrace overhead affects short calls and multithreaded scheduling. Align it with application request logs on a -ttt/clock basis.

unfinished/resumed means other trace lines appeared while a thread was blocked in a syscall.

/proc files change even while they are being read

While the fd directory is traversed, the process can close and reuse fds. VMAs can change between maps and smaps too. These look like snapshots but are not transactions over complete process state.

When an exact dump is required, define a consistency mechanism such as stopping the process, ptrace, or an application safepoint.

perf sampling is probabilistic observation

Low sample frequency can miss a short hot path; high frequency increases overhead and lost samples. Record the tradeoff among frame-pointer, DWARF, and LBR call-graph methods.

When counters multiplex, inspect time_enabled/time_running and the scaled value.

OBJECTS

Objects and lifetimes

Object	Creation and release	Values to inspect
`tracee task`	The ptrace relationship is retained from strace attach through detach/exit	stop reason, syscall state, signal
`proc file snapshot`	Looks up task objects during open/read, with a different consistency scope for each file	PID reuse, namespace, permission
`perf event fd`	Created by perf_event_open and managed through enable/disable/read/close	counter, sample ring, time scaling

FAILURE PATH

Failure conditions and common misconceptions

Observed symptom	Likely causes	How to verify
The bottleneck is invisible in strace	Userspace CPU or scheduler delay	Use perf record/sched and the wall-clock timeline
The PID target changes	A numeric PID was reused	Check pidfd and starttime
perf results are unstable	Too little sampling, multiplexing, or frequency changes	Repeat runs and inspect scaling and CPU pinning

LAB

Verify it yourself

Use strace -c and -ttT for syscall counts and individual latency, and correlate them with page-faults from perf stat.
In a version that pauses at getchar, collect /proc/PID/maps, smaps_rollup, fd, and status together.
Before perf record -g and flame graphs, verify compiler frame-pointer settings, debug symbols, and sample loss.

Run./observe_me

Tracestrace -ttT -f ./observe_me && perf stat -d ./observe_me

PRIMARY REFERENCES

Primary sources

proc(5)strace(1)perf_event_open(2)