io_uring ring and request lifetimes | Linux System Programming

QUESTION

Can a submitted buffer be reused as soon as io_uring_enter returns?

io_uring setup creates submission and completion ring memory shared by the kernel and userspace. Userspace fills an SQE, publishes the tail, and notifies the kernel with enter; after the kernel request executes, the kernel writes a CQE.

A submit syscall returning does not mean the operation is complete. The operation may refer to a user buffer and file until its CQE is consumed, so track per-request ownership and cancellation results.

STRUCTURE

Structure diagram

Figure 1. Shared SQ/CQ rings and in-flight requests

Submission Queue

head 2tail 6

0free1free2READ#413WRITE#424TIMEOUT#435NOP#446free7free

Kernel in-flight

head 0tail 3

0req#41 · buffer A1req#42 · buffer B2req#43 · timer3worker/driver

Completion Queue

head 1tail 4

0seen1CQE#422CQE#413CQE#434free5free6free7free

An SQE slot can be reused after the kernel consumes it, but the user buffer must remain valid until CQE completion. The two lifetimes differ.

CALL PATH

Call path

Figure 2. From userspace code to observable results

io_uring_setup ring fd + mmap offsets

fill SQE opcode/fd/buffer

publish SQ tail memory ordering

kernel request asynchronous/synchronous execution

CQE consume res/user_data

Draw the SQE-slot lifetime, request lifetime, user-buffer lifetime, and CQE lifetime on one overlapping timeline. Slot reuse and buffer reuse have different completion conditions.

Figure 3. Major points along the kernel-internal path

io_submit_sqes SQE copy/validate

io_init_req io_kiocb

issue file operation

io_req_complete finalize result

io_cqring_fill_event CQ publish

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

SOURCE COORDINATES

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

File	Function / structure	What to inspect
io_uring/io_uring.c	io_uring_setup(), io_uring_enter()	Create ring context and enter submission/completion paths
io_uring/io_uring.c	io_submit_sqes(), io_init_req()	Convert an SQE into a kernel request
io_uring/io_uring.c	io_req_complete_post(), io_cqring_event_overflow()	Publish a CQE and handle overflow

COMPLETE PROGRAM

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 uring_read.c -luring -o uring_read

01#include <fcntl.h>
02#include <liburing.h>
03#include <stdio.h>
04#include <string.h>
05#include <unistd.h>
06
07int main(int argc, char **argv)
08{
09    if (argc != 2)
10        return 2;
11    int fd = open(argv[1], O_RDONLY | O_CLOEXEC);
12    struct io_uring ring;
13    if (fd < 0 || io_uring_queue_init(8, &ring, 0) < 0)
14        return 1;
15
16    char buffer[256];
17    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
18    io_uring_prep_read(sqe, fd, buffer, sizeof(buffer), 0);
19    io_uring_sqe_set_data64(sqe, 0x1001);
20    if (io_uring_submit(&ring) < 0)
21        return 1;
22
23    struct io_uring_cqe *cqe;
24    if (io_uring_wait_cqe(&ring, &cqe) < 0)
25        return 1;
26    int result = cqe->res;
27    if (result > 0)
28        write(STDOUT_FILENO, buffer, (size_t)result);
29    else if (result < 0)
30        fprintf(stderr, "read: %s\n", strerror(-result));
31    io_uring_cqe_seen(&ring, cqe);
32    io_uring_queue_exit(&ring);
33    close(fd);
34    return result < 0;
35}

CODE NOTES

Code notes

Source line 13io_uring_queue_init(8

Prepares 8-entry SQ/CQ rings, a ring fd, and shared mappings. Inspect setup results for the actual CQ size and features.

Source line 16char buffer[256]

The stack frame and buffer must remain valid until read completion. Returning from the function or reusing them for another request first creates a data race.

Source line 19io_uring_sqe_set_data64

Stores an opaque key that connects the completion to an application request object. Storing only an fd number does not solve reuse.

Source line 26int result = cqe->res

A CQE result is not -1/errno like libc; it is either a successful byte count or negative errno. Pass -result to strerror.

Source line 31io_uring_cqe_seen

Advances the CQ head, returning that completion slot to the ring. Copy the result and user_data first.

DETAILS

Detailed behavior

Asynchrony depends on opcode and file type

Not every operation always runs on a separate worker or hardware-async path. Inline completion, task work, and io-wq offload can be mixed.

Measure completion latency, worker saturation, and context switches rather than only syscall reduction.

Registered resources change pinning costs

Fixed files and registered buffers can reduce per-request fd lookup and pinning, but extend registration-table and page-pin lifetimes. Account for in-flight requests during update/unregister.

Long-term pinned pages can affect reclaim and migration.

Confirm cancellation through completions too

A successful cancel SQE means the target operation was found and the cancellation request applied; handle ordering of both the target CQE and cancel CQE. A request that already completed can produce a result such as ENOENT.

For linked timeouts and multishot operations, check flags that allow one user_data to produce several CQEs.

OBJECTS

Objects and lifetimes

Object	Creation and release	Values to inspect
`io_ring_ctx`	Created by queue_init/setup and released by queue_exit/ring-fd close	SQ/CQ, task refs, worker
`SQE/io_kiocb`	Begins in a written SQ slot, becomes a kernel request, and remains in flight until completion	opcode, user_data, buffer/fd refs
`CQE`	Created when completion is published and returned to the ring by cqe_seen	res, flags, user_data

FAILURE PATH

Failure conditions and common misconceptions

Observed symptom	Likely causes	How to verify
Buffer contents are corrupted	The buffer was reused or its stack frame returned before completion	Inspect the request owner and CQE timeline
CQ overflow/stall	Completions are not consumed fast enough	Inspect CQ depth, the overflow flag, and seen calls
Errors are interpreted incorrectly	cqe->res was handled like errno	Apply the negative-errno rule

LAB

Verify it yourself

Submit reads at several offsets and use user_data to record that completion order can differ from submission order.
Compare syscalls, throughput, tail latency, and pinned memory before and after registering files/buffers.
Add a linked timeout and async cancel, then tabulate combinations of target CQEs with cancel/timeout CQEs.

Run./uring_read /etc/hostname

Tracestrace -e trace=io_uring_setup,io_uring_enter,io_uring_register,mmap,munmap ./uring_read /etc/hostname

PRIMARY REFERENCES

Primary sources

io_uring(7)Linux io_uring liburing