fcntl locks and the ioctl ABI | Linux System Programming

QUESTION

fcntl and ioctl both take an fd; what does each control?

fcntl supplies controls understood generically by the VFS, including fd-slot flags, open-file status flags, duplication, and byte-range locks. ioctl descends into per-subsystem command ABIs for terminals, block devices, network devices, and other file types.

Traditional POSIX record locks are process-associated, so closing any fd for the inode in the same process can affect the locks. Linux open-file-description locks are attached to struct file and can make ownership across threads and processes more explicit.

STRUCTURE

Structure diagram

Figure 1. Overlapping advisory locks on inode byte ranges

0163248648096

Process A · WRLCK

Process B · RDLCK

Process C · WRLCK waiting

Unlike this lock map, an ioctl argument is a device UAPI structure whose size, direction, and version must be validated for each command.

A lock covers [start, start+len), not necessarily the entire file. Nonoverlapping read locks can coexist, but a write lock conflicts with any overlap.

CALL PATH

Call path

Figure 2. From userspace code to observable results

fd check file type

fcntl/ioctl command + argument

VFS dispatch generic or f_op

lock/device change subsystem state

return result/errno

Record not only the command number, but also the argument structure's size, direction, ownership, and the object to which a lock is attached.

Figure 3. Major points along the kernel-internal path

do_fcntl cmd switch

fcntl_setlk file_lock

sys_ioctl security hook

vfs_ioctl unlocked_ioctl

driver uapi struct copy

This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.

SOURCE COORDINATES

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

File	Function / structure	What to inspect
fs/fcntl.c	do_fcntl(), f_dupfd()	Dispatch fcntl commands and distinguish fd/open-file flags
fs/locks.c	fcntl_setlk(), locks_lock_inode_wait()	Record-lock conflicts, waits, and owners
fs/ioctl.c	sys_ioctl(), do_vfs_ioctl()	Dispatch generic ioctls and file operations

COMPLETE PROGRAM

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 lock_range.c -o lock_range

01#define _GNU_SOURCE
02#include <fcntl.h>
03#include <stdio.h>
04#include <string.h>
05#include <unistd.h>
06
07int main(int argc, char **argv)
08{
09    if (argc != 2)
10        return 2;
11    int fd = open(argv[1], O_RDWR | O_CREAT | O_CLOEXEC, 0644);
12    if (fd < 0)
13        return 1;
14
15    struct flock lock;
16    memset(&lock, 0, sizeof(lock));
17    lock.l_type = F_WRLCK;
18    lock.l_whence = SEEK_SET;
19    lock.l_start = 0;
20    lock.l_len = 16;
21
22    if (fcntl(fd, F_OFD_SETLKW, &lock) < 0) {
23        perror("F_OFD_SETLKW");
24        return 1;
25    }
26    puts("locked bytes [0, 16); press Enter");
27    getchar();
28    lock.l_type = F_UNLCK;
29    int rc = fcntl(fd, F_OFD_SETLK, &lock);
30    close(fd);
31    return rc < 0;
32}

CODE NOTES

Code notes

Source line 16memset(&lock, 0

Fill the UAPI structure with 0, including padding and unused fields, so architecture-specific garbage values are not passed.

Source line 18lock.l_whence = SEEK_SET

Makes the byte-range origin the beginning of the file. With SEEK_CUR, changes to a shared file offset can affect calculation of the lock range.

Source line 20lock.l_len = 16

Places an advisory write lock only on [0,16). A length of 0 means from the starting point through EOF and any future growth.

Source line 22F_OFD_SETLKW

A Linux OFD lock that waits for a conflict to clear. Ownership belongs to the open file description, not a process PID.

Source line 28lock.l_type = F_UNLCK

Explicitly unlocks the same owner and range. An OFD lock is also released when the last struct file reference closes.

DETAILS

Detailed behavior

Advisory locks do not forcibly block I/O

They protect data only when cooperating processes obey the lock protocol. The VFS does not generally stop a process that reads or writes without taking the lock.

A database or log writer must include lock ranges and acquisition order in its application protocol.

An ioctl request number is also an ABI

The _IO, _IOR, _IOW, and _IOWR macros encode type, number, direction, and size, but the kernel does not automatically solve pointer validation or structure versioning. Compat tasks may require conversion between 32-bit and 64-bit layouts.

When designing a driver-private ioctl, specify fixed-width types, padding, reserved fields, and size/version rules.

First check whether the fd type supports the command

Regular files, pipes, sockets, and ttys do not understand the same commands. ENOTTY can mean not only that an fd is not a terminal, but more generally that the ioctl command is not implemented for it.

Use fstat and subsystem queries to identify the type, then distinguish a command failure caused by an absent device from an ABI mismatch.

OBJECTS

Objects and lifetimes

Object	Creation and release	Values to inspect
`file_lock`	Created by an fcntl lock request and attached to the inode lock tree/list	owner, range, type
`OFD owner`	Retained for the lifetime of struct file, with sharing behavior across dup/fork	fl_owner, last file ref
`ioctl argument`	Resides in user memory during the syscall and is copied by the driver with copy_from/to_user	size, alignment, reserved field

FAILURE PATH

Failure conditions and common misconceptions

Observed symptom	Likely causes	How to verify
Another writer writes despite the lock	The peer does not follow the advisory protocol	Inspect the peer's fcntl calls and I/O path
The lock is released earlier than expected	POSIX-lock semantics were confused with close semantics	Distinguish F_SETLK from F_OFD_SETLK
ioctl ENOTTY	The fd type is wrong or the command is unsupported	fstat, driver version, UAPI header

LAB

Verify it yourself

Run the example from two terminals and verify that the second process blocks until the first unlocks.
Change F_OFD_SETLKW to F_SETLKW and compare lock lifetimes when closing a duplicated fd and a separately opened fd.
Call the TIOCGWINSZ ioctl on a terminal fd and determine when redirected stdin produces ENOTTY.

Run./lock_range locked.dat

Tracestrace -e trace=openat,fcntl,ioctl,close ./lock_range locked.dat

PRIMARY REFERENCES

Primary sources

fcntl(2)ioctl(2)Linux ioctl design