Unevictable LRU Infrastructure

Unevictable LRU의 목적

Unevictable LRU 기반 구조

이 문서는 Linux memory manager의 `Unevictable LRU` 기반 구조와 이를 이용해 여러 종류의 회수 불가능한 folio를 관리하는 방법을 설명합니다.

구현이 무엇을 하는지는 code를 읽어 알 수 있으므로, 여기서는 이 mechanism의 전체적인 근거와 구현을 이끈 설계 결정의 이유, 즉 왜 그렇게 동작하는지를 설명하는 데 초점을 둡니다.

==============================
Unevictable LRU Infrastructure
==============================

.. contents:: :local:


Introduction
============

This document describes the Linux memory manager's "Unevictable LRU"
infrastructure and the use of this to manage several types of "unevictable"
folios.

The document attempts to provide the overall rationale behind this mechanism
and the rationale for some of the design decisions that drove the
implementation.  The latter design rationale is discussed in the context of an
implementation description.  Admittedly, one can obtain the implementation
details - the "what does it do?" - by reading the code.  One hopes that the
descriptions below add value by provide the answer to "why does it do that?".

다섯 번째 LRU처럼 다루는 이유

Unevictable LRU

이 기능은 회수 불가능한 folio를 추적하는 LRU를 추가해 vmscan으로부터 숨깁니다. 대용량 x86_64 시스템에서 관찰된 folio reclaim의 확장성 문제를 해결하기 위해 Larry Woodman의 patch를 바탕으로 도입되었습니다.

예를 들어 non-NUMA x86_64 시스템에 128GB memory가 있으면 node 하나에 4KB page가 3,200만 개 넘게 있습니다. 이 중 많은 page가 회수 불가능하면 vmscan은 회수 가능한 소수를 찾으려고 LRU list를 오래 훑습니다. 모든 CPU가 수 시간 또는 수일 동안 vmscan에 100% 사용되어 시스템이 응답하지 않을 수도 있습니다.

Unevictable list가 다루는 page 종류는 다음과 같습니다.

• ramfs가 소유한 page
• `noswap` mount option을 사용한 tmpfs가 소유한 page
• `SHM_LOCK`된 shared-memory 영역에 mapping된 page
• `VM_LOCKED`, 즉 `mlock()`된 VMA에 mapping된 page

앞으로 정의상 또는 상황상 회수할 수 없는 다른 조건도 이 기반 구조가 처리할 수 있습니다.

Unevictable LRU folio list

Unevictable LRU folio list라는 이름은 엄밀히 말해 사실이 아닙니다. 처음부터 LRU 순서의 list가 아니라 anonymous·file, active·inactive folio list의 동반 구조였고, 현재는 folio linked list조차 아닙니다. 그래도 익숙한 관례에 따라 source와 이 문서에서는 이를 다섯 번째 LRU folio list처럼 표현합니다.

기반 구조는 node별 `unevictable` list와 관련 folio flag인 `PG_unevictable`로 구성됩니다. `PG_lru`가 설정된 folio가 어느 LRU에 있는지 나타낸다는 점에서 `PG_unevictable`은 `PG_active`와 비슷하며 두 flag는 상호 배타적입니다.

LRU처럼 관리하면 기존 folio 조작·격리·통계 code를 그대로 재사용할 수 있습니다. 또한 memory defragmentation, workload 관리, memory hotplug를 위해 node 사이에서 회수 불가능한 folio도 migration할 수 있습니다. Kernel은 `folio_isolate_lru()`로 LRU에서 성공적으로 격리한 folio만 migration할 수 있으므로 LRU와 무관한 곳에 두면 migration이 막힙니다.

Unevictable list는 file-backed folio와 anonymous swap-backed folio를 구분하지 않습니다. 이 구분은 folio가 실제로 회수 가능할 때만 중요합니다. 이 list와 통계는 node별 LRU를 `lru_list` 배열로 만든 구조의 이점을 함께 얻습니다.

The Unevictable LRU
===================

The Unevictable LRU facility adds an additional LRU list to track unevictable
folios and to hide these folios from vmscan.  This mechanism is based on a patch
by Larry Woodman of Red Hat to address several scalability problems with folio
reclaim in Linux.  The problems have been observed at customer sites on large
memory x86_64 systems.

To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of
main memory will have over 32 million 4k pages in a single node.  When a large
fraction of these pages are not evictable for any reason [see below], vmscan
will spend a lot of time scanning the LRU lists looking for the small fraction
of pages that are evictable.  This can result in a situation where all CPUs are
spending 100% of their time in vmscan for hours or days on end, with the system
completely unresponsive.

The unevictable list addresses the following classes of unevictable pages:

 * Those owned by ramfs.

 * Those owned by tmpfs with the noswap mount option.

 * Those mapped into SHM_LOCK'd shared memory regions.

 * Those mapped into VM_LOCKED [mlock()ed] VMAs.

The infrastructure may also be able to handle other conditions that make pages
unevictable, either by definition or by circumstance, in the future.


The Unevictable LRU Folio List
------------------------------

The Unevictable LRU folio list is a lie.  It was never an LRU-ordered
list, but a companion to the LRU-ordered anonymous and file, active and
inactive folio lists; and now it is not even a folio list.  But following
familiar convention, here in this document and in the source, we often
imagine it as a fifth LRU folio list.

The Unevictable LRU infrastructure consists of an additional, per-node, LRU list
called the "unevictable" list and an associated folio flag, PG_unevictable, to
indicate that the folio is being managed on the unevictable list.

The PG_unevictable flag is analogous to, and mutually exclusive with, the
PG_active flag in that it indicates on which LRU list a folio resides when
PG_lru is set.

The Unevictable LRU infrastructure maintains unevictable folios as if they were
on an additional LRU list for a few reasons:

 (1) We get to "treat unevictable folios just like we treat other folios in the
     system - which means we get to use the same code to manipulate them, the
     same code to isolate them (for migrate, etc.), the same code to keep track
     of the statistics, etc..." [Rik van Riel]

 (2) We want to be able to migrate unevictable folios between nodes for memory
     defragmentation, workload management and memory hotplug.  The Linux kernel
     can only migrate folios that it can successfully isolate from the LRU
     lists (or "Movable" folios: outside of consideration here).  If we were to
     maintain folios elsewhere than on an LRU-like list, where they can be
     detected by folio_isolate_lru(), we would prevent their migration.

The unevictable list does not differentiate between file-backed and
anonymous, swap-backed folios.  This differentiation is only important
while the folios are, in fact, evictable.

The unevictable list benefits from the "arrayification" of the per-node LRU
lists and statistics originally proposed and posted by Christoph Lameter.

Memory control group과의 상호 작용

Unevictable LRU는 `lru_list` enum을 확장하는 방식으로 memory control group, 즉 memory controller와 연동합니다. 관련 문서는 `Documentation/admin-guide/cgroup-v1/memory.rst`를 참고하십시오.

Node별 LRU list가 enum element마다 하나씩 배열화되어 있기 때문에 memory-controller 자료 구조도 node별 unevictable list를 자동으로 갖습니다. Controller는 page가 unevictable list로 들어오고 나가는 이동을 추적합니다.

Memory cgroup에 pressure가 생겨도 controller는 unevictable list의 page를 reclaim하려 하지 않습니다. 회수 가능한 page만 처리하므로 reclaim 효율은 좋아집니다.

반대로 cgroup에 charge된 page 중 unevictable 비율이 너무 높으면 task의 회수 가능한 working set이 남은 memory에 들어가지 못할 수 있습니다. 그러면 cgroup이 thrashing에 빠지거나 OOM killer가 task를 종료할 수 있습니다.

Memory Control Group Interaction
--------------------------------

The unevictable LRU facility interacts with the memory control group [aka
memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by
extending the lru_list enum.

The memory controller data structure automatically gets a per-node unevictable
list as a result of the "arrayification" of the per-node LRU lists (one per
lru_list enum element).  The memory controller tracks the movement of pages to
and from the unevictable list.

When a memory control group comes under memory pressure, the controller will
not attempt to reclaim pages on the unevictable list.  This has a couple of
effects:

 (1) Because the pages are "hidden" from reclaim on the unevictable list, the
     reclaim process can be more efficient, dealing only with pages that have a
     chance of being reclaimed.

 (2) On the other hand, if too many of the pages charged to the control group
     are unevictable, the evictable portion of the working set of the tasks in
     the control group may not fit into the available memory.  This can cause
     the control group to thrash or to OOM-kill tasks.

Address space 전체를 회수 불가로 표시

Address space를 unevictable로 표시하기

ramfs처럼 address space에 연결된 모든 page를 evict할 수 없는 기능을 위해 `AS_UNEVICTABLE` address-space flag가 제공됩니다. Filesystem은 wrapper 함수로 이 flag를 조작합니다.

• `mapping_set_unevictable(struct address_space *mapping)`은 address space 전체를 회수 불가로 표시합니다.
• `mapping_clear_unevictable(struct address_space *mapping)`은 address space를 다시 회수 가능으로 표시합니다.
• `mapping_unevictable(struct address_space *mapping)`은 address space 전체가 회수 불가인지 질의해 true 또는 false를 반환합니다.

현재 kernel에서는 이 기능을 세 곳에서 사용합니다.

• ramfs는 inode를 만들 때 address space를 표시하며 그 표시는 inode 수명 전체에 유지됩니다.
• SYSV SHM은 `SHM_LOCK`부터 `SHM_UNLOCK`까지 address space를 표시합니다. `SHM_LOCK` 자체는 swap out된 page를 memory로 읽어오지 않으므로 application이 resident 상태를 보장하려면 page를 직접 touch해야 합니다.
• i915 driver는 pinned address space를 unpin할 때까지 표시합니다. 이 driver가 unevictable로 표시한 memory 양은 `debugfs/dri/0/i915_gem_objects`의 bounded object size와 대략 같습니다.

.. _mark_addr_space_unevict:

Marking Address Spaces Unevictable
----------------------------------

For facilities such as ramfs none of the pages attached to the address space
may be evicted.  To prevent eviction of any such pages, the AS_UNEVICTABLE
address space flag is provided, and this can be manipulated by a filesystem
using a number of wrapper functions:

 * ``void mapping_set_unevictable(struct address_space *mapping);``

        Mark the address space as being completely unevictable.

 * ``void mapping_clear_unevictable(struct address_space *mapping);``

        Mark the address space as being evictable.

 * ``int mapping_unevictable(struct address_space *mapping);``

        Query the address space, and return true if it is completely
        unevictable.

These are currently used in three places in the kernel:

 (1) By ramfs to mark the address spaces of its inodes when they are created,
     and this mark remains for the life of the inode.

 (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called.
     Note that SHM_LOCK is not required to page in the locked pages if they're
     swapped out; the application must touch the pages manually if it wants to
     ensure they're in memory.

 (3) By the i915 driver to mark pinned address space until it's unpinned. The
     amount of unevictable memory marked by i915 driver is roughly the bounded
     object size in debugfs/dri/0/i915_gem_objects.

folio_evictable()과 vmscan culling

Unevictable page 감지

`mm/internal.h`의 `folio_evictable()`은 앞의 query 함수로 `AS_UNEVICTABLE` flag를 검사해 folio가 회수 가능한지 판단합니다.

SHM 영역처럼 page가 채워진 뒤 address space를 표시하는 경우 `SHM_LOCK` 동작은 lazy할 수 있습니다. `mlock()`처럼 page table을 미리 채우거나 page를 즉시 unevictable list로 밀어 넣을 필요가 없습니다. Reclaim scan 중 vmscan이 folio를 만나면 그때 이동시킵니다.

`SHM_UNLOCK` 같은 unlock에서는 `shmctl()` 등의 해제 주체가 영역의 page를 훑고, 다른 회수 불가 조건이 없으면 unevictable list에서 구출해야 합니다. 영역 자체가 파괴되면 page를 free하는 과정에서 구출됩니다.

`folio_evictable()`은 `folio_test_mlocked()`도 호출합니다. `PG_mlocked` 상태는 folio가 `VM_LOCKED` VMA에 fault되거나 `VM_LOCKED`로 바뀌는 VMA 안에서 발견될 때 설정됩니다.

Vmscan의 unevictable folio 처리

Fault path나 `mlock()`·`mmap()` 시점에 미리 unevictable list로 옮겨진 folio는 `munlock()` 등으로 다시 evictable이 되어 구출될 때까지 vmscan이 보지 않습니다.

하지만 편의를 위해 unevictable folio가 일반 active/inactive LRU에 남는 경우도 있습니다. `shrink_active_list()`, `shrink_inactive_list()`, `shrink_folio_list()`는 이를 발견하면 scan 중인 memory cgroup과 node의 unevictable list로 우회시키는 culling을 수행합니다.

`VM_LOCKED` VMA에 mapping됐지만 `PG_mlocked`가 없는 folio는 `shrink_active_list()` 또는 `shrink_folio_list()`까지 갈 수 있습니다. 그곳에서 `folio_referenced()`나 `try_to_unmap()`이 reverse map을 걸을 때 감지하며 shrinker가 folio를 놓을 때 unevictable list로 보냅니다.

Vmscan은 culling할 folio의 lock을 놓은 뒤 `folio_isolate_lru()`의 반대인 `folio_putback_lru()`로 LRU에 다시 넣습니다. Unlock 뒤 회수 불가 조건이 바뀔 수 있으므로 `__pagevec_lru_add_fn()`이 unevictable 상태를 재확인한 뒤 올바른 list에 배치합니다.

Detecting Unevictable Pages
---------------------------

The function folio_evictable() in mm/internal.h determines whether a folio is
evictable or not using the query function outlined above [see section
:ref:`Marking address spaces unevictable <mark_addr_space_unevict>`]
to check the AS_UNEVICTABLE flag.

For address spaces that are so marked after being populated (as SHM regions
might be), the lock action (e.g. SHM_LOCK) can be lazy, and need not populate
the page tables for the region as does, for example, mlock(), nor need it make
any special effort to push any pages in the SHM_LOCK'd area to the unevictable
list.  Instead, vmscan will do this if and when it encounters the folios during
a reclamation scan.

On an unlock action (such as SHM_UNLOCK), the unlocker (e.g. shmctl()) must scan
the pages in the region and "rescue" them from the unevictable list if no other
condition is keeping them unevictable.  If an unevictable region is destroyed,
the pages are also "rescued" from the unevictable list in the process of
freeing them.

folio_evictable() also checks for mlocked folios by calling
folio_test_mlocked(), which is set when a folio is faulted into a
VM_LOCKED VMA, or found in a VMA being VM_LOCKED.


Vmscan's Handling of Unevictable Folios
---------------------------------------

If unevictable folios are culled in the fault path, or moved to the unevictable
list at mlock() or mmap() time, vmscan will not encounter the folios until they
have become evictable again (via munlock() for example) and have been "rescued"
from the unevictable list.  However, there may be situations where we decide,
for the sake of expediency, to leave an unevictable folio on one of the regular
active/inactive LRU lists for vmscan to deal with.  vmscan checks for such
folios in all of the shrink_{active|inactive|folio}_list() functions and will
"cull" such folios that it encounters: that is, it diverts those folios to the
unevictable list for the memory cgroup and node being scanned.

There may be situations where a folio is mapped into a VM_LOCKED VMA,
but the folio does not have the mlocked flag set.  Such folios will make
it all the way to shrink_active_list() or shrink_folio_list() where they
will be detected when vmscan walks the reverse map in folio_referenced()
or try_to_unmap().  The folio is culled to the unevictable list when it
is released by the shrinker.

To "cull" an unevictable folio, vmscan simply puts the folio back on
the LRU list using folio_putback_lru() - the inverse operation to
folio_isolate_lru() - after dropping the folio lock.  Because the
condition which makes the folio unevictable may change once the folio
is unlocked, __pagevec_lru_add_fn() will recheck the unevictable state
of a folio before placing it on the unevictable list.

Mlocked page 설계의 역사

MLOCKED page

Unevictable folio list는 ramfs와 SYSV SHM뿐 아니라 `mlock()`에도 유용합니다. `mlock()`은 `CONFIG_MMU=y`에서만 제공되며 NOMMU에서는 모든 mapping이 사실상 mlocked입니다.

역사

`Unevictable mlocked Pages` 기반 구조는 Nick Piggin의 RFC patch `mm: mlocked pages off LRU`에서 시작되었습니다. 이는 mlocked page를 vmscan에서 숨기려는 Christoph Lameter의 patch에 대한 대안이었습니다.

초기 patch는 `struct page`의 LRU link field 하나를 그 page를 mapping한 `VM_LOCKED` VMA의 개수로 사용했습니다. 그러나 link를 count로 쓰면 page를 LRU에서 관리할 수 없어 `folio_isolate_lru()`가 찾지 못했고 migration도 불가능했습니다.

Nick은 격리 전에 mlocked page를 LRU에 다시 넣는 방식으로 이 문제를 해결하면서 VMA count를 포기했습니다. Unevictable LRU와 합쳐진 뒤에는 `munlock` 때 reverse map을 걸어 다른 `VM_LOCKED` VMA가 남았는지 확인했습니다.

Page마다 reverse map을 걷는 것은 비효율적이고 많은 process가 같은 file을 mlock한 채 종료할 때 rmap lock의 치명적인 경합을 만들 수 있었습니다. Linux 5.18에서 LRU link field의 `mlock_count` 아이디어를 되살리되 migration을 막지 않도록 구현했습니다. 이 때문에 현재 `Unevictable LRU list`는 실제 page linked list가 아니며, meminfo용 크기만 유지합니다.

MLOCKED Pages
=============

The unevictable folio list is also useful for mlock(), in addition to ramfs and
SYSV SHM.  Note that mlock() is only available in CONFIG_MMU=y situations; in
NOMMU situations, all mappings are effectively mlocked.


History
-------

The "Unevictable mlocked Pages" infrastructure is based on work originally
posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU".
Nick posted his patch as an alternative to a patch posted by Christoph Lameter
to achieve the same objective: hiding mlocked pages from vmscan.

In Nick's patch, he used one of the struct page LRU list link fields as a count
of VM_LOCKED VMAs that map the page (Rik van Riel had the same idea three years
earlier).  But this use of the link field for a count prevented the management
of the pages on an LRU list, and thus mlocked pages were not migratable as
folio_isolate_lru() could not detect them, and the LRU list link field was not
available to the migration subsystem.

Nick resolved this by putting mlocked pages back on the LRU list before
attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs.  When
Nick's patch was integrated with the Unevictable LRU work, the count was
replaced by walking the reverse map when munlocking, to determine whether any
other VM_LOCKED VMAs still mapped the page.

However, walking the reverse map for each page when munlocking was ugly and
inefficient, and could lead to catastrophic contention on a file's rmap lock,
when many processes which had it mlocked were trying to exit.  In 5.18, the
idea of keeping mlock_count in Unevictable LRU list link field was revived and
put to work, without preventing the migration of mlocked pages.  This is why
the "Unevictable LRU list" cannot be a linked list of pages now; but there was
no use for that linked list anyway - though its size is maintained for meminfo.

PG_mlocked 설정과 해제 조건

기본 관리

`VM_LOCKED` VMA에 mapping된 mlocked page는 unevictable page의 한 종류입니다. Memory-management subsystem이 이를 인식하면 folio에 `PG_mlocked`를 설정하며 `folio_set_mlocked()`와 `folio_clear_mlocked()`로 조작합니다.

`PG_mlocked` page는 LRU에 추가될 때 unevictable list에 놓입니다. 다음 경로에서 mlocked 상태를 인식할 수 있습니다.

• `mlock()`, `mlock2()`, `mlockall()` system-call handler
• `MAP_LOCKED` 영역을 mapping하는 `mmap()` handler
• `MCL_FUTURE`로 `mlockall()`을 호출한 task가 새 영역을 mapping하는 경우
• Fault path와 `VM_LOCKED` stack segment 확장
• Vmscan의 `shrink_folio_list()`가 `folio_referenced()` 또는 `try_to_unmap()`으로 `VM_LOCKED` VMA의 page를 reclaim하려는 경우

Mlocked page는 다음 경우 unlock되어 unevictable list에서 구출됩니다.

• `munlock()` 또는 `munlockall()`로 unlock한 범위에 mapping된 경우
• 마지막 `VM_LOCKED` VMA에서 `munmap()`되거나 task exit로 unmap되는 경우
• Mmap file의 마지막 `VM_LOCKED` VMA에서 page가 truncate되는 경우
• `VM_LOCKED` VMA에서 page가 COW되기 직전

Basic Management
----------------

mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable
pages.  When such a page has been "noticed" by the memory management subsystem,
the folio is marked with the PG_mlocked flag.  This can be manipulated using
folio_set_mlocked() and folio_clear_mlocked() functions.

A PG_mlocked page will be placed on the unevictable list when it is added to
the LRU.  Such pages can be "noticed" by memory management in several places:

 (1) in the mlock()/mlock2()/mlockall() system call handlers;

 (2) in the mmap() system call handler when mmapping a region with the
     MAP_LOCKED flag;

 (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE
     flag;

 (4) in the fault path and when a VM_LOCKED stack segment is expanded; or

 (5) as mentioned above, in vmscan:shrink_folio_list() when attempting to
     reclaim a page in a VM_LOCKED VMA by folio_referenced() or try_to_unmap().

mlocked pages become unlocked and rescued from the unevictable list when:

 (1) mapped in a range unlocked via the munlock()/munlockall() system calls;

 (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including
     unmapping at task exit;

 (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file;
     or

 (4) before a page is COW'd in a VM_LOCKED VMA.

mlock 계열 system call 처리

`mlock()`·`mlock2()`·`mlockall()` system call 처리

Handler는 지정 범위의 각 VMA에 `mlock_fixup()`을 적용합니다. `mlockall()`이면 task의 active address space 전체가 대상입니다. 이 함수는 lock과 unlock 모두에 쓰며, 이미 `VM_LOCKED`인 VMA를 mlock하거나 그렇지 않은 VMA를 munlock하면 no-op으로 돌아옵니다.

VMA가 special-VMA filter를 통과하면 이웃 VMA와 merge하거나, 범위가 VMA 전체를 덮지 않을 경우 일부를 split합니다. 이미 present한 page는 `mlock_vma_pages_range()` → `walk_page_range()` → `mlock_pte_range()` → `mlock_folio()` 경로로 표시합니다.

System call이 돌아가기 전에 `do_mlock()` 또는 `mlockall()`은 `__mm_populate()`를 호출합니다. 남은 page를 `get_user_pages()`로 fault-in하고 fault되는 동안 mlocked로 표시합니다.

VMA가 `PROT_NONE`이면 `get_user_pages()`가 page를 fault-in할 수 없지만 문제는 아닙니다. 나중에 이 `VM_LOCKED` VMA에 page가 fault되면 fault path가 처리합니다. `mlock2()`의 `MLOCK_ONFAULT` 영역도 같은 방식입니다.

VMA에 fault되는 각 PTE 또는 PMD에 대해 page-add-rmap 함수가 `mlock_vma_folio()`를 호출하고, VMA가 `VM_LOCKED`이면 `mlock_folio()`를 실행합니다. 단, THP 일부의 PTE mapping은 제외합니다. 새 anonymous page라면 `folio_add_lru_vma()`가 `mlock_new_folio()`를 호출해 page가 exclusive이고 아직 LRU에 없다는 정보를 활용합니다.

`mlock_folio()`는 즉시 `PG_mlocked`를 설정하고 page를 CPU의 mlock folio batch에 넣습니다. `__mlock_folio()`가 `lru_lock` 아래에서 나머지를 일괄 처리해 `PG_unevictable`을 설정하고 `mlock_count`를 초기화한 뒤 unevictable 상태로 옮깁니다. 이미 `PG_lru`, `PG_unevictable`, `PG_mlocked` 상태이면 `mlock_count`만 증가시킵니다.

Page가 아직 LRU에 없거나 잠시 격리된 상태라면 `mlock_count`를 건드릴 수 없습니다. 이후 `__munlock_folio()`가 page를 LRU에 돌려놓을 때 0으로 설정합니다. Race 때문에 그 시점에 1로 정할 수 없으므로 page가 영구히 unevictable로 고립되는 것보다 count를 낮게 잡습니다. 일단 evictable LRU로 구출한 뒤 vmscan이 `VM_LOCKED` VMA에서 다시 찾으면 재차 mlock할 수 있습니다.

mlock()/mlock2()/mlockall() System Call Handling
------------------------------------------------

mlock(), mlock2() and mlockall() system call handlers proceed to mlock_fixup()
for each VMA in the range specified by the call.  In the case of mlockall(),
this is the entire active address space of the task.  Note that mlock_fixup()
is used for both mlocking and munlocking a range of memory.  A call to mlock()
an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED, is
treated as a no-op and mlock_fixup() simply returns.

If the VMA passes some filtering as described in "Filtering Special VMAs"
below, mlock_fixup() will attempt to merge the VMA with its neighbors or split
off a subset of the VMA if the range does not cover the entire VMA.  Any pages
already present in the VMA are then marked as mlocked by mlock_folio() via
mlock_pte_range() via walk_page_range() via mlock_vma_pages_range().

Before returning from the system call, do_mlock() or mlockall() will call
__mm_populate() to fault in the remaining pages via get_user_pages() and to
mark those pages as mlocked as they are faulted.

Note that the VMA being mlocked might be mapped with PROT_NONE.  In this case,
get_user_pages() will be unable to fault in the pages.  That's okay.  If pages
do end up getting faulted into this VM_LOCKED VMA, they will be handled in the
fault path - which is also how mlock2()'s MLOCK_ONFAULT areas are handled.

For each PTE (or PMD) being faulted into a VMA, the page add rmap function
calls mlock_vma_folio(), which calls mlock_folio() when the VMA is VM_LOCKED
(unless it is a PTE mapping of a part of a transparent huge page).  Or when
it is a newly allocated anonymous page, folio_add_lru_vma() calls
mlock_new_folio() instead: similar to mlock_folio(), but can make better
judgments, since this page is held exclusively and known not to be on LRU yet.

mlock_folio() sets PG_mlocked immediately, then places the page on the CPU's
mlock folio batch, to batch up the rest of the work to be done under lru_lock by
__mlock_folio().  __mlock_folio() sets PG_unevictable, initializes mlock_count
and moves the page to unevictable state ("the unevictable LRU", but with
mlock_count in place of LRU threading).  Or if the page was already PG_lru
and PG_unevictable and PG_mlocked, it simply increments the mlock_count.

But in practice that may not work ideally: the page may not yet be on an LRU, or
it may have been temporarily isolated from LRU.  In such cases the mlock_count
field cannot be touched, but will be set to 0 later when __munlock_folio()
returns the page to "LRU".  Races prohibit mlock_count from being set to 1 then:
rather than risk stranding a page indefinitely as unevictable, always err with
mlock_count on the low side, so that when munlocked the page will be rescued to
an evictable LRU, then perhaps be mlocked again later if vmscan finds it in a
VM_LOCKED VMA.

Special VMA 필터와 munlock

Special VMA 필터링

`mlock_fixup()`은 다음 VMA 종류를 별도로 걸러냅니다.

• `VM_IO` 또는 `VM_PFNMAP` VMA는 완전히 건너뜁니다. 뒤의 page는 본질적으로 pinned이고 많은 경우 표시할 `struct page`도 없습니다. `get_user_pages()`도 실패하므로 방문할 이유가 없습니다.
• Hugetlbfs page를 mapping한 VMA는 이미 memory에 사실상 pin되어 있어 `mlock()`할 필요가 없습니다. 그래도 `__mm_populate()`은 hugetlbfs 범위를 포함해 hugepage와 PTE를 채웁니다.
• `VM_DONTEXPAND` VMA는 VDSO나 relay-channel page처럼 대개 kernel page를 userspace에 mapping한 것입니다. 본질적으로 unevictable이고 LRU가 관리하지 않습니다. `__mm_populate()`은 이 범위의 PTE를 필요하면 채웁니다.
• `VM_MIXEDMAP` VMA에는 `VM_LOCKED`를 설정하지 않지만 `__mm_populate()`은 범위를 포함해 PTE를 채웁니다.

모든 special VMA에서 `mlock_fixup()`은 `VM_LOCKED`를 설정하지 않습니다. 따라서 이후 `munlock()`, `munmap()`, task exit에서 처리할 필요가 없고 task의 `locked_vm`에도 합산하지 않습니다.

`munlock()`·`munlockall()` system call 처리

Unlock도 mlock 계열과 같은 `mlock_fixup()`이 처리합니다. 이미 unlock된 VMA는 그대로 반환하고 special VMA는 `VM_LOCKED`가 없으므로 무시합니다.

`VM_LOCKED` VMA라면 지정 범위를 merge 또는 split한 뒤 `mlock_vma_pages_range()` → `walk_page_range()` → `mlock_pte_range()` → `munlock_folio()` 경로로 page를 unlock합니다. Mlock 때와 같은 walker를 쓰되 VMA flag가 munlock 동작임을 나타냅니다.

`munlock_folio()`는 mlock pagevec으로 `lru_lock` 아래의 작업을 모읍니다. `__munlock_folio()`는 `mlock_count`를 줄이고 0이 되면 mlocked와 unevictable flag를 지워 folio를 inactive LRU로 옮깁니다.

Folio가 아직 unevictable LRU에 도착하지 않았거나 일시 격리되었다면 `mlock_count`를 쓸 수 없어 0으로 간주합니다. Folio는 evictable LRU로 구출되고, 필요하면 vmscan이 `VM_LOCKED` VMA에서 찾아 다시 mlock합니다.

Filtering Special VMAs
----------------------

mlock_fixup() filters several classes of "special" VMAs:

1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely.  The pages behind
   these mappings are inherently pinned, so we don't need to mark them as
   mlocked.  In any case, most of the pages have no struct page in which to so
   mark the page.  Because of this, get_user_pages() will fail for these VMAs,
   so there is no sense in attempting to visit them.

2) VMAs mapping hugetlbfs page are already effectively pinned into memory.  We
   neither need nor want to mlock() these pages.  But __mm_populate() includes
   hugetlbfs ranges, allocating the huge pages and populating the PTEs.

3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages,
   such as the VDSO page, relay channel pages, etc.  These pages are inherently
   unevictable and are not managed on the LRU lists.  __mm_populate() includes
   these ranges, populating the PTEs if not already populated.

4) VMAs with VM_MIXEDMAP set are not marked VM_LOCKED, but __mm_populate()
   includes these ranges, populating the PTEs if not already populated.

Note that for all of these special VMAs, mlock_fixup() does not set the
VM_LOCKED flag.  Therefore, we won't have to deal with them later during
munlock(), munmap() or task exit.  Neither does mlock_fixup() account these
VMAs against the task's "locked_vm".


munlock()/munlockall() System Call Handling
-------------------------------------------

The munlock() and munlockall() system calls are handled by the same
mlock_fixup() function as mlock(), mlock2() and mlockall() system calls are.
If called to munlock an already munlocked VMA, mlock_fixup() simply returns.
Because of the VMA filtering discussed above, VM_LOCKED will not be set in
any "special" VMAs.  So, those VMAs will be ignored for munlock.

If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the
specified range.  All pages in the VMA are then munlocked by munlock_folio() via
mlock_pte_range() via walk_page_range() via mlock_vma_pages_range() - the same
function used when mlocking a VMA range, with new flags for the VMA indicating
that it is munlock() being performed.

munlock_folio() uses the mlock pagevec to batch up work to be done
under lru_lock by  __munlock_folio().  __munlock_folio() decrements the
folio's mlock_count, and when that reaches 0 it clears the mlocked flag
and clears the unevictable flag, moving the folio from unevictable state
to the inactive LRU.

But in practice that may not work ideally: the folio may not yet have reached
"the unevictable LRU", or it may have been temporarily isolated from it.  In
those cases its mlock_count field is unusable and must be assumed to be 0: so
that the folio will be rescued to an evictable LRU, then perhaps be mlocked
again later if vmscan finds it in a VM_LOCKED VMA.

Mlocked page의 migration과 compaction

MLOCKED page migration

Migration할 page는 LRU에서 격리한 뒤 page를 잠근 채 old mapping을 unmap하고 address-space entry를 갱신하며 내용과 상태를 복사합니다. 마지막으로 migration entry를 새 page를 가리키는 page-table entry로 바꿉니다.

Linux는 mlocked page와 다른 unevictable page의 migration을 지원합니다. 마지막 `VM_LOCKED` VMA에서 old page를 unmap할 때 `PG_mlocked`를 지우고, 새 page를 migration entry 대신 `VM_LOCKED` VMA에 map할 때 다시 설정합니다.

Mlock 때문에 unevictable이었다면 `PG_unevictable`이 `PG_mlocked`를 따라갑니다. 다른 이유라면 `PG_unevictable`을 명시적으로 복사합니다.

Migration은 같은 page의 mlock·munlock과 race할 수 있지만 old page의 모든 PTE를 unmap한 뒤 새 page를 map하고 page-table lock으로 동기화하므로 대부분 문제가 없습니다.

예외는 `mlock_vma_pages_range()`가 기존 page를 mlock하기 전에 먼저 VMA에 `VM_LOCKED`를 설정하는 구간입니다. Walker가 도달하기 전에 page가 migration되면 `mlock_count`가 두 번 증가할 수 있습니다. 이를 막기 위해 VMA를 잠시 `VM_IO`로 표시하여 `mlock_vma_folio()`가 건너뛰게 합니다.

Migration 완료 후 old·new page를 LRU에 돌려놓습니다. 성공 시 old page, 실패 시 new page인 불필요한 쪽은 migration 과정이 잡은 reference를 놓을 때 free됩니다.

MLOCKED page compaction

Memory map에서 compact 가능한 영역을 scan할 때 기본적으로 unevictable page도 이동할 수 있습니다. `/proc/sys/vm/compact_unevictable_allowed`가 이 동작을 제어하며 자세한 내용은 `Documentation/admin-guide/sysctl/vm.rst`에 있습니다. 실제 compaction은 대부분 page-migration code가 처리하므로 앞의 workflow가 그대로 적용됩니다.

Migrating MLOCKED Pages
-----------------------

A page that is being migrated has been isolated from the LRU lists and is held
locked across unmapping of the page, updating the page's address space entry
and copying the contents and state, until the page table entry has been
replaced with an entry that refers to the new page.  Linux supports migration
of mlocked pages and other unevictable pages.  PG_mlocked is cleared from the
the old page when it is unmapped from the last VM_LOCKED VMA, and set when the
new page is mapped in place of migration entry in a VM_LOCKED VMA.  If the page
was unevictable because mlocked, PG_unevictable follows PG_mlocked; but if the
page was unevictable for other reasons, PG_unevictable is copied explicitly.

Note that page migration can race with mlocking or munlocking of the same page.
There is mostly no problem since page migration requires unmapping all PTEs of
the old page (including munlock where VM_LOCKED), then mapping in the new page
(including mlock where VM_LOCKED).  The page table locks provide sufficient
synchronization.

However, since mlock_vma_pages_range() starts by setting VM_LOCKED on a VMA,
before mlocking any pages already present, if one of those pages were migrated
before mlock_pte_range() reached it, it would get counted twice in mlock_count.
To prevent that, mlock_vma_pages_range() temporarily marks the VMA as VM_IO,
so that mlock_vma_folio() will skip it.

To complete page migration, we place the old and new pages back onto the LRU
afterwards.  The "unneeded" page - old page on success, new page on failure -
is freed when the reference count held by the migration process is released.


Compacting MLOCKED Pages
------------------------

The memory map can be scanned for compactable regions and the default behavior
is to let unevictable pages be moved.  /proc/sys/vm/compact_unevictable_allowed
controls this behavior (see Documentation/admin-guide/sysctl/vm.rst).  The work
of compaction is mostly handled by the page migration code and the same work
flow as described in Migrating MLOCKED Pages will apply.

THP 부분 mlock과 MAP_LOCKED

Transparent Huge Page mlock

THP는 LRU list에서 엔트리 하나로 표현되므로 개별 subpage가 아니라 compound page 전체만 unevictable로 만들 수 있습니다. 사용자가 hugepage 일부만 mlock하고 전체를 mlock한 사용자가 없다면 나머지 부분은 reclaim 가능해야 합니다.

부분 mlock 즉시 `split_huge_page()`를 호출하면 분할 실패 때문에 system call에 새롭고 간헐적인 실패 모드가 생길 수 있습니다. 대신 PTE-mlocked hugepage를 evictable LRU에 둡니다. `VM_LOCKED` VMA 경계의 PMD를 PTE table로 split합니다.

이렇게 하면 vmscan이 hugepage에 접근할 수 있습니다. Memory pressure가 오면 page를 split하고 `VM_LOCKED` VMA에 속한 subpage만 unevictable LRU로 옮기며 나머지는 reclaim합니다.

`/proc/meminfo`의 `Unevictable`과 `Mlocked` 값에는 `VM_LOCKED` VMA에서 PTE로만 mapping된 THP 일부가 포함되지 않습니다.

`mmap(MAP_LOCKED)` system call 처리

Application은 mlock 계열 외에도 `mmap()`의 `MAP_LOCKED` flag로 memory 영역의 mlock을 요청할 수 있습니다. 중요한 차이는 `mmap()` 뒤 `mlock()`을 호출하면 `mm_populate` 실패 등으로 범위를 fault-in하지 못할 때 `ENOMEM`과 함께 실패하지만, `mmap(MAP_LOCKED)`는 실패하지 않는다는 점입니다.

`MAP_LOCKED` 영역의 page는 swap out되지 않지만 memory를 처음 fault-in할 때 major page fault가 여전히 발생할 수 있습니다.

이미 `MCL_FUTURE`로 `mlockall()`을 호출한 task의 `mmap()` 또는 heap을 늘리는 `brk()`도 새 memory를 mlock합니다. 현재 기반 구조에서 mmap handler와 address-space 확장 함수는 `populate_vma_page_range()`에 VMA와 범위를 넘겨 mlock합니다.

MLOCKING Transparent Huge Pages
-------------------------------

A transparent huge page is represented by a single entry on an LRU list.
Therefore, we can only make unevictable an entire compound page, not
individual subpages.

If a user tries to mlock() part of a huge page, and no user mlock()s the
whole of the huge page, we want the rest of the page to be reclaimable.

We cannot just split the page on partial mlock() as split_huge_page() can
fail and a new intermittent failure mode for the syscall is undesirable.

We handle this by keeping PTE-mlocked huge pages on evictable LRU lists:
the PMD on the border of a VM_LOCKED VMA will be split into a PTE table.

This way the huge page is accessible for vmscan.  Under memory pressure the
page will be split, subpages which belong to VM_LOCKED VMAs will be moved
to the unevictable LRU and the rest can be reclaimed.

/proc/meminfo's Unevictable and Mlocked amounts do not include those parts
of a transparent huge page which are mapped only by PTEs in VM_LOCKED VMAs.


mmap(MAP_LOCKED) System Call Handling
-------------------------------------

In addition to the mlock(), mlock2() and mlockall() system calls, an application
can request that a region of memory be mlocked by supplying the MAP_LOCKED flag
to the mmap() call.  There is one important and subtle difference here, though.
mmap() + mlock() will fail if the range cannot be faulted in (e.g. because
mm_populate fails) and returns with ENOMEM while mmap(MAP_LOCKED) will not fail.
The mmapped area will still have properties of the locked area - pages will not
get swapped out - but major page faults to fault memory in might still happen.

Furthermore, any mmap() call or brk() call that expands the heap by a task
that has previously called mlockall() with the MCL_FUTURE flag will result
in the newly mapped memory being mlocked.  Before the unevictable/mlock
changes, the kernel simply called make_pages_present() to allocate pages
and populate the page table.

To mlock a range of memory under the unevictable/mlock infrastructure,
the mmap() handler and task address space expansion functions call
populate_vma_page_range() specifying the vma and the address range to mlock.

Unmap과 truncate의 mlock_count 정리

`munmap()`·`exit()`·`exec()` system call 처리

명시적 `munmap()`이나 `exit()`·`exec()` 내부 unmap으로 mlocked 영역을 제거할 때, 해당 page를 mapping한 마지막 `VM_LOCKED` VMA를 없앤다면 page도 munlock해야 합니다.

VMA에서 unmap하는 각 PTE 또는 PMD에 대해 `folio_remove_rmap_*()`은 `munlock_vma_folio()`를 호출하고, VMA가 `VM_LOCKED`이면 `munlock_folio()`를 호출합니다. THP 일부의 PTE mapping은 제외됩니다.

`munlock_folio()`는 pagevec으로 작업을 모아 `lru_lock` 아래의 `__munlock_folio()`가 처리하게 합니다. `mlock_count`가 0이 되면 mlocked·unevictable flag를 지우고 inactive LRU로 옮깁니다. Folio가 아직 list에 없거나 격리 중이면 count를 0으로 보고 evictable LRU로 구출합니다.

MLOCKED page truncate

File truncation이나 hole punching은 삭제할 page를 userspace에서 강제로 unmap합니다. Truncation은 대상 file page에서 COW된 private anonymous page도 unmap하고 삭제합니다.

이 과정에서도 `folio_remove_rmap_*()` → `munlock_vma_folio()` → `munlock_folio()` 경로로 mlocked page를 unlock하고 삭제할 수 있습니다.

동시에 `munlock()`이 실행되면 race가 생길 수 있습니다. `mlock_vma_pages_range()`는 present page 전체를 munlock하기 전에 VMA의 `VM_LOCKED`를 먼저 지웁니다. Walker가 page에 도달하기 전에 truncation 또는 hole punch가 unmap하면 이 VMA의 mlocked page로 인식하지 못해 `mlock_count`에서 빠지지 않습니다.

드문 경우 완전히 unmap된 page에 `PG_mlocked`가 남을 수 있습니다. `release_pages()` 또는 `__page_cache_release()`가 free하기 전에 flag와 통계를 정리합니다. 이 사건은 보통 0인 `/proc/vmstat`의 `unevictable_pgs_cleared`에 집계됩니다.

munmap()/exit()/exec() System Call Handling
-------------------------------------------

When unmapping an mlocked region of memory, whether by an explicit call to
munmap() or via an internal unmap from exit() or exec() processing, we must
munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages.
Before the unevictable/mlock changes, mlocking did not mark the pages in any
way, so unmapping them required no processing.

For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls
munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED
(unless it was a PTE mapping of a part of a transparent huge page).

munlock_folio() uses the mlock pagevec to batch up work to be done
under lru_lock by  __munlock_folio().  __munlock_folio() decrements the
folio's mlock_count, and when that reaches 0 it clears the mlocked flag
and clears the unevictable flag, moving the folio from unevictable state
to the inactive LRU.

But in practice that may not work ideally: the folio may not yet have reached
"the unevictable LRU", or it may have been temporarily isolated from it.  In
those cases its mlock_count field is unusable and must be assumed to be 0: so
that the folio will be rescued to an evictable LRU, then perhaps be mlocked
again later if vmscan finds it in a VM_LOCKED VMA.


Truncating MLOCKED Pages
------------------------

File truncation or hole punching forcibly unmaps the deleted pages from
userspace; truncation even unmaps and deletes any private anonymous pages
which had been Copied-On-Write from the file pages now being truncated.

Mlocked pages can be munlocked and deleted in this way: like with munmap(),
for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls
munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED
(unless it was a PTE mapping of a part of a transparent huge page).

However, if there is a racing munlock(), since mlock_vma_pages_range() starts
munlocking by clearing VM_LOCKED from a VMA, before munlocking all the pages
present, if one of those pages were unmapped by truncation or hole punch before
mlock_pte_range() reached it, it would not be recognized as mlocked by this VMA,
and would not be counted out of mlock_count.  In this rare case, a page may
still appear as PG_mlocked after it has been fully unmapped: and it is left to
release_pages() (or __page_cache_release()) to clear it and update statistics
before freeing (this event is counted in /proc/vmstat unevictable_pgs_cleared,
which is usually 0).

shrink_*_list()의 최종 교정

`shrink_*_list()`의 page reclaim

Vmscan의 `shrink_active_list()`는 `!page_evictable(page)`인 명백한 unevictable page를 골라 unevictable list로 우회시킵니다. 이 함수가 보는 것은 active/inactive LRU까지 들어온 page뿐이며, `PG_unevictable`이 이미 설정됐다면 애초에 unevictable list에 있어 보이지 않습니다.

일반 LRU에 남아 있을 수 있는 unevictable page의 예는 다음과 같습니다.

• 처음 할당될 때 일반 LRU에 놓인 ramfs page
• `SHM_LOCK`된 shared-memory page. `shmctl(SHM_LOCK)`은 page를 할당하거나 fault-in하지 않으므로 application이 처음 접근할 때 일반 경로로 들어올 수 있습니다.
• `VM_LOCKED` VMA에 여전히 mapping됐지만 사건 순서 때문에 `mlock_count`가 너무 낮아 일찍 munlock된 page

`shrink_inactive_list()`와 `shrink_folio_list()`도 inactive list에서 명백히 회수 불가능한 page를 찾아 해당 memory cgroup과 node의 unevictable list로 보냅니다.

`shrink_active_list()` 또는 `shrink_folio_list()`가 호출하는 rmap의 `folio_referenced_one()`과 `shrink_folio_list()`가 호출하는 `try_to_unmap_one()`은 마지막 종류의 page를 검사합니다. 아직 `VM_LOCKED` VMA에 mapping돼 있으면 `mlock_vma_folio()`로 상태를 교정하고, shrinker가 page를 놓을 때 unevictable list로 보냅니다.

Page Reclaim in shrink_*_list()
-------------------------------

vmscan's shrink_active_list() culls any obviously unevictable pages -
i.e. !page_evictable(page) pages - diverting those to the unevictable list.
However, shrink_active_list() only sees unevictable pages that made it onto the
active/inactive LRU lists.  Note that these pages do not have PG_unevictable
set - otherwise they would be on the unevictable list and shrink_active_list()
would never see them.

Some examples of these unevictable pages on the LRU lists are:

 (1) ramfs pages that have been placed on the LRU lists when first allocated.

 (2) SHM_LOCK'd shared memory pages.  shmctl(SHM_LOCK) does not attempt to
     allocate or fault in the pages in the shared memory region.  This happens
     when an application accesses the page the first time after SHM_LOCK'ing
     the segment.

 (3) pages still mapped into VM_LOCKED VMAs, which should be marked mlocked,
     but events left mlock_count too low, so they were munlocked too early.

vmscan's shrink_inactive_list() and shrink_folio_list() also divert obviously
unevictable pages found on the inactive lists to the appropriate memory cgroup
and node unevictable list.

rmap's folio_referenced_one(), called via vmscan's shrink_active_list() or
shrink_folio_list(), and rmap's try_to_unmap_one() called via shrink_folio_list(),
check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_folio()
to correct them.  Such pages are culled to the unevictable list when released
by the shrinker.