회수할 memory가 부족하다고 모든 task가 더 빨리 scan하면 상황이 좋아질까요? 이미 모든 후보가 writeback 중이거나, 다른 reclaimer가 가져간 상태라면 CPU만 더 쓰게 됩니다. reclaim throttling은 진행을 만드는 주체에게 시간을 주고 reserve가 완전히 고갈되는 일을 피하려는 조절입니다.
기다리는 이유마다 queue가 다릅니다
reclaim_throttle()은 node의 wait queue를 reason별로 사용합니다. WRITEBACK은 backing write 진행을 기다리는 상황, ISOLATED는 너무 많은 후보가 local reclaimer에게 묶인 상황, NOPROGRESS는 scan해도 결과가 없는 상황입니다. congestion을 별도로 다루는 구현도 있습니다. 이들은 timeout과 wakeup 근거가 다르므로 하나의 전역 sleep으로 합쳐 생각하면 안 됩니다.
caller의 역할도 구분해야 합니다. reclaim의 shrink_node()는 writeback·congestion 관찰을, shrink_inactive_list()는 isolation 과다를, shrink_zones()의 helper는 no-progress를 근거로 기다립니다. 고전 v6.0의 do_writepages()도 writeback callback이 -ENOMEM을 반환하고 WB_SYNC_ALL로 계속해야 하는 경우 local node의 WRITEBACK queue에서 기다린 뒤 재시도합니다. 단순히 write syscall이면 항상 throttle하는 조건은 아닙니다. 해당 writeback loop를 함께 보면 allocation 실패와 writeback 의존성이 연결되는 이유를 알 수 있습니다.
compaction도 migration 후보를 isolate하므로 같은 ISOLATED queue를 사용할 수 있습니다. v6.0의 isolate_migratepages_block()은 isolation 과다를 보았을 때 아직 이동하지 않은 후보가 남아 있거나 MIGRATE_ASYNC이면 기다리지 않고 -EAGAIN으로 돌아갑니다. 그 밖에 기다린 뒤 fatal signal이 생기면 -EINTR입니다. reclaim driver의 특별한 양수 반환과 같은 protocol이 아닙니다. compaction의 실제 caller 조건을 보면 공통 wait queue를 사용해도 caller의 재시도 정책은 달라질 수 있다는 점이 드러납니다.
non-kswapd kernel thread나 I/O worker를 application과 똑같이 멈추면 reclaim에 필요한 일을 하는 주체가 사라질 수 있습니다. 예를 들어 writeback 완료를 기다리면서 그 write를 수행할 worker까지 같은 throttle에 넣으면 진행 보장이 무너집니다. 적용 대상의 예외는 성능 취향이 아니라 dependency를 보존하는 조건입니다.
Writeback 진전은 대기자 수와 함께 셉니다
WRITEBACK에서는 대기 중인 reclaimer 수와 대기 시작 이후 writeback 진전을 추적합니다. v6.18의 관련 accounting은 완료된 base page 수가 SWAP_CLUSTER_MAX * nr_throttled를 초과하면 queue를 깨우는 식입니다. folio 하나가 완료되었다는 사실만으로 모든 waiter에게 충분한 진전이 생긴 것은 아니므로 batch와 대기자 규모를 함께 봅니다.
wait queue는 timeout 전에도 깨울 수 있습니다. v6.18의 WRITEBACK은 HZ/10, ISOLATED는 HZ/50, NOPROGRESS·CONGESTED는 skip 검사를 통과한 경우 1 jiffy의 timeout을 사용합니다. 이 값은 scheduling까지 포함한 정확한 대기시간이나 I/O 완료 deadline이 아닙니다. timeout으로 돌아왔다고 writeback이 끝난 것도 아닙니다. caller는 조건을 다시 판단해야 하며, wait entry와 waiter count도 모든 종료 경로에서 정리해야 합니다.
NOPROGRESS는 backing이 병목일 때 의미가 있습니다
consider_reclaim_throttle()은 scan 대비 실제 reclaimed가 충분하면 NOPROGRESS waiter를 깨울 수 있습니다. 설명하는 v6.18 조건은 nr_reclaimed > (nr_scanned >> 3)입니다. 1/8이라는 heuristic을 정확히 보아야 하며, shift 기호가 사라진 문장을 '두 장 회수'로 읽으면 안 됩니다.
반대로 아주 적극적인 priority에서도 전혀 회수하지 못했다면 잠시 기다리는 것이 유리한지 봅니다. skip_throttle_noprogress()는 kswapd가 이미 반복 실패해 도움이 되지 않는 경우, write-pending data가 회수 가능량에 비해 작아 기다릴 실익이 낮은 경우 등을 거릅니다. no-progress라는 결과 하나로 항상 sleep하지는 않습니다.
구체적으로 consider_reclaim_throttle()은 앞의 progress 조건을 만족하면 waiter를 깨우고 반환합니다. 그 밖에는 kswapd와 cgroup reclaim을 NOPROGRESS 대상에서 제외하고, priority == 1 && nr_reclaimed == 0일 때 대기를 고려합니다. priority 1을 'LRU 전체를 반드시 다 scan한 상태'로 읽으면 안 됩니다. 기본 scan량의 shift와 실제 scan budget은 여전히 별도 조건입니다.
skip_throttle_noprogress()는 managed zone들의 reclaimable과 NR_ZONE_WRITE_PENDING을 합산합니다. kswapd 실패 한도에 도달했거나 2 * write_pending <= reclaimable이면 이 대기를 건너뜁니다. kswapd가 영구 종료되었다는 뜻이 아니라 지금 그 도움만 기다려서는 진전이 없다는 판단입니다. dirty·writeback이 많다는 사실과 이미 실제 제출된 I/O 양도 같은 값이 아닙니다.
ISOLATED 대기는 shared LRU를 지나치게 비운 direct reclaimer의 수를 조절합니다. timeout을 지나거나 isolation이 줄어 wakeup을 받으면 다시 조건을 확인합니다. 이 queue의 진행은 dirty data가 disk에 도착하는 것과 다른 사건입니다.
Direct reclaim 시작 전에는 reserve도 봅니다
direct reclaim은 내부 allocation을 위해 reserve에 접근할 수 있습니다. backing이 너무 느린 상황에서 수많은 caller가 이 작업에 들어가면 reserve까지 바닥날 위험이 있습니다. throttle_direct_reclaim()은 이 경우 먼저 kswapd의 진전을 기다리게 합니다.
kernel thread와 fatal signal이 pending인 task는 특별 취급합니다. 필수 worker를 멈추거나 곧 memory를 반환할 task의 exit를 불필요하게 늦춰서는 안 됩니다. allocation이 사용할 수 있는 zone 중 ordinary reserve를 평가할 범위를 찾아 판단하며, 이런 zone을 찾을 수 없는 상황을 무조건 reserve 고갈로 간주하지 않습니다.
filesystem 진입이 금지된 context에서는 direct·background reclaim이 서로 의존하는 일을 피하기 위해 timeout을 둡니다. 그 밖의 경로도 signal과 wait condition을 보존해야 합니다. 대기 후 fatal signal 때문에 direct reclaim을 중단하고 caller를 재시도·종료 경로로 돌리는 경우의 반환값은 실제로 회수한 page 수가 아닙니다.
ordinary reserve를 볼 node는 사용 가능한 zonelist·nodemask를 따라 만나는 첫 ZONE_NORMAL 이하 zone의 node입니다. 모든 node가 좋아질 때까지 기다리는 정책이 아닙니다. network swap의 buffer allocation이 사용할 수 없는 HIGHMEM 같은 zone을 reserve 회복의 근거로 삼지 않으려는 범위입니다. 적합한 zone을 찾지 못하면 reserve 고갈을 가정하지 않고 false를 반환해 direct reclaim을 계속합니다.
__GFP_FS가 없으면 wait_event_interruptible_timeout(..., HZ)를 사용하고, filesystem 진입이 허용되면 wait_event_killable()로 reserve 조건을 기다립니다. 전자는 일반 signal이나 timeout으로도 돌아올 수 있습니다. 대기 후 fatal signal이 있는 경우에만 true로 direct reclaim 중단을 알리므로, true가 reserve 회복의 증명이라는 해석도 틀립니다.
allow_direct_reclaim은 언제 기다림을 끝낼까요
고전 v6.0의 이 helper는 managed이고 회수 가능량이 있는 ZONE_NORMAL 이하 zone의 free 양과 minimum reserve를 모읍니다. v6.18에서는 회수 가능량이 없더라도 free page까지 0인 zone은 합산에 포함하며, !zone_reclaimable_pages(zone) && NR_FREE_PAGES != 0일 때만 제외합니다. 따라서 현대의 조건을 단순히 'reclaimable이 0이면 제외'라고 줄이면 reserve 합산이 달라집니다. reserve가 아예 없는 설정이라면 침범할 reserve도 없으므로 진행을 허용합니다. free 양이 합산 reserve의 절반을 엄격히 넘는 free_pages > reserve / 2가 진행 조건입니다.
kswapd가 실패 한도에 도달한 경우에도 기다리기만 해서는 해결되지 않으므로 direct reclaim을 허용합니다. 아직 기다려야 한다면 background 요청이 적절한 zone 범위를 향하도록 만들고 kswapd를 깨웁니다.
이 대기는 일반 reclaim reason queue와 별도의 direct-reclaim wait queue를 사용합니다. balance_pgdat()가 reserve 회복을 확인할 때 깨우고, prepare_kswapd_sleep()도 잠들기 전에 waiter를 깨워 남겨 두지 않습니다. sleep 상태가 끝났다는 사실보다 무엇이 조건을 바꿨는지를 함께 읽어야 합니다.
전체 code로 ownership과 종료 경로를 확인합니다
대기 여부를 결정하는 helper와 실제 wait entry를 등록·해제하는 함수는 역할이 다릅니다. 아래에서는 v6.0과 v6.18의 전체 정의를 통해 검사 순서, signal의 특별한 반환, waiter accounting의 정리를 함께 확인할 수 있습니다.
v6.0 consider_reclaim_throttle
v6_0_consider_reclaim_throttle.c
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/vmscan.c:3425-3453.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/vmscan.c#L3425
* License text: GPL-2.0.txt (in this directory).
*/
static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)
{
/*
* If reclaim is making progress greater than 12% efficiency then
* wake all the NOPROGRESS throttled tasks.
*/
if (sc->nr_reclaimed > (sc->nr_scanned >> 3)) {
wait_queue_head_t *wqh;
wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS];
if (waitqueue_active(wqh))
wake_up(wqh);
return;
}
/*
* Do not throttle kswapd or cgroup reclaim on NOPROGRESS as it will
* throttle on VMSCAN_THROTTLE_WRITEBACK if there are too many pages
* under writeback and marked for immediate reclaim at the tail of the
* LRU.
*/
if (current_is_kswapd() || cgroup_reclaim(sc))
return;
/* Throttle if making no progress at high prioities. */
if (sc->priority == 1 && !sc->nr_reclaimed)
reclaim_throttle(pgdat, VMSCAN_THROTTLE_NOPROGRESS);
}
v6.0 reclaim_throttle
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/vmscan.c:1104-1171.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/vmscan.c#L1104
* License text: GPL-2.0.txt (in this directory).
*/
void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)
{
wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];
long timeout, ret;
DEFINE_WAIT(wait);
/*
* Do not throttle IO workers, kthreads other than kswapd or
* workqueues. They may be required for reclaim to make
* forward progress (e.g. journalling workqueues or kthreads).
*/
if (!current_is_kswapd() &&
current->flags & (PF_IO_WORKER|PF_KTHREAD)) {
cond_resched();
return;
}
/*
* These figures are pulled out of thin air.
* VMSCAN_THROTTLE_ISOLATED is a transient condition based on too many
* parallel reclaimers which is a short-lived event so the timeout is
* short. Failing to make progress or waiting on writeback are
* potentially long-lived events so use a longer timeout. This is shaky
* logic as a failure to make progress could be due to anything from
* writeback to a slow device to excessive references pages at the tail
* of the inactive LRU.
*/
switch(reason) {
case VMSCAN_THROTTLE_WRITEBACK:
timeout = HZ/10;
if (atomic_inc_return(&pgdat->nr_writeback_throttled) == 1) {
WRITE_ONCE(pgdat->nr_reclaim_start,
node_page_state(pgdat, NR_THROTTLED_WRITTEN));
}
break;
case VMSCAN_THROTTLE_CONGESTED:
fallthrough;
case VMSCAN_THROTTLE_NOPROGRESS:
if (skip_throttle_noprogress(pgdat)) {
cond_resched();
return;
}
timeout = 1;
break;
case VMSCAN_THROTTLE_ISOLATED:
timeout = HZ/50;
break;
default:
WARN_ON_ONCE(1);
timeout = HZ;
break;
}
prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
ret = schedule_timeout(timeout);
finish_wait(wqh, &wait);
if (reason == VMSCAN_THROTTLE_WRITEBACK)
atomic_dec(&pgdat->nr_writeback_throttled);
trace_mm_vmscan_throttled(pgdat->node_id, jiffies_to_usecs(timeout),
jiffies_to_usecs(timeout - ret),
reason);
}
v6.0 skip_throttle_noprogress
v6_0_skip_throttle_noprogress.c
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/vmscan.c:1071-1102.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/vmscan.c#L1071
* License text: GPL-2.0.txt (in this directory).
*/
static bool skip_throttle_noprogress(pg_data_t *pgdat)
{
int reclaimable = 0, write_pending = 0;
int i;
/*
* If kswapd is disabled, reschedule if necessary but do not
* throttle as the system is likely near OOM.
*/
if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES)
return true;
/*
* If there are a lot of dirty/writeback pages then do not
* throttle as throttling will occur when the pages cycle
* towards the end of the LRU if still under writeback.
*/
for (i = 0; i < MAX_NR_ZONES; i++) {
struct zone *zone = pgdat->node_zones + i;
if (!managed_zone(zone))
continue;
reclaimable += zone_reclaimable_pages(zone);
write_pending += zone_page_state_snapshot(zone,
NR_ZONE_WRITE_PENDING);
}
if (2 * write_pending <= reclaimable)
return true;
return false;
}
v6.0 allow_direct_reclaim
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/vmscan.c:3672-3710.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/vmscan.c#L3672
* License text: GPL-2.0.txt (in this directory).
*/
static bool allow_direct_reclaim(pg_data_t *pgdat)
{
struct zone *zone;
unsigned long pfmemalloc_reserve = 0;
unsigned long free_pages = 0;
int i;
bool wmark_ok;
if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES)
return true;
for (i = 0; i <= ZONE_NORMAL; i++) {
zone = &pgdat->node_zones[i];
if (!managed_zone(zone))
continue;
if (!zone_reclaimable_pages(zone))
continue;
pfmemalloc_reserve += min_wmark_pages(zone);
free_pages += zone_page_state(zone, NR_FREE_PAGES);
}
/* If there are no reserves (unexpected config) then do not throttle */
if (!pfmemalloc_reserve)
return true;
wmark_ok = free_pages > pfmemalloc_reserve / 2;
/* kswapd must be awake if processes are being throttled */
if (!wmark_ok && waitqueue_active(&pgdat->kswapd_wait)) {
if (READ_ONCE(pgdat->kswapd_highest_zoneidx) > ZONE_NORMAL)
WRITE_ONCE(pgdat->kswapd_highest_zoneidx, ZONE_NORMAL);
wake_up_interruptible(&pgdat->kswapd_wait);
}
return wmark_ok;
}
v6.0 throttle_direct_reclaim
v6_0_throttle_direct_reclaim.c
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.0, mm/vmscan.c:3721-3799.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.0/mm/vmscan.c#L3721
* License text: GPL-2.0.txt (in this directory).
*/
static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
nodemask_t *nodemask)
{
struct zoneref *z;
struct zone *zone;
pg_data_t *pgdat = NULL;
/*
* Kernel threads should not be throttled as they may be indirectly
* responsible for cleaning pages necessary for reclaim to make forward
* progress. kjournald for example may enter direct reclaim while
* committing a transaction where throttling it could forcing other
* processes to block on log_wait_commit().
*/
if (current->flags & PF_KTHREAD)
goto out;
/*
* If a fatal signal is pending, this process should not throttle.
* It should return quickly so it can exit and free its memory
*/
if (fatal_signal_pending(current))
goto out;
/*
* Check if the pfmemalloc reserves are ok by finding the first node
* with a usable ZONE_NORMAL or lower zone. The expectation is that
* GFP_KERNEL will be required for allocating network buffers when
* swapping over the network so ZONE_HIGHMEM is unusable.
*
* Throttling is based on the first usable node and throttled processes
* wait on a queue until kswapd makes progress and wakes them. There
* is an affinity then between processes waking up and where reclaim
* progress has been made assuming the process wakes on the same node.
* More importantly, processes running on remote nodes will not compete
* for remote pfmemalloc reserves and processes on different nodes
* should make reasonable progress.
*/
for_each_zone_zonelist_nodemask(zone, z, zonelist,
gfp_zone(gfp_mask), nodemask) {
if (zone_idx(zone) > ZONE_NORMAL)
continue;
/* Throttle based on the first usable node */
pgdat = zone->zone_pgdat;
if (allow_direct_reclaim(pgdat))
goto out;
break;
}
/* If no zone was usable by the allocation flags then do not throttle */
if (!pgdat)
goto out;
/* Account for the throttling */
count_vm_event(PGSCAN_DIRECT_THROTTLE);
/*
* If the caller cannot enter the filesystem, it's possible that it
* is due to the caller holding an FS lock or performing a journal
* transaction in the case of a filesystem like ext[3|4]. In this case,
* it is not safe to block on pfmemalloc_wait as kswapd could be
* blocked waiting on the same lock. Instead, throttle for up to a
* second before continuing.
*/
if (!(gfp_mask & __GFP_FS))
wait_event_interruptible_timeout(pgdat->pfmemalloc_wait,
allow_direct_reclaim(pgdat), HZ);
else
/* Throttle until kswapd wakes the process */
wait_event_killable(zone->zone_pgdat->pfmemalloc_wait,
allow_direct_reclaim(pgdat));
if (fatal_signal_pending(current))
return true;
out:
return false;
}
v6.18 consider_reclaim_throttle
v6_18_consider_reclaim_throttle.c
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.18, mm/vmscan.c:6222-6250.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.18/mm/vmscan.c#L6222
* License text: GPL-2.0.txt (in this directory).
*/
static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)
{
/*
* If reclaim is making progress greater than 12% efficiency then
* wake all the NOPROGRESS throttled tasks.
*/
if (sc->nr_reclaimed > (sc->nr_scanned >> 3)) {
wait_queue_head_t *wqh;
wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS];
if (waitqueue_active(wqh))
wake_up(wqh);
return;
}
/*
* Do not throttle kswapd or cgroup reclaim on NOPROGRESS as it will
* throttle on VMSCAN_THROTTLE_WRITEBACK if there are too many pages
* under writeback and marked for immediate reclaim at the tail of the
* LRU.
*/
if (current_is_kswapd() || cgroup_reclaim(sc))
return;
/* Throttle if making no progress at high prioities. */
if (sc->priority == 1 && !sc->nr_reclaimed)
reclaim_throttle(pgdat, VMSCAN_THROTTLE_NOPROGRESS);
}
v6.18 reclaim_throttle
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.18, mm/vmscan.c:540-607.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.18/mm/vmscan.c#L540
* License text: GPL-2.0.txt (in this directory).
*/
void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)
{
wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];
long timeout, ret;
DEFINE_WAIT(wait);
/*
* Do not throttle user workers, kthreads other than kswapd or
* workqueues. They may be required for reclaim to make
* forward progress (e.g. journalling workqueues or kthreads).
*/
if (!current_is_kswapd() &&
current->flags & (PF_USER_WORKER|PF_KTHREAD)) {
cond_resched();
return;
}
/*
* These figures are pulled out of thin air.
* VMSCAN_THROTTLE_ISOLATED is a transient condition based on too many
* parallel reclaimers which is a short-lived event so the timeout is
* short. Failing to make progress or waiting on writeback are
* potentially long-lived events so use a longer timeout. This is shaky
* logic as a failure to make progress could be due to anything from
* writeback to a slow device to excessive referenced folios at the tail
* of the inactive LRU.
*/
switch(reason) {
case VMSCAN_THROTTLE_WRITEBACK:
timeout = HZ/10;
if (atomic_inc_return(&pgdat->nr_writeback_throttled) == 1) {
WRITE_ONCE(pgdat->nr_reclaim_start,
node_page_state(pgdat, NR_THROTTLED_WRITTEN));
}
break;
case VMSCAN_THROTTLE_CONGESTED:
fallthrough;
case VMSCAN_THROTTLE_NOPROGRESS:
if (skip_throttle_noprogress(pgdat)) {
cond_resched();
return;
}
timeout = 1;
break;
case VMSCAN_THROTTLE_ISOLATED:
timeout = HZ/50;
break;
default:
WARN_ON_ONCE(1);
timeout = HZ;
break;
}
prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
ret = schedule_timeout(timeout);
finish_wait(wqh, &wait);
if (reason == VMSCAN_THROTTLE_WRITEBACK)
atomic_dec(&pgdat->nr_writeback_throttled);
trace_mm_vmscan_throttled(pgdat->node_id, jiffies_to_usecs(timeout),
jiffies_to_usecs(timeout - ret),
reason);
}
v6.18 skip_throttle_noprogress
v6_18_skip_throttle_noprogress.c
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.18, mm/vmscan.c:512-538.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.18/mm/vmscan.c#L512
* License text: GPL-2.0.txt (in this directory).
*/
static bool skip_throttle_noprogress(pg_data_t *pgdat)
{
int reclaimable = 0, write_pending = 0;
int i;
struct zone *zone;
/*
* If kswapd is disabled, reschedule if necessary but do not
* throttle as the system is likely near OOM.
*/
if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES)
return true;
/*
* If there are a lot of dirty/writeback folios then do not
* throttle as throttling will occur when the folios cycle
* towards the end of the LRU if still under writeback.
*/
for_each_managed_zone_pgdat(zone, pgdat, i, MAX_NR_ZONES - 1) {
reclaimable += zone_reclaimable_pages(zone);
write_pending += zone_page_state_snapshot(zone,
NR_ZONE_WRITE_PENDING);
}
if (2 * write_pending <= reclaimable)
return true;
return false;
}
v6.18 allow_direct_reclaim
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.18, mm/vmscan.c:6487-6521.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.18/mm/vmscan.c#L6487
* License text: GPL-2.0.txt (in this directory).
*/
static bool allow_direct_reclaim(pg_data_t *pgdat)
{
struct zone *zone;
unsigned long pfmemalloc_reserve = 0;
unsigned long free_pages = 0;
int i;
bool wmark_ok;
if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES)
return true;
for_each_managed_zone_pgdat(zone, pgdat, i, ZONE_NORMAL) {
if (!zone_reclaimable_pages(zone) && zone_page_state_snapshot(zone, NR_FREE_PAGES))
continue;
pfmemalloc_reserve += min_wmark_pages(zone);
free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES);
}
/* If there are no reserves (unexpected config) then do not throttle */
if (!pfmemalloc_reserve)
return true;
wmark_ok = free_pages > pfmemalloc_reserve / 2;
/* kswapd must be awake if processes are being throttled */
if (!wmark_ok && waitqueue_active(&pgdat->kswapd_wait)) {
if (READ_ONCE(pgdat->kswapd_highest_zoneidx) > ZONE_NORMAL)
WRITE_ONCE(pgdat->kswapd_highest_zoneidx, ZONE_NORMAL);
wake_up_interruptible(&pgdat->kswapd_wait);
}
return wmark_ok;
}
v6.18 throttle_direct_reclaim
v6_18_throttle_direct_reclaim.c
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
* Swap reorganised 29.12.95, Stephen Tweedie.
* kswapd added: 7.1.96 sct
* Removed kswapd_ctl limits, and swap out as many pages as needed
* to bring the system back to freepages.high: 2.4.97, Rik van Riel.
* Zone aware kswapd started 02/00, Kanoj Sarcar (kanoj@sgi.com).
* Multiqueue VM started 5.8.00, Rik van Riel.
*/
/* Extracted unchanged from Linux v6.18, mm/vmscan.c:6532-6610.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.18/mm/vmscan.c#L6532
* License text: GPL-2.0.txt (in this directory).
*/
static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
nodemask_t *nodemask)
{
struct zoneref *z;
struct zone *zone;
pg_data_t *pgdat = NULL;
/*
* Kernel threads should not be throttled as they may be indirectly
* responsible for cleaning pages necessary for reclaim to make forward
* progress. kjournald for example may enter direct reclaim while
* committing a transaction where throttling it could forcing other
* processes to block on log_wait_commit().
*/
if (current->flags & PF_KTHREAD)
goto out;
/*
* If a fatal signal is pending, this process should not throttle.
* It should return quickly so it can exit and free its memory
*/
if (fatal_signal_pending(current))
goto out;
/*
* Check if the pfmemalloc reserves are ok by finding the first node
* with a usable ZONE_NORMAL or lower zone. The expectation is that
* GFP_KERNEL will be required for allocating network buffers when
* swapping over the network so ZONE_HIGHMEM is unusable.
*
* Throttling is based on the first usable node and throttled processes
* wait on a queue until kswapd makes progress and wakes them. There
* is an affinity then between processes waking up and where reclaim
* progress has been made assuming the process wakes on the same node.
* More importantly, processes running on remote nodes will not compete
* for remote pfmemalloc reserves and processes on different nodes
* should make reasonable progress.
*/
for_each_zone_zonelist_nodemask(zone, z, zonelist,
gfp_zone(gfp_mask), nodemask) {
if (zone_idx(zone) > ZONE_NORMAL)
continue;
/* Throttle based on the first usable node */
pgdat = zone->zone_pgdat;
if (allow_direct_reclaim(pgdat))
goto out;
break;
}
/* If no zone was usable by the allocation flags then do not throttle */
if (!pgdat)
goto out;
/* Account for the throttling */
count_vm_event(PGSCAN_DIRECT_THROTTLE);
/*
* If the caller cannot enter the filesystem, it's possible that it
* is due to the caller holding an FS lock or performing a journal
* transaction in the case of a filesystem like ext[3|4]. In this case,
* it is not safe to block on pfmemalloc_wait as kswapd could be
* blocked waiting on the same lock. Instead, throttle for up to a
* second before continuing.
*/
if (!(gfp_mask & __GFP_FS))
wait_event_interruptible_timeout(pgdat->pfmemalloc_wait,
allow_direct_reclaim(pgdat), HZ);
else
/* Throttle until kswapd wakes the process */
wait_event_killable(zone->zone_pgdat->pfmemalloc_wait,
allow_direct_reclaim(pgdat));
if (fatal_signal_pending(current))
return true;
out:
return false;
}
v6.18 wake_throttle_isolated
v6_18_wake_throttle_isolated.c
/* SPDX-License-Identifier: GPL-2.0-or-later */
/* internal.h: mm/ internal definitions
*
* Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
* Written by David Howells (dhowells@redhat.com)
*/
/* Extracted unchanged from Linux v6.18, mm/internal.h:420-427.
* Complete function definition; surrounding kernel context is required.
* Source: https://github.com/torvalds/linux/blob/v6.18/mm/internal.h#L420
* License text: GPL-2.0.txt (in this directory).
*/
static inline void wake_throttle_isolated(pg_data_t *pgdat)
{
wait_queue_head_t *wqh;
wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED];
if (waitqueue_active(wqh))
wake_up(wqh);
}
참고 자료
- Lorenzo Stoakes, The Linux Memory Manager, Early Access 2025, Chapter 11, Reclaim Throttling.
- Linux v6.18 mm/vmscan.c: throttle queue, wakeup accounting, direct-reclaim reserve 판단.