swapped PTE에 접근하면 kernel은 내용을 찾고, 사용할 수 있는지 검증하고, page table을 복구합니다. cache에 folio가 있다는 사실은 첫 단계의 일부만 해결합니다. 내용 준비, entry identity, CoW ownership, write permission은 별도로 확인해야 합니다. 여기서는 v6.12 do_swap_page()를 기준으로 설명합니다.
먼저 entry 종류와 area lifetime을 확인합니다
non-present software PTE가 모두 ordinary swap area를 뜻하지는 않습니다. migration·device-private·hardware-poison·PTE marker는 각자의 처리로 갑니다. ordinary swap이라면 get_swap_device()로 area metadata를 안전하게 사용할 수 있는 기간을 확보합니다. 이 보호를 swapoff 전체가 영원히 금지된다는 뜻으로 확대하면 안 됩니다.
swap_cache_get_folio()가 folio를 반환하면 caller는 object의 reference를 얻습니다. folio lock을 얻은 것은 아닙니다. cache miss라면 direct read 또는 readahead 경로에서 folio를 마련합니다. 새 folio가 있어도 read I/O가 아직 끝나지 않았다면 uptodate하지 않을 수 있습니다.
Cache를 우회하는 read에도 slot 보호가 필요합니다
SWP_SYNCHRONOUS_IO 성격의 area이고 swap 참조 수가 하나인 경우에는 새 folio를 swap cache에 삽입하지 않고 직접 읽는 최적화가 있습니다. 그러나 swapcache_prepare()로 해당 entry의 cache 보유 상태를 임시 확보합니다.
이 보호가 없다면 다른 thread가 먼저 swap-in을 끝내고 slot을 반환한 뒤 새 swap-out이 같은 type·offset을 재사용할 수 있습니다. 원래 PTE와 값이 같아도 내용은 다른 ABA 상황입니다. pte_same()만으로 검출할 수 없으므로 direct read도 slot lifetime protocol을 생략하지 않습니다. 이 임시 pin은 PTE lock을 해제한 뒤 clear하고 waiter를 깨우는 순서를 지켜야 합니다.
Allocation 실패도 PTE를 다시 보고 판단합니다
folio를 마련하지 못했다면 PTE lock 아래 original PTE와 현재 값을 비교합니다. 다른 thread가 이미 fault를 해결했다면 자신의 allocation 실패를 그 주소의 OOM으로 보고해서는 안 됩니다. original swap PTE가 그대로일 때 VM_FAULT_OOM을 보고할 수 있습니다.
cache-miss 경로의 major-fault accounting도 physical disk read 횟수와 동일하지 않습니다. compressed·RAM backend, 이미 시작된 read와의 경쟁 등이 있으므로 major라는 이름만으로 어떤 storage I/O가 발생했는지 확정할 수 없습니다.
Reference, lock, uptodate는 세 가지 보장입니다
folio_lock_or_retry()는 fault의 lock·retry protocol에 따라 기다리거나 상위 fault handler에 retry를 요청합니다. I/O가 끝나면 folio가 unlock되어 내용 준비를 관찰할 수 있지만, lock을 얻었다는 사실만으로 read 성공을 보장하지는 않습니다. uptodate를 확인해야 합니다.
cache에서 찾은 folio라면 아직 swap cache인지, 여전히 같은 entry에 연결되어 있는지 재검사합니다. swapoff나 cache 제거가 그 사이 상태를 바꿀 수 있습니다. 그 뒤 PTE lock을 얻어 pte_same()을 다시 검사합니다. reference는 object 수명을 보호하고, folio lock은 folio 상태를 보호하며, PTE lock은 지금 복구할 mapping을 보호합니다.
uptodate가 아니면 내용 대신 조용히 zero page를 설치하지 않고 적절한 fault error를 반환합니다. KSM은 read fault에서도 복사가 필요할 수 있으므로 ordinary folio 재사용과 별도의 branch가 있습니다. large folio를 여러 PTE로 복구하는 경우에도 전체 범위가 여전히 같은 entry sequence인지 확인해야 합니다.
Exclusive라도 writable이 자동으로 되지는 않습니다
swap PTE의 exclusive 상태, cache와 다른 새 folio인지, refcount로 독점 사용을 확인할 수 있는지를 조합합니다. pending LRU batch가 가진 임시 reference가 독점 판정을 방해할 수 있어 필요한 경우 drain합니다. 이것은 외부 pin을 무시하려는 처리가 아니라 끝난 pending operation의 reference를 정리하는 것입니다.
SWP_STABLE_WRITES backend에서 folio가 writeback 중이면 exclusive marker가 있어도 이를 writable 재사용 근거로 쓰지 않을 수 있습니다. I/O가 읽는 memory를 동시에 수정하지 못하도록 하기 위해서입니다. anon-exclusive는 같은 mm의 CPU가 하나라는 뜻이 아닙니다.
안전하게 재사용할 수 있더라도 VMA의 write 권한, userfaultfd write protection, soft-dirty tracking 요구를 충족해야 writable PTE를 만들 수 있습니다. read fault에서도 독점과 permission 조건이 맞으면 미리 writable하게 만들 수 있으므로 'writable 복구는 write fault에서만 가능'하다는 절대 규칙도 맞지 않습니다. 처리되지 않은 write fault는 do_wp_page()에서 CoW를 추가 수행할 수 있습니다.
PTE 복구와 slot 정리는 함께 하되 같은 일은 아닙니다
성공 경로는 architecture metadata를 복구한 뒤 swap PTE 참조 accounting을 줄이고, 조건이 맞으면 cache를 정리합니다. should_try_to_free_swap()은 swap cache folio인지 먼저 확인한 뒤 swap pressure, locked VMA·mlocked 상태를 봅니다. write fault에서 KSM이 아니며 reference가 caller와 cache 보유분인 1 + folio_nr_pages()에 해당하는 경우도 제거 시도에 유리합니다. 이 helper는 시도할 이유를 고를 뿐, folio_free_swap()의 writeback·남은 swap 참조 검사를 면제하지 않습니다.
이어 RSS와 swap-entry 수를 바꾸고 soft-dirty·uffd-wp를 반영한 present PTE, rmap과 folio reference를 설치합니다.
swap_free_nr()가 해당 PTE의 참조를 줄여도 cache 보유나 다른 swap PTE가 남으면 slot은 즉시 재사용되지 않습니다. writeback 중 slot이 무조건 풀린다고 단정해서도 안 됩니다. direct-read 임시 cache pin도 별도로 남아 있을 수 있습니다.
error·retry 경로는 어느 lock과 reference를 확보했는지에 따라 정리합니다. PTE lock, folio lock, folio reference, 별도 swapcache folio, swap-device 보호, direct-read cache pin을 획득한 만큼 반환해야 합니다. 성공 경로의 중간에서 빠져나갔다고 나머지 정리를 생략하면 lifetime이 깨집니다.
고전 경로와 v6.12 경로를 따로 읽습니다
아래 두 전체 함수는 각각 official v6.0과 v6.12입니다. 본문의 기준은 v6.12이며, v6.0은 page 중심 API와 단일-PTE 복구 경로를 비교하기 위한 code입니다. upstream notice와 GPL-2.0을 함께 제공합니다.
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/memory.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* demand-loading started 01.12.91 - seems it is high on the list of
* things wanted, and it should be easy to implement. - Linus
*/
/*
* Ok, demand-loading was easy, shared pages a little bit tricker. Shared
* pages started 02.12.91, seems to work. - Linus.
*
* Tested sharing by executing about 30 /bin/sh: under the old kernel it
* would have taken more than the 6M I have free, but it worked well as
* far as I could see.
*
* Also corrected some "invalidate()"s - I wasn't doing enough of them.
*/
/*
* Real VM (paging to/from disk) started 18.12.91. Much more work and
* thought has to go into this. Oh, well..
* 19.12.91 - works, somewhat. Sometimes I get faults, don't know why.
* Found it. Everything seems to work now.
* 20.12.91 - Ok, making the swap-device changeable like the root.
*/
/*
* 05.04.94 - Multi-page memory management added for v1.1.
* Idea by Alex Bligh (alex@cconcepts.co.uk)
*
* 16.07.99 - Support of BIGMEM added by Gerhard Wichert, Siemens AG
* (Gerhard.Wichert@pdb.siemens.de)
*
* Aug/Sep 2004 Changed to four level page tables (Andi Kleen)
*/
/*
* Official Linux v6.0: mm/memory.c
* https://github.com/torvalds/linux/blob/v6.0/mm/memory.c#L3718
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
vm_fault_t do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL, *swapcache;
struct swap_info_struct *si = NULL;
rmap_t rmap_flags = RMAP_NONE;
bool exclusive = false;
swp_entry_t entry;
pte_t pte;
int locked;
vm_fault_t ret = 0;
void *shadow = NULL;
if (!pte_unmap_same(vmf))
goto out;
entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
vmf->address);
} else if (is_device_exclusive_entry(entry)) {
vmf->page = pfn_swap_entry_to_page(entry);
ret = remove_device_exclusive_entry(vmf);
} else if (is_device_private_entry(entry)) {
vmf->page = pfn_swap_entry_to_page(entry);
ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else if (is_swapin_error_entry(entry)) {
ret = VM_FAULT_SIGBUS;
} else if (is_pte_marker_entry(entry)) {
ret = handle_pte_marker(vmf);
} else {
print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
}
/* Prevent swapoff from happening to us. */
si = get_swap_device(entry);
if (unlikely(!si))
goto out;
page = lookup_swap_cache(entry, vma, vmf->address);
swapcache = page;
if (!page) {
if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
__swap_count(entry) == 1) {
/* skip swapcache */
page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma,
vmf->address);
if (page) {
__SetPageLocked(page);
__SetPageSwapBacked(page);
if (mem_cgroup_swapin_charge_page(page,
vma->vm_mm, GFP_KERNEL, entry)) {
ret = VM_FAULT_OOM;
goto out_page;
}
mem_cgroup_swapin_uncharge_swap(entry);
shadow = get_shadow_from_swap_cache(entry);
if (shadow)
workingset_refault(page_folio(page),
shadow);
lru_cache_add(page);
/* To provide entry to swap_readpage() */
set_page_private(page, entry.val);
swap_readpage(page, true, NULL);
set_page_private(page, 0);
}
} else {
page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE,
vmf);
swapcache = page;
}
if (!page) {
/*
* Back out if somebody else faulted in this pte
* while we released the pte lock.
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
goto unlock;
}
/* Had to read the page from swap area: Major fault */
ret = VM_FAULT_MAJOR;
count_vm_event(PGMAJFAULT);
count_memcg_event_mm(vma->vm_mm, PGMAJFAULT);
} else if (PageHWPoison(page)) {
/*
* hwpoisoned dirty swapcache pages are kept for killing
* owner processes (which may be unknown at hwpoison time)
*/
ret = VM_FAULT_HWPOISON;
goto out_release;
}
locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags);
if (!locked) {
ret |= VM_FAULT_RETRY;
goto out_release;
}
if (swapcache) {
/*
* Make sure try_to_free_swap or swapoff did not release the
* swapcache from under us. The page pin, and pte_same test
* below, are not enough to exclude that. Even if it is still
* swapcache, we need to check that the page's swap has not
* changed.
*/
if (unlikely(!PageSwapCache(page) ||
page_private(page) != entry.val))
goto out_page;
/*
* KSM sometimes has to copy on read faults, for example, if
* page->index of !PageKSM() pages would be nonlinear inside the
* anon VMA -- PageKSM() is lost on actual swapout.
*/
page = ksm_might_need_to_copy(page, vma, vmf->address);
if (unlikely(!page)) {
ret = VM_FAULT_OOM;
page = swapcache;
goto out_page;
}
/*
* If we want to map a page that's in the swapcache writable, we
* have to detect via the refcount if we're really the exclusive
* owner. Try removing the extra reference from the local LRU
* pagevecs if required.
*/
if ((vmf->flags & FAULT_FLAG_WRITE) && page == swapcache &&
!PageKsm(page) && !PageLRU(page))
lru_add_drain();
}
cgroup_throttle_swaprate(page, GFP_KERNEL);
/*
* Back out if somebody else already faulted in this pte.
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
ret = VM_FAULT_SIGBUS;
goto out_nomap;
}
/*
* PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte
* must never point at an anonymous page in the swapcache that is
* PG_anon_exclusive. Sanity check that this holds and especially, that
* no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity
* check after taking the PT lock and making sure that nobody
* concurrently faulted in this page and set PG_anon_exclusive.
*/
BUG_ON(!PageAnon(page) && PageMappedToDisk(page));
BUG_ON(PageAnon(page) && PageAnonExclusive(page));
/*
* Check under PT lock (to protect against concurrent fork() sharing
* the swap entry concurrently) for certainly exclusive pages.
*/
if (!PageKsm(page)) {
/*
* Note that pte_swp_exclusive() == false for architectures
* without __HAVE_ARCH_PTE_SWP_EXCLUSIVE.
*/
exclusive = pte_swp_exclusive(vmf->orig_pte);
if (page != swapcache) {
/*
* We have a fresh page that is not exposed to the
* swapcache -> certainly exclusive.
*/
exclusive = true;
} else if (exclusive && PageWriteback(page) &&
data_race(si->flags & SWP_STABLE_WRITES)) {
/*
* This is tricky: not all swap backends support
* concurrent page modifications while under writeback.
*
* So if we stumble over such a page in the swapcache
* we must not set the page exclusive, otherwise we can
* map it writable without further checks and modify it
* while still under writeback.
*
* For these problematic swap backends, simply drop the
* exclusive marker: this is perfectly fine as we start
* writeback only if we fully unmapped the page and
* there are no unexpected references on the page after
* unmapping succeeded. After fully unmapped, no
* further GUP references (FOLL_GET and FOLL_PIN) can
* appear, so dropping the exclusive marker and mapping
* it only R/O is fine.
*/
exclusive = false;
}
}
/*
* Remove the swap entry and conditionally try to free up the swapcache.
* We're already holding a reference on the page but haven't mapped it
* yet.
*/
swap_free(entry);
if (should_try_to_free_swap(page, vma, vmf->flags))
try_to_free_swap(page);
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS);
pte = mk_pte(page, vma->vm_page_prot);
/*
* Same logic as in do_wp_page(); however, optimize for pages that are
* certainly not shared either because we just allocated them without
* exposing them to the swapcache or because the swap entry indicates
* exclusivity.
*/
if (!PageKsm(page) && (exclusive || page_count(page) == 1)) {
if (vmf->flags & FAULT_FLAG_WRITE) {
pte = maybe_mkwrite(pte_mkdirty(pte), vma);
vmf->flags &= ~FAULT_FLAG_WRITE;
ret |= VM_FAULT_WRITE;
}
rmap_flags |= RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
if (pte_swp_uffd_wp(vmf->orig_pte)) {
pte = pte_mkuffd_wp(pte);
pte = pte_wrprotect(pte);
}
vmf->orig_pte = pte;
/* ksm created a completely new copy */
if (unlikely(page != swapcache && swapcache)) {
page_add_new_anon_rmap(page, vma, vmf->address);
lru_cache_add_inactive_or_unevictable(page, vma);
} else {
page_add_anon_rmap(page, vma, vmf->address, rmap_flags);
}
VM_BUG_ON(!PageAnon(page) || (pte_write(pte) && !PageAnonExclusive(page)));
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte);
unlock_page(page);
if (page != swapcache && swapcache) {
/*
* Hold the lock to avoid the swap entry to be reused
* until we take the PT lock for the pte_same() check
* (to avoid false positives from pte_same). For
* further safety release the lock after the swap_free
* so that the swap count won't change under a
* parallel locked swapcache.
*/
unlock_page(swapcache);
put_page(swapcache);
}
if (vmf->flags & FAULT_FLAG_WRITE) {
ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
}
/* No need to invalidate - it was non-present before */
update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
if (si)
put_swap_device(si);
return ret;
out_nomap:
pte_unmap_unlock(vmf->pte, vmf->ptl);
out_page:
unlock_page(page);
out_release:
put_page(page);
if (page != swapcache && swapcache) {
unlock_page(swapcache);
put_page(swapcache);
}
if (si)
put_swap_device(si);
return ret;
}
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/memory.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* demand-loading started 01.12.91 - seems it is high on the list of
* things wanted, and it should be easy to implement. - Linus
*/
/*
* Ok, demand-loading was easy, shared pages a little bit tricker. Shared
* pages started 02.12.91, seems to work. - Linus.
*
* Tested sharing by executing about 30 /bin/sh: under the old kernel it
* would have taken more than the 6M I have free, but it worked well as
* far as I could see.
*
* Also corrected some "invalidate()"s - I wasn't doing enough of them.
*/
/*
* Real VM (paging to/from disk) started 18.12.91. Much more work and
* thought has to go into this. Oh, well..
* 19.12.91 - works, somewhat. Sometimes I get faults, don't know why.
* Found it. Everything seems to work now.
* 20.12.91 - Ok, making the swap-device changeable like the root.
*/
/*
* 05.04.94 - Multi-page memory management added for v1.1.
* Idea by Alex Bligh (alex@cconcepts.co.uk)
*
* 16.07.99 - Support of BIGMEM added by Gerhard Wichert, Siemens AG
* (Gerhard.Wichert@pdb.siemens.de)
*
* Aug/Sep 2004 Changed to four level page tables (Andi Kleen)
*/
/*
* Official Linux v6.12: mm/memory.c
* https://github.com/torvalds/linux/blob/v6.12/mm/memory.c#L4200
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
vm_fault_t do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct folio *swapcache, *folio = NULL;
DECLARE_WAITQUEUE(wait, current);
struct page *page;
struct swap_info_struct *si = NULL;
rmap_t rmap_flags = RMAP_NONE;
bool need_clear_cache = false;
bool exclusive = false;
swp_entry_t entry;
pte_t pte;
vm_fault_t ret = 0;
void *shadow = NULL;
int nr_pages;
unsigned long page_idx;
unsigned long address;
pte_t *ptep;
if (!pte_unmap_same(vmf))
goto out;
entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
vmf->address);
} else if (is_device_exclusive_entry(entry)) {
vmf->page = pfn_swap_entry_to_page(entry);
ret = remove_device_exclusive_entry(vmf);
} else if (is_device_private_entry(entry)) {
if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
/*
* migrate_to_ram is not yet ready to operate
* under VMA lock.
*/
vma_end_read(vma);
ret = VM_FAULT_RETRY;
goto out;
}
vmf->page = pfn_swap_entry_to_page(entry);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (unlikely(!vmf->pte ||
!pte_same(ptep_get(vmf->pte),
vmf->orig_pte)))
goto unlock;
/*
* Get a page reference while we know the page can't be
* freed.
*/
get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
put_page(vmf->page);
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else if (is_pte_marker_entry(entry)) {
ret = handle_pte_marker(vmf);
} else {
print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
}
/* Prevent swapoff from happening to us. */
si = get_swap_device(entry);
if (unlikely(!si))
goto out;
folio = swap_cache_get_folio(entry, vma, vmf->address);
if (folio)
page = folio_file_page(folio, swp_offset(entry));
swapcache = folio;
if (!folio) {
if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
__swap_count(entry) == 1) {
/* skip swapcache */
folio = alloc_swap_folio(vmf);
if (folio) {
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
nr_pages = folio_nr_pages(folio);
if (folio_test_large(folio))
entry.val = ALIGN_DOWN(entry.val, nr_pages);
/*
* Prevent parallel swapin from proceeding with
* the cache flag. Otherwise, another thread
* may finish swapin first, free the entry, and
* swapout reusing the same entry. It's
* undetectable as pte_same() returns true due
* to entry reuse.
*/
if (swapcache_prepare(entry, nr_pages)) {
/*
* Relax a bit to prevent rapid
* repeated page faults.
*/
add_wait_queue(&swapcache_wq, &wait);
schedule_timeout_uninterruptible(1);
remove_wait_queue(&swapcache_wq, &wait);
goto out_page;
}
need_clear_cache = true;
mem_cgroup_swapin_uncharge_swap(entry, nr_pages);
shadow = get_shadow_from_swap_cache(entry);
if (shadow)
workingset_refault(folio, shadow);
folio_add_lru(folio);
/* To provide entry to swap_read_folio() */
folio->swap = entry;
swap_read_folio(folio, NULL);
folio->private = NULL;
}
} else {
folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE,
vmf);
swapcache = folio;
}
if (!folio) {
/*
* Back out if somebody else faulted in this pte
* while we released the pte lock.
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (likely(vmf->pte &&
pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
ret = VM_FAULT_OOM;
goto unlock;
}
/* Had to read the page from swap area: Major fault */
ret = VM_FAULT_MAJOR;
count_vm_event(PGMAJFAULT);
count_memcg_event_mm(vma->vm_mm, PGMAJFAULT);
page = folio_file_page(folio, swp_offset(entry));
} else if (PageHWPoison(page)) {
/*
* hwpoisoned dirty swapcache pages are kept for killing
* owner processes (which may be unknown at hwpoison time)
*/
ret = VM_FAULT_HWPOISON;
goto out_release;
}
ret |= folio_lock_or_retry(folio, vmf);
if (ret & VM_FAULT_RETRY)
goto out_release;
if (swapcache) {
/*
* Make sure folio_free_swap() or swapoff did not release the
* swapcache from under us. The page pin, and pte_same test
* below, are not enough to exclude that. Even if it is still
* swapcache, we need to check that the page's swap has not
* changed.
*/
if (unlikely(!folio_test_swapcache(folio) ||
page_swap_entry(page).val != entry.val))
goto out_page;
/*
* KSM sometimes has to copy on read faults, for example, if
* page->index of !PageKSM() pages would be nonlinear inside the
* anon VMA -- PageKSM() is lost on actual swapout.
*/
folio = ksm_might_need_to_copy(folio, vma, vmf->address);
if (unlikely(!folio)) {
ret = VM_FAULT_OOM;
folio = swapcache;
goto out_page;
} else if (unlikely(folio == ERR_PTR(-EHWPOISON))) {
ret = VM_FAULT_HWPOISON;
folio = swapcache;
goto out_page;
}
if (folio != swapcache)
page = folio_page(folio, 0);
/*
* If we want to map a page that's in the swapcache writable, we
* have to detect via the refcount if we're really the exclusive
* owner. Try removing the extra reference from the local LRU
* caches if required.
*/
if ((vmf->flags & FAULT_FLAG_WRITE) && folio == swapcache &&
!folio_test_ksm(folio) && !folio_test_lru(folio))
lru_add_drain();
}
folio_throttle_swaprate(folio, GFP_KERNEL);
/*
* Back out if somebody else already faulted in this pte.
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
goto out_nomap;
if (unlikely(!folio_test_uptodate(folio))) {
ret = VM_FAULT_SIGBUS;
goto out_nomap;
}
/* allocated large folios for SWP_SYNCHRONOUS_IO */
if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
unsigned long nr = folio_nr_pages(folio);
unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
pte_t *folio_ptep = vmf->pte - idx;
pte_t folio_pte = ptep_get(folio_ptep);
if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) ||
swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
goto out_nomap;
page_idx = idx;
address = folio_start;
ptep = folio_ptep;
goto check_folio;
}
nr_pages = 1;
page_idx = 0;
address = vmf->address;
ptep = vmf->pte;
if (folio_test_large(folio) && folio_test_swapcache(folio)) {
int nr = folio_nr_pages(folio);
unsigned long idx = folio_page_idx(folio, page);
unsigned long folio_start = address - idx * PAGE_SIZE;
unsigned long folio_end = folio_start + nr * PAGE_SIZE;
pte_t *folio_ptep;
pte_t folio_pte;
if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start)))
goto check_folio;
if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end)))
goto check_folio;
folio_ptep = vmf->pte - idx;
folio_pte = ptep_get(folio_ptep);
if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) ||
swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
goto check_folio;
page_idx = idx;
address = folio_start;
ptep = folio_ptep;
nr_pages = nr;
entry = folio->swap;
page = &folio->page;
}
check_folio:
/*
* PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte
* must never point at an anonymous page in the swapcache that is
* PG_anon_exclusive. Sanity check that this holds and especially, that
* no filesystem set PG_mappedtodisk on a page in the swapcache. Sanity
* check after taking the PT lock and making sure that nobody
* concurrently faulted in this page and set PG_anon_exclusive.
*/
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));
BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));
/*
* Check under PT lock (to protect against concurrent fork() sharing
* the swap entry concurrently) for certainly exclusive pages.
*/
if (!folio_test_ksm(folio)) {
exclusive = pte_swp_exclusive(vmf->orig_pte);
if (folio != swapcache) {
/*
* We have a fresh page that is not exposed to the
* swapcache -> certainly exclusive.
*/
exclusive = true;
} else if (exclusive && folio_test_writeback(folio) &&
data_race(si->flags & SWP_STABLE_WRITES)) {
/*
* This is tricky: not all swap backends support
* concurrent page modifications while under writeback.
*
* So if we stumble over such a page in the swapcache
* we must not set the page exclusive, otherwise we can
* map it writable without further checks and modify it
* while still under writeback.
*
* For these problematic swap backends, simply drop the
* exclusive marker: this is perfectly fine as we start
* writeback only if we fully unmapped the page and
* there are no unexpected references on the page after
* unmapping succeeded. After fully unmapped, no
* further GUP references (FOLL_GET and FOLL_PIN) can
* appear, so dropping the exclusive marker and mapping
* it only R/O is fine.
*/
exclusive = false;
}
}
/*
* Some architectures may have to restore extra metadata to the page
* when reading from swap. This metadata may be indexed by swap entry
* so this must be called before swap_free().
*/
arch_swap_restore(folio_swap(entry, folio), folio);
/*
* Remove the swap entry and conditionally try to free up the swapcache.
* We're already holding a reference on the page but haven't mapped it
* yet.
*/
swap_free_nr(entry, nr_pages);
if (should_try_to_free_swap(folio, vma, vmf->flags))
folio_free_swap(folio);
add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);
add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages);
pte = mk_pte(page, vma->vm_page_prot);
if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
if (pte_swp_uffd_wp(vmf->orig_pte))
pte = pte_mkuffd_wp(pte);
/*
* Same logic as in do_wp_page(); however, optimize for pages that are
* certainly not shared either because we just allocated them without
* exposing them to the swapcache or because the swap entry indicates
* exclusivity.
*/
if (!folio_test_ksm(folio) &&
(exclusive || folio_ref_count(folio) == 1)) {
if ((vma->vm_flags & VM_WRITE) && !userfaultfd_pte_wp(vma, pte) &&
!pte_needs_soft_dirty_wp(vma, pte)) {
pte = pte_mkwrite(pte, vma);
if (vmf->flags & FAULT_FLAG_WRITE) {
pte = pte_mkdirty(pte);
vmf->flags &= ~FAULT_FLAG_WRITE;
}
}
rmap_flags |= RMAP_EXCLUSIVE;
}
folio_ref_add(folio, nr_pages - 1);
flush_icache_pages(vma, page, nr_pages);
vmf->orig_pte = pte_advance_pfn(pte, page_idx);
/* ksm created a completely new copy */
if (unlikely(folio != swapcache && swapcache)) {
folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
folio_add_lru_vma(folio, vma);
} else if (!folio_test_anon(folio)) {
/*
* We currently only expect small !anon folios which are either
* fully exclusive or fully shared, or new allocated large
* folios which are fully exclusive. If we ever get large
* folios within swapcache here, we have to be careful.
*/
VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
} else {
folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address,
rmap_flags);
}
VM_BUG_ON(!folio_test_anon(folio) ||
(pte_write(pte) && !PageAnonExclusive(page)));
set_ptes(vma->vm_mm, address, ptep, pte, nr_pages);
arch_do_swap_page_nr(vma->vm_mm, vma, address,
pte, pte, nr_pages);
folio_unlock(folio);
if (folio != swapcache && swapcache) {
/*
* Hold the lock to avoid the swap entry to be reused
* until we take the PT lock for the pte_same() check
* (to avoid false positives from pte_same). For
* further safety release the lock after the swap_free
* so that the swap count won't change under a
* parallel locked swapcache.
*/
folio_unlock(swapcache);
folio_put(swapcache);
}
if (vmf->flags & FAULT_FLAG_WRITE) {
ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
}
/* No need to invalidate - it was non-present before */
update_mmu_cache_range(vmf, vma, address, ptep, nr_pages);
unlock:
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
/* Clear the swap cache pin for direct swapin after PTL unlock */
if (need_clear_cache) {
swapcache_clear(si, entry, nr_pages);
if (waitqueue_active(&swapcache_wq))
wake_up(&swapcache_wq);
}
if (si)
put_swap_device(si);
return ret;
out_nomap:
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
out_page:
folio_unlock(folio);
out_release:
folio_put(folio);
if (folio != swapcache && swapcache) {
folio_unlock(swapcache);
folio_put(swapcache);
}
if (need_clear_cache) {
swapcache_clear(si, entry, nr_pages);
if (waitqueue_active(&swapcache_wq))
wake_up(&swapcache_wq);
}
if (si)
put_swap_device(si);
return ret;
}
v6.0의 direct read는 임시 page_private()로 entry를 전달하는 형태입니다. 이 tag에는 v6.12의 need_clear_cache reservation protocol이 없으므로 두 함수의 direct path를 같은 구현으로 설명하지 않습니다. v6.12는 area reference 아래 swapcache_prepare(entry, nr_pages)를 확보하며 경쟁이면 wait queue에 잠시 등록해 쉬고 종료합니다. 확보한 cache pin만 뒤의 cleanup에서 clear하고 waiter를 깨웁니다.
v6.12 large direct folio는 allocation 크기에 맞춰 entry를 정렬하고, PTL 아래 전체 PTE sequence가 예상한 연속 swap offset인지 재검사합니다. cache-hit large folio를 넓게 복구할 때도 VMA·PMD 경계와 sequence가 맞아야 합니다. 그 검사가 안 되면 cache folio 전체를 무조건 매핑하지 않고 단일 page 복구를 유지합니다. direct large folio는 잘못된 sequence를 발견하면 mapping을 설치하지 않습니다.
재사용 판정과 남은 write fault는 다릅니다
두 tag의 cache 제거 시도 helper도 나란히 읽어 보십시오. v6.0의 작은 page 기준 count 2를 모든 large folio의 고정값이라고 일반화하면 안 됩니다. v6.12는 1 + folio_nr_pages()와 비교합니다.
v6_0_should_try_to_free_swap.c
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/memory.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* demand-loading started 01.12.91 - seems it is high on the list of
* things wanted, and it should be easy to implement. - Linus
*/
/*
* Ok, demand-loading was easy, shared pages a little bit tricker. Shared
* pages started 02.12.91, seems to work. - Linus.
*
* Tested sharing by executing about 30 /bin/sh: under the old kernel it
* would have taken more than the 6M I have free, but it worked well as
* far as I could see.
*
* Also corrected some "invalidate()"s - I wasn't doing enough of them.
*/
/*
* Real VM (paging to/from disk) started 18.12.91. Much more work and
* thought has to go into this. Oh, well..
* 19.12.91 - works, somewhat. Sometimes I get faults, don't know why.
* Found it. Everything seems to work now.
* 20.12.91 - Ok, making the swap-device changeable like the root.
*/
/*
* 05.04.94 - Multi-page memory management added for v1.1.
* Idea by Alex Bligh (alex@cconcepts.co.uk)
*
* 16.07.99 - Support of BIGMEM added by Gerhard Wichert, Siemens AG
* (Gerhard.Wichert@pdb.siemens.de)
*
* Aug/Sep 2004 Changed to four level page tables (Andi Kleen)
*/
/*
* Official Linux v6.0: mm/memory.c
* https://github.com/torvalds/linux/blob/v6.0/mm/memory.c#L3638
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static inline bool should_try_to_free_swap(struct page *page,
struct vm_area_struct *vma,
unsigned int fault_flags)
{
if (!PageSwapCache(page))
return false;
if (mem_cgroup_swap_full(page) || (vma->vm_flags & VM_LOCKED) ||
PageMlocked(page))
return true;
/*
* If we want to map a page that's in the swapcache writable, we
* have to detect via the refcount if we're really the exclusive
* user. Try freeing the swapcache to get rid of the swapcache
* reference only in case it's likely that we'll be the exlusive user.
*/
return (fault_flags & FAULT_FLAG_WRITE) && !PageKsm(page) &&
page_count(page) == 2;
}
v6_12_should_try_to_free_swap.c
// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/memory.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* demand-loading started 01.12.91 - seems it is high on the list of
* things wanted, and it should be easy to implement. - Linus
*/
/*
* Ok, demand-loading was easy, shared pages a little bit tricker. Shared
* pages started 02.12.91, seems to work. - Linus.
*
* Tested sharing by executing about 30 /bin/sh: under the old kernel it
* would have taken more than the 6M I have free, but it worked well as
* far as I could see.
*
* Also corrected some "invalidate()"s - I wasn't doing enough of them.
*/
/*
* Real VM (paging to/from disk) started 18.12.91. Much more work and
* thought has to go into this. Oh, well..
* 19.12.91 - works, somewhat. Sometimes I get faults, don't know why.
* Found it. Everything seems to work now.
* 20.12.91 - Ok, making the swap-device changeable like the root.
*/
/*
* 05.04.94 - Multi-page memory management added for v1.1.
* Idea by Alex Bligh (alex@cconcepts.co.uk)
*
* 16.07.99 - Support of BIGMEM added by Gerhard Wichert, Siemens AG
* (Gerhard.Wichert@pdb.siemens.de)
*
* Aug/Sep 2004 Changed to four level page tables (Andi Kleen)
*/
/*
* Official Linux v6.12: mm/memory.c
* https://github.com/torvalds/linux/blob/v6.12/mm/memory.c#L3921
* Exact definition/declaration; not a reconstruction of a book listing.
* See GPL-2.0.txt. Not a standalone translation unit.
*/
static inline bool should_try_to_free_swap(struct folio *folio,
struct vm_area_struct *vma,
unsigned int fault_flags)
{
if (!folio_test_swapcache(folio))
return false;
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
return true;
/*
* If we want to map a page that's in the swapcache writable, we
* have to detect via the refcount if we're really the exclusive
* user. Try freeing the swapcache to get rid of the swapcache
* reference only in case it's likely that we'll be the exlusive user.
*/
return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) &&
folio_ref_count(folio) == (1 + folio_nr_pages(folio));
}
v6.12는 architecture metadata를 slot 참조 감소 전에 복구합니다. 이어 복구할 page 수만큼 swap 참조와 RSS 통계를 바꾸고, present PTE의 soft-dirty·uffd-wp를 구성합니다. 재사용 가능한 non-KSM folio여도 writable 조건은 VM_WRITE, userfaultfd 보호, soft-dirty write-protect 요구를 따릅니다. read fault에서도 이 조건이 맞으면 writable을 미리 허용할 수 있지만, write fault가 끝내 처리되지 않았다면 do_wp_page()로 이어집니다.
성공 시 caller가 얻었던 folio reference는 설치한 mapping의 수명으로 이어지고, 여러 PTE를 설치하면 나머지 reference를 추가합니다. KSM copy로 새 folio가 생겼다면 별도의 anon rmap·LRU 준비를 하고, 옛 swapcache folio는 PTE 동일성 확인과 swap 참조 감소가 끝날 때까지 locked로 유지합니다. 새 folio만 살리고 옛 cache lock을 일찍 풀면 slot 재사용 방지가 약해질 수 있습니다.
Cleanup label은 확보한 자원의 역순을 보여 줍니다
PTE가 바뀌었거나 uptodate가 아니면 out_nomap에서 PTL을 정리하고, out_page에서 folio를 unlock하고, out_release에서 reference를 반환합니다. folio_lock_or_retry()가 retry를 반환한 경우에는 자신이 얻지 못한 folio lock을 unlock하지 않습니다. 별도 KSM swapcache가 있으면 그 lock과 reference도 따로 반환합니다.
성공 경로의 do_wp_page()는 자체 PTL 계약을 따르므로 그 뒤 단순히 공통 unlock label로 떨어뜨리지 않습니다. direct-read cache pin은 PTL이 풀린 뒤에 clear하고 device-user reference는 그 이후 반환합니다. 이 순서를 포함한 전체 함수가 있어야 goto out과 goto out_release를 같은 cleanup으로 오해하지 않을 수 있습니다.
참고 자료
- Lorenzo Stoakes, The Linux Memory Manager, Early Access 2025, Chapter 12, Page Fault on a Swapped Out Page.
- Linux v6.12 mm/memory.c:
do_swap_page(),should_try_to_free_swap(). - Linux v6.12 mm/swap_state.c: cache lookup와 read 준비.