fork 뒤 child가 이미 CoW한 page라면 parent의 folio에서 child VMA가 검색되어도 child는 그 folio를 사용하지 않을 수 있습니다. rmap index가 잘못된 것은 아닙니다. rmap은 후보를 좁히며, 현재 PFN mapping은 page-table lock 아래 다시 확인하도록 설계되어 있습니다.
walk control이 작업을 바꿉니다
rmap_walk()는 anonymous, file과 KSM 등의 종류에 따라 경로를 고릅니다. rmap_walk_locked()는 필요한 rmap lock을 caller가 보유했다는 다른 계약입니다. control의 arg는 caller state, invalid_vma는 건너뛸 후보, rmap_one은 후보별 작업, done은 조기 종료 판단입니다. callback의 false 반환과 done의 true 반환은 모두 종료에 관계하지만 서로 다른 질문입니다.
try_lock은 root anon_vma 또는 file interval lock의 contention에서 기다리지 않을 수 있게 합니다. contended는 검색 결과가 없었다는 뜻과 lock 때문에 끝났다는 뜻을 구별합니다. reclaim처럼 다른 task의 진전을 막지 않아야 하는 사용처에서 중요합니다.
anonymous와 file의 검색 좌표
anonymous walk는 mapping tag를 확인해 anon_vma를 얻고 root의 read 보호 아래 folio offset 범위와 겹치는 AVC를 찾습니다. file walk는 안정된 folio와 address_space를 확인하고 i_mmap의 해당 offset 범위를 검색합니다. file mapping pointer를 얻었다고 잠금이나 수명 검사가 없어지는 것은 아닙니다.
후보 VMA마다 folio index를 vm_pgoff와 비교하여 VA로 변환합니다. 범위를 벗어나거나 overflow하는 변환을 허용해서는 안 됩니다. compound folio에서는 VMA와 겹치는 일부만 해당할 수 있습니다. callback은 이 주소를 이용해 실제 entry를 확인합니다.
page_vma_mapped_walk의 두 종류 state
입력에는 PFN 시작, base-page 수, logical pgoff, VMA, VA와 flags가 있습니다. 출력에는 찾은 PMD·PTE와 획득한 PTE lock이 담깁니다. 정상 mapping을 찾으면 true와 함께 lock을 넘기므로 caller가 안정된 entry를 조작할 수 있습니다. 더 찾을 것이 없으면 lock을 정리하고 false를 돌려줍니다.
loop를 중간에 끝내는 caller는 page_vma_mapped_walk_done()으로 남은 상태를 정리해야 합니다. true를 받았다는 이유만으로 일반 iterator처럼 break하면 lock이 남을 수 있습니다. PVMW_SYNC는 더 엄격한 lookup을, PVMW_MIGRATION은 present PFN 대신 migration entry 검색을 요청합니다.
table을 걸은 뒤 PFN도 비교합니다
PGD·P4D·PUD·PMD를 따라 내려가며 비어 있는 level은 그 level의 다음 경계로 건너뜁니다. 다음 주소 계산이 wrap하면 무한 loop가 되지 않도록 종료 state로 옮깁니다. PTE를 얻고 lock을 잡은 뒤 상위 PMD가 바뀌지 않았는지도 재확인이 필요할 수 있습니다.
마지막 검사는 present 여부만이 아닙니다. PTE가 가리키는 PFN이 찾는 folio의 PFN 범위와 맞는지 확인합니다. migration과 device-private 등에는 대응 entry 판정이 더해집니다. “VMA offset이 맞는다”와 “현재 이 folio가 연결됐다”가 비로소 여기서 구분됩니다.
reclaim의 referenced 검사, unmap, writeback의 clean/write-protect, migration은 이 machinery에 서로 다른 callback을 얹습니다. migration entry는 swap-like encoding을 사용하더라도 실제 swap disk 위치와 같지 않습니다. 같은 walk를 쓴다는 사실이 각 작업의 lock·entry 계약까지 같게 만들지는 않습니다.
Dispatch와 control의 전체 정의
다음 code는 official Linux v6.0의 전체 정의를 사용합니다. Kernel context source이고 standalone 실행 예제가 아닙니다. 책 전체의 exact tag를 확인했다거나, 뒤에 제시한 v6.18 link와 API·layout이 같다고 주장하지 않겠습니다. 원래 notice와 COPYING, GPL-2.0를 함께 보존했습니다.
Dispatch는 KSM·anonymous·file을 구별하지만 locked variant에는 KSM을 그대로 적용할 수 없습니다. Callback control의 pointer와 반환형도 생략하지 않고 보겠습니다. rmap_one의 false와 done의 true는 서로 반대 방향의 판정값으로 loop를 멈춥니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:2494-2502
* Exact source lines, not a restored book listing or standalone program.
*/
void rmap_walk(struct folio *folio, struct rmap_walk_control *rwc)
{
if (unlikely(folio_test_ksm(folio)))
rmap_walk_ksm(folio, rwc);
else if (folio_test_anon(folio))
rmap_walk_anon(folio, rwc, false);
else
rmap_walk_file(folio, rwc, false);
}
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:2505-2513
* Exact source lines, not a restored book listing or standalone program.
*/
void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc)
{
/* no ksm support for now */
VM_BUG_ON_FOLIO(folio_test_ksm(folio), folio);
if (folio_test_anon(folio))
rmap_walk_anon(folio, rwc, true);
else
rmap_walk_file(folio, rwc, true);
}
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/rmap.h:390-404
* Exact source lines, not a restored book listing or standalone program.
*/
struct rmap_walk_control {
void *arg;
bool try_lock;
bool contended;
/*
* Return false if page table scanning in rmap_walk should be stopped.
* Otherwise, return true.
*/
bool (*rmap_one)(struct folio *folio, struct vm_area_struct *vma,
unsigned long addr, void *arg);
int (*done)(struct folio *folio);
struct anon_vma *(*anon_lock)(struct folio *folio,
struct rmap_walk_control *rwc);
bool (*invalid_vma)(struct vm_area_struct *vma, void *arg);
};
Anonymous walker의 lock과 후보 검색
Locked caller가 직접 보장하는 경우와 default lock helper를 통하는 경우를 구분합니다. Root lock을 잡았다는 사실만으로 callback이 찾는 page-table mapping이 현재도 존재한다고 보장되지는 않습니다. Walker는 [pgoff_start, pgoff_start + nr_pages - 1]의 inclusive 구간에서 후보를 찾고 callback으로 실제 작업을 넘깁니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:2393-2431
* Exact source lines, not a restored book listing or standalone program.
*/
static void rmap_walk_anon(struct folio *folio,
struct rmap_walk_control *rwc, bool locked)
{
struct anon_vma *anon_vma;
pgoff_t pgoff_start, pgoff_end;
struct anon_vma_chain *avc;
if (locked) {
anon_vma = folio_anon_vma(folio);
/* anon_vma disappear under us? */
VM_BUG_ON_FOLIO(!anon_vma, folio);
} else {
anon_vma = rmap_walk_anon_lock(folio, rwc);
}
if (!anon_vma)
return;
pgoff_start = folio_pgoff(folio);
pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root,
pgoff_start, pgoff_end) {
struct vm_area_struct *vma = avc->vma;
unsigned long address = vma_address(&folio->page, vma);
VM_BUG_ON_VMA(address == -EFAULT, vma);
cond_resched();
if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg))
continue;
if (!rwc->rmap_one(folio, vma, address, rwc->arg))
break;
if (rwc->done && rwc->done(folio))
break;
}
if (!locked)
anon_vma_unlock_read(anon_vma);
}
// SPDX-License-Identifier: GPL-2.0-only
/* Upstream Linux v6.0: mm/util.c:778-785
* Exact source lines, not a restored book listing or standalone program.
*/
struct anon_vma *folio_anon_vma(struct folio *folio)
{
unsigned long mapping = (unsigned long)folio->mapping;
if ((mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
return NULL;
return (void *)(mapping - PAGE_MAPPING_ANON);
}
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:2352-2382
* Exact source lines, not a restored book listing or standalone program.
*/
static struct anon_vma *rmap_walk_anon_lock(struct folio *folio,
struct rmap_walk_control *rwc)
{
struct anon_vma *anon_vma;
if (rwc->anon_lock)
return rwc->anon_lock(folio, rwc);
/*
* Note: remove_migration_ptes() cannot use folio_lock_anon_vma_read()
* because that depends on page_mapped(); but not all its usages
* are holding mmap_lock. Users without mmap_lock are required to
* take a reference count to prevent the anon_vma disappearing
*/
anon_vma = folio_anon_vma(folio);
if (!anon_vma)
return NULL;
if (anon_vma_trylock_read(anon_vma))
goto out;
if (rwc->try_lock) {
anon_vma = NULL;
rwc->contended = true;
goto out;
}
anon_vma_lock_read(anon_vma);
out:
return anon_vma;
}
anon_vma_interval_tree_foreach.h
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/mm.h:2572-2574
* Exact source lines, not a restored book listing or standalone program.
*/
#define anon_vma_interval_tree_foreach(avc, root, start, last) \
for (avc = anon_vma_interval_tree_iter_first(root, start, last); \
avc; avc = anon_vma_interval_tree_iter_next(avc, start, last))
folio_anon_vma()는 anonymous bit가 하나라도 켜졌는지만 보는 검사가 아닙니다. PAGE_MAPPING_FLAGS로 mask한 결과 전체가 PAGE_MAPPING_ANON과 같은지 비교하므로 KSM·movable 표현과 구분됩니다. Lock helper의 주석은 migration caller가 page_mapped()에 의존할 수 없는 경우도 설명합니다. 그런 caller의 별도 reference 계약을 생략하고 임의의 folio에 이 함수를 호출해서는 안 됩니다.
File walker는 folio가 locked여야 합니다
File walker의 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio)는 unlocked이면 invariant 위반이라는 뜻입니다. Locked인 경우를 거절하는 조건이 아닙니다. Folio lock은 truncate가 mapping pointer를 지우거나 address_space를 해제·재사용하는 동안 이 walker가 잘못된 rwsem에 접근하지 않도록 하는 수명 보장의 일부입니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:2441-2492
* Exact source lines, not a restored book listing or standalone program.
*/
static void rmap_walk_file(struct folio *folio,
struct rmap_walk_control *rwc, bool locked)
{
struct address_space *mapping = folio_mapping(folio);
pgoff_t pgoff_start, pgoff_end;
struct vm_area_struct *vma;
/*
* The page lock not only makes sure that page->mapping cannot
* suddenly be NULLified by truncation, it makes sure that the
* structure at mapping cannot be freed and reused yet,
* so we can safely take mapping->i_mmap_rwsem.
*/
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
if (!mapping)
return;
pgoff_start = folio_pgoff(folio);
pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
if (!locked) {
if (i_mmap_trylock_read(mapping))
goto lookup;
if (rwc->try_lock) {
rwc->contended = true;
return;
}
i_mmap_lock_read(mapping);
}
lookup:
vma_interval_tree_foreach(vma, &mapping->i_mmap,
pgoff_start, pgoff_end) {
unsigned long address = vma_address(&folio->page, vma);
VM_BUG_ON_VMA(address == -EFAULT, vma);
cond_resched();
if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg))
continue;
if (!rwc->rmap_one(folio, vma, address, rwc->arg))
goto done;
if (rwc->done && rwc->done(folio))
goto done;
}
done:
if (!locked)
i_mmap_unlock_read(mapping);
}
folio_mapping()의 swapcache 반환값은 swap file 자체의 address_space와 같다는 뜻이 아닙니다. Slab folio에 대한 NULL 반환도 이 helper 하나가 모든 slab race를 해결한다는 보장은 아닙니다. Anonymous tag가 있는 pointer를 file mapping으로 넘기지 않는 분류와 caller의 lock 계약을 나눠 읽겠습니다.
// SPDX-License-Identifier: GPL-2.0-only
/* Upstream Linux v6.0: mm/util.c:799-815
* Exact source lines, not a restored book listing or standalone program.
*/
struct address_space *folio_mapping(struct folio *folio)
{
struct address_space *mapping;
/* This happens if someone calls flush_dcache_page on slab page */
if (unlikely(folio_test_slab(folio)))
return NULL;
if (unlikely(folio_test_swapcache(folio)))
return swap_address_space(folio_swap_entry(folio));
mapping = folio->mapping;
if ((unsigned long)mapping & PAGE_MAPPING_FLAGS)
return NULL;
return mapping;
}
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/mm.h:2555-2557
* Exact source lines, not a restored book listing or standalone program.
*/
#define vma_interval_tree_foreach(vma, root, start, last) \
for (vma = vma_interval_tree_iter_first(root, start, last); \
vma; vma = vma_interval_tree_iter_next(vma, start, last))
주소 변환은 겹침과 overflow도 처리합니다
Base page의 경우에는 index 차이를 VA로 옮기면 됩니다. Compound folio가 VMA 왼쪽에서 시작하되 일부만 겹친다면 callback의 시작 주소는 VMA 시작으로 잡습니다. 전혀 겹치지 않거나 address 계산이 범위를 벗어나면 unsigned long 표현의 -EFAULT를 반환합니다. 단순한 arithmetic 예시와 이 전체 helper의 실패 계약을 혼동하지 않으셔야 합니다.
/* SPDX-License-Identifier: GPL-2.0-or-later */
/* internal.h: mm/ internal definitions
*
* Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
* Written by David Howells (dhowells@redhat.com)
*/
/* Upstream Linux v6.0: mm/internal.h:578-583
* Exact source lines, not a restored book listing or standalone program.
*/
static inline unsigned long
vma_address(struct page *page, struct vm_area_struct *vma)
{
VM_BUG_ON_PAGE(PageKsm(page), page); /* KSM page->index unusable */
return vma_pgoff_address(page_to_pgoff(page), compound_nr(page), vma);
}
/* SPDX-License-Identifier: GPL-2.0-or-later */
/* internal.h: mm/ internal definitions
*
* Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
* Written by David Howells (dhowells@redhat.com)
*/
/* Upstream Linux v6.0: mm/internal.h:552-571
* Exact source lines, not a restored book listing or standalone program.
*/
static inline unsigned long
vma_pgoff_address(pgoff_t pgoff, unsigned long nr_pages,
struct vm_area_struct *vma)
{
unsigned long address;
if (pgoff >= vma->vm_pgoff) {
address = vma->vm_start +
((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
/* Check for address beyond vma (or wrapped through 0?) */
if (address < vma->vm_start || address >= vma->vm_end)
address = -EFAULT;
} else if (pgoff + nr_pages - 1 >= vma->vm_pgoff) {
/* Test above avoids possibility of wrap to 0 on 32-bit */
address = vma->vm_start;
} else {
address = -EFAULT;
}
return address;
}
출처: v6.0 mm/rmap.c, v6.0 mm/util.c, v6.0 mm/internal.h, v6.0 mm.h.
Iterator를 끝까지 읽어야 lock의 주인이 보입니다
DEFINE_FOLIO_VMA_WALK는 검색할 folio의 PFN 범위와 logical offset을 초기화합니다. page_vma_mapped_walk()가 true를 반환하면 pte 또는 pmd만 얻은 것이 아니라 그 entry를 보호할 ptl도 넘겨받습니다. 다음 호출로 반복하거나, 중간 종료라면 page_vma_mapped_walk_done()을 호출하셔야 합니다. 후자는 PTE mapping과 spinlock을 정리하는 helper이지 folio reference를 put하는 함수는 아닙니다.
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/rmap.h:316-326
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
struct page_vma_mapped_walk {
unsigned long pfn;
unsigned long nr_pages;
pgoff_t pgoff;
struct vm_area_struct *vma;
unsigned long address;
pmd_t *pmd;
pte_t *pte;
spinlock_t *ptl;
unsigned int flags;
};
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/rmap.h:338-346
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
#define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \
struct page_vma_mapped_walk name = { \
.pfn = folio_pfn(_folio), \
.nr_pages = folio_nr_pages(_folio), \
.pgoff = folio_pgoff(_folio), \
.vma = _vma, \
.address = _address, \
.flags = _flags, \
}
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/rmap.h:348-355
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw)
{
/* HugeTLB pte is set to the relevant page table entry without pte_mapped. */
if (pvmw->pte && !is_vm_hugetlb_page(pvmw->vma))
pte_unmap(pvmw->pte);
if (pvmw->ptl)
spin_unlock(pvmw->ptl);
}
전체 iterator에서는 HugeTLB, THP, devmap, migration 분기를 생략하지 않았습니다. 일반 PTE 경로만 읽더라도 이 분기가 왜 같은 output field를 다르게 사용하는지 확인할 수 있습니다. READ_ONCE(*pmd)는 snapshot의 compiler 재접근을 제한하는 것이며, 그 자체가 table의 수명을 reference로 고정하거나 이후 lock 아래 재검사를 대신하지 않습니다.
// SPDX-License-Identifier: GPL-2.0
/* Upstream Linux v6.0: mm/page_vma_mapped.c:151-289
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
{
struct vm_area_struct *vma = pvmw->vma;
struct mm_struct *mm = vma->vm_mm;
unsigned long end;
pgd_t *pgd;
p4d_t *p4d;
pud_t *pud;
pmd_t pmde;
/* The only possible pmd mapping has been handled on last iteration */
if (pvmw->pmd && !pvmw->pte)
return not_found(pvmw);
if (unlikely(is_vm_hugetlb_page(vma))) {
struct hstate *hstate = hstate_vma(vma);
unsigned long size = huge_page_size(hstate);
/* The only possible mapping was handled on last iteration */
if (pvmw->pte)
return not_found(pvmw);
/* when pud is not present, pte will be NULL */
pvmw->pte = huge_pte_offset(mm, pvmw->address, size);
if (!pvmw->pte)
return false;
pvmw->ptl = huge_pte_lock(hstate, mm, pvmw->pte);
if (!check_pte(pvmw))
return not_found(pvmw);
return true;
}
end = vma_address_end(pvmw);
if (pvmw->pte)
goto next_pte;
restart:
do {
pgd = pgd_offset(mm, pvmw->address);
if (!pgd_present(*pgd)) {
step_forward(pvmw, PGDIR_SIZE);
continue;
}
p4d = p4d_offset(pgd, pvmw->address);
if (!p4d_present(*p4d)) {
step_forward(pvmw, P4D_SIZE);
continue;
}
pud = pud_offset(p4d, pvmw->address);
if (!pud_present(*pud)) {
step_forward(pvmw, PUD_SIZE);
continue;
}
pvmw->pmd = pmd_offset(pud, pvmw->address);
/*
* Make sure the pmd value isn't cached in a register by the
* compiler and used as a stale value after we've observed a
* subsequent update.
*/
pmde = READ_ONCE(*pvmw->pmd);
if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde) ||
(pmd_present(pmde) && pmd_devmap(pmde))) {
pvmw->ptl = pmd_lock(mm, pvmw->pmd);
pmde = *pvmw->pmd;
if (!pmd_present(pmde)) {
swp_entry_t entry;
if (!thp_migration_supported() ||
!(pvmw->flags & PVMW_MIGRATION))
return not_found(pvmw);
entry = pmd_to_swp_entry(pmde);
if (!is_migration_entry(entry) ||
!check_pmd(swp_offset(entry), pvmw))
return not_found(pvmw);
return true;
}
if (likely(pmd_trans_huge(pmde) || pmd_devmap(pmde))) {
if (pvmw->flags & PVMW_MIGRATION)
return not_found(pvmw);
if (!check_pmd(pmd_pfn(pmde), pvmw))
return not_found(pvmw);
return true;
}
/* THP pmd was split under us: handle on pte level */
spin_unlock(pvmw->ptl);
pvmw->ptl = NULL;
} else if (!pmd_present(pmde)) {
/*
* If PVMW_SYNC, take and drop THP pmd lock so that we
* cannot return prematurely, while zap_huge_pmd() has
* cleared *pmd but not decremented compound_mapcount().
*/
if ((pvmw->flags & PVMW_SYNC) &&
transhuge_vma_suitable(vma, pvmw->address) &&
(pvmw->nr_pages >= HPAGE_PMD_NR)) {
spinlock_t *ptl = pmd_lock(mm, pvmw->pmd);
spin_unlock(ptl);
}
step_forward(pvmw, PMD_SIZE);
continue;
}
if (!map_pte(pvmw))
goto next_pte;
this_pte:
if (check_pte(pvmw))
return true;
next_pte:
do {
pvmw->address += PAGE_SIZE;
if (pvmw->address >= end)
return not_found(pvmw);
/* Did we cross page table boundary? */
if ((pvmw->address & (PMD_SIZE - PAGE_SIZE)) == 0) {
if (pvmw->ptl) {
spin_unlock(pvmw->ptl);
pvmw->ptl = NULL;
}
pte_unmap(pvmw->pte);
pvmw->pte = NULL;
goto restart;
}
pvmw->pte++;
if ((pvmw->flags & PVMW_SYNC) && !pvmw->ptl) {
pvmw->ptl = pte_lockptr(mm, pvmw->pmd);
spin_lock(pvmw->ptl);
}
} while (pte_none(*pvmw->pte));
if (!pvmw->ptl) {
pvmw->ptl = pte_lockptr(mm, pvmw->pmd);
spin_lock(pvmw->ptl);
}
goto this_pte;
} while (pvmw->address < end);
return false;
}
step_forward()는 다음 level 경계로 진행하다 address가 wrap하면 ULONG_MAX 종료 state를 만듭니다. vma_address_end()는 찾는 folio 범위의 끝을 VMA 안에서 제한합니다. 이 end는 loop의 exclusive bound이며, 앞서 interval tree에서 사용한 inclusive last와 다릅니다.
// SPDX-License-Identifier: GPL-2.0
/* Upstream Linux v6.0: mm/page_vma_mapped.c:120-125
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
static void step_forward(struct page_vma_mapped_walk *pvmw, unsigned long size)
{
pvmw->address = (pvmw->address + size) & ~(size - 1);
if (!pvmw->address)
pvmw->address = ULONG_MAX;
}
/* SPDX-License-Identifier: GPL-2.0-or-later */
/* internal.h: mm/ internal definitions
*
* Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
* Written by David Howells (dhowells@redhat.com)
*/
/* Upstream Linux v6.0: mm/internal.h:589-605
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
static inline unsigned long vma_address_end(struct page_vma_mapped_walk *pvmw)
{
struct vm_area_struct *vma = pvmw->vma;
pgoff_t pgoff;
unsigned long address;
/* Common case, plus ->pgoff is invalid for KSM */
if (pvmw->nr_pages == 1)
return pvmw->address + PAGE_SIZE;
pgoff = pvmw->pgoff + pvmw->nr_pages;
address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
/* Check for address beyond vma (or wrapped through 0?) */
if (address < vma->vm_start || address > vma->vm_end)
address = vma->vm_end;
return address;
}
map_pte()의 false는 단순히 allocation 실패를 뜻하지 않습니다. entry 종류와 sync 요구에 따라 검색할 가치가 없는 경우도 있습니다. check_pte()는 migration·device-private entry를 포함해 원하는 PFN 범위인지 판정합니다. unsigned subtraction을 이용한 범위 검사는 present 여부 검사보다 더 구체적인 질문입니다. 정상 mapping을 찾았더라도 이 caller가 원하는 종류의 entry인지까지 읽어야 합니다.
// SPDX-License-Identifier: GPL-2.0
/* Upstream Linux v6.0: mm/page_vma_mapped.c:16-54
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
static bool map_pte(struct page_vma_mapped_walk *pvmw)
{
pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address);
if (!(pvmw->flags & PVMW_SYNC)) {
if (pvmw->flags & PVMW_MIGRATION) {
if (!is_swap_pte(*pvmw->pte))
return false;
} else {
/*
* We get here when we are trying to unmap a private
* device page from the process address space. Such
* page is not CPU accessible and thus is mapped as
* a special swap entry, nonetheless it still does
* count as a valid regular mapping for the page (and
* is accounted as such in page maps count).
*
* So handle this special case as if it was a normal
* page mapping ie lock CPU page table and returns
* true.
*
* For more details on device private memory see HMM
* (include/linux/hmm.h or mm/hmm.c).
*/
if (is_swap_pte(*pvmw->pte)) {
swp_entry_t entry;
/* Handle un-addressable ZONE_DEVICE memory */
entry = pte_to_swp_entry(*pvmw->pte);
if (!is_device_private_entry(entry) &&
!is_device_exclusive_entry(entry))
return false;
} else if (!pte_present(*pvmw->pte))
return false;
}
}
pvmw->ptl = pte_lockptr(pvmw->vma->vm_mm, pvmw->pmd);
spin_lock(pvmw->ptl);
return true;
}
// SPDX-License-Identifier: GPL-2.0
/* Upstream Linux v6.0: mm/page_vma_mapped.c:75-108
* Exact source lines; kernel context, not a standalone program or book restoration.
*/
static bool check_pte(struct page_vma_mapped_walk *pvmw)
{
unsigned long pfn;
if (pvmw->flags & PVMW_MIGRATION) {
swp_entry_t entry;
if (!is_swap_pte(*pvmw->pte))
return false;
entry = pte_to_swp_entry(*pvmw->pte);
if (!is_migration_entry(entry) &&
!is_device_exclusive_entry(entry))
return false;
pfn = swp_offset(entry);
} else if (is_swap_pte(*pvmw->pte)) {
swp_entry_t entry;
/* Handle un-addressable ZONE_DEVICE memory */
entry = pte_to_swp_entry(*pvmw->pte);
if (!is_device_private_entry(entry) &&
!is_device_exclusive_entry(entry))
return false;
pfn = swp_offset(entry);
} else {
if (!pte_present(*pvmw->pte))
return false;
pfn = pte_pfn(*pvmw->pte);
}
return (pfn - pvmw->pfn) < pvmw->nr_pages;
}
위 정의는 Linux v6.0 mm/page_vma_mapped.c, rmap.h의 고정 source입니다. 특정 historical x86 경로에서 pte_unmap()이 가벼워 보이더라도 다른 architecture·version의 mapping/RCU 계약까지 no-op이라고 확대하지 않겠습니다. 이 글의 code는 compile·runtime 검증한 standalone program이 아닙니다.
참고 자료
Lorenzo Stoakes, The Linux Memory Manager, Chapter 7 §7.0.15–§7.0.17. Linux v6.18 mm/rmap.c, mm/page_vma_mapped.c, include/linux/rmap.h를 참고하실 수 있습니다.