anonymous VMA를 만들었다고 곧바로 anon_vma까지 필요하지는 않습니다. zero page만 읽거나 아직 접근하지 않았다면 새 anonymous folio의 계보가 없을 수 있습니다. 실제로 필요한 순간에 준비하면 mapping마다 중간 객체를 무조건 allocation하는 비용을 줄일 수 있습니다.
AVC는 재사용하더라도 새로 필요합니다
anon_vma_prepare()는 이미 준비됐는지 확인하고 필요할 때 실제 준비 경로로 갑니다. 먼저 AVC를 확보합니다. 이웃의 anon_vma를 재사용해도 새 VMA와 그 객체 사이의 연결 하나는 필요하기 때문입니다. 재사용 후보가 없으면 anon_vma 자체도 allocation합니다.
anon_vma slab constructor는 rwsem과 빈 tree를 초기화합니다. 개별 allocation에서는 수명 reference와 active·child count, 자기 parent·root를 준비합니다. constructor가 cache object를 처음 만들 때 하는 초기화와 매번 사용자에게 넘길 때의 초기화를 같은 시점으로 생각하면 재사용 상태를 잘못 읽게 됩니다.
lock 아래에서 설치 경쟁을 다시 확인합니다
다른 thread가 같은 VMA에서 먼저 준비를 끝낼 수 있습니다. root write lock과 active pointer 설치를 보호하는 동기화 아래에서 아직 NULL인지 다시 검사합니다. 이미 연결됐으면 먼저 allocation한 여분 AVC·anon_vma는 반환합니다. allocation 성공만으로 설치가 보장되는 것은 아닙니다.
연결할 때에는 AVC의 양 끝을 채우고 VMA list와 anon_vma interval tree에 함께 넣습니다. 이 중 한쪽만 연결한 state가 외부 walk에 정상 결과처럼 보이면 안 됩니다. allocation 실패도 새 object와 재사용 object의 reference 계약을 구분하여 unwind해야 합니다.
이웃 anon_vma 재사용은 VMA merge가 아닙니다
find_mergeable_anon_vma() 계열은 인접 VMA에서 compatible한 객체를 찾습니다. 주소의 인접성, NUMA policy, file identity와 논리 offset 연속성이 필요합니다. anonymous rmap 관점에서는 READ·WRITE·EXEC 같은 기본 접근 차이가 동일 backing 계보를 금지할 이유는 아니므로 VMA 자체의 merge보다 flags 비교가 느슨할 수 있습니다. soft-dirty 차이도 같은 의미의 구조 차이가 아닙니다.
반대로 복잡한 fork ancestry를 단순한 이웃 관계로 가져오지 않도록 AVC list의 singular 조건을 검사합니다. 따라서 같은 anon_vma를 공유하는 두 VMA도 protection이 달라 별개 VMA로 남을 수 있습니다. “anon_vma mergeable”과 “VMA mergeable”은 다른 predicate입니다.
왜 임의의 먼 객체까지 합치지 않는가
논리 offset을 쓰면 떨어진 VA도 같은 anon_vma에서 표현할 수 있습니다. 그러나 compatible 객체를 주소 공간 전체에서 찾아 합치려면 추가 검색과 ancestry 조정이 필요합니다. mremap() 뒤에는 현재 VA와 논리 index도 다를 수 있습니다. kernel은 모든 가능한 통합을 수행하기보다 저렴하게 확인할 수 있는 이웃의 재사용을 선택합니다.
이 절충 덕분에 첫 fault는 필요할 때만 metadata를 만들면서도 split·protection 변경으로 생긴 단순 이웃 관계를 불필요하게 여러 계보로 쪼개지 않습니다. 실제 performance를 볼 때 data page allocation뿐 아니라 이런 metadata allocation과 lock 경쟁도 함께 고려해야 합니다.
설치 경쟁과 실패 경로를 전체 코드로 확인합니다
여기서는 official Linux v6.0의 전체 function을 읽습니다. 책의 exact source tag가 별도로 증명됐다는 뜻은 아니며, vm_next·vm_prev를 사용하는 historical 구현을 v6.18 구현으로 바꿔 표시하지 않습니다. 아래 code는 kernel-context source이고 standalone build·실행은 하지 않았습니다. 원래 copyright notice와 GPL 조건은 COPYING 및 GPL-2.0에 보존했습니다.
Fast path는 이미 준비된 pointer가 있으면 끝납니다. 실제 준비 함수는 AVC allocation 실패, anon_vma allocation 실패, 다른 thread가 먼저 설치한 경우를 모두 다룹니다. allocated와 avc를 NULL로 바꾸는 것은 단순한 초기화가 아니라 성공적으로 설치한 object의 소유권을 cleanup 경로에서 빼는 표시입니다.
/* SPDX-License-Identifier: GPL-2.0 */
/* Upstream Linux v6.0: include/linux/rmap.h:154-160
* Exact source lines, not a restored book listing or standalone program.
*/
static inline int anon_vma_prepare(struct vm_area_struct *vma)
{
if (likely(vma->anon_vma))
return 0;
return __anon_vma_prepare(vma);
}
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:187-233
* Exact source lines, not a restored book listing or standalone program.
*/
int __anon_vma_prepare(struct vm_area_struct *vma)
{
struct mm_struct *mm = vma->vm_mm;
struct anon_vma *anon_vma, *allocated;
struct anon_vma_chain *avc;
might_sleep();
avc = anon_vma_chain_alloc(GFP_KERNEL);
if (!avc)
goto out_enomem;
anon_vma = find_mergeable_anon_vma(vma);
allocated = NULL;
if (!anon_vma) {
anon_vma = anon_vma_alloc();
if (unlikely(!anon_vma))
goto out_enomem_free_avc;
anon_vma->num_children++; /* self-parent link for new root */
allocated = anon_vma;
}
anon_vma_lock_write(anon_vma);
/* page_table_lock to protect against threads */
spin_lock(&mm->page_table_lock);
if (likely(!vma->anon_vma)) {
vma->anon_vma = anon_vma;
anon_vma_chain_link(vma, avc, anon_vma);
anon_vma->num_active_vmas++;
allocated = NULL;
avc = NULL;
}
spin_unlock(&mm->page_table_lock);
anon_vma_unlock_write(anon_vma);
if (unlikely(allocated))
put_anon_vma(allocated);
if (unlikely(avc))
anon_vma_chain_free(avc);
return 0;
out_enomem_free_avc:
anon_vma_chain_free(avc);
out_enomem:
return -ENOMEM;
}
새 객체라도 root lock을 잡는 이유는 RCU 아래에서 stale pointer를 관찰한 reader가 같은 cache object의 rwsem에 접근할 수 있기 때문입니다. Allocation 직후라는 사실만으로 다른 관찰자가 전혀 없다고 가정하지 않습니다. Root write lock 뒤 page_table_lock을 잡고 vma->anon_vma를 다시 검사하는 순서를 읽어 보시면, allocation과 설치가 별개의 단계라는 점이 드러납니다.
AVC cache의 alloc/free wrapper와 anon_vma의 per-allocation 초기화, slab constructor도 빠짐없이 구분하겠습니다. Constructor의 빈 tree·rwsem 준비와 anon_vma_alloc()의 refcount 1 설정은 호출 빈도와 목적이 다릅니다. Slab object를 얻을 때마다 constructor가 다시 실행된다고 읽어서는 안 됩니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:139-142
* Exact source lines, not a restored book listing or standalone program.
*/
static inline struct anon_vma_chain *anon_vma_chain_alloc(gfp_t gfp)
{
return kmem_cache_alloc(anon_vma_chain_cachep, gfp);
}
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:144-147
* Exact source lines, not a restored book listing or standalone program.
*/
static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain)
{
kmem_cache_free(anon_vma_chain_cachep, anon_vma_chain);
}
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:89-107
* Exact source lines, not a restored book listing or standalone program.
*/
static inline struct anon_vma *anon_vma_alloc(void)
{
struct anon_vma *anon_vma;
anon_vma = kmem_cache_alloc(anon_vma_cachep, GFP_KERNEL);
if (anon_vma) {
atomic_set(&anon_vma->refcount, 1);
anon_vma->num_children = 0;
anon_vma->num_active_vmas = 0;
anon_vma->parent = anon_vma;
/*
* Initialise the anon_vma root to point to itself. If called
* from fork, the root will be reset to the parents anon_vma.
*/
anon_vma->root = anon_vma;
}
return anon_vma;
}
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:450-457
* Exact source lines, not a restored book listing or standalone program.
*/
static void anon_vma_ctor(void *data)
{
struct anon_vma *anon_vma = data;
init_rwsem(&anon_vma->rwsem);
atomic_set(&anon_vma->refcount, 0);
anon_vma->rb_root = RB_ROOT_CACHED;
}
이웃 재사용의 predicate를 하나씩 대조합니다
v6.0은 next를 먼저 보고 prev를 봅니다. 먼 VMA까지 검색하지 않는 것은 표현할 수 없어서가 아니라 검색 비용과 한 anon_vma에 모이는 VMA 수를 제한하려는 선택입니다. 다음 세 함수는 후보 선택, pointer의 한 번 관찰, compatibility 검사를 나눕니다.
// SPDX-License-Identifier: GPL-2.0-only
/*
* mm/mmap.c
*
* Written by obz.
*
* Address space accounting code <alan@lxorguk.ukuu.org.uk>
*/
/* Upstream Linux v6.0: mm/mmap.c:1276-1302
* Exact source lines, not a restored book listing or standalone program.
*/
struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma)
{
struct anon_vma *anon_vma = NULL;
/* Try next first. */
if (vma->vm_next) {
anon_vma = reusable_anon_vma(vma->vm_next, vma, vma->vm_next);
if (anon_vma)
return anon_vma;
}
/* Try prev next. */
if (vma->vm_prev)
anon_vma = reusable_anon_vma(vma->vm_prev, vma->vm_prev, vma);
/*
* We might reach here with anon_vma == NULL if we can't find
* any reusable anon_vma.
* There's no absolute need to look only at touching neighbours:
* we could search further afield for "compatible" anon_vmas.
* But it would probably just be a waste of time searching,
* or lead to too many vmas hanging off the same anon_vma.
* We're trying to allow mprotect remerging later on,
* not trying to minimize memory used for anon_vmas.
*/
return anon_vma;
}
// SPDX-License-Identifier: GPL-2.0-only
/*
* mm/mmap.c
*
* Written by obz.
*
* Address space accounting code <alan@lxorguk.ukuu.org.uk>
*/
/* Upstream Linux v6.0: mm/mmap.c:1257-1266
* Exact source lines, not a restored book listing or standalone program.
*/
static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old, struct vm_area_struct *a, struct vm_area_struct *b)
{
if (anon_vma_compatible(a, b)) {
struct anon_vma *anon_vma = READ_ONCE(old->anon_vma);
if (anon_vma && list_is_singular(&old->anon_vma_chain))
return anon_vma;
}
return NULL;
}
// SPDX-License-Identifier: GPL-2.0-only
/*
* mm/mmap.c
*
* Written by obz.
*
* Address space accounting code <alan@lxorguk.ukuu.org.uk>
*/
/* Upstream Linux v6.0: mm/mmap.c:1226-1233
* Exact source lines, not a restored book listing or standalone program.
*/
static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *b)
{
return a->vm_end == b->vm_start &&
mpol_equal(vma_policy(a), vma_policy(b)) &&
a->vm_file == b->vm_file &&
!((a->vm_flags ^ b->vm_flags) & ~(VM_ACCESS_FLAGS | VM_SOFTDIRTY)) &&
b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT);
}
READ_ONCE()는 같은 expression에서 pointer를 재관찰하지 않도록 하며 lock 전체를 대체하지 않습니다. vm_file은 같은 inode라는 느슨한 조건이 아니라 이 코드의 pointer equality입니다. Offset 연속성도 b.vm_pgoff = a.vm_pgoff + (b.vm_start-a.vm_start)/PAGE_SIZE로 직접 검사합니다. Address가 인접하다는 사실만 보고 다른 논리 offset까지 합치지 않습니다. 반면 vm_ops->close()로 VMA 자체의 merge가 막혀도 anon_vma sharing은 별도의 조건일 수 있습니다.
연결 helper는 두 pointer를 채우고 list와 interval tree 양쪽에 node를 넣습니다. 이 함수를 lock 없는 독립 API처럼 호출해도 된다는 뜻은 아닙니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* page->flags PG_locked (lock_page) * (see hugetlbfs below)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* folio_lock_memcg move_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* * hugetlbfs PageHuge() pages take locks in this order:
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* page->flags PG_locked (lock_page)
*/
/* Upstream Linux v6.0: mm/rmap.c:149-157
* Exact source lines, not a restored book listing or standalone program.
*/
static void anon_vma_chain_link(struct vm_area_struct *vma,
struct anon_vma_chain *avc,
struct anon_vma *anon_vma)
{
avc->vma = vma;
avc->anon_vma = anon_vma;
list_add(&avc->same_vma, &vma->anon_vma_chain);
anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
}
비교 tag v6.18의 준비 함수
같은 이름의 전체 function을 v6.18에서도 별도로 보겠습니다. 이 version에는 mmap_assert_locked(mm)가 명시돼 있습니다. 준비·race 재확인·unused object 반환의 큰 구조는 유지되지만, 이것을 근거로 인접 VMA 탐색 구현과 VMA layout까지 v6.0과 같다고 결론 내리지는 않습니다. v6.18을 현재 최신 release라고 부르는 것도 아닙니다.
/*
* mm/rmap.c - physical to virtual reverse mappings
*
* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
* Released under the General Public License (GPL).
*
* Simple, low overhead reverse mapping scheme.
* Please try to keep this thing as modular as possible.
*
* Provides methods for unmapping each kind of mapped page:
* the anon methods track anonymous pages, and
* the file methods track pages belonging to an inode.
*
* Original design by Rik van Riel <riel@conectiva.com.br> 2001
* File methods by Dave McCracken <dmccr@us.ibm.com> 2003, 2004
* Anonymous methods by Andrea Arcangeli <andrea@suse.de> 2004
* Contributions by Hugh Dickins 2003, 2004
*/
/*
* Lock ordering in mm:
*
* inode->i_rwsem (while writing or truncating, not reading or faulting)
* mm->mmap_lock
* mapping->invalidate_lock (in filemap_fault)
* folio_lock
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below)
* vma_start_write
* mapping->i_mmap_rwsem
* anon_vma->rwsem
* mm->page_table_lock or pte_lock
* swap_lock (in swap_duplicate, swap_info_get)
* mmlist_lock (in mmput, drain_mmlist and others)
* mapping->private_lock (in block_dirty_folio)
* i_pages lock (widely used)
* lruvec->lru_lock (in folio_lruvec_lock_irq)
* inode->i_lock (in set_page_dirty's __mark_inode_dirty)
* bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty)
* sb_lock (within inode_lock in fs/fs-writeback.c)
* i_pages lock (widely used, in set_page_dirty,
* in arch-dependent flush_dcache_mmap_lock,
* within bdi.wb->list_lock in __sync_single_inode)
*
* anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
*
* hugetlbfs PageHuge() take locks in this order:
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
* vma_lock (hugetlb specific lock for pmd_sharing)
* mapping->i_mmap_rwsem (also used for hugetlb pmd sharing)
* folio_lock
*/
/* Upstream Linux v6.18: mm/rmap.c:185-232
* Exact source lines, not a restored book listing or standalone program.
*/
int __anon_vma_prepare(struct vm_area_struct *vma)
{
struct mm_struct *mm = vma->vm_mm;
struct anon_vma *anon_vma, *allocated;
struct anon_vma_chain *avc;
mmap_assert_locked(mm);
might_sleep();
avc = anon_vma_chain_alloc(GFP_KERNEL);
if (!avc)
goto out_enomem;
anon_vma = find_mergeable_anon_vma(vma);
allocated = NULL;
if (!anon_vma) {
anon_vma = anon_vma_alloc();
if (unlikely(!anon_vma))
goto out_enomem_free_avc;
anon_vma->num_children++; /* self-parent link for new root */
allocated = anon_vma;
}
anon_vma_lock_write(anon_vma);
/* page_table_lock to protect against threads */
spin_lock(&mm->page_table_lock);
if (likely(!vma->anon_vma)) {
vma->anon_vma = anon_vma;
anon_vma_chain_link(vma, avc, anon_vma);
anon_vma->num_active_vmas++;
allocated = NULL;
avc = NULL;
}
spin_unlock(&mm->page_table_lock);
anon_vma_unlock_write(anon_vma);
if (unlikely(allocated))
put_anon_vma(allocated);
if (unlikely(avc))
anon_vma_chain_free(avc);
return 0;
out_enomem_free_avc:
anon_vma_chain_free(avc);
out_enomem:
return -ENOMEM;
}
출처: v6.0 mm/rmap.c, v6.0 mm/mmap.c, v6.18 mm/rmap.c. Source의 line·SHA와 비교 tag의 범위는 code manifest에 분리해 기록했습니다.
참고 자료
Lorenzo Stoakes, The Linux Memory Manager, Chapter 7 §7.0.5–§7.0.7. Linux v6.18 mm/rmap.c, 이웃 재사용 검사는 mm/vma.c에 있습니다.