VM_BIND locking

문서 범위와 DRM GPUVM helper

이 문서는 userptr `mmu_notifier` locking을 포함해 VM_BIND locking을 올바르게 구현하는 데 필요한 조건을 설명합니다. 가장 단순한 구현이 모든 userptr mapping과 external·shared object mapping을 매번 순회하는 비용을 줄이는 최적화도 다룹니다.

또한 recoverable page fault를 구현할 때 필요한 VM_BIND locking을 별도 절에서 설명합니다.

VM_BIND driver를 위한 DRM GPUVM helper 집합은 이 문서의 locking 상당 부분을 구현하지만 전부는 아닙니다. 특히 현재 userptr 구현이 빠져 있습니다.

이 문서는 DRM GPUVM 내부 구현을 상세히 설명하지 않으며 `drm_gpuvm` 참조 문서가 그 역할을 합니다. VM_BIND를 구현하는 driver는 DRM GPUVM helper를 사용하고 공통 기능이 부족하면 helper 자체를 확장하는 것이 강하게 권장됩니다.

.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)

===============
VM_BIND locking
===============

This document attempts to describe what's needed to get VM_BIND locking right,
including the userptr mmu_notifier locking. It also discusses some
optimizations to get rid of the looping through of all userptr mappings and
external / shared object mappings that is needed in the simplest
implementation. In addition, there is a section describing the VM_BIND locking
required for implementing recoverable pagefaults.

The DRM GPUVM set of helpers
============================

There is a set of helpers for drivers implementing VM_BIND, and this
set of helpers implements much, but not all of the locking described
in this document. In particular, it is currently lacking a userptr
implementation. This document does not intend to describe the DRM GPUVM
implementation in detail, but it is covered in :ref:`its own
documentation <drm_gpuvm>`. It is highly recommended for any driver
implementing VM_BIND to use the DRM GPUVM helpers and to extend it if
common functionality is missing.

Nomenclature

용어, lock 역할과 ordering

`gpu_vm`은 metadata를 포함한 virtual GPU address space 추상화로 보통 client(DRM file-private) 또는 execution context마다 하나입니다. `gpu_vma`는 `gpu_vm` 안의 GPU address range와 metadata를 나타내며 backing store는 GEM object 또는 process CPU address space에도 mapping된 anonymous·page-cache page일 수 있습니다.

`gpu_vm_bo`는 GEM object와 VM의 연관 관계입니다. GEM object는 `gpu_vm_bo` list를 갖고 각 `gpu_vm_bo`는 `gpu_vma` list를 갖습니다. `userptr gpu_vma`는 anonymous 또는 page-cache page를 backing store로 쓰는 `gpu_vma`입니다.

`revalidating`은 backing store 최신 version을 resident로 만들고 `gpu_vma` page-table entry가 그 backing store를 가리키도록 확인하는 과정입니다.

`dma_fence`는 `struct completion`과 비슷하게 GPU activity를 추적하고 activity가 끝나면 signal합니다. `dma_resv`는 `gpu_vm` 또는 GEM object의 여러 dma-fence를 추적하는 reservation object입니다. Fence를 추가할 때 잡아야 하는 lock과 fence array·list를 포함하며 여러 dma_resv를 임의 순서로 deadlock 없이 lock할 수 있는 유형입니다. 자세한 규칙은 `/driver-api/dma-buf`의 DMA Fences와 Reservation Objects 절을 따릅니다.

`exec function`은 영향받는 `gpu_vma`를 revalidate하고 GPU command batch를 submit한 뒤 activity dma-fence를 모든 관련 dma_resv에 등록합니다. Compute·long-running mode에서는 revalidation worker가 exec function 역할을 할 수도 있습니다.

`local object`는 단일 VM 안에서만 mapping되는 GEM object이므로 `gpu_vm`의 dma_resv를 공유합니다. `external object` 또는 shared object는 여러 `gpu_vm`과 다른 driver가 backing storage를 공유할 수 있어 자체 dma_resv가 필요합니다.

Local object가 `gpu_vm->resv`를 공유하는 것이 VM_BIND의 중요한 장점입니다. Local GEM object 수가 매우 많아도 exec sequence 전체를 atomic하게 만드는 데 lock 하나면 충분합니다.

`gpu_vm->lock`은 `gpu_vma`를 추적하는 VM data structure와 선택적으로 userptr `gpu_vma` list를 보호합니다. CPU mm의 `mmap_lock`에 대응하며 rwsem을 쓰면 여러 reader가 VM tree를 동시에 순회할 수 있지만 효과는 driver마다 다릅니다.

`userptr_seqlock`은 실제 seqlock이 아니라 `mm/mmu_notifier.c`가 seqcount와 비슷하다고 설명하는 collision-retry read/write lock입니다. 여러 write side가 동시에 잡을 수 있습니다. Read critical section은 `mmu_interval_read_begin()`과 `mmu_interval_read_retry()` 사이이며 write side가 active이면 begin이 sleep합니다. Core mm은 MMU interval invalidation notifier를 호출하는 동안 write side를 잡습니다.

`gpu_vm->resv` lock은 rebind가 필요한 `gpu_vma` list, 모든 local GEM object의 residency, 보통 evicted·external object list까지 보호합니다. `gpu_vm->userptr_notifier_lock`은 VM별 rwsem으로 exec에서 read mode, MMU notifier invalidation에서 write mode로 잡습니다.

`gem_object->gpuva_lock`은 GEM object의 `gpu_vm_bo` list를 보호합니다. 보통 GEM object dma_resv와 같지만 driver가 다른 lock을 쓸 수도 있습니다. 일부 구현은 evicted·external list를 갱신하려고 `gpu_vm` list spinlock을 쓰며, dma_resv ordering 위반 없이 순회하려면 특별한 iteration 방식이 필요합니다.

============

* ``gpu_vm``: Abstraction of a virtual GPU address space with
  meta-data. Typically one per client (DRM file-private), or one per
  execution context.
* ``gpu_vma``: Abstraction of a GPU address range within a gpu_vm with
  associated meta-data. The backing storage of a gpu_vma can either be
  a GEM object or anonymous or page-cache pages mapped also into the CPU
  address space for the process.
* ``gpu_vm_bo``: Abstracts the association of a GEM object and
  a VM. The GEM object maintains a list of gpu_vm_bos, where each gpu_vm_bo
  maintains a list of gpu_vmas.
* ``userptr gpu_vma or just userptr``: A gpu_vma, whose backing store
  is anonymous or page-cache pages as described above.
* ``revalidating``: Revalidating a gpu_vma means making the latest version
  of the backing store resident and making sure the gpu_vma's
  page-table entries point to that backing store.
* ``dma_fence``: A struct dma_fence that is similar to a struct completion
  and which tracks GPU activity. When the GPU activity is finished,
  the dma_fence signals. Please refer to the ``DMA Fences`` section of
  the :doc:`dma-buf doc </driver-api/dma-buf>`.
* ``dma_resv``: A struct dma_resv (a.k.a reservation object) that is used
  to track GPU activity in the form of multiple dma_fences on a
  gpu_vm or a GEM object. The dma_resv contains an array / list
  of dma_fences and a lock that needs to be held when adding
  additional dma_fences to the dma_resv. The lock is of a type that
  allows deadlock-safe locking of multiple dma_resvs in arbitrary
  order. Please refer to the ``Reservation Objects`` section of the
  :doc:`dma-buf doc </driver-api/dma-buf>`.
* ``exec function``: An exec function is a function that revalidates all
  affected gpu_vmas, submits a GPU command batch and registers the
  dma_fence representing the GPU command's activity with all affected
  dma_resvs. For completeness, although not covered by this document,
  it's worth mentioning that an exec function may also be the
  revalidation worker that is used by some drivers in compute /
  long-running mode.
* ``local object``: A GEM object which is only mapped within a
  single VM. Local GEM objects share the gpu_vm's dma_resv.
* ``external object``: a.k.a shared object: A GEM object which may be shared
  by multiple gpu_vms and whose backing storage may be shared with
  other drivers.

Locks and locking order
=======================

One of the benefits of VM_BIND is that local GEM objects share the gpu_vm's
dma_resv object and hence the dma_resv lock. So, even with a huge
number of local GEM objects, only one lock is needed to make the exec
sequence atomic.

The following locks and locking orders are used:

* The ``gpu_vm->lock`` (optionally an rwsem). Protects the gpu_vm's
  data structure keeping track of gpu_vmas. It can also protect the
  gpu_vm's list of userptr gpu_vmas. With a CPU mm analogy this would
  correspond to the mmap_lock. An rwsem allows several readers to walk
  the VM tree concurrently, but the benefit of that concurrency most
  likely varies from driver to driver.
* The ``userptr_seqlock``. This lock is taken in read mode for each
  userptr gpu_vma on the gpu_vm's userptr list, and in write mode during mmu
  notifier invalidation. This is not a real seqlock but described in
  ``mm/mmu_notifier.c`` as a "Collision-retry read-side/write-side
  'lock' a lot like a seqcount. However this allows multiple
  write-sides to hold it at once...". The read side critical section
  is enclosed by ``mmu_interval_read_begin() /
  mmu_interval_read_retry()`` with ``mmu_interval_read_begin()``
  sleeping if the write side is held.
  The write side is held by the core mm while calling mmu interval
  invalidation notifiers.
* The ``gpu_vm->resv`` lock. Protects the gpu_vm's list of gpu_vmas needing
  rebinding, as well as the residency state of all the gpu_vm's local
  GEM objects.
  Furthermore, it typically protects the gpu_vm's list of evicted and
  external GEM objects.
* The ``gpu_vm->userptr_notifier_lock``. This is an rwsem that is
  taken in read mode during exec and write mode during a mmu notifier
  invalidation. The userptr notifier lock is per gpu_vm.
* The ``gem_object->gpuva_lock`` This lock protects the GEM object's
  list of gpu_vm_bos. This is usually the same lock as the GEM
  object's dma_resv, but some drivers protects this list differently,
  see below.
* The ``gpu_vm list spinlocks``. With some implementations they are needed
  to be able to update the gpu_vm evicted- and external object
  list. For those implementations, the spinlocks are grabbed when the
  lists are manipulated. However, to avoid locking order violations
  with the dma_resv locks, a special scheme is needed when iterating
  over the lists.

gpu_vm_bo·gpu_vma 보호와 lifetime

GEM object의 `gpu_vm_bo` list와 각 `gpu_vm_bo`의 `gpu_vma` list는 `gem_object->gpuva_lock`이 보호합니다. 보통 이 lock은 GEM object의 dma_resv와 같지만, dma-fence signaling critical section 안에서 list에 접근해야 하는 driver는 그 경로에서 잡을 수 있는 별도 lock을 선택할 수 있습니다.

별도 lock을 선택한 driver는 `gpu_vm_bo`와 `gpu_vma` list를 순회하면서 추가로 잡는 lock이 ordering을 위반하지 않도록 더 주의해야 합니다.

DRM GPUVM helper는 관련 상황에서 이 lock이 잡혀 있는지 lockdep assertion을 제공하며, 실제로 어떤 lock을 사용하는지 helper에 알리는 `drm_gem_gpuva_set_lock()`도 제공합니다.

각 `gpu_vm_bo`는 underlying GEM object에 대한 reference-counted pointer를, 각 `gpu_vma`는 `gpu_vm_bo`에 대한 reference-counted pointer를 보유합니다.

그러나 GEM object의 `gpu_vm_bo` list와 그 아래 `gpu_vma` list를 순회하는 동안 `gem_object->gpuva_lock`을 놓으면 안 됩니다. 순회 자체가 각 attached `gpu_vma` reference를 따로 잡지 않으므로 lock을 놓는 순간 entry가 예고 없이 사라질 수 있습니다. 별도 scheme도 가능하지만 추가 복잡성이 필요하며 이 문서 범위 밖입니다.

DRM GPUVM 구현에서는 각 `gpu_vm_bo`와 `gpu_vma`가 `gpu_vm` reference count도 보유합니다. Circular reference를 피하려면 `gpu_vm` destructor에서 `gpu_vma` cleanup을 수행하면 안 됩니다.

Driver는 보통 별도 `gpu_vm` close function에서 해당 VM을 사용하는 GPU execution을 abort하고 모든 `gpu_vma`를 unmap한 뒤 page-table memory를 release합니다.

.. _gpu_vma lifetime:

Protection and lifetime of gpu_vm_bos and gpu_vmas
==================================================

The GEM object's list of gpu_vm_bos, and the gpu_vm_bo's list of gpu_vmas
is protected by the ``gem_object->gpuva_lock``, which is typically the
same as the GEM object's dma_resv, but if the driver
needs to access these lists from within a dma_fence signalling
critical section, it can instead choose to protect it with a
separate lock, which can be locked from within the dma_fence signalling
critical section. Such drivers then need to pay additional attention
to what locks need to be taken from within the loop when iterating
over the gpu_vm_bo and gpu_vma lists to avoid locking-order violations.

The DRM GPUVM set of helpers provide lockdep asserts that this lock is
held in relevant situations and also provides a means of making itself
aware of which lock is actually used: :c:func:`drm_gem_gpuva_set_lock`.

Each gpu_vm_bo holds a reference counted pointer to the underlying GEM
object, and each gpu_vma holds a reference counted pointer to the
gpu_vm_bo. When iterating over the GEM object's list of gpu_vm_bos and
over the gpu_vm_bo's list of gpu_vmas, the ``gem_object->gpuva_lock`` must
not be dropped, otherwise, gpu_vmas attached to a gpu_vm_bo may
disappear without notice since those are not reference-counted. A
driver may implement its own scheme to allow this at the expense of
additional complexity, but this is outside the scope of this document.

In the DRM GPUVM implementation, each gpu_vm_bo and each gpu_vma
holds a reference count on the gpu_vm itself. Due to this, and to avoid circular
reference counting, cleanup of the gpu_vm's gpu_vmas must not be done from the
gpu_vm's destructor. Drivers typically implements a gpu_vm close
function for this cleanup. The gpu_vm close function will abort gpu
execution using this VM, unmap all gpu_vmas and release page-table memory.

Revalidation and eviction of local objects

Local object revalidation과 eviction

이 절의 code는 간소화한 pseudo-code입니다. 실제 구현에 필요한 dma_resv deadlock-avoidance algorithm과 dma_resv fence memory 예약은 생략되어 있습니다.

VM_BIND에서 GPU가 `gpu_vm`을 사용해 실행하는 동안 모든 local object는 resident여야 하고 이를 가리키는 유효한 `gpu_vma`가 설정되어 있어야 합니다. 따라서 GPU command buffer submission 앞에는 보통 revalidation section이 옵니다.

Revalidation은 `gpu_vm->resv`를 lock하고 evict list의 각 `gpu_vm_bo` backing object를 validate합니다. Local object는 `gpu_vm->resv`를 공유하므로 `gpu_vm_bo`의 `gpu_vma` list를 보호하는 reservation lock도 이미 잡혀 있습니다.

각 VMA를 별도 rebind list로 옮겨 page table을 rebind한 뒤 list에서 제거합니다. 별도 `gpu_vm` rebind list가 필요한 이유는 buffer object를 map하지 않는 userptr `gpu_vma`도 rebind 대상일 수 있기 때문입니다.

Validation 뒤에는 기존 `gpu_vm->resv` fence를 GPU job dependency로 추가하고 job을 submit한 다음 새 job dma-fence를 reservation에 등록하고 unlock합니다.

Local object eviction은 object LRU에서 대상을 얻어 `obj->resv`를 lock하고 모든 `gpu_vm_bo`를 VM evict list에 넣습니다. Eviction job에 기존 dependency를 연결하고 submit fence를 reservation에 추가한 뒤 unlock하고 object reference를 놓습니다.

Local object는 `obj->resv == gpu_vm->resv`이므로 object lock이 VM evict list도 보호합니다. Eviction 동안 local object의 dma_resv가 잡혀 있어 새 exec function이 eviction과 race할 수 없습니다.

VM_BIND에서는 eviction 전에 `gpu_vma`를 unbind할 필요가 없습니다. Driver가 eviction blit·copy를 GPU idle 뒤에 실행하거나 이전 GPU activity에 dependency로 연결해야 하고, freed memory에 대한 다음 GPU access 앞의 exec revalidation이 모든 VMA를 다시 bind하기 때문입니다.

Exec마다 VMA 일부만 rebind하는 구현이라면 선택되지 않은 모든 VMA는 workload submit 전에 반드시 unbind해야 합니다.

   dma_resv_lock(gpu_vm->resv);

   // Validation section starts here.
   for_each_gpu_vm_bo_on_evict_list(&gpu_vm->evict_list, &gpu_vm_bo) {
           validate_gem_bo(&gpu_vm_bo->gem_bo);

           // The following list iteration needs the Gem object's
           // dma_resv to be held (it protects the gpu_vm_bo's list of
           // gpu_vmas, but since local gem objects share the gpu_vm's
           // dma_resv, it is already held at this point.
           for_each_gpu_vma_of_gpu_vm_bo(&gpu_vm_bo, &gpu_vma)
                  move_gpu_vma_to_rebind_list(&gpu_vma, &gpu_vm->rebind_list);
   }

   for_each_gpu_vma_on_rebind_list(&gpu vm->rebind_list, &gpu_vma) {
           rebind_gpu_vma(&gpu_vma);
           remove_gpu_vma_from_rebind_list(&gpu_vma);
   }
   // Validation section ends here, and job submission starts.

   add_dependencies(&gpu_job, &gpu_vm->resv);
   job_dma_fence = gpu_submit(&gpu_job));

   add_dma_fence(job_dma_fence, &gpu_vm->resv);
   dma_resv_unlock(gpu_vm->resv);

   obj = get_object_from_lru();

   dma_resv_lock(obj->resv);
   for_each_gpu_vm_bo_of_obj(obj, &gpu_vm_bo);
           add_gpu_vm_bo_to_evict_list(&gpu_vm_bo, &gpu_vm->evict_list);

   add_dependencies(&eviction_job, &obj->resv);
   job_dma_fence = gpu_submit(&eviction_job);
   add_dma_fence(&obj->resv, job_dma_fence);

   dma_resv_unlock(&obj->resv);
   put_object(obj);

==========================================

Note that in all the code examples given below we use simplified
pseudo-code. In particular, the dma_resv deadlock avoidance algorithm
as well as reserving memory for dma_resv fences is left out.

Revalidation
____________
With VM_BIND, all local objects need to be resident when the gpu is
executing using the gpu_vm, and the objects need to have valid
gpu_vmas set up pointing to them. Typically, each gpu command buffer
submission is therefore preceded with a re-validation section:

.. code-block:: C

   dma_resv_lock(gpu_vm->resv);

   // Validation section starts here.
   for_each_gpu_vm_bo_on_evict_list(&gpu_vm->evict_list, &gpu_vm_bo) {
           validate_gem_bo(&gpu_vm_bo->gem_bo);

           // The following list iteration needs the Gem object's
           // dma_resv to be held (it protects the gpu_vm_bo's list of
           // gpu_vmas, but since local gem objects share the gpu_vm's
           // dma_resv, it is already held at this point.
           for_each_gpu_vma_of_gpu_vm_bo(&gpu_vm_bo, &gpu_vma)
                  move_gpu_vma_to_rebind_list(&gpu_vma, &gpu_vm->rebind_list);
   }

   for_each_gpu_vma_on_rebind_list(&gpu vm->rebind_list, &gpu_vma) {
           rebind_gpu_vma(&gpu_vma);
           remove_gpu_vma_from_rebind_list(&gpu_vma);
   }
   // Validation section ends here, and job submission starts.

   add_dependencies(&gpu_job, &gpu_vm->resv);
   job_dma_fence = gpu_submit(&gpu_job));

   add_dma_fence(job_dma_fence, &gpu_vm->resv);
   dma_resv_unlock(gpu_vm->resv);

The reason for having a separate gpu_vm rebind list is that there
might be userptr gpu_vmas that are not mapping a buffer object that
also need rebinding.

Eviction
________

Eviction of one of these local objects will then look similar to the
following:

.. code-block:: C

   obj = get_object_from_lru();

   dma_resv_lock(obj->resv);
   for_each_gpu_vm_bo_of_obj(obj, &gpu_vm_bo);
           add_gpu_vm_bo_to_evict_list(&gpu_vm_bo, &gpu_vm->evict_list);

   add_dependencies(&eviction_job, &obj->resv);
   job_dma_fence = gpu_submit(&eviction_job);
   add_dma_fence(&obj->resv, job_dma_fence);

   dma_resv_unlock(&obj->resv);
   put_object(obj);

Note that since the object is local to the gpu_vm, it will share the gpu_vm's
dma_resv lock such that ``obj->resv == gpu_vm->resv``.
The gpu_vm_bos marked for eviction are put on the gpu_vm's evict list,
which is protected by ``gpu_vm->resv``. During eviction all local
objects have their dma_resv locked and, due to the above equality, also
the gpu_vm's dma_resv protecting the gpu_vm's evict list is locked.

With VM_BIND, gpu_vmas don't need to be unbound before eviction,
since the driver must ensure that the eviction blit or copy will wait
for GPU idle or depend on all previous GPU activity. Furthermore, any
subsequent attempt by the GPU to access freed memory through the
gpu_vma will be preceded by a new exec function, with a revalidation
section which will make sure all gpu_vmas are rebound. The eviction
code holding the object's dma_resv while revalidating will ensure a
new exec function may not race with the eviction.

A driver can be implemented in such a way that, on each exec function,
only a subset of vmas are selected for rebind.  In this case, all vmas that are
*not* selected for rebind must be unbound before the exec
function workload is submitted.

Locking with external buffer objects

External buffer object locking

External buffer object는 여러 `gpu_vm`이 공유할 수 있으므로 특정 `gpu_vm`의 reservation object를 공유할 수 없고 자체 dma_resv를 가져야 합니다.

하나 이상의 `gpu_vma`로 VM에 bind된 external object는 VM별 list에 넣습니다. 이 list는 `gpu_vm->resv` 또는 `Spinlock iteration` 절의 `gpu_vm` list spinlock이 보호합니다.

VM reservation을 lock하면 external object list를 안전하게 순회하며 각 external object의 dma_resv를 lock할 수 있습니다. List spinlock을 사용하면 더 정교한 iteration scheme이 필요합니다.

External object를 evict할 때는 그 object가 bind된 모든 `gpu_vm`의 `gpu_vm_bo`를 각 VM evict list에 넣어야 합니다. 그러나 이 시점에는 보통 object 자체 dma_resv만 잡혀 있고 VM dma_resv는 잡혀 있지 않습니다.

Eviction 시점에 `ww_acquire` context가 있다면 VM dma_resv도 잡을 수 있지만 비싼 `ww_mutex` rollback을 유발할 수 있습니다. 단순한 방법은 evicted GEM object의 모든 `gpu_vm_bo`에 `evicted` bool을 표시하는 것입니다. 이 bool은 object dma_resv가 정식으로 보호합니다.

다음 exec에서 external object list를 순회할 때 VM dma_resv와 object dma_resv를 모두 잡습니다. 이때 `evicted` 표시를 확인해 해당 `gpu_vm_bo`를 VM evict list로 옮긴 뒤 backing object validate와 VMA rebind를 수행합니다.

Job submit 후 같은 job dma-fence를 `gpu_vm->resv`뿐 아니라 모든 external object dma_resv에도 등록해야 합니다. 그 다음 모든 reservation lock을 해제합니다.

Shared-object-aware eviction은 local object면 즉시 VM evict list에 추가하고 external object면 `evicted` bool만 표시합니다. 그런 다음 eviction job fence를 object dma_resv에 등록합니다.

   dma_resv_lock(gpu_vm->resv);

   // External object list is protected by the gpu_vm->resv lock.
   for_each_gpu_vm_bo_on_extobj_list(gpu_vm, &gpu_vm_bo) {
           dma_resv_lock(gpu_vm_bo.gem_obj->resv);
           if (gpu_vm_bo_marked_evicted(&gpu_vm_bo))
                   add_gpu_vm_bo_to_evict_list(&gpu_vm_bo, &gpu_vm->evict_list);
   }

   for_each_gpu_vm_bo_on_evict_list(&gpu_vm->evict_list, &gpu_vm_bo) {
           validate_gem_bo(&gpu_vm_bo->gem_bo);

           for_each_gpu_vma_of_gpu_vm_bo(&gpu_vm_bo, &gpu_vma)
                  move_gpu_vma_to_rebind_list(&gpu_vma, &gpu_vm->rebind_list);
   }

   for_each_gpu_vma_on_rebind_list(&gpu vm->rebind_list, &gpu_vma) {
           rebind_gpu_vma(&gpu_vma);
           remove_gpu_vma_from_rebind_list(&gpu_vma);
   }

   add_dependencies(&gpu_job, &gpu_vm->resv);
   job_dma_fence = gpu_submit(&gpu_job));

   add_dma_fence(job_dma_fence, &gpu_vm->resv);
   for_each_external_obj(gpu_vm, &obj)
          add_dma_fence(job_dma_fence, &obj->resv);
   dma_resv_unlock_all_resv_locks();

   obj = get_object_from_lru();

   dma_resv_lock(obj->resv);
   for_each_gpu_vm_bo_of_obj(obj, &gpu_vm_bo)
           if (object_is_vm_local(obj))
                add_gpu_vm_bo_to_evict_list(&gpu_vm_bo, &gpu_vm->evict_list);
           else
                mark_gpu_vm_bo_evicted(&gpu_vm_bo);

   add_dependencies(&eviction_job, &obj->resv);
   job_dma_fence = gpu_submit(&eviction_job);
   add_dma_fence(&obj->resv, job_dma_fence);

   dma_resv_unlock(&obj->resv);
   put_object(obj);

====================================

Since external buffer objects may be shared by multiple gpu_vm's they
can't share their reservation object with a single gpu_vm. Instead
they need to have a reservation object of their own. The external
objects bound to a gpu_vm using one or many gpu_vmas are therefore put on a
per-gpu_vm list which is protected by the gpu_vm's dma_resv lock or
one of the :ref:`gpu_vm list spinlocks <Spinlock iteration>`. Once
the gpu_vm's reservation object is locked, it is safe to traverse the
external object list and lock the dma_resvs of all external
objects. However, if instead a list spinlock is used, a more elaborate
iteration scheme needs to be used.

At eviction time, the gpu_vm_bos of *all* the gpu_vms an external
object is bound to need to be put on their gpu_vm's evict list.
However, when evicting an external object, the dma_resvs of the
gpu_vms the object is bound to are typically not held. Only
the object's private dma_resv can be guaranteed to be held. If there
is a ww_acquire context at hand at eviction time we could grab those
dma_resvs but that could cause expensive ww_mutex rollbacks. A simple
option is to just mark the gpu_vm_bos of the evicted gem object with
an ``evicted`` bool that is inspected before the next time the
corresponding gpu_vm evicted list needs to be traversed. For example, when
traversing the list of external objects and locking them. At that time,
both the gpu_vm's dma_resv and the object's dma_resv is held, and the
gpu_vm_bo marked evicted, can then be added to the gpu_vm's list of
evicted gpu_vm_bos. The ``evicted`` bool is formally protected by the
object's dma_resv.

The exec function becomes

.. code-block:: C

   dma_resv_lock(gpu_vm->resv);

   // External object list is protected by the gpu_vm->resv lock.
   for_each_gpu_vm_bo_on_extobj_list(gpu_vm, &gpu_vm_bo) {
           dma_resv_lock(gpu_vm_bo.gem_obj->resv);
           if (gpu_vm_bo_marked_evicted(&gpu_vm_bo))
                   add_gpu_vm_bo_to_evict_list(&gpu_vm_bo, &gpu_vm->evict_list);
   }

   for_each_gpu_vm_bo_on_evict_list(&gpu_vm->evict_list, &gpu_vm_bo) {
           validate_gem_bo(&gpu_vm_bo->gem_bo);

           for_each_gpu_vma_of_gpu_vm_bo(&gpu_vm_bo, &gpu_vma)
                  move_gpu_vma_to_rebind_list(&gpu_vma, &gpu_vm->rebind_list);
   }

   for_each_gpu_vma_on_rebind_list(&gpu vm->rebind_list, &gpu_vma) {
           rebind_gpu_vma(&gpu_vma);
           remove_gpu_vma_from_rebind_list(&gpu_vma);
   }

   add_dependencies(&gpu_job, &gpu_vm->resv);
   job_dma_fence = gpu_submit(&gpu_job));

   add_dma_fence(job_dma_fence, &gpu_vm->resv);
   for_each_external_obj(gpu_vm, &obj)
          add_dma_fence(job_dma_fence, &obj->resv);
   dma_resv_unlock_all_resv_locks();

And the corresponding shared-object aware eviction would look like:

.. code-block:: C

   obj = get_object_from_lru();

   dma_resv_lock(obj->resv);
   for_each_gpu_vm_bo_of_obj(obj, &gpu_vm_bo)
           if (object_is_vm_local(obj))
                add_gpu_vm_bo_to_evict_list(&gpu_vm_bo, &gpu_vm->evict_list);
           else
                mark_gpu_vm_bo_evicted(&gpu_vm_bo);

   add_dependencies(&eviction_job, &obj->resv);
   job_dma_fence = gpu_submit(&eviction_job);
   add_dma_fence(&obj->resv, job_dma_fence);

   dma_resv_unlock(&obj->resv);
   put_object(obj);

dma_resv 없이 gpu_vm list 순회

일부 driver는 VM evict·external object list에 접근할 때 항상 `gpu_vm` dma_resv를 잡습니다. 그러나 dma-fence signaling critical path의 asynchronous state update처럼 dma_resv 없이 접근해야 하는 driver도 있습니다.

이 경우 list 조작은 spinlock으로 보호할 수 있습니다. 하지만 각 item을 처리하려면 상위 sleeping lock이 필요하므로 이미 방문한 item을 임시 private list로 옮기고, item마다 처리하는 동안 spinlock을 놓아야 합니다.

Pseudo-code는 첫 entry를 `still_in_list`로 옮기고 `list_entry_get_unless_zero()`로 reference를 확보한 뒤 spinlock을 놓고 `process(entry)`를 실행합니다. 다시 spinlock을 잡아 reference를 놓고 반복한 뒤 임시 list를 원래 list tail에 splice합니다.

추가 lock과 atomic operation 비용 때문에 dma_resv 밖에서 list에 접근하지 않아도 되는 driver, 특히 item 수가 많을 것으로 예상되는 driver는 이 scheme을 피하는 편이 좋을 수 있습니다.

반대로 item 수가 적거나 iteration 빈도가 낮거나 각 처리 자체의 비용이 크면 atomic overhead는 대체로 무시할 수 있습니다.

Iteration 중 item이 원래 list에서 임시로 빠지므로 이 전체 iteration은 외부 lock 또는 semaphore가 보호해야 합니다. Local `still_in_list`도 `gpu_vm->list_lock`의 보호 대상이며 iteration과 동시에 local list에서 item이 제거될 수 있음을 고려해야 합니다.

관련 구현은 `drm_gpuvm_locking` 참조와 내부 `get_next_vm_bo_from_list()` function에서 확인할 수 있습니다.

    struct list_head still_in_list;

    INIT_LIST_HEAD(&still_in_list);

    spin_lock(&gpu_vm->list_lock);
    do {
            struct list_head *entry = list_first_entry_or_null(&gpu_vm->list, head);

            if (!entry)
                    break;

            list_move_tail(&entry->head, &still_in_list);
            list_entry_get_unless_zero(entry);
            spin_unlock(&gpu_vm->list_lock);

            process(entry);

            spin_lock(&gpu_vm->list_lock);
            list_entry_put(entry);
    } while (true);

    list_splice_tail(&still_in_list, &gpu_vm->list);
    spin_unlock(&gpu_vm->list_lock);

.. _Spinlock iteration:

Accessing the gpu_vm's lists without the dma_resv lock held
===========================================================

Some drivers will hold the gpu_vm's dma_resv lock when accessing the
gpu_vm's evict list and external objects lists. However, there are
drivers that need to access these lists without the dma_resv lock
held, for example due to asynchronous state updates from within the
dma_fence signalling critical path. In such cases, a spinlock can be
used to protect manipulation of the lists. However, since higher level
sleeping locks need to be taken for each list item while iterating
over the lists, the items already iterated over need to be
temporarily moved to a private list and the spinlock released
while processing each item:

.. code block:: C

    struct list_head still_in_list;

    INIT_LIST_HEAD(&still_in_list);

    spin_lock(&gpu_vm->list_lock);
    do {
            struct list_head *entry = list_first_entry_or_null(&gpu_vm->list, head);

            if (!entry)
                    break;

            list_move_tail(&entry->head, &still_in_list);
            list_entry_get_unless_zero(entry);
            spin_unlock(&gpu_vm->list_lock);

            process(entry);

            spin_lock(&gpu_vm->list_lock);
            list_entry_put(entry);
    } while (true);

    list_splice_tail(&still_in_list, &gpu_vm->list);
    spin_unlock(&gpu_vm->list_lock);

Due to the additional locking and atomic operations, drivers that *can*
avoid accessing the gpu_vm's list outside of the dma_resv lock
might want to avoid also this iteration scheme. Particularly, if the
driver anticipates a large number of list items. For lists where the
anticipated number of list items is small, where list iteration doesn't
happen very often or if there is a significant additional cost
associated with each iteration, the atomic operation overhead
associated with this type of iteration is, most likely, negligible. Note that
if this scheme is used, it is necessary to make sure this list
iteration is protected by an outer level lock or semaphore, since list
items are temporarily pulled off the list while iterating, and it is
also worth mentioning that the local list ``still_in_list`` should
also be considered protected by the ``gpu_vm->list_lock``, and it is
thus possible that items can be removed also from the local list
concurrently with list iteration.

Please refer to the :ref:`DRM GPUVM locking section
<drm_gpuvm_locking>` and its internal
:c:func:`get_next_vm_bo_from_list` function.


userptr gpu_vmas

Userptr mapping과 exec revalidation

Userptr `gpu_vma`는 buffer object 대신 CPU mm의 anonymous page 또는 file page-cache page range를 GPU virtual address에 직접 map합니다.

Bind 때 `pin_user_pages()`로 page를 pin하고 unbind 때 unpin하는 단순 방식은 한 userspace process가 system memory 전체를 pin하는 denial-of-service vector가 됩니다. 올바른 accounting이 있는 특수 use case에서는 가능하지만 일반 해법은 아닙니다.

일반적인 구현은 필요한 page reference를 얻고 CPU mm이 page를 unmap하기 직전에 MMU notifier로 통지받습니다. GPU에 read-only로 map하지 않았다면 page를 dirty로 표시한 뒤 reference를 놓습니다.

MMU notifier가 CPU mm의 page 제거를 알리면 notifier 안에서 VM idle을 기다려 GPU access를 중단해야 합니다. 다음 GPU access 전에는 GPU page table에서 old page를 unmap하고 새 page reference를 다시 얻어야 합니다.

Core mm이 page를 laundry할 때도 unmap notification이 오므로 다음 GPU access 전에 다시 dirty 표시할 수 있습니다. GPU driver에 필요하지 않은 NUMA accounting notification도 유사하게 오지만 특정 notification만 제외하기는 어렵습니다. Device DMA용 MMU notifier 등록은 `mmu-notifier-registration-case` 참조를 따릅니다.

`get_user_pages()`로 `struct page` reference를 얻는 작업은 dma_resv lock 아래에서 할 수 없습니다. CPU page fault를 resolve할 때 잡는 `mmap_lock`과 dma_resv lock의 ordering을 위반하기 때문입니다. 따라서 userptr VMA list는 예제의 `gpu_vm->lock` 같은 outer lock이 보호해야 합니다.

Exec revalidation은 필요할 때 `gpu_vm->lock`을 write mode로 잡고 `mmu_interval_read_begin()`으로 sequence를 얻습니다. Saved sequence와 다르면 새 page pointer를 얻고 잠시 `gpu_vm->resv`를 잡아 VMA를 revalidate list에 추가한 뒤 saved sequence를 갱신합니다.

일반 revalidation과 dependency 추가 뒤 `gpu_vm->userptr_notifier_lock`을 read mode로 잡고 `mmu_interval_read_retry()`를 실행합니다. Sequence가 invalidated됐으면 notifier lock을 놓고 처음부터 retry합니다.

Retry가 필요 없으면 GPU job을 submit하고 job dma-fence를 VM과 모든 external object reservation에 등록합니다. Reservation locks, notifier read lock, VM outer lock 순서로 해제합니다.

`mmu_interval_read_begin()`과 `mmu_interval_read_retry()` 사이가 `userptr_seqlock` read-side critical section입니다. 예제는 VMA 하나만 보이지만 실제로는 VM의 모든 userptr `gpu_vma`를 순회해 검사합니다.

Userptr MMU invalidation notifier는 reclaim context에서 호출될 수 있으므로 ordering 위반을 피하려면 notifier 안에서 어떤 dma_resv lock이나 `gpu_vm->lock`도 잡을 수 없습니다.

   // Exclusive locking mode here is strictly needed only if there are
   // invalidated userptr gpu_vmas present, to avoid concurrent userptr
   // revalidations of the same userptr gpu_vma.
   down_write(&gpu_vm->lock);
   retry:

   // Note: mmu_interval_read_begin() blocks until there is no
   // invalidation notifier running anymore.
   seq = mmu_interval_read_begin(&gpu_vma->userptr_interval);
   if (seq != gpu_vma->saved_seq) {
           obtain_new_page_pointers(&gpu_vma);
           dma_resv_lock(&gpu_vm->resv);
           add_gpu_vma_to_revalidate_list(&gpu_vma, &gpu_vm);
           dma_resv_unlock(&gpu_vm->resv);
           gpu_vma->saved_seq = seq;
   }

   // The usual revalidation goes here.

   // Final userptr sequence validation may not happen before the
   // submission dma_fence is added to the gpu_vm's resv, from the POW
   // of the MMU invalidation notifier. Hence the
   // userptr_notifier_lock that will make them appear atomic.

   add_dependencies(&gpu_job, &gpu_vm->resv);
   down_read(&gpu_vm->userptr_notifier_lock);
   if (mmu_interval_read_retry(&gpu_vma->userptr_interval, gpu_vma->saved_seq)) {
          up_read(&gpu_vm->userptr_notifier_lock);
          goto retry;
   }

   job_dma_fence = gpu_submit(&gpu_job));

   add_dma_fence(job_dma_fence, &gpu_vm->resv);

   for_each_external_obj(gpu_vm, &obj)
          add_dma_fence(job_dma_fence, &obj->resv);

   dma_resv_unlock_all_resv_locks();
   up_read(&gpu_vm->userptr_notifier_lock);
   up_write(&gpu_vm->lock);

================

A userptr gpu_vma is a gpu_vma that, instead of mapping a buffer object to a
GPU virtual address range, directly maps a CPU mm range of anonymous-
or file page-cache pages.
A very simple approach would be to just pin the pages using
pin_user_pages() at bind time and unpin them at unbind time, but this
creates a Denial-Of-Service vector since a single user-space process
would be able to pin down all of system memory, which is not
desirable. (For special use-cases and assuming proper accounting pinning might
still be a desirable feature, though). What we need to do in the
general case is to obtain a reference to the desired pages, make sure
we are notified using a MMU notifier just before the CPU mm unmaps the
pages, dirty them if they are not mapped read-only to the GPU, and
then drop the reference.
When we are notified by the MMU notifier that CPU mm is about to drop the
pages, we need to stop GPU access to the pages by waiting for VM idle
in the MMU notifier and make sure that before the next time the GPU
tries to access whatever is now present in the CPU mm range, we unmap
the old pages from the GPU page tables and repeat the process of
obtaining new page references. (See the :ref:`notifier example
<Invalidation example>` below). Note that when the core mm decides to
laundry pages, we get such an unmap MMU notification and can mark the
pages dirty again before the next GPU access. We also get similar MMU
notifications for NUMA accounting which the GPU driver doesn't really
need to care about, but so far it has proven difficult to exclude
certain notifications.

Using a MMU notifier for device DMA (and other methods) is described in
:ref:`the pin_user_pages() documentation <mmu-notifier-registration-case>`.

Now, the method of obtaining struct page references using
get_user_pages() unfortunately can't be used under a dma_resv lock
since that would violate the locking order of the dma_resv lock vs the
mmap_lock that is grabbed when resolving a CPU pagefault. This means
the gpu_vm's list of userptr gpu_vmas needs to be protected by an
outer lock, which in our example below is the ``gpu_vm->lock``.

The MMU interval seqlock for a userptr gpu_vma is used in the following
way:

.. code-block:: C

   // Exclusive locking mode here is strictly needed only if there are
   // invalidated userptr gpu_vmas present, to avoid concurrent userptr
   // revalidations of the same userptr gpu_vma.
   down_write(&gpu_vm->lock);
   retry:

   // Note: mmu_interval_read_begin() blocks until there is no
   // invalidation notifier running anymore.
   seq = mmu_interval_read_begin(&gpu_vma->userptr_interval);
   if (seq != gpu_vma->saved_seq) {
           obtain_new_page_pointers(&gpu_vma);
           dma_resv_lock(&gpu_vm->resv);
           add_gpu_vma_to_revalidate_list(&gpu_vma, &gpu_vm);
           dma_resv_unlock(&gpu_vm->resv);
           gpu_vma->saved_seq = seq;
   }

   // The usual revalidation goes here.

   // Final userptr sequence validation may not happen before the
   // submission dma_fence is added to the gpu_vm's resv, from the POW
   // of the MMU invalidation notifier. Hence the
   // userptr_notifier_lock that will make them appear atomic.

   add_dependencies(&gpu_job, &gpu_vm->resv);
   down_read(&gpu_vm->userptr_notifier_lock);
   if (mmu_interval_read_retry(&gpu_vma->userptr_interval, gpu_vma->saved_seq)) {
          up_read(&gpu_vm->userptr_notifier_lock);
          goto retry;
   }

   job_dma_fence = gpu_submit(&gpu_job));

   add_dma_fence(job_dma_fence, &gpu_vm->resv);

   for_each_external_obj(gpu_vm, &obj)
          add_dma_fence(job_dma_fence, &obj->resv);

   dma_resv_unlock_all_resv_locks();
   up_read(&gpu_vm->userptr_notifier_lock);
   up_write(&gpu_vm->lock);

The code between ``mmu_interval_read_begin()`` and the
``mmu_interval_read_retry()`` marks the read side critical section of
what we call the ``userptr_seqlock``. In reality, the gpu_vm's userptr
gpu_vma list is looped through, and the check is done for *all* of its
userptr gpu_vmas, although we only show a single one here.

The userptr gpu_vma MMU invalidation notifier might be called from
reclaim context and, again, to avoid locking order violations, we can't
take any dma_resv lock nor the gpu_vm->lock from within it.

MMU invalidation과 효율적인 userptr iteration

Invalidation notifier는 exec function이 새 sequence를 보고 물러나거나 이미 제출된 job의 dma-fence를 notifier가 기다리도록 만들어야 합니다.

Notifier는 `gpu_vm->userptr_notifier_lock`을 write mode로 잡고 `mmu_interval_set_seq()`로 현재 sequence를 invalid value로 설정한 뒤 unlock합니다. 이 시점부터 exec의 final retry check는 반드시 실패하므로 새 job을 성공적으로 submit할 수 없습니다.

모든 invalidation callback이 끝나면 MMU notifier core가 sequence를 다시 valid value로 바꿉니다. 그 전에 old page에 대한 GPU access를 멈춰야 하므로 notifier는 `gpu_vm->resv`의 `DMA_RESV_USAGE_BOOKKEEP` fence를 `MAX_SCHEDULE_TIMEOUT`까지 기다립니다.

Invalidation notifier가 반환하면 GPU는 더 이상 old userptr page에 접근하지 않으며 새 GPU submission이 성공하려면 page binding을 다시 수행해야 합니다.

Userptr list가 커지면 exec마다 모든 VMA의 saved sequence를 확인하는 방식은 비효율적입니다. 해결책은 invalidated userptr VMA만 별도 VM list에 넣고 exec마다 그 list만 검사하는 것입니다.

MMU notifier에서는 `gpu_vm->lock`이나 `gpu_vm->resv` 같은 outer lock을 잡을 수 없으므로 invalidated list에는 앞의 spinlock iteration scheme이 잘 맞습니다. 다만 iteration 중 list가 완전함을 보장하려면 여전히 `gpu_vm->lock`을 잡아야 합니다.

이 별도 list를 사용하면 exec retry check는 invalidated list가 비었는지 확인하는 단순한 검사로 바뀝니다.

  bool gpu_vma_userptr_invalidate(userptr_interval, cur_seq)
  {
          // Make sure the exec function either sees the new sequence
          // and backs off or we wait for the dma-fence:

          down_write(&gpu_vm->userptr_notifier_lock);
          mmu_interval_set_seq(userptr_interval, cur_seq);
          up_write(&gpu_vm->userptr_notifier_lock);

          // At this point, the exec function can't succeed in
          // submitting a new job, because cur_seq is an invalid
          // sequence number and will always cause a retry. When all
          // invalidation callbacks, the mmu notifier core will flip
          // the sequence number to a valid one. However we need to
          // stop gpu access to the old pages here.

          dma_resv_wait_timeout(&gpu_vm->resv, DMA_RESV_USAGE_BOOKKEEP,
                                false, MAX_SCHEDULE_TIMEOUT);
          return true;
  }

.. _Invalidation example:
.. code-block:: C

  bool gpu_vma_userptr_invalidate(userptr_interval, cur_seq)
  {
          // Make sure the exec function either sees the new sequence
          // and backs off or we wait for the dma-fence:

          down_write(&gpu_vm->userptr_notifier_lock);
          mmu_interval_set_seq(userptr_interval, cur_seq);
          up_write(&gpu_vm->userptr_notifier_lock);

          // At this point, the exec function can't succeed in
          // submitting a new job, because cur_seq is an invalid
          // sequence number and will always cause a retry. When all
          // invalidation callbacks, the mmu notifier core will flip
          // the sequence number to a valid one. However we need to
          // stop gpu access to the old pages here.

          dma_resv_wait_timeout(&gpu_vm->resv, DMA_RESV_USAGE_BOOKKEEP,
                                false, MAX_SCHEDULE_TIMEOUT);
          return true;
  }

When this invalidation notifier returns, the GPU can no longer be
accessing the old pages of the userptr gpu_vma and needs to redo the
page-binding before a new GPU submission can succeed.

Efficient userptr gpu_vma exec_function iteration
_________________________________________________

If the gpu_vm's list of userptr gpu_vmas becomes large, it's
inefficient to iterate through the complete lists of userptrs on each
exec function to check whether each userptr gpu_vma's saved
sequence number is stale. A solution to this is to put all
*invalidated* userptr gpu_vmas on a separate gpu_vm list and
only check the gpu_vmas present on this list on each exec
function. This list will then lend itself very-well to the spinlock
locking scheme that is
:ref:`described in the spinlock iteration section <Spinlock iteration>`, since
in the mmu notifier, where we add the invalidated gpu_vmas to the
list, it's not possible to take any outer locks like the
``gpu_vm->lock`` or the ``gpu_vm->resv`` lock. Note that the
``gpu_vm->lock`` still needs to be taken while iterating to ensure the list is
complete, as also mentioned in that section.

If using an invalidated userptr list like this, the retry check in the
exec function trivially becomes a check for invalidated list empty.

Locking at bind and unbind time

Bind·unbind 시 linking과 lifetime

GEM object-backed `gpu_vma`를 bind할 때 각 VMA를 `gpu_vm_bo`와 연결하고, 그 `gpu_vm_bo`를 GEM object의 `gpu_vm_bo` list와 필요하면 VM external object list에 추가해야 합니다. 이를 `gpu_vma` linking이라고 합니다.

Linking에는 보통 `gpu_vm->lock`과 `gem_object->gpuva_lock`이 모두 필요합니다. Unlinking할 때도 같은 lock을 잡아야 합니다.

이 규칙은 `gpu_vm->resv` 또는 GEM object dma_resv 아래에서 `gpu_vma`를 순회하는 동안 해당 iteration lock을 놓지 않는 한 VMA가 살아 있도록 보장합니다.

Userptr `gpu_vma`를 destroy할 때도 outer `gpu_vm->lock`을 잡아야 합니다. 그렇지 않으면 invalidated userptr list를 순회할 때 VMA lifetime을 보장할 수 없습니다.

===============================

At bind time, assuming a GEM object backed gpu_vma, each
gpu_vma needs to be associated with a gpu_vm_bo and that
gpu_vm_bo in turn needs to be added to the GEM object's
gpu_vm_bo list, and possibly to the gpu_vm's external object
list. This is referred to as *linking* the gpu_vma, and typically
requires that the ``gpu_vm->lock`` and the ``gem_object->gpuva_lock``
are held. When unlinking a gpu_vma the same locks should be held,
and that ensures that when iterating over ``gpu_vmas`, either under
the ``gpu_vm->resv`` or the GEM object's dma_resv, that the gpu_vmas
stay alive as long as the lock under which we iterate is not released. For
userptr gpu_vmas it's similarly required that during vma destroy, the
outer ``gpu_vm->lock`` is held, since otherwise when iterating over
the invalidated userptr list as described in the previous section,
there is nothing keeping those userptr gpu_vmas alive.

Locking for recoverable page-fault page-table updates

Recoverable page-fault의 page-table locking

Recoverable page fault locking은 두 조건을 보장해야 합니다. 첫째, page를 system 또는 allocator에 reuse 용도로 돌려줄 때 남은 GPU mapping이 없어야 하고 모든 GPU TLB가 flush되어야 합니다. 둘째, `gpu_vma`의 unmapping과 mapping이 race하면 안 됩니다.

GPU PTE unmapping 또는 zapping은 outer lock을 잡기 어렵거나 불가능한 context에서 일어나는 경우가 많습니다. 따라서 mapping과 unmapping 양쪽에서 잡는 새 lock을 만들거나, unmapping 시 이미 잡는 lock을 mapping 때도 잡아야 합니다.

Userptr `gpu_vma`는 MMU invalidation notifier의 zapping 동안 `userptr_seqlock` write side가 잡혀 있습니다. Mapping 동안 `userptr_seqlock`과 `gpu_vm->userptr_notifier_lock`을 모두 read mode로 잡으면 zapping과 race하지 않습니다.

GEM object-backed `gpu_vma`의 zapping은 GEM object dma_resv 아래에서 수행됩니다. 해당 object를 가리키는 VMA의 page table을 populate할 때도 같은 dma_resv를 잡으면 race를 막을 수 있습니다.

Mapping 일부를 lock 없이 dma-fence 아래에서 asynchronous하게 수행한다면 zapping은 page table을 수정하기 전에 관련 lock 아래에서 그 dma-fence가 signal되기를 기다려야 합니다.

Page-table memory를 free하는 구조 변경에는 outer lock이 필요할 수 있습니다. 따라서 GPU PTE zapping은 보통 page-table 또는 page-directory entry를 0으로 만들고 TLB를 flush하는 데 집중하며, page-table memory 해제는 unbind 또는 rebind 시점으로 미룹니다.

=====================================================

There are two important things we need to ensure with locking for
recoverable page-faults:

* At the time we return pages back to the system / allocator for
  reuse, there should be no remaining GPU mappings and any GPU TLB
  must have been flushed.
* The unmapping and mapping of a gpu_vma must not race.

Since the unmapping (or zapping) of GPU ptes is typically taking place
where it is hard or even impossible to take any outer level locks we
must either introduce a new lock that is held at both mapping and
unmapping time, or look at the locks we do hold at unmapping time and
make sure that they are held also at mapping time. For userptr
gpu_vmas, the ``userptr_seqlock`` is held in write mode in the mmu
invalidation notifier where zapping happens. Hence, if the
``userptr_seqlock`` as well as the ``gpu_vm->userptr_notifier_lock``
is held in read mode during mapping, it will not race with the
zapping. For GEM object backed gpu_vmas, zapping will take place under
the GEM object's dma_resv and ensuring that the dma_resv is held also
when populating the page-tables for any gpu_vma pointing to the GEM
object, will similarly ensure we are race-free.

If any part of the mapping is performed asynchronously
under a dma-fence with these locks released, the zapping will need to
wait for that dma-fence to signal under the relevant lock before
starting to modify the page-table.

Since modifying the
page-table structure in a way that frees up page-table memory
might also require outer level locks, the zapping of GPU ptes
typically focuses only on zeroing page-table or page-directory entries
and flushing TLB, whereas freeing of page-table memory is deferred to
unbind or rebind time.