I915 VM_BIND feature design and use cases

VM_BIND persistent mapping과 TLB

`DRM_I915_GEM_VM_BIND/UNBIND` ioctl은 UMD가 GEM BO 전체 또는 일부를 지정 VM의 GPU virtual address에 bind·unbind하게 합니다. Persistent mapping은 여러 execbuf submission 사이에 유지되므로 과거 execbuf처럼 매번 필요한 mapping list를 제출하지 않습니다.

Bind/unbind 완료를 알리는 timeline out fence를 요청할 수 있습니다. Feature version은 `I915_PARAM_VM_BIND_VERSION`으로 알리고, VM 생성 때 `I915_VM_CREATE_FLAGS_USE_VM_BIND` extension으로 opt-in해야 합니다.

서로 다른 CPU thread에서 동시에 실행한 VM_BIND/UNBIND ioctl 사이에는 ordering이 없습니다. 유효한 out fence를 지정하면 operation 일부를 asynchronous하게 수행할 수 있습니다.

Feature는 같은 physical page에 여러 VA를 mapping하는 aliasing, BO 일부만 mapping하는 partial binding, GPU error dump의 persistent mapping capture, 별도 uAPI 없는 userptr GEM object를 지원합니다.

i915는 각 submission과 object page release 때 TLB를 flush합니다. VM_BIND/UNBIND 자체는 추가 flush를 하지 않습니다. 새 mapping은 이후 submission의 working set에는 들어가지만 현재 실행 중인 batch에는 들어가지 않습니다. 후자는 추가 TLB flush가 필요하지만 지원하지 않습니다.

==========================================
I915 VM_BIND feature design and use cases
==========================================

VM_BIND feature
================
DRM_I915_GEM_VM_BIND/UNBIND ioctls allows UMD to bind/unbind GEM buffer
objects (BOs) or sections of a BOs at specified GPU virtual addresses on a
specified address space (VM). These mappings (also referred to as persistent
mappings) will be persistent across multiple GPU submissions (execbuf calls)
issued by the UMD, without user having to provide a list of all required
mappings during each submission (as required by older execbuf mode).

The VM_BIND/UNBIND calls allow UMDs to request a timeline out fence for
signaling the completion of bind/unbind operation.

VM_BIND feature is advertised to user via I915_PARAM_VM_BIND_VERSION.
User has to opt-in for VM_BIND mode of binding for an address space (VM)
during VM creation time via I915_VM_CREATE_FLAGS_USE_VM_BIND extension.

VM_BIND/UNBIND ioctl calls executed on different CPU threads concurrently are
not ordered. Furthermore, parts of the VM_BIND/UNBIND operations can be done
asynchronously, when valid out fence is specified.

VM_BIND features include:

* Multiple Virtual Address (VA) mappings can map to the same physical pages
  of an object (aliasing).
* VA mapping can map to a partial section of the BO (partial binding).
* Support capture of persistent mappings in the dump upon GPU error.
* Support for userptr gem objects (no special uapi is required for this).

TLB flush consideration
------------------------
The i915 driver flushes the TLB for each submission and when an object's
pages are released. The VM_BIND/UNBIND operation will not do any additional
TLB flush. Any VM_BIND mapping added will be in the working set for subsequent
submissions on that VM and will not be in the working set for currently running
batches (which would require additional TLB flushes, which is not supported).

execbuf3와 VM_PRIVATE BO

VM_BIND mode VM은 과거 execbuf binding mode를 지원하지 않습니다. 처리 방식이 `drm_i915_gem_execbuffer2`와 크게 달라 새 `execbuf3` ioctl과 `drm_i915_gem_execbuffer3`를 사용합니다.

Execbuf3는 execlist를 받지 않아 implicit sync를 지원하지 않습니다. Object dependency는 dma-buf sync-file export API가 충족할 것으로 기대합니다. Execbuf3는 VM_BIND mode에서만 동작하고 VM_BIND mode도 submission에 execbuf3만 사용합니다.

Execbuf3 호출 시점에 VM_BIND로 해당 VM에 mapping된 모든 BO를 submission에 필요하다고 봅니다. Batch는 execbuf2의 object handle 대신 GPU address로 직접 지정합니다. In/out/submit fence, fence array, default GEM context 같은 과거 기능도 지원하지 않습니다.

VA allocation은 i915가 아니라 user가 완전히 관리하므로 driver VA assignment·eviction은 적용되지 않습니다. Object active 여부도 `i915_vma` active reference 대신 `dma-resv` object로 추적합니다.

따라서 relocation, VA eviction, VMA lookup table, implicit sync 같은 execbuf2 전용 code를 execbuf3에 사용하지 않습니다. Execbuf3 handling은 별도 file에 두고 정말 공통인 기능만 공유합니다.

기본 Shared BO는 여러 VM에 mapping하고 dma-buf export할 수 있습니다. 매 execbuf submission에서 VM에 mapping된 모든 shared BO의 dma-resv fence list에 request fence를 추가합니다.

`I915_GEM_CREATE_EXT_VM_PRIVATE` flag로 특정 VM 전용 BO를 만들 수 있습니다. Private BO는 그 VM에만 mapping할 수 있고 dma-buf export할 수 없습니다. VM의 모든 private BO가 dma-resv object 하나를 공유하므로 submission마다 fence list 하나만 갱신합니다. 이미 mapping된 fast path latency는 private BO 수에 대해 O(1)입니다.

Execbuf ioctl in VM_BIND mode
-------------------------------
A VM in VM_BIND mode will not support older execbuf mode of binding.
The execbuf ioctl handling in VM_BIND mode differs significantly from the
older execbuf2 ioctl (See struct drm_i915_gem_execbuffer2).
Hence, a new execbuf3 ioctl has been added to support VM_BIND mode. (See
struct drm_i915_gem_execbuffer3). The execbuf3 ioctl will not accept any
execlist. Hence, no support for implicit sync. It is expected that the below
work will be able to support requirements of object dependency setting in all
use cases:

"dma-buf: Add an API for exporting sync files"
(https://lwn.net/Articles/859290/)

The new execbuf3 ioctl only works in VM_BIND mode and the VM_BIND mode only
works with execbuf3 ioctl for submission. All BOs mapped on that VM (through
VM_BIND call) at the time of execbuf3 call are deemed required for that
submission.

The execbuf3 ioctl directly specifies the batch addresses instead of as
object handles as in execbuf2 ioctl. The execbuf3 ioctl will also not
support many of the older features like in/out/submit fences, fence array,
default gem context and many more (See struct drm_i915_gem_execbuffer3).

In VM_BIND mode, VA allocation is completely managed by the user instead of
the i915 driver. Hence all VA assignment, eviction are not applicable in
VM_BIND mode. Also, for determining object activeness, VM_BIND mode will not
be using the i915_vma active reference tracking. It will instead use dma-resv
object for that (See `VM_BIND dma_resv usage`_).

So, a lot of existing code supporting execbuf2 ioctl, like relocations, VA
evictions, vma lookup table, implicit sync, vma active reference tracking etc.,
are not applicable for execbuf3 ioctl. Hence, all execbuf3 specific handling
should be in a separate file and only functionalities common to these ioctls
can be the shared code where possible.

VM_PRIVATE objects
-------------------
By default, BOs can be mapped on multiple VMs and can also be dma-buf
exported. Hence these BOs are referred to as Shared BOs.
During each execbuf submission, the request fence must be added to the
dma-resv fence list of all shared BOs mapped on the VM.

VM_BIND feature introduces an optimization where user can create BO which
is private to a specified VM via I915_GEM_CREATE_EXT_VM_PRIVATE flag during
BO creation. Unlike Shared BOs, these VM private BOs can only be mapped on
the VM they are private to and can't be dma-buf exported.
All private BOs of a VM share the dma-resv object. Hence during each execbuf
submission, they need only one dma-resv fence list updated. Thus, the fast
path (where required mappings are already bound) submission latency is O(1)
w.r.t the number of VM private BOs.

Lock A/B/C와 LRU

Locking 설계는 legacy execlist execbuf, 새 VM_BIND, GPU page fault가 있는 VM_BIND, 미래 system allocator를 함께 지원합니다. 앞의 두 mode는 backing storage residency에 dma_fence를 사용하지만 page-fault·system allocator mode는 dma_fence를 사용하지 않습니다.

Lock-A는 `vm_bind` mutex로 bind list를 보호하며 bind/unbind ioctl, execbuf path, mapping release에서 잡습니다. GPU page fault를 지원하면 여러 handler가 mapping lookup을 병렬 수행하도록 rwsem의 read side로 바꿀 수 있습니다. Legacy execbuf binding은 이 lock이 필요 없습니다.

Lock-B는 object의 dma-resv lock입니다. `i915_vma` state를 보호하고 async worker의 VMA bind/unbind와 object fence-list update 동안 유지합니다. 한 VM의 private BO는 dma-resv object를 공유합니다. 미래 system allocator는 HMM이 정한 locking을 사용합니다.

Lock-C는 eviction·userptr invalidation으로 invalidated된 VMA list 같은 VM list를 보호하는 spinlock입니다.

GPU page fault mode의 execbuf path는 이 lock을 하나도 잡지 않고 새 batch-buffer address를 ring에 기록한 뒤 scheduler를 실행합니다. Fault handler가 Lock-A read side, backing storage에 맞는 Lock-B(dma_resv 또는 HMM/core MM), page-table race용 추가 Lock-D를 잡습니다. Fault mode는 VM list를 조작하지 않아 Lock-C가 필요 없습니다.

VM_BIND object가 적절한 LRU tag를 갖도록 하고 execbuf latency를 줄이기 위해 bulk LRU movement를 지원해야 합니다. Page-table page도 VM별 persistent mapping과 비슷하며 VM이 execbuf로 active해질 때 memory에 pin하므로 page-table page의 bulk LRU movement도 필요합니다.

VM_BIND locking hierarchy
-------------------------
The locking design here supports the older (execlist based) execbuf mode, the
newer VM_BIND mode, the VM_BIND mode with GPU page faults and possible future
system allocator support (See `Shared Virtual Memory (SVM) support`_).
The older execbuf mode and the newer VM_BIND mode without page faults manages
residency of backing storage using dma_fence. The VM_BIND mode with page faults
and the system allocator support do not use any dma_fence at all.

VM_BIND locking order is as below.

1) Lock-A: A vm_bind mutex will protect vm_bind lists. This lock is taken in
   vm_bind/vm_unbind ioctl calls, in the execbuf path and while releasing the
   mapping.

   In future, when GPU page faults are supported, we can potentially use a
   rwsem instead, so that multiple page fault handlers can take the read side
   lock to lookup the mapping and hence can run in parallel.
   The older execbuf mode of binding do not need this lock.

2) Lock-B: The object's dma-resv lock will protect i915_vma state and needs to
   be held while binding/unbinding a vma in the async worker and while updating
   dma-resv fence list of an object. Note that private BOs of a VM will all
   share a dma-resv object.

   The future system allocator support will use the HMM prescribed locking
   instead.

3) Lock-C: Spinlock/s to protect some of the VM's lists like the list of
   invalidated vmas (due to eviction and userptr invalidation) etc.

When GPU page faults are supported, the execbuf path do not take any of these
locks. There we will simply smash the new batch buffer address into the ring and
then tell the scheduler run that. The lock taking only happens from the page
fault handler, where we take lock-A in read mode, whichever lock-B we need to
find the backing storage (dma_resv lock for gem objects, and hmm/core mm for
system allocator) and some additional locks (lock-D) for taking care of page
table races. Page fault mode should not need to ever manipulate the vm lists,
so won't ever need lock-C.

VM_BIND LRU handling
---------------------
We need to ensure VM_BIND mapped objects are properly LRU tagged to avoid
performance degradation. We will also need support for bulk LRU movement of
VM_BIND objects to avoid additional latencies in execbuf path.

The page table pages are similar to VM_BIND mapped objects (See
`Evictable page table allocations`_) and are maintained per VM and needs to
be pinned in memory when VM is made active (ie., upon an execbuf call with
that VM). So, bulk LRU movement of page table pages is also needed.

dma_resv usage와 Mesa 효과

모든 VM_BIND mapped object에 fence를 추가해야 합니다. Execbuf submission마다 과도한 synchronization을 막기 위해 `DMA_RESV_USAGE_BOOKKEEP` usage로 추가하며 explicit dependency 설정에서는 `DMA_RESV_USAGE_READ` 또는 `DMA_RESV_USAGE_WRITE`로 override할 수 있습니다.

`DRM_I915_GEM_WAIT`와 `DRM_I915_GEM_BUSY` ioctl은 BOOKKEEP fence를 검사하지 않으므로 batch 종료 확인에 쓰면 안 됩니다. 대신 execbuf3 out fence를 사용해야 합니다.

Object active 여부는 deprecated된 `i915_vma` active reference가 아니라 `dma_resv_test_signaled()`와 `dma_resv_wait_timeout()` 같은 dma-resv API로 판단합니다. 이는 현재 TTM backend와 통합하기도 쉽습니다.

VM_BIND는 Vulkan과 Iris의 Mesa CPU overhead를 줄여 CPU-bound application 성능을 개선할 수 있고 Vulkan Sparse Resources 구현도 가능하게 합니다. GPU 성능이 높아질수록 CPU overhead 절감 효과가 커집니다.

VM_BIND dma_resv usage
-----------------------
Fences needs to be added to all VM_BIND mapped objects. During each execbuf
submission, they are added with DMA_RESV_USAGE_BOOKKEEP usage to prevent
over sync (See enum dma_resv_usage). One can override it with either
DMA_RESV_USAGE_READ or DMA_RESV_USAGE_WRITE usage during explicit object
dependency setting.

Note that DRM_I915_GEM_WAIT and DRM_I915_GEM_BUSY ioctls do not check for
DMA_RESV_USAGE_BOOKKEEP usage and hence should not be used for end of batch
check. Instead, the execbuf3 out fence should be used for end of batch check
(See struct drm_i915_gem_execbuffer3).

Also, in VM_BIND mode, use dma-resv apis for determining object activeness
(See dma_resv_test_signaled() and dma_resv_wait_timeout()) and do not use the
older i915_vma active reference tracking which is deprecated. This should be
easier to get it working with the current TTM backend.

Mesa use case
--------------
VM_BIND can potentially reduce the CPU overhead in Mesa (both Vulkan and Iris),
hence improving performance of CPU-bound applications. It also allows us to
implement Vulkan's Sparse Resources. With increasing GPU hardware performance,
reducing CPU overhead becomes more impactful.

Compute·fault·debug·SVM use case

Long-running compute는 합리적 시간 안에 끝나야 하는 dma-fence와 맞지 않으므로 user/memory fence를 사용하고 dma-fence는 kernel 내부 소비로 제한합니다.

GPU page fault가 없으면 buffer invalidation 때 kernel이 long-running context를 suspend(preempt)하고 invalidation·BO revalidation을 끝낸 뒤 resume합니다. Context별 preempt fence는 누군가 wait할 때 활성화되어 context preemption을 trigger합니다.

User/Memory Fence는 `<address, value>` pair입니다. Signal할 때 지정 virtual address에 value를 쓰고 waiting process를 깨웁니다. GPU 또는 bind-completion async worker가 signal할 수 있고 새 user-fence-wait ioctl로 기다립니다.

Low Latency Submission은 compute UMD가 execbuf를 거치지 않고 GPU job을 직접 제출하게 합니다. VM_BIND가 execbuf와 동기화되지 않기 때문에 가능하며, 직접 제출할 job의 mapping도 VM_BIND/UNBIND로 관리합니다.

Debug event interface를 사용하면 debugger process가 다른 debugged process가 생성하고 VM_BIND로 GPU에 attach한 resource를 추적하고 조작할 수 있습니다.

미래 GPU page fault는 VM_BIND mode에서만 지원합니다. Legacy와 non-fault VM_BIND는 residency에 dma-fence를 쓰지만 fault mode는 PTE 설치·제거·invalidation으로 residency를 관리하므로 dma-fence를 사용하지 않습니다.

VM_BIND는 placement·atomicity hint를 BO가 아니라 mapping별로 설정합니다. On-demand page fault에서는 sub-BO placement hint가 더 중요합니다. Cache/CLOS도 mapping별로 설정할 수 있습니다.

Page-table allocation을 evictable하게 만들고 VM_BIND object처럼 관리합니다. 차이는 page-table page에는 `i915_vma`가 없고 swap-in 뒤 parent-page link를 갱신해야 한다는 점입니다.

SVM은 HMM interface로 GEM BO abstraction 없이 system memory를 직접 VM_BIND mapping합니다. GPU page fault가 활성화된 경우에만 지원합니다.

Other VM_BIND use cases
========================

Long running Compute contexts
------------------------------
Usage of dma-fence expects that they complete in reasonable amount of time.
Compute on the other hand can be long running. Hence it is appropriate for
compute to use user/memory fence (See `User/Memory Fence`_) and dma-fence usage
must be limited to in-kernel consumption only.

Where GPU page faults are not available, kernel driver upon buffer invalidation
will initiate a suspend (preemption) of long running context, finish the
invalidation, revalidate the BO and then resume the compute context. This is
done by having a per-context preempt fence which is enabled when someone tries
to wait on it and triggers the context preemption.

User/Memory Fence
~~~~~~~~~~~~~~~~~~
User/Memory fence is a <address, value> pair. To signal the user fence, the
specified value will be written at the specified virtual address and wakeup the
waiting process. User fence can be signaled either by the GPU or kernel async
worker (like upon bind completion). User can wait on a user fence with a new
user fence wait ioctl.

Here is some prior work on this:
https://patchwork.freedesktop.org/patch/349417/

Low Latency Submission
~~~~~~~~~~~~~~~~~~~~~~~
Allows compute UMD to directly submit GPU jobs instead of through execbuf
ioctl. This is made possible by VM_BIND is not being synchronized against
execbuf. VM_BIND allows bind/unbind of mappings required for the directly
submitted jobs.

Debugger
---------
With debug event interface user space process (debugger) is able to keep track
of and act upon resources created by another process (debugged) and attached
to GPU via vm_bind interface.

GPU page faults
----------------
GPU page faults when supported (in future), will only be supported in the
VM_BIND mode. While both the older execbuf mode and the newer VM_BIND mode of
binding will require using dma-fence to ensure residency, the GPU page faults
mode when supported, will not use any dma-fence as residency is purely managed
by installing and removing/invalidating page table entries.

Page level hints settings
--------------------------
VM_BIND allows any hints setting per mapping instead of per BO. Possible hints
include placement and atomicity. Sub-BO level placement hint will be even more
relevant with upcoming GPU on-demand page fault support.

Page level Cache/CLOS settings
-------------------------------
VM_BIND allows cache/CLOS settings per mapping instead of per BO.

Evictable page table allocations
---------------------------------
Make pagetable allocations evictable and manage them similar to VM_BIND
mapped objects. Page table pages are similar to persistent mappings of a
VM (difference here are that the page table pages will not have an i915_vma
structure and after swapping pages back in, parent page link needs to be
updated).

Shared Virtual Memory (SVM) support
------------------------------------
VM_BIND interface can be used to map system memory directly (without gem BO
abstraction) using the HMM interface. SVM is only supported with GPU page
faults enabled.

VM_BIND UAPI kernel-doc

VM_BIND UAPI의 structure와 ioctl 계약은 `Documentation/gpu/rfc/i915_vm_bind.h`의 `kernel-doc`에서 포함합니다.

VM_BIND UAPI
=============

.. kernel-doc:: Documentation/gpu/rfc/i915_vm_bind.h