Memory-to-memory Stateless Video Decoder 인터페이스

Stateless decoder와 request API

Stateless decoder는 처리한 frame 사이에 decoding state를 보관하지 않습니다. 각 frame을 이전·이후 frame과 독립적으로 decode하며, client가 decoding state를 유지하고 매 decode request에 필요한 상태를 모두 제공해야 합니다.

Stateful decoder에서는 hardware와 driver가 state를 유지하므로 client는 encoded stream을 넣고 display order로 decoded frame을 꺼내면 됩니다. Stateless 방식은 decoder/client 순서가 단순한 대신 일관된 state 유지와 reference 관리 복잡성이 user space로 이동합니다.

Stateless decoder는 media request API를 사용합니다. `VIDIOC_REQBUFS` 또는 `VIDIOC_CREATE_BUFS`로 `OUTPUT` queue를 조회할 때 `V4L2_BUF_CAP_SUPPORTS_REQUESTS` capability를 반드시 노출해야 합니다.

H.264 multi-slice처럼 decoded frame 하나가 여러 decode request의 결과가 될 수 있는 format을 지원하면 `OUTPUT` queue에 `V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`도 노출해야 합니다.

.. SPDX-License-Identifier: GPL-2.0

.. _stateless_decoder:

**************************************************
Memory-to-memory Stateless Video Decoder Interface
**************************************************

A stateless decoder is a decoder that works without retaining any kind of state
between processed frames. This means that each frame is decoded independently
of any previous and future frames, and that the client is responsible for
maintaining the decoding state and providing it to the decoder with each
decoding request. This is in contrast to the stateful video decoder interface,
where the hardware and driver maintain the decoding state and all the client
has to do is to provide the raw encoded stream and dequeue decoded frames in
display order.

This section describes how user-space ("the client") is expected to communicate
with stateless decoders in order to successfully decode an encoded stream.
Compared to stateful codecs, the decoder/client sequence is simpler, but the
cost of this simplicity is extra complexity in the client which is responsible
for maintaining a consistent decoding state.

Stateless decoders make use of the :ref:`media-request-api`. A stateless
decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its
``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or :c:func:`VIDIOC_CREATE_BUFS`
are invoked.

Depending on the encoded formats supported by the decoder, a single decoded
frame may be the result of several decode requests (for instance, H.264 streams
with multiple slices per frame). Decoders that support such formats must also
expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their
``OUTPUT`` queue.

Coded/raw format과 profile 조회

지원 coded format은 `OUTPUT` queue에서 `VIDIOC_ENUM_FMT`로 열거합니다. Driver는 현재 `CAPTURE` format과 관계없이 지원하는 OUTPUT format 전체를 반환해야 하고, H.264 profile 같은 codec-specific capability control은 hardware가 실제 지원하는 값으로 제한해야 합니다.

지원 raw format은 `CAPTURE` queue에서 `VIDIOC_ENUM_FMT`로 열거합니다. 이 결과는 현재 활성 `OUTPUT` format에 맞는 항목만 포함하며 codec-dependent control 값에 따라 달라질 수 있습니다.

Client는 CAPTURE format을 열거하기 전에 필요한 codec control을 설정해야 합니다. 설정하지 않으면 default control 값으로 계산한 raw format 목록이 반환되어 실제 media를 decode하는 데 쓸 수 없을 수 있습니다.

특정 format의 지원 resolution은 원하는 `pixel_format`을 `struct v4l2_frmsizeenum`에 넣고 `VIDIOC_ENUM_FRAMESIZES`로 조회합니다. 현재 OUTPUT format의 profile과 level은 각각의 control을 `VIDIOC_QUERYCTRL`로 조회할 수 있습니다.

Querying capabilities
=====================

1. To enumerate the set of coded formats supported by the decoder, the client
   calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue.

   * The driver must always return the full set of supported ``OUTPUT`` formats,
     irrespective of the format currently set on the ``CAPTURE`` queue.

   * Simultaneously, the driver must restrain the set of values returned by
     codec-specific capability controls (such as H.264 profiles) to the set
     actually supported by the hardware.

2. To enumerate the set of supported raw formats, the client calls
   :c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue.

   * The driver must return only the formats supported for the format currently
     active on the ``OUTPUT`` queue.

   * Depending on the currently set ``OUTPUT`` format, the set of supported raw
     formats may depend on the value of some codec-dependent controls.
     The client is responsible for making sure that these controls are set
     before querying the ``CAPTURE`` queue. Failure to do so will result in the
     default values for these controls being used, and a returned set of formats
     that may not be usable for the media the client is trying to decode.

3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
   resolutions for a given format, passing desired pixel format in
   :c:type:`v4l2_frmsizeenum`'s ``pixel_format``.

4. Supported profiles and levels for the current ``OUTPUT`` format, if
   applicable, may be queried using their respective controls via
   :c:func:`VIDIOC_QUERYCTRL`.

초기화 1~3: OUTPUT과 기본 CAPTURE

첫 단계는 `VIDIOC_S_FMT`로 `OUTPUT` queue의 coded format을 설정하는 것입니다. 적절한 OUTPUT `type`, coded `pixelformat`, stream에서 parsing한 coded `width`와 `height`가 필수이며 나머지 필드는 표준 의미를 따릅니다.

OUTPUT format을 바꾸면 현재 CAPTURE format도 바뀔 수 있습니다. Driver는 새 OUTPUT format에서 decoded resolution과 colorimetry 등을 도출해 CAPTURE format을 갱신하므로, client가 특정 raw format을 원하면 그 뒤에 조정해야 합니다.

두 번째 단계에서는 `VIDIOC_S_EXT_CTRLS`로 CAPTURE format 열거에 필요한 parsed header 등 OUTPUT codec의 모든 control을 설정합니다.

세 번째 단계는 CAPTURE queue에 `VIDIOC_G_FMT`를 호출해 decoded destination buffer format을 얻는 것입니다. 필수 입력은 CAPTURE `type`이고, driver는 decoded frame의 `width`, `height`, `pixelformat`, multiplanar이면 `num_planes`, 표준 의미의 `sizeimage`와 `bytesperline`을 반환합니다.

반환 pixelformat은 현재 OUTPUT에서 hardware가 지원하는 어느 raw format이든 될 수 있습니다. Driver는 추가 RGB conversion이 필요한 상황에서 YUV를 고르는 것처럼 현재 구성에 최적인 format을 선택하는 것이 권장됩니다.

Initialization
==============

1. Set the coded format on the ``OUTPUT`` queue via :c:func:`VIDIOC_S_FMT`.

   * **Required fields:**

     ``type``
         a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.

     ``pixelformat``
         a coded pixel format.

     ``width``, ``height``
         coded width and height parsed from the stream.

     other fields
         follow standard semantics.

   .. note::

      Changing the ``OUTPUT`` format may change the currently set ``CAPTURE``
      format. The driver will derive a new ``CAPTURE`` format from the
      ``OUTPUT`` format being set, including resolution, colorimetry
      parameters, etc. If the client needs a specific ``CAPTURE`` format,
      it must adjust it afterwards.

2. Call :c:func:`VIDIOC_S_EXT_CTRLS` to set all the controls (parsed headers,
   etc.) required by the ``OUTPUT`` format to enumerate the ``CAPTURE`` formats.

3. Call :c:func:`VIDIOC_G_FMT` for ``CAPTURE`` queue to get the format for the
   destination buffers parsed/decoded from the bytestream.

   * **Required fields:**

     ``type``
         a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.

   * **Returned fields:**

     ``width``, ``height``
         frame buffer resolution for the decoded frames.

     ``pixelformat``
         pixel format for decoded frames.

     ``num_planes`` (for _MPLANE ``type`` only)
         number of planes for pixelformat.

     ``sizeimage``, ``bytesperline``
         as per standard semantics; matching frame buffer format.

   .. note::

      The value of ``pixelformat`` may be any pixel format supported for the
      ``OUTPUT`` format, based on the hardware capabilities. It is suggested
      that the driver chooses the preferred/optimal format for the current
      configuration. For example, a YUV format may be preferred over an RGB
      format, if an additional conversion step would be required for RGB.

초기화 4~5: CAPTURE format 선택

선택 단계로 CAPTURE queue에서 `VIDIOC_ENUM_FMT`를 호출하면 현재 OUTPUT format과 control에 지원되는 대안 raw format을 찾을 수 있습니다. Decoder 전체가 지원하는 모든 format이 아니라 현재 구성에서 가능한 subset만 반환합니다.

예를 들어 hardware가 1920x1088 이하에서 YUV와 RGB를 지원하고 그보다 큰 resolution에서는 YUV만 지원한다면, 작은 OUTPUT 크기에서는 두 종류를 열거하지만 큰 크기에서는 RGB를 반환하지 않습니다.

Client는 CAPTURE `type`, 원하는 raw `pixelformat`, decoded frame buffer `width`와 `height`를 `VIDIOC_S_FMT`로 설정할 수 있습니다. 보통 G_FMT 크기를 유지하지만 hardware composition이나 scaling을 지원하면 다른 크기를 고를 수 있습니다.

CAPTURE S_FMT 뒤에는 buffer 크기와 layout이 달라졌을 수 있으므로 반드시 초기화 3단계의 CAPTURE `VIDIOC_G_FMT`를 다시 호출해 최신 정보를 얻어야 합니다.

4. *[optional]* Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on
   the ``CAPTURE`` queue. The client may use this ioctl to discover which
   alternative raw formats are supported for the current ``OUTPUT`` format and
   select one of them via :c:func:`VIDIOC_S_FMT`.

   .. note::

      The driver will return only formats supported for the currently selected
      ``OUTPUT`` format and currently set controls, even if more formats may be
      supported by the decoder in general.

      For example, a decoder may support YUV and RGB formats for
      resolutions 1920x1088 and lower, but only YUV for higher resolutions (due
      to hardware limitations). After setting a resolution of 1920x1088 or lower
      as the ``OUTPUT`` format, :c:func:`VIDIOC_ENUM_FMT` may return a set of
      YUV and RGB pixel formats, but after setting a resolution higher than
      1920x1088, the driver will not return RGB pixel formats, since they are
      unsupported for this resolution.

5. *[optional]* Choose a different ``CAPTURE`` format than suggested via
   :c:func:`VIDIOC_S_FMT` on ``CAPTURE`` queue. It is possible for the client to
   choose a different format than selected/suggested by the driver in
   :c:func:`VIDIOC_G_FMT`.

    * **Required fields:**

      ``type``
          a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.

      ``pixelformat``
          a raw pixel format.

      ``width``, ``height``
         frame buffer resolution of the decoded stream; typically unchanged from
         what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different
         if the hardware supports composition and/or scaling.

   After performing this step, the client must perform step 3 again in order
   to obtain up-to-date information about the buffers size and layout.

초기화 6~9: Buffer, request와 streaming

여섯 번째 단계는 OUTPUT queue에 `VIDIOC_REQBUFS`를 호출해 source bytestream buffer를 할당하는 것입니다. `count`는 0보다 큰 요청 수, `type`은 OUTPUT 종류, `memory`는 표준 의미를 사용합니다. Driver는 format에 필요한 최소 수와 요청 수를 고려해 실제 `count`를 같거나 더 크게 조정할 수 있으므로 반환값을 확인합니다.

일곱 번째 단계는 CAPTURE queue의 decoded destination buffer 할당입니다. Client는 reference frame 등을 고려해 stream decode에 필요한 최소 수를 계산하고 그 이상을 요청해야 합니다. `V4L2_MEMORY_USERPTR`는 CAPTURE buffer에서 지원되지 않습니다.

Driver는 현재 format과 stream configuration, 요청 수를 바탕으로 필요한 최소 CAPTURE count로 조정하고 실제 할당 수를 반환합니다. Client는 OUTPUT과 CAPTURE 모두 요청값이 그대로 적용된다고 가정하면 안 됩니다.

여덟 번째 단계에서 media device의 `MEDIA_IOC_REQUEST_ALLOC`으로 request를 할당합니다. 일반적으로 OUTPUT buffer마다 request 하나를 준비합니다. 마지막 아홉 번째 단계에서 `VIDIOC_STREAMON`으로 OUTPUT과 CAPTURE queue를 모두 시작합니다.

6. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on
   ``OUTPUT`` queue.

    * **Required fields:**

      ``count``
          requested number of buffers to allocate; greater than zero.

      ``type``
          a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.

      ``memory``
          follows standard semantics.

    * **Returned fields:**

      ``count``
          actual number of buffers allocated.

    * If required, the driver will adjust ``count`` to be equal or bigger to the
      minimum of required number of ``OUTPUT`` buffers for the given format and
      requested count. The client must check this value after the ioctl returns
      to get the actual number of buffers allocated.

7. Allocate destination (raw format) buffers via :c:func:`VIDIOC_REQBUFS` on the
   ``CAPTURE`` queue.

    * **Required fields:**

      ``count``
          requested number of buffers to allocate; greater than zero. The client
          is responsible for deducing the minimum number of buffers required
          for the stream to be properly decoded (taking e.g. reference frames
          into account) and pass an equal or bigger number.

      ``type``
          a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.

      ``memory``
          follows standard semantics. ``V4L2_MEMORY_USERPTR`` is not supported
          for ``CAPTURE`` buffers.

    * **Returned fields:**

      ``count``
          adjusted to allocated number of buffers, in case the codec requires
          more buffers than requested.

    * The driver must adjust count to the minimum of required number of
      ``CAPTURE`` buffers for the current format, stream configuration and
      requested count. The client must check this value after the ioctl
      returns to get the number of buffers allocated.

8. Allocate requests (likely one per ``OUTPUT`` buffer) via
    :c:func:`MEDIA_IOC_REQUEST_ALLOC` on the media device.

9. Start streaming on both ``OUTPUT`` and ``CAPTURE`` queues via
    :c:func:`VIDIOC_STREAMON`.

Frame request와 HOLD_CAPTURE_BUF

각 frame마다 client는 적어도 하나의 request를 제출합니다. Request에는 현재 codec configuration이 요구하는 양의 encoded data를 담은 OUTPUT buffer와 그 data를 decode하는 데 필요한 codec-specific metadata control 전체가 연결됩니다.

대개 OUTPUT data 한 단위는 frame 하나지만 codec에 따라 다른 크기를 허용하거나 요구합니다. 정확한 buffer 내용과 request control 집합은 활성 coded pixelformat 및 codec-specific extended control 문서를 따릅니다.

현재 request 뒤에 같은 decoded frame을 완성할 추가 request가 올 수 있다면 OUTPUT buffer에 `V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`를 설정해야 합니다. 그러면 부분 decode된 CAPTURE buffer를 dequeue하지 않고 다음 request에서 재사용합니다. 다음 OUTPUT timestamp가 바뀌면 held buffer를 dequeue 가능 상태로 만들고 새 CAPTURE buffer를 사용합니다.

Decoding
========

For each frame, the client is responsible for submitting at least one request to
which the following is attached:

* The amount of encoded data expected by the codec for its current
  configuration, as a buffer submitted to the ``OUTPUT`` queue. Typically, this
  corresponds to one frame worth of encoded data, but some formats may allow (or
  require) different amounts per unit.
* All the metadata needed to decode the submitted encoded data, in the form of
  controls relevant to the format being decoded.

The amount of data and contents of the source ``OUTPUT`` buffer, as well as the
controls that must be set on the request, depend on the active coded pixel
format and might be affected by codec-specific extended controls, as stated in
documentation of each format.

If there is a possibility that the decoded frame will require one or more
decode requests after the current one in order to be produced, then the client
must set the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag on the ``OUTPUT``
buffer. This will result in the (potentially partially) decoded ``CAPTURE``
buffer not being made available for dequeueing, and reused for the next decode
request if the timestamp of the next ``OUTPUT`` buffer has not changed.

OUTPUT·control 연결과 request 제출

첫 단계는 `VIDIOC_QBUF`로 request의 encoded data 단위를 담은 OUTPUT buffer 하나를 queue하는 것입니다. `index`, OUTPUT `type`, 실제 encoded byte 수인 `bytesused`가 필요합니다.

`flags`에는 `V4L2_BUF_FLAG_REQUEST_FD`를 반드시 설정하고 마지막 decode request인지 확실하지 않으면 `V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`도 설정합니다. `request_fd`는 해당 decoding request descriptor입니다.

`timestamp`는 frame마다 고유하며 decoded CAPTURE buffer로 전파되고 다른 frame이 reference할 때도 사용됩니다. 한 frame에 여러 request가 있으면 모든 OUTPUT buffer timestamp가 같아야 합니다. Timestamp가 바뀌면 held CAPTURE buffer를 dequeue하고 새 buffer에서 현재 request를 처리합니다.

두 번째 단계는 `VIDIOC_S_EXT_CTRLS`입니다. `which`를 `V4L2_CTRL_WHICH_REQUEST_VAL`, `request_fd`를 같은 request descriptor로 설정하고 frame decode에 필요한 codec control 전체를 연결합니다. 여러 번 호출하거나 기존 값을 덮어쓸 수 있으며 request 제출 순간의 최종 control state를 사용합니다.

OUTPUT buffer queue와 control 설정의 순서는 서로 바꿀 수 있습니다. 세 번째 단계에서 request FD에 `MEDIA_REQUEST_IOC_QUEUE`를 호출해 제출합니다.

OUTPUT buffer가 없거나 필수 control이 빠지면 `-ENOENT`, OUTPUT buffer가 둘 이상이면 `-EINVAL`입니다. Non-zero 반환은 해당 request에서 CAPTURE buffer가 생성되지 않는다는 뜻입니다.

CAPTURE buffer는 request에 포함하지 않고 독립적으로 queue합니다. 반환 순서는 display order가 아니라 OUTPUT에 coded frame을 제출한 decode order입니다.

A typical frame would thus be decoded using the following sequence:

1. Queue an ``OUTPUT`` buffer containing one unit of encoded bytestream data for
   the decoding request, using :c:func:`VIDIOC_QBUF`.

    * **Required fields:**

      ``index``
          index of the buffer being queued.

      ``type``
          type of the buffer.

      ``bytesused``
          number of bytes taken by the encoded data frame in the buffer.

      ``flags``
          the ``V4L2_BUF_FLAG_REQUEST_FD`` flag must be set. Additionally, if
          we are not sure that the current decode request is the last one needed
          to produce a fully decoded frame, then
          ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set.

      ``request_fd``
          must be set to the file descriptor of the decoding request.

      ``timestamp``
          must be set to a unique value per frame. This value will be propagated
          into the decoded frame's buffer and can also be used to use this frame
          as the reference of another. If using multiple decode requests per
          frame, then the timestamps of all the ``OUTPUT`` buffers for a given
          frame must be identical. If the timestamp changes, then the currently
          held ``CAPTURE`` buffer will be made available for dequeuing and the
          current request will work on a new ``CAPTURE`` buffer.

2. Set the codec-specific controls for the decoding request, using
   :c:func:`VIDIOC_S_EXT_CTRLS`.

    * **Required fields:**

      ``which``
          must be ``V4L2_CTRL_WHICH_REQUEST_VAL``.

      ``request_fd``
          must be set to the file descriptor of the decoding request.

      other fields
          other fields are set as usual when setting controls. The ``controls``
          array must contain all the codec-specific controls required to decode
          a frame.

   .. note::

      It is possible to specify the controls in different invocations of
      :c:func:`VIDIOC_S_EXT_CTRLS`, or to overwrite a previously set control, as
      long as ``request_fd`` and ``which`` are properly set. The controls state
      at the moment of request submission is the one that will be considered.

   .. note::

      The order in which steps 1 and 2 take place is interchangeable.

3. Submit the request by invoking :c:func:`MEDIA_REQUEST_IOC_QUEUE` on the
   request FD.

    If the request is submitted without an ``OUTPUT`` buffer, or if some of the
    required controls are missing from the request, then
    :c:func:`MEDIA_REQUEST_IOC_QUEUE` will return ``-ENOENT``. If more than one
    ``OUTPUT`` buffer is queued, then it will return ``-EINVAL``.
    :c:func:`MEDIA_REQUEST_IOC_QUEUE` returning non-zero means that no
    ``CAPTURE`` buffer will be produced for this request.

``CAPTURE`` buffers must not be part of the request, and are queued
independently. They are returned in decode order (i.e. the same order as coded
frames were submitted to the ``OUTPUT`` queue).

오류 전파와 reference buffer 수명

Runtime decode 오류는 dequeue된 CAPTURE buffer의 `V4L2_BUF_FLAG_ERROR`로 알립니다. 오류가 있는 decoded reference frame을 뒤 frame이 참조하면 그 frame에도 ERROR flag를 설정하지만 decoder는 손상 가능성이 있는 frame 생성을 계속 시도합니다.

Stateless decoder는 buffer 관리를 수행하지 않습니다. Dequeue한 CAPTURE buffer는 다시 queue하기 전까지 client가 compositing, display 또는 다른 frame의 reference로 사용할 수 있음을 보장할 뿐입니다.

Frame을 reference로 지정할 때 timestamp를 nanosecond로 변환해 codec-dependent control의 관련 멤버에 저장합니다. 반드시 `v4l2_timeval_to_ns`를 사용합니다. Frame의 encoded data unit이 모두 OUTPUT에 성공적으로 제출되면 그 timestamp로 즉시 reference할 수 있습니다.

Reference frame을 담은 decoded buffer는 이를 참조하는 frame이 모두 decode될 때까지 새 decode target으로 재사용하면 안 됩니다. 가장 안전한 방법은 모든 referencing frame을 dequeue할 때까지 reference buffer를 queue하지 않는 것입니다.

Driver가 CAPTURE buffer를 queue 순서대로 처리한다고 보장한다면 더 일찍 재queue할 수 있습니다. Reference의 영향을 받는 모든 frame request가 queue되었고, 그 referencing frame 전체를 수용할 만큼 CAPTURE buffer가 뒤에 queue된 두 조건을 모두 충족해야 합니다.

Decode request를 queue할 때 driver는 reference frame 관련 resource의 reference count를 증가시킵니다. 따라서 client는 이후 필요하지 않은 reference buffer의 DMABUF file descriptor를 닫아도 request가 resource 수명을 유지합니다.

Runtime decoding errors are signaled by the dequeued ``CAPTURE`` buffers
carrying the ``V4L2_BUF_FLAG_ERROR`` flag. If a decoded reference frame has an
error, then all following decoded frames that refer to it also have the
``V4L2_BUF_FLAG_ERROR`` flag set, although the decoder will still try to
produce (likely corrupted) frames.

Buffer management while decoding
================================
Contrary to stateful decoders, a stateless decoder does not perform any kind of
buffer management: it only guarantees that dequeued ``CAPTURE`` buffers can be
used by the client for as long as they are not queued again. "Used" here
encompasses using the buffer for compositing or display.

A dequeued capture buffer can also be used as the reference frame of another
buffer.

A frame is specified as reference by converting its timestamp into nanoseconds,
and storing it into the relevant member of a codec-dependent control structure.
The :c:func:`v4l2_timeval_to_ns` function must be used to perform that
conversion. The timestamp of a frame can be used to reference it as soon as all
its units of encoded data are successfully submitted to the ``OUTPUT`` queue.

A decoded buffer containing a reference frame must not be reused as a decoding
target until all the frames referencing it have been decoded. The safest way to
achieve this is to refrain from queueing a reference buffer until all the
decoded frames referencing it have been dequeued. However, if the driver can
guarantee that buffers queued to the ``CAPTURE`` queue are processed in queued
order, then user-space can take advantage of this guarantee and queue a
reference buffer when the following conditions are met:

1. All the requests for frames affected by the reference frame have been
   queued, and

2. A sufficient number of ``CAPTURE`` buffers to cover all the decoded
   referencing frames have been queued.

When queuing a decoding request, the driver will increase the reference count of
all the resources associated with reference frames. This means that the client
can e.g. close the DMABUF file descriptors of reference frame buffers if it
won't need them afterwards.

Seeking과 pausing

Seek하려면 새 stream 위치의 input buffer를 사용해 request를 제출하면 됩니다. 다만 새 위치에서 resolution이 달라졌다면 dynamic resolution change 절차를 수행해야 합니다.

Codec에 따라 H.264 SPS/PPS 같은 picture parameter도 달라질 수 있으므로 client가 새 위치에 유효한 state를 decoder에 제공해야 합니다. Seek 전 위치에서 늦게 반환되는 CAPTURE buffer는 무시할 수 있습니다.

Pause는 OUTPUT queue에 buffer를 더 queue하지 않는 것으로 충분합니다. Source bytestream data가 없으면 처리할 작업이 없어 codec이 idle 상태로 유지됩니다.

Seeking
=======
In order to seek, the client just needs to submit requests using input buffers
corresponding to the new stream position. It must however be aware that
resolution may have changed and follow the dynamic resolution change sequence in
that case. Also depending on the codec used, picture parameters (e.g. SPS/PPS
for H.264) may have changed and the client is responsible for making sure that a
valid state is sent to the decoder.

The client is then free to ignore any returned ``CAPTURE`` buffer that comes
from the pre-seek position.

Pausing
=======

In order to pause, the client can just cease queuing buffers onto the ``OUTPUT``
queue. Without source bytestream data, there is no data to process and the codec
will remain idle.

Dynamic resolution change

Client가 stream에서 resolution change를 발견하면 새 크기로 초기화 절차를 다시 수행합니다. 마지막 request가 `V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`로 CAPTURE buffer를 보유 중이면 `V4L2_DEC_CMD_FLUSH`를 보내 held buffer를 dequeue하게 합니다.

제출한 모든 request가 완료될 때까지 기다리고 대응 OUTPUT buffer를 dequeue합니다. 그 다음 OUTPUT과 CAPTURE 모두 `VIDIOC_STREAMOFF`를 호출합니다.

CAPTURE queue에 `count = 0`인 `VIDIOC_REQBUFS`를 호출해 모든 destination buffer를 해제합니다. OUTPUT buffer 할당 단계만 제외하고 새 resolution으로 초기화 절차를 반복합니다. Resolution 제약 때문에 CAPTURE pixelformat도 바꿔야 할 수 있습니다.

Dynamic resolution change
=========================

If the client detects a resolution change in the stream, it will need to perform
the initialization sequence again with the new resolution:

1. If the last submitted request resulted in a ``CAPTURE`` buffer being
   held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the
   last frame is not available on the ``CAPTURE`` queue. In this case, a
   ``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver
   dequeue the held ``CAPTURE`` buffer.

2. Wait until all submitted requests have completed and dequeue the
   corresponding output buffers.

3. Call :c:func:`VIDIOC_STREAMOFF` on both the ``OUTPUT`` and ``CAPTURE``
   queues.

4. Free all ``CAPTURE`` buffers by calling :c:func:`VIDIOC_REQBUFS` on the
   ``CAPTURE`` queue with a buffer count of zero.

5. Perform the initialization sequence again (minus the allocation of
   ``OUTPUT`` buffers), with the new resolution set on the ``OUTPUT`` queue.
   Note that due to resolution constraints, a different format may need to be
   picked on the ``CAPTURE`` queue.

Drain과 held CAPTURE 회수

마지막 request가 `V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`로 CAPTURE buffer를 보유해 마지막 frame을 아직 dequeue할 수 없다면 `V4L2_DEC_CMD_FLUSH`를 보냅니다. Driver는 held CAPTURE buffer를 dequeue 가능 상태로 만듭니다.

그 뒤 stateless decoder의 drain은 제출한 모든 request가 완료될 때까지 기다리는 것으로 끝납니다. Stateful decoder처럼 별도의 내부 decode state를 비우는 복잡한 절차는 없습니다.

Drain
=====

If the last submitted request resulted in a ``CAPTURE`` buffer being
held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the
last frame is not available on the ``CAPTURE`` queue. In this case, a
``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver
dequeue the held ``CAPTURE`` buffer.

After that, in order to drain the stream on a stateless decoder, the client
just needs to wait until all the submitted requests are completed.