Media Request API

설계 목적과 per-frame configuration

Request API는 stateless codec, 복잡한 camera pipeline, Android Codec v2 같은 현대 장치와 API의 요구 사항을 V4L2가 처리하도록 설계되었습니다.

같은 pipeline에 속한 여러 장치가 frame마다 재구성되고 긴밀히 협력해야 하며, stateless codec은 특정 frame에 control을 적용하는 per-frame control이 필요합니다. 최초 사용 사례는 V4L2지만 Media Controller를 사용하는 다른 subsystem으로도 확장할 수 있습니다.

Request API가 없으면 매 frame마다 pipeline 활동을 모두 비우고 다음 frame용으로 재설정한 뒤 buffer를 queue하고 전부 dequeue 가능해질 때까지 기다려야 합니다. 실제로 한 번에 buffer 하나만 queue하는 셈이라 buffer queue의 이점을 잃습니다.

Request는 특정 pipeline configuration, 즉 Media Controller topology와 각 media entity의 설정을 특정 buffer와 결합합니다. Userspace는 서로 다른 설정을 가진 여러 task를 미리 schedule하고, 필요한 시점에 정확한 설정이 적용되도록 할 수 있습니다. 완료 시점의 configuration value도 읽을 수 있습니다.

.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
.. c:namespace:: MC

.. _media-request-api:

Request API
===========

The Request API has been designed to allow V4L2 to deal with requirements of
modern devices (stateless codecs, complex camera pipelines, ...) and APIs
(Android Codec v2). One such requirement is the ability for devices belonging to
the same pipeline to reconfigure and collaborate closely on a per-frame basis.
Another is support of stateless codecs, which require controls to be applied
to specific frames (aka 'per-frame controls') in order to be used efficiently.

While the initial use-case was V4L2, it can be extended to other subsystems
as well, as long as they use the media controller.

Supporting these features without the Request API is not always possible and if
it is, it is terribly inefficient: user-space would have to flush all activity
on the media pipeline, reconfigure it for the next frame, queue the buffers to
be processed with that configuration, and wait until they are all available for
dequeuing before considering the next frame. This defeats the purpose of having
buffer queues since in practice only one buffer would be queued at a time.

The Request API allows a specific configuration of the pipeline (media
controller topology + configuration for each media entity) to be associated with
specific buffers. This allows user-space to schedule several tasks ("requests")
with different configurations in advance, knowing that the configuration will be
applied when needed to get the expected result. Configuration values at the time
of request completion are also available for reading.

일반 사용, 할당과 준비

Request API는 Media Controller API를 확장하고 subsystem별 API와 협력합니다. Media Controller device node에서 request를 할당한 뒤 opaque한 request fd로 수명 주기를 관리합니다.

Request에 저장된 configuration data, buffer handle과 처리 결과는 request 지원이 추가된 subsystem별 API로 접근합니다. V4L2에서는 명시적인 `request_fd` parameter를 받는 ioctl이 그 역할을 합니다.

Userspace는 media device node에 `MEDIA_IOC_REQUEST_ALLOC`을 호출해 request fd를 받습니다. Pipeline을 채워 운용하려면 보통 여러 request를 할당합니다.

지원되는 표준 V4L2 ioctl에 request fd를 전달하면 명령을 즉시 적용하지 않고 해당 request의 일부로 저장합니다. Request에 queue한 buffer도 request 자체가 queue될 때까지 일반 buffer queue에 들어가지 않습니다.

General Usage
-------------

The Request API extends the Media Controller API and cooperates with
subsystem-specific APIs to support request usage. At the Media Controller
level, requests are allocated from the supporting Media Controller device
node. Their life cycle is then managed through the request file descriptors in
an opaque way. Configuration data, buffer handles and processing results
stored in requests are accessed through subsystem-specific APIs extended for
request support, such as V4L2 APIs that take an explicit ``request_fd``
parameter.

Request Allocation
------------------

User-space allocates requests using :ref:`MEDIA_IOC_REQUEST_ALLOC`
for the media device node. This returns a file descriptor representing the
request. Typically, several such requests will be allocated.

Request Preparation
-------------------

Standard V4L2 ioctls can then receive a request file descriptor to express the
fact that the ioctl is part of said request, and is not to be applied
immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC` for a list of ioctls that
support this. Configurations set with a ``request_fd`` parameter are stored
instead of being immediately applied, and buffers queued to a request do not
enter the regular buffer queue until the request itself is queued.

제출 제약과 동기화

Configuration과 buffer를 모두 지정하면 request fd에 `MEDIA_REQUEST_IOC_QUEUE`를 호출합니다. Request에는 buffer가 하나 이상 있어야 하며 없으면 `ENOENT`입니다. Queue된 request는 더 이상 수정할 수 없습니다.

Memory-to-memory device에서는 request를 OUTPUT buffer에만 사용할 수 있고 CAPTURE buffer에는 사용할 수 없습니다. Capture buffer를 request에 추가하면 `EBADR`입니다.

여러 entity의 configuration을 포함한 request에서는 driver들이 buffer 처리 전에 요청된 pipeline topology를 적용하도록 동기화할 수 있습니다. Hardware 한계 때문에 완전한 atomicity가 불가능할 수 있어 Media Controller driver는 best effort로 구현합니다.

Request queue와 direct buffer queue는 섞을 수 없습니다. 먼저 사용한 방식은 `VIDIOC_STREAMOFF`를 호출하거나 device를 닫을 때까지 고정되며, 반대 방식을 시도하면 `EBUSY`입니다.

Request 없이 설정한 control은 request 사용 여부와 관계없이 즉시 적용됩니다. 같은 control을 request를 통해서도 설정하고 직접 설정하기도 하면 동작이 정의되지 않습니다.

Request Submission
------------------

Once the configuration and buffers of the request are specified, it can be
queued by calling :ref:`MEDIA_REQUEST_IOC_QUEUE` on the request file descriptor.
A request must contain at least one buffer, otherwise ``ENOENT`` is returned.
A queued request cannot be modified anymore.

.. caution::
   For :ref:`memory-to-memory devices <mem2mem>` you can use requests only for
   output buffers, not for capture buffers. Attempting to add a capture buffer
   to a request will result in an ``EBADR`` error.

If the request contains configurations for multiple entities, individual drivers
may synchronize so the requested pipeline's topology is applied before the
buffers are processed. Media controller drivers do a best effort implementation
since perfect atomicity may not be possible due to hardware limitations.

.. caution::

   It is not allowed to mix queuing requests with directly queuing buffers:
   whichever method is used first locks this in place until
   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` is called or the device is
   :ref:`closed <func-close>`. Attempts to directly queue a buffer when earlier
   a buffer was queued via a request or vice versa will result in an ``EBUSY``
   error.

Controls can still be set without a request and are applied immediately,
regardless of whether a request is in use or not.

.. caution::

   Setting the same control through a request and also directly can lead to
   undefined behavior!

완료 판정, 결과 조회와 수명

Userspace는 request fd를 `poll()`해 완료를 기다릴 수 있습니다. 연관된 모든 buffer가 dequeue 가능하고 모든 control이 완료 시점 값으로 갱신되면 request가 완료된 것으로 간주합니다.

Request 전체가 완료되기 전이라도 중간에 사용 가능해진 buffer는 독립적으로 dequeue할 수 있습니다. Buffer 반환과 request completion은 같은 시점일 필요가 없습니다.

완료된 request에는 실행 후 device state가 들어 있습니다. `request_fd`와 함께 `VIDIOC_G_EXT_CTRLS`를 호출해 그 상태를 읽습니다. 아직 실행 중인 request는 driver가 control value를 바꿀 수 있으므로 조회하면 `EBUSY`입니다.

완료된 request는 버리거나 재사용할 수 있습니다. Request fd를 `close()`하면 그 fd는 즉시 사용할 수 없지만 kernel이 request 사용을 마칠 때까지 object는 해제되지 않습니다. Queue 직후 닫았다면 driver 완료 뒤 해제됩니다.

`MEDIA_REQUEST_IOC_REINIT`은 상태를 전부 지워 방금 할당한 것과 같은 request로 만듭니다. 이전 상태는 하나도 유지되지 않습니다.

User-space can :c:func:`poll()` a request file descriptor in
order to wait until the request completes. A request is considered complete
once all its associated buffers are available for dequeuing and all the
associated controls have been updated with the values at the time of completion.
Note that user-space does not need to wait for the request to complete to
dequeue its buffers: buffers that are available halfway through a request can
be dequeued independently of the request's state.

A completed request contains the state of the device after the request was
executed. User-space can query that state by calling
:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` with the request file
descriptor. Calling :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` for a
request that has been queued but not yet completed will return ``EBUSY``
since the control values might be changed at any time by the driver while the
request is in flight.

.. _media-request-life-time:

Recycling and Destruction
-------------------------

Finally, a completed request can either be discarded or be reused. Calling
:c:func:`close()` on a request file descriptor will make
that file descriptor unusable and the request will be freed once it is no
longer in use by the kernel. That is, if the request is queued and then the
file descriptor is closed, then it won't be freed until the driver completed
the request.

The :ref:`MEDIA_REQUEST_IOC_REINIT` will clear a request's state and make it
available again. No state is retained by this operation: the request is as
if it had just been allocated.

Codec device 예제

Memory-to-memory codec에서는 OUTPUT buffer에 적용할 특정 control을 request로 결합해 여러 OUTPUT buffer를 미리 queue할 수 있습니다. 완료 시점의 volatile control도 request에서 읽을 수 있습니다.

먼저 request를 할당하고 `V4L2_CTRL_WHICH_REQUEST_VAL`과 `request_fd`로 control을 저장한 뒤, OUTPUT buffer에 `V4L2_BUF_FLAG_REQUEST_FD`와 같은 fd를 설정해 queue합니다.

struct v4l2_buffer buf;
struct v4l2_ext_controls ctrls;
int req_fd;
...
if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
    return errno;
...
ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
ctrls.request_fd = req_fd;
if (ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
    return errno;
...
buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
buf.request_fd = req_fd;
if (ioctl(codec_fd, VIDIOC_QBUF, &buf))
    return errno;

M2M에서는 CAPTURE buffer에 report할 per-frame setting이 없으므로 Request API를 CAPTURE buffer에 사용할 수 없습니다.

준비가 끝나면 request fd에 `MEDIA_REQUEST_IOC_QUEUE`를 호출합니다. 이후 poll로 전체 완료를 기다리거나 일반 `VIDIOC_DQBUF`로 준비된 CAPTURE buffer부터 받을 수 있습니다.

if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
    return errno;

struct v4l2_buffer buf;
memset(&buf, 0, sizeof(buf));
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
if (ioctl(codec_fd, VIDIOC_DQBUF, &buf))
    return errno;

예제는 OUTPUT buffer 하나마다 CAPTURE buffer 하나가 나온다고 단순화하지만 실제 codec은 이 비율을 보장하지 않습니다.

Request fd에서 `POLLPRI`를 기다린 뒤 `VIDIOC_G_EXT_CTRLS`에 `V4L2_CTRL_WHICH_REQUEST_VAL`과 request fd를 전달하면 완료 시점 control을 읽습니다. Capture buffer 생성 직후 volatile control을 확인할 때 유용합니다.

struct pollfd pfd = { .events = POLLPRI, .fd = req_fd };
poll(&pfd, 1, -1);
...
ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
ctrls.request_fd = req_fd;
if (ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls))
    return errno;

마지막에는 `MEDIA_REQUEST_IOC_REINIT`으로 재사용하거나 `close(req_fd)`로 폐기합니다.

if (ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT))
    return errno;

close(req_fd);

Example for a Codec Device
--------------------------

For use-cases such as :ref:`codecs <mem2mem>`, the request API can be used
to associate specific controls to
be applied by the driver for the OUTPUT buffer, allowing user-space
to queue many such buffers in advance. It can also take advantage of requests'
ability to capture the state of controls when the request completes to read back
information that may be subject to change.

Put into code, after obtaining a request, user-space can assign controls and one
OUTPUT buffer to it:

.. code-block:: c

	struct v4l2_buffer buf;
	struct v4l2_ext_controls ctrls;
	int req_fd;
	...
	if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
		return errno;
	...
	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
	ctrls.request_fd = req_fd;
	if (ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
		return errno;
	...
	buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
	buf.request_fd = req_fd;
	if (ioctl(codec_fd, VIDIOC_QBUF, &buf))
		return errno;

Note that it is not allowed to use the Request API for CAPTURE buffers
since there are no per-frame settings to report there.

Once the request is fully prepared, it can be queued to the driver:

.. code-block:: c

	if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
		return errno;

User-space can then either wait for the request to complete by calling poll() on
its file descriptor, or start dequeuing CAPTURE buffers. Most likely, it will
want to get CAPTURE buffers as soon as possible and this can be done using a
regular :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`:

.. code-block:: c

	struct v4l2_buffer buf;

	memset(&buf, 0, sizeof(buf));
	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
	if (ioctl(codec_fd, VIDIOC_DQBUF, &buf))
		return errno;

Note that this example assumes for simplicity that for every OUTPUT buffer
there will be one CAPTURE buffer, but this does not have to be the case.

We can then, after ensuring that the request is completed via polling the
request file descriptor, query control values at the time of its completion via
a call to :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
This is particularly useful for volatile controls for which we want to
query values as soon as the capture buffer is produced.

.. code-block:: c

	struct pollfd pfd = { .events = POLLPRI, .fd = req_fd };
	poll(&pfd, 1, -1);
	...
	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
	ctrls.request_fd = req_fd;
	if (ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls))
		return errno;

Once we don't need the request anymore, we can either recycle it for reuse with
:ref:`MEDIA_REQUEST_IOC_REINIT`...

.. code-block:: c

	if (ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT))
		return errno;

... or close its file descriptor to completely dispose of it.

.. code-block:: c

	close(req_fd);

Simple capture device 예제

Simple capture device에서는 특정 CAPTURE buffer에 적용할 control을 request로 지정할 수 있습니다. 이는 M2M codec의 CAPTURE 제한과 다르며 device 유형을 구분해야 합니다.

Request를 할당하고 `VIDIOC_S_EXT_CTRLS`로 control을 저장한 뒤 CAPTURE buffer에 `V4L2_BUF_FLAG_REQUEST_FD`와 request fd를 넣어 queue합니다.

struct v4l2_buffer buf;
struct v4l2_ext_controls ctrls;
int req_fd;
...
if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
    return errno;
...
ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
ctrls.request_fd = req_fd;
if (ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
    return errno;
...
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
buf.request_fd = req_fd;
if (ioctl(camera_fd, VIDIOC_QBUF, &buf))
    return errno;

완전히 준비된 request를 `MEDIA_REQUEST_IOC_QUEUE`로 제출합니다. 이후 buffer dequeue, 완료 대기, control 조회와 request 재사용은 M2M 예제와 같습니다.

if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
    return errno;

Example for a Simple Capture Device
-----------------------------------

With a simple capture device, requests can be used to specify controls to apply
for a given CAPTURE buffer.

.. code-block:: c

	struct v4l2_buffer buf;
	struct v4l2_ext_controls ctrls;
	int req_fd;
	...
	if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
		return errno;
	...
	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
	ctrls.request_fd = req_fd;
	if (ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
		return errno;
	...
	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
	buf.request_fd = req_fd;
	if (ioctl(camera_fd, VIDIOC_QBUF, &buf))
		return errno;

Once the request is fully prepared, it can be queued to the driver:

.. code-block:: c

	if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
		return errno;

User-space can then dequeue buffers, wait for the request completion, query
controls and recycle the request as in the M2M example above.