요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
=========================================
Rockchip Image Signal Processor (rkisp1)
=========================================
Introduction
============
This file documents the driver for the Rockchip ISP1 that is part of RK3288
and RK3399 SoCs. The driver is located under drivers/media/platform/rockchip/
rkisp1 and uses the Media-Controller API.
Revisions
=========
There exist multiple smaller revisions to this ISP that got introduced in
later SoCs. Revisions can be found in the enum :c:type:`rkisp1_cif_isp_version`
in the UAPI and the revision of the ISP inside the running SoC can be read
in the field hw_revision of struct media_device_info as returned by
ioctl MEDIA_IOC_DEVICE_INFO.
Versions in use are:
- RKISP1_V10: used at least in rk3288 and rk3399
- RKISP1_V11: declared in the original vendor code, but not used
- RKISP1_V12: used at least in rk3326 and px30
- RKISP1_V13: used at least in rk1808
Topology
========
.. _rkisp1_topology_graph:
.. kernel-figure:: rkisp1.dot
:alt: Diagram of the default media pipeline topology
:align: center
The driver has 4 video devices:
- rkisp1_mainpath: capture device for retrieving images, usually in higher
resolution.
- rkisp1_selfpath: capture device for retrieving images.
- rkisp1_stats: a metadata capture device that sends statistics.
- rkisp1_params: a metadata output device that receives parameters
configurations from userspace.
The driver has 3 subdevices:
- rkisp1_resizer_mainpath: used to resize and downsample frames for the
mainpath capture device.
- rkisp1_resizer_selfpath: used to resize and downsample frames for the
selfpath capture device.
- rkisp1_isp: is connected to the sensor and is responsible for all the isp
operations.
rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes
-------------------------------------------------------------
Those are the `mainpath` and `selfpath` capture devices to capture frames.
Those entities are the DMA engines that write the frames to memory.
The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded
stream and it is able to convert it to RGB. The selfpath is not able to
capture bayer formats.
The mainpath can capture both bayer and YUV formats but it is not able to
capture RGB formats.
Both capture videos support
the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`.
rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes
----------------------------------------------------------------------------
Those are resizer entities for the mainpath and the selfpath. Those entities
can scale the frames up and down and also change the YUV sampling (for example
YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad.
The resizers entities can only operate on YUV:4:2:2 format
(MEDIA_BUS_FMT_YUYV8_2X8).
The mainpath capture device supports capturing video in bayer formats. In that
case the resizer of the mainpath is set to 'bypass' mode - it just forward the
frame without operating on it.
rkisp1_isp - Image Signal Processing Subdevice Node
---------------------------------------------------
This is the isp entity. It is connected to the sensor on sink pad 0 and
receives the frames using the CSI-2 protocol. It is responsible of configuring
the CSI-2 protocol. It has a cropping capability on sink pad 0 that is
connected to the sensor and on source pad 2 connected to the resizer entities.
Cropping on sink pad 0 defines the image region from the sensor.
Cropping on source pad 2 defines the region for the Image Stabilizer (IS).
.. _rkisp1_stats:
rkisp1_stats - Statistics Video Node
------------------------------------
The statistics video node outputs the 3A (auto focus, auto exposure and auto
white balance) statistics, and also histogram statistics for the frames that
are being processed by the rkisp1 to userspace applications.
Using these data, applications can implement algorithms and re-parameterize
the driver through the rkisp_params node to improve image quality during a
video stream.
The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and
userspace should set
:ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-rk-isp1-stat-3a>` as the
dataformat.
.. _rkisp1_params:
rkisp1_params - Parameters Video Node
-------------------------------------
The rkisp1_params video node receives a set of parameters from userspace
to be applied to the hardware during a video stream, allowing userspace
to dynamically modify values such as black level, cross talk corrections
and others.
The ISP driver supports two different parameters configuration methods, the
`fixed parameters format` or the `extensible parameters format`.
When using the `fixed parameters` method the buffer format is defined by struct
:c:type:`rkisp1_params_cfg`, and userspace should set
:ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-rk-isp1-params>` as the
dataformat.
When using the `extensible parameters` method the buffer format is defined by
struct :c:type:`rkisp1_ext_params_cfg`, and userspace should set
:ref:`V4L2_META_FMT_RK_ISP1_EXT_PARAMS <v4l2-meta-fmt-rk-isp1-ext-params>` as
the dataformat.
Capturing Video Frames Example
==============================
In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is
imx219.
The following commands can be used to capture video from the selfpath video
node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping
capabilities possible, (see explanation right below)
.. code-block:: bash
# set the links
"media-ctl" "-d" "platform:rkisp1" "-r"
"media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]"
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]"
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]"
# set format for imx219 4-0010:0
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]'
# set format for rkisp1_isp pads:
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]'
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]'
# set format for rkisp1_resizer_selfpath pads:
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]'
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]'
# set format for rkisp1_selfpath:
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800,"
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P"
# start streaming:
v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10"
In the above example the sensor is configured to bayer format:
`SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the
same mbus format and dimensions as the sensor, otherwise streaming will fail
with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`.
In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`.
The cropping dimensions are automatically propagated to be the format of the
isp source pad `rkisp1_isp:2`. Another cropping operation is configured on
the isp source pad: `(0,0)/1500x1100`.
The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format
`YUYV8_2X8/1500x1100` in order to match the format on the other side of the
link. In addition a cropping `(300,400)/1400x1000` is configured on it.
The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to
format `YUYV8_2X8/900x800`. That means that the resizer first crop a window
of `(300,400)/1400x100` from the received frame and then scales this window
to dimension `900x800`.
Note that the above example does not uses the stats-params control loop.
Therefore the capture frames will not go through the 3A algorithms and
probably won't have a good quality, and can even look dark and greenish.
Configuring Quantization
========================
The driver supports limited and full range quantization on YUV formats,
where limited is the default.
To switch between one or the other, userspace should use the Colorspace
Conversion API (CSC) for subdevices on source pad 2 of the
isp (`rkisp1_isp:2`). The quantization configured on this pad is the
quantization of the captured video frames on the mainpath and selfpath
video nodes.
Note that the resizer and capture entities will always report
``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full
range on `rkisp1_isp:2`. So in order to get the configured quantization,
application should get it from pad `rkisp1_isp:2`.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
Rockchip ISP1 driver 소개
1-15이 문서는 GPL-2.0 라이선스를 따릅니다. Rockchip ISP1은 RK3288과 RK3399 SoC에 포함되어 있으며, driver는 `drivers/media/platform/rockchip/rkisp1`에 있고 Media-Controller API를 사용합니다.
ISP revision
16-31후기 SoC에는 이 ISP의 여러 소규모 revision이 도입되었습니다. Revision은 UAPI의 `rkisp1_cif_isp_version` enum에서 확인할 수 있습니다. 실행 중인 SoC의 ISP revision은 `MEDIA_IOC_DEVICE_INFO` ioctl이 반환하는 `media_device_info` struct의 `hw_revision` field에서 읽습니다.
| Revision | 확인된 SoC 또는 출처 | 상태 |
|---|---|---|
| RKISP1_V10 | rk3288, rk3399 | 사용 |
| RKISP1_V11 | Original vendor code | 선언되었지만 사용되지 않음 |
| RKISP1_V12 | rk3326, px30 | 사용 |
| RKISP1_V13 | rk1808 | 사용 |
Media topology
32-59원문의 `rkisp1.dot` graph는 sensor, ISP, main/self resizer, 네 video device의 기본 media pipeline topology를 나타냅니다.
| Entity | Device node | Type | 역할 |
|---|---|---|---|
| rkisp1_isp | /dev/v4l-subdev0 | Subdevice | Sensor CSI-2 입력, ISP operation, crop |
| rkisp1_resizer_mainpath | /dev/v4l-subdev1 | Subdevice | Mainpath resize/downsample/crop |
| rkisp1_resizer_selfpath | /dev/v4l-subdev2 | Subdevice | Selfpath resize/downsample/crop |
| rkisp1_mainpath | /dev/video0 | Capture | 주로 고해상도 image frame |
| rkisp1_selfpath | /dev/video1 | Capture | Image frame |
| rkisp1_stats | /dev/video2 | Metadata capture | 3A와 histogram statistics |
| rkisp1_params | /dev/video3 | Metadata output | Userspace parameter configuration |
ISP source pad 2는 두 resizer로 분기하고 source pad 3은 statistics로 이어집니다. Parameter metadata는 ISP sink pad 1로 들어갑니다.
rkisp1_mainpath와 rkisp1_selfpath
60-72`mainpath`와 `selfpath`는 frame을 capture하는 device입니다. 이 entity들은 frame을 memory에 기록하는 DMA engine입니다.
Selfpath video device는 YUV/RGB format을 capture할 수 있습니다. Input은 YUV encoded stream이며 RGB로 변환할 수 있지만 Bayer format은 capture할 수 없습니다. Mainpath는 Bayer와 YUV를 capture할 수 있지만 RGB는 capture할 수 없습니다.
| Entity | Bayer | YUV | RGB | 비고 |
|---|---|---|---|---|
| rkisp1_mainpath | 지원 | 지원 | 미지원 | Bayer에서는 resizer bypass |
| rkisp1_selfpath | 미지원 | 지원 | 지원 | YUV input을 RGB로 변환 가능 |
두 capture video device는 모두 `V4L2_CAP_IO_MC` capability를 지원합니다.
Mainpath와 selfpath resizer
73-83두 resizer entity는 mainpath와 selfpath용입니다. Frame을 upscale/downscale하고 YUV sampling을 변경할 수 있으며(예: YUV4:2:2 → YUV4:2:0), sink pad에서 cropping도 지원합니다.
Resizer entity는 YUV 4:2:2 format인 `MEDIA_BUS_FMT_YUYV8_2X8`에서만 동작합니다. Mainpath capture device가 Bayer format을 capture할 때 mainpath resizer는 `bypass` mode로 설정되어 frame을 처리하지 않고 그대로 전달합니다.
rkisp1_isp subdevice
84-92`rkisp1_isp`는 ISP entity입니다. Sink pad 0에서 sensor와 연결되어 CSI-2 protocol로 frame을 받고 CSI-2 protocol 설정을 담당합니다. Sensor에 연결된 sink pad 0과 resizer entity에 연결된 source pad 2에서 cropping을 지원합니다.
Sink pad 0의 crop은 sensor image region을 정의하고, source pad 2의 crop은 Image Stabilizer(IS)가 사용할 region을 정의합니다.
rkisp1_stats video node
93-107`rkisp1_stats` video node는 rkisp1이 처리 중인 frame의 3A(auto focus, auto exposure, auto white balance) statistics와 histogram statistics를 userspace application에 출력합니다.
Application은 이 data로 algorithm을 구현하고 video stream 중 `rkisp1_params` node를 통해 driver parameter를 다시 설정하여 image quality를 개선할 수 있습니다.
Buffer format은 `rkisp1_stat_buffer` struct로 정의되며 userspace는 dataformat을 `V4L2_META_FMT_RK_ISP1_STAT_3A`로 설정해야 합니다.
rkisp1_params video node
108-129`rkisp1_params` video node는 video stream 중 hardware에 적용할 parameter set을 userspace에서 받습니다. 이를 통해 userspace는 black level, cross talk correction 등의 값을 동적으로 변경할 수 있습니다.
ISP driver는 `fixed parameters format`과 `extensible parameters format`의 두 configuration 방식을 지원합니다.
| Configuration method | Buffer structure | V4L2 dataformat |
|---|---|---|
| Fixed parameters | rkisp1_params_cfg | V4L2_META_FMT_RK_ISP1_PARAMS |
| Extensible parameters | rkisp1_ext_params_cfg | V4L2_META_FMT_RK_ISP1_EXT_PARAMS |
Selfpath capture example
130-166다음 예시는 `rkisp1_isp` pad 0에 `imx219` sensor가 연결된 환경에서 selfpath video node로 900x800 planar YUV 4:2:2 video를 capture합니다. 가능한 모든 cropping capability를 사용합니다.
# set the links
"media-ctl" "-d" "platform:rkisp1" "-r"
"media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]"
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]"
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]"
# set format for imx219 4-0010:0
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]'
# set format for rkisp1_isp pads:
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]'
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]'
# set format for rkisp1_resizer_selfpath pads:
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]'
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]'
# set format for rkisp1_selfpath:
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800,"
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P"
# start streaming:
v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10"Format과 crop propagation
167-189Sensor는 Bayer format `SRGGB10_1X10/1640x1232`로 설정됩니다. `rkisp1_isp:0` pad의 mbus format과 dimension은 sensor와 같아야 하며, 다르면 streaming이 `EPIPE` error로 실패합니다. 따라서 이 pad도 `SRGGB10_1X10/1640x1232`로 설정하고 `(0,0)/1600x1200` crop을 적용합니다.
Crop dimension은 ISP source pad `rkisp1_isp:2`의 format으로 자동 전파됩니다. 이어서 ISP source pad에 `(0,0)/1500x1100` crop을 하나 더 설정합니다.
Resizer sink pad `rkisp1_resizer_selfpath`는 link 반대쪽 format과 일치하도록 `YUYV8_2X8/1500x1100`으로 설정해야 하며, 여기에 `(300,400)/1400x1000` crop을 적용합니다.
Resizer source pad `rkisp1_resizer_selfpath:1`은 `YUYV8_2X8/900x800`으로 설정합니다. 즉 resizer는 받은 frame에서 `(300,400)/1400x100` window를 먼저 crop한 다음 `900x800`으로 scale합니다.
원문 line 183은 crop window를 `(300,400)/1400x100`으로 적지만 example command line 156은 `(300,400)/1400x1000`을 사용합니다. 이 불일치를 원문 그대로 보존했습니다.
이 예시는 stats-params control loop를 사용하지 않습니다. 따라서 capture frame은 3A algorithm을 거치지 않으며 image quality가 좋지 않거나 어둡고 녹색을 띨 수 있습니다.
Quantization 설정
190-204Driver는 YUV format에서 limited range와 full range quantization을 지원하며 default는 limited입니다.
둘 사이를 전환하려면 userspace가 ISP source pad 2(`rkisp1_isp:2`)에서 subdevice용 Colorspace Conversion API(CSC)를 사용해야 합니다. 이 pad에 설정한 quantization이 mainpath와 selfpath video node에서 capture한 video frame의 quantization입니다.
Resizer와 capture entity는 `rkisp1_isp:2`에서 full range를 설정하더라도 항상 `V4L2_QUANTIZATION_DEFAULT`를 보고합니다. 따라서 application은 실제 설정된 quantization을 `rkisp1_isp:2` pad에서 가져와야 합니다.
rkisp1 pipeline 요약
rkisp1.rst:1-204Sensor frame은 `rkisp1_isp`에서 CSI-2 수신과 image processing을 거친 뒤 mainpath/selfpath resizer와 DMA capture node로 분기합니다. Statistics metadata는 userspace 3A loop로 나가고 parameter metadata는 반대로 ISP 설정으로 들어옵니다.