요약·해설과 원문, 전문 번역을 서로 분리했습니다. API 이름, symbol, source path는 원문 표기를 사용합니다.
1. 요약·해설
원문의 핵심 논리와 kernel programming 관점의 보충 설명입니다. 아래의 전문 번역과는 별도로 작성했습니다.
2. 영어 원문 전체
번역 기준이 된 Linux v6.18.37 원문입니다. 줄 번호는 이 버전의 파일 좌표입니다.
원문 전체 펼치기
.. SPDX-License-Identifier: GPL-2.0
i.MX Video Capture Driver
=========================
Introduction
------------
The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
handles the flow of image frames to and from capture devices and
display devices.
For image capture, the IPU contains the following internal subunits:
- Image DMA Controller (IDMAC)
- Camera Serial Interface (CSI)
- Image Converter (IC)
- Sensor Multi-FIFO Controller (SMFC)
- Image Rotator (IRT)
- Video De-Interlacing or Combining Block (VDIC)
The IDMAC is the DMA controller for transfer of image frames to and from
memory. Various dedicated DMA channels exist for both video capture and
display paths. During transfer, the IDMAC is also capable of vertical
image flip, 8x8 block transfer (see IRT description), pixel component
re-ordering (for example UYVY to YUYV) within the same colorspace, and
packed <--> planar conversion. The IDMAC can also perform a simple
de-interlacing by interweaving even and odd lines during transfer
(without motion compensation which requires the VDIC).
The CSI is the backend capture unit that interfaces directly with
camera sensors over Parallel, BT.656/1120, and MIPI CSI-2 buses.
The IC handles color-space conversion, resizing (downscaling and
upscaling), horizontal flip, and 90/270 degree rotation operations.
There are three independent "tasks" within the IC that can carry out
conversions concurrently: pre-process encoding, pre-process viewfinder,
and post-processing. Within each task, conversions are split into three
sections: downsizing section, main section (upsizing, flip, colorspace
conversion, and graphics plane combining), and rotation section.
The IPU time-shares the IC task operations. The time-slice granularity
is one burst of eight pixels in the downsizing section, one image line
in the main processing section, one image frame in the rotation section.
The SMFC is composed of four independent FIFOs that each can transfer
captured frames from sensors directly to memory concurrently via four
IDMAC channels.
The IRT carries out 90 and 270 degree image rotation operations. The
rotation operation is carried out on 8x8 pixel blocks at a time. This
operation is supported by the IDMAC which handles the 8x8 block transfer
along with block reordering, in coordination with vertical flip.
The VDIC handles the conversion of interlaced video to progressive, with
support for different motion compensation modes (low, medium, and high
motion). The deinterlaced output frames from the VDIC can be sent to the
IC pre-process viewfinder task for further conversions. The VDIC also
contains a Combiner that combines two image planes, with alpha blending
and color keying.
In addition to the IPU internal subunits, there are also two units
outside the IPU that are also involved in video capture on i.MX:
- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
interface. This is a Synopsys DesignWare core.
- Two video multiplexers for selecting among multiple sensor inputs
to send to a CSI.
For more info, refer to the latest versions of the i.MX5/6 reference
manuals [#f1]_ and [#f2]_.
Features
--------
Some of the features of this driver include:
- Many different pipelines can be configured via media controller API,
that correspond to the hardware video capture pipelines supported in
the i.MX.
- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
- Concurrent independent streams, by configuring pipelines to multiple
video capture interfaces using independent entities.
- Scaling, color-space conversion, horizontal and vertical flip, and
image rotation via IC task subdevs.
- Many pixel formats supported (RGB, packed and planar YUV, partial
planar YUV).
- The VDIC subdev supports motion compensated de-interlacing, with three
motion compensation modes: low, medium, and high motion. Pipelines are
defined that allow sending frames to the VDIC subdev directly from the
CSI. There is also support in the future for sending frames to the
VDIC from memory buffers via output/mem2mem devices.
- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
problems with the ADV718x video decoders.
Topology
--------
The following shows the media topologies for the i.MX6Q SabreSD and
i.MX6Q SabreAuto. Refer to these diagrams in the entity descriptions
in the next section.
The i.MX5/6 topologies can differ upstream from the IPUv3 CSI video
multiplexers, but the internal IPUv3 topology downstream from there
is common to all i.MX5/6 platforms. For example, the SabreSD, with the
MIPI CSI-2 OV5640 sensor, requires the i.MX6 MIPI CSI-2 receiver. But
the SabreAuto has only the ADV7180 decoder on a parallel bt.656 bus, and
therefore does not require the MIPI CSI-2 receiver, so it is missing in
its graph.
.. _imx6q_topology_graph:
.. kernel-figure:: imx6q-sabresd.dot
:alt: Diagram of the i.MX6Q SabreSD media pipeline topology
:align: center
Media pipeline graph on i.MX6Q SabreSD
.. kernel-figure:: imx6q-sabreauto.dot
:alt: Diagram of the i.MX6Q SabreAuto media pipeline topology
:align: center
Media pipeline graph on i.MX6Q SabreAuto
Entities
--------
imx6-mipi-csi2
--------------
This is the MIPI CSI-2 receiver entity. It has one sink pad to receive
the MIPI CSI-2 stream (usually from a MIPI CSI-2 camera sensor). It has
four source pads, corresponding to the four MIPI CSI-2 demuxed virtual
channel outputs. Multiple source pads can be enabled to independently
stream from multiple virtual channels.
This entity actually consists of two sub-blocks. One is the MIPI CSI-2
core. This is a Synopsys Designware MIPI CSI-2 core. The other sub-block
is a "CSI-2 to IPU gasket". The gasket acts as a demultiplexer of the
four virtual channels streams, providing four separate parallel buses
containing each virtual channel that are routed to CSIs or video
multiplexers as described below.
On i.MX6 solo/dual-lite, all four virtual channel buses are routed to
two video multiplexers. Both CSI0 and CSI1 can receive any virtual
channel, as selected by the video multiplexers.
On i.MX6 Quad, virtual channel 0 is routed to IPU1-CSI0 (after selected
by a video mux), virtual channels 1 and 2 are hard-wired to IPU1-CSI1
and IPU2-CSI0, respectively, and virtual channel 3 is routed to
IPU2-CSI1 (again selected by a video mux).
ipuX_csiY_mux
-------------
These are the video multiplexers. They have two or more sink pads to
select from either camera sensors with a parallel interface, or from
MIPI CSI-2 virtual channels from imx6-mipi-csi2 entity. They have a
single source pad that routes to a CSI (ipuX_csiY entities).
On i.MX6 solo/dual-lite, there are two video mux entities. One sits
in front of IPU1-CSI0 to select between a parallel sensor and any of
the four MIPI CSI-2 virtual channels (a total of five sink pads). The
other mux sits in front of IPU1-CSI1, and again has five sink pads to
select between a parallel sensor and any of the four MIPI CSI-2 virtual
channels.
On i.MX6 Quad, there are two video mux entities. One sits in front of
IPU1-CSI0 to select between a parallel sensor and MIPI CSI-2 virtual
channel 0 (two sink pads). The other mux sits in front of IPU2-CSI1 to
select between a parallel sensor and MIPI CSI-2 virtual channel 3 (two
sink pads).
ipuX_csiY
---------
These are the CSI entities. They have a single sink pad receiving from
either a video mux or from a MIPI CSI-2 virtual channel as described
above.
This entity has two source pads. The first source pad can link directly
to the ipuX_vdic entity or the ipuX_ic_prp entity, using hardware links
that require no IDMAC memory buffer transfer.
When the direct source pad is routed to the ipuX_ic_prp entity, frames
from the CSI can be processed by one or both of the IC pre-processing
tasks.
When the direct source pad is routed to the ipuX_vdic entity, the VDIC
will carry out motion-compensated de-interlace using "high motion" mode
(see description of ipuX_vdic entity).
The second source pad sends video frames directly to memory buffers
via the SMFC and an IDMAC channel, bypassing IC pre-processing. This
source pad is routed to a capture device node, with a node name of the
format "ipuX_csiY capture".
Note that since the IDMAC source pad makes use of an IDMAC channel,
pixel reordering within the same colorspace can be carried out by the
IDMAC channel. For example, if the CSI sink pad is receiving in UYVY
order, the capture device linked to the IDMAC source pad can capture
in YUYV order. Also, if the CSI sink pad is receiving a packed YUV
format, the capture device can capture a planar YUV format such as
YUV420.
The IDMAC channel at the IDMAC source pad also supports simple
interweave without motion compensation, which is activated if the source
pad's field type is sequential top-bottom or bottom-top, and the
requested capture interface field type is set to interlaced (t-b, b-t,
or unqualified interlaced). The capture interface will enforce the same
field order as the source pad field order (interlaced-bt if source pad
is seq-bt, interlaced-tb if source pad is seq-tb).
For events produced by ipuX_csiY, see ref:`imx_api_ipuX_csiY`.
Cropping in ipuX_csiY
---------------------
The CSI supports cropping the incoming raw sensor frames. This is
implemented in the ipuX_csiY entities at the sink pad, using the
crop selection subdev API.
The CSI also supports fixed divide-by-two downscaling independently in
width and height. This is implemented in the ipuX_csiY entities at
the sink pad, using the compose selection subdev API.
The output rectangle at the ipuX_csiY source pad is the same as
the compose rectangle at the sink pad. So the source pad rectangle
cannot be negotiated, it must be set using the compose selection
API at sink pad (if /2 downscale is desired, otherwise source pad
rectangle is equal to incoming rectangle).
To give an example of crop and /2 downscale, this will crop a
1280x960 input frame to 640x480, and then /2 downscale in both
dimensions to 320x240 (assumes ipu1_csi0 is linked to ipu1_csi0_mux):
.. code-block:: none
media-ctl -V "'ipu1_csi0_mux':2[fmt:UYVY2X8/1280x960]"
media-ctl -V "'ipu1_csi0':0[crop:(0,0)/640x480]"
media-ctl -V "'ipu1_csi0':0[compose:(0,0)/320x240]"
Frame Skipping in ipuX_csiY
---------------------------
The CSI supports frame rate decimation, via frame skipping. Frame
rate decimation is specified by setting the frame intervals at
sink and source pads. The ipuX_csiY entity then applies the best
frame skip setting to the CSI to achieve the desired frame rate
at the source pad.
The following example reduces an assumed incoming 60 Hz frame
rate by half at the IDMAC output source pad:
.. code-block:: none
media-ctl -V "'ipu1_csi0':0[fmt:UYVY2X8/640x480@1/60]"
media-ctl -V "'ipu1_csi0':2[fmt:UYVY2X8/640x480@1/30]"
Frame Interval Monitor in ipuX_csiY
-----------------------------------
See ref:`imx_api_FIM`.
ipuX_vdic
---------
The VDIC carries out motion compensated de-interlacing, with three
motion compensation modes: low, medium, and high motion. The mode is
specified with the menu control V4L2_CID_DEINTERLACING_MODE. The VDIC
has two sink pads and a single source pad.
The direct sink pad receives from an ipuX_csiY direct pad. With this
link the VDIC can only operate in high motion mode.
When the IDMAC sink pad is activated, it receives from an output
or mem2mem device node. With this pipeline, the VDIC can also operate
in low and medium modes, because these modes require receiving
frames from memory buffers. Note that an output or mem2mem device
is not implemented yet, so this sink pad currently has no links.
The source pad routes to the IC pre-processing entity ipuX_ic_prp.
ipuX_ic_prp
-----------
This is the IC pre-processing entity. It acts as a router, routing
data from its sink pad to one or both of its source pads.
This entity has a single sink pad. The sink pad can receive from the
ipuX_csiY direct pad, or from ipuX_vdic.
This entity has two source pads. One source pad routes to the
pre-process encode task entity (ipuX_ic_prpenc), the other to the
pre-process viewfinder task entity (ipuX_ic_prpvf). Both source pads
can be activated at the same time if the sink pad is receiving from
ipuX_csiY. Only the source pad to the pre-process viewfinder task entity
can be activated if the sink pad is receiving from ipuX_vdic (frames
from the VDIC can only be processed by the pre-process viewfinder task).
ipuX_ic_prpenc
--------------
This is the IC pre-processing encode entity. It has a single sink
pad from ipuX_ic_prp, and a single source pad. The source pad is
routed to a capture device node, with a node name of the format
"ipuX_ic_prpenc capture".
This entity performs the IC pre-process encode task operations:
color-space conversion, resizing (downscaling and upscaling),
horizontal and vertical flip, and 90/270 degree rotation. Flip
and rotation are provided via standard V4L2 controls.
Like the ipuX_csiY IDMAC source, this entity also supports simple
de-interlace without motion compensation, and pixel reordering.
ipuX_ic_prpvf
-------------
This is the IC pre-processing viewfinder entity. It has a single sink
pad from ipuX_ic_prp, and a single source pad. The source pad is routed
to a capture device node, with a node name of the format
"ipuX_ic_prpvf capture".
This entity is identical in operation to ipuX_ic_prpenc, with the same
resizing and CSC operations and flip/rotation controls. It will receive
and process de-interlaced frames from the ipuX_vdic if ipuX_ic_prp is
receiving from ipuX_vdic.
Like the ipuX_csiY IDMAC source, this entity supports simple
interweaving without motion compensation. However, note that if the
ipuX_vdic is included in the pipeline (ipuX_ic_prp is receiving from
ipuX_vdic), it's not possible to use interweave in ipuX_ic_prpvf,
since the ipuX_vdic has already carried out de-interlacing (with
motion compensation) and therefore the field type output from
ipuX_vdic can only be none (progressive).
Capture Pipelines
-----------------
The following describe the various use-cases supported by the pipelines.
The links shown do not include the backend sensor, video mux, or mipi
csi-2 receiver links. This depends on the type of sensor interface
(parallel or mipi csi-2). So these pipelines begin with:
sensor -> ipuX_csiY_mux -> ...
for parallel sensors, or:
sensor -> imx6-mipi-csi2 -> (ipuX_csiY_mux) -> ...
for mipi csi-2 sensors. The imx6-mipi-csi2 receiver may need to route
to the video mux (ipuX_csiY_mux) before sending to the CSI, depending
on the mipi csi-2 virtual channel, hence ipuX_csiY_mux is shown in
parenthesis.
Unprocessed Video Capture:
--------------------------
Send frames directly from sensor to camera device interface node, with
no conversions, via ipuX_csiY IDMAC source pad:
-> ipuX_csiY:2 -> ipuX_csiY capture
IC Direct Conversions:
----------------------
This pipeline uses the preprocess encode entity to route frames directly
from the CSI to the IC, to carry out scaling up to 1024x1024 resolution,
CSC, flipping, and image rotation:
-> ipuX_csiY:1 -> 0:ipuX_ic_prp:1 -> 0:ipuX_ic_prpenc:1 -> ipuX_ic_prpenc capture
Motion Compensated De-interlace:
--------------------------------
This pipeline routes frames from the CSI direct pad to the VDIC entity to
support motion-compensated de-interlacing (high motion mode only),
scaling up to 1024x1024, CSC, flip, and rotation:
-> ipuX_csiY:1 -> 0:ipuX_vdic:2 -> 0:ipuX_ic_prp:2 -> 0:ipuX_ic_prpvf:1 -> ipuX_ic_prpvf capture
Usage Notes
-----------
To aid in configuration and for backward compatibility with V4L2
applications that access controls only from video device nodes, the
capture device interfaces inherit controls from the active entities
in the current pipeline, so controls can be accessed either directly
from the subdev or from the active capture device interface. For
example, the FIM controls are available either from the ipuX_csiY
subdevs or from the active capture device.
The following are specific usage notes for the Sabre* reference
boards:
i.MX6Q SabreLite with OV5642 and OV5640
---------------------------------------
This platform requires the OmniVision OV5642 module with a parallel
camera interface, and the OV5640 module with a MIPI CSI-2
interface. Both modules are available from Boundary Devices:
- https://boundarydevices.com/product/nit6x_5mp
- https://boundarydevices.com/product/nit6x_5mp_mipi
Note that if only one camera module is available, the other sensor
node can be disabled in the device tree.
The OV5642 module is connected to the parallel bus input on the i.MX
internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
receiver, and the four virtual channel outputs from the receiver are
routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
OV5640 must not share the same i2c slave address.
The following basic example configures unprocessed video capture
pipelines for both sensors. The OV5642 is routed to ipu1_csi0, and
the OV5640, transmitting on MIPI CSI-2 virtual channel 1 (which is
imx6-mipi-csi2 pad 2), is routed to ipu1_csi1. Both sensors are
configured to output 640x480, and the OV5642 outputs YUYV2X8, the
OV5640 UYVY2X8:
.. code-block:: none
# Setup links for OV5642
media-ctl -l "'ov5642 1-0042':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Setup links for OV5640
media-ctl -l "'ov5640 1-0040':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':2 -> 'ipu1_csi1':0[1]"
media-ctl -l "'ipu1_csi1':2 -> 'ipu1_csi1 capture':0[1]"
# Configure pads for OV5642 pipeline
media-ctl -V "'ov5642 1-0042':0 [fmt:YUYV2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:YUYV2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/640x480 field:none]"
# Configure pads for OV5640 pipeline
media-ctl -V "'ov5640 1-0040':0 [fmt:UYVY2X8/640x480 field:none]"
media-ctl -V "'imx6-mipi-csi2':2 [fmt:UYVY2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi1':2 [fmt:AYUV32/640x480 field:none]"
Streaming can then begin independently on the capture device nodes
"ipu1_csi0 capture" and "ipu1_csi1 capture". The v4l2-ctl tool can
be used to select any supported YUV pixelformat on the capture device
nodes, including planar.
i.MX6Q SabreAuto with ADV7180 decoder
-------------------------------------
On the i.MX6Q SabreAuto, an on-board ADV7180 SD decoder is connected to the
parallel bus input on the internal video mux to IPU1 CSI0.
The following example configures a pipeline to capture from the ADV7180
video decoder, assuming NTSC 720x480 input signals, using simple
interweave (unconverted and without motion compensation). The adv7180
must output sequential or alternating fields (field type 'seq-bt' for
NTSC, or 'alternate'):
.. code-block:: none
# Setup links
media-ctl -l "'adv7180 3-0021':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 3-0021':0 [fmt:UYVY2X8/720x480 field:seq-bt]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
# Configure "ipu1_csi0 capture" interface (assumed at /dev/video4)
v4l2-ctl -d4 --set-fmt-video=field=interlaced_bt
Streaming can then begin on /dev/video4. The v4l2-ctl tool can also be
used to select any supported YUV pixelformat on /dev/video4.
This example configures a pipeline to capture from the ADV7180
video decoder, assuming PAL 720x576 input signals, with Motion
Compensated de-interlacing. The adv7180 must output sequential or
alternating fields (field type 'seq-tb' for PAL, or 'alternate').
.. code-block:: none
# Setup links
media-ctl -l "'adv7180 3-0021':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 3-0021':0 [fmt:UYVY2X8/720x576 field:seq-tb]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x576]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x576]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]"
# Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2)
v4l2-ctl -d2 --set-fmt-video=field=none
Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be
used to select any supported YUV pixelformat on /dev/video2.
This platform accepts Composite Video analog inputs to the ADV7180 on
Ain1 (connector J42).
i.MX6DL SabreAuto with ADV7180 decoder
--------------------------------------
On the i.MX6DL SabreAuto, an on-board ADV7180 SD decoder is connected to the
parallel bus input on the internal video mux to IPU1 CSI0.
The following example configures a pipeline to capture from the ADV7180
video decoder, assuming NTSC 720x480 input signals, using simple
interweave (unconverted and without motion compensation). The adv7180
must output sequential or alternating fields (field type 'seq-bt' for
NTSC, or 'alternate'):
.. code-block:: none
# Setup links
media-ctl -l "'adv7180 4-0021':0 -> 'ipu1_csi0_mux':4[1]"
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 4-0021':0 [fmt:UYVY2X8/720x480 field:seq-bt]"
media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
# Configure "ipu1_csi0 capture" interface (assumed at /dev/video0)
v4l2-ctl -d0 --set-fmt-video=field=interlaced_bt
Streaming can then begin on /dev/video0. The v4l2-ctl tool can also be
used to select any supported YUV pixelformat on /dev/video0.
This example configures a pipeline to capture from the ADV7180
video decoder, assuming PAL 720x576 input signals, with Motion
Compensated de-interlacing. The adv7180 must output sequential or
alternating fields (field type 'seq-tb' for PAL, or 'alternate').
.. code-block:: none
# Setup links
media-ctl -l "'adv7180 4-0021':0 -> 'ipu1_csi0_mux':4[1]"
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 4-0021':0 [fmt:UYVY2X8/720x576 field:seq-tb]"
media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x576]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x576]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]"
# Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2)
v4l2-ctl -d2 --set-fmt-video=field=none
Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be
used to select any supported YUV pixelformat on /dev/video2.
This platform accepts Composite Video analog inputs to the ADV7180 on
Ain1 (connector J42).
i.MX6Q SabreSD with MIPI CSI-2 OV5640
-------------------------------------
Similarly to i.MX6Q SabreLite, the i.MX6Q SabreSD supports a parallel
interface OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640
module. The OV5642 connects to i2c bus 1 and the OV5640 to i2c bus 2.
The device tree for SabreSD includes OF graphs for both the parallel
OV5642 and the MIPI CSI-2 OV5640, but as of this writing only the MIPI
CSI-2 OV5640 has been tested, so the OV5642 node is currently disabled.
The OV5640 module connects to MIPI connector J5. The NXP part number
for the OV5640 module that connects to the SabreSD board is H120729.
The following example configures unprocessed video capture pipeline to
capture from the OV5640, transmitting on MIPI CSI-2 virtual channel 0:
.. code-block:: none
# Setup links
media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Configure pads
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY2X8/640x480]"
media-ctl -V "'imx6-mipi-csi2':1 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0_mux':0 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/640x480]"
Streaming can then begin on "ipu1_csi0 capture" node. The v4l2-ctl
tool can be used to select any supported pixelformat on the capture
device node.
To determine what is the /dev/video node correspondent to
"ipu1_csi0 capture":
.. code-block:: none
media-ctl -e "ipu1_csi0 capture"
/dev/video0
/dev/video0 is the streaming element in this case.
Starting the streaming via v4l2-ctl:
.. code-block:: none
v4l2-ctl --stream-mmap -d /dev/video0
Starting the streaming via Gstreamer and sending the content to the display:
.. code-block:: none
gst-launch-1.0 v4l2src device=/dev/video0 ! kmssink
The following example configures a direct conversion pipeline to capture
from the OV5640, transmitting on MIPI CSI-2 virtual channel 0. It also
shows colorspace conversion and scaling at IC output.
.. code-block:: none
# Setup links
media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# Configure pads
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY2X8/640x480]"
media-ctl -V "'imx6-mipi-csi2':1 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:ARGB8888_1X32/800x600]"
# Set a format at the capture interface
v4l2-ctl -d /dev/video1 --set-fmt-video=pixelformat=RGB3
Streaming can then begin on "ipu1_ic_prpenc capture" node.
To determine what is the /dev/video node correspondent to
"ipu1_ic_prpenc capture":
.. code-block:: none
media-ctl -e "ipu1_ic_prpenc capture"
/dev/video1
/dev/video1 is the streaming element in this case.
Starting the streaming via v4l2-ctl:
.. code-block:: none
v4l2-ctl --stream-mmap -d /dev/video1
Starting the streaming via Gstreamer and sending the content to the display:
.. code-block:: none
gst-launch-1.0 v4l2src device=/dev/video1 ! kmssink
Known Issues
------------
1. When using 90 or 270 degree rotation control at capture resolutions
near the IC resizer limit of 1024x1024, and combined with planar
pixel formats (YUV420, YUV422p), frame capture will often fail with
no end-of-frame interrupts from the IDMAC channel. To work around
this, use lower resolution and/or packed formats (YUYV, RGB3, etc.)
when 90 or 270 rotations are needed.
File list
---------
drivers/staging/media/imx/
include/media/imx.h
include/linux/imx-media.h
References
----------
.. [#f1] http://www.nxp.com/assets/documents/data/en/reference-manuals/IMX6DQRM.pdf
.. [#f2] http://www.nxp.com/assets/documents/data/en/reference-manuals/IMX6SDLRM.pdf
Authors
-------
- Steve Longerbeam <steve_longerbeam@mentor.com>
- Philipp Zabel <kernel@pengutronix.de>
- Russell King <linux@armlinux.org.uk>
Copyright (C) 2012-2017 Mentor Graphics Inc.
3. 한국어 전문 번역
영어 원문의 문단 순서와 의미를 유지한 전체 번역입니다. 코드, 함수명, symbol과 URL은 원문 표기를 유지합니다.
i.MX Video Capture Driver
1-5이 문서는 GPL-2.0 라이선스를 따르는 i.MX video capture driver 안내입니다.
IPU capture unit 소개
6-74Freescale i.MX5/6에는 capture device와 display device 사이의 image frame 흐름을 처리하는 IPU(Image Processing Unit)가 있습니다.
Image capture를 위해 IPU에는 다음 내부 subunit이 포함됩니다.
- Image DMA Controller(IDMAC)
- Camera Serial Interface(CSI)
- Image Converter(IC)
- Sensor Multi-FIFO Controller(SMFC)
- Image Rotator(IRT)
- Video De-Interlacing or Combining Block(VDIC)
IDMAC은 memory와 image frame을 주고받는 DMA controller입니다. Video capture 및 display path마다 전용 DMA channel이 있습니다. 전송 중 vertical image flip, IRT와 연계한 8x8 block transfer, 같은 colorspace 안의 pixel component 순서 변경(예: UYVY에서 YUYV), packed·planar 변환도 수행할 수 있습니다.
IDMAC은 전송 중 짝수·홀수 line을 서로 엮는 단순 de-interlacing도 수행합니다. 이 방식에는 VDIC가 필요한 motion compensation이 없습니다.
CSI는 Parallel, BT.656/1120, MIPI CSI-2 bus를 통해 camera sensor와 직접 연결되는 backend capture unit입니다.
IC는 color-space conversion, downscale·upscale resize, horizontal flip, 90/270도 rotation을 처리합니다.
IC에는 동시에 conversion을 수행할 수 있는 세 개의 독립 task가 있습니다. Pre-process encoding, pre-process viewfinder, post-processing입니다. 각 task의 conversion은 downsizing section, main section(upsizing, flip, colorspace conversion, graphics plane combining), rotation section으로 나뉩니다.
IPU는 IC task operation을 time-share합니다. Time slice 단위는 downsizing section에서 8 pixel burst 하나, main processing section에서 image line 하나, rotation section에서 image frame 하나입니다.
SMFC는 독립 FIFO 네 개로 이루어지며, 각각 IDMAC channel 네 개를 통해 sensor에서 capture한 frame을 memory로 동시에 전송할 수 있습니다.
IRT는 90도와 270도 image rotation을 한 번에 8x8 pixel block 단위로 수행합니다. IDMAC이 vertical flip과 연계해 8x8 block transfer 및 block reorder를 처리하여 이 동작을 지원합니다.
VDIC는 interlaced video를 progressive로 변환하며 low, medium, high motion compensation mode를 지원합니다. De-interlace한 frame은 추가 변환을 위해 IC pre-process viewfinder task로 보낼 수 있습니다. VDIC의 Combiner는 alpha blending과 color keying으로 두 image plane을 결합합니다.
IPU 외부에도 i.MX video capture에 관여하는 unit 두 개가 있습니다.
- MIPI CSI-2 bus camera sensor용 MIPI CSI-2 Receiver. Synopsys DesignWare core입니다.
- 여러 sensor input 중 CSI로 보낼 입력을 고르는 video multiplexer 두 개
자세한 내용은 최신 i.MX5/6 reference manual `f1`, `f2`를 참조하십시오.
Driver 기능
75-104이 driver의 주요 기능은 다음과 같습니다.
- i.MX hardware video capture pipeline에 대응하는 다양한 pipeline을 Media Controller API로 구성
- Parallel, BT.565, MIPI CSI-2 interface 지원
- 독립 entity를 사용하는 여러 video capture interface pipeline으로 동시 독립 stream 구성
- IC task subdev를 통한 scaling, color-space conversion, horizontal·vertical flip, image rotation
- RGB, packed·planar YUV, partial planar YUV 등 다양한 pixel format
- VDIC subdev의 low·medium·high motion compensated de-interlacing. CSI에서 VDIC로 직접 보내는 pipeline을 제공하며 향후 output/mem2mem device의 memory buffer 입력도 지원할 예정
- ADV718x video decoder의 vertical sync 문제를 보정하는 Frame Interval Monitor(FIM)
SabreSD와 SabreAuto topology
105-133다음은 i.MX6Q SabreSD와 i.MX6Q SabreAuto의 media topology입니다. 다음 entity 설명에서 이 구조를 참조합니다.
i.MX5/6 topology는 IPUv3 CSI video multiplexer의 upstream에서는 platform마다 다르지만 그 지점 이후의 IPUv3 내부 topology는 공통입니다. SabreSD의 MIPI CSI-2 OV5640 sensor는 i.MX6 MIPI CSI-2 receiver가 필요합니다. SabreAuto는 parallel BT.656 bus의 ADV7180 decoder만 사용하므로 MIPI CSI-2 receiver가 없습니다.
원문 figure source는 `Documentation/admin-guide/media/imx6q-sabresd.dot`와 `Documentation/admin-guide/media/imx6q-sabreauto.dot`입니다.
OV5640의 네 virtual channel은 두 mux와 두 CSI direct input으로 분기되며, 각 CSI 이후에는 공통 IPUv3 capture processing path가 이어집니다.
ADV7180 parallel BT.656 입력은 ipu1_csi0_mux를 거쳐 IPU1 CSI0에 들어가며 이후 공통 IPUv3 분기를 사용합니다.
imx6-mipi-csi2 entity
134-161`imx6-mipi-csi2`는 MIPI CSI-2 receiver entity입니다. 보통 MIPI CSI-2 camera sensor에서 stream을 받는 sink pad 하나와, demux한 네 virtual channel output에 대응하는 source pad 네 개가 있습니다. 여러 source pad를 활성화해 여러 virtual channel을 독립적으로 stream할 수 있습니다.
이 entity는 Synopsys DesignWare MIPI CSI-2 core와 `CSI-2 to IPU gasket` 두 sub-block으로 구성됩니다. Gasket은 네 virtual channel stream을 demultiplex하고, 각 channel을 담는 네 parallel bus를 CSI 또는 video multiplexer로 보냅니다.
i.MX6 solo/dual-lite에서는 네 virtual channel bus가 두 video multiplexer로 연결되어 CSI0과 CSI1이 선택된 어떤 virtual channel도 받을 수 있습니다.
i.MX6 Quad에서는 virtual channel 0이 video mux 선택 후 IPU1-CSI0으로, channel 1과 2가 각각 IPU1-CSI1과 IPU2-CSI0으로 고정 연결됩니다. Channel 3은 video mux 선택 후 IPU2-CSI1로 갑니다.
ipuX_csiY_mux entity
162-182`ipuX_csiY_mux`는 video multiplexer입니다. Parallel interface camera sensor 또는 `imx6-mipi-csi2` entity의 MIPI CSI-2 virtual channel 중 하나를 고르는 sink pad가 둘 이상이며, 선택한 입력을 `ipuX_csiY` CSI entity로 보내는 source pad 하나가 있습니다.
i.MX6 solo/dual-lite에는 video mux entity 두 개가 있습니다. 하나는 IPU1-CSI0 앞에서 parallel sensor와 네 MIPI CSI-2 virtual channel, 총 다섯 sink pad 중 하나를 고릅니다. 다른 하나도 IPU1-CSI1 앞에서 같은 다섯 입력 중 하나를 고릅니다.
i.MX6 Quad에는 video mux entity 두 개가 있습니다. IPU1-CSI0 앞의 mux는 parallel sensor와 MIPI CSI-2 virtual channel 0 중 하나를, IPU2-CSI1 앞의 mux는 parallel sensor와 virtual channel 3 중 하나를 선택합니다.
ipuX_csiY entity
183-224`ipuX_csiY`는 CSI entity입니다. 앞에서 설명한 video mux 또는 MIPI CSI-2 virtual channel에서 입력을 받는 sink pad 하나가 있습니다.
Source pad는 두 개입니다. 첫 번째 direct source pad는 IDMAC memory buffer transfer 없이 hardware link로 `ipuX_vdic` 또는 `ipuX_ic_prp`에 직접 연결할 수 있습니다.
Direct source pad를 `ipuX_ic_prp`에 연결하면 CSI frame을 IC pre-processing task 하나 또는 둘 모두에서 처리할 수 있습니다. `ipuX_vdic`에 연결하면 VDIC가 high motion mode의 motion-compensated de-interlace를 수행합니다.
두 번째 source pad는 IC pre-processing을 우회해 SMFC와 IDMAC channel로 video frame을 memory buffer에 직접 보냅니다. 이 pad는 `ipuX_csiY capture` 형식의 이름을 가진 capture device node로 연결됩니다.
IDMAC source pad는 IDMAC channel을 사용하므로 같은 colorspace 안에서 pixel 순서를 바꿀 수 있습니다. CSI sink가 UYVY를 받더라도 capture device는 YUYV로 capture할 수 있고, packed YUV 입력을 YUV420 같은 planar YUV로 받을 수도 있습니다.
IDMAC channel은 motion compensation 없는 simple interweave도 지원합니다. Source pad field type이 sequential top-bottom 또는 bottom-top이고 capture interface를 interlaced로 요청하면 활성화됩니다. Capture interface는 source pad와 같은 field order를 강제합니다. 즉 seq-bt는 interlaced-bt, seq-tb는 interlaced-tb가 됩니다.
`ipuX_csiY`가 생성하는 event는 `imx_api_ipuX_csiY`를 참조하십시오.
ipuX_csiY crop과 /2 downscale
225-251CSI는 들어오는 raw sensor frame을 crop할 수 있습니다. `ipuX_csiY` sink pad에서 crop selection subdev API로 구현합니다.
Width와 height를 서로 독립적으로 정확히 절반으로 downscale할 수도 있습니다. 이 기능은 sink pad의 compose selection subdev API로 구현합니다.
`ipuX_csiY` source pad의 output rectangle은 sink pad의 compose rectangle과 같습니다. 따라서 source pad rectangle은 negotiate할 수 없고 sink pad의 compose selection API로 설정해야 합니다. /2 downscale이 없으면 source pad rectangle은 incoming rectangle과 같습니다.
다음 예는 `ipu1_csi0`이 `ipu1_csi0_mux`와 연결됐다고 가정하고, 1280x960 입력을 640x480으로 crop한 뒤 두 dimension을 /2 downscale하여 320x240으로 만듭니다.
media-ctl -V "'ipu1_csi0_mux':2[fmt:UYVY2X8/1280x960]"
media-ctl -V "'ipu1_csi0':0[crop:(0,0)/640x480]"
media-ctl -V "'ipu1_csi0':0[compose:(0,0)/320x240]"ipuX_csiY frame skipping
252-268CSI는 frame skipping으로 frame rate decimation을 지원합니다. Sink pad와 source pad의 frame interval을 설정해 decimation 비율을 지정하면 `ipuX_csiY`가 원하는 source pad frame rate에 가장 가까운 CSI frame skip 설정을 적용합니다.
다음 예는 60Hz 입력 frame rate를 IDMAC output source pad에서 절반으로 줄입니다.
media-ctl -V "'ipu1_csi0':0[fmt:UYVY2X8/640x480@1/60]"
media-ctl -V "'ipu1_csi0':2[fmt:UYVY2X8/640x480@1/30]"ipuX_csiY Frame Interval Monitor
269-273Frame Interval Monitor는 `imx_api_FIM`을 참조하십시오.
ipuX_vdic entity
274-292VDIC는 low, medium, high motion의 세 motion compensation mode로 de-interlacing을 수행합니다. Mode는 `V4L2_CID_DEINTERLACING_MODE` menu control로 지정합니다. Sink pad 두 개와 source pad 하나가 있습니다.
Direct sink pad는 `ipuX_csiY` direct pad에서 입력을 받으며 이 link에서는 high motion mode만 사용할 수 있습니다.
IDMAC sink pad를 활성화하면 output 또는 mem2mem device node에서 입력을 받습니다. Low와 medium mode는 memory buffer에서 frame을 받아야 하므로 이 pipeline에서 사용할 수 있습니다. 하지만 output 또는 mem2mem device가 아직 구현되지 않아 현재 이 sink pad에는 link가 없습니다.
Source pad는 IC pre-processing entity `ipuX_ic_prp`로 연결됩니다.
ipuX_ic_prp entity
293-309`ipuX_ic_prp`는 IC pre-processing entity이며 sink pad의 data를 source pad 하나 또는 둘로 보내는 router 역할을 합니다.
Sink pad 하나는 `ipuX_csiY` direct pad 또는 `ipuX_vdic`에서 입력을 받을 수 있습니다. Source pad 하나는 pre-process encode task `ipuX_ic_prpenc`, 다른 하나는 pre-process viewfinder task `ipuX_ic_prpvf`로 갑니다.
Sink가 `ipuX_csiY`에서 입력을 받으면 두 source pad를 동시에 활성화할 수 있습니다. `ipuX_vdic`에서 받으면 de-interlaced frame을 처리할 수 있는 viewfinder task 쪽 source pad만 활성화할 수 있습니다.
ipuX_ic_prpenc entity
310-325`ipuX_ic_prpenc`는 IC pre-processing encode entity입니다. `ipuX_ic_prp`에서 오는 sink pad 하나와 `ipuX_ic_prpenc capture` 형식의 capture device node로 가는 source pad 하나가 있습니다.
Color-space conversion, downscale·upscale resize, horizontal·vertical flip, 90/270도 rotation을 수행합니다. Flip과 rotation은 표준 V4L2 control로 제공합니다.
`ipuX_csiY` IDMAC source와 마찬가지로 motion compensation 없는 simple de-interlace와 pixel reordering도 지원합니다.
ipuX_ic_prpvf entity
326-346`ipuX_ic_prpvf`는 IC pre-processing viewfinder entity입니다. `ipuX_ic_prp`에서 오는 sink pad 하나와 `ipuX_ic_prpvf capture` 형식의 capture device node로 가는 source pad 하나가 있습니다.
동작은 `ipuX_ic_prpenc`와 같아서 동일한 resize·CSC operation과 flip·rotation control을 제공합니다. `ipuX_ic_prp`가 `ipuX_vdic`에서 입력을 받으면 de-interlaced frame을 받아 처리합니다.
Motion compensation 없는 simple interweave도 지원합니다. 다만 pipeline에 `ipuX_vdic`가 포함되면 VDIC가 이미 motion-compensated de-interlacing을 수행해 output field type이 progressive인 `none`만 가능하므로 `ipuX_ic_prpvf` interweave를 사용할 수 없습니다.
Capture pipeline의 sensor 구간
347-366다음 use case의 link에는 backend sensor, video mux, MIPI CSI-2 receiver가 포함되지 않습니다. 이 부분은 sensor interface가 parallel인지 MIPI CSI-2인지에 따라 달라집니다.
MIPI CSI-2 virtual channel에 따라 receiver가 CSI로 직접 가거나 video mux를 거칠 수 있습니다.
변환 없는 video capture
367-374Conversion 없이 sensor frame을 `ipuX_csiY` IDMAC source pad를 통해 camera device interface node로 직접 보냅니다.
IDMAC source pad 2의 direct memory capture 경로입니다.
IC direct conversion
375-383Preprocess encode entity를 사용해 CSI frame을 IC로 직접 보내며 최대 1024x1024 scaling, CSC, flip, image rotation을 수행합니다.
CSI direct pad에서 PRP encode capture까지의 경로입니다.
Motion-compensated de-interlace
384-393CSI direct pad의 frame을 VDIC로 보내 high motion mode의 motion-compensated de-interlacing, 최대 1024x1024 scaling, CSC, flip, rotation을 수행합니다.
VDIC와 PRP viewfinder를 거치는 capture 경로입니다.
사용 시 control 상속
394-408설정을 돕고 video device node에서만 control을 접근하는 기존 V4L2 application과 호환되도록 capture device interface는 현재 pipeline의 활성 entity에서 control을 상속합니다. 따라서 control은 subdev 또는 활성 capture device interface에서 접근할 수 있습니다.
예를 들어 FIM control은 `ipuX_csiY` subdev 또는 활성 capture device 양쪽에서 사용할 수 있습니다. 다음은 Sabre reference board별 사용 참고 사항입니다.
i.MX6Q SabreLite: OV5642와 OV5640
409-462이 platform에는 parallel camera interface의 OmniVision OV5642 module과 MIPI CSI-2 interface의 OV5640 module이 필요합니다. 두 module은 Boundary Devices에서 제공합니다.
- OV5642 module
https://boundarydevices.com/product/nit6x_5mp - MIPI OV5640 module
https://boundarydevices.com/product/nit6x_5mp_mipi
Camera module 하나만 있다면 device tree에서 다른 sensor node를 비활성화할 수 있습니다.
OV5642 module은 IPU1 CSI0으로 가는 i.MX 내부 video mux의 parallel bus input과 I²C bus 2에 연결됩니다.
MIPI CSI-2 OV5640 module은 i.MX 내부 MIPI CSI-2 receiver에 연결됩니다. Receiver의 vc0은 IPU1 CSI0 mux, vc1은 IPU1 CSI1, vc2는 IPU2 CSI0, vc3은 IPU2 CSI1 mux로 갑니다. OV5640도 SabreLite의 I²C bus 2를 사용하므로 OV5642와 같은 I²C slave address를 공유하면 안 됩니다.
다음 예는 두 sensor의 unprocessed capture pipeline을 구성합니다. OV5642는 `ipu1_csi0`, MIPI CSI-2 virtual channel 1(pad 2)의 OV5640은 `ipu1_csi1`로 보냅니다. 두 sensor는 640x480을 출력하며 OV5642는 YUYV2X8, OV5640은 UYVY2X8입니다.
# Setup links for OV5642
media-ctl -l "'ov5642 1-0042':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Setup links for OV5640
media-ctl -l "'ov5640 1-0040':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':2 -> 'ipu1_csi1':0[1]"
media-ctl -l "'ipu1_csi1':2 -> 'ipu1_csi1 capture':0[1]"
# Configure pads for OV5642 pipeline
media-ctl -V "'ov5642 1-0042':0 [fmt:YUYV2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:YUYV2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/640x480 field:none]"
# Configure pads for OV5640 pipeline
media-ctl -V "'ov5640 1-0040':0 [fmt:UYVY2X8/640x480 field:none]"
media-ctl -V "'imx6-mipi-csi2':2 [fmt:UYVY2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi1':2 [fmt:AYUV32/640x480 field:none]"두 sensor는 같은 I²C bus를 사용하지만 서로 다른 CSI capture node로 독립 streaming합니다.
이후 `ipu1_csi0 capture`와 `ipu1_csi1 capture` node에서 독립적으로 streaming을 시작할 수 있습니다. `v4l2-ctl`로 planar format을 포함한 지원 YUV pixelformat을 선택할 수 있습니다.
i.MX6Q SabreAuto: ADV7180
463-520i.MX6Q SabreAuto의 onboard ADV7180 SD decoder는 IPU1 CSI0으로 가는 내부 video mux의 parallel bus input에 연결됩니다.
다음은 NTSC 720x480 입력을 conversion과 motion compensation 없이 simple interweave로 capture하는 예입니다. ADV7180은 sequential 또는 alternating field를 출력해야 하며 NTSC sequential field type은 `seq-bt`입니다.
# Setup links
media-ctl -l "'adv7180 3-0021':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 3-0021':0 [fmt:UYVY2X8/720x480 field:seq-bt]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
# Configure "ipu1_csi0 capture" interface (assumed at /dev/video4)
v4l2-ctl -d4 --set-fmt-video=field=interlaced_bt이후 `/dev/video4`에서 streaming을 시작하고 `v4l2-ctl`로 지원 YUV pixelformat을 선택할 수 있습니다.
다음은 PAL 720x576 입력에 motion-compensated de-interlacing을 적용하는 예입니다. ADV7180은 PAL의 `seq-tb` 또는 `alternate` field를 출력해야 합니다.
# Setup links
media-ctl -l "'adv7180 3-0021':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 3-0021':0 [fmt:UYVY2X8/720x576 field:seq-tb]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x576]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x576]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]"
# Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2)
v4l2-ctl -d2 --set-fmt-video=field=noneADV7180 입력은 raw interweave capture 또는 VDIC motion-compensated de-interlace path로 구성할 수 있습니다.
이후 `/dev/video2`에서 streaming을 시작하고 `v4l2-ctl`로 지원 YUV pixelformat을 선택할 수 있습니다. 이 platform은 J42 connector의 Ain1에서 ADV7180으로 들어오는 Composite Video analog input을 받습니다.
i.MX6DL SabreAuto: ADV7180
521-578i.MX6DL SabreAuto에서도 onboard ADV7180 SD decoder가 IPU1 CSI0으로 가는 내부 video mux의 parallel bus input에 연결됩니다.
다음은 NTSC 720x480 입력을 simple interweave로 capture하는 예입니다. ADV7180 field type은 `seq-bt` 또는 `alternate`여야 합니다.
# Setup links
media-ctl -l "'adv7180 4-0021':0 -> 'ipu1_csi0_mux':4[1]"
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 4-0021':0 [fmt:UYVY2X8/720x480 field:seq-bt]"
media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
# Configure "ipu1_csi0 capture" interface (assumed at /dev/video0)
v4l2-ctl -d0 --set-fmt-video=field=interlaced_bt이후 `/dev/video0`에서 streaming을 시작하고 `v4l2-ctl`로 YUV pixelformat을 선택할 수 있습니다.
다음은 PAL 720x576 입력에 motion-compensated de-interlacing을 적용하는 예입니다. ADV7180 field type은 `seq-tb` 또는 `alternate`여야 합니다.
# Setup links
media-ctl -l "'adv7180 4-0021':0 -> 'ipu1_csi0_mux':4[1]"
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# Configure pads
media-ctl -V "'adv7180 4-0021':0 [fmt:UYVY2X8/720x576 field:seq-tb]"
media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x576]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x576]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]"
# Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2)
v4l2-ctl -d2 --set-fmt-video=field=none이후 `/dev/video2`에서 streaming을 시작할 수 있습니다. 이 platform도 J42 connector의 Ain1에서 ADV7180 Composite Video analog input을 받습니다.
i.MX6Q SabreSD: MIPI CSI-2 OV5640
579-681i.MX6Q SabreLite와 마찬가지로 SabreSD는 IPU1 CSI0의 parallel OV5642 module과 MIPI CSI-2 OV5640 module을 지원합니다. OV5642는 I²C bus 1, OV5640은 I²C bus 2에 연결됩니다.
SabreSD device tree에는 두 sensor의 OF graph가 모두 있지만 현재 문서 작성 시점에는 MIPI CSI-2 OV5640만 시험되어 OV5642 node는 비활성화되어 있습니다. OV5640은 MIPI connector J5에 연결되며 SabreSD용 NXP module part number는 H120729입니다.
다음 예는 MIPI CSI-2 virtual channel 0의 OV5640에서 unprocessed capture pipeline을 구성합니다.
# Setup links
media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# Configure pads
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY2X8/640x480]"
media-ctl -V "'imx6-mipi-csi2':1 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0_mux':0 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/640x480]"이후 `ipu1_csi0 capture` node에서 streaming을 시작하며 `v4l2-ctl`로 지원 pixelformat을 선택할 수 있습니다.
`ipu1_csi0 capture`에 해당하는 `/dev/video` node는 다음과 같이 확인합니다.
media-ctl -e "ipu1_csi0 capture"
/dev/video0이 경우 streaming element는 `/dev/video0`입니다. `v4l2-ctl`로 시작하는 명령은 다음과 같습니다.
v4l2-ctl --stream-mmap -d /dev/video0GStreamer로 streaming을 시작해 display로 보내는 명령은 다음과 같습니다.
gst-launch-1.0 v4l2src device=/dev/video0 ! kmssink다음 예는 MIPI CSI-2 virtual channel 0의 OV5640에서 direct conversion pipeline을 구성하고 IC output에서 colorspace conversion과 scaling을 수행합니다.
# Setup links
media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# Configure pads
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY2X8/640x480]"
media-ctl -V "'imx6-mipi-csi2':1 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/640x480]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:ARGB8888_1X32/800x600]"
# Set a format at the capture interface
v4l2-ctl -d /dev/video1 --set-fmt-video=pixelformat=RGB3이후 `ipu1_ic_prpenc capture` node에서 streaming을 시작할 수 있습니다.
`ipu1_ic_prpenc capture`에 해당하는 `/dev/video` node는 다음과 같이 확인합니다.
media-ctl -e "ipu1_ic_prpenc capture"
/dev/video1이 경우 streaming element는 `/dev/video1`입니다. `v4l2-ctl` 명령은 다음과 같습니다.
v4l2-ctl --stream-mmap -d /dev/video1GStreamer로 display에 보내는 명령은 다음과 같습니다.
gst-launch-1.0 v4l2src device=/dev/video1 ! kmssink알려진 문제
682-692IC resizer 한계인 1024x1024에 가까운 capture resolution에서 90도 또는 270도 rotation을 planar pixel format(YUV420, YUV422p)과 함께 사용하면 IDMAC channel의 end-of-frame interrupt 없이 capture가 자주 실패합니다.
90도 또는 270도 rotation이 필요하면 더 낮은 resolution 또는 packed format(YUYV, RGB3 등)을 사용하십시오.
관련 source file
693-699- `drivers/staging/media/imx/`
- `include/media/imx.h`
- `include/linux/imx-media.h`
Reference manual
700-706작성자와 저작권
707-714- Steve Longerbeam <steve_longerbeam@mentor.com>
- Philipp Zabel <kernel@pengutronix.de>
- Russell King <linux@armlinux.org.uk>
Copyright (C) 2012-2017 Mentor Graphics Inc.
IPUv3 capture architecture
imx.rst:1-714IDMAC, CSI, IC, SMFC, IRT, VDIC의 역할부터 Media Controller entity 연결, 세 가지 capture path, SabreLite·SabreAuto·SabreSD의 실제 `media-ctl` 구성까지 이어지는 구현 안내입니다.
Sensor 입력은 CSI 뒤에서 raw capture, IC conversion 또는 VDIC de-interlace 경로로 분기됩니다.