eXtended Image Sensor Controller (XISC) and MIPI CSI-2 pipeline
Introduction
This page describes how to use and understand the MIPI CSI-2 pipeline in SAM MPU products. The MIPI CSI-2 pipeline is a way of capturing image from a sensor using a serial interface, named CSI-2
(camera serial interface 2) which is standardized by the MIPI Alliance.
Previous MPU products had a simple parallel pipeline, in which there was a parallel sensor (like ov2640, ov7740) connected directly through I/O pins to the MPU to the Image Sensor Controller or Image Sensor Interface hardware block.
This page describes the pipeline for serial camera image acquisition in newer products, like SAMA7G5, which include complete hardware support for this.
Hardware blocks
Please refer to the product datasheet in the IMAGE SUBSYSTEM chapters, where each hardware block is described.Software pipeline
- The software pipeline is formed out of specific pipeline entities.
- Entities are defined as software constructs that perform a simple video task. It can be for example acquisition, conversion, scaling or outputting video.
- Each entity can have a variable number of pads. Pads are ways to send or receive data to or from an entity. Pads can be either sink pads, or source pads.
- A sink pad by definition is a pad where you can send data to (sink data to).
- A source pad by definition is a pad that produces (it's a source of) data.
- Pads can be linked together through a link
- You can imagine links like a flow of data. It is then logical to assume that a link is always between a source pad and a sink pad.
Entities may or may not correspond to real hardware blocks that process data. Some of them may be pure software implementations.
SAMA7G5 media graph
In the graph above, you can see the entities , pads, and links for the SAMA7G5 media pipeline.
This information can be retrieved using the command line tool media-ctl from the v4l2-utils package.
# media-ctl -p
Media controller API version 5.15.0
Media device information
------------------------
driver atmel_isc_mc
model microchip,sama7g5-isc
serial
bus info platform:microchip-sama7g5-xisc
hw revision 0x220
driver version 5.15.0
Device topology
- entity 1: atmel_isc_scaler (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:unknown/3264x2464
crop.bounds:(0,0)/3264x2464
crop:(0,0)/3264x2464]
<- "csi2dc":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:unknown/3264x2464]
-> "atmel_isc_base":0 [ENABLED,IMMUTABLE]
- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb]
<- "dw-csi.0":1 [ENABLED]
pad1: Source
[fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb]
-> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]
- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:SBGGR8_1X8/0x0]
<- "imx219 1-0010":0 [ENABLED]
pad1: Source
[fmt:SBGGR8_1X8/0x0]
-> "csi2dc":0 [ENABLED]
- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(8,8)/3280x2464
crop:(8,8)/3280x2464]
-> "dw-csi.0":0 [ENABLED]
- entity 24: atmel_isc_base (1 pad, 1 link)
type Node subtype V4L flags 1
device node name /dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]
- In this media-ctl output we can find out all the entities in the pipeline, pads and links between them, together with the current configured format for each pad. It is important to notice that each entity has a type.
- We can see that one of them is Sensor , the imx219 entity. This is the Sony imx219 sensor.
- We can see that one of them is V4L , the video4linux2 entity. This corresponds to the /dev/video0 classic v4l2 video node.
Querying subdevice capabilities
- Entities can have a subdevice associated.
- A subdevice is a /dev/v4l-subdev node that can be queried for information regarding the subdevice capabilities.
- The media-ctl -p command will show for each entity, if there is a v4l2 subdevice associated or not, and which index each subdevice has.
- Let's query our subdevices for capabilities.
Enum mbus codes
- One important capability that each subdevice must have, is a list of mbus (media bus) codes supported. The media bus code is a number that represents the format of the data that is supported by the subdevice.
- A media entity associated with a subdevice can have multiple pads, and each pad can have different media buses supported.
- For example one entity can be a format converter, then it's only logical that it's a different set of mbus codes is supported on the source pad compared to the sink pad. A list of mbus codes can be found here
- For example one entity can be a format converter, then it's only logical that it's a different set of mbus codes is supported on the source pad compared to the sink pad.
# v4l2-ctl -d /dev/v4l-subdev3 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
0x3014: MEDIA_BUS_FMT_SRGGB8_1X8
- We queried subdev3 to see the mbus codes supported by this pad (pad number 0).
- We see that there are two codes supported: 0x300f and 0x3014 . These correspond to SRGGB10_1x10 and SRGGB8_1X8 .
# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
0x3001: MEDIA_BUS_FMT_SBGGR8_1X8
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
0x3008: MEDIA_BUS_FMT_SBGGR12_1X12
0x3019: MEDIA_BUS_FMT_SBGGR14_1X14
0x301d: MEDIA_BUS_FMT_SBGGR16_1X16
0x1009: MEDIA_BUS_FMT_RGB666_1X18
0x1007: MEDIA_BUS_FMT_RGB565_2X8_BE
0x1008: MEDIA_BUS_FMT_RGB565_2X8_LE
0x1003: MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE
0x1004: MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE
0x1001: MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE
0x1002: MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE
0x100c: MEDIA_BUS_FMT_RGB888_2X12_LE
0x100b: MEDIA_BUS_FMT_RGB888_2X12_BE
0x100a: MEDIA_BUS_FMT_RGB888_1X24
0x2010: MEDIA_BUS_FMT_VYUY8_1X16
0x2019: MEDIA_BUS_FMT_VYUY10_2X10
0x2001: MEDIA_BUS_FMT_Y8_1X8
0x200a: MEDIA_BUS_FMT_Y10_1X10
# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 1
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=1)
0x3001: MEDIA_BUS_FMT_SBGGR8_1X8
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
0x3008: MEDIA_BUS_FMT_SBGGR12_1X12
0x3019: MEDIA_BUS_FMT_SBGGR14_1X14
0x301d: MEDIA_BUS_FMT_SBGGR16_1X16
0x1009: MEDIA_BUS_FMT_RGB666_1X18
0x1007: MEDIA_BUS_FMT_RGB565_2X8_BE
0x1008: MEDIA_BUS_FMT_RGB565_2X8_LE
0x1003: MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE
0x1004: MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE
0x1001: MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE
0x1002: MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE
0x100c: MEDIA_BUS_FMT_RGB888_2X12_LE
0x100b: MEDIA_BUS_FMT_RGB888_2X12_BE
0x100a: MEDIA_BUS_FMT_RGB888_1X24
0x2010: MEDIA_BUS_FMT_VYUY8_1X16
0x2019: MEDIA_BUS_FMT_VYUY10_2X10
0x2001: MEDIA_BUS_FMT_Y8_1X8
0x200a: MEDIA_BUS_FMT_Y10_1X10
- We queried both pads of the subdev2 and obtained the possible mbus codes for this subdevice on each of its pads.
- It's good to see that the source pad of the subdevice3 has codes that are also found in the list of codes for the sink pad of the subdevice2.
- This means that most likely a common format can be found for the link between them.
- It's good to see that the source pad of the subdevice3 has codes that are also found in the list of codes for the sink pad of the subdevice2.
# v4l2-ctl -d /dev/v4l-subdev1 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300a: MEDIA_BUS_FMT_SGRBG10_1X10
0x300e: MEDIA_BUS_FMT_SGBRG10_1X10
# v4l2-ctl -d /dev/v4l-subdev1 --list-subdev-mbus-codes 1
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=1)
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300a: MEDIA_BUS_FMT_SGRBG10_1X10
0x300e: MEDIA_BUS_FMT_SGBRG10_1X10
#
- We queried both pads of the subdev1 and obtained the possible mbus codes for this subdevice on each of its pads.
# v4l2-ctl -d /dev/v4l-subdev0 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300e: MEDIA_BUS_FMT_SGBRG10_1X10
0x300a: MEDIA_BUS_FMT_SGRBG10_1X10
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
# v4l2-ctl -d /dev/v4l-subdev0 --list-subdev-mbus-codes 1
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=1)
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300e: MEDIA_BUS_FMT_SGBRG10_1X10
0x300a: MEDIA_BUS_FMT_SGRBG10_1X10
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
#
- We queried both pads of the subdev0 and obtained the possible mbus codes for this subdevice on each of its pads.
Enum frame sizes
- One other important capability that a subdevice can have, is a list of supported frame sizes, or in other words, resolutions. Resolutions can be a fixed discrete size, or a continuous size, in which a minimum and a maximum of a frame are specified.
- A media entity associated with a subdevice can have multiple pads, and each pad can have different frame sizes supported.
- For example one entity can be a scaler, then it's only logical that it's a different set of frame sizes is supported on the source pad compared to the sink pad.
# v4l2-ctl -d /dev/v4l-subdev3 --list-subdev-framesizes pad=0,code=0x300f
ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0)
Size Range: 3280x2464 - 3280x2464
Size Range: 1920x1080 - 1920x1080
Size Range: 1640x1232 - 1640x1232
Size Range: 640x480 - 640x480
- We queried the subdevice3 to display the framesizes for the given pad, and given mbus code.
- According to one mbus code or another, a subdevice can have different possible frame sizes.
# v4l2-ctl -d /dev/v4l-subdev3 --list-subdev-framesizes pad=0,code=0x3001 ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0) #
- We queried the subdevice3 to display the framesizes for the given pad, and an unsupported mbus code.
- No frame size was returned.
Some subdevices do not support querying for frame size. This means that the frame size is unknown. When configuring the subdevice, one has to be careful if the configuration with a given frame size was successful or not.
Query the subdevice2 for frame size:
# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-framesizes pad=0
ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0)
Size Range: 16x16 - 4000x3000
#
- We queried the subdevice2 to display the framesizes for the given pad.
- We obtained a range of frame sizes from 16x16 to 4000x3000. This means any frame size that fits is supported.
Get current subdevice format (per pad)
- It is important to distinguish between mbus code and format. A format includes an mbus code, but a format also means a frame size, colorspace, and other type of information.
- We can query the subdevice to return the current configured format per pad.
# v4l2-ctl -d /dev/v4l-subdev3 --get-subdev-fmt 0
ioctl: VIDIOC_SUBDEV_G_FMT (pad=0)
Width/Height : 3280/2464
Mediabus Code : 0x300f (MEDIA_BUS_FMT_SRGGB10_1X10)
Field : None
Colorspace : sRGB
Transfer Function : sRGB
YCbCr/HSV Encoding: ITU-R 601
Quantization : Full Range
#
- We notice the pad number, the mbus code and the frame size that is currently configured on the subdevice. A summary of this format can be seen in media-ctl -p output
# v4l2-ctl -d /dev/v4l-subdev2 --get-subdev-fmt 0
ioctl: VIDIOC_SUBDEV_G_FMT (pad=0)
Width/Height : 0/0
Mediabus Code : 0x3001 (MEDIA_BUS_FMT_SBGGR8_1X8)
Field : Any
Colorspace : Default
Transfer Function : Default (maps to Rec. 709)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
# v4l2-ctl -d /dev/v4l-subdev2 --get-subdev-fmt 1
ioctl: VIDIOC_SUBDEV_G_FMT (pad=1)
Width/Height : 0/0
Mediabus Code : 0x3001 (MEDIA_BUS_FMT_SBGGR8_1X8)
Field : Any
Colorspace : Default
Transfer Function : Default (maps to Rec. 709)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
#
- We notice that the frame size is not set correctly. This has to be set for this subdevice such that streaming works.
Configuring subdevice format
- Once we know how to query the subdevices for different settings, we need to know also how to change these according to what we would like to do.
media-ctl -d /dev/media0 --set-v4l2 '"imx219 1-0010":0[fmt:SRGGB10_1X10/3280x2464]' media-ctl -d /dev/media0 --set-v4l2 '"dw-csi.0":0[fmt:SRGGB10_1X10/3280x2464]' media-ctl -d /dev/media0 --set-v4l2 '"csi2dc":0[fmt:SRGGB10_1X10/3280x2464]' media-ctl -d /dev/media0 --set-v4l2 '"atmel_isc_scaler":0[fmt:SRGGB10_1X10/3280x2464]'The above sequence of commands will configure the 4 given entities with the format specified.
The format string and specification is identical with the one printed by media-ctl -p
We can observe the difference in media-ctl output:
# media-ctl -p
Media controller API version 5.15.0
Media device information
------------------------
driver atmel_isc_mc
model microchip,sama7g5-isc
serial
bus info platform:microchip-sama7g5-xisc
hw revision 0x220
driver version 5.15.0
Device topology
- entity 1: atmel_isc_scaler (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464
crop.bounds:(0,0)/3264x2464
crop:(0,0)/3264x2464]
<- "csi2dc":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:SRGGB10_1X10/3264x2464]
-> "atmel_isc_base":0 [ENABLED,IMMUTABLE]
- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
<- "dw-csi.0":1 [ENABLED]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
-> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]
- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464]
<- "imx219 1-0010":0 [ENABLED]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464]
-> "csi2dc":0 [ENABLED]
- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(8,8)/3280x2464
crop:(8,8)/3280x2464]
-> "dw-csi.0":0 [ENABLED]
- entity 24: atmel_isc_base (1 pad, 1 link)
type Node subtype V4L flags 1
device node name /dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]
All links now have frame size and mbus code configured.
- We can configure the pipeline with a different frame size and the same mbus code. Here is an example:
media-ctl -d /dev/media0 --set-v4l2 '"imx219 1-0010":0[fmt:SRGGB10_1X10/1920x1080]' media-ctl -d /dev/media0 --set-v4l2 '"dw-csi.0":0[fmt:SRGGB10_1X10/1920x1080]' media-ctl -d /dev/media0 --set-v4l2 '"csi2dc":0[fmt:SRGGB10_1X10/1920x1080]' media-ctl -d /dev/media0 --set-v4l2 '"atmel_isc_scaler":0[fmt:SRGGB10_1X10/1920x1080]'
- And the corresponding media-ctl -p output:
# media-ctl -p
Media controller API version 5.15.0
Media device information
------------------------
driver atmel_isc_mc
model microchip,sama7g5-isc
serial
bus info platform:microchip-sama7g5-xisc
hw revision 0x220
driver version 5.15.0
Device topology
- entity 1: atmel_isc_scaler (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:SRGGB10_1X10/1920x1080
crop.bounds:(0,0)/3264x2464
crop:(0,0)/3264x2464]
<- "csi2dc":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:SRGGB10_1X10/1920x1080]
-> "atmel_isc_base":0 [ENABLED,IMMUTABLE]
- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
<- "dw-csi.0":1 [ENABLED]
pad1: Source
[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
-> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]
- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:SRGGB10_1X10/1920x1080]
<- "imx219 1-0010":0 [ENABLED]
pad1: Source
[fmt:SRGGB10_1X10/1920x1080]
-> "csi2dc":0 [ENABLED]
- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(8,8)/3280x2464
crop:(688,700)/1920x1080]
-> "dw-csi.0":0 [ENABLED]
- entity 24: atmel_isc_base (1 pad, 1 link)
type Node subtype V4L flags 1
device node name /dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]
Configuring top video driver
- In v4l2 , the top video driver is the driver that registers the /dev/video node.
- This driver also registers the /dev/media node, which is the main device of the associated media controller
- Video device and media device cannot be intertwined, they perform different tasks.
- You could consider the media device and video device like two devices that complete each other for full functionality access and tweaking for the video capture pipeline
- Video node is used to configure the video device. This is usually the top driver that performs the video capture and includes the way to access the video frame.
- Media node is the node that allows to access all entities, pads and links, and informs the userspace about the topology, status, and allows topology configuration and reconfiguration Device tree is still needed to describe the hardware on the SoC / board. Device tree is a description of the hardware. The drivers themselves may or may not register media entities for their hardware devices, or, could register entities that do not have any specific hardware device associated (pure software entities), or a piece of hardware could translate into multiple entities. It's the drivers' choice what to expose or not expose to userspace. Thus, the media topology is a view from userspace of the available video pipeline.
- The top video driver, /dev/video can be queried for information and configured using the v4l2-ctl tool.
Querying the top video driver
- We can query the top video driver using v4l2-ctl
- Let's see what formats the top video driver can output to user space:
# v4l2-ctl -d /dev/video0 --list-formats
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'BG10' (10-bit Bayer BGBG/GRGR)
[1]: 'GB10' (10-bit Bayer GBGB/RGRG)
[2]: 'BA10' (10-bit Bayer GRGR/BGBG)
[3]: 'RG10' (10-bit Bayer RGRG/GBGB)
[4]: 'AR12' (16-bit ARGB 4-4-4-4)
[5]: 'AR15' (16-bit ARGB 1-5-5-5)
[6]: 'RGBP' (16-bit RGB 5-6-5)
[7]: 'AR24' (32-bit BGRA 8-8-8-8)
[8]: 'XR24' (32-bit BGRX 8-8-8-8)
[9]: 'YU12' (Planar YUV 4:2:0)
[10]: 'UYVY' (UYVY 4:2:2)
[11]: 'VYUY' (VYUY 4:2:2)
[12]: 'YUYV' (YUYV 4:2:2)
[13]: '422P' (Planar YUV 4:2:2)
[14]: 'GREY' (8-bit Greyscale)
[15]: 'Y10 ' (10-bit Greyscale)
[16]: 'Y16 ' (16-bit Greyscale)
#
- We can query the video node also for possible resolutions:
# v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'BG10' (10-bit Bayer BGBG/GRGR)
Size: Continuous 16x16 - 3264x2464
[1]: 'GB10' (10-bit Bayer GBGB/RGRG)
Size: Continuous 16x16 - 3264x2464
[2]: 'BA10' (10-bit Bayer GRGR/BGBG)
Size: Continuous 16x16 - 3264x2464
[3]: 'RG10' (10-bit Bayer RGRG/GBGB)
Size: Continuous 16x16 - 3264x2464
[4]: 'AR12' (16-bit ARGB 4-4-4-4)
Size: Continuous 16x16 - 3264x2464
[5]: 'AR15' (16-bit ARGB 1-5-5-5)
Size: Continuous 16x16 - 3264x2464
[6]: 'RGBP' (16-bit RGB 5-6-5)
Size: Continuous 16x16 - 3264x2464
[7]: 'AR24' (32-bit BGRA 8-8-8-8)
Size: Continuous 16x16 - 3264x2464
[8]: 'XR24' (32-bit BGRX 8-8-8-8)
Size: Continuous 16x16 - 3264x2464
[9]: 'YU12' (Planar YUV 4:2:0)
Size: Continuous 16x16 - 3264x2464
[10]: 'UYVY' (UYVY 4:2:2)
Size: Continuous 16x16 - 3264x2464
[11]: 'VYUY' (VYUY 4:2:2)
Size: Continuous 16x16 - 3264x2464
[12]: 'YUYV' (YUYV 4:2:2)
Size: Continuous 16x16 - 3264x2464
[13]: '422P' (Planar YUV 4:2:2)
Size: Continuous 16x16 - 3264x2464
[14]: 'GREY' (8-bit Greyscale)
Size: Continuous 16x16 - 3264x2464
[15]: 'Y10 ' (10-bit Greyscale)
Size: Continuous 16x16 - 3264x2464
[16]: 'Y16 ' (16-bit Greyscale)
Size: Continuous 16x16 - 3264x2464
- We see that the video device can output any resolution from 16x16 up to 3264x2464 .
- This means that any frame size that fits, it's fine from the video node perspective.
- We will see below how to get the current frame size. The actual frame size is usually given by the sensor. The top video device cannot create frames, frames are created by sensors. However, entities in the pipeline can scale, adjust, crop and compose the frame into a new frame at a different frame size.
- How to see the complete format that is configured now ?
# v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
Width/Height : 640/480
Pixel Format : 'BG10' (10-bit Bayer BGBG/GRGR)
Field : None
Bytes per Line : 1280
Size Image : 614400
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
#
- This is the format that the video node will output to the userspace.
- We notice the pixel format (FOURCC codification), resolution, bytes for each line and the total image size.
Configuring the top video driver
- How to change the video format ?
# v4l2-ctl -d /dev/video0 --set-fmt-video width=3464,height=2464,pixelformat=RGBP
# v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
Width/Height : 3264/2464
Pixel Format : 'RGBP' (16-bit RGB 5-6-5)
Field : None
Bytes per Line : 6528
Size Image : 16084992
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
#
- Format was changed as requested A list of possible pixelformat FOURCC values can be obtained from the output of the command v4l2-ctl --list-formats
Capture a frame
- Once the pipeline is configured correctly, we can use different tools to capture a frame from the video device.
- If one of the elements in the pipeline is not configured correctly, we will have errors.
- The top video driver will assert the situation at the start streaming step.
- If the subdevice is not configured to a compatible format, capture will fail.
Using v4l2-ctl to capture a frame
# v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=RGBP_3264_2464.raw --stream-count=1 < # ls -la RGBP_3264_2464.raw -rw-r--r-- 1 root root 16084992 Jan 1 00:00 RGBP_3264_2464.raw #
- The raw frame has been captured with v4l2-ctl .
Converting raw frame to png
- v4l2-ctl only captures raw frames
- We can use another tool like ffmpeg to convert the raw frame into a usual format photo.
- For example, from BAYER 8 bit RGGB raw format :
ffmpeg -f rawvideo -s 1920x1080 -pix_fmt bayer_rggb8 -i frame.raw result.png
To see ffmpg supported pixel formats, try this:
# ffmpeg -pix_fmts
Using fswebcam to capture a frame
- fswebcam can still be used to take a photo, once the pipeline is correctly configured
fswebcam -p RGB565 -r 3280x2464 -S 20 bigRGB565.png fswebcam -p YUYV -r 3280x2464 -S 20 bigYUYV.png fswebcam -p UYVY -r 3280x2464 -S 20 bigUYVY.png fswebcam -p VYUY -r 3280x2464 -S 20 bigVYUY.png fswebcam -p ABGR32 -r 3280x2464 -S 20 bigABGR32.png fswebcam -p Y16 -r 3280x2464 -S 20 bigY16.png fswebcam -p GREY -r 3280x2464 -S 20 bigGREY.png fswebcam -p YUV420P -r 3280x2464 -S 20 bigYUV420P.png
libcamera
- libcamera is a userspace library that configures a media pipeline and allows higher level application to stop hassling with the media controller configuration
- libcamera is actually a library, so other applications have to link with libcamera in order to use it
- basically libcamera is a layer between /dev/video and /dev/media and the user
- libcamera also provides few test apps that are linked with libcamera, and can be used to validate the functionality
libcamera cam app
- The cam application is a basic libcamera app that can capture a frame.
- Using cam application to test if the camera is available:
# cam -l [3:09:29.602118600] [272] INFO IPAManager ipa_manager.cpp:138 libcamera is not installed. Adding '//src/ipa' to the IPA search path [3:09:29.602612000] [272] WARN IPAManager ipa_manager.cpp:149 No IPA found in '/usr/lib/libcamera' [3:09:29.602773400] [272] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+58645-2021.08-dirty (2021-10-08T16:10:25+03:00) [3:09:29.610875800] [274] WARN CameraSensor camera_sensor.cpp:197 'imx219 1-0010': Recommended V4L2 control 0x009a0922 not supported [3:09:29.611025000] [274] WARN CameraSensor camera_sensor.cpp:249 'imx219 1-0010': The sensor kernel driver needs to be fixed [3:09:29.611092200] [274] WARN CameraSensor camera_sensor.cpp:251 'imx219 1-0010': See Documentation/sensor_driver_requirements.rst in the libcamera sources for more information [3:09:29.617962400] [274] WARN CameraSensor camera_sensor.cpp:414 'imx219 1-0010': Failed to retrieve the camera location [3:09:29.620663000] [274] WARN V4L2 v4l2_pixelformat.cpp:283 Unsupported V4L2 pixel format Y10 [3:09:29.620766200] [274] WARN V4L2 v4l2_pixelformat.cpp:283 Unsupported V4L2 pixel format Y16 [3:09:29.620828600] [274] WARN V4L2 v4l2_pixelformat.cpp:283 Unsupported V4L2 pixel format AR12 [3:09:29.620886800] [274] WARN V4L2 v4l2_pixelformat.cpp:283 Unsupported V4L2 pixel format AR15 Available cameras: 1: 'imx219' (/base/soc/flexcom@e2818000/i2c@600/camera@10) #
- cam app has identified the camera and displays it to the user with the name of the video sensor (imx219 in our case)
- camera number 1 is imx219
|
|||||||||||||||
r3 - 25 Oct 2021 - 15:52:24 - EugenHristev
Linux4SAM
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
Microchip® and others, are registered trademarks or trademarks of Microchip Technology Inc. and its subsidiaries. 
Arm® and others are registered trademarks or trademarks of Arm Limited (or its affiliates). Other terms and product names may be trademarks of others.
Ideas, requests, contributions ? Connect to LinksToCommunities page.