RAW8 capture settings for nvcamerasrc

Hi,
I set a sensor to provide RAW8 and I’m able to get the frame properly using v4l2src. Since I need to perform the debayering using the ISP, I want to use nvcamerasrc for capturing. Checking the device-tree settings for another sensor that I found on the kernel that provided RAW12, I concluded that in order to correctly set the mode on nvcamerasrc I needed to set the following values on the particular mode on the sensor that has RAW8:

mode0 {
+				mode_type = "bayer";
+				pixel_phase = "rggb";
+				dynamic_pixel_bit_depth = "8";
+				csi_pixel_bit_depth = "8";
+				pixel_t = "bayer_rggb8";
+

...
}

When I launch a gstreamer pipeline using that mode, I’m able to see that nvcamerasrc recognizes properly that the sensor is going to provide RAW8

1640 x 1232 FR=30.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=8 DynPixelBitDepth=8

However, I’m only seeing a black frame at the output from nvcamerasrc. In the past as stated before, I have been able to capture and debayer RAW12, and it seems like the setting above are the ones that configures the mode properly. I got the following questions:

1- Does Nvcamerasrc supports debayering RAW8?
2- Am I missing some extra setting for enabling debayering of RAW8 in Nvcamerasrc ?

Kind regards,
JJ

hello josejich,

your modification to change the device tree bit depth may not the correct method to configure as a RAW8 output.
since sensor output format was determine by the sensor side; device tree make the correct settings to correspond that.

may i know why you need RAW8 format?
thanks

Hi JerryChang,

It seems like nvcamerasrc it was using more resources that I would like to when the sensor provided RAW10 and nvcamerasrc was debayering to I420, so I would like to test the performance when the camera provides RAW8 and nvcamerasrc performs the debayering to I420.

I checked that the sensor was providing RAW8 properly using the v4l2src path, so the sensor settings seems to be correct. I’m wondering if the settings I set on the device-tree are the correct to get nvcamerasrc to perform the RAW8 to I420 conversion.

Thanks again!,
JJ

hello josejich,

  1. i understand you would like to evaluate the performance between difference input format.
    however, please double confirm all the settings were correct.
    generally speaking, sensor output format would be determine by the its register settings.
    for example.
    you could also check our reference driver, imx185
    please check the register settings of 0x333e/0x333f, which 0x0A0A means RAW10 and 0x0C0C means RAW12.
    please contact your sensor vendor or check the sensor spec to make sure sensor output as your expect.

  2. moreover, may i know what’s your performance evaluate criteria? clock rate?
    what’s the tool you’re using to monitor the performance?
    thanks

Hi JerryChang,

Thanks for your answer:

  1. We already have the sensor putting 8 bits on the output and we were able to verify this capturing with v4l2src.

  2. The customer is running another algorithm in the background with the images, when using more than 8 bits per pixel we don’t get 30fps, when using 8 bits per pixel we get 30fps. So this sounds like a memory bandwidth issue. We would like to use the ISP too so we want to use nvcamerasrc with the 8bits support, could you confirm if this is possible with nvcamerasrc? or is the binary or ISP calibration file wired to other values and not 8 bits mode?

Thanks,
-David

hello DavidSoto,

our reference camera driver is RAW10 output but we’re able to get 30-fps result.

could you help analysis with v4l2-ctl

v4l2-ctl --list-formats-ext

and also argus_camera

./argus_camera -i --sensormode=0 --outputsize=2592x1944 --framerate=30 --kpi

Hi JerryChang,

Thanks for your response, sorry, maybe I confused you with my previous answer. When using RAW10 or RAW12 and only nvcamerasrc with the 5 or 6 cameras it is possible to get 30fps. Something like:
nvcamerasrc ! fakesink in 5 or 6 cameras

So nvcamerasrc is fine. But when adding the algorithm of the customer, it is not possible to reach 30fps, I know that this indicates that maybe the customer algorithm is not efficient enough to keep the framerate up, something like:

nvcamerasrc ! customAlg ! fakesink in 5 or 6 cameras

However, as part of the optimization process the customer wants to reduce the amount of data per pixel, and instead of capturing RAW10 or RAW12 they want to capture RAW8. For this, we added the support in the camera and verified that something like this is working at 30fps:

v4l2src ! customAlg ! fakesink in 5 or 6 cameras

where now customAlg is also doing the debayering in CUDA and some other functions, all in RAW8. But I keep thinking that if we are able to get nvcamerasrc working with RAW8, it will be even more efficient because the ISP is pretty powerful.

Is there any limitation on nvcamerasrc causing to not work when the input sensor gives RAW8?

Thanks,
-David

Hi,

I also set a sensor to provide Bayer RAW8 and I’m able to get the frame properly using v4l2src. But when I have tried to use nvcamerasrc for capturing, I am getting a black frame.

I am using the same device tree ISP configuration described previously.

In the past I have configured Bayer RAW12 for another sensor and for this case, the configuration applied was:

mode0 {
. . .
                    dynamic_pixel_bit_depth = "12";
                    csi_pixel_bit_depth = "12";
                    pixel_t = "bayer_grbg12";
. . .

This change was enough to configure the ISP properly to process RAW12.

Do you know if I need to apply some extra change to be able to process Bayer RAW8 format?

Regards,
-Manuel

https://developer.ridgerun.com/wiki/index.php?title=RidgeRun_Professional_Services

hello manuel.leiva,

thanks for bringing this issue up.
could you run the camera daemon manually and share the background message to us.
execute below command at the terminal before launching camera for capture,

sudo /usr/sbin/nvcamera-daemon

also, please share both capture commands with v4l2src and nvcamerasrc
thanks

Hello JerryChang,

Thanks for your response, These are the messages displayed.

nvidia@tegra-ubuntu:~$ sudo /usr/sbin/nvcamera-daemon






NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
PCLHW_DTParser
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 447: Error: Invalid isp config attribute element: "mwbCCT.sunlight=6100"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 450: Error: Invalid isp config attribute element: "mwbCCT.tungsten=2950"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 453: Error: Invalid isp config attribute element: "mwbCCT.flash=7300"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 454: Error: Invalid isp config attribute element: "mwbCCT.horizon=3000"
LSC: LSC surface is not based on full res!
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 447: Error: Invalid isp config attribute element: "mwbCCT.sunlight=6100"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 450: Error: Invalid isp config attribute element: "mwbCCT.tungsten=2950"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 453: Error: Invalid isp config attribute element: "mwbCCT.flash=7300"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 454: Error: Invalid isp config attribute element: "mwbCCT.horizon=3000"
LSC: LSC surface is not based on full res!
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 447: Error: Invalid isp config attribute element: "mwbCCT.sunlight=6100"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 450: Error: Invalid isp config attribute element: "mwbCCT.tungsten=2950"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 453: Error: Invalid isp config attribute element: "mwbCCT.flash=7300"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 454: Error: Invalid isp config attribute element: "mwbCCT.horizon=3000"
LSC: LSC surface is not based on full res!
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 447: Error: Invalid isp config attribute element: "mwbCCT.sunlight=6100"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 450: Error: Invalid isp config attribute element: "mwbCCT.tungsten=2950"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 453: Error: Invalid isp config attribute element: "mwbCCT.flash=7300"
NvCameraIspConfigFileLoad: Config file "camera_overrides.isp" Line 454: Error: Invalid isp config attribute element: "mwbCCT.horizon=3000"
LSC: LSC surface is not based on full res!
PowerServiceCore:handleRequests: timePassed = 1594

This is the command used to capture from nvcamerasrc

DISPLAY=:0 gst-launch-1.0 nvcamerasrc sensor-id=0 auto-exposure=off fpsRange="30 30" ! 'video/x-raw(memory:NVMM),width=(int)4656,height=(int)3496,format=(string)I420,framerate=(fraction)21/1' ! nvvidconv ! 'video/x-raw, width=640, height=480' ! xvimagesink

This is the command used to capture with v4l2src

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4656,height=3496, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to image_8bit.raw

This is the information provided by v4l2-ctl related to the video format:

nvidia@tegra-ubuntu:~$ v4l2-ctl --all | grep "Format Video" -A 10
Format Video Capture:
	Width/Height      : 4656/3496
	Pixel Format      : 'RGGB'
	Field             : None
	Bytes per Line    : 4864
	Size Image        : 17004544
	Colorspace        : sRGB
	Transfer Function : Default
	YCbCr Encoding    : Default
	Quantization      : Default
	Flags             :

Thanks.

hello manuel.leiva,

thanks for sharing the information.

since you had dump the raw8 files with v4l2-ctl.
could you also analysis the raw files with 3rdparty tools, such as 7yuv.
let’s confirmed the raw file is correct by converting to an image or reading its hex values.
thanks

JerryChang,

I have tried to convert the image captured with 7yuv, but at least with the evaluation version, bayer 8bit/pixel is not supported (only was supported 16bit/pixel) , Then I have tried with another tool called Bayer2rgb (see RAW Image Compression | Color Space Conversion | RidgeRun)

Using this command:

./bayer2rgb --input=image_8bit.raw --output=pic.tiff --width=4864 --height=3496 --bpp=8 --first=RGGB --method=BILINEAR --tiff

I got this image:
https://drive.google.com/file/d/1KvkWILva6WQJB_cv4IptTqbbjNIoloX0/view?usp=sharing

The image has a black area on the right, since v4l2-core is assigning ‘Bytes per Line: 4864’ but the image width is 4656.

Thanks.

This is the image:

hello manuel.leiva/ DavidSoto,

thanks for the confirmation,
we’re going to investigate this internally.

beside gstreamer pipeline, did you also got black preview frames with libargus?

moreover,
since there are different drivers between TX1 and TX2.
do you have any chances to confirm the status for TX2.
thanks

Hi JerryChang,

I have tried to capture using libargus (TX1) but I am getting the same black frame.

Also I was checking the “Sensor Driver Programming Guide” and I have found that there is not a definition for bayer 8bpp.

pixel_t: Specifies the readout pixel pattern of the sensor.
The following examples shows the values for Bayer sensors with the “bggr” pixel pattern:
• 10 bit: bayer_bggr
• 12 bit: bayer_bggr12
• 14 bit: bayer_bggr14

Based on this documentation I think that there is not support for Bayer8.

Thanks.

hi all,

we would like to investigate this internally.
could you share the raw files dumped by RAW8 sensors.
for example, please dump raw files as below command.

v4l2-ctl -d /dev/camera/video0 --set-fmt-video=width=1920,height=1080,maxframerate=30,pixelformat=RG8 --stream-mmap --stream-count=300 --stream-to=/test.raw8

thanks!

Hi JerryChang,

You can download example images from these links:

Image: one frame: https://drive.google.com/file/d/1yXwrHO14njxm3nEzQ3Vq6MZ99-1kLn9J/view?usp=sharing
Size: 17M
Command:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4656,height=3496, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to image_8bit.raw

Image 10 frames: image_8bit.raw - Google Drive
size: 163M
Command:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4656,height=3496,pixelformat=RG8 --stream-mmap --stream-count=10 --stream-to=image_8bit_10_frames.raw

Thanks.

I also tried switching my sensor to RAW8 and just get black from the output. Capturing a frame with Yavta shows data, however. Interested to hear how this develops.

Any updates on the Bayer8 front?

I was also wondering if we need to change settings in tegra-camera-platform. Specifically the min_bits_per_pixel and vi_peak_byte_per_pixel, and isp_peak_byte_per_pixel.

Here’s the dmesg output when I try to use bayer_rggb8:

[ 142.819129] daxc02 1-0010: daxc02_power_on
[ 143.065486] extract_pixel_format: Need to extend formatbayer_rggb8
[ 143.071779] daxc02 1-0010: Unsupported pixel format
[ 143.076743] daxc02 1-0010: Failed to read image properties
[ 143.084002] daxc02 1-0010: daxc02_power_off

and the corresponding function in sensor_common.c:

static int extract_pixel_format(
	const char *pixel_t, u32 *format)
{
	size_t size = strnlen(pixel_t, OF_MAX_STR_LEN);

	if (strncmp(pixel_t, "bayer_bggr10", size) == 0)
		*format = V4L2_PIX_FMT_SBGGR10;
	else if (strncmp(pixel_t, "bayer_rggb10", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB10;
	else if (strncmp(pixel_t, "bayer_bggr12", size) == 0)
		*format = V4L2_PIX_FMT_SBGGR12;
	else if (strncmp(pixel_t, "bayer_rggb12", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB12;
	else if (strncmp(pixel_t, "bayer_wdr_pwl_rggb12", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB12;
	else if (strncmp(pixel_t, "bayer_xbggr10p", size) == 0)
		*format = V4L2_PIX_FMT_XBGGR10P;
	else if (strncmp(pixel_t, "bayer_xrggb10p", size) == 0)
		*format = V4L2_PIX_FMT_XRGGB10P;
	else {
		pr_err("%s: Need to extend format%s\n", __func__, pixel_t);
		return -EINVAL;
	}

	return 0;
}

Doesn’t look like 8 bit is supported (or at least not broken out).