Tegra Multimedia APIs could get raw image via v4l2 only at 15fps

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Dear all,

When I tried to run 12_camera_v4l2_cuda in tegra_multimedia_api examples, I could get only 15 FPS as below

----------- Element = renderer0 -----------
Total Profiling time = 6.85006
Average FPS = 15.0364
Total units processed = 104
Num. of late units = 103

I also tried to measure polling time then result was same.

Polling Time : 0.066571 
Polling Time : 0.066521 
Polling Time : 0.066563 
Polling Time : 0.066589 
Polling Time : 0.066581 
Polling Time : 0.066580 
Polling Time : 0.066573 
Polling Time : 0.066569 
Polling Time : 0.066564 
Polling Time : 0.066583

But when I stream camera with gstreamer, it worked look like 30 fps

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,width=1280,height=720,framerate=30/1,format=(string)UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)128, height=(int)72, format=(string)N
V12' ! nvoverlaysink -e

I’m wondering this is camera driver issue or I need to change any configuration of tegra_multimedia_api.

Thanks
Vu Nguyen

2

Vu Nguyen,
Sorry for late, are u use 1080p for test?

I don’t think it is camera dirver issue, but it may be caused by buffer copy in APP side.

It is better if you can do some tirage to know which module is the bottleneck.

3

Hi waynezhu,

I used only 720p for testing.

I don’t think it is caused by buffer copy in APP side because I did test with below command :

v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600

then result still was 15fps.

I also checked my driver setting to confirm that v4l2 setting in driver was set to 30fps. I still didn’t figure out the reason.

Here is MIPI output

800×480 18.9 KB

Thanks,
Vu Nguyen

4

Vu Nguyen,
Are you connecting to FPGA or sensor?

Can you change fps to 20, does it get 10 fps in commandline?

5

Hi waynezhu,

I’m connecting to sensor. I tried to test on TX1 and TX2, camera driver same for both but the result when I run this command is different

v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600

TX1 return 30fps
TX2 return 15fps

So I think there is difference related v4l2 between TX1 and TX2. Please help me confirm v4l2 on TX2.

Thanks

6

Hi Vu Nguyen,
Which sensor do you use?
So you see different framerate by using v4l2-ctl on TX1 and TX2?

7

Hi DaneLLL,

I’m using OV10635 image sensor (via Serializer/Deserializer). There is only one baseboard and I just replace TX1 and TX2 modules each other so I could confirm that it was not hardware issue.

Camera driver for TX1 and TX2 also almost same so I think the most different is kernel (3.10 vs 4.x).

The result return from this command

v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600

as above post
TX1 return 30fps
TX2 return 15fps

Thanks
Vu Nguyen

8

Update

The result still was 15fps even though below command return 30fps

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
9

hello forever3000,

please try below commands to boost the system performance (VI clock and ISP clock)

sudo ./jetson_clocks.sh
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate
echo max_rate > /sys/kernel/debug/bpmp/debug/clk/vi/rate
echo max_rate > /sys/kernel/debug/bpmp/debug/clk/isp/rate
10

Hi JerryChang,

Thanks. I did as you prompt but result still was same.

11

Hi Vu,
Looks like 1920x1080p30 is not shown in ‘v4l2-ctl --list-formats-ext’. It shows 1280x720p30, is this mode working?

And for TX1, do you run r28.1?

12

Hi DaneLLL,

Yes my image sensor is 1280x720p30.
For TX1, I’m running r24.2. I will try r28.1 for TX1.

Thanks
Vu Nguyen

13

Hi DaneLLL,

I tried to test TX1 with 28.1 kernel. Driver was same because I just copied from TX2 and pasted to TX1, hardware baseboard also same just replace TX2 by TX1 module. And the result was:

TX1: return 30 fps
TX2: return 15 fps

Thanks

14

Hi DaneLLL,

Is there anyone at your side tried to test with YUV image sensor? I still couldn’t figure out why TX2 can run only at 15fps.

Thanks

15

Hi forever3000,
Could you try command “v4l2-ctl --list-formats-ext -d /dev/video0” and paste the result?
We can achieve 72 fps when using 1920x1080 mode with our sensor.

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext -d /dev/video0 
ioctl: VIDIOC_ENUM_FMT
Index       : 0
Type        : Video Capture
Pixel Format: 'UYVY'
Name        : UYVY 4:2:2
Size: Discrete 4224x3156
Interval: Discrete 0.050s (20.000 fps)
Size: Discrete 3840x2160
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.014s (72.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.014s (72.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.017s (60.000 fps)

nvidia@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat="UYVY" --device=/dev/video0  --stream-count=300 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.27 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.13 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.06 fps
<<<<<<<<<<
16

Hi WayneWWW,

This is my result

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext -d /dev/video0
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2

nvidia@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat="UYVY" --device=/dev/video0 --stream-count=300 --stream-mmap
<<<<<<<<<<<<<<<<< 15.09 fps
<<<<<<<<<<<<<<< 15.04 fps
<<<<<<<<<<<<<<< 15.03 fps
<<<<<<<<<<<<<<< 15.02 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<

Which camera was you running test?

Thanks
Vu Nguyen

17

Hi Vu,
Please share your device tree for reference.

18

Hi Vu,
Please also check if you have following change to support UYVY:

diff --git a/drivers/media/platform/tegra/camera/camera_common.c b/drivers/media/platform/tegra/camera/camera_common.c
index c0fabbd..cd43d34 100644
--- a/drivers/media/platform/tegra/camera/camera_common.c
+++ b/drivers/media/platform/tegra/camera/camera_common.c
@@ -34,6 +34,11 @@
 
 static const struct camera_common_colorfmt camera_common_color_fmts[] = {
 	{
+		MEDIA_BUS_FMT_UYVY8_1X16,
+		V4L2_COLORSPACE_SRGB,
+		V4L2_PIX_FMT_UYVY,
+	},
+	{
 		MEDIA_BUS_FMT_SRGGB12_1X12,
 		V4L2_COLORSPACE_SRGB,
 		V4L2_PIX_FMT_SRGGB12,
diff --git a/drivers/media/platform/tegra/camera/sensor_common.c b/drivers/media/platform/tegra/camera/sensor_common.c
index 62941cf..7bd9fbe 100644
--- a/drivers/media/platform/tegra/camera/sensor_common.c
+++ b/drivers/media/platform/tegra/camera/sensor_common.c
@@ -111,6 +111,8 @@ static int extract_pixel_format(
 		*format = V4L2_PIX_FMT_XBGGR10P;
 	else if (strncmp(pixel_t, "bayer_xrggb10p", size) == 0)
 		*format = V4L2_PIX_FMT_XRGGB10P;
+	else if (strncmp(pixel_t, "uyvy", size) == 0)
+		*format = V4L2_PIX_FMT_UYVY;
 	else {
 		pr_err("%s: Need to extend format%s\n", __func__, pixel_t);
 		return -EINVAL;
diff --git a/drivers/media/platform/tegra/camera/vi/vi4_formats.h b/drivers/media/platform/tegra/camera/vi/vi4_formats.h
index bbdbbab..c40854c 100644
--- a/drivers/media/platform/tegra/camera/vi/vi4_formats.h
+++ b/drivers/media/platform/tegra/camera/vi/vi4_formats.h
@@ -125,8 +125,6 @@ static const struct tegra_video_format vi4_video_formats[] = {
 	/* YUV422 */
 	TEGRA_VIDEO_FORMAT(YUV422, 16, UYVY8_1X16, 2, 1, T_U8_Y8__V8_Y8,
 				YUV422_8, UYVY, "YUV 4:2:2"),
-	TEGRA_VIDEO_FORMAT(YUV422, 16, UYVY8_1X16, 1, 1, T_Y8__V8U8_N422,
-				YUV422_8, NV16, "NV16"),
 	TEGRA_VIDEO_FORMAT(YUV422, 16, UYVY8_2X8, 2, 1, T_U8_Y8__V8_Y8,
 				YUV422_8, UYVY, "YUV 4:2:2 UYVY"),
 	TEGRA_VIDEO_FORMAT(YUV422, 16, VYUY8_2X8, 2, 1, T_T_V8_Y8__U8_Y8,
19

Hi DaneLLL,

Yes I already applied but result was still same.

20

Hi Vu,
Please share your device tree for reference.