libargus repeating or interpolating frames? - 240fps support

Hi,

We are using the IMX274 sensor to capture in TX1, this sensor supports 1280x540@240fps and 4K@60fps. We are mainly interested in the 1280x540@240fps mode.

a) When trying to capture with libargus we are able to get 240fps for 1280x540, however, checking the driver debug information libargus is setting the mode in 4K@60fps. How is possible that we get the desired resolution and framerate if the driver is configuring the camera for 4K@60fps? is libargus able to downscale in the ISP and repeat or interpolate frames?

b) When using nvcamerasrc we have seen that nvcamerasrc is able to provide frames for resolutions even not supported by the sensor if they are lower than one of the supported resolutions. For instance, I can get 720p if the sensor supports 1080p, is nvcamerasrc able to downscale?

c) When trying to capture pure (not downscaled) 1280x540 coming from the camera, nvcamerasrc hangs and the camera daemon as well, it seems that not data is being captured by the CSI. Is the tegra able to capture 1280x540?

Thanks
-David

Hi David
a.) Check the log may not the final resolution. Put some message to kernel and check should be good.
b.) Yes, nvcamerasrc able to downscale.
c.) Yes, it able to capture 1280x540.

Hi ShaneCCC,

a) We are actually checking that it is the latest mode set but we will continue debugging it, just to confirm, there is not way for libargus to increase the framerate such that it is higher than the videosource or camera in this case?

b) Understood, that explains that it downscales instead of switching the mode.

c) Understood. Thanks.

I will keep you informed with the progress,

-David

Could you try set the frameraterange.

./apps/camera/modules/Dispatcher.cpp:    PROPAGATE_ERROR(m_sensorFrameRateRange.set(unifiedSensorFrameRateRange));

Hi,

We found how to set libargus in the correct mode and why the camera daemon was not happy when trying to capture 240fps, one register in the camera driver was not setting the correct crop size and therefore the resolution expected by VI was not the same given by the camera due the padding in the buffer. Anyway, now it works, even with the modified binary of nvcamerasrc, thanks for your help!

thanks for the finding DavidSoto,

one register in the camera driver was not setting the correct crop size
could you point out which register was not setup correctly?
thanks

Hi JerryChang,

It is the horizontal crop size register in the camera sensor for the mode supporting 240fps,

-David

Hi,

I thought that it was important to provide more details, just to be clear we are using the IMX274, in the 1280x540 mode with the TX1 and we are now able to capture ~240fps (the sensor is not stable on the value but is okay) using gstreamer and libargus, passing the data through the ISP to get YUV frames. The change mentioned above was required in the driver in order to make it work properly with the camera daemon,
ubuntu@tegra-ubuntu:~$ GST_DEBUG=WARNING gst-launch-1.0 -v nvcamerasrc sensor-id=0 fpsRange="240 240" ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)540, format=(string)I420, framerate=(fraction)240/1' ! perf ! queue ! nvvidconv ! video/x-raw ! xvimagesink 
Setting pipeline to PAUSED ...

Available Sensor modes : 
3840 x 2160 FR=60.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1920 x 1080 FR=60.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 540 FR=240.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...

NvCameraSrc: Trying To Set Default Camera Resolution. Selected 1280x540 FrameRate = 240.000000 ...

New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstPerf:perf0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstPerf:perf0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ framerate\=\(fraction\)240/1\,\ format\=\(string\)UYVY"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ framerate\=\(fraction\)240/1\,\ format\=\(string\)UYVY"
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ framerate\=\(fraction\)240/1\,\ format\=\(string\)UYVY"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ framerate\=\(fraction\)240/1\,\ format\=\(string\)UYVY"
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
GST-PERF INFO -->  Timestamp: 0:13:29.281982177; Bps: 0; fps: 0.0 
GST-PERF INFO -->  Timestamp: 0:13:30.286708457; Bps: 772; fps: 213.14 
GST-PERF INFO -->  Timestamp: 0:13:31.289609830; Bps: 774; fps: 233.53 
GST-PERF INFO -->  Timestamp: 0:13:32.289851751; Bps: 776; fps: 235.0 
GST-PERF INFO -->  Timestamp: 0:13:33.291798446; Bps: 775; fps: 234.76 
GST-PERF INFO -->  Timestamp: 0:13:34.295496007; Bps: 773; fps: 235.29 
GST-PERF INFO -->  Timestamp: 0:13:35.295873155; Bps: 776; fps: 235.0 
GST-PERF INFO -->  Timestamp: 0:13:36.298405889; Bps: 774; fps: 234.53 
GST-PERF INFO -->  Timestamp: 0:13:37.302135346; Bps: 773; fps: 235.29 
GST-PERF INFO -->  Timestamp: 0:13:38.304414042; Bps: 774; fps: 234.53 
GST-PERF INFO -->  Timestamp: 0:13:39.304604974; Bps: 776; fps: 231.0 
GST-PERF INFO -->  Timestamp: 0:13:40.307313141; Bps: 774; fps: 236.52 
GST-PERF INFO -->  Timestamp: 0:13:41.310724562; Bps: 773; fps: 236.29 
GST-PERF INFO -->  Timestamp: 0:13:42.313956205; Bps: 773; fps: 232.30 
GST-PERF INFO -->  Timestamp: 0:13:43.317905372; Bps: 773; fps: 237.28

Hey David,

This is an old post but is this frame rate possible with leopard imaging cameras for the TX1/TX2? They use CSI - MIPI. Thank you so much for your time and any insight you can provide.

Hi, Yes, we got it working with with the cameras from leopard after fixing the driver.

-David