Severe frame dropping with multiple cameras on Xavier/GStreamer

adev99 · September 16, 2019, 11:59am

Hi there,

I am running a 5-sensor (12MP/30fps) setup on Xavier w/ latest L4T R32.2.1 (problem also observed on R32.1). The image sensor driver is set up specifically for 5 sensors, and each sensor has been individually verified to provide a solid 30fps at 12MP. However, when attempting multiple simultaneous camera streams, I am seeing some strange bandwidth limitation with nvarguscamerasrc in GStreamer, even with the simplest of pipelines, as shown here:

gst-launch-1.0
nvarguscamerasrc sensor-id=0 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=1 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=2 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=3 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=4 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! fakesink

This pipeline is not a problem on the TX2 with R28/nvcamerasrc using the same image sensors, and a very similar driver; all 5 sensors run steady at 30 fps, simultaneously. But on Xavier/R32/nvarguscamerasrc, I am getting a lot of frame drops, with each camera seeing unstable frame rates of 15-20 fps (as observed from gst-shark tracer data).

If I reduce the number of cameras, or reduce nvarguscamerasrc’s output resolution, the problem can be reduced or eliminated, indicating there is some unusual bandwidth limitation that should not be present. I should note that the driver only has one mode (4056x3040x30fps), so the fact that I can get a solid 30fps on all 5 cameras, when specifying a lower resolution in the caps (meaning the ISP has to do the scaling), indicates that the driver is successfully supplying 4056x3040x30fps to the ISP for all 5 sensors, simultaneously.

Hope you have some insight into this problem. All 8 of Xavier’s CPUs are running, mode is MAXN, and jetson_clocks.sh has been run/stored to maximize all CPU/GPU clock frequencies.

DaneLLL · September 17, 2019, 2:35am

Hi,
Please enable the property in nvarguscamerasrc:

maxperf             : set or unset the max performace
                    flags: readable, writable
                    Boolean. Default: false

adev99 · September 17, 2019, 7:47pm

Hi DaneLLL,

Thanks for the reply, that did indeed allow all sensors to run at 30 fps with the example pipeline. However, if I add in some simple NV12 → I420 colour conversion to each camera output using nvvidconv, I again see frame drops, with performance that does not match what we’ve measured on nvcamerasrc/TX2/R28 (can run all 5 cameras at 30fps with colour conversion). Is there a way to optimize nvvidconv as well?

DaneLLL · September 18, 2019, 4:51am

Hi,
The hardware engine(VIC) of nvvidconv should always run at max clocks. Looks not expected if you observe worse performance on Xavier. We need to check if we can reproduce the issue. We don’t have cameras with 4056x3040, but will try with what we have.

For more information, you also launch all cameras in one process on R28? Please share the command you run on R28.

adev99 · September 24, 2019, 1:05am

Hi Dane,

Here is the pipeline I run on TX2/R28:

gst-launch-1.0
nvcamerasrc sensor-id=0 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvcamerasrc sensor-id=1 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvcamerasrc sensor-id=2 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvcamerasrc sensor-id=3 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvcamerasrc sensor-id=4 wbmode=3 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink

Each camera runs at 30fps no problem (maybe a dropped frame here or there).

Here is the equivalent pipeline I run on Xavier/R32:

gst-launch-1.0
nvarguscamerasrc sensor-id=0 wbmode=3 maxperf=true num-buffers=300 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=1 wbmode=3 maxperf=true num-buffers=300 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=2 wbmode=3 maxperf=true num-buffers=300 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=3 wbmode=3 maxperf=true num-buffers=300 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink
nvarguscamerasrc sensor-id=4 wbmode=3 maxperf=true num-buffers=300 ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=NV12, framerate=(fraction)30/1’ ! queue max-size-buffers=4 leaky=2 ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, format=I420, framerate=(fraction)30/1’ ! fakesink

Each camera has erratic frame rates that average ~21fps.

Thank you for your assistance.

DaneLLL · September 24, 2019, 6:46am

Hi,
For more information, please replace ‘fakesink’ with ‘fpsdisplaysink video-sink=fakesink text-overlay=false’ and run with ‘-v’ to print out the frame rate information.

We don’t observe the issue but the board we have is six ov5693 running in 2592x1944. It still has a gap in required bandwidth comparing to yours.

adev99 · September 24, 2019, 7:43pm

Hi Dane,

I tried as you suggested, and the results matched my previous results using gst-shark (erratic frame rates going between 17 and 25 fps, averaging ~21fps.

Yes, unfortunately six ov5693’s running is only half the bandwidth we are using (30MP per set of frames, vs 60MP per set), which I believe is the reason you are not seeing the issue.

DaneLLL · September 25, 2019, 1:53am

Hi,
For confirmation, you have flashed jetson-xavier-maxn.conf?

JerryChang · September 25, 2019, 5:22am

hello adev99,

could you please narrow down the issue,

let’s using v4l2 standard controls to exclude other component to check the CSI capability first.
for example,

$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000
$ v4l2-ctl -d /dev/video1 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000
$ v4l2-ctl -d /dev/video2 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000
$ v4l2-ctl -d /dev/video3 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000
$ v4l2-ctl -d /dev/video4 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000

please reduce the number of cameras in the same pipeline, for example, please check you still see frame drops by launching only dual camera.
you might also try to exclude video converter, please remove nvvidconv component and use fakesink to analysis the frame-rate.