Realsense camera unstable

pkolomiets · March 21, 2019, 10:15am

Hi,
I’m trying to use multiple Realsense cameras (D435) with AGX Xavier.
But even connecting single camera to either Type-C or Type-A port
in 1280x720 at 15fps mode with rgb+left+right+depth streams on
I get many incomplete frames and “USB SCP Overflow” kind of error eventually.
Or sometimes I get problems with simple 1280x720x30fps rgb stream.

My config: JP 4.1.1, Ubuntu 18.04.

I tried to patch kernel image for Realsense following [url]Intel RealSense D435i on NVIDIA Jetson AGX Xavier - JetsonHacks. Doesn’t help.
I tried using more powerful 90W power adapter from TX2, different firmwares and library versions for Realsense, different Xavier’s nvpmodes, different USB ports.
Nothing helps.
The problem is also mentioned here
[url]Intel RealSense D435i on NVIDIA Jetson AGX Xavier - JetsonHacks,
solved by using PCI->USB expansion card.
But it looks kind of ugly to use PCI instead of three USB 3.0 ports.

On host machine it’s not a problem to use even 2-3 Realsense cameras connected to 1 port,
so the USB 3.0 bandwidth should be enough.
So, what can be the reason? Not enough power supplied through USB ports (Realsense needs 3.5W)?
Thanks.

DaneLLL · March 22, 2019, 5:00am

Hi,
This is a known issue. We are trying but it is tough to keep 100% compliance of all USB devices since we cannot buy USB cameras of all brands. Also some devices may be with its own drivers/frameworks.

If you have to use this camera and need further support, please contact us.

DaneLLL · April 9, 2019, 6:29am

Please try attachment.
0001-media-uvc-restrict-urb_num-no-less-than-UVC_URBS.zip (1.31 KB)

pkolomiets · April 12, 2019, 8:28am

Tried the patch. Doesn’t help at all.

DaneLLL · April 12, 2019, 8:33am

Hi,
We have verified it working on r32.1. Are you on r32.1?

DaneLLL · April 12, 2019, 9:04am

Hi,
Our command for verification is

$ sudo jetson_clocks.sh
$ sudo nvpmodel -m 0
$ gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=YUY2,framerate=30/1,width=1280,height=720 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvoverlaysink

If you run a different case, please share more detail so that we can reproduce it first.

pkolomiets · April 12, 2019, 9:19am

Yes, I’m on r32.1. I’m checking with realsense-viewer and 1280x720x30fps rgb fails after half a minute. I’ve checked with your gstreamer command and it fails too (freezes silently), though after a longer time (several minutes).
Can you check with realsense-viewer, especially depth + rgb?

DaneLLL · April 12, 2019, 9:33am

Hi,
For USB cameras, we usually run gstreamer, tegra_multimedia_api and v4l2-ctl. If you don’t use gstreamer and tegra_multimedia_api, can you please share v4l2-ctl commands for reproducing the issue?

We don’t have experience of using realsense-viewer.

pkolomiets · April 12, 2019, 9:41am

Then, maybe it’s time to try realsense-viewer, which would be the most straightforward way to reproduce the error? Also can you try to render depth stream through gstreamer? Most of the errors come with depth enabled.
My final application uses ROS’s wrapper for librealsense, so no v4l2-ctl commands.

jgentry · April 14, 2019, 11:14pm

Dane, Is there any documentation for setting up the patch to run RealSense on the Xavier? I’m just running into the set-up issues that are found throughout the forum.

Is there an est. future Jetson release that will be compatible with the RealSense line of cameras?

If using RealSense is going to be a headache until a future release, is there a good method for streaming with the cameras on Ubuntu?

Thanks in advance for the help.

DaneLLL · April 15, 2019, 1:40am

Hi,
We now support USB cameras through USB UVC. Please consider using cameras from our partners:

simfend · May 29, 2019, 8:11am

Will this make the RealSense (D435) viable? And how?

We just bought two of them for our Jetson TX2 and it would be sad, if the “USB SCP Overflow” error will still occure.

DaneLLL · May 29, 2019, 9:01am

Hi,
We are evaluating to have further relationship with the camera vendor. Please realize x86 systems are the first priority to the vendor. Let’s see if we could make some breakthrough.

DaneLLL · May 30, 2019, 6:26am

For TX2, the user’s sharing might help you:
[url]https://devtalk.nvidia.com/default/topic/1052039/jetson-tx2/how-to-install-intel-realsense-d435i-driver-on-tx2-jetpack4-2-/post/5344538/#5344538[/url]

pkolomiets · May 30, 2019, 7:49am

It’s not about patching the kernel (no problem to patch), it’s about stability.

DaneLLL · July 24, 2019, 9:28am

Hi,
On r32.2/Xavier, please execute following steps and try again:

1. Replace xusb_sil_rel_fw_Xavier
    1. copy xusb_sil_rel_fw_Xavier to pendrive
    2. back up original firmware
    mv /lib/firmware/tegra19x_xusb_firmware /lib/firmware/tegra19x_xusb_firmware_ori
    3. copy xusb_sil_rel_fw_Xavier to /lib/firmware
    cp <WHERE_YOU_MOUNT_PENDRIVE>/xusb_sil_rel_fw_Xavier /lib/firmware/tegra19x_xusb_firmware
    4. reboot device
    5. The fw timestamp should be:
        root@tegra-ubuntu:/sys/class/tegra-firmware/3610000.xhci# cat version 
        3610000.xhci: Firmware timestamp: 2019-07-16 08:23:26 UTC, Version: 60.05 release
2. Remove all the USB device and confirm that xhci enters ELPG
    1. Check "tegra-xusb 3610000.xhci: entering ELPG done" in kernel log
3. Increase falcon clock freq
    1. sudo su
    2. cd /sys/kernel/debug/bpmp/debug/clk/xusb_falcon
    3. echo 1 > state
    4. echo 408000000 > rate
    5. cat rate    ---------> To make sure the rate is 408000000

The firmware is attached.
On r32.1, you also need the patch

xusb_sil_rel_fw_Xavier.zip (75.4 KB)

atyshka · July 25, 2019, 3:22pm

Big thanks to the Jetson team for working to resolve this issue! I can confirm your directions greatly improve performance on Xavier with Jetpack 4.2.1. I still notice occasional incomplete frames when I push the limits with four full-res streams, but these errors only happen at about 1 Hz. With 4 streams at 30fps, these dropped frames are now within a 1% failure rate, which is more than good enough for most applications. We’d all appreciate it if these changes were incorporated into the next Jetpack release so that we don’t have to make this patch manually. Other than that I’d consider this issue solved!

yutaka.takaoka · July 31, 2019, 6:03pm

Hi @DaneLLL
I replaced the FW for attached one but the fw_timestamp was still like “2018-03-29 14:24:42 UTC” even though device was rebooted…
Any help would be appreciated.

yutaka.takaoka · July 31, 2019, 10:02pm

Hi,
Even for the above timestamp, it seems to work better than before. However I am still seeing some incomplete frames. When I use an external powered USB hub, it’s much better.
Do you have any plan to improve it furthermore?

pkolomiets · August 7, 2019, 8:53am

Hi,
On r32.1, after step 2 cannot find “tegra-xusb 3610000.xhci: entering ELPG done” in kernel log.
Instead, have this

tegra-xusb 3610000.xhci: Direct firmware load for tegra19x_xusb_firmware failed with error -2
tegra-xusb 3610000.xhci: Falling back to user helper
...
tegra-xusb 3610000.xhci: cannot find firmware....retry after 1 second

The full log attached.

UPDATE: Haven’t managed to make “entering ELPG done” whatever I tried.
However, with increased falcon clock freq, three full 1280x720 depth+infra1+infra2+color streams run well at 30fps, with occasional incomplete frames.
And, the latter is true with the old firmware too! So, what is the change introduced by the new firmware?
Also, after every reboot, falcon clock freq is reset to default value. Is there a way to make the change permanent?
Thanks.

kern.log (719 KB)