USB 3 transfer failures

Hi Nvidia devs,

We have several TX1 users reporting libfreenect2 cannot correctly receive USB 3 transfers. This application receives 30 isochronous frames per second from Kinect, and each frame consists of about 100 active packets, and idle empty packets the rest of the time.

They get a lot of USB isochronous packets with status -EXDEV, which apparently means no data is transmitted, that is, some packets in the middle get no data. Their dmesg also shows a lot of tegra-xhci: WARN Event TRB for slot 3 ep 8 with no TDs queued?. This results in them being unable to decode the image frames and use the Kinect.

Two reports with respective dmesg and their system setup:

https://github.com/OpenKinect/libfreenect2/issues/581#issuecomment-188870962
https://github.com/OpenKinect/libfreenect2/issues/480#issuecomment-160745936

One user is able to get libfreenect2 work via a uPD720202 PCIe USB 3.0 extension card. None of our users were able to get it work using the built-in USB 3.0 port.

I see a similar report from a few years ago here from xhci_hcd driver:
https://bbs.archlinux.org/viewtopic.php?id=157557

I don’t think it is relevant, but could possibly be related…reports mentioned from other users on other hardware listed being on battery from a laptop. I’m not positive but it may be possible battery operation limits supplied power when devices are powered via USB. Since some of the reports I saw on the web mention errors starting upon plugging in other devices which would be higher current draws (e.g., the above link has this occur when connecting a USB flash drive), I would start by my usual USB step of seeing if use of a powered HUB makes a difference.

In your link I see these: “packetsize or sequence doesn’t match!”, “subpacket too large”, “image data too short!”. It seems fairly obvious data is being truncated. I do not have a kinect, nor do I have a USB analyzer, so it is limited what I can suggest. However, here is a quote I found particularly relevant from this JetsonHacks.com URL:
https://github.com/OpenKinect/libfreenect2/issues/480#issuecomment-160745936
[i]Since the first article, there were a couple issues that have been addressed in xlz’s repository.

First, xlz reverted a previous libfreenect2 commit to fix MAX_ISO_BUFFER_LENGTH which was causing libusb issues.

Second, libfreenect2 installs libopencv-dev, which overwrites the CUDA accelerated opencv4tegra library. The install script on the How-To removes the libopencv-dev and adds libjpeg-turbo8-dev which installs the ‘turbojpeg.h’ header file.[/i]

That MAX_ISO_BUFFER_LENGTH seems very suspicious under these circumstances, and kernel used may actually have the same issue. After testing with a powered USB HUB, I would go look for the MAX_ISO_BUFFER_LENGTH and see what an increase might do to solve this (and I suspect this will be the fix).

I’m trying to abstract away the higher level details of libfreenect2 so you don’t have to worry about the warnings.

I suggested to a user to patch libusb to revert MAX_ISO_BUFFER_LENGTH from 49152 * 128 (libusb upstream value, doesn’t work for uPD720202) to 49152 (the value that works for uPD720202). The user reported this had no effect on tegra-xhci.

The user: Libfreenect2 not working on Jetson TX1 · Issue #581 · OpenKinect/libfreenect2 · GitHub

Jetson TK1’s tegra-xhci works fine though. I’m definitely looking at kernel issues with tegra-xhci.

From drivers/usb/core/urb.c:

* in the past.  When a packet is assigned in this way to a slot that has
 * already expired, the packet is not transmitted and the corresponding
 * usb_iso_packet_descriptor's status field will return -EXDEV.  If this
 * would happen to all the packets in the URB, submission fails with a
 * -EXDEV error code.

What happens here is that some packets in the middle of a transfer get -EXDEV.

There were some discussions in the past regarding USB3 buffer options for a JTK1 (prior to JTX1 existing) using kernel boot parameters to make buffer size testing easier…I don’t recall right now where that was. But it really sounds like the root hub has had buffers fill and any kind of recovery from the missing data was incorrect. This could perhaps be a case of failing to resend data, or resending data too late (expired). Having a larger Jetson-side buffer still seems to be a step in the right direction.

I’m just wondering what buffer size the kinect actually needs, and perhaps setting MAX_ISO_BUFFER_LENGTH based on that. How were the people experimenting with the kinect altering buffer size? Is it via kernel command line parameter, editing driver code, etc? What was the “49152 * 128” and other size settings based on?

I’m not sure if MAX_ISO_BUFFER_LENGTH is entirely relevant to this issue.

128 is the maximum packets per transfer allowed by the kernel. 49152 is the maximum bytes per packets allowed by the kernel. (All isochronous)

In drivers/usb/core/devio.c:

case USBDEVFS_URB_TYPE_ISO:
        /* arbitrary limit */
        if (uurb->number_of_packets < 1 ||
            uurb->number_of_packets > 128)
                return -EINVAL;

...
/*
 * arbitrary limit need for USB 3.0
 * bMaxBurst (0~15 allowed, 1~16 packets)
 * bmAttributes (bit 1:0, mult 0~2, 1~3 packets)
 * sizemax: 1024 * 16 * 3 = 49152
 */
if (isopkt[u].length > 49152) {
        ret = -EINVAL;
        goto error;
}

The way libfreenect2 does with this is to queue 60 transfers of 8 packets of 0x8400 bytes each. tegra-xhci on TK1 is apparently able to do it. uPD720202 with TX1 is unable to handle it (unless a transfer is split into 49152-byte pieces). tegra-xhci on TX1 is unable to handle this, as reported above.

Until trying more buffer size or using a protocol analyzer, so on, there isn’t much I can add. But do keep in mind that part of the issue could be not emptying the buffer in time to make it available for the next kinect data transfer…if that’s the case, then having more buffer would give the end user app or other drivers time to empty the first buffer. Without really digging into the USB chain of code to know things like how the memory is allocated, what might cause it to block (e.g., threading issues or demanding a flush before putting in new data, or maybe even some odd memory alignment requirement using more memory than actual packet sizes); it’s just easier to try a bigger buffer and see what happens. Don’t forget that although other USB devices might not use much data, they probably do lock out some fixed amount of memory as well, making it unavailable for the kinect. Try double, triple, or even quadruple of USB buffer…I can’t do the experimenting since I don’t have a kinect.

I have obtained a TX1 and checked several things.

It is not caused by autosuspend.

It seems when the USB isochronous transfer packet size is >= 32768, the packet will get -EXDEV. If the packet size is 31*1024, data is received. This seems to be bugs in Tegra XHCI kernel drivers.

That seems like a large packet for isochronous, but I don’t know what standards would call for. I’m looking at a Logitech cam (not a high res one), and here’s the list of wMaxPacketSize (not even remotely close to 32k):

wMaxPacketSize     0x0040  1x 64 bytes
        wMaxPacketSize     0x00c0  1x 192 bytes
        wMaxPacketSize     0x0180  1x 384 bytes
        wMaxPacketSize     0x0200  1x 512 bytes
        wMaxPacketSize     0x0280  1x 640 bytes
        wMaxPacketSize     0x0320  1x 800 bytes
        wMaxPacketSize     0x03b0  1x 944 bytes
        wMaxPacketSize     0x0a80  2x 640 bytes
        wMaxPacketSize     0x0b20  2x 800 bytes
        wMaxPacketSize     0x0be0  2x 992 bytes
        wMaxPacketSize     0x1380  3x 896 bytes
        wMaxPacketSize     0x13fc  3x 1020 bytes
        wMaxPacketSize     0x0044  1x 68 bytes
        wMaxPacketSize     0x0064  1x 100 bytes
        wMaxPacketSize     0x0084  1x 132 bytes

Would you please post the output of “sudo lsusb -vvv” for that one device?

Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84  EP 4 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
        bMaxBurst              10
        Mult                    2

33792 = (Mult+1)*(bMaxBurst+1)*wMaxPacketSize, which is also known as wBytesPerInterval.

I said “packet size” because it’s the “packet” within a URB.

FWIW the same setup with the same device works fine on TK1 L4T 21.4.

That’s straight out of the usb.org docs, it should work. Looking at the github URL you mentioned, it seems the uPD720202 extension card works, and this card should have core USB code in common in the kernel. I have no means to measure packet timing to see if ISO queues are failing (-EXDEV) due to delays…this seems like a possibility which could show in the integrated USB3 but function correctly for an add-on card. I believe there is a bug here within the kernel, but I have no means of tracking this further.

Hi xlz,

A warn message, “tegra-xhci: WARN Event TRB for slot 3 ep 8 with no TDs queued?” which mentioned above might be related to an issue, and could be fixed already.
Because we don’t have details info, I am not sure it is relevant.

Would you please try with TX1 L4T R23.2 to see if it can improve your case?

Thanks

Hi kay,

The L4T version tested above is # R23 (release), REVISION: 2.0, GCID: 6630372, BOARD: t210ref, EABI: hard, DATE: Tue Feb 9 01:49:02 UTC 2016, the kernel version is Linux version 3.10.67-g458d45c (buildbrain@mobile-u64-630) (gcc version 4.9 20140514 (prerelease) (GCC) ) #1 SMP PREEMPT Mon Feb 8 17:44:18 PST 2016.

Hi kay,

I used TX1 L4T R23.2 and could repro the problem too. are there any updates about this? We planned to use Kinect2 on TX1 and need this to be fixed soon. Thanks for the help!

Hello xlz, kay,

I also have purchased a Jetson TX1 and am attempting to use it with a Kinect v2 (aswell as a ZED and Duo MLX). I have followed the progress here and on the related github forum (see latest post for my log files) [url]https://github.com/OpenKinect/libfreenect2/issues/581[/url] and am eager to hear about progress.

Kind Regards,

Sorry for the late reply.

We’re still investigating this issue. Once any finding, I will do the update.

Thanks

Same issue here… any updates?

This became available after Kay could post it, but we have a firmware patch that should resolve this.

You can replace the USB firmware file by unzipping and copying the unzipped file to your /lib/firmware/ directory with the following command:

cp xusb_sil_rel_fw-test /lib/firmware/tegra21x_xusb_firmware

Then reboot the system. Let us know if this resolves the issue or if any subsequent issues are encountered.

xusb_sil_rel_fw-test.zip (73.2 KB)

After some light testing, this appears to fix the transfer issue.
Thank you very much!
There appears to be issues with the RGB color stream, but I don’t know if this is on the raw data side or the application side just yet.
Update: The RGB issue is on the application side with conversion of color formats. It be working!

Kangalow: is this on 23.2 or 24.1?

Regards