PCIe cards known to work for TX1

I could not have our FPGA cards working for TX1 under L4T 24.1, one card could not be detected by “lspci”, another was detected by “lspci” but could not be accessed be software even driver was natively built and installed without error. Those cards have be tested on Intel based CPUs without issue.

I’d like to test off-the-shelf PCIe cards to make sure it’s not L4T issue.

There are some links for (mini) PCIe cards tested and untested for TK1, but not for TX1.

Thanks in advance for sharing your inputs.

As far as I can tell the PCIe interface in 24.1 appears to be broken.

I finally did get build in 24.1 and when I got it going the debug USB/UART port started sending odd errors about the miniPCIe USB3 card, SYBA SD-MPE20142 I had stuck in a M.2 slot extender, which did work in 23.2. (note I did fix the error in 24.1 file drivers/base/Kconfig line 234 missing end quote.)
If I removed the miniPCie card the USB/UART port stop sending error messages.

On my custom FPGA card in the standard PCIe port I was not able to get any read/write DMA to work in 24.1. It can do ioctl commands , but the DRAM does not connect into PCIe bus right to do the file ops read and write DMA to the card. In 23.1 I can get PCIe read/write DMA working but not ioctl to it.
I added these vmalloc=256M cma=128M coherent-pool=96M in the /boot/extlinux extlinux.conf to get 23.1 DMA to work for PCIe using dma_alloc_coherent.

Between 24.1 and 23.2 there are changes made to the drivers/base files that do the DMA memory allocation. If the error drivers/base/Kconfig is left in place it appears that it does not build the object files in drivers/base, so I think no ones really testing it.

My best bet is to try and figure out why my ioctl commands do not work in 23.2 and forget using 24.1 until someone other then me gets into fixing it. (I actually have iio commands working in 23.2 to my card so can use that instead of ioctl if I need to.)

What really need to be done is convince the management a nvidia that it is not acceptable to release a development product that is not fully test with good support docs; that PCIe in the TX1 needs to be supported and how to make it work well document by them, not forgoing to third party documentation etc.

Jayd

Thanks again for sharing your experiences with 23.2 and 24.1. I completely agree with you that Nvidia should test basic hardware for any releases. It seems PCIe is broken and V4L for on board camera is not working. At lease 24.1 is a big stop forward for 64 bit user space which allows me to build driver/apps natively, but there are some steps back. I believe Nvidia should test PCie with hardware known to work for Linux. In our case, the FPGA cards and driver/app have been working for Intel CPUs many years under Linux. I hope Nvidia support can step in to answer some questions and fix the issues.

If someone can make off-the-shelf or custom FPGA cards working for TX1, we’ll definitely be interested in using them in our projects.

I can’t find a card that will work with 23.2. Tried two multiport serial cards and a sound and wifi card. Wouldn’t recognize any of them. I was told that if they try and throttle down past 2.5G/s transfer rate they don’t show up. The slowest the interface will run is 2.5G/s apparently at the present time. Haven’t tried on 24.1 yet. I just needed more serial ports but my controllers have can bus as well so I found a SPI to can converter. Managed to get the devices created so I think this will work for me. But it would be nice to have better support on the pcie interface. Although unless you build some kind of custom box there is no place to secure the cards and the tab on the bottom of the bracket has to come off as well. Mini itx case they stick up quite a bit. I used a nylon spacer and long bolt but it was still a little shaky. And the cover has to stay off. Having worked with the TK1’s I thought the whole interface was broken when I did a lspci the first time as nothing shows up. Hopefully when they fix it for 24.1 it will be able to throttle down some more as well.

Dan