Force rescan of PCIe bus?

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Is there a way to force the PCIe bus to re-scan all devices after boot? This is quite useful for developing FPGA based PCIe devices, which are rebooted frequently in dev.

The internet says the proper way is this, but it seems to have no effect on the TX1:

echo 1 > /sys/bus/pci/rescan
2

I have not tried to rescan, but was your echo command done sudo? If not, then it’d probably fail for lack of permissions.

3

It is with sudo.

4

There are couple of ways to handle this (i.e. end point being an FPGA and comes up slowly) situation

  1. There is an entry in device tree “nvidia,boot-detect-delay” which can be used to delay enumeration by a specified time
  2. Make PCIe host controller driver as a loadable module (pci-tegra.ko) and insmod it only after FPGA is ready
  3. If you can tell me the version being used here (L4T 23 / L4T 24.2 Etc
), I can provide a patch with which we can have only root ports getting enumerated (in case end point is not ready) and later ‘rescan’ can be used to rescan the bus and add newly found devices to hierarchy.
5

The pci module unload/reload is a great suggestion, however, when I change the Tegra PCI controller to a module in menuconfig, the “make modules” step fails with “disable_scx_states [drivers/pci/host/pci-tegra.ko] undefined!”

I am running L4T 24.2 but can upgrade to 24.2.1 if need be. A rescan patch that works with /sys/bus/pci/rescan would be terrific, bringing it in line with other pci controller drivers.

6

following patch can be used to have only root ports enumerated in the beginning and after making sure that the end point is ready, “echo 1 > /sys/bus/pci/rescan” can be run to add end points to hierarchy.

diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c
index 6c070a9..af85890 100644
--- a/drivers/pci/host/pci-tegra.c
+++ b/drivers/pci/host/pci-tegra.c
@@ -2278,6 +2278,11 @@ static void tegra_pcie_check_ports(struct tegra_pcie *pcie)
            continue;
        }
        dev_info(pcie->dev, "link %u down, ignoring\n", port->index);
+       if (port->status == 0){
+           port->status = 1;
+           pcie->num_ports++;
+           continue;
+       }
        tegra_pcie_port_disable(port);
    }
 #if defined(CONFIG_ARCH_TEGRA_21x_SOC)
7

I tried this change on xavier L4T 32.1, which doesn’t work.

8

Could you please share the dmesg log with this patch? and also ‘sudo lspci -vv’ output.

9

Hello @vidyas!
I’m having the same problem with scan of pci device. I tried to edit that “pci-tegra.c” file in system, could you please tell to me how can I compile and build update that file for the pci driver in jetson tx2?
Thank you in advance,

10

That’s part of the kernel, so you’d end up building a kernel with a matching configuration to the running system. The hard part is that I don’t know if the edit will change the base kernel, or instead change a module. Out of safety I would recommend building and installing both a new kernel Image and modules. You would need to change the CONFIG_LOCALVERSION to something else so that the module directory changes. You could add the new Image with an alternate name (probably based on the CONFIG_LOCALVERSION) and simply add a second boot entry to “/boot/extlinux/extlinux.conf”, then pick that entry with serial console during boot.

The documents for your L4T release have the cross compile information. If you use command “head -n 1 /etc/nv_tegra_release” you can find your L4T version. The list of L4T versions (leading to documents and kernel source) are here:
https://developer.nvidia.com/linux-tegra

Some information on doing this natively, and explaining CONFIG_LOCALVERSION, can be found here:
https://forums.developer.nvidia.com/t/attempts-to-set-up-ecryptfs-and-fscrypt-failed-and-failed-and-failed/119083/2

The above URL emphasizes matching the CONFIG_LOCALVERSION, but in this case you might want your CONFIG_LOCALVERSION to be something like:
CONFIG_LOCALVERSION="-pci-test"

I highly recommend leaving the old kernel there and accessible via serial console picking. In that above URL it doesn’t mention “/boot/extlinux/extlinux.conf”, but basically, if you add the Image file with a new name to “/boot”, and copy the original part of the “/boot/extlinux/extlinux.conf” for the default Image, then you simply pick the one you want and don’t worry about if your edit worked or not. An example extlinux.conf edit to make both kernels available (if you have serial console set up during boot):

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

LABEL pci-test
      MENU LABEL pci-test
      LINUX /boot/Image-pci-test
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

You’d then interrupt boot at the correct moment (it is only a very short window to hit a key to interrupt boot
if you interrupt too early just type “boot” and then interrupt again at the kernel pick phase) and use “2” to pick the second kernel label.

1 Like
11

hi @linuxdev,

Thank you so much for your instruction. Do you any document or tutorial how to compile and update pci-driver in jetson tx2?

My problem is Jetson tx2 can detect an external PCIe device, however when I connect with FPGA device via PCIe it was not detected. if you have any advice, could you please share to me?

My log error as below:
tony@dovanhuong:~$ dmesg | grep pci
[ 0.488359] iommu: Adding device 10003000.pcie-controller to group 49
[ 0.488386] arm-smmu: forcing sodev map for 10003000.pcie-controller
[ 1.141635] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
[ 1.142814] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[ 1.143256] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
[ 1.147004] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[ 1.574753] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[ 1.983000] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[ 2.390663] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[ 2.392701] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
[ 2.799308] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 3.207451] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 3.649205] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 3.655800] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring
[ 3.690028] ehci-pci: EHCI PCI platform driver
[ 3.690068] ohci-pci: OHCI PCI platform driver
[ 3.878089] tegra-pcie 10003000.pcie-controller: PCIE: no end points detected
[ 3.892630] tegra-pcie 10003000.pcie-controller: PCIE: Disable power rails
tony@dovanhuong:~$ head -n 1 /etc/nv_tegra_release
R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t186ref, EABI: aarch64, DATE: Tue Dec 10 07:03:07 UTC 2019

Thank you in advance,

12

That part is not documented except to the extent that there is online content regarding kernel compile, and perhaps also kernel coding information online.

It is common for an FPGA to boot slower then the PCIe bus of a Jetson. There is nothing unusual about that. The trouble is that the PCIe bus is scanning before the FPGA is ready to respond.

On most PCs you’ll find the PCI bridge is active even if there is nothing detected. Then, if the FPGA is on such a bus, the bridge will see the FPGA as it does become available (but it will not be a cold plug detect, it will be a delayed hot plug detect).

On a Jetson the bridge and PCIe circuitry is stopped if it does not see anything at initial startup. This saves electrical power. You can do as the topic asks: Manually rescan the bus after that delay when you know the FPGA is active.

Another option is to power up the FPGA prior to powering up the Jetson.

I think most replies in this thread are rather old, but probably still accurate. The rest is feeling comfortable compiling a kernel and installing it to test. As long as you leave the old kernel in place and use serial console in combination with edits to “/boot/extlinux/extlinux.conf” there isn’t much risk. You do need to make sure your CONFIG_LOCALVERSION in the kernel config is not “-tegra” if you are going to install both kernel and modules and preserve the old kernel (and if you are editing the kernel in places where it might have an effect on both the kernel and a module, then this is what you want to do).

1 Like
13

This file (pci-tegra.c ) is used till TX2 and for Jetson Xavier, a different driver (pcie-tegra.c is used). The procedure to keep root ports alive (even when there are no endpoint devices enumerated) in Jetson Xavier is different compared to TX1/TX2.

14

For anyone wondering just HOW the Xavier is different compared to TX1/2, here is a patch that doesn’t shutdown the root port when the link is down:

diff --git a/drivers/pci/dwc/pcie-tegra.c b/drivers/pci/dwc/pcie-tegra.c
index 6f3919f85..2c7e06a41 100644
--- a/drivers/pci/dwc/pcie-tegra.c
+++ b/drivers/pci/dwc/pcie-tegra.c
@@ -3343,11 +3343,6 @@ static int tegra_pcie_config_rp(struct tegra_pcie_dw *pcie)

pcie->link_state = tegra_pcie_dw_link_up(&pcie->pci);

-       if (!pcie->link_state && pcie->power_down_en) {
-               ret = -ENOMEDIUM;
-               goto fail_host_init;
-       }
-
name = kasprintf(GFP_KERNEL, "pcie-%u", pcie->cid);
if (!name) {
ret = -ENOMEM;

It appears that you could also edit the device tree to remove “nvidia,enable-power-down”. This has the advantage of allowing you to disable power down on a single Root Port, instead of keeping all of them enabled.

1 Like