Use PCIe to communicate between two Xaviers: RP phy dosen't up

Hi

I use two xavier connected by pcie(x1, gen-1).

What I had done.

–1–
Follow https://devtalk.nvidia.com/default/topic/1048582/jetson-agx-xavier/the-problem-about-xavier-pcie-endpoint-mode/. set ep ODMDATA=0x9191000

–2–
swap tx and rx with a cable
tx <–> rx, prst <–> prst, wake <–> wake, clkreq <–> clkreq, clkref <–> clkref

–3–
ep log

uisee@uisee-xavier-02:~$ dmesg | grep -i pci
[    0.000000]     PCI I/O : 0xffffffbefee00000 - 0xffffffbeffe00000   (    16 MB)
[    0.958765] iommu: Adding device 141a0000.pcie_ep to group 0
[    0.959822] iommu: Adding device 14180000.pcie to group 1
[    0.961093] iommu: Adding device 14100000.pcie to group 2
[    0.962101] iommu: Adding device 14140000.pcie to group 3
[    1.037882] GPIO line 490 (pcie-reg-enable) hogged as output/high
[    1.037989] GPIO line 289 (pcie-reg-enable) hogged as output/high
[    1.337731] PCI: CLS 0 bytes, default 64
[    4.344789] Intel(R) 10GbE PCI Express Linux Network Driver - version 4.6.4
[    4.356394] ehci-pci: EHCI PCI platform driver
[    4.356464] ohci-pci: OHCI PCI platform driver
[    7.619977] tegra-pcie-dw 141a0000.pcie_ep: invalid max-speed (err=-22), set to Gen-1
[    7.619981] tegra-pcie-dw 141a0000.pcie_ep: Setting init speed to max speed
[    7.621325] tegra-pcie-dw 14180000.pcie: Setting init speed to max speed
[    7.622166] OF: PCI: host bridge /pcie@14180000 ranges:
[    7.622178] OF: PCI:    IO 0x38100000..0x381fffff -> 0x38100000
[    7.622184] OF: PCI:   MEM 0x38200000..0x39ffffff -> 0x38200000
[    7.622188] OF: PCI:   MEM 0x1800000000..0x1bffffffff -> 0x1800000000
[    8.131545] tegra-pcie-dw 14180000.pcie: link is down
[    8.132925] tegra-pcie-dw 14180000.pcie: PCI host bridge to bus 0000:00
[    8.134013] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.135036] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[    8.136151] pci_bus 0000:00: root bus resource [mem 0x38200000-0x39ffffff]
[    8.137209] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1bffffffff pref]
[    8.138306] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[    8.138471] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    8.140051] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    8.141123] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.142451] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    8.143551] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    8.143661] aer 0000:00:00.0:pcie002: service driver aer loaded
[    8.143856] pcie_pme 0000:00:00.0:pcie001: unloading service driver pcie_pme
[    8.143900] aer 0000:00:00.0:pcie002: unloading service driver aer
[    8.144010] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[    8.145563] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[    8.145842] tegra-pcie-dw 14180000.pcie: PCIe link is not up...!
[    8.152669] tegra-pcie-dw 14100000.pcie: Setting init speed to max speed
[    8.159310] OF: PCI: host bridge /pcie@14100000 ranges:
[    8.163632] OF: PCI:    IO 0x30100000..0x301fffff -> 0x30100000
[    8.169474] OF: PCI:   MEM 0x30200000..0x31ffffff -> 0x30200000
[    8.175294] OF: PCI:   MEM 0x1200000000..0x123fffffff -> 0x1200000000
[    8.288868] tegra-pcie-dw 14100000.pcie: link is up
[    8.290217] tegra-pcie-dw 14100000.pcie: PCI host bridge to bus 0001:00
[    8.291356] pci_bus 0001:00: root bus resource [bus 00-ff]
[    8.292422] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[    8.293620] pci_bus 0001:00: root bus resource [mem 0x30200000-0x31ffffff]
[    8.294732] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x123fffffff pref]
[    8.295961] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[    8.296155] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    8.297904] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[    8.298012] pci 0001:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    8.298095] pci 0001:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    8.298157] pci 0001:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    8.298235] pci 0001:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    8.298297] pci 0001:01:00.0: reg 0x20: [io  0x800000-0x80000f]
[    8.298358] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[    8.298419] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[    8.298750] pci 0001:01:00.0: PME# supported from D3hot
[    8.309600] pci 0001:00:00.0: BAR 14: assigned [mem 0x30200000-0x302fffff]
[    8.310810] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    8.311976] pci 0001:01:00.0: BAR 6: assigned [mem 0x30200000-0x3020ffff pref]
[    8.313222] pci 0001:01:00.0: BAR 5: assigned [mem 0x30210000-0x302101ff]
[    8.314465] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    8.315710] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    8.316947] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    8.318196] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    8.319435] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    8.320669] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    8.321909] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    8.323143] pci 0001:00:00.0:   bridge window [mem 0x30200000-0x302fffff]
[    8.329296] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.338016] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    8.346886] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
[    8.353481] pci 0001:01:00.0: Signaling PME through PCIe PME interrupt
[    8.360007] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
[    8.360166] aer 0001:00:00.0:pcie002: service driver aer loaded
[    8.386644] tegra-pcie-dw 14140000.pcie: Setting init speed to max speed
[    8.394088] OF: PCI: host bridge /pcie@14140000 ranges:
[    8.398493] OF: PCI:    IO 0x34100000..0x341fffff -> 0x34100000
[    8.404244] OF: PCI:   MEM 0x34200000..0x35ffffff -> 0x34200000
[    8.410355] OF: PCI:   MEM 0x1280000000..0x12bfffffff -> 0x1280000000
[    8.928143] tegra-pcie-dw 14140000.pcie: link is down
[    8.929697] tegra-pcie-dw 14140000.pcie: PCI host bridge to bus 0003:00
[    8.931450] pci_bus 0003:00: root bus resource [bus 00-ff]
[    8.933391] pci_bus 0003:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[    8.934763] pci_bus 0003:00: root bus resource [mem 0x34200000-0x35ffffff]
[    8.936047] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12bfffffff pref]
[    8.937386] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[    8.937606] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    8.939471] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    8.940726] pci 0003:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.942339] pcieport 0003:00:00.0: Signaling PME through PCIe PME interrupt
[    8.943630] pcie_pme 0003:00:00.0:pcie001: service driver pcie_pme loaded
[    8.943753] aer 0003:00:00.0:pcie002: service driver aer loaded
[    8.943942] pcie_pme 0003:00:00.0:pcie001: unloading service driver pcie_pme
[    8.943987] aer 0003:00:00.0:pcie002: unloading service driver aer
[    8.944079] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[    8.945540] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[    8.945813] tegra-pcie-dw 14140000.pcie: PCIe link is not up...!
uisee@uisee-xavier-02:~$

–4–
rc log

uisee@uisee-xavier-01:~$ dmesg | grep -i pci
[    0.000000]     PCI I/O : 0xffffffbefee00000 - 0xffffffbeffe00000   (    16 MB)
[    0.913131] iommu: Adding device 14180000.pcie to group 0
[    0.914244] iommu: Adding device 14100000.pcie to group 1
[    0.915275] iommu: Adding device 14140000.pcie to group 2
[    0.916263] iommu: Adding device 141a0000.pcie to group 3
[    0.991724] GPIO line 490 (pcie-reg-enable) hogged as output/high
[    0.991820] GPIO line 289 (pcie-reg-enable) hogged as output/high
[    1.318471] PCI: CLS 0 bytes, default 64
[    4.322711] Intel(R) 10GbE PCI Express Linux Network Driver - version 4.6.4
[    4.333785] ehci-pci: EHCI PCI platform driver
[    4.333845] ohci-pci: OHCI PCI platform driver
[    6.805965] tegra-pcie-dw 14180000.pcie: Setting init speed to max speed
[    6.813333] OF: PCI: host bridge /pcie@14180000 ranges:
[    6.817538] OF: PCI:    IO 0x38100000..0x381fffff -> 0x38100000
[    6.823306] OF: PCI:   MEM 0x38200000..0x39ffffff -> 0x38200000
[    6.829503] OF: PCI:   MEM 0x1800000000..0x1bffffffff -> 0x1800000000
[    7.346922] tegra-pcie-dw 14180000.pcie: link is down
[    7.347303] tegra-pcie-dw 14180000.pcie: PCI host bridge to bus 0000:00
[    7.347436] pci_bus 0000:00: root bus resource [bus 00-ff]
[    7.347534] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[    7.347693] pci_bus 0000:00: root bus resource [mem 0x38200000-0x39ffffff]
[    7.347830] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1bffffffff pref]
[    7.347985] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[    7.348151] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    7.348896] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    7.349003] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    7.349424] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    7.349557] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    7.349757] aer 0000:00:00.0:pcie002: service driver aer loaded
[    7.349911] pcie_pme 0000:00:00.0:pcie001: unloading service driver pcie_pme
[    7.350017] aer 0000:00:00.0:pcie002: unloading service driver aer
[    7.350123] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[    7.350384] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[    7.350592] tegra-pcie-dw 14180000.pcie: PCIe link is not up...!
[    7.351493] tegra-pcie-dw 14100000.pcie: Setting init speed to max speed
[    7.352672] OF: PCI: host bridge /pcie@14100000 ranges:
[    7.352784] OF: PCI:    IO 0x30100000..0x301fffff -> 0x30100000
[    7.352913] OF: PCI:   MEM 0x30200000..0x31ffffff -> 0x30200000
[    7.358960] OF: PCI:   MEM 0x1200000000..0x123fffffff -> 0x1200000000
[    7.473726] tegra-pcie-dw 14100000.pcie: link is up
[    7.474119] tegra-pcie-dw 14100000.pcie: PCI host bridge to bus 0001:00
[    7.474246] pci_bus 0001:00: root bus resource [bus 00-ff]
[    7.474345] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[    7.474514] pci_bus 0001:00: root bus resource [mem 0x30200000-0x31ffffff]
[    7.474633] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x123fffffff pref]
[    7.474799] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[    7.474946] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    7.475734] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[    7.475840] pci 0001:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    7.475901] pci 0001:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    7.475962] pci 0001:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    7.476040] pci 0001:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    7.476101] pci 0001:01:00.0: reg 0x20: [io  0x800000-0x80000f]
[    7.476179] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[    7.476240] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[    7.476570] pci 0001:01:00.0: PME# supported from D3hot
[    7.486419] pci 0001:00:00.0: BAR 14: assigned [mem 0x30200000-0x302fffff]
[    7.486563] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    7.486691] pci 0001:01:00.0: BAR 6: assigned [mem 0x30200000-0x3020ffff pref]
[    7.486840] pci 0001:01:00.0: BAR 5: assigned [mem 0x30210000-0x302101ff]
[    7.487012] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    7.487206] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    7.487408] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    7.487591] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    7.488474] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    7.494886] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    7.499994] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    7.506467] pci 0001:00:00.0:   bridge window [mem 0x30200000-0x302fffff]
[    7.513141] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    7.522000] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    7.530737] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
[    7.537346] pci 0001:01:00.0: Signaling PME through PCIe PME interrupt
[    7.543838] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
[    7.543968] aer 0001:00:00.0:pcie002: service driver aer loaded
[    7.570577] tegra-pcie-dw 14140000.pcie: Setting init speed to max speed
[    7.578054] OF: PCI: host bridge /pcie@14140000 ranges:
[    7.582358] OF: PCI:    IO 0x34100000..0x341fffff -> 0x34100000
[    7.588199] OF: PCI:   MEM 0x34200000..0x35ffffff -> 0x34200000
[    7.594022] OF: PCI:   MEM 0x1280000000..0x12bfffffff -> 0x1280000000
[    8.114591] tegra-pcie-dw 14140000.pcie: link is down
[    8.114956] tegra-pcie-dw 14140000.pcie: PCI host bridge to bus 0003:00
[    8.115098] pci_bus 0003:00: root bus resource [bus 00-ff]
[    8.115195] pci_bus 0003:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[    8.115379] pci_bus 0003:00: root bus resource [mem 0x34200000-0x35ffffff]
[    8.115512] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12bfffffff pref]
[    8.115683] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[    8.115833] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    8.116527] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    8.116639] pci 0003:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.117072] pcieport 0003:00:00.0: Signaling PME through PCIe PME interrupt
[    8.117196] pcie_pme 0003:00:00.0:pcie001: service driver pcie_pme loaded
[    8.117305] aer 0003:00:00.0:pcie002: service driver aer loaded
[    8.117453] pcie_pme 0003:00:00.0:pcie001: unloading service driver pcie_pme
[    8.117497] aer 0003:00:00.0:pcie002: unloading service driver aer
[    8.117571] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[    8.117839] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[    8.118081] tegra-pcie-dw 14140000.pcie: PCIe link is not up...!
[    8.118727] tegra-pcie-dw 141a0000.pcie: Setting init speed to max speed
[    8.119950] OF: PCI: host bridge /pcie@141a0000 ranges:
[    8.120095] OF: PCI:    IO 0x3a100000..0x3a1fffff -> 0x3a100000
[    8.120978] OF: PCI:   MEM 0x3a200000..0x3bffffff -> 0x3a200000
[    8.126887] OF: PCI:   MEM 0x1c00000000..0x1fffffffff -> 0x1c00000000
[    8.646535] tegra-pcie-dw 141a0000.pcie: link is down
[    8.646895] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[    8.647040] pci_bus 0005:00: root bus resource [bus 00-ff]
[    8.647159] pci_bus 0005:00: root bus resource [io  0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[    8.647324] pci_bus 0005:00: root bus resource [mem 0x3a200000-0x3bffffff]
[    8.647456] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1fffffffff pref]
[    8.647809] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[    8.647975] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    8.648664] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    8.648773] pci 0005:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.649202] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[    8.649331] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[    8.649434] aer 0005:00:00.0:pcie002: service driver aer loaded
[    8.649572] pcie_pme 0005:00:00.0:pcie001: unloading service driver pcie_pme
[    8.649610] aer 0005:00:00.0:pcie002: unloading service driver aer
[    8.649711] pci_bus 0005:01: busn_res: [bus 01-ff] is released
[    8.649934] pci_bus 0005:00: busn_res: [bus 00-ff] is released
[    8.650138] tegra-pcie-dw 141a0000.pcie: PCIe link is not up...!
uisee@uisee-xavier-01:~$

Thanks

Can you check if the EP system is able to observe pex_rst toggle coming from host? you can check that by dumping the interrupt count for ‘pex_rst’ interrupt in ‘cat /proc/interrupts’.

Hi

Thanks for your reply.

uisee@uisee-xavier-02:~$ cat /proc/interrupts | grep pex_rst
415: 0 0 0 0 tegra-gpio 217 Edge pex_rst

So, there is no PEX_RST toggle observed by endpoint mode. Can you please check PEX_RST routing from host to endpoint?
Also, another way to check in SW whether or not routing is present is to do the following on host.

  1. Remove device-tree entry “nvidia,enable-power-down” from the node “pcie@141a0000”. This is to make sure that PCIe root port is at least available for SW to play around with even though there is no PCIe link up with endpoint mode.
  2. Once the console is available, try toggling the PEX_RST line
    busybox devmem 0x141a0000 w
    in the read value, make bit-0 as ‘0’ and write it back to drive ‘0’ on PEX_RST line and
    in the read value, make bit-0 as ‘1’ and write it back to drive ‘1’ on PEX_RST line.

When a particular value (be it ‘0’ or ‘1’) on PEX_RST, the same value should be observed on the other side i.e. in endpoint mode console, dump the value at same register (i.e. 0x141a0000) and look for bit-7 which reflects the state of PEX_RST as observed by endpoint mode controller.

Hi

Follow your suggestion.

Remove “nvidia,enable-power-down”

Using devmem set the pex_rst on rp, I got the set value on prst pin by multimeter.

But on ep, using devmem 0x141a0000 w, trigger a crash

[   99.422049] CPU3: SError detected, daif=140, spsr=0x80000000, mpidr=80000101, esr=be000000
[   99.422064] CPU1: SError detected, daif=140, spsr=0x60000000, mpidr=80000001, esr=be000000
[   99.422071] CPU2: SError detected, daif=140, spsr=0x20000000, mpidr=80000100, esr=be000000
[   99.424223] CPU:0, Error:CBB-NOC@0x2300000,irq=485
[   99.424299] **************************************
[   99.424301] * For more Internal Decode Help
[   99.424302] *     http://nv/cbberr
[   99.424303] * NVIDIA userID is required to access
[   99.424304] **************************************
[   99.424307] CPU:1, Error:CBB-NOC
[   99.424310] 	Error Logger		: 0
[   99.424321] 	ErrLog0			: 0x80030000
[   99.424326] 	  Transaction Type	: RD  - Read, Incrementing
[   99.424329] 	  Error Code		: SLV
[   99.424331] 	  Error Source		: Target
[   99.424340] 	  Error Description	: Target error detected by CBB slave
[   99.424355] 	  AXI2APB_1 bridge error: SFIFONE - Status FIFO Not Empty interrupt
[   99.424364] 	  AXI2APB_1 bridge error: SLV - SLVERR interrupt
[   99.424364] 	  Packet header Lock	: 0
[   99.424366] 	  Packet header Len1	: 3
[   99.424368] 	  NOC protocol version	: version >= 2.7
[   99.424369] 	ErrLog1			: 0x320022
[   99.424371] 	ErrLog2			: 0x0
[   99.424373] 	  RouteId		: 0x320022
[   99.424376] 	  InitFlow		: ccroc_p2ps/I/ccroc_p2ps
[   99.424381] 	  Targflow		: axis_satellite_grout/T/axis_satellite_grout
[   99.424382] 	  TargSubRange		: 0
[   99.424383] 	  SeqId			: 0
[   99.424385] 	ErrLog3			: 0x141a0000
[   99.424386] 	ErrLog4			: 0x0
[   99.424471] 	  Address		: 0x141a0000 -- appl + 0x0
[   99.424473] 	ErrLog5			: 0x889f850
[   99.424475] 	  Non-Modify		: 0x1
[   99.424476] 	  AXI ID		: 0x11
[   99.424478] 	  Master ID		: CCPLEX
[   99.424480] 	  Security Group(GRPSEC): 0x7e
[   99.424482] 	  Cache			: 0x0 -- Non-cacheable/Non-Bufferable)
[   99.424488] 	  Protection		: 0x2 -- Unprivileged, Non-Secure, Data Access
[   99.424489] 	  FALCONSEC		: 0x0
[   99.424491] 	  Virtual Queuing Channel(VQC): 0x0
[   99.424494] 	**************************************
[   99.424503] ras_ccplex_serr_callback: Scanning CCPLEX Error Records for Uncorrectable Errors
[   99.424556] **************************************
[   99.424559] RAS Error in SCF:IOB:
[   99.424561] 	Status = 0xf4009604
[   99.424563] 	IERR = CBB Interface Error: 0x96
[   99.424567] 	SERR = Assertion Failure: 0x4
[   99.424568] 	Uncorrectable (this is fatal)
[   99.424581] 	MISC0 = 0x40
[   99.424582] 	MISC1 = 0x264a444463
[   99.424589] 	ADDR = 0x80000000141a0000
[   99.424604] **************************************
[   99.424611] ras_corecluster_serr_callback:Scanning CoreCluster Error Records for Uncorrectable Errors
[   99.424630] **************************************
[   99.424632] RAS Error in L2:
[   99.424633] 	Status = 0xf4006612
[   99.424635] 	IERR = SCF to L2 Slave Error Read: 0x66
[   99.424641] 	SERR = Error response from slave: 0x12
[   99.424642] 	Uncorrectable (this is fatal)
[   99.424653] 	MISC0 = 0x100000000400000
[   99.424654] 	MISC1 = 0x40240000000
[   99.424660] 	ADDR = 0x80000000141a0000
[   99.424669] **************************************
[   99.424711] ras_core_serr_callback: Scanning Core Error Records for Uncorrectable Errors
[   99.424942] Bad mode in Error handler detected on CPU1, code 0xbe000000 -- SError
[   99.424954] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[   99.424964] ras_ccplex_serr_callback: Scanning CCPLEX Error Records for Uncorrectable Errors
[   99.426047] Modules linked in: bnep
[   99.426048] ras_corecluster_serr_callback:Scanning CoreCluster Error Records for Uncorrectable Errors
[   99.426074]  fuse nvs_bmi160 nvs nvgpu bluedroid_pm ip_tables x_tables
[   99.426107] CPU: 1 PID: 7899 Comm: devmem2 Not tainted 4.9.140-tegra #1
[   99.426110] Hardware name: jetson-xavier (DT)
[   99.426115] task: ffffffc37cd2b800 task.stack: ffffffc3a7a48000
[   99.426123] ras_core_serr_callback: Scanning Core Error Records for Uncorrectable Errors
[   99.426128] PC is at 0x400c60
[   99.426130] LR is at 0x400bf0
[   99.426133] pc : [<0000000000400c60>] lr : [<0000000000400bf0>] pstate: 60000000
[   99.426138] sp : 0000007fd3587cf0
[   99.426143] x29: 0000007fd3587cf0 x28: 0000000000000000 
[   99.426146] x27: 0000000000000000 x26: 0000000000000000 
[   99.426149] x25: 0000000000000000 x24: 0000000000000000 
[   99.426152] x23: 0000000000000000 x22: 0000000000000000 
[   99.426156] x21: 00000000004008c0 x20: 0000000000000000 
[   99.426159] x19: 0000000000400e50 x18: 0000007fd3587bae 
[   99.426162] x17: 0000007f9541bdc8 x16: 0000000000412058 
[   99.426165] x15: 0000000000000010 x14: 0000000000000001 
[   99.426168] x13: 0000000000000000 x12: 0000000000000020 
[   99.426171] x11: 0000007fd3587bb8 x10: 0000000000000000 
[   99.426179] x9 : 00000000fffffff6 x8 : 0000000000000040 
[   99.426183] x7 : 000000000000000a x6 : 000000000000002e 
[   99.426186] x5 : 0000000036051287 x4 : 0000000000000000 
[   99.426189] x3 : 0000000000000000 x2 : 0000000000000001 
[   99.426192] x1 : 0000000000400c2c x0 : 0000000000000000 

[   99.426201] Process devmem2 (pid: 7899, stack limit = 0xffffffc3a7a48000)
[   99.426213] ---[ end trace efdf9f1ec733c6a5 ]---
[   99.428647] ras_ccplex_serr_callback: Scanning CCPLEX Error Records for Uncorrectable Errors
[   99.428694] ras_corecluster_serr_callback:Scanning CoreCluster Error Records for Uncorrectable Errors
[   99.428755] ras_core_serr_callback: Scanning Core Error Records for Uncorrectable Errors
[   99.825328] **************************************
[   99.830104] * For more Internal Decode Help
[   99.833864] *     http://nv/cbberr
[   99.837356] * NVIDIA userID is required to access
[   99.841909] **************************************
[   99.846722] CPU:0, Error:CBB-NOC
[   99.850153] 	Error Logger		: 0
[   99.852864] 	ErrLog0			: 0x80030000
[   99.856698] 	  Transaction Type	: RD  - Read, Incrementing
[   99.861949] 	  Error Code		: SLV
[   99.865532] 	  Error Source		: Target
[   99.868951] 	  Error Description	: Target error detected by CBB slave
[   99.875359] 	  AXI2APB_1 bridge error: SFIFONE - Status FIFO Not Empty interrupt
[   99.882789] 	  Packet header Lock	: 0
[   99.886188] 	  Packet header Len1	: 3
[   99.890120] 	  NOC protocol version	: version >= 2.7
[   99.895106] 	ErrLog1			: 0x320022
[   99.898344] 	ErrLog2			: 0x0
[   99.901231] 	  RouteId		: 0x320022
[   99.904649] 	  InitFlow		: ccroc_p2ps/I/ccroc_p2ps
[   99.909634] 	  Targflow		: axis_satellite_grout/T/axis_satellite_grout
[   99.916038] 	  TargSubRange		: 0
[   99.919170] 	  SeqId			: 0
[   99.922057] 	ErrLog3			: 0x141a0000
[   99.925298] 	ErrLog4			: 0x0
[   99.928321] 	  Address		: 0x141a0000 -- appl + 0x0
[   99.933001] 	ErrLog5			: 0x889f850
[   99.936667] 	  Non-Modify		: 0x1
[   99.939821] 	  AXI ID		: 0x11
[   99.943144] 	  Master ID		: CCPLEX
[   99.946556] 	  Security Group(GRPSEC): 0x7e
[   99.950667] 	  Cache			: 0x0 -- Non-cacheable/Non-Bufferable)
[   99.956199] 	  Protection		: 0x2 -- Unprivileged, Non-Secure, Data Access
[   99.963268] 	  FALCONSEC		: 0x0
[   99.966330] 	  Virtual Queuing Channel(VQC): 0x0
[   99.971080] 	**************************************
[   99.978724] CPU0: SError detected, daif=140, spsr=0x20400145, mpidr=80000000, esr=56000000
[   99.984523] ras_ccplex_serr_callback: Scanning CCPLEX Error Records for Uncorrectable Errors
[   99.992260] ras_corecluster_serr_callback:Scanning CoreCluster Error Records for Uncorrectable Errors
[  100.001509] ras_core_serr_callback: Scanning Core Error Records for Uncorrectable Errors

Thanks

Can you please tell me the jetpack version you are using?

Hi

Tegra_Linux_Sample-Root-Filesystem_R32.1.0_aarch64.tbz2

Jetson_Linux_R32.1.0_aarch64.tbz2

uisee@uisee-xavier-02:~$ uname -a
Linux uisee-xavier-02 4.9.140-tegra #1 SMP PREEMPT Wed Mar 13 00:30:11 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux

Hi,

Please follow below steps to bring up Tegra EP,

  1. Boot the endpoint Jetson system.
  2. Run the following commands to configure and enable PCIe endpoint mode
    cd /sys/kernel/config/pci_ep/
    mkdir functions/pci_epf_nv_test/func1
    echo 0x10de > functions/pci_epf_nv_test/func1/vendorid
    echo 0x0001 > functions/pci_epf_nv_test/func1/deviceid
    ln -s functions/pci_epf_nv_test/func1 controllers/141a0000.pcie_ep/
    echo 1 > controllers/141a0000.pcie_ep/start
  3. Boot the root port system. You must execute this after above step.

Manikanta

Hi, Manikanta

Thanks for your replay!

Follow your step, there is no ep, when lspci on master.

rp info

[    9.359091] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[    9.359248] pci_bus 0005:00: root bus resource [bus 00-ff]
[    9.359352] pci_bus 0005:00: root bus resource [io  0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[    9.359529] pci_bus 0005:00: root bus resource [mem 0x3a200000-0x3bffffff]
[    9.359655] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1fffffffff pref]
[    9.359816] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[    9.359969] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    9.360531] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    9.360644] pci 0005:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    9.360978] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[    9.361116] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[    9.361202] aer 0005:00:00.0:pcie002: service driver aer loaded

uisee@uisee-xavier-01:~$ lspci
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
uisee@uisee-xavier-01:~$

ep info

[   57.692420] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x3f707a000
[   57.692449] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[   57.692558] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0xffffff800bd0b000


uisee@uisee-xavier-02:/sys/kernel/config/pci_ep/functions/pci_epf_nv_test/func1$ cat deviceid 
0x0001
uisee@uisee-xavier-02:/sys/kernel/config/pci_ep/functions/pci_epf_nv_test/func1$ cat vendorid 
0x10de
uisee@uisee-xavier-02:/sys/kernel/config/pci_ep/functions/pci_epf_nv_test/func1$ cd ../../../

uisee@uisee-xavier-02:/sys/kernel/config/pci_ep/controllers/141a0000.pcie_ep$ cat start 
1

Thanks!

Hi,

Please share complete kernel log of both root port and endpoint.
Also “cat /proc/interrupts | grep pex_rst” output from end point.

Manikanta

Hi, Manikanta

Change another cable, follow you step, find ep, when lspci on master.

Another question, how to start the virtual ethernet over pcie?

Thanks!

uisee@uisee-xavier-01:~$ lspci
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0005:01:00.0 RAM memory: NVIDIA Corporation Device 0001
uisee@uisee-xavier-01:~$
[   72.769851] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x3f6d96000
[   72.769884] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[   72.769924] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0xffffff800bc85000
[   95.131970] tegra-pcie-dw 141a0000.pcie_ep: EP init done
uisee@uisee-xavier-02:~$ 
uisee@uisee-xavier-02:~$ cat /proc/interrupts | grep pex_rst
415:          2          0          0          0  tegra-gpio 217 Edge      pex_rst
uisee@uisee-xavier-02:~$

Hi,

Instead of pci_epf_nv_test, you have to use pci_epf_tvnet driver in the steps mentioned in comment #8.
Following are the complete commands.
cd /sys/kernel/config/pci_ep/
mkdir functions/pci_epf_tvnet/func1
#echo 0x104c > functions/pci_epf_tvnet/func1/vendorid
#echo 0xb500 > functions/pci_epf_tvnet/func1/deviceid
echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
ln -s functions/pci_epf_tvnet/func1 controllers/141a0000.pcie_ep/
echo 1 > controllers/141a0000.pcie_ep/start

Once the host is booted, you can find eth1 interface on both side. Bring up both the interfaces and assign IP addresses in same subnet.

Manikanta

Hi, Manikanta

Something is strange.

After 300 sec, the eth1 up.

[  362.452362] tegradc 15220000.nvdisplay: blank - powerdown
[  424.008601] pci_epf_tvnet pci_epf_tvnet.0: tvnet_open: PCIe link is not up
[  473.571014] pci_epf_tvnet pci_epf_tvnet.0: tvnet_open: PCIe link is not up
[  473.581270] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  473.600211] pci_epf_tvnet pci_epf_tvnet.0: tvnet_open: PCIe link is not up
[  473.609735] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  473.628403] pci_epf_tvnet pci_epf_tvnet.0: tvnet_open: PCIe link is not up
[  473.637794] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  473.659132] pci_epf_tvnet pci_epf_tvnet.0: tvnet_open: PCIe link is not up
[  473.669469] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  482.287445] tegra-pcie-dw 141a0000.pcie_ep: EP init done
[  773.451716] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  773.451794] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  966.747892] NOHZ: local_softirq_pending 08
[  966.822238] NOHZ: local_softirq_pending 08
[  966.839511] NOHZ: local_softirq_pending 08
[  967.711765] NOHZ: local_softirq_pending 08
[  968.735855] NOHZ: local_softirq_pending 08
[  968.747695] NOHZ: local_softirq_pending 08

Hi,

This is expected,

  • We can bring up eth1 interface only after root port enumerates the device, below print signifies it

[ 473.659132] pci_epf_tvnet pci_epf_tvnet.0: tvnet_open: PCIe link is not up
[ 482.287445] tegra-pcie-dw 141a0000.pcie_ep: EP init done /* PCIe link is up and eth1 interface is ready */

  • Since there no Ethernet PHY, we simulated Ethernet link up in SW. Ethernet link comes up after eth1 interface is up on both sides. I believe you brought up eth1 interface on host side after some delay.

Manikanta

Hi, Manikanta

My script lack “echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts”

Thanks!

With the above command ,Ihave generated eth1 on both sides.
But the speed at which they transfer files directly is only 12.5M/s.
Is there any way to improve it ?

您好,能否分享下您PCI通信使用的PCIE (X1,gen-1)电路板,最好有链接,非常感谢