How to configure pcix1 in tx2i

i have followed OEM DG & adaptation guide for updating required things in dts & cfg for custom carrier board with jetson tx2i.

  • ODMDATA = 0x90000 in p2700-common.conf
  • tegra186-quill-p3489-1000-a00-00-ucm1.dtb where pcie nodes,usb lane nodes, xhci nodes & overrides modified accordingly.
  • boot log looks like following

    nvidia@tegra-ubuntu:~$ dmesg | grep pci
    [    0.000000] Kernel command line: root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 pci=noaer memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 nvdumper_reserved=0x2372e0000 gpt tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.2.1 androidboot.serialno=0421918050217 bl_prof_dataptr=0x10000@0x237040000 sdhci_tegra.en_boot_part_access=1 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4
    [    0.251987] GPIO line 459 (pcie-lane2-mux) hogged as output/low
    [    0.255109] iommu: Adding device 10003000.pcie-controller to group 50
    [    0.416160] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
    [    0.418005] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
    [    0.418496] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
    [    0.420729] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
    [    0.846180] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
    [    1.250236] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
    [    1.654301] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
    [    1.656317] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
    [    2.058210] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
    [    2.462284] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
    [    2.623359] tegra-pcie 10003000.pcie-controller: PCI host bridge to bus 0000:00
    [    2.623364] pci_bus 0000:00: root bus resource [mem 0x50100000-0x57ffffff]
    [    2.623368] pci_bus 0000:00: root bus resource [mem 0x58000000-0x7fffffff pref]
    [    2.623371] pci_bus 0000:00: root bus resource [bus 00-ff]
    [    2.623374] pci_bus 0000:00: root bus resource [io  0x1000-0xffff]
    [    2.623401] pci 0000:00:03.0: [10de:10e6] type 01 class 0x060400
    [    2.623484] pci 0000:00:03.0: PME# supported from D0 D1 D2 D3hot D3cold
    [    2.623837] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [    2.643475] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
    [    2.643497] pci 0000:00:03.0: PCI bridge to [bus 01]
    [    2.643661] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
    [    2.643667] pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded
    

    it is showing invalid pcie-host bridge configurations.

    attached extracted dts file

    am i missing any configurations/modifications ? please share the detail steps to configure pcix1 for tx2i
    tegra186-quill-p3489-1000-a00-00-ucm1.txt (352 KB)

    Is this a nvidia devkit or custom carrier board?
    Could you use the same pcie setting in nv devkit and see if your pcie device can work or not?

    it is custom carrier board.

    we have pcie switch, we cant use it on dev kit. but device is fine, since we are able to configure switch ports from u-boot (i2c). if device is not proper. following o/p should not show when lspci is executed but we are getting this o/p

    PCI bridge: NVIDIA Corporation Device
    

    please share the required modifications steps for enabling pcix1 …

    How to resolve bridge configuration invalid ([bus 00-00]), reconfiguring error

    Does lspci -vvv show this device?

    here is the log

    nvidia@tegra-ubuntu:~$ sudo lspci -vvv
    00:03.0 PCI bridge: NVIDIA Corporation Device 10e6 (rev a1) (prog-if 00 [Normal decode])
    	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
    	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    	Latency: 0
    	Interrupt: pin A routed to IRQ 388
    	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    	I/O behind bridge: 0000f000-00000fff
    	Memory behind bridge: fff00000-000fffff
    	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
    	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
    		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    	Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
    	Capabilities: [48] Power Management version 3
    		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
    		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    	Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
    		Address: 0000000000000000  Data: 0000
    	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
    		Mapping Address Base: 00000000fee00000
    	Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
    		DevCap:	MaxPayload 128 bytes, PhantFunc 0
    			ExtTag+ RBE+
    		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
    			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    			MaxPayload 128 bytes, MaxReadReq 512 bytes
    		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
    		LnkCap:	Port #2, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
    			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
    		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
    			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
    		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
    			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
    		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
    			Control: AttnInd Off, PwrInd On, Power- Interlock-
    		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
    			Changed: MRL- PresDet+ LinkState+
    		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
    		RootCap: CRSVisible-
    		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
    		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
    		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
    		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
    			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
    			 Compliance De-emphasis: -6dB
    		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
    			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    	Capabilities: [100 v1] Advanced Error Reporting
    		UESta:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
    		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
    		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
    		CESta:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
    		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
    		AERCap:	First Error Pointer: 0e, GenCap+ CGenEn- ChkCap+ ChkEn-
    	Capabilities: [140 v1] L1 PM Substates
    		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
    			  PortCommonModeRestoreTime=30us PortTPowerOnTime=70us
    	Kernel driver in use: pcieport
    

    my switch device id not displaying only class code it is displaying

    Could you also try other pcie devices and see if get detected? If other device can get detected, then it is not a lane mapping issue.

    i have removed the pcie switch and directly connected frame grabber card( which is working on nxp evaluation board) through x1. with this change nvidia bridge also not detecting

    software used here is same as used for pcie switch where nvidia bridge detected. can i assume required lane mappings & configurations required for pcie are correct?

    Im not able to check lane mapping configurations since

    devmem2
    

    command not available. how to confirm my lane mappings for pcie x1 ??

    i referred following link
    https://devtalk.nvidia.com/default/topic/1023160/jetson-tx2/tx2-using-pcix1-instead-of-usb3-default-config-1-/2#reply

    thank you.

    Hi,

    Please download devmem2 tool from apt-get.

    [b]i have removed the pcie switch and directly connected frame grabber card( which is working on nxp evaluation board) through x1. with this change nvidia bridge also not detecting

    software used here is same as used for pcie switch where nvidia bridge detected. can i assume required lane mappings & configurations required for pcie are correct?[/b]

    Nope, if even other pcie devices are not detected. We could not tell whether it is hardware issue or software issue.

    Please use devmem2 to check register value and check if runtime device tree is correct.

    To check runtime device tree, please check the nodes under /proc/device-tree on your device.

    please find the o/p of devmem REG read.

    nvidia@tegra-ubuntu:~$ sudo busybox devmem 0x02520284
    0x00000201
    nvidia@tegra-ubuntu:~$ sudo busybox devmem 0x02530284
    0x00000201
    nvidia@tegra-ubuntu:~$ sudo busybox devmem 0x02540284
    0x00000201
    nvidia@tegra-ubuntu:~$ sudo busybox devmem 0x02550284
    0x00000201
    nvidia@tegra-ubuntu:~$ sudo busybox devmem 0x02560284
    0x00000201
    nvidia@tegra-ubuntu:~$ sudo busybox devmem 0x02570284
    0x00000202
    nvidia@tegra-ubuntu:~$
    

    referred link-https://devtalk.nvidia.com/default/topic/1002494/jetson-tx2/usb-lane-mapping/post/5125933/#5125933

    are my lane mappings as required for pcie x1 configurations??

    Hi,

    I just reviewed all the comments here. Actually, to use config #1, you just need to change ODMDATA = 0x90000 and plugin-manage would handle the rest. You don’t even need to change the dts/dtb.

    If same configuration can work on devkit but not your board, please check the hardware design.

    thank you for the information. i updated odm data in p2771-0000.conf.common it reflected in devmem Reg read.

    pinmux xls modification also not required ???

    in pinmux xls i modified following

  • uphy usb lines to pex
  • QSPI IO2 GPIO high to low
  • in tx2 adaptation guide, it suggests to modify DT for changing from usb to pex1
    Please confirm whether DT & pinmux needs to be changed ?

    Hi,

    in tx2 adaptation guide, it suggests to modify DT for changing from usb to pex1
    This is for general cases like config #3~6. Original device tree from jetpack supports config #1 and #2 by just using different odmdata value.

    This is due to a mechanism called plugin-manager which would override the content of device tree when ODMDATA is changed. You could check if below node is existing in your DT. (yes, I see it)

    fragment-500-e3325-pcie {
    

    Please note that only config #1 and #2 have such function because devkit supports them.

    Also, even devkit does not need to change the pinmux when changing usecases. Thus, I think it applies to your case too. You could verify my comment by using devkit first.

    thank you so much WayneWWW

    i have successfully updated odm data and then varified lane mapping on tx2 devkit as well as my custom carrier card with tx2i, it got reflected & dtb is getting override.

    i was getting continuous error

    8.323092] aer 0000:00:03.0:pcie02: service driver aer loaded
    [    8.328274] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0060
    [    8.329539] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0018(Receiver ID)
    [    8.329542] pcieport 0000:00:03.0:   device [10de:10e6] error status/mask=00000001/00002000
    [    8.329705] pcieport 0000:00:03.0:    [ 0] Receiver Error
    

    updated cmdline with

    pci=nomsi
    

    then error stopped.

    one bridge ditected

    updated source code driver/pci/host/pci-tegra.c for disabling power down
    two bridges detected

    but still it is not detecting my device on custom carrier card.

    please find the following log

    nvidia@tegra-ubuntu:~$ dmesg : | grep pci
    [    0.000000] Kernel command line: root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 pci=nomsi memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 nvdumper_reserved=0x2372e0000 gpt tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.2.1 androidboot.serialno=0421918050030 bl_prof_dataptr=0x10000@0x237040000 sdhci_tegra.en_boot_part_access=1 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4
    [    0.142720] node /plugin-manager/fragment-500-e3325-pcie match with board >=3489-0000-200
    [    0.142737] node /plugin-manager/fragment-500-e3325-pcie match with odm-data enable-pcie-on-uphy-lane0
    [    0.264405] GPIO line 459 (pcie-lane2-mux) hogged as output/low
    [    0.267633] iommu: Adding device 10003000.pcie-controller to group 50
    [    6.752051] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
    [    6.762753] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
    [    6.771158] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
    [    6.782107] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
    [   17.214812] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
    [   17.622796] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
    [   18.034817] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
    [   18.042814] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
    [   18.275172] tegra-pcie 10003000.pcie-controller: PCI host bridge to bus 0000:00
    [   18.282506] pci_bus 0000:00: root bus resource [mem 0x50100000-0x57ffffff]
    [   18.289410] pci_bus 0000:00: root bus resource [mem 0x58000000-0x7fffffff pref]
    [   18.296744] pci_bus 0000:00: root bus resource [bus 00-ff]
    [   18.302266] pci_bus 0000:00: root bus resource [io  0x1000-0xffff]
    [   18.308490] pci 0000:00:01.0: [10de:10e5] type 01 class 0x060400
    [   18.314585] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
    [   18.327051] pci 0000:00:03.0: [10de:10e6] type 01 class 0x060400
    [   18.333163] pci 0000:00:03.0: PME# supported from D0 D1 D2 D3hot D3cold
    [   18.345608] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [   18.353643] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [   18.361807] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
    [   18.402946] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
    [   18.409609] pci 0000:00:01.0: PCI bridge to [bus 01]
    [   18.414615] pci 0000:00:03.0: PCI bridge to [bus 02]
    [   18.419768] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
    [   18.426765] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded
    [   18.433660] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
    [   18.440659] pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded
    

    lspci o/p

    nvidia@tegra-ubuntu:~$ lspci
    00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1)
    00:03.0 PCI bridge: NVIDIA Corporation Device 10e6 (rev a1)
    

    thank you so much It worked !!