PCIe Rootport Controller ID Issue on Nvidia Jetson Orin Nano Dev Board

Issue Overview

Users are experiencing difficulties when attempting to connect two Nvidia Jetson Orin Nano boards via PCIe. The specific problem arises when configuring one board as a PCIe Endpoint (EP) and the other as a Root Port (RP). The main challenges include:

• Identifying the correct PCIe Rootport Controller ID for the Orin Nano
• Configuring the M.2 key M slot for PCIe connectivity
• Absence of expected PCIe controller addresses in the system debug information
• Uncertainty about Jetpack support for this specific use case

The issue occurs during the setup process, particularly when trying to replicate the configuration steps designed for the Jetson AGX on the Orin Nano. Users are unable to locate the expected PCIe controller ID (0004) or the address pcie@14160000 in the /sys/kernel/debug directory.

Possible Causes

1. Hardware Differences: The Jetson Orin Nano may have a different PCIe controller configuration compared to the Jetson AGX, leading to mismatched controller IDs and addresses.

2. Unsupported Use Case: The specific configuration of connecting two Orin Nanos via PCIe might not be officially supported by the Jetpack release, potentially causing unexpected behavior or missing functionality.

3. Incorrect HSIO Configuration: The High-Speed I/O (HSIO) settings for the PCIe controllers might be incorrectly configured, leading to the absence of expected controller IDs.

4. Software Limitations: The current Jetpack version may not fully support or expose the necessary PCIe controller information for this specific use case on the Orin Nano.

5. Device Tree Misconfiguration: The device tree might not be properly configured to expose or enable the required PCIe controller for the M.2 key M slot on the Orin Nano.

Troubleshooting Steps, Solutions & Fixes

1. Verify Hardware Compatibility:

   • Ensure that both Jetson Orin Nano boards are using compatible hardware revisions.
   • Confirm that the M.2 key M slots on both boards are properly configured for PCIe operation.

2. Check Jetpack and Firmware Versions:

   • Update both Orin Nano boards to the latest Jetpack version available.
   • Verify that all firmware components are up-to-date.

3. HSIO Configuration:

   • For the Endpoint (EP) device, use HSIO 41 configuration:

     sudo flash.sh jetson-orin-nano-devkit mmcblk0p1
     

   • For the Root Port (RP) device, use HSIO 0 configuration:

     sudo flash.sh jetson-orin-nano-devkit mmcblk0p1
     

4. Endpoint Configuration:
   Execute the following commands on the EP device:

   modprobe pci-epf-dma-test
   cd /sys/kernel/config/pci_ep/
   mkdir functions/tegra_pcie_dma_epf/func1
   echo 0x10de > functions/tegra_pcie_dma_epf/func1/vendorid
   echo 0x229a > functions/tegra_pcie_dma_epf/func1/deviceid
   echo 16 > functions/tegra_pcie_dma_epf/func1/msi_interrupts
   ln -s functions/tegra_pcie_dma_epf/func1 controllers/141a0000.pcie-ep/
   echo 1 > controllers/141a0000.pcie-ep/start
   

5. Identify Correct Controller ID:

   • On the RP device, examine the contents of /sys/kernel/debug/tegra_pcie:

     ls /sys/kernel/debug/tegra_pcie
     

   • Look for entries related to the M.2 key M slot or C4 controller.

6. Check Device Tree:

   • Examine the device tree for PCIe controller entries:

     dtc -I fs /sys/firmware/devicetree/base | grep -i pcie
     

   • Look for entries related to the M.2 slot or C4 controller.

7. Manually Probe PCIe Controller:
   If the controller is not automatically detected, try manually probing it:

   echo 1 > /sys/bus/pci/rescan
   

8. Consult Nvidia Developer Resources:

   • Check the Nvidia Jetson Linux Developer Guide for any updates or errata related to PCIe configuration on Orin Nano.
   • Consider posting a question on the Nvidia Developer Forums for official support.

9. Alternative Connection Methods:
   If PCIe connectivity remains problematic, consider alternative high-speed connection methods supported by the Orin Nano, such as USB 3.0 or Ethernet.

10. Debug Information Collection:
    Gather comprehensive debug information to assist in troubleshooting:

    sudo tegrastats
    dmesg | grep -i pcie
    lspci -vvv
    

It’s worth emphasizing that connecting two Orin Nanos via PCIe may not be an officially supported use case in the current Jetpack release. Users should be prepared for potential limitations or the need for custom development to achieve the desired functionality.