Keeping a Specific PCI Root Port Enabled on Nvidia Jetson Orin Nano

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing difficulties with re-enumerating PCI devices after boot, particularly when working with FPGA-based PCIe devices that require a bit-stream to be programmed before they appear. The standard method of rescanning the PCIe bus using the command echo 1 > /sys/bus/pci/rescan appears to be ineffective on this platform. This issue is crucial for developers working with FPGA-based PCIe devices that are frequently rebooted during the development process.

Possible Causes

1. Default Power Management Behavior: The Jetson Orin Nano’s PCIe controller may be designed to power down unused ports after boot, preventing re-enumeration.

2. Driver Limitations: The current PCIe driver (pcie-tegra194.c) may not support dynamic re-enumeration of devices post-boot.

3. FPGA Programming Timing: The FPGA may not be programmed quickly enough after boot for the system to detect it during the initial PCIe scan.

4. Device Tree Specification: The absence of specific properties in the Device Tree Specification (DTS) for the Tegra234 (Orin Nano) may prevent easy configuration of PCIe power management.

Troubleshooting Steps, Solutions & Fixes

1. Modify Device Tree Overlay:
   The most effective and recommended solution is to add a specific property to the Device Tree Overlay:

   • Add the property nvidia,disable-power-down to the PCIe root port you want to keep enabled.
   • This method allows for keeping a specific PCI root port enabled without modifying the driver code.

   Example Device Tree Overlay snippet:

   &{/pcie@14100000} {
       nvidia,disable-power-down;
   };
   

2. Driver Modification (Not Recommended):
   If the Device Tree Overlay method doesn’t work, a less desirable alternative involves modifying the PCIe driver:

   • Edit the file drivers/pci/controller/dwc/pcie-tegra194.c
   • Comment out the following lines around line 1675:

     // pcie->link_state = tegra_pcie_dw_link_up(&pcie->pci);
     // if (!pcie->link_state) {
     //     ret = -ENOMEDIUM;
     //     goto fail_host_init;
     // }
     

   • This modification prevents the driver from checking the link state, potentially allowing for re-enumeration.
   • Note: This method affects all PCIe ports and is not recommended for production use.

3. Verify FPGA Programming:
   Ensure that the FPGA is programmed with the correct bit-stream before attempting to re-enumerate PCIe devices:

   • Implement a delay or a check in your boot process to confirm FPGA programming completion.
   • Use appropriate FPGA programming tools and verify the programming status before proceeding with PCIe operations.

4. Check for Updated Drivers:
   Regularly check for updates to the Jetson Orin Nano BSP (Board Support Package) or NVIDIA drivers:

   • Visit the NVIDIA Developer website for the latest Jetson Linux releases.
   • Update your system to the latest available version, as newer releases may include improvements to PCIe handling.

5. Custom Kernel Module:
   If the above solutions are insufficient, consider developing a custom kernel module:

   • The module could implement a sysfs interface to trigger re-enumeration of specific PCIe ports on demand.
   • This approach requires advanced Linux kernel development skills and thorough testing.

6. Consult NVIDIA Support:
   If the issue persists or you need further assistance:

   • Reach out to NVIDIA’s developer support channels.
   • Provide detailed information about your setup, including the specific FPGA model and any custom hardware configurations.

By implementing these solutions, particularly the Device Tree Overlay method, users should be able to keep specific PCI root ports enabled on their Nvidia Jetson Orin Nano, facilitating the development process with FPGA-based PCIe devices.