PCIe Link Speed Issue on Custom Jetson Orin Nano Carrier Board

Issue Overview

Users developing custom carrier boards for the Nvidia Jetson Orin Nano are experiencing unexpected PCIe link speeds when connecting Gen4-compatible M.2 SSDs. Specifically, the PCIe interface (PCIE0, Gen4, x4) is operating at 8GT/s instead of the expected 16GT/s, even when using hardware capable of higher speeds. This issue occurs during hardware testing and affects the ability to fully utilize the PCIe Gen4 capabilities of the system. The problem persists regardless of the connected SSD, suggesting it may be related to the Jetson Orin Nano’s power management or configuration rather than the storage device itself.

Possible Causes

1. Power Management Features: The Jetson Orin Nano may be implementing power-saving measures that reduce PCIe link speeds when the GPU is idle to conserve energy.

2. BIOS/Firmware Settings: There might be default configurations in the system’s BIOS or firmware that limit PCIe speeds under certain conditions.

3. Driver Issues: The PCIe controller drivers may not be properly configured to enable full Gen4 speeds.

4. Hardware Limitations: There could be physical limitations on the custom carrier board that prevent achieving higher link speeds.

5. Thermal Constraints: The system might be throttling PCIe speeds to manage heat generation, especially if the custom carrier board doesn’t have adequate cooling.

6. Software Configuration: JetPack or Linux kernel settings may be limiting PCIe performance for stability or compatibility reasons.

Troubleshooting Steps, Solutions & Fixes

1. Verify Hardware Compatibility

   • Ensure that both the M.2 SSD and the PCIe lanes on the custom carrier board are properly designed for Gen4 operation.
   • Double-check the PCB layout and signal integrity of the custom carrier board to rule out hardware limitations.

2. Check PCIe Link Status

   • Use the following command to view detailed PCIe link information:

     sudo lspci -vvv
     

   • Look for the "LnkSta:" field to confirm current link speed and width.

3. Disable Power Management

   • Edit the GRUB configuration file:

     sudo nano /etc/default/grub
     

   • Add the following kernel parameter to the GRUB_CMDLINE_LINUX_DEFAULT line:

     pcie_aspm=off
     

   • Update GRUB and reboot:

     sudo update-grub
     sudo reboot
     

4. Force PCIe Gen4 Mode

   • Use the following commands to force Gen4 mode on the PCIe root port:

     sudo su
     echo 4 > /sys/bus/pci/devices/0000:00:00.0/max_link_speed
     echo 1 > /sys/bus/pci/devices/0000:00:00.0/rescan
     

   • Replace 0000:00:00.0 with the appropriate PCIe root port address for your system.

5. Update JetPack and Drivers

   • Ensure you’re running the latest version of JetPack (currently 6.0).
   • Check for any available updates:

     sudo apt update
     sudo apt upgrade
     

6. Monitor Thermal Performance

   • Use the tegrastats command to monitor system temperatures and clock speeds:

     tegrastats
     

   • If thermal throttling is occurring, improve cooling on the custom carrier board.

7. Adjust PCIe Clock Gating

   • Disable PCIe clock gating by adding the following to /etc/modprobe.d/pcie-clk-gating.conf:

     options pcie_aspm clock_gating=0
     

   • Reboot the system after making this change.

8. Check for BIOS/Firmware Updates

   • Visit the NVIDIA Jetson developer website to check for any available firmware updates for the Jetson Orin Nano.

9. Consult NVIDIA Developer Support

   • If the issue persists after trying these steps, consider reaching out to NVIDIA’s developer support or posting in their official forums for more specialized assistance.

10. Test with GPU Load

    • Create a simple CUDA program or use a benchmark tool to put load on the GPU.
    • Monitor PCIe link speed while the GPU is active to determine if the issue is related to power-saving features.

Remember that forcing maximum PCIe speeds may increase power consumption and heat generation. Ensure proper thermal management when running at higher speeds, especially on custom carrier boards that may not have the same cooling capabilities as official developer kits.