Display Driver Loading Issue on Jetson Orin Nano Dev Kit with Custom Kernel

Issue Overview

Users are experiencing difficulties loading display drivers on a Jetson Orin Nano Developer Kit board after building a custom kernel for Jetson Linux 36.3 (JP 6.0). The primary symptoms include:

• Failure to load nvidia.ko and nvidia_modeset.ko modules
• Error messages during boot related to display probe failures
• Missing devices in /dev directory
• Inability to run CUDA containers

The issue occurs specifically with the custom-built kernel, while the default Jetpack kernel functions correctly. The problem appears to be related to the NVIDIA driver’s inability to access required BPMP (Boot and Power Management Processor) data.

Possible Causes

1. Incorrect Kernel Configuration: The custom kernel build may have missing or incorrect configurations essential for NVIDIA driver functionality.

2. BPMP Firmware Mismatch: The built kernel might not be compatible with the installed BPMP firmware version.

3. Module Loading Order: The sequence in which kernel modules are loaded could be affecting the NVIDIA driver initialization.

4. Missing or Incompatible Device Tree: The device tree used in the custom kernel build might lack necessary entries for proper driver initialization.

5. Conflicting Kernel Modules: As suggested in the error message, conflicting modules like nouveau, rivafb, nvidiafb, or rivatv might be interfering with NVIDIA driver loading.

6. Incomplete or Incorrect Build Process: Despite no apparent errors during the build, some critical steps or components might have been missed or incorrectly configured.

Troubleshooting Steps, Solutions & Fixes

1. Verify Kernel Configuration:

   • Compare your kernel .config file with the one from the working Jetpack kernel.
   • Ensure all NVIDIA-related options are correctly set.
   • Pay special attention to BPMP-related configurations.

2. Check BPMP Firmware Compatibility:

   • Verify the BPMP firmware version on your system:

     sudo cat /sys/firmware/bpmp/firmware_version
     

   • Ensure your kernel build is compatible with this version.

3. Adjust Module Loading Order:

   • Create or modify /etc/modules-load.d/modules.conf to specify the correct loading order.
   • Ensure NVIDIA modules are loaded after essential system modules.

4. Review and Update Device Tree:

   • Compare your device tree (DTB) with the one from the working Jetpack kernel.
   • Ensure all necessary NVIDIA and display-related entries are present.
   • Rebuild and flash the updated device tree if changes are made.

5. Check for Conflicting Modules:

   • List loaded modules: lsmod
   • Unload potentially conflicting modules:

     sudo rmmod nouveau rivafb nvidiafb rivatv
     

   • Attempt to load NVIDIA modules manually:

     sudo modprobe nvidia
     sudo modprobe nvidia_modeset
     

6. Rebuild Kernel and Modules:

   • Ensure you’re using the correct source code and build tools.
   • Follow NVIDIA’s official documentation meticulously.
   • Pay special attention to the Out-of-Tree (OOT) module build process.

7. Verify NVIDIA Driver Compatibility:

   • Check if the NVIDIA driver version is compatible with your custom kernel version.
   • Consider updating or downgrading the NVIDIA driver if necessary.

8. Analyze Kernel Logs:

   • Review dmesg output for more detailed error messages:

     dmesg | grep -i nvidia
     

   • Look for any BPMP-related errors or warnings.

9. Check CUDA Compatibility:

   • Verify that the CUDA version is compatible with your custom kernel.
   • Rebuild CUDA samples to ensure compatibility:

     cd /usr/local/cuda/samples
     sudo make
     

10. Restore Default Kernel:

    • If issues persist, consider temporarily reverting to the default Jetpack kernel to isolate the problem.

11. Consult NVIDIA Developer Forums:

    • If the issue remains unresolved, consider posting detailed logs and your kernel configuration on the NVIDIA Developer Forums for expert assistance.

Remember to backup important data before making significant system changes. If you’re unsure about any step, consult the official NVIDIA Jetson documentation or seek help from the developer community.