CUDA Failure on Nvidia Jetson Orin Nano Dev Board with L4T 36.3.0 and JetPack 6.0

Issue Overview

Users are experiencing CUDA failures when running the deviceQuery sample on Nvidia Jetson Orin Nano modules (both 4GB and 8GB variants) with a custom carrier board. The issue occurs specifically with L4T 36.3.0 and JetPack 6.0. When executing ./deviceQuery, an error message is displayed indicating a CUDA driver version mismatch. Interestingly, the same setup works correctly with an Orin NX module, suggesting a potential compatibility issue specific to the Orin Nano.

The error message states:

CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

This problem impacts the ability to use CUDA-enabled applications and development tools on the affected Orin Nano devices, potentially hindering GPU-accelerated tasks and development workflows.

Possible Causes

1. Driver Version Mismatch: The error message explicitly mentions a version mismatch between the CUDA driver and runtime. This could be due to incompatible or incorrectly installed drivers for the Orin Nano.

2. Firmware Incompatibility: The Orin Nano may require a specific firmware version that is not properly aligned with the installed L4T and JetPack versions.

3. Hardware-Specific Issue: Since the problem occurs only on Orin Nano modules and not on Orin NX, there might be a hardware-specific compatibility issue or limitation with the current software stack.

4. Custom Carrier Board Compatibility: Although the custom carrier board works with Orin NX, it’s possible that it has some compatibility issues specific to the Orin Nano modules.

5. Incorrect CUDA Installation: The CUDA toolkit or related components might not be correctly installed or configured for the Orin Nano modules.

6. Power Management Issues: The Orin Nano might have different power requirements or management features that are not properly addressed in the current setup.

Troubleshooting Steps, Solutions & Fixes

1. Verify CUDA Installation:

   • Ensure that CUDA is properly installed on the Orin Nano.
   • Run the following command to check CUDA version:

     nvcc --version
     

   • Verify that the installed CUDA version matches the one expected for L4T 36.3.0 and JetPack 6.0.

2. Check Driver Versions:

   • Use the following command to check the NVIDIA driver version:

     cat /proc/driver/nvidia/version
     

   • Compare this with the CUDA version to ensure compatibility.

3. Update L4T and JetPack:

   • Check for any available updates for L4T and JetPack that might address Orin Nano compatibility issues.
   • Follow the official NVIDIA documentation to perform the update if newer versions are available.

4. Reinstall CUDA Toolkit:

   • Uninstall the current CUDA installation:

     sudo apt-get --purge remove "*cuda*" "*cublas*" "*cufft*" "*cufile*" "*curand*" "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*"
     

   • Reboot the system.
   • Reinstall CUDA toolkit following NVIDIA’s official instructions for Jetson devices.

5. Check Power Management Settings:

   • Investigate power management settings specific to Orin Nano.
   • Use the nvpmodel tool to check and adjust power modes:

     sudo nvpmodel -q
     sudo nvpmodel -m <mode_number>
     

6. Verify Carrier Board Compatibility:

   • Review the custom carrier board specifications and ensure it meets all requirements for Orin Nano modules.
   • Check for any firmware updates or configuration changes needed for the carrier board to fully support Orin Nano.

7. Examine System Logs:

   • Check system logs for any relevant error messages:

     dmesg | grep -i cuda
     journalctl -xe | grep -i nvidia
     

8. Compare with Reference Hardware:

   • If possible, test the Orin Nano module on a reference carrier board to isolate whether the issue is related to the custom carrier board or the module itself.

9. Contact NVIDIA Support:

   • If the issue persists after trying the above steps, consider reaching out to NVIDIA developer support for Jetson-specific guidance.
   • Provide them with detailed information about your setup, including:
     • Exact model numbers of Orin Nano modules
     • Custom carrier board specifications
     • Complete output of deviceQuery
     • Any relevant system logs

10. Community Forums:

    • Search NVIDIA Developer Forums for similar issues reported by other Orin Nano users.
    • Post your specific problem, including all troubleshooting steps attempted, to seek advice from the community.

Remember to document all changes and test results throughout the troubleshooting process. This will help in identifying patterns and potentially uncovering the root cause of the CUDA failure on Orin Nano modules.