cuDNN Version Reporting Issues on Nvidia Jetson Orin Nano Dev Board

Issue Overview

Users of the Nvidia Jetson Orin Nano Dev board are experiencing issues with the cuDNN version reported by the jtop utility. Specifically, despite having installed cuDNN version 8.9.7 alongside CUDA version 11.8, jtop incorrectly displays the cuDNN version as 1.0.0. This issue occurs during the verification process after installation, and users have reported similar problems when running example code from the NVIDIA cuDNN documentation, leading to errors. The inconsistency in version reporting raises concerns about compatibility between CUDA and cuDNN versions, particularly since some users have noted that pre-installed packages may not align with their manually installed versions.

Symptoms:

• jtop shows cuDNN version as 1.0.0 instead of the installed version (8.9.7).
• Errors occur when running example code from the NVIDIA documentation.

Context:

• The issue arises after manual installations of CUDA and cuDNN on JetPack 5 and JetPack 6.
• Users have downgraded CUDA to version 11.4 and cuDNN to version 8.6.0 but still see incorrect version reporting.

Impact:

This problem affects user experience by creating confusion regarding the installed libraries’ compatibility and functionality, potentially hindering development efforts.

Possible Causes

1. Hardware Incompatibilities: Certain configurations of the Jetson Orin Nano may not fully support newer versions of CUDA and cuDNN.
2. Software Bugs: There could be bugs in the jtop utility or the underlying libraries that prevent accurate reporting.
3. Configuration Errors: Incorrect environment variables or paths may lead to misreporting of installed software versions.
4. Driver Issues: Outdated or incompatible drivers could affect how libraries are loaded and reported.
5. Manual Installation Conflicts: Manual installations may conflict with pre-installed packages, leading to inconsistencies.
6. Compatibility Issues: Specific combinations of CUDA and cuDNN versions may not be compatible with each other or with other installed software (e.g., TensorRT, PyTorch).

Troubleshooting Steps, Solutions & Fixes

1. Verify Installation Paths:

   • Check if the correct paths for CUDA and cuDNN are set in your environment variables:

     echo $CUDA_HOME
     echo $LD_LIBRARY_PATH
     

   • Ensure they point to your manually installed versions.

2. Check Installed Versions:

   • Use the following command to verify installed cuDNN files:

     ls /usr/local/cuda/lib64 | grep libcudnn
     

   • Ensure that the expected version files (e.g., libcudnn.so.x.x.x) are present.

3. Reinstall cuDNN:

   • If discrepancies persist, consider reinstalling cuDNN:

     sudo apt-get remove --purge libcudnn*
     sudo apt-get install libcudnn8=8.x.x.x-1+cuda11.x
     

4. Compatibility Check:

   • Confirm compatibility between your installed versions of CUDA, cuDNN, and any other frameworks like TensorRT or PyTorch.
   • Refer to NVIDIA’s compatibility documentation for guidance.

5. Update Drivers:

   • Ensure that you are using the latest drivers for your Jetson Orin Nano by checking for updates through SDK Manager.

6. Use Official Packages:

   • If you manually installed CUDA or cuDNN, consider using official packages provided by NVIDIA through SDK Manager to avoid conflicts.

7. Testing Different Versions:

   • If issues persist, try downgrading to known compatible versions (e.g., CUDA 11.4 with cuDNN 8.6) to see if this resolves the reporting issue.

8. Consult Community Forums:

   • Engage with the NVIDIA developer forums for additional insights or shared experiences from other users facing similar issues.

Recommended Approach

Several users have found success by reverting to earlier versions of CUDA and cuDNN that are known to work well together, particularly CUDA 11.4 with cuDNN 8.6, as indicated in user replies within the forum discussion.

Unresolved Aspects

The root cause of why jtop reports incorrect versions remains unclear, particularly if it is a bug in the utility itself or a deeper compatibility issue within the Jetson ecosystem that requires further investigation from NVIDIA developers or community support channels.

By following these troubleshooting steps and solutions, users should be able to diagnose and potentially resolve their issues related to cuDNN version reporting on their Nvidia Jetson Orin Nano Dev board effectively.