CUDA Error: CUBLAS_STATUS_NOT_INITIALIZED on Jetson Orin Nano

Issue Overview

Users are experiencing a RuntimeError related to CUDA when attempting to run the YOLOv7 object detection model on the Nvidia Jetson Orin Nano 4GB. The specific error message reads: "CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)." This issue arises during the execution of a Python script using PyTorch with the command:

python3 detect.py --weights yolov7.pt --conf 0.25 --img-size 640 --source resized.jpg --device 0

Symptoms

• The script fails to load the model, throwing a CUDA-related error.
• Users have confirmed that CUDA is recognized as available by running torch.cuda.is_available(), which returns True.

Context

• The issue occurs specifically when running object detection tasks using YOLOv7.
• Users have reported trying various versions of PyTorch (2.1.0a0, 2.1.1, and 2.0.0) and TorchVision (0.16.0, 0.16.1, 0.16.2) without resolution.
• The JetPack version installed is 5.1.1.

Impact

This problem significantly hinders the ability to perform real-time object detection, affecting user experience and functionality in applications relying on GPU acceleration.

Possible Causes

1. Hardware Limitations: Insufficient GPU memory could lead to initialization failures during model loading.

   • Users have noted that similar errors may occur when the device runs out of memory.

2. Software Bugs: Incompatibilities between different versions of PyTorch and TorchVision can cause runtime errors.

   • Users reported switching versions without success.

3. Configuration Errors: Incorrect installation or configuration of the CUDA toolkit or PyTorch can lead to such errors.

   • Ensuring that the correct versions are installed for the JetPack version is crucial.

4. Driver Issues: Problems with GPU drivers can prevent proper initialization of CUDA resources.

   • Users should verify that the appropriate drivers are installed and functioning.

5. Environmental Factors: External factors such as power supply issues or overheating may impact performance.

   • Monitoring system temperatures and power stability is recommended.

6. User Errors: Misconfigurations in code or environment settings can lead to unexpected errors.

   • Double-checking environment variables and installation paths may help.

Troubleshooting Steps, Solutions & Fixes

Step-by-Step Instructions

1. Check GPU Memory Usage:

   • Use jtop or nvidia-smi to monitor GPU memory usage before running your script.
   • If memory is low, consider reducing batch sizes or image sizes in your script.

2. Verify CUDA Installation:

   • Run nvcc --version to confirm that CUDA is properly installed and matches your JetPack version.

3. Test with DeviceQuery:

   • Clone the CUDA samples repository and run the deviceQuery utility:

     git clone https://github.com/NVIDIA/cuda-samples.git
     cd cuda-samples/Samples/1_Utilities/deviceQuery
     make
     ./deviceQuery
     

   • This will help verify if the GPU is functioning correctly.

4. Reinstall PyTorch with Correct Versions:

   • Ensure you are using compatible versions of PyTorch and TorchVision for your JetPack:

     export TORCH_INSTALL=https://developer.download.nvidia.cn/compute/redist/jp/v511/pytorch/torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl
     pip install --no-cache-dir $TORCH_INSTALL
     

5. Check for Conflicting Packages:

   • If you have previously installed conflicting packages, consider purging them:

     sudo apt purge <package-name>
     

6. Reflash Jetson Orin Nano:

   • If issues persist after trying the above steps, consider reflashing your device with a fresh installation of JetPack via SDK Manager to ensure all components are correctly installed.

7. Monitor System Temperature and Power Supply:

   • Ensure that your device is not overheating and that it has a stable power supply during operation.

Recommended Approach

If multiple users reported success with a particular solution, such as reinstalling PyTorch or checking memory usage, it should be highlighted as a recommended approach for troubleshooting this issue.

Additional Resources

• NVIDIA Developer Forums
• JetPack Documentation
• PyTorch Installation Guide for Jetson

Unresolved Aspects

Some users have noted that despite following troubleshooting steps, they continue to encounter issues related to CUDA initialization, indicating potential deeper compatibility problems with specific versions of libraries or hardware configurations that may require further investigation by NVIDIA support teams or community forums.