CUDA Error: No Kernel Image Available for Execution on Jetson Orin Nano

Issue Overview

Users deploying the VMamba (Visual State Space Models) on the Nvidia Jetson Orin Nano development board are encountering a CUDA error. The specific error message states: "no kernel image is available for execution on the device." This issue occurs when attempting to run the VMamba model using a Python script. The problem persists even when setting the CUDA_LAUNCH_BLOCKING environment variable to 1. The issue was observed on a system running Jetpack 6.0 rev.1, installed via SDK Manager 2.1.0.11682 x86_64.

Possible Causes

1. Incompatible CUDA architecture: The CUDA kernels in the installed packages may not be compiled for the specific compute capability of the Jetson Orin Nano.

2. Incorrect PyTorch installation: The PyTorch installation might lack proper CUDA support for the Jetson platform.

3. Mismatched CUDA versions: There could be a mismatch between the CUDA version used to compile the PyTorch wheels and the CUDA version on the Jetson device.

4. Incomplete dependency installation: Some required CUDA-enabled dependencies might be missing or incorrectly installed.

Troubleshooting Steps, Solutions & Fixes

1. Verify PyTorch and TorchVision installation:

   • Ensure that you are using the prebuilt wheels specifically designed for Jetson platforms.
   • Install PyTorch and TorchVision using the following commands:

     wget https://nvidia.box.com/shared/static/ssf2v7pf5i245fk4i0q926hy4imzs2ph.whl -O torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl
     pip install torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl
     
     wget https://nvidia.box.com/shared/static/vdctxqejdda5qznh7bmsolt8rhtiwvrw.whl -O torchvision-0.15.1a0+d0d4b0b-cp38-cp38-linux_aarch64.whl
     pip install torchvision-0.15.1a0+d0d4b0b-cp38-cp38-linux_aarch64.whl
     

2. Set correct CUDA architecture flags:

   • When installing or compiling packages that depend on CUDA, ensure that the correct Compute Capability for the Jetson Orin Nano is specified.
   • For the Jetson Orin Nano, use the following architecture flags:

     arch=compute_87,code=sm_87
     

   • Reinstall the affected packages with these flags set.

3. Rebuild CUDA kernels:

   • If you have access to the source code of the VMamba repository, try rebuilding the CUDA kernels with the correct architecture flags.
   • Navigate to the directory containing the CUDA source files and run:

     nvcc -arch=sm_87 -code=sm_87 -c your_cuda_file.cu -o your_cuda_file.o
     

4. Check CUDA compatibility:

   • Verify that the installed CUDA version is compatible with your Jetson Orin Nano and the PyTorch version you’re using.
   • Run the following command to check the CUDA version:

     nvcc --version
     

5. Ensure all dependencies are correctly installed:

   • Review the VMamba repository’s requirements and ensure all dependencies are installed correctly for the Jetson platform.
   • Pay special attention to CUDA-enabled libraries and ensure they are compiled for the correct architecture.

6. Update Jetpack and CUDA drivers:

   • Ensure you have the latest Jetpack version installed on your Jetson Orin Nano.
   • Update CUDA drivers to the latest version compatible with your Jetpack installation.

7. Check for known issues:

   • Visit the VMamba GitHub repository and check for any known issues or pull requests related to Jetson compatibility.
   • Look for any specific instructions or workarounds for deploying on Jetson platforms.

8. Use NVIDIA’s official PyTorch installation guide:

   • If problems persist, follow NVIDIA’s official guide for installing PyTorch on Jetson platforms:
     Installing PyTorch for Jetson Platform – NVIDIA Docs

By following these steps, particularly setting the correct CUDA architecture flags (arch=compute_87, code=sm_87) and reinstalling the dependent packages, users should be able to resolve the "no kernel image is available for execution on the device" error when deploying VMamba on the Jetson Orin Nano.