Nsight Compute (ncu) Compatibility Issue with Nvidia Jetson Orin Nano

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing difficulties when attempting to use Nsight Compute (ncu) for profiling their CUDA applications. The primary symptom is that when running the ncu command to profile a program, users receive an error message stating that profiling is not supported on their device. This issue occurs during the setup and usage of development tools on the Jetson Orin Nano platform.

Specifically, when executing the command ncu ./"program", the following error is encountered:

==ERROR== Profiling is not supported on device 0. To find out supported GPUs refer --list-chips option.

Furthermore, when users attempt to list supported chips using ncu --list-chips, the Orin Nano’s chip (ga10b) is not included in the output, leading to confusion about compatibility.

This problem impacts the ability of developers to profile and optimize their CUDA applications on the Jetson Orin Nano, potentially hindering development and performance tuning efforts.

Possible Causes

1. Outdated Software Version: The installed version of Nsight Compute may not include support for the Jetson Orin Nano’s ga10b chip.

2. Incomplete JetPack Installation: The JetPack SDK might not have been fully or correctly installed, leading to missing components necessary for Nsight Compute to function properly.

3. Configuration Issues: System configuration files or environment variables may not be set correctly to recognize the Orin Nano’s hardware.

4. Driver Incompatibility: The CUDA drivers installed on the system might not be compatible with the version of Nsight Compute being used.

5. Hardware Recognition Problem: The system may not be correctly identifying the Orin Nano’s hardware, causing Nsight Compute to fail in recognizing the device.

Troubleshooting Steps, Solutions & Fixes

1. Verify JetPack Version:
   Ensure that you are using JetPack 5.1.2 or later, which includes support for the Orin Nano.

   sudo apt-cache show nvidia-jetpack
   

2. Update Nsight Compute:
   If you’re using an older version of Nsight Compute, update it to the latest version compatible with your JetPack installation.

   sudo apt update
   sudo apt install nvidia-nsight-compute
   

3. Check CUDA Installation:
   Verify that CUDA is correctly installed and recognized:

   nvcc --version
   

4. Verify Chip Support:
   Run the following command to check if ga10b is listed:

   sudo /opt/nvidia/nsight-compute/2022.2.1/ncu --list-chips
   

   If ga10b is not listed, you may need to update your Nsight Compute installation.

5. Reinstall JetPack:
   If the issue persists, consider performing a clean installation of JetPack 5.1.2 or later to ensure all components are correctly installed.

6. Use Nsight Systems as an Alternative:
   If Nsight Compute (ncu) is not working, you can use Nsight Systems (nsys) for profiling, which seems to be functional on the Orin Nano:

   nsys profile ./your_application
   

7. Check System Logs:
   Examine system logs for any error messages related to CUDA or Nsight Compute:

   sudo dmesg | grep -i cuda
   sudo dmesg | grep -i nsight
   

8. Verify Environment Variables:
   Ensure that the necessary environment variables are set correctly:

   echo $PATH
   echo $LD_LIBRARY_PATH
   

   Make sure that the CUDA and Nsight Compute paths are included.

9. Contact Nvidia Support:
   If none of the above steps resolve the issue, consider reaching out to Nvidia’s support team or posting on their official developer forums for more specialized assistance.

10. Check for Known Issues:
    Review the Nvidia Developer website and forums for any known issues or bugs related to Nsight Compute on the Jetson Orin Nano platform.

Remember to reboot your system after making any significant changes to ensure all updates take effect. If you continue to experience issues with Nsight Compute, consider using alternative profiling tools or waiting for future updates that may address compatibility with the Orin Nano.