Jetson Orin Nano Insanely Slow Inference Speed

Issue Overview

Users of the Nvidia Jetson Orin Nano Dev board, particularly those utilizing the 8GB model, have reported significantly slow inference speeds when running AI models, specifically with the Ollama framework. Symptoms include:

• Inconsistent Performance: Users are experiencing inference speeds of approximately 1 token per 10 to 15 seconds, which is markedly slower than expected. For comparison, similar models run on Raspberry Pi yield much faster performance, comparable to GPT-4 speeds.

• Context of Issue: The problem arises during the execution of inference tasks, even when attempting to utilize the GPU. Users have tested multiple software versions (R35.5.3 and R36.2) and are considering the latest release (R36.3) for potential improvements.

• Frequency: This issue appears to be consistent among multiple users, indicating it is not an isolated incident.

• Impact on User Experience: The slow inference speed renders the device nearly unusable for intended applications, leading to frustration and questions regarding hardware functionality or user error.

Possible Causes

Several potential causes for the slow inference speeds have been identified:

• Hardware Limitations: The Jetson Orin Nano may not be fully optimized for certain AI models, leading to slower performance than anticipated.

• Software Bugs or Conflicts: Issues within the Ollama framework or incompatibilities with specific versions of Jetson’s software could lead to degraded performance.

• Configuration Errors: Improper settings or configurations during setup may hinder the GPU’s ability to perform optimally.

• Driver Issues: Outdated or incorrect drivers could prevent the GPU from functioning at its full capacity.

• Environmental Factors: External conditions such as power supply stability or thermal management could affect performance.

• User Misconfigurations: Users may be misconfiguring their environments or not utilizing the appropriate settings for optimal performance.

Troubleshooting Steps, Solutions & Fixes

To address the slow inference speed on the Jetson Orin Nano, follow these comprehensive troubleshooting steps:

1. Check Software Version:

   • Ensure you are using the latest version of the Jetson software (currently R36.3). Update if necessary.
   • Command to check version:

     cat /etc/nv_tegra_release
     

2. Verify GPU Utilization:

   • Confirm that the GPU is being utilized during inference tasks.
   • Use the following command to monitor GPU usage:

     jtop
     

3. Test Different Models:

   • Experiment with various AI models to determine if the issue is model-specific.
   • Reference documentation or tutorials provided by Nvidia for recommended models that work efficiently with Jetson hardware.

4. Reconfigure Ollama Settings:

   • Review and adjust configuration settings in Ollama to ensure they are optimized for Jetson hardware.
   • Consult Ollama’s documentation for specific configuration tips related to Nvidia devices.

5. Driver Update:

   • Check for any available driver updates that may enhance performance.
   • Command to update drivers:

     sudo apt update && sudo apt upgrade
     

6. Isolation Testing:

   • Test with a different power supply or under different environmental conditions (cooler temperatures) to rule out external factors affecting performance.

7. Community Resources:

   • Engage with community forums or resources such as jetson-ai-lab.com for additional insights and shared experiences from other users facing similar issues.

8. Performance Expectations:

   • Understand that while users report that Orin Nano can achieve around 16 tokens/sec with optimized models like Llama2 7B, actual performance may vary based on specific configurations and workloads.

9. Document Findings:

   • Keep a log of changes made and their effects on performance to identify what works best for your setup.

10. Further Investigation:

    • If issues persist despite following these steps, consider reaching out to Nvidia support or community forums for advanced troubleshooting assistance.

By following these steps, users can systematically diagnose and potentially resolve slow inference speeds on the Nvidia Jetson Orin Nano Dev board, enhancing their overall experience with this powerful device.