Troubleshooting LLaVA Demo on Nvidia Jetson Orin Nano Dev Board

Issue Overview

Users are experiencing difficulties when attempting to run the LLaVA (Large Language and Vision Assistant) demo on the Nvidia Jetson Orin Nano Developer Board. The specific problem occurs when executing a Python script to run the video query functionality of the nano_llm package. The script abruptly terminates, suggesting a potential memory-related issue on the Orin Nano platform.

Possible Causes

1. Insufficient Memory: The Jetson Orin Nano may be running out of available memory when trying to load and run the large language model (VILA1.5-3b).

2. Resource Allocation: The default system configuration might not be optimized for running memory-intensive AI models, leading to resource constraints.

3. Software Conflicts: There could be conflicts between the installed packages or dependencies required for running the LLaVA demo.

4. Hardware Limitations: The Jetson Orin Nano’s specifications might be insufficient for the particular model or configuration being used.

Troubleshooting Steps, Solutions & Fixes

1. Disable ZRAM:
   ZRAM is a compressed RAM disk that can sometimes interfere with memory-intensive applications. Disabling it may free up more physical memory for the LLaVA demo.

2. Mount SWAP:
   Adding swap space can provide additional virtual memory to the system, potentially alleviating out-of-memory issues.

3. Disable Desktop UI:
   Turning off the graphical user interface can free up significant system resources for AI tasks.

To implement these optimizations, follow the steps outlined in the Jetson Containers documentation:

# Navigate to the appropriate directory
cd path/to/jetson-containers

# Follow the setup instructions
cat docs/setup.md

4. Test with Terminal-Based Programs:
   Before running the full video query demo, try running simpler terminal-based programs to isolate the issue:

# Run the chat example
python3 -m nano_llm.chat

# Run the vision example
python3 -m nano_llm.vision.example

If these programs run successfully, it may indicate that the issue is specific to the video query functionality or related to the additional resources required for video processing.

5. Reduce Model Size or Complexity:
   If the issue persists, consider using a smaller or less complex model. For example, you could try a different model from the Efficient-Large-Model collection that may have lower memory requirements.

6. Update Software and Drivers:
   Ensure that all software components, including the Jetson system software, CUDA toolkit, and any relevant drivers, are up to date. This can sometimes resolve compatibility issues or bugs that may be causing memory-related problems.

7. Monitor Resource Usage:
   Use system monitoring tools to observe memory usage, CPU load, and GPU utilization while running the demo. This can provide insights into where the bottleneck occurs:

# Monitor system resources
htop

# Monitor GPU usage
tegrastats

8. Adjust Script Parameters:
   Modify the script parameters to reduce memory usage. For example:

python3 -m nano_llm.agents.video_query --api=mlc \
--model Efficient-Large-Model/VILA1.5-3b \
--max-context-len 128 \  # Reduced from 256
--max-new-tokens 16 \    # Reduced from 32
--video-input /dev/video0 \
--video-output webrtc://@:8554/output \
--nanodb /data/nanodb/coco/2017

9. Check for Memory Leaks:
   If the issue occurs after prolonged use, there might be a memory leak in the application. Consider using memory profiling tools to identify any potential leaks.

10. Consult Nvidia Developer Forums:
    If the problem persists after trying these solutions, consider posting a detailed description of your issue, including system specifications and steps you’ve already taken, on the Nvidia Developer Forums for more specialized assistance.