Live LLaVA Not Working on Jetson Orin Nano 8G

Issue Overview

Users are experiencing difficulties running Live LLaVA on the Nvidia Jetson Orin Nano 8G developer board. The specific problem occurs when attempting to execute the Live LLaVA demo using the VILA-2.7b model. When users run the provided command, the system becomes unresponsive and eventually reboots automatically. This issue is preventing users from successfully implementing and utilizing the Live LLaVA functionality on their Jetson Orin Nano devices.

Possible Causes

1. Insufficient Memory: The Jetson Orin Nano 8G may not have enough available memory to handle the Live LLaVA demo with the VILA-2.7b model.

2. ZRAM Configuration: The default ZRAM settings might be interfering with the memory allocation required for the Live LLaVA demo.

3. Swap Space: Inadequate swap space could be limiting the system’s ability to manage memory effectively for this resource-intensive application.

4. Vision API Compatibility: The default vision API settings may not be compatible with the current system configuration or the VILA-2.7b model.

5. Model Quantization: The initial run of the model might be failing during the quantization process, which is necessary for optimal performance on the Jetson platform.

Troubleshooting Steps, Solutions & Fixes

1. Disable ZRAM and Mount Additional Swap

   To address potential memory issues, try disabling ZRAM and mounting additional swap space:

   sudo swapoff -a
   sudo systemctl disable nvzramconfig
   sudo systemctl mask nvzramconfig
   

   Then, create and mount a swap file:

   sudo fallocate -l 4G /mnt/4GB.swap
   sudo chmod 600 /mnt/4GB.swap
   sudo mkswap /mnt/4GB.swap
   sudo swapon /mnt/4GB.swap
   

   Add the following line to /etc/fstab to make the swap persistent:

   /mnt/4GB.swap  none  swap  sw 0  0
   

2. Use Alternative Vision API

   If the system is still getting stuck after addressing memory issues, try using the Hugging Face vision API instead of the default one:

   jetson-containers run $(autotag nano_llm) \
     python3 -m nano_llm.agents.video_query --api=mlc \
       --model Efficient-Large-Model/VILA-2.7b \
       --max-context-len 256 \
       --max-new-tokens 32 \
       --vision-api hf \
       --video-input /dev/video0 \
       --video-output webrtc://@:8554/output
   

3. Test with Sample Images

   Before attempting to use live video input, try running the model on some test images to ensure the quantization process completes successfully:

   jetson-containers run $(autotag nano_llm) \
     python3 -m nano_llm.agents.video_query --api=mlc \
       --model Efficient-Large-Model/VILA-2.7b \
       --max-context-len 256 \
       --max-new-tokens 32 \
       --vision-api hf \
       --prompt /data/prompts/images.json
   

4. Optimize System Resources

   Continue using the sudo init 3 command to boot into a console environment and reduce memory usage from graphical interfaces. Connect to the Jetson Orin Nano via SSH for running the Live LLaVA demo.

5. Check for Updates

   Ensure that you have the latest versions of the Jetson containers and the nano_llm package:

   jetson-containers pull nano_llm
   

6. Monitor System Resources

   While running the Live LLaVA demo, monitor system resources using tools like top or htop to identify any potential bottlenecks or excessive resource usage.

7. Adjust Model Parameters

   If the issue persists, try adjusting the model parameters such as reducing the max-context-len or max-new-tokens to lower the memory requirements:

   jetson-containers run $(autotag nano_llm) \
     python3 -m nano_llm.agents.video_query --api=mlc \
       --model Efficient-Large-Model/VILA-2.7b \
       --max-context-len 128 \
       --max-new-tokens 16 \
       --vision-api hf \
       --video-input /dev/video0 \
       --video-output webrtc://@:8554/output
   

8. Check for Hardware Issues

   If none of the above solutions work, consider checking for potential hardware issues. Ensure that the Jetson Orin Nano 8G is not overheating and that the power supply is stable and sufficient.

By following these troubleshooting steps and applying the suggested fixes, users should be able to resolve the Live LLaVA functionality issues on their Nvidia Jetson Orin Nano 8G developer boards. If problems persist, further investigation may be needed, potentially involving Nvidia support or the Jetson developer community.