Getting NanoLLM Up and Running on Nvidia Jetson Orin Nano

Issue Overview

Users are facing challenges while trying to run the NanoLLM library for video streaming demos on the Nvidia Jetson Orin Nano. The specific symptoms and context of the problem include:

• Symptoms: Users encounter ModuleNotFoundError when attempting to run a script from the NanoLLM library, specifically indicating that the module local_llm cannot be found. The error traceback points to issues with module imports during execution.

• Context: The issue arises after upgrading to JetPack 6 and while following tutorials for running ViLA models using Docker containers. Users are attempting to execute a command to run a specific model but are confused about the transition from local_llm to NanoLLM.

• Hardware/Software Specifications: The setup involves an Nvidia Jetson Orin Nano with JetPack 6, and users are utilizing Docker containers to run the NanoLLM library.

• Frequency: This appears to be a common issue among users transitioning from local_llm to NanoLLM, particularly when using different model versions.

• Impact on User Experience: The inability to run the desired models can significantly hinder development efforts and experimentation with generative AI projects on the Jetson platform.

Possible Causes

Several factors may contribute to the issues described:

• Missing Dependencies: The transition from local_llm to NanoLLM may not have included all necessary dependencies, leading to missing modules during execution.

• Incomplete Transition: Lingering references to local_llm in the codebase could cause confusion and errors when users attempt to run scripts that expect a fully transitioned library.

• Memory Limitations: Users have reported that running certain models, particularly larger ones, may exceed the available RAM (8GB) on the Orin Nano, causing hangs or crashes during execution.

• Configuration Issues: Users may not be using the correct commands or configurations needed for specific models, particularly when switching between different versions of ViLA models.

Troubleshooting Steps, Solutions & Fixes

To resolve issues related to getting NanoLLM up and running on the Nvidia Jetson Orin Nano, follow these steps:

1. Verify Installation and Dependencies:

   • Ensure that all dependencies for NanoLLM are correctly installed. If you suspect missing modules, reinstall or update the library:

     pip install --upgrade nano_llm
     

2. Check for Module Availability:

   • Confirm whether local_llm is still referenced in your code or scripts. If it is, update your scripts to use nano_llm instead.

3. Run Updated Example Scripts:

   • Use example scripts provided in the updated repository. For instance, try running:

     jetson-containers run $(autotag nano_llm) \
       python3 -m nano_llm.chat --api=mlc \
       --model Efficient-Large-Model/VILA-2.7b \
       --max-context-len 768 \
       --max-new-tokens 128 \
       --prompt /data/prompts/images.json 
     

4. Test Different Models:

   • If you experience memory issues or hangs with larger models (e.g., VILA-7B), consider testing smaller models first or running in console mode without video streaming:

     jetson-containers run $(autotag nano_llm) \
       python3 -m nano_llm.chat --api=mlc \
       --model liuhaotian/llava-v1.6-vicuna-7b
     

5. Monitor System Resources:

   • Keep an eye on system memory usage while running models. Use tools like htop or free -h to monitor RAM usage and ensure you are not exceeding available resources.

6. Consult Documentation and Updates:

   • Regularly check for updates in the NanoLLM documentation or GitHub repository for any changes or fixes related to module references and usage instructions.

7. Community Support:

   • Engage with community forums or GitHub issues for additional support if problems persist. Other users may have encountered similar issues and can provide insights or solutions.

8. Unresolved Aspects:

   • Users may still face unique challenges based on their specific configurations or setups that are not covered in common solutions. Further investigation may be required if standard troubleshooting does not resolve the problem.

By following these steps, users should be able to troubleshoot and successfully run NanoLLM on their Nvidia Jetson Orin Nano for video streaming demos and other applications.