MiniGPT-4 on Jetson Orin Nano 8Gb Dev Kit Not Working

Issue Overview

Users have reported difficulties running the MiniGPT-4 model on the Nvidia Jetson Orin Nano 8GB Dev Kit, particularly when following a tutorial for setup. The main symptoms include:

• Command Line Output: After executing the installation and running commands, users see extensive output related to model loading but ultimately return to the command line without any further action or errors, indicating that the application did not execute as expected.

• Memory Issues: It appears that the board runs out of memory during the execution of the MiniGPT-4 model, which is a common issue given the resource constraints of the Orin Nano 8GB.

• Context of Failure: The problem occurs primarily during the execution of the MiniGPT-4 commands after successful installation via Docker. Users have successfully run other applications, such as "text-generation-webui," indicating that the issue is specific to MiniGPT-4.

• Frequency and Impact: This issue seems to be consistent among users attempting to run MiniGPT-4, significantly impacting user experience as it prevents them from utilizing this AI model effectively.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Limitations: The Orin Nano 8GB may not have sufficient memory to handle the demands of the MiniGPT-4 model, leading to memory exhaustion.

• Software Bugs or Conflicts: Updates to either the model or associated libraries (like llama.cpp) may introduce bugs or conflicts that could prevent proper execution.

• Configuration Errors: Incorrect configurations in Docker or within the application settings may lead to failures during execution.

• Driver Issues: Incompatibilities or outdated drivers could hinder performance and functionality.

• Environmental Factors: Factors such as power supply issues or overheating might impact performance.

• User Errors: Misconfigurations during setup or execution commands could also lead to failures.

Troubleshooting Steps, Solutions & Fixes

To address the issues experienced with running MiniGPT-4 on the Jetson Orin Nano, users can follow these troubleshooting steps and solutions:

1. Check Memory Usage:

   • Monitor memory usage while running commands using:

     free -h
     

   • If memory is low, consider implementing a swap file.

2. Create and Enable Swap Space:

   • Follow these steps to create a swap file:

     sudo fallocate -l 4G /swapfile
     sudo chmod 600 /swapfile
     sudo mkswap /swapfile
     sudo swapon /swapfile
     

   • Add it to /etc/fstab for persistence:

     /swapfile none swap sw 0 0
     

3. Disable ZRAM:

   • If ZRAM is enabled, consider disabling it temporarily to free up resources.

4. Run Alternative Models:

   • As suggested by other users, consider using smaller models like VILA-2.7b or VILA1.5-3b instead of MiniGPT-4. Use the following commands:

     jetson-containers run $(autotag nano_llm)
     python3 -m nano_llm.chat --api=mlc --model Efficient-Large-Model/VILA1.5-3b --max-context-len 256 --max-new-tokens 32
     

5. Reimage NVMe Card:

   • If problems persist, reimage your NVMe card and apply RAM optimization settings as discussed in forum replies.

6. Update Drivers and Software:

   • Ensure that all drivers and software are up-to-date by checking for updates in JetPack and relevant libraries.

7. Run Commands with Debugging:

   • Execute commands with additional logging or debugging information enabled to capture more detailed error messages:

     python3 -m nano_llm.chat --api=mlc --model Efficient-Large-Model/VILA1.5-3b --max-context-len 256 --max-new-tokens 32 --verbose
     

8. Consult Documentation:

   • Refer to official documentation for Jetson containers and any specific requirements for running AI models on Jetson hardware.

9. Community Support:

   • Engage with community forums for additional support and shared experiences from other users facing similar issues.

By following these steps, users should be able to troubleshoot their issues effectively and find a suitable workaround for running AI models on their Nvidia Jetson Orin Nano Dev Kit.