Starting text-generation-webui from jetson-containers on Jetson Orin Nano throws error

Issue Overview

Users are experiencing a series of permission-related errors while attempting to run the text-generation-webui on the Nvidia Jetson Orin Nano Developer Kit. The issue manifests during the execution of the command ./run.sh $(./autotag text-generation-webui), resulting in multiple error messages, including:

• Permission denied errors related to /tmp/nv_jetson_model.
• Errors indicating failures in memory management, such as NvRmMemMgrInit failed and NvRmGpuLibOpen failed.
• A Docker runtime error stating failed to create task for container: failed to create shim task.

The problem occurs in the context of running a specific application (text-generation-webui) using Docker, and it appears consistently across attempts. The user has noted that running the script with sudo produces similar errors, indicating that the issue may not solely be related to user permissions.

The system specifications provided include:

• Operating System: Ubuntu 22.04.3 LTS
• Kernel Version: 5.15.122-tegra
• CUDA Version: 12.2.140
• Docker Version: 25.0.3
• Jetpack Version: 6.0

The impact of this issue significantly hinders the user’s ability to utilize the Jetson Orin Nano for their intended application, leading to frustration and delays in development.

Possible Causes

The following potential causes may be contributing to the observed issues:

• Hardware Incompatibilities or Defects: If the Jetson Orin Nano hardware is not functioning correctly, it could lead to memory management errors.

• Software Bugs or Conflicts: Compatibility issues between different software components (e.g., CUDA, Docker, Jetpack) can result in runtime failures.

• Configuration Errors: Incorrect configurations in Docker or Jetpack settings may prevent proper access to necessary resources.

• Driver Issues: Outdated or improperly installed drivers can cause permission and functionality problems.

• Environmental Factors: Issues such as insufficient power supply or overheating can affect performance and lead to errors.

• User Errors or Misconfigurations: If the user has not followed setup instructions correctly, this could lead to permission-related issues.

Troubleshooting Steps, Solutions & Fixes

To resolve the issues encountered while running text-generation-webui, users can follow these troubleshooting steps:

1. Check Permissions:

   • Ensure that the user has appropriate permissions for accessing /tmp/nv_jetson_model. Use:

     ls -l /tmp/nv_jetson_model
     

2. Run with Elevated Privileges:

   • Although running with sudo did not resolve the issue for some users, it is still worth trying:

     sudo ./run.sh $(./autotag text-generation-webui)
     

3. Verify Jetpack Installation:

   • Ensure that Jetpack is correctly installed and configured. Users have reported success after reflashing their device with a compatible version (e.g., Jetpack 5.1.3). Follow these steps:
     • Reflash using Nvidia SDK Manager.
     • Select an appropriate version compatible with CUDA and Docker.

4. Check CUDA Compatibility:

   • Verify that the CUDA version is compatible with both Docker and Jetpack versions being used:

     nvcc --version
     

5. Test Other Containers:

   • Attempt running other known containers (e.g., l4t-jetpack) to isolate whether the issue is specific to text-generation-webui:

     docker run --rm nvidia/l4t-base:latest nvidia-smi
     

6. Update Docker and Dependencies:

   • Ensure that Docker and all related dependencies are up-to-date. Check for updates using:

     sudo apt update && sudo apt upgrade
     

7. Review Logs for Detailed Errors:

   • Check Docker logs for more detailed error messages that might provide insight into what is failing:

     docker logs <container_id>
     

8. Consider Environmental Factors:

   • Ensure that the power supply is adequate and that there are no overheating issues affecting performance.

9. Documentation and Community Support:

   • Refer to Nvidia’s official documentation for troubleshooting guidance and check community forums for similar issues and solutions.

10. Best Practices:

    • Regularly update all software components.
    • Document changes made during troubleshooting for future reference.

By following these steps, users should be able to diagnose and potentially resolve the issues they are facing with their Nvidia Jetson Orin Nano Developer Kit when attempting to run text-generation-webui.