Running Docker with CUDA >=11.8

Issue Overview

Users are experiencing issues when attempting to run Docker containers with CUDA version 11.8 on the Jetson Orin Nano Dev board using L4T (Linux for Tegra) version 35.4. The specific error message encountered is:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier: requirements not met: unsatisfied condition: cuda>=11.8 (cuda=11.4): unknown.

This indicates that the container is unable to meet the required CUDA version, which leads to confusion about whether only CUDA 11.4 images can run on L4T 35.4. Users have reported that even after installing CUDA 11.8, the issue persists, suggesting a deeper compatibility problem.

The context of the problem arises during the setup of Docker containers specifically designed for CUDA applications, where users expect compatibility between the host system and the containerized environment.

Possible Causes

• Hardware Incompatibilities: The Jetson Orin Nano may have specific hardware requirements that are not met by certain Docker images.

• Software Bugs or Conflicts: There may be bugs in the Docker or NVIDIA Container Runtime that prevent proper execution of containers requiring higher CUDA versions.

• Configuration Errors: The user may not have configured Docker or the NVIDIA runtime correctly for use with CUDA 11.8.

• Driver Issues: Incompatibility between installed drivers and the requested CUDA version could lead to failure in launching containers.

• Environmental Factors: Issues such as insufficient power supply or overheating could affect performance and compatibility.

• User Errors or Misconfigurations: Users might be using Docker images intended for desktop GPUs rather than those optimized for Jetson devices.

Troubleshooting Steps, Solutions & Fixes

1. Verify Docker Image Compatibility:

   • Ensure that you are using a Docker image compatible with Jetson devices. For instance, avoid using images designed for desktop GPUs (dGPU).
   • Recommended image for Jetson devices is nvidia/cuda:11.8.0-devel-ubuntu20.04.

2. Check Installed CUDA Version:

   • Run the following command to verify the installed CUDA version:

     nvcc --version
     

   • Ensure that it reflects CUDA 11.8.

3. Set Up Compatibility Libraries:

   • If you encounter compatibility issues, export the compatibility folder using:

     export LD_LIBRARY_PATH=/usr/local/cuda-11.8/compat
     

4. Test CUDA Outside of Docker:

   • Before running in a container, confirm that CUDA 11.8 works properly outside of Docker by executing a simple CUDA program.

5. Update NVIDIA Drivers and Runtime:

   • Ensure that you have the latest NVIDIA drivers and runtime installed.
   • Check for updates on the NVIDIA Developer website.

6. Inspect Docker Configuration:

   • Verify your Docker configuration settings, particularly those related to NVIDIA runtime integration.
   • Check if you have set up Docker to use NVIDIA as a default runtime in /etc/docker/daemon.json:

     {
       "runtimes": {
         "nvidia": {
           "path": "nvidia-container-runtime",
           "runtimeArgs": []
         }
       },
       "default-runtime": "nvidia"
     }
     

7. Consult Documentation and Community Resources:

   • Refer to the official NVIDIA documentation for guidance on setting up CUDA with Docker.
   • Review community forums for similar issues and solutions shared by other users.

8. Testing Different Configurations:

   • If problems persist, try running different configurations (e.g., older versions of CUDA or different base images) to isolate the issue.

9. Log Analysis:

   • Review logs generated by Docker and NVIDIA Container Runtime for any additional error messages that could provide insights into the failure.

10. Best Practices:

    • Regularly update both your Jetson software stack and your Docker images.
    • Always verify compatibility between your host system’s software versions and those required by your container images.

By following these troubleshooting steps, users can systematically address issues related to running Docker containers with CUDA on the Jetson Orin Nano Dev board while ensuring optimal performance and compatibility.