System Throttled Due to Over-Current Issue Faced (Jetpack 6)

Issue Overview

Users have reported encountering a “System throttled due to over-current” error while running TensorRT engine files in Docker containers on the Nvidia Jetson Orin Nano Dev board. This issue manifests as a warning symbol near the MAXN power mode indicator on the screen, accompanied by high CPU usage (89-100%) and a system temperature around 60°C. The problem typically occurs during the execution of inference scripts, particularly when attempting to maximize performance. Users express concern about potential consequences of this throttling, such as reduced performance or increased temperatures, and some have noted that the Jetson device fails to detect the GPU despite showing GPU load in the Jetson Power GUI.

Possible Causes

1. Protection Mechanism: The throttling is a built-in safety feature designed to prevent damage from excessive power draw.
2. Hardware Limitations: The current hardware configuration may not support the required power levels for high-performance tasks.
3. Software Bugs: Issues within the JetPack version (6) or Docker container could lead to improper resource management.
4. Configuration Errors: Incorrect settings in power management or Docker configurations may exacerbate power draw issues.
5. Driver Issues: Outdated or incompatible drivers could prevent proper GPU detection and utilization.
6. Environmental Factors: High ambient temperatures or inadequate cooling solutions can contribute to overheating and subsequent throttling.
7. User Errors: Misconfigurations in Docker settings or TensorRT engine parameters might lead to excessive resource consumption.

Troubleshooting Steps, Solutions & Fixes

1. Diagnosing the Problem:

   • Use nvidia-smi to check GPU status and utilization.
   • Monitor CPU and GPU temperatures using system monitoring tools.

2. Adjust Power Modes:

   • To change power modes, access the Jetson Power GUI and select a lower power mode to prevent throttling.
   • Command example:

     sudo nvpmodel -m <mode_number>
     

   • Recommended modes for reduced power draw include modes 0 (MAXN) or 1 (5W).

3. Isolate the Issue:

   • Test with different Docker images or run TensorRT outside of Docker to compare performance.
   • Verify if the issue persists with minimal workloads.

4. Update Drivers and Software:

   • Ensure that all drivers are up-to-date by checking for updates in the SDK Manager.
   • Consider downgrading to JetPack 5.x if issues persist with JetPack 6.

5. Cooling Solutions:

   • Ensure adequate cooling; consider adding heatsinks or fans if temperatures exceed safe operating limits.

6. Check Configuration Files:

   • Review Docker configuration settings for resource limits that may be set too high.
   • Example configuration adjustments in docker-compose.yml:

     deploy:
       resources:
         limits:
           cpus: '0.5'
           memory: 512M
     

7. Best Practices for Future Prevention:

   • Regularly monitor system performance metrics during high-load operations.
   • Keep software updated and review Nvidia’s documentation for any changes regarding power management.

8. Documentation and Resources:

   • Refer to Nvidia’s official documentation for Jetson Orin Nano on power management and driver updates.
   • For further assistance, consider engaging with community forums or Nvidia support channels.

By following these steps, users can effectively address the “System throttled due to over-current” issue on their Nvidia Jetson Orin Nano Dev board while optimizing their setup for better performance during intensive tasks.