System throttled due to Over-current on Jetson Orin Nano

Issue Overview

Users of the Jetson Orin Nano are experiencing system throttling due to over-current events, particularly when running resource-intensive applications like isaac_ros_dnn_stereo_depth. The issue manifests as OC (over-current) events, specifically on Channel 3 (OC3), which represents VDD_IN instant power. This problem occurs even when the device is running in 15W power mode, suggesting that the current power configuration may not be sufficient for the workload.

Possible Causes

1. Instantaneous high current draw: The application (isaac_ros_dnn_stereo_depth) may be causing sudden spikes in power consumption that exceed the set limits.

2. Inadequate power configuration: The default power settings may not be optimized for the specific workload, leading to over-current events.

3. Hardware limitations: The custom board being used might have different power characteristics compared to the official dev kit.

4. Software/firmware mismatch: Incompatibilities between the JetPack version (5.1.2) and the custom board could be contributing to the issue.

Troubleshooting Steps, Solutions & Fixes

1. Verify the current power configuration:
   Run the following command to check the current power mode:

   nvpmodel -q --verbose
   

2. Create a custom power mode:
   Add the following configuration to /etc/nvpmodel.conf:

   < POWER_MODEL ID=2 NAME=Custom >
   CPU_ONLINE CORE_0 1
   CPU_ONLINE CORE_1 1
   CPU_ONLINE CORE_2 1
   CPU_ONLINE CORE_3 1
   CPU_ONLINE CORE_4 1
   CPU_ONLINE CORE_5 1
   FBP_POWER_GATING FBP_PG_MASK 2
   TPC_POWER_GATING TPC_PG_MASK 240
   GPU_POWER_CONTROL_ENABLE GPU_PWR_CNTL_EN on
   CPU_A78_0 MIN_FREQ 729600
   CPU_A78_0 MAX_FREQ 1190400
   CPU_A78_1 MIN_FREQ 729600
   CPU_A78_1 MAX_FREQ 1190400
   GPU MIN_FREQ 0
   GPU MAX_FREQ 624750000
   GPU_POWER_CONTROL_DISABLE GPU_PWR_CNTL_DIS auto
   EMC MAX_FREQ 0
   

3. Apply the new power mode:
   Execute the following commands:

   sudo nvpmodel -m 2
   sudo reboot
   

4. Verify the new power configuration:
   After reboot, run:

   nvpmodel -q --verbose
   

5. Monitor power consumption:
   Use the following command to check for OC events:

   grep "" /sys/class/hwmon/hwmon*/oc*
   

6. Analyze power logs:
   If the issue persists, collect and analyze power logs to identify specific components or operations causing high current draw.

7. Adjust application resource usage:
   If possible, optimize the isaac_ros_dnn_stereo_depth application to reduce instantaneous power consumption.

8. Check for firmware updates:
   Ensure that you are using the latest firmware and JetPack version compatible with your custom board.

9. Consider hardware modifications:
   If the issue persists, you may need to evaluate the power delivery system of your custom board and potentially make hardware modifications to handle the power requirements of your application.

10. Consult NVIDIA support:
    If none of the above steps resolve the issue, reach out to NVIDIA support with detailed logs and information about your custom board for further assistance.