Jetson Orin Nano Sudden Reboot During Video Processing with TensorRT

Issue Overview

Users report experiencing sudden reboots of the Nvidia Jetson Orin Nano when executing Python scripts that involve video recording and performing inference using TensorRT. The issue occurs specifically when using the X264 software encoder, which is necessary for achieving higher frame rates (above 10 FPS). The reboots typically happen after a few frames are processed, while the same setup works without issues when using MJPG encoding, albeit at a significantly lower frame rate of around 5 FPS.

Symptoms:

• Reboots: The device reboots almost instantly when running the script with X264 encoding.
• Performance: The script runs successfully with MJPG encoding but fails to meet the required frame rate.
• Inconsistent Behavior: Some devices reboot after running for over an hour when saving images as JPEG, while others do not exhibit this behavior.

Context:

• The problem arises during video processing tasks that require real-time inference.
• Users are unable to upgrade their JetPack version due to deployment constraints.

Hardware/Software Specifications:

• Device: Nvidia Jetson Orin Nano
• Camera: IMX477 CSI camera sensor
• Libraries:
  • cuda-python==12.3.0
  • tensorrt==8.5.2.2
  • numpy==1.17.4
  • OpenCV version 4.5.4

Impact:

The issue severely affects the ability to record high-quality video while performing inference, which is critical for user applications requiring continuous operation.

Possible Causes

1. Thermal Issues: Overheating may trigger thermal throttling or shutdowns.

   • Explanation: High processing loads from video encoding and inference can generate significant heat, potentially exceeding safe operating temperatures.

2. Power Supply Instability: Insufficient power delivery could lead to unexpected reboots.

   • Explanation: The demands of video processing and TensorRT inference may exceed what the power supply can provide.

3. Driver or Software Bugs: Incompatibilities or bugs in the software stack may cause instability.

   • Explanation: Issues within TensorRT or OpenCV libraries could lead to crashes under specific conditions.

4. Configuration Errors: Incorrect settings in the video pipeline or TensorRT engine configuration.

   • Explanation: Misconfigured parameters might lead to resource conflicts or memory issues.

5. User Errors: Misuse of APIs or incorrect implementation of scripts.

   • Explanation: Errors in code logic could inadvertently trigger system failures.

Troubleshooting Steps, Solutions & Fixes

1. Check System Status:

   • Run sudo tegrastats to monitor GPU temperature and performance metrics before running the script.
   • Look for any abnormal temperature spikes or resource usage patterns.

2. Test Different Encoders:

   • Consider using a different encoder (e.g., JPEG) temporarily to see if it stabilizes performance while maintaining acceptable frame rates.

3. Upgrade JetPack Version:

   • If possible, upgrade to JetPack version 6.0GA or later, as it may contain bug fixes and performance improvements.
   • Note that some users have reported success upgrading without physically reflashing SD cards.

4. Reduce Load on GPU:

   • Simplify the script by reducing the resolution or frame rate temporarily to determine if load reduction prevents reboots.
   • Example modification in the script:

     cap = cv2.VideoCapture('nvarguscamerasrc ! ... framerate=15/1 ! ...')
     

5. Monitor Power Supply:

   • Ensure that the power supply is adequate for high-load scenarios; consider testing with a different power source if available.

6. Isolate Components:

   • Test with different camera sensors or configurations to rule out hardware-specific issues.
   • If available, try using another Jetson Orin Nano unit to see if the issue persists.

7. Code Review and Debugging:

   • Review code for potential logical errors that could lead to crashes.
   • Utilize logging within the script to capture state before crashes occur.

8. Consult Documentation and Community Resources:

   • Refer to Nvidia’s official documentation for any specific notes on known issues with TensorRT and video processing on Jetson devices.
   • Engage with community forums for similar experiences and solutions shared by other users.

9. Consider Hardware Upgrades:

   • If consistent issues arise, evaluate whether upgrading to a model with better thermal management (like Orin NX) could be beneficial.

10. Best Practices for Future Prevention:

    • Regularly update software libraries and firmware as updates may resolve existing bugs.
    • Implement proper cooling solutions (e.g., fans) if operating in high-temperature environments.

By following these steps, users may be able to diagnose and remedy the sudden reboot issue effectively while optimizing their use of the Nvidia Jetson Orin Nano for video processing tasks with TensorRT.