Camera connected to Jetson Orin Nano stops after long test: Seeking solutions for nvargus-daemon errors

Issue Overview

Users of the Nvidia Jetson Orin Nano have reported issues with their connected cameras, specifically the IMX477 4K Camera from Arducam, ceasing to function after prolonged usage. The failure can occur unpredictably, with instances noted between 1 hour to 48 hours of continuous operation. The primary error is associated with the nvargus-daemon, where the system logs indicate a timeout in frame acquisition, specifically returning an Argus::STATUS_TIMEOUT status. The error messages also reference various states of invalidity and resource conflicts, indicating that the camera resources are already in use or that there is a timeout in processing frames. This issue appears to be exacerbated under heavy system load but can also occur during lighter tasks using GStreamer commands. Users have noted that similar setups on the Jetson Xavier NX do not exhibit this problem.

Relevant Hardware and Software Specifications:

• Jetson Module: P3767-0003 (No SD card slot model)
• Carrier Board: J401 Carrier Board from Seeed Studio
• Camera: IMX477 4K Camera from Arducam
• Jetpack Version: 5.1.2
• L4T Version: 35.4.1

Impact on User Experience:

The inability to maintain a stable camera connection significantly hampers the user experience, particularly for applications requiring continuous video feed or image capture, leading to potential data loss and operational disruptions.

Possible Causes

1. Hardware Incompatibilities or Defects:

   • The combination of the Jetson Orin Nano and specific camera models may not be fully compatible, leading to resource conflicts.

2. Software Bugs or Conflicts:

   • Issues within the nvargus-daemon or GStreamer could lead to improper handling of camera resources, resulting in timeouts.

3. Configuration Errors:

   • Incorrect settings in the camera or system configuration may prevent proper operation under sustained loads.

4. Driver Issues:

   • Outdated or incompatible drivers for the camera or Jetson module could lead to instability.

5. Environmental Factors:

   • High temperatures or insufficient power supply during operation may cause hardware failures.

6. User Errors or Misconfigurations:

   • Improper setup of the camera or incorrect command usage could lead to operational issues.

Troubleshooting Steps, Solutions & Fixes

Step-by-Step Instructions for Diagnosing the Problem:

1. Check System Load:

   • Monitor CPU and memory usage during camera operation using commands like top or htop.

2. Review Logs:

   • Examine nvargus-daemon logs for specific error messages related to timeouts or resource conflicts.
   • Use the command:

     journalctl -u nvargus-daemon
     

3. Test Camera Connection:

   • Disconnect and reconnect the camera, ensuring proper seating in the CAM0 port.

Commands for Gathering System Information:

• To check current camera settings and status:

  v4l2-ctl --list-devices
  

• To view current clock rates for VI/CSI/ISP:

  cat /sys/kernel/debug/bpmp/debug/clk/vi/rate
  

Methods to Isolate the Issue:

• Test with different cameras if available (e.g., IMX219) to determine if the issue is specific to IMX477.
• Run a simple GStreamer command to check if it consistently fails under light load.

Potential Fixes or Workarounds:

1. Increase Clock Rates:

   • Boost VI/CSI/ISP clocks using commands provided in the forum discussion:

     sudo su
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
     

2. Update Jetpack and Drivers:

   • Ensure you are using the latest stable version of Jetpack (consider moving to JetPack 6.x if it addresses these issues).

3. Reduce System Load:

   • If possible, limit other processes running concurrently with camera operations.

4. Implement Error Handling in Code:

   • Modify your application code to handle Argus::STATUS_TIMEOUT gracefully and attempt reconnection.

5. Power Supply Check:

   • Ensure that your power supply meets the requirements for both the Jetson module and connected peripherals.

Best Practices for Prevention:

• Regularly update software and drivers.
• Monitor system performance metrics during critical operations.
• Conduct thorough testing under various loads before deploying applications.

Unresolved Aspects and Further Investigation Needed:

• The root cause of why similar setups on Jetson Xavier NX do not exhibit this issue remains unclear.
• Ongoing updates from Nvidia regarding fixes in future JetPack releases are anticipated but not confirmed at this time.

This document serves as a comprehensive guide for troubleshooting and resolving issues related to camera connectivity on the Nvidia Jetson Orin Nano platform, based on user experiences shared in forum discussions.