Multi Camera Acquisition Crashes with Argus SCF_AutocontrolACSync Error

Issue Overview

Users are experiencing crashes in their applications that utilize multiple cameras with the Nvidia Jetson Orin Nano Dev board. The specific error reported is related to the nvargus-daemon, indicating a timeout issue during image acquisition. The error messages include:

• SCF: Error Timeout: (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
• SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
• Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)

This issue typically occurs after several hours of operation, particularly during the stop/start sequence of camera streaming, which leads to instability and critical failures in the session. Users have reported that the problem appears consistently after 2-3 hours of continuous operation, indicating a potential timing or resource management issue within the Argus framework. The setup involves two cameras streaming approximately 100 images in a loop.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Incompatibilities: Users have noted that unstable serdes links may lead to signal drops, causing similar errors. If using MIPI cameras, cable length or interference may contribute to this problem.

• Software Bugs or Conflicts: The error may stem from bugs within the patched versions of Argus, fusacap, and scf libraries being used (e.g., LIBARGUS hash: b6f55f6cb2de004bcb4a5238f9ed3938).

• Configuration Errors: Improper configuration during the camera setup or incorrect handling of camera sessions could lead to timeout errors.

• Driver Issues: The nvargus-daemon may not be properly handling requests due to driver issues or internal timing problems.

• Environmental Factors: External conditions such as temperature fluctuations or power supply inconsistencies could impact performance.

• User Errors or Misconfigurations: Mismanagement of capture sessions and commands might lead to failures, especially if stop commands are missed.

Troubleshooting Steps, Solutions & Fixes

To address the issue effectively, users can follow these troubleshooting steps:

1. Check Hardware Configuration:

   • Ensure that the camera setup is stable and that cables are not too long or subjected to interference.
   • Use CRC error checks or link status registers to assess the quality of the serdes link.

2. Review Software Versions:

   • Confirm that you are using compatible versions of JetPack (e.g., JP5.1.2) and ensure all libraries are properly patched.
   • Verify library hashes against known stable versions.

3. Enable Detailed Logging:

   • Increase logging levels for Argus by using:

     sudo service nvargus-daemon stop
     sudo su
     export enableCamPclLogs=5
     export enableCamScfLogs=5
     /usr/sbin/nvargus-daemon
     

   • Analyze logs for any additional errors or warnings that may provide insight into timing issues.

4. Test with Different Configurations:

   • Isolate the problem by testing with only one camera at a time or by changing camera models.
   • Experiment with different capture session configurations and commands.

5. Implement Workarounds:

   • If crashes occur during the stop/start sequence, consider introducing delays between stopping and starting sessions to allow for proper resource management.
   • Use alternative methods for stopping the sensor if current commands fail.

6. Update Firmware and Drivers:

   • Ensure that all firmware is up-to-date, including any patches specific to your hardware configuration.
   • Check for updates on Nvidia’s developer forums regarding driver improvements.

7. Monitor System Resources:

   • Use system monitoring tools to check CPU and memory usage during operation to identify any resource bottlenecks.

8. Reach Out for Support:

   • If issues persist, consider reaching out on Nvidia forums or contacting support for assistance with detailed logs and system specifications.

By following these steps and recommendations, users can potentially resolve the SCF_AutocontrolACSync error and enhance the stability of their multi-camera acquisition systems on the Nvidia Jetson Orin Nano Dev board.