Argus Image Acquisition Crashes After a Few Days

Issue Overview

Users of the Nvidia Jetson Orin Nano Dev board have reported recurring crashes in applications that utilize the Argus library for image acquisition from multiple cameras. The issue manifests as a series of error messages, notably including:

• SCF: Error InvalidState: Corr Error Received for sensor 2 .. Continuing!
• SCF: Error ResourceAlreadyInUse
• SCF: Error Timeout
• SCF: Error InvalidState: Capture Scheduler not running

These errors typically occur after a few days of continuous operation, even under normal conditions without excessive load. The affected system is running JetPack version 5.1.2 and Argus version 0.99.3.3 in a multi-process configuration.

The crashes lead to failures in the iCaptureSession->isRepeating() function, which users report as non-recoverable, severely impacting their applications’ functionality. Additionally, users have noted potential GPU-related DMA errors in the dmesg logs, indicating possible hardware or driver issues.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Incompatibilities or Defects: Continuous operation may expose underlying hardware issues, particularly with the GPU or camera interfaces.

• Software Bugs or Conflicts: The specific versions of JetPack and Argus being used may contain unresolved bugs that affect stability over prolonged use.

• Configuration Errors: Incorrect settings in the application or libraries could lead to resource conflicts or improper state management.

• Driver Issues: Outdated or incompatible drivers may cause conflicts that result in resource errors and timeouts.

• Environmental Factors: Power supply instability or overheating could lead to hardware malfunctions over time.

• User Errors or Misconfigurations: Improper implementation of the Argus API or incorrect camera initialization sequences may contribute to the issue.

Troubleshooting Steps, Solutions & Fixes

To address the crashing issue, users can follow these comprehensive troubleshooting steps:

1. Check System Logs:

   • Use the command dmesg to review logs for GPU-related errors or other system warnings that might provide additional context.

2. Update Libraries:

   • Replace existing libraries with updated versions known to improve stability:
     • libnvfusacap_35.4.1.so
     • libnvargus.so.0201
     • libnvscf.so.0201
   • Users have reported increased stability after replacing these libraries.

3. Monitor Application Behavior:

   • Run the application and monitor its performance over time to identify when crashes occur.
   • Use logging mechanisms to capture detailed error messages leading up to a crash.

4. Test Different Configurations:

   • Experiment with different camera configurations (e.g., resolution, frame rate) to see if certain settings exacerbate the issue.

5. Isolate Hardware Components:

   • If possible, test with different cameras or peripherals to rule out hardware-specific issues.
   • Check power supply stability and ensure that it meets operational requirements.

6. Reinstall JetPack:

   • If issues persist, consider reinstalling JetPack and ensuring all components are correctly configured.

7. Seek Community Support:

   • Engage with online forums and communities for additional insights or similar experiences from other users.
   • Share findings and solutions that may benefit others facing similar problems.

8. Best Practices for Future Prevention:

   • Regularly update software and libraries as new versions become available.
   • Implement error handling in applications to gracefully manage unexpected states and recover where possible.
   • Conduct regular system health checks to identify potential issues before they lead to crashes.

Users have noted that after implementing library updates, they experienced extended periods of stability (up to 2.5 days), suggesting that this may be a viable solution until further updates are released in future JetPack versions.

Unresolved aspects of this issue include whether these fixes will be incorporated into future JetPack releases and if there are any additional patches available from Nvidia that specifically address these stability concerns. Further investigation into the root causes of GPU DMA errors may also be warranted.