Camera Image Stops After Prolonged Use on Jetson Orin Nano

Issue Overview

Users are experiencing an issue where the camera feed stops unexpectedly after prolonged use on the Nvidia Jetson Orin Nano development board. This problem occurs specifically when capturing long-duration camera footage. The issue is not consistent and seems to depend on the specific GStreamer pipeline being used. Some pipelines can run for days without issues, while others stop within 1 to 8 hours.

Key details:

• Environment: Jetson Orin Nano with JetPack 5.1.2
• The problem is intermittent and not easily reproducible
• When the issue occurs, an error message is displayed indicating a connection reset
• The same error message appears when physically disconnecting the MIPI flat cable
• The issue was not observed on a Jetson Xavier NX with JetPack 5.0.2

Possible Causes

1. Hardware issues:

   • Loose or faulty MIPI cable connection
   • Overheating of the camera module or Jetson board

2. Software bugs:

   • GStreamer pipeline instability in JetPack 5.1.2
   • Memory leaks in the camera driver or GStreamer elements

3. Resource exhaustion:

   • Running out of memory or buffer space during long operations

4. Power management issues:

   • Inconsistent power delivery to the camera module

5. Driver compatibility:

   • Potential incompatibility between the camera driver and JetPack 5.1.2

Troubleshooting Steps, Solutions & Fixes

1. Verify hardware connections:

   • Ensure the MIPI cable is securely connected to both the camera and the Jetson board
   • Try using a different MIPI cable to rule out cable issues

2. Monitor system temperature:

   • Use the tegrastats command to monitor CPU and GPU temperatures during long runs
   • Improve cooling if temperatures are consistently high

3. Test with different GStreamer pipelines:

   • Use the pipelines that have shown stability for long durations:

     gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=20/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=20/1" ! nvvidconv ! xvimagesink
     

   • Compare behavior with pipelines that exhibit the issue to isolate problematic elements

4. Enable detailed GStreamer debugging:

   • Set the GST_DEBUG environment variable:

     export GST_DEBUG=nvarguscamerasrc:8,*:4
     

   • Run your pipeline and capture the full debug output to a file for analysis

5. Check system logs:

   • Examine /var/log/syslog and dmesg output for any relevant error messages or warnings

6. Test with a previous JetPack version:

   • If possible, try reverting to JetPack 5.0.2 to see if the issue persists

7. Update to the latest JetPack:

   • Check if a newer version of JetPack is available and update if possible

8. Investigate MMAPI:

   • As suggested in the forum, verify the behavior using the argus_camera example from MMAPI

9. Implement error handling and recovery:

   • Modify your application to detect pipeline failures and attempt to restart the camera feed

10. Memory management:

    • Monitor memory usage during long runs using free -m or top
    • Implement periodic pipeline restarts if memory usage grows over time

11. Power management:

    • Disable power management features temporarily to rule out power-related issues:

      sudo nvpmodel -m 0
      sudo jetson_clocks
      

12. File a bug report:

    • If the issue persists after trying these steps, file a detailed bug report with NVIDIA, including:
      • Full system information (JetPack version, kernel version, etc.)
      • Detailed steps to reproduce the issue
      • GStreamer debug logs
      • Any relevant system logs

By systematically working through these steps, you should be able to isolate the cause of the camera stopping issue and either find a workaround or gather enough information for NVIDIA to assist in resolving the problem.