Orin Nano Devkit CAM0 interface video capture intermittently “freezes”

Issue Overview

Users are experiencing intermittent freezing of video capture on the CAM0 interface of the Nvidia Jetson Orin Nano DevKit while using a MIPI CSI-2 compliant custom camera. The camera captures video at 90Hz through the v4l2 interface, with failures occurring during streaming. This issue has been reported across multiple devkits, indicating a potential systemic problem rather than an isolated incident.

The symptoms include:

• Freezing of video capture: The capture process fails without recovery, requiring users to restart the capture application.
• Trace logs indicate problems: Users have noted that the trace logs show errors such as "CHANSEL_PXL_EOF" not being received, and the channel resets due to "tegra_channel_capture_setup" errors.
• Environment: The issue occurs during normal operation of a mature custom camera system, which has been deployed in the field successfully.

The impact on user experience includes disrupted video feeds and increased downtime due to the need for manual intervention to restart the capture process.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Incompatibilities or Defects: There may be compatibility issues between the custom camera and the Orin Nano DevKit, or defects in either device.

• Software Bugs or Conflicts: The Jetson Linux version (36.3) or specific drivers might contain bugs that affect video capture stability.

• Configuration Errors: Incorrect settings in the camera configuration or v4l2 parameters could lead to capture failures.

• Driver Issues: Outdated or improperly configured drivers may cause communication problems between the camera and the devkit.

• Environmental Factors: Power supply issues or overheating could lead to instability during high-demand operations like video streaming.

• User Errors or Misconfigurations: Improper setup of the camera or software parameters by users could contribute to the problem.

Troubleshooting Steps, Solutions & Fixes

To diagnose and potentially resolve the freezing issue with CAM0, follow these troubleshooting steps:

1. Check Hardware Connections:

   • Ensure all physical connections between the camera and devkit are secure.
   • Test with different cables if possible.

2. Review Trace Logs:

   • Use the following commands to enable tracing and gather logs:

     modprobe rtcpu_debug
     echo 1 > /sys/kernel/debug/tracing/tracing_on
     echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
     echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
     echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
     echo 3 > /sys/kernel/debug/camrtc/log-level
     echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
     echo > /sys/kernel/debug/tracing/trace
     

   • Analyze dmesg and kern.log for any relevant error messages.

3. Test Different Configurations:

   • Attempt to adjust MIPI timing parameters as incorrect settings may lead to communication issues.
   • Check if changing sensor settings or modes affects performance.

4. Boost Clock Rates:

   • Increase clock rates for various components using:

     sudo su
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
     

   • Monitor if this change stabilizes the video stream.

5. Restart Capture Application:

   • If freezing occurs, terminate the capture application, wait briefly, and restart it. This has been noted as a temporary workaround by users.

6. Update Software and Drivers:

   • Ensure that you are using the latest version of Jetson Linux (JetPack) and check for any available driver updates that might fix known issues.

7. Consult Documentation:

   • Review Nvidia’s official documentation for any specific notes on using custom cameras with the Orin Nano DevKit.

8. Gather Additional Debugging Information:

   • If problems persist, consider collecting more detailed logs from NVCSI (Nvidia Camera Serial Interface) for further analysis.

9. Consider Environmental Factors:

   • Ensure that power supply is stable and within specifications, and monitor device temperatures during operation.

10. Engage with Community Support:

    • If unresolved, consider reaching out on forums or Nvidia support channels with detailed logs and descriptions of troubleshooting steps already taken.

By following these steps, users should be able to diagnose and potentially resolve issues related to intermittent video capture freezes on CAM0 of the Nvidia Jetson Orin Nano DevKit.