Multiple Camera Streaming Issue on Nvidia Jetson Orin Nano Dev Board

Issue Overview

Users are experiencing difficulties streaming multiple cameras simultaneously on the Nvidia Jetson Orin Nano Dev board. The issue specifically occurs when attempting to stream more than one virtual channel on the same CSI interface using mode1. While streaming works fine with a single camera on each CSI interface, starting a second camera on the same interface results in errors. The problem affects both mode0 and mode1 configurations.

Key observations:

• Streaming 8 streams simultaneously on mode0 works correctly
• One camera running mode1 on each CSI interface functions properly
• The issue arises when attempting to start a second camera stream on the same CSI interface
• V4L2 streaming works for multiple cameras, but GStreamer pipeline using nvarguscamerasrc fails

Possible Causes

1. VI Stride Alignment: Initial investigations revealed a VI stride alignment issue, which was partially resolved by adjusting the sensor output dimensions.

2. CSI Interface Configuration: The problem may be related to how multiple virtual channels are configured on a single CSI interface.

3. Driver Bugs: Recent VI driver bugs have been observed in L4T 35.5.0, potentially contributing to the streaming issues.

4. Firmware Issues: The initial "General error queue is out of sync with frame queue" error was resolved with a firmware update, suggesting potential firmware-related causes.

5. Resource Contention: Multiple cameras on the same CSI interface may be competing for limited resources.

6. Timing and Synchronization: There could be issues with proper timing and synchronization when multiple cameras are active on the same interface.

7. Clock Settings: Insufficient clock rates for various components (VI, ISP, NVCSI, EMC) might limit the system’s ability to handle multiple camera streams.

Troubleshooting Steps, Solutions & Fixes

1. Update RCE Firmware:
   Apply the provided RCE firmware update (camera-rtcpu-t234-rce.img.r35.5.asynchronous) to resolve the "General error queue is out of sync with frame queue" error.

2. Apply VI Driver Patches:
   Implement the kernel patches mentioned in Topic 305007 to address potential VI driver bugs in L4T 35.5.0.

3. Adjust VI Stride Alignment:
   Modify the sensor output dimensions to ensure compatibility with the 64-byte packet length:

   • For mode0: Crop the output to 1920×1080
   • For mode1: Crop the output to 960×540

4. Investigate CSI Interface Configuration:
   Review and potentially adjust the CSI interface configuration in the device tree, focusing on virtual channel settings and lane configurations.

5. Analyze Trace Logs:
   Examine the provided trace logs (trace-gst.log, trace-v4l2.log) to identify specific points of failure or unusual behavior when attempting to stream multiple cameras.

6. Implement Delayed Camera Initialization:
   Try launching one camera stream and waiting for a few minutes before starting the second stream to see if timing-related issues are a factor.

7. Boost System Clocks:
   Increase clock rates for critical components to ensure sufficient processing power:

   sudo su
   echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
   echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
   echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
   echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
   cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
   cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
   cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
   cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
   

8. Review Virtual Channel IDs:
   Double-check that the virtual channel IDs (vc-id) in the device tree are correctly configured for each camera and do not conflict.

9. Analyze nvargus-daemon Logs:
   Enable verbose logging for nvargus-daemon and analyze the output for specific errors or warnings related to multiple camera initialization:

   export enableCamInfiniteTimeout=1
   export enableCamCoreLogs=0
   export enableCamPclLogs=1
   export enableCamScfLogs=1
   /usr/sbin/nvargus-daemon
   

10. Test with Different GStreamer Elements:
    Experiment with alternative GStreamer pipelines that bypass nvarguscamerasrc, using elements like v4l2src instead, to isolate whether the issue is specific to the nvarguscamerasrc implementation.

11. Check for Resource Limitations:
    Monitor system resources (CPU, memory, bandwidth) during multi-camera streaming attempts to identify potential bottlenecks.

12. Investigate Sensor-specific Settings:
    Review the IMX662 sensor configuration in the device tree, ensuring that all necessary settings for multi-camera operation are correctly specified.

If the issue persists after trying these steps, consider reaching out to Nvidia developer support with detailed logs and configuration information for further assistance.