CAM0 Port Latency Issue with IMX219 Stereo Camera on Nvidia Jetson Orin Nano

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing a latency problem with stereo camera setups using IMX219 sensors. Specifically, the camera connected to the CAM0 port consistently lags behind the one connected to CAM1 when displaying video feeds side-by-side. This issue persists even when switching the physical connections of the cameras or altering the display order in software. The problem occurs during video capture and display using GStreamer and OpenCV on a system running Jetpack 6.0-b52.

Possible Causes

1. Hardware-level synchronization: The CAM0 and CAM1 ports may not be properly synchronized at the hardware level, causing a consistent delay in one feed.

2. GStreamer pipeline configuration: The GStreamer pipelines used for capturing video from the two cameras might not be optimized for synchronous operation.

3. OpenCV processing delay: The method of capturing and processing frames in OpenCV could introduce latency, especially if not optimized for parallel operation.

4. Driver or firmware issues: There might be driver or firmware-related problems specific to the Jetson Orin Nano or the IMX219 camera module causing the synchronization issue.

5. Resource allocation: The system might be allocating resources unevenly between the two camera ports, leading to performance differences.

Troubleshooting Steps, Solutions & Fixes

1. Use synchronized applications:

   • Install and run the syncStereo sample application to test if the cameras can be synchronized at a lower level.

   Executing Argus Sample: argus_syncstereo
   

   • If syncStereo works correctly, refer to its implementation to incorporate synchronization mechanisms into your own code.

2. Implement multi-threading:

   • Run the cameras in separate threads to improve synchronization.
   • Refer to the dual camera example provided by JetsonHacksNano:
     CSI-Camera/dual_camera.py

3. Optimize GStreamer pipeline:

   • Experiment with different GStreamer pipeline configurations to minimize latency.
   • Consider using the nvarguscamerasrc element with additional synchronization options if available.

4. Update Jetpack and drivers:

   • Ensure you are using the latest version of Jetpack and have all the necessary drivers updated.
   • Check for any firmware updates for the IMX219 camera module.

5. Adjust capture and processing logic:

   • Instead of using separate grab() and retrieve() calls, try using read() to capture frames from both cameras.
   • Implement a frame dropping mechanism to maintain synchronization if one camera falls behind.

6. Use hardware synchronization if available:

   • Check if your IMX219 camera module supports hardware-level synchronization and enable it if possible.

7. Monitor system resources:

   • Use system monitoring tools to check if there are any resource bottlenecks affecting one camera port more than the other.

8. Experiment with different camera settings:

   • Try adjusting frame rates, resolutions, or other camera parameters to see if it affects the synchronization issue.

Here’s a modified version of the code incorporating some of these suggestions:

import cv2 as cv
import numpy as np
import threading

def capture_frames(cam, frames):
    while True:
        ret, frame = cam.read()
        if ret:
            frames.append(frame)
        if len(frames) > 10:  # Keep only the last 10 frames
            frames.pop(0)

gst0 = "nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=15/1 ! nvvidconv ! appsink -v"
gst1 = "nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=15/1 ! nvvidconv ! appsink -v"

cam0 = cv.VideoCapture(gst0, cv.CAP_GSTREAMER)
cam1 = cv.VideoCapture(gst1, cv.CAP_GSTREAMER)

frames0, frames1 = [], []

thread0 = threading.Thread(target=capture_frames, args=(cam0, frames0))
thread1 = threading.Thread(target=capture_frames, args=(cam1, frames1))

thread0.start()
thread1.start()

while True:
    if len(frames0) > 0 and len(frames1) > 0:
        frame0 = frames0[-1]
        frame1 = frames1[-1]
        
        imOut = np.hstack((frame0, frame1))
        cv.imshow("cam0 vs. cam1", imOut)
    
    key = cv.waitKey(1)
    if key == ord('q'):
        break

cv.destroyAllWindows()
cam0.release()
cam1.release()
thread0.join()
thread1.join()

This modified code uses separate threads for capturing frames from each camera and maintains a small buffer of recent frames. It then displays the most recent frame from each camera side-by-side, which should help reduce the perceived latency between the two feeds.