Corrupted data when streaming multiple streams in parallel

Issue Overview

Users are experiencing data corruption when attempting to stream multiple video sources in parallel using the Nvidia Jetson Orin Nano Dev board. The specific symptoms include:

• Corrupted images: When streaming from one source, the video is fine, but launching multiple streams simultaneously results in corrupted frames.

• Context: The issue arises during the execution of a GStreamer pipeline designed for handling real-time streaming. The pipeline is as follows:

  cap = cv2.VideoCapture(
      f"rtspsrc location={video_source} latency=0 ! rtph264depay ! nvv4l2decoder disable-dpb=false enable-max-performance=true ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw format=BGR ! appsink max-buffers=1 drop=true",
      cv2.CAP_GSTREAMER,
  )
  

• Hardware and Software Specifications:

  • GStreamer version: 1.16.3
  • OpenCV version: 4.8.0
  • L4J version: 35.4.1
  • Jetpack version: 5.1.2

• Frequency of Issue: The problem occurs consistently when attempting to stream multiple Full HD (FHD) streams at a frame rate of 15 fps.

• Impact on User Experience: The corruption of video frames significantly affects the usability of the system for applications requiring reliable video streaming.

Possible Causes

Several potential causes for the issue have been identified:

• Hardware Limitations: Although users reported spare CPU capacity, there may still be bottlenecks in processing power when handling multiple streams.

• Network Bandwidth: Insufficient network bandwidth could lead to data corruption when streaming multiple high-definition video sources concurrently.

• Software Configuration:

  • Setting latency=0 may not be optimal for multi-stream scenarios, potentially leading to dropped frames or corrupted data.
  • The use of disable-dpb=false could result in frames being processed out of order, especially under heavy load.

• Driver Issues: Compatibility or bugs within the GStreamer or OpenCV versions being used may contribute to the problem.

Troubleshooting Steps, Solutions & Fixes

To address the issue of corrupted data during parallel streaming, follow these troubleshooting steps and possible solutions:

1. Run CPU at Maximum Clock Speed:

   • Execute the command:

     sudo jetson_clocks
     

   • This command ensures that CPU cores are running at maximum frequency, potentially alleviating processing bottlenecks.

2. Check System Status:

   • Use the following command to monitor system performance:

     sudo tegrastats
     

   • This will provide insights into resource usage and help identify any bottlenecks.

3. Adjust Latency Settings:

   • Experiment with different latency settings by modifying the pipeline to avoid setting latency=0. This may stabilize stream processing.

4. Assess Network Bandwidth:

   • Test network conditions to ensure sufficient bandwidth is available for multiple streams.
   • Consider saving streams to files and then decoding them using both hardware and software decoders for comparison.

5. Isolate Streams:

   • Run tests with fewer streams (e.g., two or three) to determine if the issue persists with lower loads, which can help isolate whether the problem is related to hardware limitations or network issues.

6. Update Software Components:

   • Ensure that all software components (GStreamer, OpenCV, Jetpack) are updated to their latest versions as newer releases may contain bug fixes and performance improvements.

7. Review Configuration Settings:

   • Double-check all configuration settings in the GStreamer pipeline for potential misconfigurations that could lead to data corruption.

8. Best Practices for Future Streaming:

   • When setting up future streaming applications, consider starting with lower resolutions or frame rates and gradually increasing them while monitoring performance.
   • Regularly check for updates and community discussions regarding known issues with specific versions of software being used.

By following these steps, users can diagnose and potentially resolve issues related to corrupted data when streaming multiple video sources on the Nvidia Jetson Orin Nano Dev board. Further investigation may be required if problems persist after trying these solutions.