Software Encoding with x264enc on Jetson Orin Nano: Latency Issues and Solutions

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing significant latency issues when streaming multiple video feeds simultaneously using GStreamer pipelines and OpenCV. The problem becomes particularly noticeable when attempting to stream three video feeds concurrently, while two streams work almost in real-time. The user is utilizing an OpenCV program that initializes three GStreamer pipelines to capture video streams and transmit them to a Windows host PC using imagezmq.

The Jetson Orin Nano, running Jetpack 6.0, only supports software encoding, which adds complexity to the task of optimizing video streaming performance. The user is specifically interested in implementing x264enc for encoding to reduce data transmission and potentially alleviate latency issues.

Possible Causes

1. Bandwidth Limitations: The current setup may be overwhelming the available network bandwidth, especially when transmitting three uncompressed video streams simultaneously.

2. Processing Overhead: The Jetson Orin Nano’s CPU might be struggling to handle the processing required for multiple uncompressed video streams.

3. Software Encoding Bottleneck: As the Orin Nano only supports software encoding, this process could be introducing additional latency, particularly when handling multiple streams.

4. Inefficient Pipeline Configuration: The current GStreamer pipeline configuration may not be optimized for the Jetson Orin Nano’s capabilities or the specific use case.

5. Network Congestion: If the network between the Jetson Orin Nano and the Windows host PC is congested, it could contribute to the observed latency.

Troubleshooting Steps, Solutions & Fixes

1. Implement UDP Streaming with x264enc:
   Instead of using imagezmq, consider using UDP streaming with x264enc for each pipeline. This approach can potentially reduce latency and improve overall performance. Use the following command structure for each pipeline:

   gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=640,height=360,framerate=30/1 ! videoconvert ! queue max-size-buffers=2 ! x264enc ! h264parse ! rtph264pay ! udpsink host=<HOST_IP> port=5000 sync=0
   

   Replace <HOST_IP> with the IP address of your Windows host PC. Use different ports (e.g., 5000, 5001, 5002) for each pipeline.

2. Optimize Encoding Parameters:
   Experiment with x264enc parameters to find the optimal balance between video quality and encoding speed. For example:

   x264enc tune=zerolatency speed-preset=ultrafast bitrate=500
   

   Adjust the bitrate value based on your network capabilities and quality requirements.

3. Consider RTSP Streaming:
   As an alternative to UDP, you might want to explore RTSP (Real-Time Streaming Protocol) for potentially better performance and easier management of multiple streams.

4. Network Optimization:

   • Ensure you’re using a wired Ethernet connection if possible, as it typically provides lower latency than Wi-Fi.
   • Check for any network congestion or bandwidth limitations on your local network.

5. Performance Monitoring:
   Use the tegrastats command on the Jetson Orin Nano to monitor CPU, GPU, and memory usage during streaming. This can help identify potential bottlenecks:

   tegrastats
   

6. Gradual Testing:
   Start with a single stream and gradually add more, monitoring performance at each step. This can help isolate the point at which latency becomes problematic.

7. Pipeline Optimization:
   Experiment with different queue sizes and buffer settings in your GStreamer pipeline. For example:

   queue max-size-buffers=3 max-size-time=100000000 max-size-bytes=10485760
   

8. Update Jetpack and Drivers:
   Ensure you’re running the latest version of Jetpack and have the most recent drivers installed for your Jetson Orin Nano.

9. Explore Hardware Acceleration Options:
   While the Orin Nano supports software encoding, investigate if there are any partial hardware acceleration features available for video processing that could offload some work from the CPU.

By implementing these solutions and continuing to fine-tune your setup, you should be able to achieve improved performance when streaming multiple video feeds from your Jetson Orin Nano to your Windows host PC.