Camera Streaming Delay with GStreamer on Jetson Orin

Issue Overview

Users are experiencing significant latency and low frame rates when streaming video from a web camera using GStreamer on the Jetson AGX Orin platform. The issue occurs when creating a pipeline with an appsink to retrieve frames. As the streaming continues, the latency increases progressively, and the frame rate drops noticeably. Additionally, despite using the "nvv4l2decoder" element, GPU utilization remains unexpectedly low, suggesting potential inefficiencies in hardware acceleration.

Possible Causes

1. Inefficient Pipeline Configuration: The current GStreamer pipeline may not be optimized for low-latency streaming on the Jetson platform.

2. Hardware Acceleration Issues: The "nvv4l2decoder" element might not be properly utilizing the GPU, leading to increased CPU load and latency.

3. Buffer Management: Improper buffer handling in the pipeline could cause frame accumulation and increasing latency.

4. Network-related Delays: If streaming over RTSP, network conditions or server-side issues could contribute to latency.

5. Application-level Processing: The callback function used to retrieve and process frames might be introducing additional delays.

6. Resource Contention: Other processes or system configurations might be limiting the resources available for video processing.

Troubleshooting Steps, Solutions & Fixes

1. Simplify and Test the Pipeline:

   • Use gst-launch-1.0 to test a simplified pipeline without the application:

     gst-launch-1.0 rtspsrc location="rtsp://admin:[email protected]:554/h264/main/ch1/main/av_stream" ! queue ! h264parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! ximagesink
     

   • This helps isolate whether the issue is in the pipeline configuration or the application code.

2. Optimize Pipeline Elements:

   • Ensure rtspsrc is configured for low latency:

     g_object_set (G_OBJECT (data.source), "latency", 0, NULL);
     g_object_set (G_OBJECT (data.source), "buffer-mode", 0, NULL);
     

   • Use nvv4l2decoder with performance-enhancing options:

     g_object_set (G_OBJECT (data.avdec), "disable-dpb", true, NULL);
     g_object_set (G_OBJECT (data.avdec), "enable-max-performance", true, NULL);
     

3. Improve Buffer Management:

   • Adjust queue sizes and buffering:

     data.queue = gst_element_factory_make ("queue", "Queue");
     g_object_set (G_OBJECT (data.queue), "max-size-buffers", 1, NULL);
     

4. Optimize Appsink Configuration:

   • Configure appsink for low-latency operation:

     g_object_set (data.sink, "sync", false, NULL);
     g_object_set (data.sink, "async", false, NULL);
     g_object_set (data.sink, "max-buffers", 1, NULL);
     g_object_set (data.sink, "drop", true, NULL);
     

5. Enable Hardware Acceleration:

   • Ensure CUDA support is enabled in librealsense if using RealSense cameras:
     • Build librealsense from source with CUDA support enabled.
     • Follow the Jetson-specific installation instructions: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

6. Profile and Monitor Performance:

   • Use nvgstcapture-1.0 for performance analysis:

     nvgstcapture-1.0 --camsrc=0 --cap-dev-node=0 --capture-auto
     

   • Monitor GPU usage with tegrastats or jetson_stats.

7. Optimize Network Streaming:

   • If possible, use a wired connection instead of Wi-Fi.
   • Reduce the video resolution or frame rate if network bandwidth is limited.

8. Update Software:

   • Ensure you’re using the latest JetPack version compatible with your Jetson AGX Orin.
   • Update GStreamer and related NVIDIA plugins to the latest versions.

9. Check System Resources:

   • Monitor CPU, GPU, and memory usage during streaming.
   • Close unnecessary applications to free up system resources.

10. Investigate Application-level Processing:

    • Profile the callback function (new_sample) to ensure it’s not introducing significant delays.
    • Consider moving heavy processing to separate threads.

If the issue persists after trying these solutions, consider reaching out to NVIDIA developer support or consulting the Jetson community forums for more specific assistance tailored to your exact use case and configuration.