Gstreamer Pipeline Issue: PosixMemMap:84 mmap failed

Issue Overview

Users are experiencing a crash in their application using a Gstreamer pipeline on the Nvidia Jetson Orin Nano Dev board. The issue occurs after running the application for some time (approximately 1.5 hours) and appears to be related to one of the elements in the pipeline. The application is designed to process video streams from multiple cameras, apply face and person detection, and send the processed video through UDP for display in a browser.

Key details:

• Hardware: Jetson Orin Nano
• DeepStream Version: 6.3
• JetPack Version: 5.1.2
• The pipeline handles multiple camera inputs (usually 4)
• Face and person detection are applied to the video streams
• The processed video is sent via UDP for browser display

Possible Causes

1. Camera Source Instability: The issue may be originating from the camera sources, which could be unstable during long-running operations.

2. Memory Management Issues: The error message "PosixMemMap:84 mmap failed" suggests a problem with memory mapping, which could be due to resource exhaustion or memory leaks.

3. Pipeline Configuration: Incorrect configuration of Gstreamer elements, such as nvstreammux or nvdsosd, may lead to instability over time.

4. Hardware Limitations: The Jetson Orin Nano may be struggling to handle the processing load of multiple high-resolution video streams over extended periods.

5. Software Conflicts: Potential conflicts between different versions of libraries or drivers used in the DeepStream and JetPack environments.

Troubleshooting Steps, Solutions & Fixes

1. Isolate Camera Source Issues:

   • Replace v4l2src with videotestsrc is-live=1 in the pipeline to determine if the issue is specific to the cameras.
   • Run the modified pipeline for an extended period to see if the crash still occurs.

2. Optimize Pipeline Configuration:

   • Set batch-size=1 for nvstreammux when using a single source.
   • Remove nvvidconv and use only nvvideoconvert in DeepStream SDK.
   • Adjust the batched-push-timeout value in nvstreammux (e.g., 33333 or 40000).

3. Verify Hardware Compatibility:

   • Ensure that the cameras and their resolutions are compatible with the Jetson Orin Nano’s capabilities.
   • Consider reducing the resolution or frame rate of the input streams if the hardware is overloaded.

4. Update Software and Drivers:

   • Verify that you are using the latest compatible versions of DeepStream and JetPack.
   • Check for any available updates for camera drivers or Gstreamer plugins.

5. Memory Management:

   • Monitor memory usage during runtime to identify potential leaks.
   • Consider implementing periodic pipeline restarts to mitigate long-term memory issues.

6. Gstreamer Pipeline Debugging:

   • Use the GST_DEBUG environment variable to enable more verbose logging:

     export GST_DEBUG=3
     

   • Analyze the logs to identify any specific elements causing issues.

7. Simplified Pipeline Testing:
   Try running a simplified version of the pipeline to isolate the problem:

   gst-launch-1.0 -e nvstreammux name=m batch-size=4 width=960 height=544 batched-push-timeout=33333 sync_inputs=1 live-source=1 ! \
   nvinfer config-file-path=/path/to/config.txt ! nvmultistreamtiler width=1920 height=1080 rows=2 columns=2 ! \
   nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvjpegenc quality=30 ! udpsink host=<IP_ADDRESS> port=5005 sync=0 \
   videotestsrc is-live=1 ! "video/x-raw,format=UYVY,width=1920,height=1536,framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=I420, width=960,height=544" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! m.sink_0 \
   videotestsrc is-live=1 ! "video/x-raw,format=UYVY,width=1920,height=1536,framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=I420, width=960,height=544" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! m.sink_1 \
   videotestsrc is-live=1 ! "video/x-raw,format=UYVY,width=1920,height=1536,framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=I420, width=960,height=544" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! m.sink_2 \
   videotestsrc is-live=1 ! "video/x-raw,format=UYVY,width=1920,height=1536,framerate=30/1" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=I420, width=960,height=544" ! nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! m.sink_3
   

8. nvdsosd Element Investigation:

   • If the issue persists, try removing the nvdsosd element from the pipeline temporarily to see if it resolves the crash.
   • If removing nvdsosd fixes the issue, investigate potential conflicts or configuration problems with this element.

9. Long-term Stability Testing:

   • Run the pipeline for extended periods (3+ days) to ensure long-term stability.
   • Monitor system resources and logs during these extended runs to identify any gradual degradation or resource exhaustion.

By systematically applying these troubleshooting steps and solutions, you should be able to identify and resolve the underlying cause of the Gstreamer pipeline issue on your Nvidia Jetson Orin Nano Dev board.