Kernel-Level Memory Leak in Nvidia Jetson Orin Nano with nvarguscamerasrc

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing a kernel-level memory leak when using the nvarguscamerasrc element in GStreamer pipelines. This issue occurs specifically with JetPack 6.0 and DeepStream 7, affecting the kmalloc-256 memory allocation. The problem persists even after applying a previously recommended patch for a similar memory leak issue.

The leak is observed when running a simple GStreamer pipeline:

gst-launch-1.0 nvarguscamerasrc ! fakesink

The kmemleak tool reveals an unreferenced object in the kernel memory, associated with the EglStrmComm process.

Possible Causes

1. Software Bug: The issue appears to be a known bug in the current version of JetPack (6.0) or the NVIDIA drivers.

2. Incomplete Previous Fix: The patch applied for a previous memory leak may not have fully addressed all aspects of the problem, leading to this new manifestation.

3. Driver Incompatibility: There might be an incompatibility between the current NVIDIA GPU driver (version 540.3.0) and the nvarguscamerasrc element.

4. Memory Management Issue: The problem could be rooted in how the kernel or NVIDIA drivers handle memory allocation and deallocation for camera streams.

Troubleshooting Steps, Solutions & Fixes

1. Confirm the Issue:
   Run the simple GStreamer pipeline and monitor system memory usage over time to verify the leak:

   gst-launch-1.0 nvarguscamerasrc ! fakesink
   

   In another terminal, periodically check memory usage:

   free -m
   

2. Apply Kmemleak:
   Use the kmemleak tool to identify the specific memory objects that are leaking:

   echo scan > /sys/kernel/debug/kmemleak
   cat /sys/kernel/debug/kmemleak
   

3. Check for Updates:
   Although a full upgrade to JetPack 6.1 is not possible due to DeepStream 7 incompatibility, check for any available patches or updates for JetPack 6.0 that might address this issue.

4. Workaround Attempts:

   • Try using a different camera source if possible, to determine if the issue is specific to nvarguscamerasrc.
   • Implement a periodic pipeline restart in your application to mitigate the long-term effects of the leak.

5. Report the Issue:
   If not already done, report this specific kmalloc-256 leak to NVIDIA’s developer forums or support channels, providing:

   • Detailed system information (JetPack version, DeepStream version, CUDA version)
   • The GStreamer pipeline that reproduces the issue
   • Output from kmemleak

6. Monitor for Future Updates:
   Keep an eye on NVIDIA’s release notes and forums for announcements about fixes in upcoming JetPack or DeepStream versions.

7. Temporary Memory Management:
   If possible, implement a watchdog in your application that monitors memory usage and takes corrective action (e.g., restarting the pipeline or the entire application) when memory consumption exceeds a certain threshold.

8. Investigate Alternative Camera Interfaces:
   If feasible for your project, consider using alternative camera interfaces or APIs that might not exhibit the same memory leak issue.

9. Kernel Module Unloading:
   As a last resort, you might try unloading and reloading the relevant kernel modules periodically, though this is not a recommended long-term solution:

   sudo modprobe -r nvidia-drm nvidia-modeset nvidia
   sudo modprobe nvidia
   

It is worth underlining that this issue is acknowledged by NVIDIA, and they are currently investigating it. The most reliable solution will likely come in the form of an official update or patch from NVIDIA. In the meantime, users should implement temporary workarounds and closely monitor their system’s memory usage when using nvarguscamerasrc.