JetPack 6.0 (Rev 2) Kernel Memory Leak

Issue Overview

Users are experiencing a kernel memory leak on the Nvidia Jetson Orin Nano Developer Kit when running applications on JetPack 6.0, including the recent Rev 2 release. The problem manifests as an increase in the memory allocated from the kmalloc-128 pool, which does not get released even after terminating the application.

The issue occurs specifically when running a DeepStream sample application that processes multiple RTSP feeds (4 x 1080p 30fps). Initially, the slab count starts at approximately 3,000 and increases to nearly 4000 upon application start. Over time, particularly after several days of operation, this number can escalate to over 140,000 slabs, leading to complete memory exhaustion.

Rebooting the board is currently the only method to recover this memory, as stopping the application or restarting Docker does not free up the allocated memory. This issue has been consistently observed across different runs and configurations, significantly impacting user experience by limiting the operational time before requiring a reboot.

Possible Causes

• Kernel Bugs: The memory leak may stem from bugs within the kernel itself, particularly in how memory is managed and released in the context of containerized applications.

• Driver Issues: If specific kernel drivers are involved, they might not be handling memory allocation and deallocation correctly.

• Configuration Errors: Misconfigurations in the application or environment settings could contribute to improper memory management.

• Software Conflicts: Interactions between different software components (e.g., Docker and DeepStream SDK) might lead to unexpected behavior regarding memory usage.

• User Errors: Incorrect implementation of the application logic could inadvertently lead to memory leaks.

Troubleshooting Steps, Solutions & Fixes

1. Monitor Memory Usage:

   • Use commands like cat /proc/slabinfo to monitor slab usage over time.
   • Track changes in slab count before and after running your application.

2. Test Kernel Module Reloading:

   • If your application depends on specific kernel drivers:

     sudo rmmod <module_name>
     sudo modprobe <module_name>
     

   • This may temporarily release leaked memory and help identify if a particular driver is responsible for the leak.

3. Isolate the Problem:

   • Create a minimal version of your application that reproduces the leak.
   • Test this isolated version to confirm whether it exhibits similar memory usage patterns.

4. Check for Updates:

   • Regularly check for updates or patches related to JetPack and DeepStream SDK that may address known issues.
   • Engage with NVIDIA support or community forums for any unreleased patches that might resolve this issue.

5. Investigate Related Issues:

   • Review discussions related to similar problems, such as those involving Triton Server and DeepStream SDK.
   • Look for any patches referenced in these discussions that could be applicable.

6. Best Practices for Future Prevention:

   • Ensure your applications are optimized for memory management.
   • Regularly update your development environment and dependencies to incorporate fixes from NVIDIA.
   • Implement logging mechanisms to track resource allocation and deallocation within your applications.

7. Documentation & Resources:

   • Refer to NVIDIA’s official documentation for JetPack and DeepStream SDK for any guidelines on managing resources effectively.
   • Utilize community forums for shared experiences and solutions from other developers facing similar issues.

8. Unresolved Aspects:

   • The exact cause of the leak remains unclear, necessitating further investigation by developers.
   • Additional testing with various configurations or hardware setups may be required to isolate contributing factors effectively.

By following these steps, users can better diagnose and potentially mitigate the kernel memory leak issue experienced on the Nvidia Jetson Orin Nano Developer Kit while using JetPack 6.0.