Feeding RAW Camera Data Directly to CUDA on Jetson Orin Nano

Issue Overview

Users are seeking a method to efficiently feed RAW camera data directly into a CUDA pipeline on the Jetson Orin Nano, bypassing the Integrated Signal Processor (ISP) and minimizing context switches. The specific use case involves an RGBIR camera not supported by the ISP, requiring custom ISP-style processing in CUDA. The goal is to achieve optimal performance by avoiding unnecessary CPU involvement and userspace code execution.

Possible Causes

1. Limited API options: Jetpack-based systems may have restricted APIs for accessing RAW camera data compared to Drive OS.
2. V4L2 limitations: The current V4L2 API requires userspace intervention, causing additional context switches and potential performance bottlenecks.
3. Lack of direct hardware-to-GPU data transfer mechanisms: The absence of a straightforward method to trigger GPU execution without CPU involvement.
4. ISP incompatibility: The RGBIR camera’s incompatibility with the built-in ISP necessitates custom processing.

Troubleshooting Steps, Solutions & Fixes

1. Explore MMAPI (Multimedia API) samples:

   • Install MMAPI using the command:

     sudo apt install nvidia-l4t-jetson-multimedia-api
     

   • Examine the 12_camera_v4l2_cuda sample in the MMAPI for guidance on camera-to-CUDA workflows.

2. Investigate Argus samples:

   • Navigate to /usr/src/jetson_multimedia_api/argus/samples/cudaBayerDemosaic/ for relevant demonstrations.
   • This sample may provide insights into efficient RAW data processing with CUDA.

3. Consider EGL streams:

   • While not directly supported for RAW camera data in Jetpack, EGL streams might offer performance benefits if a workaround can be found.

4. Optimize buffer management:

   • Implement a single buffer system where camera hardware writes directly, and CUDA reads from it to minimize data transfer overhead.

5. Explore advanced synchronization methods:

   • Investigate the TRM (Technical Reference Manual) for information on using sync points to trigger GPU execution without CPU involvement.

6. Research Drive OS APIs:

   • Study the APIs available in Drive OS that achieve direct hardware-to-GPU data transfer, as they might provide inspiration for custom solutions or future Jetpack features.

7. Custom CUDA kernel development:

   • Develop specialized CUDA kernels to perform ISP-like operations efficiently for the RGBIR camera data.

8. Minimize V4L2 overhead:

   • If V4L2 must be used, optimize the implementation to reduce context switches and CPU involvement as much as possible.

9. Contact NVIDIA support:

   • Reach out to NVIDIA developer support for guidance on potential undocumented features or upcoming solutions for direct RAW data to CUDA pipelines.

10. Community collaboration:

    • Engage with the Jetson developer community to explore potential workarounds or custom driver solutions that might enable more direct hardware-to-GPU data flow.

While a perfect solution for direct RAW camera data to CUDA transfer without context switches may not be immediately available in Jetpack, these steps provide a foundation for optimizing performance and exploring potential workarounds. Continue to monitor NVIDIA’s documentation and forums for updates that may address this specific use case in future releases.