V4L2 Driver Timeout Issues with Xilinx FPGA and Orin Nano DevKit

Issue Overview

Users are experiencing continuous timeouts when attempting to stream images via V4L2 on an Nvidia Jetson Orin Nano DevKit with a Xilinx FPGA. The FPGA outputs MIPI CSI-2 data in RGB888 format at 1920×1200 resolution and 15 fps. The driver loads correctly and the device path is detected, but streaming fails to start successfully. The issue persists even after attempts to reset the FPGA at the start of streaming.

Possible Causes

1. Timing mismatch: The FPGA starts sending image data as soon as it’s powered on, potentially causing synchronization issues with the V4L2 driver.

2. Pixel format mismatch: The FPGA sends RGB888 data, but the V4L2 image format is set to RGBA8888, which may not be supported by the VI (Video Input) subsystem.

3. CSI-2 signaling issues: The receiver may not be recognizing the proper LP (Low Power) sequence required for CSI-2 communication.

4. Driver configuration: Incorrect settings in the device tree or driver code may be causing incompatibilities.

5. Power sequencing: The FPGA power supply is independent of the AP (Application Processor), which may lead to timing and synchronization problems.

6. Hardware limitations: The VI subsystem may have limitations in supporting the specific pixel format or resolution being used.

Troubleshooting Steps, Solutions & Fixes

1. Verify pixel format settings:

   • Check the device tree settings for mode_type, pixel_phase, and csi_pixel_bit_depth.
   • Ensure these values correctly compose the intended pixel format.
   • Verify that csi_pixel_bit_depth is set to 8 for RGB888 use-case.

2. Adjust streaming parameters:

   v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1200,pixelformat=AB24 --stream-mmap --set-ctrl bypass_mode=0 --stream-skip=100 --stream-count=500
   

   Use the --stream-skip option to skip initial frames, which may help with synchronization.

3. Implement FPGA reset:

   • Add logic to reset the FPGA at the start of streaming in the fpga_s_stream function.
   • This can help re-initialize the FPGA and potentially resolve synchronization issues.

4. Check CSI-2 signaling:

   • Verify that the FPGA is sending the correct LP sequence (LP11->LP01->LP00->LP11) before transitioning to high-speed signaling.
   • Use an oscilloscope or logic analyzer to monitor the CSI-2 lanes if possible.

5. Modify driver power sequencing:

   • Implement logic to enable FPGA power or reset after the software calls s_stream().
   • This may require coordination with the hardware team to add necessary control signals.

6. Gather and analyze VI tracing logs:

   • Enable detailed VI tracing and capture logs during streaming attempts.
   • Look for specific error messages or timing anomalies in the trace output.

7. Verify V4L2 format support:

   • Check the supported formats using:

     v4l2-ctl -d /dev/video1 --list-formats-ext
     

   • Ensure that the AB24 (32-bit RGBA 8-8-8-8) format is listed and supported.

8. Test with different GStreamer pipelines:

   gst-launch-1.0 v4l2src device=/dev/video1 ! 'video/x-raw,format=RGBA,width=1920,height=1200,framerate=15/1' ! nvvidconv ! fakesink
   

   or

   gst-launch-1.0 nvv4l2camerasrc device=/dev/video1 ! 'video/x-raw(memory:NVMM),format=(string)RGBA, width=(int)1920, height=(int)1200,framerate=(fraction)15/1' ! nvvidconv ! fakesink
   

9. Review and update driver code:

   • Ensure that the start_streaming and stop_streaming functions in the driver are properly implemented, not just logging success.
   • Verify that the driver correctly handles power management and streaming control, even if actual power control is not possible.

10. Consult Nvidia documentation:

    • Review the Jetson Linux Driver Package documentation for any known issues or limitations with custom CSI-2 inputs.
    • Check for any errata or application notes related to VI subsystem usage with external FPGA sources.

If the issue persists after trying these steps, consider reaching out to Nvidia developer support or the Jetson community forums with detailed logs and hardware setup information for further assistance.