General error queue is out of sync with frame queue on Nvidia Jetson Orin Nano

Issue Overview

Users are experiencing a significant issue when connecting an IMX715 sensor to the Nvidia Jetson Orin Nano 8GB via GMSL (Gigabit Multimedia Serial Link). The primary symptoms include a low frame rate of received images and various errors logged in the kernel. Specifically, users have reported the following errors:

• Kernel Logs:
  • "General error queue is out of sync with frame queue."
  • "uncorr_err: request timed out after 2500 ms"
  • "corr_err: discarding frame 0"

These errors occur during the initialization and data capture phases, indicating potential synchronization issues between the sensor and the Jetson board. The problem seems to manifest consistently during operation, severely impacting the user experience and functionality of applications relying on real-time image processing.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Incompatibilities: The IMX715 sensor may not be fully compatible with the Jetson Orin Nano’s hardware configuration or firmware.
• Software Bugs: There could be bugs in the driver or firmware related to the tegracam sensor that affect its performance.
• Configuration Errors: Incorrect settings in the device tree or camera configuration might lead to improper data handling.
• Driver Issues: Outdated or incompatible drivers for the IMX715 sensor could lead to synchronization problems.
• Environmental Factors: Issues such as power supply fluctuations or overheating could affect performance.
• User Errors: Misconfigurations during setup or incorrect connections could lead to operational failures.

Troubleshooting Steps, Solutions & Fixes

To address the issue, users can follow these comprehensive troubleshooting steps:

1. Check Connections:

   • Ensure that all physical connections between the IMX715 sensor and Jetson Orin Nano are secure and correctly configured.

2. Review Device Tree Configuration:

   • Verify that the embedded_metadata_height in the device tree is correctly configured. Incorrect values can lead to mismatches in data processing.

3. Increase Clock Rates:

   • Users have reported success by boosting clock rates. Execute the following commands to lock clock rates:

     sudo su
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
     echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
     

   • After executing these commands, check if there is any improvement in frame rates.

4. Examine Kernel Logs:

   • Continuously monitor kernel logs for any new errors after making changes:

     dmesg | grep imx715
     

5. Test Different Configurations:

   • If possible, test with different configurations of GMSL settings or different sensors to isolate whether the issue is specific to the IMX715.

6. Update Drivers and Firmware:

   • Ensure that all relevant drivers and firmware for both the Jetson Orin Nano and IMX715 are up-to-date. Check Nvidia’s official documentation for updates.

7. Check Power Supply:

   • Make sure that the power supply meets the requirements for both the Jetson board and connected peripherals.

8. Consult Documentation:

   • Refer to Nvidia’s official documentation regarding camera integration with Jetson devices for any specific configuration recommendations.

9. Seek Community Support:

   • If issues persist, consider posting detailed logs and configurations on forums like Nvidia Developer Forums for community assistance.

Recommended Approach

Based on user feedback, increasing clock rates has shown some positive results in addressing synchronization issues. Users should prioritize this step while also ensuring that their configurations align with best practices outlined in Nvidia’s documentation.

Unresolved Aspects

While several solutions have been proposed, some users still report inconsistencies in performance even after applying fixes. Further investigation into potential firmware bugs or deeper hardware compatibility issues may be necessary as more users share their experiences and solutions.