Disable PCIE IOMMU, if MSI Coming Then an Error

Issue Overview

Users are experiencing errors related to the Nvidia Jetson Orin Nano Dev board when attempting to disable the System Memory Management Unit (SMMU) in the device tree. The main symptoms include:

• Errors Encountered: Upon receiving a Message Signaled Interrupt (MSI) from a custom Xilinx FPGA endpoint device, users report the following errors:
  • Unexpected global fault, this could be serious
  • MC request violates VPR requirements
• Context of Occurrence: The issue arises specifically after disabling SMMU, which users have done by modifying the device tree node for PCIe. This action is necessary for other operational requirements but leads to these critical errors.
• Hardware/Software Specifications: Users are operating on Jetson Linux version 35.4.1.
• Frequency of Issue: The problem appears consistently when SMMU is disabled, indicating a direct correlation between this action and the errors.
• Impact on User Experience: The inability to disable SMMU while still receiving interrupts significantly hampers functionality, affecting both development and deployment of applications reliant on the FPGA.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Incompatibilities: Disabling SMMU may lead to incompatibilities with certain hardware configurations, particularly with custom FPGA implementations that expect SMMU to be active.
• Software Bugs or Conflicts: There may be underlying bugs in the Jetson Linux kernel or driver conflicts that arise when SMMU is disabled.
• Configuration Errors: Incorrectly modifying the device tree or not properly configuring related settings may lead to unexpected behaviors.
• Driver Issues: The drivers responsible for handling PCIe communications might not support operations without SMMU enabled, leading to faults.
• Environmental Factors: Power supply issues or thermal conditions could exacerbate problems when SMMU is disabled.
• User Misconfigurations: Unintended consequences of user modifications to the device tree could result in system instability.

Troubleshooting Steps, Solutions & Fixes

To address the issue effectively, follow these comprehensive troubleshooting steps:

1. Re-enable SMMU Temporarily:

   • Confirm that re-enabling SMMU resolves the errors. This establishes that the issue is directly linked to its deactivation.

2. Verify Device Tree Modifications:

   • Double-check modifications made to the device tree node for PCIe. Ensure that no other critical parameters were inadvertently altered.

3. Update Drivers and Firmware:

   • Ensure that all drivers and firmware are up-to-date. Use the following command to check for updates:

     sudo apt-get update && sudo apt-get upgrade
     

4. Testing with Different Configurations:

   • Experiment with different configurations by testing with various hardware setups or using alternative PCIe devices to isolate whether the issue is specific to the FPGA.

5. Consult Documentation:

   • Review Nvidia’s official documentation for Jetson Orin Nano regarding PCIe and SMMU configurations. This can provide insights into supported configurations and known issues.

6. Utilize Diagnostic Tools:

   • Use diagnostic tools such as dmesg to gather logs that provide additional context around errors when they occur:

     dmesg | grep -i error
     

7. Contact Support Forums:

   • Engage with Nvidia’s developer forums for additional support from other users who may have encountered similar issues.

8. Consider Alternative Solutions:

   • If disabling SMMU is essential for your application, consider consulting Nvidia’s technical support for possible patches or workarounds specific to your use case.

9. Best Practices for Future Prevention:

   • Maintain backups of working configurations before making changes.
   • Document all modifications made to device trees and configurations for easier troubleshooting in the future.

10. Unresolved Aspects:

    • Further investigation may be needed into whether there are software patches available from Nvidia that specifically address this issue when SMMU is disabled.

By following these steps, users can systematically troubleshoot and potentially resolve issues related to disabling SMMU on their Nvidia Jetson Orin Nano Dev boards while ensuring continued functionality with their applications.