UEFI Does Not Recognize Micron NVMe as Bootable

Issue Overview

Users are experiencing issues with the Nvidia Jetson Orin Nano Dev board where the UEFI firmware does not recognize a Micron NVMe drive as a bootable device. This problem arises during the boot process after successfully flashing the JetPack r35.4.1 on a custom carrier card. The NVMe drive, previously used as storage on a Xavier NX, is now intended to serve as the primary boot drive.

Symptoms and Errors

• The NVMe drive is detected in the UEFI shell when running PCI commands, but it fails to boot.
• UEFI logs indicate errors related to the NVMe controller, specifically:
  • "NVMe controller is disabled with status [Success]."
  • "Failed to Init Controller (Device Error)."
• The logs also point to issues with the number of I/O Completion Queues being created.

Context

• The issue occurs during the boot process after flashing JetPack.
• The NVMe drive in question is soldered onto the carrier card, preventing users from swapping it out for testing.
• Users have successfully used this carrier card with other Nvidia models (TX2 NX and Xavier NX) but have not encountered this problem before.

Frequency and Impact

The issue appears to be specific to certain Micron NVMe drives, as some users have reported success with different models. The inability to boot from the NVMe drive significantly impacts user experience, hindering system functionality and potentially delaying project timelines.

Possible Causes

• Hardware Incompatibilities or Defects: The specific Micron NVMe model may not be fully compatible with the Orin Nano’s UEFI implementation.

• Software Bugs or Conflicts: There may be bugs in the UEFI firmware or JetPack that affect NVMe drive recognition.

• Configuration Errors: Incorrect settings in UEFI related to NVMe configuration could prevent proper initialization.

• Driver Issues: Outdated or incorrect drivers for the NVMe controller may lead to communication failures.

• Environmental Factors: Power supply issues or thermal conditions might affect hardware performance and detection.

• User Errors or Misconfigurations: Improper setup or flashing procedures could contribute to the problem.

Troubleshooting Steps, Solutions & Fixes

Step-by-Step Troubleshooting

1. Boot from USB Drive:

   • Use a USB drive to boot the device and check if the NVMe drive is recognized as /dev/nvme0n1.
   • This helps determine if the issue lies within UEFI or with the NVMe drive itself.

2. Firmware Update:

   • Check for available firmware updates for the Micron NVMe drive.
   • Users have reported success after updating to version "M05" from "2100AI-AT_FFU_to_MU05.zip".

3. Modify EDK2 Source Code:

   • If comfortable with coding, modify the EDK2 source code by reducing the loop upper bound in NvmeCreateIoCompletionQueue and NvmeCreateIoSubmissionQueue functions from NVME_MAX_QUEUES to 2.
   • This change has reportedly allowed some users to bypass the issue temporarily.

   Example patch snippet:

   for (index = 0; index < 2; index++) {
       // Original code logic
   }
   

4. Check Compatibility:

   • Validate compatibility of the Micron NVMe model with Jetson Orin Nano.
   • Testing with a different development kit may provide insights into whether it’s a hardware-specific issue.

5. Collect Logs for Analysis:

   • Enable detailed logging in UEFI and collect boot logs for further analysis.
   • Look specifically for error messages related to I/O completion queues.

6. Consult Documentation:

   • Review Nvidia’s official documentation for any known issues or additional configuration steps required for using NVMe drives with Jetson Orin Nano.

Recommended Approaches

• Firmware Update: Prioritize updating firmware based on user reports of success.

• Source Code Modification: Consider implementing code changes if other solutions do not resolve the issue, but proceed with caution due to potential side effects on other drives.

Best Practices for Prevention

• Regularly check for firmware updates for both the Jetson board and connected devices.

• Maintain backups of working configurations before making changes in UEFI or source code.

Unresolved Aspects

Further investigation may be needed regarding:

• The specific compatibility of various Micron NVMe models with Jetson Orin Nano.

• Long-term effects of any source code modifications on system stability and performance.