Jetson Orin Nano fails to mount rootfs when booting from NVMe

Issue Overview

Users are experiencing issues with the Nvidia Jetson Orin Nano Developer Kit where the system fails to mount the root filesystem (rootfs) when attempting to boot from an NVMe storage device. The symptoms include:

• The boot process initiates but fails to mount the rootfs, leading to repeated attempts and eventual system reboot after 50 failed attempts.
• The kernel logs indicate I/O timeouts with the NVMe driver, suggesting that the I/O Memory Management Unit (IOMMU) may be blocking certain writes.
• The root filesystem is confirmed to be functional since it can be mounted successfully when booting from an SD card.
• The issue appears to occur consistently during boot attempts from NVMe, causing significant delays and manual intervention to continue booting.

The hardware specifications include:

• Kernel version: 5.15.136-tegra
• NVMe disk: PNY CS1030 (PCIe Gen3, 250GB)

This issue impacts user experience by requiring manual intervention for successful booting and potentially leading to system instability or operational delays.

Possible Causes

Several potential causes have been identified for the mounting issue:

1. Hardware Incompatibility: The PNY CS1030 NVMe disk may not be fully compatible with the Jetson Orin Nano, as indicated by user reports of successful boots with other brands like Samsung 980 PRO.

2. Driver Issues: The nvme driver may not be functioning correctly due to misconfigurations or bugs in the current kernel version, leading to I/O timeouts.

3. Configuration Errors: Incorrect settings in the BIOS or within the boot configuration (e.g., using the wrong boot method) could prevent proper communication with the NVMe device.

4. Environmental Factors: Power supply issues or thermal conditions may affect device performance, although these were not explicitly mentioned in user reports.

5. User Errors: Misconfigurations during flashing or setup processes could lead to improper initialization of the NVMe device.

Troubleshooting Steps, Solutions & Fixes

To address the mounting issue, users can follow these troubleshooting steps and solutions:

1. Verify Hardware Compatibility:

   • Test with different NVMe disks known to work well with Jetson devices, such as the Samsung 980 PRO.
   • If possible, obtain a spare NVMe disk for further testing.

2. Check BIOS Settings:

   • Ensure that secure boot is disabled in BIOS settings.
   • Review other relevant BIOS configurations that may affect device initialization.

3. Reflash JetPack:

   • Use the SDK Manager to reflash JetPack 6 onto the NVMe stick, ensuring that no customizations are made during installation.
   • Confirm that firmware updates are applied correctly prior to flashing.

4. Boot Method Testing:

   • Experiment with both “ExtLinux” and “Kernel Partition” boot methods to determine if one method yields better results than the other.

5. Manual Boot Process:

   • If automatic mounting fails, drop into a shell during boot and manually mount the rootfs using commands like:

     mount /dev/nvme0n1p1 /mnt
     

   • After mounting, perform a root pivot and continue booting from the mounted filesystem.

6. Kernel Log Analysis:

   • Review kernel logs for specific error messages related to I/O timeouts or driver issues using:

     dmesg | grep nvme
     

7. Driver Module Check:

   • Ensure that the nvme driver module is included in the initrd image by checking /lib/modules/ directory after flashing.

8. Update Kernel and Initrd:

   • If issues persist after updates, consider rolling back to a previous stable kernel version if available.

9. Preventive Measures:

   • Regularly update firmware and software components.
   • Maintain backups of working configurations for quick recovery.

10. Documentation and Community Resources:

    • Refer to Nvidia’s official documentation for Jetson Orin Nano regarding troubleshooting and compatibility guidelines.
    • Engage with community forums for shared experiences and solutions.

Recommended Approach

Users have reported success with switching to a compatible NVMe disk like the Samsung 980 PRO, which resolved their mounting issues effectively. This solution should be prioritized for those facing similar problems.

Unresolved Aspects

Further investigation may be needed regarding potential IOMMU misconfigurations that could lead to blocking writes during initialization, as well as ongoing compatibility assessments between various NVMe devices and the Jetson Orin Nano platform.