Overlayfs mount fail on Jetson Orin Nano after enabling overlayfs

Issue Overview

Some users are experiencing an issue where the Jetson Orin Nano Dev board fails to boot and mount the root filesystem after enabling overlayfs on an A/B redundant partition system. Specifically, after flashing the A/B redundant partition system and successfully logging in, enabling overlayfs with the nv_overlayfs_config -e command and rebooting causes the system to fail to mount the root filesystem from the overlayfs. The watchdog is also not triggered in this state.

The issue seems to occur consistently on custom carrier boards, but has not been verified on the official Jetson Orin Nano Dev Kit. The problem appears to be related to a timing issue in the initrd when attempting to mount the rootfs after enabling overlayfs.

Possible Causes

Based on the provided information, the likely causes of this overlayfs mounting failure are:

• A timing issue or race condition in the initrd process that mounts the root filesystem from overlayfs after it is enabled. The initrd may not be waiting long enough for certain prerequisites before attempting the overlayfs mount.

• Potential incompatibilities or misconfigurations between the A/B partition setup and the overlayfs feature on the Jetson Orin Nano, especially on custom carrier boards. The official Dev Kit has not been tested for comparison.

• Bugs or issues in the underlying overlayfs implementation or integration with the Orin Nano’s boot process and partition layout. There may be assumptions about partition naming, ordering, or timing that break in this scenario.

Troubleshooting Steps, Solutions & Fixes

The Nvidia team has been able to reproduce this issue and are actively working on a fix. Based on their latest update, a timing issue in the initrd rootfs mounting has been identified and a patch is being developed and tested.

In the meantime, affected users can try the following workarounds:

1. Avoid using nv_overlayfs_config -e to enable overlayfs if the A/B redundant partition layout is in use. Stick to the default non-overlayfs configuration.

2. If possible, test with the official Jetson Orin Nano Dev Kit instead of a custom carrier board to see if the issue is specific to the custom hardware configuration.

3. Experiment with adding boot delay parameters to the kernel command line or initrd to allow more time for the overlayfs root mount. Increasing the rootwait timeout may help.

4. Watch for the upcoming JP6 release from Nvidia which should include a proper fix for the initrd timing issue that causes the overlayfs mount to fail on boot.

The Nvidia team has committed to including the fix for this overlayfs mount issue in the next major JP6 release for the Jetson Orin Nano. Once that update is available, upgrading to the latest JP6 software should resolve the problem when enabling overlayfs on an A/B redundant partition system.