Jetson Orin Nano: Issues with A/B Boot Redundancy and Kernel Panic

Issue Overview

Users are experiencing issues with the A/B boot redundancy feature on the Nvidia Jetson Orin Nano, particularly after flashing JetPack 6.0. The primary concern is that the system fails to switch to the backup root filesystem (Rootfs-B) when the primary filesystem (Rootfs-A) becomes corrupted or unbootable.

• Symptoms: Users report encountering kernel panic errors during boot attempts after corrupting Rootfs-A. The error messages indicate that the system cannot mount the filesystem, leading to a failure to switch to Rootfs-B as expected.

• Context: The issue arises during testing of the A/B redundancy feature after intentionally corrupting the primary root filesystem to verify that the system can recover by switching to the backup.

• Hardware/Software Specifications:

  • Device: Nvidia Jetson Orin Nano
  • JetPack Version: 6.0 (with L4T version r36.3.0)

• Frequency: This issue appears to be consistently reported by users who have recently upgraded their systems or tested A/B redundancy features.

• Impact on User Experience: The inability to switch to a backup filesystem can lead to significant downtime and disrupt development workflows, particularly for applications requiring high availability.

Possible Causes

Several factors may contribute to the failure of A/B boot redundancy:

• Kernel Configuration Changes: Changes in kernel behavior between JetPack versions may affect how A/B redundancy operates, particularly regarding how filesystems are mounted during boot.

• Filesystem Corruption Handling: The current implementation may only allow for switching to Rootfs-B when Rootfs-A is completely unbootable due to corruption, rather than handling cases where it is still mountable but has missing dependencies.

• Driver or Firmware Bugs: There may be bugs in the drivers or firmware that prevent proper detection and switching of boot slots under certain conditions.

• User Errors During Testing: Users may not be following the correct procedures for simulating corruption or may not understand how the boot process works in relation to A/B redundancy.

Troubleshooting Steps, Solutions & Fixes

To address issues related to A/B boot redundancy and kernel panic on the Nvidia Jetson Orin Nano, follow these steps:

1. Verify Current Firmware and JetPack Versions:

   • Confirm that you are using the correct versions of JetPack and L4T. Check for any available updates that might resolve known issues:

     sudo apt-get update
     sudo apt-get upgrade
     

2. Check Device Tree Configuration:

   • Review your device tree settings to ensure they are correctly configured for A/B booting. Ensure that both root filesystems are defined properly.

3. Simulate Filesystem Corruption:

   • To test A/B redundancy, simulate corruption of Rootfs-A using commands like:

     sudo dd if=/dev/zero of=/dev/nvme0n1p1 bs=1M count=1
     

   • This should cause a mount failure and trigger a switch to Rootfs-B if functioning correctly.

4. Monitor Boot Logs:

   • Use a serial console connection to capture detailed boot logs. This will help identify where the boot process is failing:

     screen /dev/ttyUSB0 115200
     

5. Test Recovery Mechanisms:

   • After simulating corruption, observe whether the system attempts to switch to Rootfs-B after three failed attempts (the default retry setting).

6. Check for Kernel Panic Messages:

   • Review logs for kernel panic messages and other errors that occur during boot. This can provide insights into what is causing the failure:

     dmesg | grep -i panic
     

7. Consult Documentation and Community Resources:

   • Refer to Nvidia’s official documentation regarding A/B redundancy and troubleshooting boot issues.
   • Engage with community forums for additional insights from users who may have faced similar challenges.

8. Unresolved Aspects:

   • Users may still need clarification regarding specific configurations required for successful operation of A/B redundancy.
   • Further investigation into potential updates or patches that could improve support for this feature may also be necessary if standard troubleshooting does not resolve the problem.

By following these steps, users should be able to troubleshoot and potentially resolve issues related to A/B boot redundancy and kernel panic on their Nvidia Jetson Orin Nano devices after installing JetPack 6.0.