Jetson Orin Nano Dev Board Boot Failure After OTA Upgrade

Issue Overview

Users are experiencing boot failures on the Nvidia Jetson Orin Nano Dev board after performing an over-the-air (OTA) upgrade from L4T version 35.5.0 to 36.3. The issue occurs during the boot process, with the system getting stuck at an early stage. Specifically, users report seeing the following error message:

ASSERT [FvbNorFlashStandaloneMm] /out/nvidia/optee.t234-uefi/StandaloneMmOptee_RELEASE/edk2-nvidia/Silicon/NVIDIA/Drivers/FvbNorFlashDxe/FvbNorFlashStandaloneMm.c(937): ((BOOLEAN)(0==1))

This problem affects both encrypted and non-encrypted setups, as well as configurations with and without redundant rootfs enabled.

Possible Causes

1. UEFI Firmware Bug: The issue appears to be related to a known bug in the UEFI firmware of R36.3.

2. Incompatible EKS Image: Some users reported success after regenerating the eks_t234.img file, suggesting that an incompatibility between the old and new EKS images might be contributing to the problem.

3. OTA Update Process Issues: The OTA upgrade process may not be correctly handling all necessary components, particularly for fused devices or when dealing with encrypted setups.

4. Standalone MM OptEE Binary Mismatch: The problem might be related to inconsistencies in the Standalone MM OptEE binary, which is part of the UEFI firmware.

5. OP-TEE Related Issues: Some of the boot failures seem to occur after OP-TEE initialization, indicating potential problems with the OP-TEE configuration or compatibility.

Troubleshooting Steps, Solutions & Fixes

1. Use Initrd Flash Method:
   As a temporary workaround, use the initrd flash method to flash your board with JP6.0GA + disk-encryption + redundant rootfs enabled:

   sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" -u rsa.pem -v sbk.key --network usb0 jetson-orin-nano-devkit internal
   

2. Regenerate EKS Image:
   Try regenerating the eks_t234.img file for R36.3 L4T. This has resolved the issue for some users:

   # Update sym2_t234.key in example.sh
   echo "f0e0d0c0b0a001020304050607080900" > sym2_t234.key
   # Generate new eks_t234.img
   ./example.sh
   

3. Apply UEFI Patches:
   Apply the following patches to the UEFI firmware and rebuild:

   • Add a null version of the FwVariableLib
   • Don’t try to erase early vars partition in Jetson
   • Reset the meas buffer after computing the first measurement

   Build the updated UEFI firmware:

   edk2_docker edk2-nvidia/Platform/NVIDIA/StandaloneMmOptee/build.sh
   

4. Rebuild OP-TEE and TOS:
   Rebuild the OP-TEE and TOS images with the updated Standalone MM OptEE binary:

   # Build ATF and OP-TEE
   # Generate TOS image
   ./gen_tos_part_img.py --monitor ./atf_build/arm-trusted-firmware/build/tegra/t234/release/bl31.bin --os ./optee/build/t234/core/tee-raw.bin --dtb ./optee/tegra234-optee.dtb --tostype optee ./tos.img
   

5. Flash Updated Components:
   Flash the updated components to the device:

   sudo ./flash.sh -k A_secure-os -c bootloader/generic/cfg/flash_t234_qspi.xml jetson-orin-nano-devkit nvme0n1p1
   

6. Check for Updated Releases:
   Keep an eye out for official updates from NVIDIA, as they are aware of the issue and are working on a fix for the next release.

7. Verify OTA Package Generation:
   For fused devices, ensure you’re using the correct command to generate the OTA package:

   sudo -E ROOTFS_AB=1 ROOTFS_ENC=1 ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -u rsa.pem -v sbk.key --external-device nvme0n1 -i sym2_t234.key jetson-orin-nano-devkit R35-5
   

8. Monitor Boot Logs:
   If the issue persists, collect detailed boot logs to identify at which exact point the boot process fails. This information can be crucial for further debugging.

If none of these steps resolve the issue, it’s recommended to wait for the next official release from NVIDIA, which is expected to include a fix for this problem. In the meantime, users can consider reverting to the previous working version (35.5.0) if possible.