UEFI Boot Issues on Nvidia Jetson Orin Nano Custom Board

Issue Overview

The Nvidia Jetson Orin Nano 4GB SKU, when used on a custom carrier board as a robot controller, is experiencing intermittent boot failures. The issue manifests as the UEFI entering the Boot Maintenance Manager during startup, requiring user input on a headless system. This problem occurs randomly across different robots, even after applying custom UEFI modifications to disable the boot wait feature. The issue is causing significant productivity problems at customer sites.

Key points:

• Custom carrier board with BSP based on 35.4.1 L4T
• Headless system with no need for boot wait option
• Problem persists after disabling boot wait in UEFI
• Inconsistent occurrence across robots
• Debug port shows UEFI menu when boot fails

Possible Causes

1. Hardware Design Flaw: The RX line on the debug UART is floating, potentially causing noise and triggering unwanted UEFI behavior.

2. Incomplete UEFI Update: The modified UEFI binaries may not be properly applied to both boot slots (A and B).

3. Software Configuration Issues: Despite disabling the boot wait feature, other configurations might still be causing the UEFI to enter the menu.

4. Peripheral Interference: Connected devices on USB or UART ports might be sending data that triggers the UEFI menu.

5. Bootloader Partition Inconsistency: Updates applied to only one bootloader partition (A or B) could lead to inconsistent behavior.

Troubleshooting Steps, Solutions & Fixes

1. Update Both Bootloader Partitions:

   • Flash both A_cpu_bootloader and B_cpu_bootloader partitions:

     sudo ./flash.sh -k A_cpu-bootloader -c bootloader/t186ref/cfg/flash_t234_qspi.xml jetson-orin-nano-rapyuta-jiri nvme0n1p1
     sudo ./flash.sh -k B_cpu-bootloader -c bootloader/t186ref/cfg/flash_t234_qspi.xml jetson-orin-nano-rapyuta-jiri nvme0n1p1
     

   • Alternatively, reflash the entire board to ensure all changes are applied:

     sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 1
     

2. Verify UEFI Modifications:

   • Ensure the following changes are applied in the UEFI source:
     • Disable boot wait feature
     • Set auto boot time to 0 seconds
   • Rebuild UEFI with these modifications:

     export EDK2_DEV_IMAGE="ghcr.io/tianocore/containers/ubuntu-22-dev:latest"
     export EDK2_USER_ARGS="-v \"${HOME}\":\"${HOME}\" -e EDK2_DOCKER_USER_HOME=\"${HOME}\""
     export EDK2_BUILD_ROOT="${HOME}/build"
     export EDK2_BUILDROOT_ARGS="-v \"${EDK2_BUILD_ROOT}\":\"${EDK2_BUILD_ROOT}\""
     alias edk2_docker="docker run -it --rm -w \"\$(pwd)\" ${EDK2_BUILDROOT_ARGS} ${EDK2_USER_ARGS} \"${EDK2_DEV_IMAGE}\""
     edk2_docker init_edkrepo_conf
     edk2_docker edkrepo manifest-repos add nvidia https://github.com/NVIDIA/edk2-edkrepo-manifest.git main nvidia
     edk2_docker edkrepo clone nvidia-uefi-r35.4.1-updates NVIDIA-Platforms r35.4.1-updates
     edk2_docker edk2-nvidia/Platform/NVIDIA/Jetson/build.sh
     

3. Hardware Mitigation:

   • Add a weak pull-up resistor to the RX line on the 3.3V side to prevent floating.
   • For future hardware iterations, ensure proper termination of the debug UART lines.

4. Isolate Peripheral Interference:

   • Temporarily disconnect all USB and UART devices except for the debug port.
   • Reconnect devices one by one to identify if any specific device triggers the UEFI menu.

5. Debug with UEFI Debug Binary:

   • Flash a debug version of the UEFI binary.
   • Capture and analyze logs to understand why the system still enters the UEFI menu.

6. Verify Boot Chain Configuration:

   • Ensure that both A and B slots for the bootloader are properly configured, even if rootfs A/B is disabled.

7. Perform Extensive Testing:

   • After applying fixes, test multiple power cycles on various robots to ensure consistency.
   • Document any patterns or specific conditions that might trigger the issue.

If the issue persists after implementing these solutions, consider reaching out to Nvidia support for further assistance or investigating potential hardware-specific issues on the custom carrier board.