Flashing Jetson Orin Nano Devkit NVMe fails with BL_CARVEOUT error

Issue Overview

Users are experiencing difficulties when attempting to flash the Jetson Orin Nano Developer Kit with JetPack 5.1.1 using the NVMe storage option. The flashing process initially fails with USB write timeouts, and after resolving connectivity issues, the device fails to boot due to a BL_CARVEOUT error. This issue appears to be specific to the Jetson Orin Nano Devkit when using NVMe storage, as the same process works for the Xavier NX Devkit.

Possible Causes

1. Incorrect flashing command syntax
2. Incompatibility between the flashing method and the Orin Nano hardware
3. Memory allocation issues in the bootloader
4. Misconfiguration of the EMMC_CFG in the board configuration file
5. Changes in the flashing process between Xavier and Orin platforms

Troubleshooting Steps, Solutions & Fixes

1. Use the correct network interface:

   • Ensure that the --network usb0 parameter is included in the flashing command.
   • The Orin platform uses a different networking method (NFS) compared to the Xavier (RNDIS USB Mass Storage).

2. Create a custom board configuration:

   • Create a custom .conf file (e.g., jetson-orin-nx-devkit-custom-nvme.conf) with the following content:

     source "${LDK_DIR}/p3768-0000+p3767-0000.conf";
     EMMC_CFG="flash_t234_qspi.xml";
     

   • Place this file in the L4T top directory.

3. Use the modified flashing command:

   sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml --network usb0 jetson-orin-nano-devkit-custom-nvme internal
   sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only
   

   • Note: You may need to unplug and reconnect the board between the two commands.

4. Choose the appropriate XML configuration file:

   • For A/B Support: tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml
   • For Single Filesystem: tools/kernel_flash/flash_l4t_nvme.xml

5. If the issue persists, collect full UART logs for both the failing and successful flashing attempts, as requested by NVIDIA support.

6. After successfully booting the device, check the boot control configuration:

   cat /etc/nv_boot_control.conf
   

7. If you encounter the BL_CARVEOUT error after flashing:

   [0000.444] E> BL_CARVEOUT: Failed to allocate memory of size 0x8000000 for CO:31.
   [0000.452] C> Task 0x0 failed (err: 0x49490003)
   

   This may indicate a memory allocation issue in the bootloader. Ensure that your custom rootfs or kernel modifications are not interfering with the bootloader’s memory requirements.

8. Consider using the two-step flashing process (–no-flash followed by –flash-only) for troubleshooting, as it allows you to prepare flash files without the board attached and then flash multiple times without recreating the data.

If the issue persists after trying these steps, it’s recommended to work with NVIDIA support to investigate further, providing them with the requested logs and configuration details.