Jetson Orin Nano RT Kernel Boot Issues

Issue Overview

Users are experiencing difficulties booting the Nvidia Jetson Orin Nano development board with a custom-built RT (Real-Time) kernel. The main symptoms include:

• The board fails to boot after flashing a custom RT kernel
• In some cases, the board gets stuck at a black screen with a flashing cursor
• The issue occurs specifically with RT-enabled kernels, while non-RT kernels boot normally
• The problem affects both JetPack 6.0 Developer Preview (DP) and JetPack 5.1.3 versions

These boot issues significantly impact the usability of the Jetson Orin Nano for real-time applications, preventing users from leveraging the RT capabilities of the system.

Possible Causes

1. Kernel-initrd mismatch: The NVMe drivers in the initrd (initial RAM disk) do not match the custom RT kernel, causing boot failures.

2. Incorrect flashing procedure: Users may be using incorrect commands or methods to flash the custom kernel onto the device.

3. JetPack version incompatibility: Some issues may be specific to certain JetPack versions, particularly the Developer Preview releases.

4. Incomplete kernel customization process: Users might be missing crucial steps in the kernel customization and signing process.

5. Hardware-specific issues: Some problems may be related to the specific hardware configuration of the Jetson Orin Nano.

6. Host system compatibility: The Ubuntu version on the host machine (e.g., Ubuntu 22.04) may affect the ability to use certain tools like SDK Manager.

Troubleshooting Steps, Solutions & Fixes

1. Use the correct JetPack version:

   • For stability, use JetPack 5.1.3 instead of 6.0 DP.
   • Download packages from: https://developer.nvidia.com/embedded/jetson-linux-r3550
   • Follow the Quick Start guide to extract and flash the image.

2. Update initrd to match the RT kernel:

   • Unpack the initrd following this guide: https://docs.nvidia.com/jetson/archives/r36.2/DeveloperGuide/SD/FlashingSupport.html#modifying-the-ram-disk
   • Replace /lib/modules/($uname -r)/ in the unpacked folder with your RT kernel modules (e.g., replace 5.15.122-tegra with 5.15.122-rt-tegra).
   • Update both rootfs/boot/initrd and bootloader/l4t_initrd.img.
   • Repack the initrd and flash again.

3. Use the correct flashing command:

   sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal
   

4. Set up a serial debug console:

   • Follow the guide at: https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/
   • Use the serial console to diagnose boot issues and execute commands during early boot stages.

5. Fix black screen issue (for Orin):

   • After flashing, connect to the serial console.
   • Log in and execute: sudo depmod -a
   • Reboot the system.

6. If using Ubuntu 22.04 on the host machine:

   • Download individual packages instead of using SDK Manager.
   • Follow the manual flashing procedure in the Quick Start guide.

7. For persistent issues:

   • Dump the serial console log for further analysis.
   • Share the log with Nvidia support or on the developer forums for expert assistance.

8. Wait for official RT kernel support:

   • Nvidia plans to provide pre-built RT kernels on their APT server in future releases.
   • This will eliminate the need for manual kernel building and simplify the process.

9. Consider using non-RT kernel temporarily:

   • If RT features are not immediately critical, use the standard kernel until official RT support is available.

10. Stay updated:

    • Regularly check the Nvidia Developer forums and documentation for updates on RT kernel support and known issues.
    • Be prepared to update to newer JetPack versions as they become available, which may resolve some of the current issues.