Recurring SD Card Corruption on Nvidia Jetson Orin Nano Dev Board

Issue Overview

Users of the Nvidia Jetson Orin Nano Developer Kit are experiencing recurring SD card corruptions. The issue occurs after safe shutdowns or reboots, without unplugging the power cable. The problem has been observed with multiple types and brands of SD cards, including SanDisk, Kingston, and Samsung. The corruption happens intermittently, often after copying new kernel and modules via SSH, running the sync command, and then rebooting. This issue significantly impacts user experience, as it requires reflashing the board each time corruption occurs.

Possible Causes

1. File System Stress: The corruption may be triggered by file system operations between reboots, such as copying large amounts of data.

2. Hardware Incompatibility: There might be an underlying hardware issue affecting SD card compatibility with the Orin Nano Dev Board.

3. Software Bug: A potential bug in the kernel or bootloader could be causing improper handling of SD card operations during shutdown or reboot.

4. Power Management Issues: Inconsistent power delivery during shutdown or reboot processes might lead to corruption.

5. SD Card Controller Problems: The SD card controller on the Orin Nano might have issues that manifest as file system corruption.

6. Firmware or Driver Bugs: There could be bugs in the firmware or drivers responsible for SD card interactions.

Troubleshooting Steps, Solutions & Fixes

1. Verify with Stock Image:

   • Flash the board with a pure image from SDK Manager without any custom kernels or DTBs.
   • Test if the issue persists with the stock image to isolate whether the problem is related to custom modifications.

2. File System Check:

   • Before shutting down the system, run a file system check:

     sudo fsck -y /dev/mmcblk1p1
     

   • Note: This cannot be done on a live system as the partition is mounted. Perform this check on a host machine if necessary.

3. Stress Testing:

   • Perform stress tests by writing a few hundred megabytes to the SD card between reboots to replicate the issue.
   • Use a script to automate the process of copying files and rebooting multiple times.

4. Serial Console Logging:

   • Set up a serial console to capture detailed boot logs, including information not available in standard system logs.
   • Follow the guide at JetsonHacks for setting up a serial debug console.

5. Analyze Logs:

   • Examine syslog, kern.log, and dmesg outputs for any error messages or warnings related to SD card or file system issues.
   • Pay special attention to entries related to EXT4 file system operations and SD card controller messages.

6. Check File System Details:

   • When the system is working correctly, run the following commands and note the output:

     df -H -T
     lsblk -f
     

   • Compare these outputs with those from a corrupted state to identify discrepancies.

7. Alternative Storage Options:

   • If possible, test the system with alternative storage options, such as eMMC or NVMe, to determine if the issue is specific to SD cards.

8. Update Firmware and Drivers:

   • Ensure that the Jetson Orin Nano is running the latest firmware and drivers available from NVIDIA.
   • Check for any known issues or bug fixes in the release notes of recent updates.

9. Power Supply Verification:

   • Verify that the power supply is stable and meets the requirements of the Jetson Orin Nano.
   • Try using a different power supply to rule out power-related issues.

10. SD Card Benchmarking:

    • Perform read/write benchmarks on the SD card to ensure it’s performing as expected.
    • Use tools like dd or specialized SD card benchmarking software.

11. Kernel Parameter Adjustments:

    • Experiment with kernel parameters related to SD card and file system handling.
    • Consult NVIDIA documentation for recommended settings.

12. File System Alternatives:

    • Consider testing with alternative file systems like F2FS, which is optimized for flash storage devices.

If the issue persists after trying these steps, it’s recommended to contact NVIDIA support directly, providing them with all the collected logs and details of the troubleshooting steps performed. The recurring nature of this problem across multiple users suggests it might be a broader issue that requires attention from NVIDIA’s development team.