Low NVMe SSD Write Speed on Jetson Orin Nano Module

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing significantly low write speeds when using an NVMe SSD connected via PCIe2 interface. The specific symptoms include:

• The NVMe SSD is used as the boot disk and is the only storage on the board
• The design follows the EVM (Evaluation Module) board layout
• Read speeds are acceptable at an average of 803MB/s
• Write speeds are extremely low, averaging only 98.3MB/s
• The PCIe interface is configured as Gen3 x2 using Orin UPHY2 Lanes 1:0
• The SSD is connected via an M.2 M-Key slot

This issue significantly impacts the overall storage performance of the system, potentially affecting boot times, application loading, and data-intensive operations.

Possible Causes

1. PCIe Lane Configuration: The use of only two PCIe lanes (x2) might be limiting the bandwidth available for write operations.

2. NVMe Driver Issues: Suboptimal or outdated NVMe drivers could be causing poor write performance.

3. SSD Firmware: Outdated or incompatible SSD firmware might not be fully utilizing the capabilities of the Jetson Orin Nano’s PCIe interface.

4. Thermal Throttling: If the SSD is overheating, it may be throttling its performance to protect itself.

5. Power Management: Inadequate power delivery or aggressive power-saving settings could be limiting the SSD’s write performance.

6. File System Configuration: Suboptimal file system settings or inappropriate choice of file system might be affecting write speeds.

7. Hardware Defect: There could be a physical issue with the SSD or the PCIe interface on the development board.

8. Benchmark Methodology: The method used for measuring write speeds might not be accurately representing real-world performance.

Troubleshooting Steps, Solutions & Fixes

1. Verify PCIe Configuration:

   • Check the PCIe link status to ensure it’s running at Gen3 x2 as expected:

     sudo lspci -vvv
     

   • Look for the "LnkSta:" line to confirm the link speed and width.

2. Update NVMe Drivers and SSD Firmware:

   • Check for updates to the NVMe drivers in the Jetson Linux package.
   • Visit the SSD manufacturer’s website to download and install the latest firmware for your specific model.

3. Monitor SSD Temperature:

   • Install and use the nvme-cli tool to check SSD temperature:

     sudo apt-get install nvme-cli
     sudo nvme smart-log /dev/nvme0
     

   • If temperatures are high, improve cooling or consider adding a heatsink to the SSD.

4. Check Power Management Settings:

   • Review and adjust power management settings in the UEFI/BIOS if available.
   • Ensure the power supply can deliver sufficient current for both the Jetson module and the NVMe SSD.

5. Optimize File System:

   • If using ext4, consider enabling the noatime mount option to reduce write operations:

     sudo nano /etc/fstab
     # Add 'noatime' to the options for the NVMe partition
     

   • Experiment with different file systems like XFS or F2FS, which may offer better performance for SSDs.

6. Benchmark with Different Tools:

   • Use various benchmarking tools to get a comprehensive view of performance:

     sudo apt-get install fio
     fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
     

7. Check for Hardware Issues:

   • Try the SSD in a different system to rule out hardware defects.
   • If possible, test a different NVMe SSD in the Jetson Orin Nano to isolate the issue.

8. Adjust NVMe Controller Settings:

   • Use nvme-cli to modify NVMe controller settings:

     sudo nvme id-ctrl /dev/nvme0
     sudo nvme set-feature /dev/nvme0 -f 7 -v <value>
     

   • Experiment with different values for the Number of Queues and Queue Depth.

9. Investigate Kernel Parameters:

   • Check and adjust relevant kernel parameters:

     sudo sysctl -a | grep nvme
     

   • Consider increasing the nvme_core.io_timeout value if timeouts are occurring.

10. Consult NVIDIA Developer Forums:

    • If the issue persists, create a detailed post on the NVIDIA Developer Forums, including:
      • Exact hardware specifications
      • Jetson Linux version
      • Output of diagnostic commands
      • Benchmarking results

Remember to backup important data before making significant changes to system configuration or firmware. If none of these solutions resolve the issue, it may be necessary to contact NVIDIA support or consider the possibility of a hardware limitation or defect in the specific combination of the Jetson Orin Nano and the NVMe SSD being used.