Intermittent Data Loss on Nvidia Jetson Orin Nano UART (ttyTHS1)

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing intermittent data loss when receiving long messages on the /dev/ttyTHS1 UART interface. The problem occurs across various baud rates, including high-speed communications up to 4.3 Mbaud. Specific symptoms include:

• Occasional corruption of data, typically starting from the 38th byte
• Random insertion of 2, 4, or 6 zero bytes at the beginning of corrupted messages
• Skipping of 1-8 bytes in the middle of the message
• Issue occurs approximately every 30 seconds or 4000 messages
• Problem persists even after disabling nvgetty service

The issue appears to be more prevalent on Orin NX modules compared to Xavier NX modules, which can handle higher baud rates without data loss.

Possible Causes

1. Buffer Overflow: The symptoms suggest a potential overflow in the receive buffer, especially at higher baud rates or with longer messages.

2. Clock Frequency Differences: The default clock frequency for Orin Nano/NX may be lower than that of Xavier NX, potentially affecting UART performance at high speeds.

3. Hardware Flow Control: Lack of hardware flow control (RTS/CTS) may contribute to data loss at higher baud rates.

4. Software or Driver Issues: The problem may be related to software handling of the UART, possibly in the device drivers or lower-level UART management.

5. DMA or Interrupt Handling: There could be issues with Direct Memory Access (DMA) or interrupt handling for the UART, leading to missed data at high speeds.

Troubleshooting Steps, Solutions & Fixes

1. Enable Hardware Flow Control:

   • Implement RTS/CTS flow control in your UART configuration.
   • This has shown significant improvement, allowing stable operation up to 4.3 Mbaud.

2. Adjust Baud Rate:

   • If possible, reduce the baud rate to 115200 or lower, where the issue is less prevalent.
   • For higher speeds, stay below 4.3 Mbaud with hardware flow control enabled.

3. Buffer Size Optimization:

   • Experiment with different buffer sizes in your application.
   • Try reducing message length to under 40 bytes if possible, as issues are less common with shorter messages.

4. UART Configuration Check:

   • Verify your UART configuration using the following command:

     sudo stty -F /dev/ttyTHS1 -a
     

   • Ensure settings like speed, parity, and flow control are correct.

5. Disable nvgetty Service:

   • If not already done, disable the nvgetty service:

     systemctl disable nvgetty
     

6. JetPack Version Update:

   • Ensure you are using the latest JetPack version (at least 5.1.2 was mentioned in the discussion).
   • Check for any available updates or patches for UART-related issues.

7. Custom Device Tree Modifications:

   • If you’re using a custom carrier board, review and potentially modify the device tree entries related to UART configuration.

8. Loopback Testing:

   • Perform loopback tests to isolate the issue:
     • Connect UART TX to RX
     • Use a script similar to the one provided in the forum to send and receive data.
   • This can help determine if the problem is in the Jetson hardware/software or external devices.

9. Oscilloscope Analysis:

   • Use an oscilloscope to verify the integrity of the UART signals at various baud rates.
   • Look for any anomalies in the waveforms, especially during data loss events.

10. Consider Alternative Communication Methods:

    • If high-speed serial communication is crucial and cannot be resolved, consider using alternative interfaces like SPI or Ethernet for data transfer.

11. Report to NVIDIA Support:

    • If the issue persists after trying these solutions, consider reporting the problem to NVIDIA support with detailed reproduction steps and system information.

Remember to thoroughly test any changes in a non-production environment before implementing them in your final setup. The most effective solution appears to be enabling hardware flow control and limiting the baud rate to 4.3 Mbaud or lower for Orin NX/Nano modules.