Wifi traffic delays UDP messages over Ethernet

Issue Overview

Users of the Nvidia Jetson Orin Nano Developer Kit have reported experiencing significant delays in receiving UDP messages over the Ethernet interface (eth0) when unrelated traffic is generated on the Wi-Fi network (wlan0). The specific symptoms include:

• Delayed Responses: Typically, responses from custom robotic hardware connected to eth0 are received within 0.1 to 0.2 ms, but delays spike to approximately 2 ms when Wi-Fi traffic is introduced.
• Measurement Tools: Delays are measured using tools like tshark and wireshark, indicating a notable increase in latency during high Wi-Fi traffic conditions.
• Setup Context: The issue occurs in a setup where the Jetson Orin Nano is running Tegra release 35.5.0 and Jetpack 5.1.3, and the user is utilizing a high-priority thread to manage UDP communications.

This problem can severely impact real-time communication with robotic systems, leading to degraded performance and potential operational failures.

Possible Causes

Several potential causes have been identified for the observed delay spikes:

• Kernel or OS Settings: Misconfigurations in kernel parameters or OS settings could lead to improper prioritization of network traffic, affecting UDP message timing.

• Network Configuration Errors: Overlapping IP addresses or incorrect subnet masks between eth0 and wlan0 may cause routing issues, resulting in delays.

• Driver Issues: Outdated or incompatible network drivers could contribute to performance issues under load.

• Environmental Factors: High levels of traffic on wlan0 may saturate the available bandwidth, leading to increased latency on eth0.

• User Errors: Misconfigurations in network settings or thread priorities could also be responsible for the delays.

Troubleshooting Steps, Solutions & Fixes

To diagnose and potentially resolve the issue of increased UDP message delays, follow these comprehensive troubleshooting steps:

1. Verify Network Configuration:

   • Ensure that there are no overlapping IP addresses between eth0 and wlan0.
   • Check subnet masks for both interfaces to confirm they are correctly configured.

   Example configuration using netplan:

   network:
     ethernets:
       eth0:
         optional: true
         addresses: [192.168.2.1/24]
     version: 2
   
     wifis:
       wlan0:
         optional: true
         dhcp4: no
         addresses: [192.168.1.211/24]
         access-points:
            [...]
     version: 2
   

2. Test with Different Traffic Loads:

   • Use iperf to generate varying levels of traffic on wlan0 and measure the impact on UDP response times on eth0.
   • Command example:

   iperf -c [ip] -b 1000M
   

3. Monitor System Performance:

   • Utilize monitoring tools like htop or nload to observe CPU and network load during tests.

4. Adjust Thread Priorities:

   • Review and adjust thread priorities in your application to ensure that UDP handling threads are given higher priority over other tasks.

5. Update Drivers and Firmware:

   • Check for any available updates for network drivers or firmware related to the Jetson Orin Nano. Install updates as necessary.

6. Isolate Hardware Issues:

   • If possible, connect eth0 to a different host PC instead of the robotic hardware and replicate the issue to rule out hardware-specific problems.

7. Consult Documentation:

   • Review Nvidia’s official documentation for any known issues or recommended configurations related to networking on the Jetson Orin Nano.

8. Best Practices for Future Prevention:

   • Regularly monitor network configurations and performance metrics.
   • Keep software and drivers updated to mitigate potential bugs.

If users continue to experience issues after following these steps, further investigation may be necessary, including examining kernel logs or seeking additional support from community forums or Nvidia support channels.