Jetson Orin Nano

Issue Overview

Users have reported a recurring issue with the Nvidia Jetson Orin Nano Dev board where the system becomes unresponsive during shutdown after conducting power on/off stress tests, specifically around 600 cycles. The problem appears to be linked to the GPU, as indicated by error messages in the logs associated with GPU submission failures.

Symptoms

• System hangs or becomes unresponsive during shutdown.
• Frequent error messages in the logs related to GPU operations, particularly with nvgpu and gk20a.
• Specific errors include:
  • failed to host gk20a to submit gpfifo
  • failed to power on gpu

Context

• Occurs during power cycling stress tests.
• SDK version in use is Jetson 35.3.1.
• Users are testing on a carrier board configuration.

Frequency and Impact

• The issue appears to be consistent under the described conditions, impacting user experience significantly by preventing normal shutdown procedures and potentially leading to system instability.

Possible Causes

• Hardware Incompatibilities or Defects: The carrier board may have compatibility issues with the Jetson Orin Nano or defects that lead to instability under stress.

• Software Bugs or Conflicts: The specific SDK version (35.3.1) may contain bugs that affect GPU functionality during shutdown.

• Configuration Errors: Incorrect configurations in the software settings may lead to improper handling of GPU resources.

• Driver Issues: Outdated or incompatible drivers could prevent proper communication between the operating system and the GPU.

• Environmental Factors: Power supply instability or overheating during stress testing could contribute to system failures.

• User Errors or Misconfigurations: Improper setup or usage patterns could inadvertently trigger these issues.

Troubleshooting Steps, Solutions & Fixes

1. Update SDK Version:

   • Test with the latest JetPack version (5.1.2) and L4TR (35.4.1) as suggested by users in the forum. This may resolve known issues present in earlier releases.
   • To update, follow these commands:

     sudo apt update
     sudo apt install nvidia-jetpack
     

2. Check Logs for Errors:

   • Review system logs for any additional error messages that might provide insight into the issue.
   • Use the following command to access logs:

     dmesg | grep nvgpu
     

3. Run Diagnostics:

   • Perform hardware diagnostics to ensure there are no underlying hardware issues with the carrier board or GPU.
   • Use Nvidia’s diagnostic tools if available.

4. Test Power Supply:

   • Ensure that the power supply is stable and meets the required specifications for the Jetson Orin Nano.
   • Consider using a different power source if instability is suspected.

5. Isolate Variables:

   • If possible, test with different configurations (e.g., different carrier boards or peripherals) to isolate whether the issue is hardware-specific.

6. Monitor Temperature:

   • Check for overheating during operation, especially under stress testing conditions, which could lead to system failures.
   • Use temperature monitoring tools available for Linux systems.

7. Rollback Changes:

   • If recent changes were made prior to experiencing this issue, consider rolling back those changes to see if it resolves the problem.

8. Community Feedback:

   • Engage with community forums for additional insights or shared experiences regarding similar issues.

9. Documentation and Support:

   • Refer to Nvidia’s official documentation for any known issues related to SDK versions and recommended fixes.
   • Consider reaching out directly to Nvidia support for unresolved issues after following troubleshooting steps.

By following these troubleshooting steps, users can potentially resolve the shutdown issue experienced with their Nvidia Jetson Orin Nano Dev board while also contributing valuable feedback and data back to Nvidia for further investigation into this matter.