Error Code 10: Internal Error (Could not find any implementation for node PWN(/model.0/act/Sigmoid).)

Issue Overview

Users are encountering an Error Code 10 while attempting to convert a YOLOv7 ONNX model to a TensorRT engine on the Nvidia Jetson Orin Nano Dev board. The specific error message is:

Error Code 10: Internal Error (Could not find any implementation for node PWN(/model.0/act/Sigmoid).)

Symptoms

• The error occurs during the conversion process using the export.py script with INT8 calibration.
• The issue is consistent across attempts to convert the model, leading to frustration among users.

Context

• The problem arises when converting the YOLOv7 model using TensorRT on two different machines:

  • RTX 3090 with TensorRT version 8.6.1 and CUDA version 11.4.
  • Jetson Orin Nano with TensorRT version 8.5.2-1+cuda11.4.

• Users report that while the conversion works correctly on the RTX 3090, it fails on the Jetson Orin Nano when INT8 calibration is enabled but works fine with FP16 and FP32 models.

Impact

This issue significantly hampers users’ ability to deploy optimized models on the Jetson Orin Nano, affecting their development workflow and potentially delaying project timelines.

Possible Causes

• Hardware Incompatibilities: Differences in GPU architecture between RTX 3090 and Jetson Orin Nano may lead to unsupported operations in TensorRT.

• Software Bugs or Conflicts: The discrepancy in TensorRT versions could introduce bugs or conflicts that affect model conversion.

• Configuration Errors: Incorrect settings in the export script or TensorRT configuration may prevent successful INT8 calibration.

• Driver Issues: Outdated or incompatible drivers could lead to failures in model processing.

• Environmental Factors: Memory limitations on the Jetson Orin Nano (4GB vs. 8GB) might restrict certain operations, although users have indicated that memory is not the primary issue.

• User Errors or Misconfigurations: Users may inadvertently misconfigure parameters or paths in their scripts, leading to errors during execution.

Troubleshooting Steps, Solutions & Fixes

Step-by-Step Instructions

1. Verify Software Versions:

   • Ensure that both machines are running compatible versions of TensorRT and CUDA.
   • For Jetson Orin Nano, consider upgrading to TensorRT 8.6, which is available in JetPack 6DP.

2. Check Model Compatibility:

   • Confirm that the YOLOv7 model and its operations are supported by TensorRT on the Jetson Orin Nano.
   • Review the ONNX model for unsupported nodes that may cause conversion failures.

3. Run Conversion Without INT8 Calibration:

   • As a temporary workaround, run the conversion using FP16 or FP32 settings:

     python export.py -o yolov7.onnx -e yolov7.trt --end2end --fp16
     

4. Inspect and Modify Export Script:

   • If issues persist, consider modifying the export.py script to adjust configurations for better compatibility with Jetson Orin Nano.
   • Engage with the script’s author for insights into potential changes required for successful execution.

5. Memory Management:

   • Although users reported that memory limitations are not a direct cause, ensure that workspace memory settings are optimized for your hardware configuration.
   • Set workspace memory explicitly if needed:

     builder.maxWorkspaceSize = size_in_bytes
     

6. Use trtexec for Testing:

   • Test model conversion using trtexec as a comparison method:

     /usr/src/tensorrt/bin/trtexec --onnx=yolov7.onnx --saveEngine=yolov7.engine --fp16 --int8
     

Recommended Solutions

• Upgrading TensorRT to version 8.6 has been suggested by other users as a potential fix, although some have still encountered issues post-upgrade.

• Testing with different configurations and ensuring all dependencies are up-to-date can help isolate and resolve the problem.

Best Practices for Prevention

• Regularly update software components (TensorRT, CUDA, drivers) to minimize compatibility issues.

• Thoroughly review documentation for ONNX models and TensorRT capabilities before deployment.

• Engage with community forums or support channels to share experiences and solutions regarding similar issues.

Unresolved Aspects

The discussion indicates ongoing challenges with specific models (YOLOv7, YOLOv8) on various hardware configurations, suggesting further investigation is needed into compatibility and optimization strategies specific to Jetson Orin Nano’s architecture and memory constraints.