Deploying YOLOv8 on Jetson Orin Nano: INT64 Weights and Memory Issues

Issue Overview

Users are experiencing difficulties when deploying YOLOv8 models on the Nvidia Jetson Orin Nano developer board with Jetpack 5.1.1. The main symptoms include:

• Warnings about INT64 weights being cast down to INT32
• Warnings about insufficient device memory for certain tactics
• Significantly slow inference times, particularly on the first run
• Serialization errors when attempting to use TensorRT engine files

These issues occur during the model deployment process, affecting the overall performance and usability of YOLOv8 on the Jetson Orin Nano platform. The problem appears to be consistent across different users and persists even when exporting the model in various formats.

Possible Causes

1. INT64 Weights Incompatibility: The YOLOv8 model is being exported with INT64 weights, which are not natively supported by TensorRT. This causes the system to attempt casting down to INT32.

2. Limited Device Memory: The Jetson Orin Nano has limited memory compared to desktop GPUs, causing insufficient memory for certain TensorRT tactics.

3. Environment Mismatch: The model is being trained and exported on a different system (e.g., desktop with Quadro RTX 4000) than the deployment platform (Jetson Orin Nano), potentially causing compatibility issues.

4. TensorRT Engine Serialization: Attempts to create and deserialize TensorRT engine files are failing due to environment differences between creation and inference.

5. Suboptimal Export Parameters: The model export process may not be optimized for the Jetson platform, leading to performance issues.

Troubleshooting Steps, Solutions & Fixes

1. Address INT64 Weights Warning:

   • This warning is common and usually doesn’t cause significant issues as the values are typically within INT32 range.
   • When exporting the model, try using the --int8 flag to quantize the model:

     yolo export model=yolov8n.pt format=onnx int8=True
     

2. Optimize for Limited Memory:

   • Use a smaller variant of YOLOv8 (e.g., YOLOv8n or YOLOv8s) to reduce memory requirements.
   • When exporting, use the --optimize flag to enable ONNX optimization:

     yolo export model=yolov8n.pt format=onnx optimize=True
     

3. Proper Environment Setup:

   • Ensure you’re using the correct versions of software compatible with Jetson Orin Nano:
     • Jetpack: 5.1.1
     • TensorRT: 8.5.2.2
     • ONNX Runtime: 1.15.1
     • Python: 3.8.10

4. TensorRT Engine Optimization:

   • Generate the TensorRT engine directly on the Jetson Orin Nano to ensure compatibility:

     import tensorrt as trt
     logger = trt.Logger(trt.Logger.WARNING)
     builder = trt.Builder(logger)
     network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
     parser = trt.OnnxParser(network, logger)
     success = parser.parse_from_file("path_to_your_onnx_model.onnx")
     config = builder.create_builder_config()
     config.max_workspace_size = 1 << 30  # 1GB
     engine = builder.build_engine(network, config)
     

5. Export Optimization:

   • Try different export options provided by Ultralytics:

     yolo export model=yolov8n.pt format=onnx dynamic=True
     

   • If the above doesn’t work, try simplifying the model:

     yolo export model=yolov8n.pt format=onnx simplify=True
     

6. Performance Optimization:

   • The slow first inference is likely due to TensorRT optimization. This is normal and subsequent inferences should be faster.
   • To avoid this warmup time in production, you can run a dummy inference after loading the model.

7. Memory Management:

   • Monitor the Jetson’s memory usage using tegrastats and ensure no other memory-intensive processes are running.
   • Consider using NVIDIA’s DeepStream SDK for optimized inference on Jetson platforms.

8. Update Software:

   • Regularly check for updates to Jetpack, TensorRT, and ONNX Runtime, as newer versions may include optimizations and bug fixes for Jetson platforms.

If issues persist after trying these solutions, consider reaching out to NVIDIA’s developer forums or Ultralytics’ support channels for more specific assistance tailored to your use case and model architecture.