Running Quantized TensorFlow Lite Models on Jetson Orin Nano GPU

Issue Overview

Users are experiencing difficulties running quantized TensorFlow Lite (TFLite) models on the Jetson Orin Nano GPU. Specifically, the issue arises when attempting to run uint8 quantized TFLite models. The problem occurs during the model execution phase, impacting the ability to perform benchmarking and comparisons with other platforms like EdgeTPU. This issue significantly affects the usability of the Orin Nano for certain AI applications that rely on quantized models.

Possible Causes

1. TensorFlow Lite GPU support: TFLite does not natively support GPU execution on the Jetson Orin Nano platform.

2. TensorRT compatibility: When converting TFLite models to ONNX for TensorRT execution, there’s a mismatch in integer type support. TensorRT only supports signed int8, while the model uses uint8.

3. Framework limitations: The specific requirements of uint8 quantized models may not be fully supported by the available deep learning frameworks on the Jetson platform.

4. Hardware constraints: The Jetson Orin Nano GPU may have specific limitations regarding the types of quantized operations it can efficiently execute.

Troubleshooting Steps, Solutions & Fixes

1. Attempt ONNX conversion:

   • Convert the TFLite model to ONNX format.
   • Try running the ONNX model using TensorRT.
   • If you encounter an error stating "only signed int8 is supported," proceed to the next steps.

2. Explore ONNX Runtime with CUDA:

   • Install ONNX Runtime with CUDA support for Jetson platforms.
   • Download the appropriate package from: https://www.elinux.org/Jetson_Zoo#ONNX_Runtime
   • Install the package following the provided instructions.

3. Run the model using ONNX Runtime:

   • Convert your TFLite model to ONNX if not already done.
   • Use ONNX Runtime API to load and run the model on the Jetson Orin Nano.
   • Ensure CUDA execution providers are properly configured.

4. Alternative TensorFlow installation:

   • If your model works with standard TensorFlow (non-Lite version), consider installing TensorFlow for Jetson Platform.
   • Follow the installation guide at: NVIDIA Docs – Installing TensorFlow for Jetson Platform

5. Quantization adjustment:

   • If possible, consider re-quantizing your model to use signed int8 instead of uint8.
   • This may allow you to use TensorRT directly, which has better support on the Jetson platform.

6. Benchmark and compare:

   • After successfully running the model with ONNX Runtime or an alternative solution, perform benchmarking.
   • Compare the results with EdgeTPU to ensure consistency in your experiments.

7. Stay updated:

   • Regularly check for updates to ONNX Runtime, TensorRT, and other relevant packages for the Jetson platform.
   • New versions may introduce improved support for different quantization schemes.

8. Community resources:

   • Consult the NVIDIA Developer Forums for the latest information and community-driven solutions.
   • Share your findings and successful approaches to help other users facing similar issues.

By following these steps, users should be able to run their quantized TFLite models on the Jetson Orin Nano GPU using ONNX Runtime with CUDA support. This solution provides a workaround for the limitations of TFLite and TensorRT regarding uint8 quantized models on this specific hardware platform.