JetRacer error: safeSerializationVersion failed.Version tag does not match. Note: Current Version: 0, Serialized Engine Version: 97

Issue Overview

Users are encountering an error while attempting to run the JetRacer road following model on the Nvidia Jetson Orin Nano. The specific error message is: "safeSerializationVersion failed. Version tag does not match. Note: Current Version: 0, Serialized Engine Version: 97." This issue arises after successfully installing JetCam, torch2trt, and JetRacer, and testing both the camera and IMU, which function correctly. The problem occurs when loading the road following model, indicating a potential mismatch in versions between the model and the TensorRT engine.

Symptoms

• Error message during model loading.
• Successful operation of camera and IMU.
• The torch version is located at ~/.local/lib/python3.8/site-packages, while torch2trt is at /usr/local/lib/python3.8/dist-packages.

Context

The issue appears during the execution of a notebook designed for road following, which suggests that it may be related to the conversion or compatibility of models between different versions of hardware (from Jetson Nano to Jetson Orin Nano).

Frequency

The issue seems to be consistent among users who have migrated models from older Jetson devices to the Orin Nano.

Impact

This error prevents users from executing their intended applications, significantly affecting their workflow and productivity.

Possible Causes

1. Hardware Incompatibilities:

   • The transition from Jetson Nano to Orin Nano may introduce incompatibilities in model serialization formats.

2. Software Bugs:

   • There may be bugs in TensorRT or related libraries that affect version compatibility.

3. Configuration Errors:

   • Incorrect paths or configurations for installed libraries could lead to version mismatches.

4. Driver Issues:

   • Outdated or incompatible drivers could contribute to serialization errors.

5. User Errors:

   • Users may have multiple installations of libraries like Torch, leading to confusion over which version is being used.

6. Environmental Factors:

   • Differences in system configurations or dependencies when moving from one device to another could affect performance.

Troubleshooting Steps, Solutions & Fixes

1. Remove Duplicate Installations:

   • Navigate to ~/.local/lib/python3.8/site-packages and remove any duplicate Torch installations.

   rm -rf ~/.local/lib/python3.8/site-packages/torch
   

2. Verify Torch Installation:

   • Check which version of Torch is currently active.

   python3 -c "import torch; print(torch.__version__)"
   

   Ensure it points to /usr/local/lib/python3.8/dist-packages.

3. Recreate TensorRT Engine:

   • If you have the original model file (.onnx), recreate the TensorRT engine specifically for the Orin Nano.

   # Example command to convert .pth to .onnx
   python jetson-inference/convert.py --input model.pth --output model.onnx
   

4. Run Optimization Script:

   • Execute the optimization script provided in the JetRacer repository (optimize_model.ipynb) on the Orin Nano to ensure compatibility.

5. Check Environment Variables:

   • Ensure that environment variables are correctly set to include paths for necessary libraries.

   export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.8/dist-packages
   

6. Update Drivers and Libraries:

   • Regularly check for updates for TensorRT and other libraries through Nvidia’s SDK Manager.

7. Consult Documentation:

   • Refer to Nvidia’s official documentation for any additional steps regarding model conversion and optimization specific to the Orin Nano.

8. Community Support:

   • Engage with community forums for additional insights or similar experiences shared by other users.

Recommended Approach

Multiple users reported success after removing duplicate Torch installations and recreating their models using the appropriate scripts for their new hardware setup. This method is highly recommended as a first step in resolving serialization issues.

Unresolved Aspects

Further investigation may be needed into specific version compatibility between TensorRT and models trained on older Jetson devices, as well as any potential bugs within the library that could cause serialization errors during execution on newer hardware platforms.