OSError: libmpi_cxx.so.20: cannot open shared object file during PyTorch installation on Jetson Orin Nano

Issue Overview

Users have reported encountering an OSError while attempting to compile Torchvision on the Nvidia Jetson Orin Nano Dev board, specifically receiving the error message:

OSError: libmpi_cxx.so.20: cannot open shared object file: No such file or directory

This issue arises during the installation of Torchvision, which is a library required for various computer vision tasks. The problem appears when users attempt to run commands related to PyTorch, indicating that it may not be limited to just Torchvision.

Context of the Problem

• Environment: Users are operating within a JetPack 5.1.3 environment, which is based on Ubuntu 20.04.
• Symptoms: The error occurs when running commands that depend on libmpi_cxx.so.20, which is not found in the expected locations.
• Frequency: Multiple users have reported similar issues, suggesting a systemic problem related to library dependencies.
• Impact: This error prevents users from utilizing Torchvision, significantly hindering their development work in machine learning and computer vision.

Possible Causes

1. Library Version Mismatch:

   • The specific version of the PyTorch wheel installed may not be compatible with JetPack 5.1.3, leading to missing dependencies.

2. Incorrect Installation of Dependencies:

   • Users may have installed versions of OpenMPI or other libraries that do not include the required libmpi_cxx.so.20 file.

3. Configuration Errors:

   • Misconfiguration of environment variables such as LD_LIBRARY_PATH could lead to the system being unable to locate the necessary libraries.

4. User Errors:

   • Users may inadvertently install incompatible versions or fail to follow proper installation procedures, leading to missing files.

5. Driver Issues:

   • Potential issues with drivers or system libraries that are not aligned with the installed version of JetPack.

Troubleshooting Steps, Solutions & Fixes

Step-by-Step Instructions

1. Verify JetPack Version:

   • Ensure you are using JetPack 5.1.3 by running:

     cat /etc/nv_tegra_release
     

2. Check Installed Libraries:

   • Use the following command to locate MPI libraries:

     sudo find / -name 'libmpi*'
     

3. Update Environment Variables:

   • Add OpenMPI library paths to your LD_LIBRARY_PATH:

     export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/openmpi/lib:$LD_LIBRARY_PATH
     

4. Uninstall Existing PyTorch Installation:

   • Remove any existing installations of PyTorch and related dependencies:

     pip3 uninstall torch torchvision
     

5. Reinstall PyTorch from Official Source:

   • Download and install the appropriate wheel for JetPack 5.x from the official NVIDIA repository:

     pip3 install https://developer.download.nvidia.cn/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
     

6. Purge and Reinstall OpenMPI:

   • If issues persist, purge OpenMPI and reinstall it:

     sudo apt-get purge -y libopenmpi-dev libopenmpi* openmpi-bin
     sudo apt-get install -y libopenmpi-dev openmpi-bin
     

7. Create Symlinks (if necessary):

   • If libmpi_cxx.so is still missing, create symlinks for existing versions (only if necessary):

     sudo ln -s /usr/lib/aarch64-linux-gnu/libmpi_cxx.so /usr/lib/aarch64-linux-gnu/libmpi_cxx.so.20
     

Additional Recommendations

• Reboot After Changes: After making significant changes like purging or reinstalling libraries, reboot your system to ensure all changes take effect.

• Check Compatibility: Always verify that the versions of libraries and wheels are compatible with your specific JetPack version.

Best Practices

• Regularly check for updates from NVIDIA regarding compatibility between JetPack versions and various libraries.

• Maintain a backup of working configurations before making significant changes.

Unresolved Aspects

While several users have successfully resolved their issues by following these steps, some aspects remain uncertain, particularly regarding specific library dependencies for different versions of PyTorch and their interaction with JetPack updates. Further investigation may be needed into how these dependencies evolve with future releases.