cuDNN Version Incompatibility with PyTorch 2.0.0 on Nvidia Jetson Orin Nano Dev Board

Issue Overview

Users of the Nvidia Jetson Orin Nano Dev board are experiencing difficulties when trying to install and use torchaudio with PyTorch 2.0.0. The primary issue revolves around a cuDNN version incompatibility. Specifically, users are encountering runtime errors related to mismatched cuDNN versions when attempting to import and use libraries that depend on torchaudio.

The error occurs in the context of running deep learning models, particularly when initializing the model and moving it to the GPU. The problem appears to be consistent across attempts to use torchaudio with PyTorch 2.0.0 on the Jetson platform running JetPack 5.0.2.

This incompatibility significantly impacts the user’s ability to run deep learning models that rely on torchaudio, effectively preventing the use of certain audio processing capabilities on the Jetson device.

Possible Causes

1. Version Mismatch: The primary cause of this issue is a mismatch between the cuDNN version that PyTorch 2.0.0 was compiled against (8.6.0) and the runtime version available on the Jetson device (8.4.1).

2. JetPack Version: JetPack 5.0.2, which is installed on the user’s device, predates the support for PyTorch 2.0 wheels. The PyTorch 2.0 wheels were built against JetPack 5.1, which includes a newer version of cuDNN.

3. Library Path Conflict: The error message suggests that there might be an incompatible version of cuDNN in the LD_LIBRARY_PATH, which is interfering with the bundled version that comes with PyTorch.

4. Incomplete Installation: While the user was able to install torchaudio successfully, the underlying cuDNN incompatibility wasn’t apparent until trying to use a dependent library.

Troubleshooting Steps, Solutions & Fixes

1. Downgrade PyTorch:
   The most straightforward solution is to downgrade PyTorch to version 1.13, which was the last version compatible with JetPack 5.0. To do this:

   pip uninstall torch torchaudio
   pip install torch==1.13.0 torchaudio
   

2. Upgrade JetPack:
   If you need to use PyTorch 2.0.0, consider upgrading your JetPack to version 5.1 or later. This will provide the necessary cuDNN version (8.6.0) required by PyTorch 2.0.0. Follow the official Nvidia documentation for upgrading JetPack on your Jetson device.

3. Check and Clean LD_LIBRARY_PATH:
   Inspect your LD_LIBRARY_PATH for any conflicting cuDNN versions:

   echo $LD_LIBRARY_PATH
   

   Remove any paths that might be pointing to older cuDNN versions. You can temporarily unset the variable to test:

   unset LD_LIBRARY_PATH
   

4. Manually Install Compatible cuDNN:
   If upgrading JetPack is not an option, you can try manually installing cuDNN 8.6.0. However, this approach is not recommended and may lead to system instability. If you decide to proceed:

   • Download cuDNN 8.6.0 for ARM from the NVIDIA Developer website
   • Extract and copy the files to the appropriate system directories
   • Update your LD_LIBRARY_PATH to include the new cuDNN location

5. Use Containerization:
   Consider using a container-based approach as suggested in the forum:

   • Use the l4t-pytorch container for JetPack 5 that already has torchaudio installed
   • Or, build torchaudio in a container using the provided script:

     #!/usr/bin/env bash
     set -ex
     echo "Building torchaudio ${TORCHAUDIO_VERSION}"
     
     apt-get update
     apt-get install -y --no-install-recommends \
         git \
         pkg-config \
         libffi-dev \
         libsndfile1
     
     rm -rf /var/lib/apt/lists/*
     apt-get clean
     
     git clone --branch v${TORCHAUDIO_VERSION} --recursive --depth=1 https://github.com/pytorch/audio /opt/torchaudio
     cd /opt/torchaudio
     git checkout v${TORCHAUDIO_VERSION}
     
     BUILD_SOX=1 python3 setup.py bdist_wheel --verbose --dist-dir /opt
     

6. Verify Installation:
   After applying any of the above solutions, verify the installation:

   import torch
   import torchaudio
   print(torch.__version__)
   print(torch.cuda.is_available())
   print(torch.backends.cudnn.version())
   

7. Check for Updates:
   Regularly check for updates to PyTorch, torchaudio, and JetPack, as newer versions may resolve compatibility issues.

If these solutions do not resolve the issue, consider reaching out to NVIDIA developer support or the PyTorch community forums with detailed logs and your system configuration for further assistance.