NVJPEG Decode Performance Issues on Jetson Orin Nano

Issue Overview

Users are experiencing slow JPEG decoding performance on the Nvidia Jetson Orin Nano developer kit. Specifically, the decoding speed for Full HD JPEG images is significantly lower than expected, with average latencies ranging from 87,793 to 90,901 microseconds per image. This performance is more than six times worse than anticipated, even when using the latest JetPack 5.1.3 software. The issue persists across various test scenarios, including single-stream, dual-stream, and multi-stream decoding attempts.

Possible Causes

1. Software configuration: Suboptimal settings in the JetPack or NVIDIA drivers may be limiting the JPEG decoder’s performance.

2. Hardware limitations: The Jetson Orin Nano’s NVJPEG engine might be running at a lower clock speed (115 MHz) than its maximum capability (499.2 MHz).

3. Image complexity: The specific JPEG file being used (Definitions_of_TV_standards.jpg) may be more complex than standard test images, leading to longer decoding times.

4. System load: Background processes or services might be consuming resources, affecting the decoder’s performance.

5. Power management: Incorrect power settings could be limiting the hardware’s performance capabilities.

6. Driver or firmware issues: There might be bugs or incompatibilities in the current software stack affecting the NVJPEG decoder.

Troubleshooting Steps, Solutions & Fixes

1. Verify JetPack version:
   Ensure you are running the latest JetPack version (currently 5.1.3). If not, upgrade using the NVIDIA SDK Manager.

2. Check power settings:
   Confirm that NV Power is set to 15W and that jetson_clocks is running. Use the following commands:

   sudo nvpmodel -q
   sudo jetson_clocks
   

3. Monitor system resources:
   Use tools like top or htop to check for any resource-intensive background processes that might be affecting performance.

4. Test with different JPEG files:
   Generate a standard test JPEG using the following command and compare performance:

   gst-launch-1.0 videotestsrc num-buffers=1 ! video/x-raw,width=1920,height=1080 ! jpegenc ! filesink location=test1.jpg
   

   Then run the decode test with this new file:

   ./jpeg_decode num_files 1 test1.jpg --perf
   

5. Apply NVJPEG decoder fix:
   Implement the fix mentioned in the NVIDIA Developer Forums to address a potential issue with the NvJPEGDecoder:
   https://forums.developer.nvidia.com/t/nvjpegdecoder-generates-the-same-output-if-called-twice-with-different-input-buffer/231079/7

6. Check hardware engine status:
   Verify the status of hardware engines, especially NVJPG and NVJPG1, using:

   sudo tegrastats
   

   Ensure they are not being used by other processes when your decoding test ends.

7. Test without X server:
   Try running the decoding test without the X server to eliminate any potential interference:

   sudo systemctl stop gdm3
   sudo ./jpeg_decode num_files 1 ~/Definitions_of_TV_standards.jpg --perf
   

8. Update NVIDIA drivers:
   Check for any available updates to the NVIDIA drivers specific to the Jetson Orin Nano.

9. Monitor clock speeds:
   Use tegrastats or other NVIDIA-provided tools to monitor the clock speeds of the NVJPEG engine during decoding. If it’s stuck at 115 MHz, investigate power management or thermal throttling issues.

10. File a bug report:
    If the issue persists after trying these steps, consider filing a detailed bug report with NVIDIA, including your system specifications, JetPack version, and test results.

Remember to document the results of each troubleshooting step to help identify patterns or specific conditions that affect the decoding performance. If a particular solution significantly improves performance, please share your findings with the NVIDIA developer community to help others facing similar issues.