Modifying TensorRT Sample Code for Batch Inference on Jetson Nano

Issue Overview

Users of the Nvidia Jetson Nano development board are encountering difficulties when attempting to modify the existing TensorRT sample code to perform inference with a batch size greater than one. The specific sample code in question is located at /usr/src/tensorrt/samples/python/network_api_pytorch_mnist/sample.py and model.py. By default, this code is designed to process only a single image with dimensions of 28×28 pixels (784 elements total) during inference. Users are seeking guidance on how to adapt this code to handle multiple images simultaneously, effectively increasing the batch size for more efficient processing.

Possible Causes

1. Limited Understanding of TensorRT API: Users may not be familiar with the specific TensorRT API calls required to modify input shapes and batch sizes.

2. Sample Code Limitations: The provided sample code may be intentionally simplified for demonstration purposes, not accounting for batch processing scenarios.

3. Configuration Oversight: Users might be overlooking the need to adjust input shapes and network configurations to accommodate larger batch sizes.

4. Hardware Constraints: The Jetson Nano’s hardware capabilities could potentially limit the maximum batch size that can be processed efficiently.

Troubleshooting Steps, Solutions & Fixes

1. Modify Input Shape:
   The key to enabling batch processing lies in adjusting the input shape of the network. For the MNIST dataset used in this sample, you need to modify the INPUT_SHAPE variable in sample.py.

   Change the input shape to include the desired batch size. For example, to process two images at once:

   INPUT_SHAPE = (2, 1, 28, 28)
   

   This shape represents:

   • Batch size: 2
   • Channels: 1 (grayscale images)
   • Height: 28 pixels
   • Width: 28 pixels

2. Adjust Data Generation:
   Modify the get_random_testcase() method in model.py to return multiple images. You may need to create a new method or adjust the existing one to generate a batch of images and their corresponding expected outputs.

3. Update Network Configuration:
   Ensure that the network builder in sample.py is configured to handle the new input shape. This may involve adjusting any hard-coded values related to input dimensions.

4. Verify EXPLICIT_BATCH Flag:
   Confirm that the EXPLICIT_BATCH flag is set when creating the network. This allows TensorRT to optimize for the specified batch size.

   with trt.Builder(TRT_LOGGER) as builder, builder.create_network(EXPLICIT_BATCH) as network:
       # Network definition
   

5. Adjust Output Processing:
   Update any post-processing code to handle the batch of outputs rather than a single output.

6. Performance Considerations:

   • Start with small batch sizes (e.g., 2 or 4) and gradually increase to find the optimal balance between throughput and memory usage.
   • Monitor system resources to ensure the Jetson Nano can handle the increased workload.

7. Testing and Validation:

   • After making these changes, thoroughly test the modified code with various batch sizes.
   • Verify that the results are correct for all images in the batch.

8. Documentation and Resources:

   • Refer to the official TensorRT documentation for more detailed information on working with batched inputs.
   • Explore NVIDIA’s Developer Forums or GitHub repositories for additional examples of batch processing with TensorRT on Jetson platforms.

By following these steps, users should be able to successfully modify the sample code to perform inference with batch sizes greater than one on their Jetson Nano devices. This approach has been confirmed to work by the original poster of the forum thread.