Deleting Models from Jetson-Inference on Nvidia Jetson Orin Nano

Issue Overview

Users of the Nvidia Jetson Orin Nano Development Kit (8GB) are experiencing difficulties with managing model files created during the training process using the train-ssd.py script in the jetson-inference repository. The primary symptoms include:

• Locked Folders: Users report that folders containing failed or unwanted models are locked, preventing their deletion and consuming significant memory card space.

• Training Failures: Many users have encountered issues where training processes are abruptly terminated, often indicated by "killed" errors due to memory limitations. This occurs during the setup and execution of model training for detecting specific objects, such as boats.

• Memory Management Issues: Users are unsure about how to effectively manage memory during training and whether their configurations (e.g., batch size, number of workers) are appropriate.

The issue appears to be frequent among users attempting to train models with varying dataset sizes (from 2,500 to 26,000 images) and configurations. The impact on user experience includes frustration over wasted resources and inability to proceed with new model training due to storage constraints.

Possible Causes

Several potential causes for these issues have been identified:

• Hardware Limitations: The Jetson Orin Nano has limited memory capacity (8GB), which can lead to out-of-memory errors during intensive tasks like model training.

• Software Bugs or Configuration Errors: Incorrect settings for batch size or worker threads can exacerbate memory issues. Users have reported trying various configurations without success.

• Driver Issues: Outdated or incompatible drivers may hinder performance and functionality during model training.

• Environmental Factors: Insufficient power supply or overheating could affect performance stability during intensive operations.

• User Errors: Misconfigurations in the training setup or misunderstanding of how to manage files and directories may lead to locked folders and failed trainings.

Troubleshooting Steps, Solutions & Fixes

To address the issues related to deleting models and managing training processes on the Jetson Orin Nano, users can follow these comprehensive troubleshooting steps:

1. Check Memory Status:

   • Use the command:

     sudo tegrastats
     

   • Monitor memory usage while training to identify if it is close to full capacity.

2. Adjust Training Parameters:

   • Reduce batch size and number of worker threads:
     • Set batch-size=1 and workers=0 for initial tests.
   • Start with a smaller dataset (500-1,000 images) to ensure the training process works before scaling up.

3. Delete Locked Model Files:

   • If model files are locked but need deletion:
     • Navigate to the directory using terminal commands.
     • Use rm command cautiously to remove unwanted files:

       sudo rm -rf /path/to/locked/model/folder
       

4. Utilize SWAP Memory:

   • If encountering out-of-memory errors, consider mounting SWAP memory:
     • Follow instructions from the provided GitHub link on setting up SWAP and disabling ZRAM.
   • Ensure that SWAP is correctly configured by checking available memory after setup.

5. Train on a Desktop Environment:

   • For large datasets, consider offloading training to a more powerful desktop system.
   • Clone the pytorch-ssd submodule from jetson-inference and run training locally if compatible hardware is available.

6. Experiment with Dataset Size:

   • Assess whether using fewer images (e.g., 10,000 instead of 26,000) improves model performance and reduces bias towards a single object class.

7. Documentation and Resources:

   • Refer to official documentation for Jetson Inference for further guidance on model training and optimization techniques.
   • Check for driver updates that may improve system performance.

8. Best Practices Going Forward:

   • Regularly monitor system resources during intensive tasks.
   • Maintain backups of important model files before deletion.
   • Experiment with different hyperparameters and dataset configurations iteratively for optimal results.

By following these steps, users can effectively troubleshoot their issues with model management on the Nvidia Jetson Orin Nano while optimizing their machine learning workflows.