Runtime Scheduling Error and pthread_create – Nvidia SDK Components

Issue Overview

Users of the Nvidia Jetson Orin Nano have reported encountering a runtime scheduling error specifically related to the pthread_create function when utilizing applications that include NVIDIA SDK components. The error message displayed is:

pthread_create returned 1 in file "<file directory>/<name_of_the_program>.cpp"
ETL: nullptr pointer dereference: nullpointer ETL ScopedPtr

This issue arises consistently when running personal applications that leverage these SDK components, but does not occur when the system is flashed with only the Jetson Linux OS. The problem manifests during application execution, leading to thread creation failures. Users have noted that the error is prevalent after installing the NVIDIA SDK components and persists even after attempts to purge these components from the system using commands such as sudo apt purge nvidia-jetpack followed by sudo apt autoremove.

The impact of this issue is significant, as it prevents users from executing their applications effectively, thereby hindering development and deployment processes. Users have also expressed concerns regarding the workaround of modifying kernel parameters, which is neither optimal nor permanent.

Possible Causes

Several potential causes for this issue have been identified:

• Hardware Incompatibilities or Defects: There may be compatibility issues between the Jetson Orin Nano and specific SDK components.
• Software Bugs or Conflicts: Bugs within the NVIDIA SDK or conflicts between different SDK components may lead to errors in thread management.
• Configuration Errors: Improper configurations during setup or installation of SDK components could cause runtime issues.
• Driver Issues: Outdated or incompatible drivers may affect system performance and functionality.
• Environmental Factors: Factors such as power supply inconsistencies or thermal issues could impact system stability.
• User Errors or Misconfigurations: Incorrect permissions or ownership settings on application files can prevent successful execution.

Troubleshooting Steps, Solutions & Fixes

To address the runtime scheduling error, users can follow these comprehensive troubleshooting steps:

1. Diagnose Permissions:

   • Check if the application has appropriate permissions. Use:

     sudo chown root:root <name_of_your_application>
     sudo chmod 777 <name_of_your_application>
     

   • Ensure that the application is being executed with sufficient privileges.

2. Test Thread Creation Capability:

   • Compile and run a simple test program (provided below) to verify if thread creation works independently of the main application:

     #include <stdio.h>
     #include <stdlib.h>
     #include <pthread.h>
     
     static void *_Thread(void *arg) {
         printf("Thread running!\n");
         return NULL;
     }
     
     int main(void) {
         pthread_t thread;
         int retVal = pthread_create(&thread, NULL, _Thread, NULL);
         if (retVal) {
             fprintf(stderr, "pthread_create error %d\n", retVal);
             exit(1);
         }
         pthread_join(thread, NULL);
         printf("Main run successfully\n");
         return 0;
     }
     

   • Compile with:

     gcc -o test_pthread_arm64 test_pthread_arm64.c -pthread
     

3. Modify Kernel Parameters Temporarily:

   • As a temporary workaround, execute:

     sudo sysctl -w kernel.sched_rt_runtime_us=-1
     

   • Note that this change is not permanent and will revert after a reboot.

4. Persist Kernel Parameter Changes:

   • To make kernel parameter changes persistent across reboots, add the following line to /etc/sysctl.conf:

     kernel.sched_rt_runtime_us = -1
     

   • Then run:

     sudo sysctl -p
     

5. Investigate SDK Component Removal:

   • If necessary, attempt to remove NVIDIA SDK components using:

     sudo apt purge nvidia-jetpack
     sudo apt autoremove
     

   • Reboot and test if the application runs without errors.

6. Check for Environmental Factors:

   • Ensure that power supply and thermal conditions are optimal for device operation.

7. Use strace for Debugging:

   • Utilize strace to trace system calls made by your application which can help identify where it fails:

     strace -v -s 128 ./<your_application>
     

8. Contact NVIDIA Support:

   • If issues persist despite following these steps, consider reaching out to NVIDIA support for further assistance.

9. Best Practices for Future Prevention:

   • Regularly update JetPack and SDK components.
   • Maintain backups of working configurations before making significant changes.
   • Document any modifications to system settings for future reference.

By following these steps, users should be able to diagnose and potentially resolve the pthread_create errors encountered while using applications on the Nvidia Jetson Orin Nano with SDK components installed.