Opening Multiple /dev/videoX Devices on Jetson Orin Nano: Order-Dependent Issues

Issue Overview

Users of the Nvidia Jetson Orin Nano development board are experiencing difficulties when attempting to open multiple /dev/videoX devices simultaneously. The issue specifically arises when trying to open four devices in a specific order. When opening the devices in the order of 1, 2, 3, 4, only the first device (1) can be opened successfully. Subsequent devices fail to open, generating an error message: "LIBV4L2ARGUS ERROR: CAM_CTX(0xffff9403b750) ERROR Generated. ARGUS Status : CANCELLED". However, when the devices are opened in the reverse order (4, 3, 2, 1), all devices can be opened and function normally.

This problem occurs in a custom-written test program based on the camera_sample example found in the /usr/src/jetson_multimedia_api/samples/unittest_samples/camera_unit_sample directory. The issue seems to be related to the device tree node order and how the devices are initialized.

Possible Causes

1. Device Initialization Order: The order in which the devices are initialized may affect the system’s ability to allocate resources or set up necessary configurations for each device.

2. Resource Allocation: There might be a limitation in how system resources are allocated to multiple camera devices, causing conflicts when opened in a specific order.

3. Driver Implementation: The camera driver or the V4L2 (Video4Linux2) implementation might have a bug or limitation that causes this order-dependent behavior.

4. ARGUS Framework Issues: The error message mentions ARGUS, which is NVIDIA’s camera API. There could be an issue with how ARGUS handles multiple camera contexts.

5. Device Tree Configuration: The device tree configuration might not be optimal for handling multiple camera devices simultaneously, leading to conflicts when opened in a certain order.

6. Thread Synchronization: If the custom code uses threading to open multiple devices, there might be synchronization issues affecting the device initialization process.

Troubleshooting Steps, Solutions & Fixes

1. Use ARGUS API:
   As suggested in the forum, try using the ARGUS stack instead of the V4L2 API directly. ARGUS is designed to handle multiple cameras more efficiently on NVIDIA platforms.

   Sample code locations:

   • /usr/src/jetson_multimedia_api/samples/09_argus_camera_jpeg/
   • /usr/src/jetson_multimedia_api/samples/10_argus_camera_recording/
   • /usr/src/jetson_multimedia_api/argus/

   Test these samples to see if the issue persists with the ARGUS API.

2. Implement Device Opening Delay:
   Add a small delay (e.g., 100-200ms) between opening each device. This may allow enough time for each device to initialize properly before the next one is opened.

   for (int i = 4; i >= 1; i--) {
       openDevice(i);
       std::this_thread::sleep_for(std::chrono::milliseconds(200));
   }
   

3. Check System Resources:
   Monitor system resources (CPU, memory, I/O) while opening the devices. Use tools like top, htop, or nvidia-smi to ensure there are sufficient resources available.

   watch -n 1 nvidia-smi
   

4. Verify Device Tree:
   Review and possibly modify the device tree to ensure it’s configured correctly for multiple camera usage.

   dtc -I dtb -O dts -o extracted_dt.dts /boot/dtb/kernel_tegra234-p3767-0000.dtb
   

   Analyze the extracted DTS file for camera-related nodes and their ordering.

5. Update Jetson SDK:
   Ensure you’re using the latest Jetson SDK (JetPack) version, as it may contain fixes for multi-camera issues.

   sudo apt update
   sudo apt upgrade
   

6. Check for Conflicting Camera Settings:
   Ensure that the camera settings (resolution, frame rate, format) are compatible across all devices and don’t conflict with system limitations.

7. Implement Error Handling and Retries:
   Modify your code to include robust error handling and implement a retry mechanism if a device fails to open.

   const int MAX_RETRIES = 3;
   for (int i = 4; i >= 1; i--) {
       int retries = 0;
       while (retries < MAX_RETRIES) {
           if (openDevice(i)) {
               break;
           }
           retries++;
           std::this_thread::sleep_for(std::chrono::seconds(1));
       }
       if (retries == MAX_RETRIES) {
           std::cerr << "Failed to open device " << i << " after " << MAX_RETRIES << " attempts." << std::endl;
       }
   }
   

8. Consult NVIDIA Developer Forums:
   If the issue persists, consider posting a detailed description of your problem, including your code and system configuration, on the NVIDIA Developer Forums for Jetson products.

9. Analyze ARGUS Logs:
   Enable verbose logging for the ARGUS framework to get more detailed error information.

   export ARGUS_LOG_LEVEL=LOG_DEBUG
   

   Run your application and analyze the logs for any specific errors or warnings related to device initialization.

10. Consider Hardware Inspection:
    If all software-based solutions fail, consider inspecting the hardware connections and camera modules for any physical issues or loose connections.