nvidia/RT-DETR-Hand-Detector-v1

RT-DETR Hand Detection v1.0 Overview

Description:

RT-DETR Hand Detection v1.0 adapts the Real-Time Detection Transformer (RT-DETR) architecture to detect and localize hands directly in RGB images. The model combines a convolutional backbone with an efficient hybrid encoder and a TransformerDecoder head to predict 2D hand bounding boxes and class confidence scores in a single end-to-end pass. Trained on a diverse combination of synthetic and real-world hand datasets, it provides high-throughput, accurate bounding box predictions that serve as tight regions-of-interest (ROIs) for downstream 3D hand pose estimation and gesture analysis models. RT-DETR Hand Detection v1.0 was developed for real-time human-computer interaction (HCI) and egocentric computer vision pipelines.

RT-DETR Hand Detection v1.0 was developed by NVIDIA as a part of RT-DETRv2-S.
This model is for demonstration purposes and not for production usage.

License/Terms of Use:

NVIDIA Software and Model Evaluation License

Deployment Geography:

Global

Use Case:

RT-DETR Hand Detection v1.0 could be used by researchers and engineers working on egocentric perception and human-robot interaction to detect and localize hands directly from single RGB images or video frames. The resulting 2D hand bounding boxes serve as a precise region-of-interest (ROI) crop that downstream models (e.g., EgoHand) use to predict 3D hand keypoints/mesh for the egocentric hand in the image. These upstream detections enable continuous hand tracking across human demonstration videos, providing clean visual inputs to bootstrap or supervise downstream robotics applications that rely on 3D hand pose estimation.

Reference(s):

SOMA: Unifying Parametric Human Body Models
Documentation
PyPI

Model Architecture:

Architecture Type: Transformer
Network Architecture: HGNetv2-S Backbone + Efficient Hybrid Encoder (with Discrete Sampling) + TransformerDecoder Head
This model was developed based on RT-DETRv2 (Real-Time Detection Transformer v2).
Number of model parameters: ~20M

Input:

Input Type(s): Image
Input Format(s): Red, Green, Blue (RGB)
Input Parameters: Two-Dimensional (2D)
Other Properties Related to Input: Input images are resized to the model's fixed training resolution (640×640) prior to inference; no strict minimum source resolution is enforced, though very low-resolution inputs may reduce detection accuracy.

Output:

Output Type(s): Other: Array
Output Format: Other: Floating Point
Output Parameters: One-Dimensional (1D): Per-detection [bounding box coordinates (x1, y1, x2, y2), class confidence score] Other Properties Related to Output: Reports bounding boxes for left hand and right hand (wearer-anatomical) as two separate classes, each with its own confidence score, directly from the detector's class head.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s): Not Applicable (N/A) Supported Hardware Microarchitecture Compatibility: NVIDIA Lovelace
Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

RT-DETR-v2-S

The model is exported to ONNX format for deployment and integrated as the first stage of an egocentric hand-perception pipeline: it runs inference on each input RGB frame to produce left/right hand bounding boxes with confidence scores, which are consumed directly by a downstream tracking module (temporal/identity association across frames) and by 3D hand pose/mesh estimation models that use the bounding boxes as their region-of-interest input.

Training, Testing, and Evaluation Datasets:

Training Dataset:

Data Modality: Image

Image Training Data Size: 1 Million to 1 Billion Images
Data Collection Method by dataset: Hybrid: Manually-Collected/Synthetic
Labeling Method by dataset: Hybrid: Manually-Labeled, Synthetic Properties (Quantity, Dataset Descriptions, Sensor(s)): The model is trained on a mix of synthetic and real-world hand motion image/video data, annotated with 2D bounding boxes for the left and right hands.

• In-house purchased real-world egocentric hand data: RGB image/video of hand motion captured with head-mounted egocentric cameras, sourced from 3rd-party data vendors. 20 hours of 1920p (~2M frames), 30 FPS egocentric video.
• Synthetic data generated via the in-house RigPlay/DigitalHuman rendering pipeline: photorealistic renders of hand motion with exact ground-truth 2D bounding boxes for the left and right hands.

Testing Dataset:

Data Collection Method by dataset: Manually-Collected Labeling Method by dataset: Hybrid: Manually-Collected/Synthetic Properties (Quantity, Dataset Descriptions, Sensor(s)): The testing dataset is a held-out 20% split (80% training / 20% testing) of the same mix of synthetic and real-world hand motion image/video data used for training, annotated with 2D bounding boxes for the left and right hands.

• In-house purchased real-world egocentric hand data: RGB image/video of hand motion captured with head-mounted egocentric cameras, sourced from 3rd-party data vendors. 5 hours of 1920p (~500K frames), 30 FPS egocentric video.
• Synthetic data generated via the in-house RigPlay/DigitalHuman rendering pipeline: photorealistic renders of hand motion with exact ground-truth 2D bounding boxes for the left and right hands. 5 hours of 1920p (~600K frames), 30 FPS video.

Evaluation Dataset:

Benchmark Score: Undisclosed

Data Collection Method by dataset: Manually-Collected
Labeling Method by dataset: Hybrid: Manually-Collected/Synthetic Properties (Quantity, Dataset Descriptions, Sensor(s)): * The evaluation dataset is a held-out 20% split (80% training / 20% testing/evaluation) of the same mix of synthetic and real-world hand motion image/video data used for training, annotated with 2D bounding boxes for the left and right hands.

• In-house purchased real-world egocentric hand data: RGB image/video of hand motion captured with head-mounted egocentric cameras, sourced from 3rd-party data vendors. 5 hours of 1920p (~500K frames), 30 FPS egocentric video.
• Synthetic data generated via the in-house RigPlay/DigitalHuman rendering pipeline: photorealistic renders of hand motion with exact ground-truth 2D bounding boxes for the left and right hands. 5 hours of 1920p (~600K frames), 30 FPS video.

Inference:

Engine: PyTorch Acceleration Engine: Other: None Test Hardware:

• NVIDIA Lovelace L40
• NVIDIA Lovelace L40s

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Generated by NVIDIA Model Card Generator Toolkit.