nvidia/InternVideo2-CLIP-anomaly-detection

Model Overview

Description

InternVideo2-CLIP encodes video clips and text descriptions into a shared embedding space for text-to-video retrieval and zero-shot action classification. This model card describes an NVIDIA Train, Adapt, Optimize (TAO) video_clip fine-tune of the OpenGVLab InternVideo2-CLIP-L14 model with a MobileCLIP-B text encoder. Fine-tuning uses vision Low-Rank Adaptation (LoRA) and preservation regularization on the train subset of NVIDIA PhysicalAI-Event-Videos. Retrieval evaluation uses the dataset's test subset; Kinetics-400 is reported separately as a general-domain benchmark. This model has been post-trained on the InternVideo2 CLIP-L14 model, and use of the InternVideo2 CLIP-L14 model is governed by Apache 2.0.

The model is released for non-commercial research use only. The output is an embedding, rather than a generated caption, bounding box, identity, or calibrated event probability. This model is for research and development only.

License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Internal Scientific Research and Development Model License. Additional Information: Apache 2.0 for the InternVideo2 CLIP-L14 base model, Apple Machine Learning Research Model License for the MobileCLIP-B-LT text encoder, and MIT for the CLIP tokenizer files.

Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.

Deployment Geography

Global

Use Case

Intended users are developers evaluating video retrieval and event-search systems for physical-world video collections, including warehouse, retail, traffic, and general activity footage. A text query is compared with video embeddings to rank candidate clips. Zero-shot classification compares video embeddings with prompted action-class descriptions. The model is not for use in commercial products or services.

The model is not validated as an autonomous safety decision system. Similarity scores require application-specific evaluation and should not be interpreted as proof that an event occurred.

Release Date

Hugging Face 11/03/2026 via https://huggingface.co/nvidia/InternVideo2-CLIP-anomaly-detection. This model is released on Hugging Face only; it is not published on NGC.

References(s)

• NVIDIA PhysicalAI-Event-Videos: public reference for the training and evaluation subsets and the annotation workflow.
• OpenGVLab InternVideo: upstream model family and implementation.
• InternVideo2 distillation assets: upstream vision and alignment weights.
• MobileCLIP: source of the MobileCLIP-B-LT text encoder, licensed under the Apple Machine Learning Research Model License (research-only).
• NVIDIA TAO InternVideo2-CLIP (video_clip) overview: model description and data input format for the TAO pipeline.
• NVIDIA TAO InternVideo2-CLIP training, evaluation, inference, and export guide: experiment specification, train, evaluate, inference, and export subtasks, and ONNX/TensorRT usage notes.
• Using InternVideo2-CLIP embeddings: loading the exported ONNX model and using the embeddings for text-to-video retrieval, zero-shot classification, and anomaly detection.
• InternVideo2-CLIP with TAO Deploy: TensorRT engine generation and TensorRT-side evaluation and inference.
• NVIDIA TAO PyTorch: TAO model implementation.
• Kinetics dataset: action-recognition evaluation data.

Model Architecture

Architecture Type: Transformer.
Network Architecture: InternVideo2-CLIP-L14 vision encoder with a frozen, unmodified Apple MobileCLIP-B-LT text encoder and a shared 512-dimensional embedding space.
Base Model: OpenGVLab InternVideo2-CLIP-L14.
Number of model parameters: Approximately 3.7 × 10^8 (373,342,467, of which 589,824 are LoRA parameters); excludes the preservation teacher and optimizer state.

The vision transformer has 24 blocks, 14-pixel spatial patches, a hidden width of 1,024, and 16 attention heads. A visual projection and alignment layer map video features to the text embedding space. The text tower remains frozen during this fine-tune. LoRA targets the qkv and proj modules in the last 12 vision blocks, with rank 8, alpha 16, and dropout 0.05.

The training recipe uses video-text contrastive loss, preservation regularization, bfloat16 (BF16) precision, and AdamW (learning rate 1e-3, cosine schedule, 100 warmup steps) with an effective batch of 64. Training started from the base weights, and the released checkpoint was selected on the validation set described below. The fine-tune was run with the TAO video_clip train subtask; the experiment-specification format and these recipe fields are documented in the TAO training guide. These training details do not specify deployment hardware requirements.

Input(s)

Input Type(s): Video and text; each encoder can be used independently.
Input Format(s): RGB video frames and UTF-8 text strings.
Input Parameters: Video is a temporal sequence of two-dimensional RGB frames; text is a one-dimensional token sequence.

Other Properties Related to Input:

• The measured video protocol samples eight frames uniformly across each supplied clip and resizes each frame to 224 × 224 using bicubic interpolation. Published evaluation clips are already trimmed and receive no second temporal crop.
• RGB values are converted to floating point in [0, 1] and normalized using mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225).
• The TAO adapter receives video tensors shaped [batch, 8, 3, 224, 224]; the backbone internally permutes them to channel-first video layout.
• The CLIP tokenizer produces 77-token sequences, including special tokens, with padding/truncation to the context length. Text tensors have shape [batch, 77].
• Evaluation uses English descriptions. Multilingual quality and alternative frame counts or resolutions have not been established for this checkpoint.
• Audio is not an input to the evaluated model.

Output(s)

Output Type(s): Video embedding and text embedding.
Output Format(s): Floating-point vectors.
Output Parameters: One 512-dimensional vector per video clip or text input; batched shape [batch, 512].
Other Properties Related to Output: The TAO adapter supports L2-normalized outputs. Cosine similarity ranks video-text pairs; it is not a calibrated probability. The retrieval workflow additionally produces rankings and similarity scores.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. CPU throughput and cross-hardware equivalence were not measured.

Software Integration

Runtime Engine(s): PyTorch through NVIDIA TAO video_clip (accuracy evaluation); TensorRT 10.16 with a strongly typed FP16 ONNX export (performance measurement), deployable through TAO Deploy.

Supported Hardware Microarchitecture Compatibility: NVIDIA Ampere (A100) is verified for the reported accuracy evaluation. NVIDIA Hopper (H100), Ada Lovelace (L40S), and Blackwell (RTX PRO 6000 Server Edition, GB300) were measured for TensorRT performance.

Supported Operating System(s): Linux in the recorded evaluation environment.

Integration requires a checkpoint-compatible model configuration, the base assets, tokenizer, and the matching LoRA configuration. The training checkpoint also stores preservation-teacher state; loading or packaging must handle that state consistently. System-level testing with application-specific data is required before deployment.

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

Version	Configuration
InternVideo2-CLIP-L14 TAO fine-tune v1.0 (research release, non-commercial)	Vision LoRA on the last 12 blocks, frozen text encoder, preservation regularization

The release is published as the Hugging Face repository nvidia/InternVideo2-CLIP-anomaly-detection. The upstream base model below is an evaluation reference, not an additional fine-tuned release variant.

The model integrates into an AI system as an embedding encoder: the video tower embeds clips once into a searchable index, the text tower embeds user queries at search time, and the application ranks clips by cosine similarity. Both towers run through TAO video_clip inference or as exported ONNX models inside a retrieval or event-search service.

Training, Testing, and Evaluation Datasets

Dataset Overview

All training data comes from the train subset of NVIDIA PhysicalAI-Event-Videos. All retrieval testing and evaluation data comes from its test subset. Kinetics-400 is a separate general-domain action-classification benchmark.

The train subset used for this fine-tune contains 2,743 source videos, 6,059 chunks, and 85,621 caption entries (counts from the training manifest; caption entries are training pairs, not necessarily unique text strings). The test subset has 393 clips from 166 source videos and 3,040 event queries; its 3,000 person-attribute-search (PAS) queries are excluded from all domain results reported here. General-domain evaluation uses 19,877 successfully decoded Kinetics-400 validation clips.

Training Dataset

Dataset: NVIDIA PhysicalAI-Event-Videos, train subset.
Data Modality: Video and English text.
Video Training Data Size: Less than 10,000 hours of fine-tuning video.
Text Training Data Size: Less than a Billion Tokens (85,621 caption entries, each at most 77 tokens).
Data Collection Method: Hybrid: Synthetic, Manually-Collected
Labeling Method: Hybrid: Automated, Manually-Labeled (automated pseudo-labeling produces captions and queries, with human review and correction for a subset of annotations).

Training metadata expands the caption, anomaly, and action query fields into video-text pairs. The data includes descriptions and footage of human activities, anomalous events, and physical-world scenes, including synthetic and real people.

The training manifest has no media overlap with the test gallery.

Testing Dataset

All retrieval testing data comes from the test subset of NVIDIA PhysicalAI-Event-Videos.

Data Modality: Video and English text.
Data Collection Method: Hybrid: Synthetic, Manually-Collected
Labeling Method: Hybrid: Automated, Manually-Labeled (automated query and relevance annotation with human review and correction)
Properties: 393 trimmed video clips from 166 source videos and 3,040 English natural-language event queries (264 anomaly, 1,369 activity, 64 location, 1,343 precise-moment), each with one or more labeled relevant clips. Content is surveillance-style physical-world footage of everyday human activity and anomalous events, including fights, falls, and theft, in warehouse, retail, street, and indoor scenes. Queries are short English phrases or sentences; no other languages are included.

Evaluation Dataset

All numbers below were measured on the released checkpoint.

Data Collection Method by dataset:

• PhysicalAI-Event-Videos test subset: Hybrid: Synthetic, Manually-Collected
  
• Kinetics-400: Hybrid: Automated, Manually-Collected
  

Labeling Method by dataset:

• PhysicalAI-Event-Videos test subset: Hybrid: Automated, Manually-Labeled
  
• Kinetics-400: Manually-Labeled
  

PhysicalAI-Event-Videos test subset — event-only retrieval

All retrieval evaluation data comes from the test subset of NVIDIA PhysicalAI-Event-Videos: 393 clips and 3,040 English event queries, comprising 264 anomaly, 1,369 activity, 64 location, and 1,343 precise-moment queries. All rows use the released trimmed clips and eight uniformly sampled frames. IV2CLIP uses 224-pixel inputs, BF16, and batch size four on an A100. The zero-shot Qwen3-VL-Embedding baselines use their model-specific preprocessing and resolution, BF16, and video batch size one. Their text and video instructions are respectively Represent the user's input for video anomaly retrieval. and Represent this video clip for anomaly retrieval.

For each query, all 393 clips are ranked by cosine similarity. mAP averages per-query average precision across all labeled relevant clips. Hit@K is the fraction of queries with at least one relevant clip in the first K results; it is not multi-relevance Recall@K. All query records, including repeated texts, are scored. PAS results are not reported.

Model	mAP	Hit@1	Hit@5
Upstream IV2CLIP-L14 base	54.34%	50.69%	78.49%
This model (TAO fine-tune)	63.12%	59.11%	87.04%
Qwen3-VL-Embedding-2B (zero-shot)	59.67%	57.40%	84.67%
Qwen3-VL-Embedding-8B (zero-shot)	60.58%	58.03%	85.26%

Query category	Queries	mAP	Hit@1	Hit@5
Anomaly	264	67.77%	75.76%	95.08%
Activity	1,369	69.32%	73.70%	91.23%
Location	64	99.29%	100.00%	100.00%
Precise-moment search	1,343	54.16%	39.02%	80.57%

Native retrieval metrics were independently recomputed from the saved embeddings and agreed to within 1e-15.

Protocol limitation: The result is not an independent held-out generalization estimate, because checkpoint selection used an earlier relevance revision of the same gallery.

Kinetics-400 — zero-shot action classification

Kinetics-400 contains real-world human-action video with human action-class labels. All rows use eight sampled frames, the same 400 class names, the prompt A video of a person {label}, and cosine ranking. IV2CLIP uses 224 × 224 inputs; Qwen uses model-specific preprocessing and resolution, with text and video instructions Represent this action class for matching to a video. and Represent this video for action classification. Of 19,881 available validation clips, 19,877 decoded successfully; four were excluded. Top-K accuracy measures whether the ground-truth action class is among the K highest-scoring prompted classes. These are archived evaluations under the shared protocol, not a single identical model-input pipeline; pretraining overlap with Kinetics-400 cannot be ruled out for the zero-shot models.

Model	Top-1 accuracy	Top-5 accuracy
Upstream IV2CLIP-L14 base, recorded same-protocol reference	70.11%	89.60%
This model (TAO fine-tune)	69.19%	89.64%
Qwen3-VL-Embedding-2B (zero-shot)	67.62%	88.70%
Qwen3-VL-Embedding-8B (zero-shot)	69.57%	89.94%

The fine-tune improves the measured domain retrieval score (+8.78 percentage points mAP over the base) while reducing Kinetics-400 Top-1 accuracy by 0.92 percentage points versus the recorded base reference. The Kinetics evaluation used merged LoRA weights (24 modules) on the 19,877 decodable validation clips. It did not retrain the model.

Inference

Acceleration Engine: PyTorch (accuracy results); TensorRT 10.16 (performance results)
Test Hardware:

• NVIDIA Ampere A100
• NVIDIA Blackwell RTX PRO 6000 Server Edition
• NVIDIA Hopper H100 80GB HBM3
• NVIDIA Ada Lovelace L40S
• NVIDIA Blackwell GB300 (DGX Station)

Accuracy results above are from PyTorch; domain retrieval ran on one NVIDIA Ampere A100 GPU, and the distributed Kinetics-400 evaluation ran on multiple NVIDIA Ampere A100 80-GB GPUs, which describes the evaluation setup, not a minimum inference requirement. TensorRT accuracy has not been benchmarked on the evaluation sets.

TensorRT Performance

TensorRT performance was measured with trtexec (TensorRT 10.16) in the nvcr.io/nvidia/tao/tao-toolkit:7.0.1-pyt container. The engine was built with --stronglyTyped from a ModelOpt AutoCast FP16 ONNX export (opset 23, RMSNorm layers kept in FP32) with an identical architecture. Each inference takes an 8 × 3 × 224 × 224 video clip and a 77-token text input on one GPU. The engine is about 0.76 GB.

GPU	Throughput, batch 1 (clips/s)	GPU compute, batch 1 (ms)	Throughput, batch 8 (clips/s)	Throughput, batch 32 (clips/s)	GPU compute, batch 32 (ms)
NVIDIA Blackwell RTX PRO 6000 Server Edition	101.1	9.89	162.4	157.9	202.7
NVIDIA Hopper H100 80GB HBM3	152.9	6.54	249.2	257.9	124.1
NVIDIA Ada Lovelace L40S	79.2	12.63	85.9	83.5	383.1
NVIDIA Blackwell GB300 (DGX Station)	287.2	3.48	511.0	526.8	60.7

A clip is one video-text pair. GPU compute is the mean trtexec GPU compute time per inference call.

Note: The measured engine combines the vision and text towers in one graph, so every call runs both towers. Deployment can run faster with the towers separated: video clips are embedded once at indexing time with a vision-tower engine, and only the lightweight text tower runs for each query. Separate vision-tower and text-tower ONNX files are provided for this purpose. Separate-tower performance was not measured.

How to use this model

Use a TAO installation containing the video_clip InternVideo2-CLIP-L14 implementation, the matching base assets and tokenizer, and a complete experiment specification compatible with this checkpoint. The TAO InternVideo2-CLIP overview describes the model and data input format; the training, evaluation, inference, and export guide documents the experiment specification and every subtask below. Set model.type: internvideo2-clip-l14, model.num_frames: 8, and model.image_size: 224, and restore the vision-LoRA configuration described above. The Hugging Face repository provides the merged weights (model.safetensors), the TAO checkpoint with unmerged LoRA adapters (model.pth), the model configuration (model_config.yaml), the FP32 and FP16 ONNX towers with the tokenizer (onnx/), and a SHA-256 manifest.

Train (guide): fine-tune from the base weights with the recipe described under Model Architecture.

video_clip train -e /path/to/train.yaml

Evaluate (guide): compute retrieval metrics against a query/gallery set with this checkpoint selected in the evaluation section.

video_clip evaluate -e /path/to/evaluate.yaml

Inference (guide): use inference.mode: embeddings for embedding extraction or inference.mode: retrieval for ranked retrieval with the configured query and gallery inputs.

video_clip inference -e /path/to/inference.yaml

Export and deploy (export guide, TAO Deploy, using the embeddings): export the towers to ONNX, build a TensorRT engine and run TensorRT-side evaluation and inference with TAO Deploy, and consume the ONNX embeddings in a retrieval or event-search application. Export and deployment validation must use the same tokenizer, normalization, LoRA weights, and output dimensions as the PyTorch path.

video_clip export -e /path/to/export.yaml

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility. Developers should evaluate this model against their intended domain and address potential misuse before deployment.

Users must have the necessary rights and permissions for input images and video, including content depicting people or containing intellectual property. Embedding extraction does not anonymize source content. Retrieval errors can produce false alarms or missed events; inspect source clips and validate downstream decisions in context. Commercial deployment is not permitted under the Apple MobileCLIP license; it would require a separate Apple commercial license or a replacement text encoder.

Please report model quality, risk, security vulnerabilities, or NVIDIA AI concerns through NVIDIA's reporting channel.