Ovis-VL-Embedding-2B

Ovis-VL-Embedding-2B is a compact vision-language embedding model for text, images, visual documents, video, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling cross-modal retrieval with a single 2B-scale encoder.

The model is initialized from Qwen3.5-2B. It retains the native text and vision encoders together with the shared multimodal language backbone, removes the language-modeling head, and directly uses the final-layer hidden state at the last non-padding token as the retrieval embedding. No modality-specific projection head is added.

Ovis-VL-Embedding-2B is designed for efficient multimodal search, multimodal RAG, image and visual-document retrieval, video search, recommendation, and nearest-neighbor matching under constrained serving budgets.

Model highlights

• Compact vision-language encoder: Provides strong text, image, document, and video retrieval in a deployable 2B-scale model.
• Unified multimodal retrieval: Queries and candidates may contain text, visual inputs, or interleaved text-image and text-video combinations.
• One shared embedding interface: All supported inputs are represented through last-token pooling and compared in the same cosine-similarity space.
• Dynamic-resolution visual processing: The native vision encoder handles images and sampled video frames at flexible resolutions while preserving spatial and temporal positions.
• Difficulty-aware contrastive learning: Focal embedding loss emphasizes unresolved queries with competitive negatives.
• Fine-grained embedding distillation: Teacher similarity distributions preserve relative relevance across both positive and negative candidates.
• Homogeneous-source finetuning: Task-consistent micro-batches provide informative in-batch negatives and reduce shortcuts based on modality or data format.
• Strong accuracy-efficiency trade-off: Achieves the best overall MMEB-v2 score among the compared models in its scale group while retaining a native embedding width of 2048.

Architecture

Figure 2 from the technical report. Ovis-VL-Embedding-2B encodes text, images, visual documents, and sampled video frames as one interleaved sequence. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.

The Qwen3.5-2B backbone contains 24 language layers with hidden size 2048. Its hybrid stack repeats three Gated DeltaNet layers followed by one gated full-attention layer, combining efficient long-context processing with periodic global token interaction. Multimodal positional encoding preserves temporal and two-dimensional spatial coordinates for visual tokens.

Training

Training follows three stages:

1. Multimodal contrastive pretraining. Large-scale, multi-task data establish broad alignment across text, image, document, and video inputs. The objective combines difficulty-aware focal contrastive learning with similarity-distribution distillation.
2. Full-parameter homogeneous finetuning. High-quality downstream data refine fine-grained discrimination. Each micro-batch is drawn from one dataset so that gathered candidates form task-consistent negatives.
3. Annealing Embedding Distillation. Teacher-correct examples are retained, unresolved student examples are emphasized, and confidence-adaptive forward-KL supervision transfers complementary expert capabilities.

Retrieval interface

Ovis-VL-Embedding-2B is a bi-encoder, not a cross-encoder:

1. Pair the query with the task instruction and format it with the native processor and chat template.
2. Encode queries and candidates independently.
3. Extract the final-layer hidden state at the last non-padding token.
4. L2-normalize the 2048-dimensional query and candidate embeddings.
5. Rank candidates by cosine similarity, equivalently the dot product of the normalized vectors.

No answer generation or query-candidate cross-attention is used during retrieval. Classification labels, passages, images, documents, videos, and interleaved multimodal items are all treated as candidates in the same embedding space.

Performance

MMEB-v2

MMEB-v2 evaluates vision-language embeddings over 78 datasets spanning image, video, and visual-document tasks. Ovis-VL-Embedding-2B achieves 77.46 overall, outperforming the strongest compared baseline by 2.04 points.

Group	Ovis-VL-Embedding-2B	Best compared baseline	Result
Image	80.62	77.41	+3.21
Video	67.12	68.84	-1.72
Visual document	80.47	79.86	+0.61
All 78 datasets	77.46	75.42	+2.04

The model ranks first on all four image sub-tasks, video classification, video moment retrieval, the visual-document aggregate, ViDoRe-V1, and out-of-distribution visual-document retrieval. Its strongest gains come from image understanding and document retrieval, while its compact scale retains competitive temporal-video performance.

Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 78 MMEB-v2 datasets.

Embedding dimensions

The native output width is 2048, inherited directly from the Qwen3.5-2B backbone because no embedding projection head is added. Queries and candidates must use the same preprocessing, pooling rule, dimensionality, and L2 normalization.

Intended use

The model is intended for embedding extraction and retrieval over supported unimodal or interleaved multimodal content, including:

• semantic text and cross-modal search;
• text-to-image, image-to-text, and image-to-image retrieval;
• multimodal RAG indexing and retrieval;
• visual-document and page retrieval;
• text-to-video and video retrieval;
• recommendation and nearest-neighbor matching.

Limitations

• This checkpoint does not natively support audio input. Use Ovis-Embedding-Omni-3B for audio and general omni-modal retrieval.
• This checkpoint produces retrieval embeddings; it is not intended as a text or image generation model.
• Retrieval quality depends on task-appropriate query instructions and the native preprocessing and chat template.
• Performance varies by task. Video question answering, general video retrieval, and ViDoRe-V2 remain below the strongest specialist baselines in the reported comparison.
• Benchmark scores may not directly predict performance on a new domain. Evaluate with representative queries, candidates, and retrieval metrics before deployment.

License

This model is released under the Apache 2.0 license.

Resources

• Ovis-VL-Embedding GitHub repository
• Ovis-Omni-Embedding GitHub repository

This repository contains the weights for Ovis-VL-Embedding-2B.