VidVec: Unlocking Video MLLM Embeddings for Video–Text Retrieval

Issar Tzachor · Dvir Samuel · Rami Ben-Ari
OriginAI

NeurIPS 2026

TL;DR: VidVec turns pretrained video MLLMs into video–text embedding models. Through a lightweight text-only optimization, it achieves state-of-the-art results on video–text retrieval. More broadly, it suggests that optimizing on text alone can be an efficient and effective alternative to large-scale visual supervision. VidVec-7B is the released optimized embedder.

Highlights

State-of-the-art video-text retrieval: Achieves leading performance across widely used video-text retrieval benchmarks, outperforming existing MLLM-based embedders and dedicated Video Foundation Models.
True zero-shot evaluation: Neither VidVec nor its backbone was exposed to the benchmarks' training videos, following the zero-shot protocol of InternVideo2 and VideoPrism.
Turning generative MLLMs into powerful embedders: Transforms pretrained generative MLLMs into effective video-text embedding models, unlocking their inherent retrieval capabilities.
In-context text-only optimization: Enables video-text retrieval by aligning detailed video descriptions with short summaries, outperforming its counterpart paired video-text fine-tuning.
Lightweight adaptation: Requires only ~60K text pairs and less than 30 minutes of LoRA training on 4 GPUs.

This checkpoint is VidVec-O, the output of the in-context optimization (panel c), with the LoRA adapter merged into the weights. It embeds from the final layer.

Results

VidVec-O vs. MLLM embedders

Embeddings only, no reranking. The MLLM embedders (7B) are trained on 10–100× more vision–text data; VidVec-O uses about 60K text-only pairs.

Text-to-video

Video-to-text

Two-stage VidVec vs. Video Foundation Models

Recall@1. VidVec is VidVec-O with the top 5 text-to-video candidates reranked by the calibrated MLLM head.

Method	Video–text pairs	MSR-VTT T2V	MSR-VTT V2T	MSVD T2V	MSVD V2T	VATEX T2V	VATEX V2T	DiDeMo T2V	DiDeMo V2T
InternVideo-L	12M	40.7	39.6	43.4	67.6	49.5	69.5	31.5	33.5
UMT-L	10M	42.6	38.6	49.9	75.4	–	–	48.6	49.9
LanguageBind	10M	44.8	40.9	53.9	72.0	–	–	39.9	–
VideoPrism-g	600M	52.7	51.7	–	–	62.5	77.1	–	–
PE-Core-G	22M	51.2	49.9	59.7	85.4	–	–	–	–
InternVideo2-6B	100M	55.9	53.7	59.3	83.1	71.5	85.3	57.9	57.1
VidVec	0 (text only)	56.4	54.9	61.6	85.7	70.5	89.6	62.3	56.5

Quickstart

Requirements

One CUDA GPU with bf16 support and at least 32 GB of memory.
transformers==4.46.3. The model's remote code does not run on newer releases.
The ffmpeg binary on PATH; .mp4 decoding shells out to it.

pip install transformers==4.46.3 torch accelerate decord ffmpeg-python opencv-python imageio
pip install flash-attn==2.7.4.post1 --no-build-isolation   # optional, faster attention

Usage

The inference code ships with the model, so transformers is all you need.

from transformers import AutoModel

model = AutoModel.from_pretrained("iyttor/VidVec-7B", trust_remote_code=True, device_map="cuda:0")

# one score
score = model.similarity("video.mp4", "a man is riding a horse on the beach")

# embeddings (L2-normalised, 3584-d); similarity is a dot product
v = model.encode_video("video.mp4")
t = model.encode_text(["a man is riding a horse", "a cat sleeps on a sofa"])
scores = t @ v

# rank several captions against one video by embedding similarity
for hit in model.rank("video.mp4", ["a man is riding a horse", "a cat sleeps on a sofa"]):
    print(hit["rank"], hit["similarity"], hit["text"])

The model loads in bf16 and uses FlashAttention-2 when flash-attn is installed.

Frame sampling and prompts

Every method accepts the decoding and prompt settings as keyword arguments. The values below are the defaults, used for every number in the paper:

v = model.encode_video(
    "video.mp4",
    fps=2,  # frames sampled per second
    max_frames=180,  # cap on frames per video
    video_prompt='Summarize above video in one word: ',
)
t = model.encode_text(
    "a man is riding a horse",
    text_prompt='Summarize above sentence in one word: ',
)

# similarity() and rank() take the same keywords and route each to the right encoder
score = model.similarity("video.mp4", "a man is riding a horse", fps=1, max_frames=64)

Fewer frames lower memory use and latency. Changing a prompt changes the embedding space: encode videos and captions with matching settings, and expect scores to differ from the paper's.

How the embedding is computed

A video or caption is wrapped in a "summarize in one word" prompt followed by a <emb> token. The embedding is the last-layer hidden state at the position before <emb>, L2-normalised.

Evaluation and training

Benchmark evaluation (MSR-VTT, MSVD, VATEX, DiDeMo) and our in-context text-only training recipe are at https://github.com/iyttor/VidVec/.

Acknowledgements

VidVec-7B is fine-tuned from DAMO-NLP-SG/VideoLLaMA3-7B. The code builds on VideoLLaMA3 and LamRA. Training captions come from VideoUFO.

Citation

@inproceedings{tzachor2026vidvec,
  title     = {VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval},
  author    = {Tzachor, Issar and Samuel, Dvir and Ben-Ari, Rami},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}