omura-embed-video

Finetuned projection heads (text encoder + projections + temperature) for InternVideo2-Stage2_6B-224p-f4 (with BEATs audio encoder), adapted by Omura for text→video retrieval and in-video temporal localization ("search inside a video") over the Walrus protocol's video corpus.

This repo contains only the finetuned delta (best_heads.pt, ~1.4 GB) — the base 6B backbone is not re-uploaded here. You need the original InternVideo2-6B checkpoint from OpenGVLab to use this.

What was changed

Symmetric InfoNCE finetuning on MSR-VTT (train_9k) of:

• the text encoder
• text/video projection layers
• the learned temperature

The video backbone itself is unchanged from the base InternVideo2-6B checkpoint.

Results (MSR-VTT 1K-A, text→video)

Scoring	R@1	R@5	R@10
baseline (zero-shot)	47.0	70.8	79.6
finetuned ITC	49.3	75.6	85.0
finetuned DSL	49.6	75.4	85.3

Exceeds the 85% R@10 target used for this model's evaluation. Full reproduction instructions: see BENCHMARK_REPRODUCTION.md in the omura-backend repo.

Also evaluated zero-shot (no finetuning applied to this task) for temporal localization on Charades-STA (moment retrieval): R@1 IoU@0.3 = 60.86%, R@1 IoU@0.5 = 31.24%, mIoU = 0.3585 — confirming the localization signal transfers even without task-specific tuning.

Usage

import torch, sys
sys.path.insert(0, "path/to/InternVideo2/multi_modality")  # OpenGVLab/InternVideo2 repo
# load the base 6B checkpoint per OpenGVLab's instructions, then:
heads = torch.load("best_heads.pt", map_location="cpu")
# apply heads' state dict to the corresponding text encoder / projection / temperature
# modules of the loaded base model. See iv2_finetuned.py in the omura-backend repo
# (benchmarks/eval/internvideo2/scripts/) for the exact loading code used to produce
# these results.

License

Apache 2.0 for this finetuned delta. The base InternVideo2-6B checkpoint is subject to OpenGVLab's own license terms.

Attribution

Base model: OpenGVLab/InternVideo2-Stage2_6B-224p-f4 (InternVideo2 paper). All credit for the video/audio backbone architecture and pretraining belongs to OpenGVLab. This repo is the finetuned head delta only.