Image vs video document input: large gap in text→moment retrieval quality (with measurements)
Hi, and thanks for releasing WeMM-Embedding-2B — we've been using it to build a text-to-video moment search tool (index a video in fixed-length windows, retrieve the matching time range from a text query). It works well, and multilingual queries hold up nicely (we verified Chinese / English / Japanese / Korean / Spanish / Portuguese all land on the same clip).
While optimizing our indexing pipeline we hit a result we did not expect, and we think it is worth documenting for other users.
What we compared
Our pipeline extracts keyframes per 8-second window and encodes each window as a document. We tried two encodings of the same visual content:
- A — video document: the window's keyframes muxed into a tiny mp4, then
{"video": path, "text": "Represent this video."} - B — image document: the window's first keyframe as a JPEG, then
{"image": path, "text": "Represent this image."}
Both at 360p. B is much cheaper because it skips an encode→decode round trip.
Result 1 — the two embeddings are far apart
Cosine similarity between the A-embedding and the B-embedding of the same content:
| value | |
|---|---|
| mean | 0.729 |
| min | 0.669 |
Result 2 — text retrieval is noticeably worse with image documents
Using Big Buck Bunny (public test stream, 635 s, 8-second windows) with three queries whose correct answers we had verified by hand:
| Query | A (video doc) | B (image doc) | Expected |
|---|---|---|---|
| 兔子从洞里钻出来伸懒腰 (rabbit comes out of burrow) | 00:48 ✅ | 00:48 ✅ | 00:48 |
| a butterfly flying near flowers | 01:12 ✅ | 00:16 ❌ | 01:12 |
| three small animals in a tree | 03:04 ✅ | 03:52 ❌ | ~02:24–03:04 |
A: 3/3. B: 1/3. We first suspected resolution (our thumbnails were 120px), but re-running with full 360p frames gave the same outcome.
Result 3 — the speed difference is large
Per window, same hardware (Apple M5 Max, MPS, float16):
| ms/window | |
|---|---|
| A — video document | ~900 |
| B — image document (360p) | ~127 |
So B is ~7x faster but retrieves worse. That trade-off is easy to stumble into while optimizing, which is why we think it deserves a line in the model card.
Why this surprised us
Our "video" documents are synthetic: 1–2 still keyframes stitched into a short mp4. They carry essentially the same visual information as the single image, and almost no real motion. We expected near-identical embeddings. The gap suggests the image and video paths are processed differently, and that text↔video alignment is stronger than text↔image for this checkpoint.
Questions
- Is this expected? Were text↔video and text↔image trained with different amounts of data or different objectives?
- Is there a recommended prompt or preprocessing for image documents that would close the gap? (We used
"Represent this image."from the model card.) - Would you consider noting in the model card which document modality is recommended for video-moment retrieval?
Environment
torch 2.13.0, transformers 5.2.0, sentence-transformers 6.0.0
Apple M5 Max, MPS, float16
batch_size=4 (we also measured 8/16/32 — all slower on MPS)
Happy to share the full test scripts or rerun anything on request.
中文版（Chinese version）
感谢开源 WeMM-Embedding-2B。我们用它做了一个「文字检索视频时刻」的工具（把视频按固定窗口切分建索引，用一句文字定位到对应时间段），效果不错，多语言查询也很稳（中/英/日/韩/西/葡实测都能命中同一片段）。
在优化索引流程时，我们遇到一个意外的结果，觉得值得记录下来供其他使用者参考。
对比的两种方式：同样的画面内容，分别用
- A：视频 document —— 把窗口内的关键帧封装成极短 mp4，
{"video": ..., "text": "Represent this video."} - B：图片 document —— 直接用该窗口的第一张关键帧，
{"image": ..., "text": "Represent this image."}
两者都是 360p。B 省掉了一次编码→解码往返，所以快很多。
结果 1：两种 embedding 相距较远 —— 同一内容的余弦相似度：均值 0.729，最低 0.669。
结果 2：图片 document 的文字检索明显更差 —— 用 Big Buck Bunny（公开测试流，635 秒，8 秒窗口）和三个人工确认过答案的查询：
| 查询 | A（视频） | B（图片） | 正确答案 |
|---|---|---|---|
| 兔子从洞里钻出来伸懒腰 | 00:48 ✅ | 00:48 ✅ | 00:48 |
| a butterfly flying near flowers | 01:12 ✅ | 00:16 ❌ | 01:12 |
| three small animals in a tree | 03:04 ✅ | 03:52 ❌ | ~02:24–03:04 |
A 是 3/3，B 是 1/3。我们最初怀疑是分辨率问题（缩略图只有 120px），换成完整 360p 帧后结论一致。
结果 3：速度差距很大 —— 同一硬件（Apple M5 Max，MPS，float16）下每窗耗时：视频 ~900ms，图片 ~127ms，图片快约 7 倍。
为什么意外：我们的「视频」其实是合成的——1~2 张静止关键帧拼成的短 mp4，和单张图片携带的视觉信息几乎相同、也几乎没有运动。我们本以为两者 embedding 会非常接近。这个差距说明 image 和 video 两条路径的处理方式不同，且这个 checkpoint 的 text↔video 对齐比 text↔image 更强。
想请教：
- 这是预期行为吗？text↔video 和 text↔image 的训练数据量或目标是否不同？
- 图片 document 是否有推荐的 prompt 或预处理能缩小这个差距？（我们用的是模型卡里的
"Represent this image."） - 是否考虑在模型卡里说明：做视频时刻检索时推荐用哪种 document 模态？
如果需要，可以提供完整测试脚本或按需重跑。