MetaCLIP-2 Worldwide-H — trunk-calibrated

Research artifact for "Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?"

This is facebook/metaclip-2-worldwide-huge-quickgelu with its front 4 text-encoder blocks replaced by trunk-calibrated weights. Everything else — the vision tower, the remaining 20 text blocks, and the projection head — is byte-identical to the frozen public model.

What this changes

Multilingual VL encoders trail English by 30+ pp on low-resource-language retrieval. The paper localises that gap: it is not a linear language direction at the output (erasing it with LEACE or INLP leaves retrieval within ~1.5 pp), but the pooled EOS hidden state's per-language trajectory drift with depth. Calibrating only the first 4 blocks — pulling each language's projection toward the parallel-content centroid — closes much of the gap without touching the frozen back-half.

Measured results

XM3600 (Crossmodal-3600), pinned 1,000-image subset, image-to-text R@1, bfloat16. frozen is the unmodified base model measured in the same harness.

Language	Tier	Frozen	This model	Δ
English (`en`)	HRL	69.4	71.6	+2.2
French (`fr`)	HRL	84.5	86.2	+1.7
German (`de`)	HRL	86.5	86.8	+0.3
Spanish (`es`)	HRL	78.0	77.6	-0.4
Chinese (`zh`)	HRL	74.6	77.5	+2.9
Korean (`ko`)	HRL	75.5	78.9	+3.4
Bengali (`bn`)	LRL	64.7	69.0	+4.3
Filipino (`fil`)	LRL	40.1	52.6	+12.5
Hindi (`hi`)	LRL	49.4	53.0	+3.6
Swahili (`sw`)	LRL	22.1	42.1	+20.0
Telugu (`te`)	LRL	44.3	51.9	+7.6
HRL-6 mean		78.08	79.77	+1.68
LRL-5 mean		44.12	53.72	+9.60

High-resource performance is preserved; the gain concentrates on low-resource languages.

Usage

import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("dnotitia/geometric-bottleneck-metaclip2", dtype=torch.bfloat16).cuda().eval()
processor = AutoProcessor.from_pretrained("dnotitia/geometric-bottleneck-metaclip2")

texts = ["a dog on a beach", "mbwa ufukweni"]          # English, Swahili
inputs = processor(text=texts, padding="max_length", truncation=True,
                   max_length=77, return_tensors="pt").to("cuda")
with torch.no_grad():
    z = torch.nn.functional.normalize(model.get_text_features(**inputs).float(), dim=-1)

Image encoding is unchanged from the base model — the vision tower was never trained.

Training


Calibrated blocks	front 4 of 24 text blocks
Text tower	causal, dynamic EOS (token id 2) pooling, max 77 tokens
Objective	InfoNCE + λ·(1 − cosine to parallel-content centroid)
λ_align	2.0
Learning rate	0.00012
Temperature	0.02
Steps	2000 of 2000 (warmup 200)
Weight decay	0.05
Languages	11: en, fr, de, es, zh, ko, bn, fil, hi, sw, te
Training data	CC12M caption subset, machine-translated into the 11-language pool
Precision	bfloat16

Limitations

Language-specific. Calibration is trained on these 11 languages. It does not transfer to unseen languages, and can regress on them.
Visual-vocabulary scoped. Gains are measured on image captions. Abstract prose benefits far less.
Does not reach parity. The LRL tier improves substantially but does not match HRL.
Single run. No seed variance is reported for this checkpoint.

License

These weights contain the base model's parameters, so the base model's license governs use.


Base model	`facebook/metaclip-2-worldwide-huge-quickgelu`
Pinned revision	`c139061af7b10fdb2e754b60d2b1182a3d5526c2`
License	cc-by-nc-4.0

Non-commercial. cc-by-nc-4.0 restricts this model to non-commercial use. That restriction flows from the base model and applies to these merged weights.

The calibrated front-4 blocks are released as a research artifact accompanying the paper.

Citation

@inproceedings{han2026geometric,
  title     = {Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?},
  author    = {Han, Donghoon and Moon, SungHyun and Zhakatayev, Aidyn and Cha, Junghun and Lee, SeungJae},
  booktitle = {Proceedings of EMNLP 2026},
  year      = {2026}
}

Code: https://github.com/dnotitia/geometric-bottleneck