jonghwi/ColBERT-XLMR-Mixed-Distill-enru-ru

Mixed-Distill enru-ru (English–Russian)

A cross-lingual ColBERT late-interaction retriever (XLM-RoBERTa-large backbone) for English ⇄ Russian web search. The model is distilled from a strong reranker and trained on code-switched (mixed-language) queries, following the Mixed-Distill recipe from the MiLQ paper.

The repo name encodes the training direction: enru-ru = code-switched en+ru queries → ru documents. It is designed to be robust when bilingual users issue mixed English+Russian queries against Russian-language documents.

What "Mixed-Distill" means

• Mixed — queries are code-switched (English tokens randomly mixed into Russian queries, MUSE-based, ~0.5 mixing ratio), so the model handles native, English, and mixed-language queries.
• Distill — trained with knowledge distillation (KL-divergence) from teacher relevance scores (mT5-XXL / monoT5 over mMARCO), 6-way passage scoring.

Intended use

• Queries: English, Russian, or code-switched English+Russian.
• Documents: Russian-language passages.
• Scoring: ColBERT late interaction (MaxSim over per-token embeddings).

Specs

Base model	xlm-roberta-large
Architecture	ColBERT (late interaction)
Projection dim	128
Similarity	cosine
Query max length	32
Doc max length	180
Training	KD (KLD), n-way 6, teacher: mT5-XXL/monoT5 on mMARCO

Usage

Load with the ColBERT library:

from colbert.modeling.checkpoint import Checkpoint
from colbert.infra import ColBERTConfig

ckpt = Checkpoint("<your-username>/ColBERT-XLMR-Mixed-Distill-enru-ru",
                  colbert_config=ColBERTConfig())
Q = ckpt.queryFromText(["mixed English+Russian query ..."])
D = ckpt.docFromText(["Russian document passage ..."])

Citation

@misc{kim2025milqbenchmarkingirmodels,
      title={MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries},
      author={Jonghwi Kim and Deokhyung Kang and Seonjeong Hwang and Yunsu Kim and Jungseul Ok and Gary Lee},
      year={2025},
      eprint={2505.16631},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2505.16631},
}