📚 Collection | 📝 Blogpost | 🛠️ Training and Evaluation

mSlideOn

Multilingual Sparse Retrieval Model by LightOn

SlideOn | mSlideOn | SlideOn-SAE | mSlideOn-SAE

🎯 TL;DR: A 339M-parameter multilingual sparse retrieval model built from mLateOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. mSlideOn improves on mDenseOn by 4.76 points on MIRACL and 10.19 points on MLDR, and reaches 74.78 on MTEB Code, surpassing both mDenseOn and mLateOn. It brings multilingual, long-context, and code retrieval to sparse representations compatible with inverted-index search, with Matryoshka training for adjustable query and document sparsity.

About the SlideOn / mSlideOn Family

With DenseOn and LateOn, then mDenseOn and mLateOn, we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

SlideOn starts from DenseOn for English retrieval. mSlideOn starts from mLateOn for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our blog post for the full method, results, and ablations.

mSlideOn

mSlideOn encodes queries and documents into 32,768-dimensional nonnegative sparse vectors. Its sparse encoder lifts mLateOn's normalized 128-dimensional token representations to 768 dimensions and applies a residual MLP before selecting the top-32 latent activations and applying ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the dot product between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

mSlideOn notably:

• Reaches 56.00 on BEIR-15, the highest score among the multilingual sparse models in our comparison.
• Reaches 63.21 on MIRACL target languages and 62.78 across all 18 languages, improving on mDenseOn by 4.76 points on the full benchmark.
• Reaches 75.79 on MLDR target languages and 61.68 across all 13 languages, outperforming every dense baseline in our comparison.
• Reaches 74.78 on MTEB Code, ahead of mDenseOn (71.53) and mLateOn (73.48).
• Supports inputs of up to 8,192 tokens and independently adjustable query and document sparsity.

Results

Headline Results (nDCG@10)

Scores are average nDCG@10 on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use unpruned query and document vectors. Dense and late-interaction comparisons follow our mDenseOn and mLateOn blog post.

MIRACL tgt covers Arabic, English, French, German, and Spanish; MLDR tgt covers Arabic, English, French, German, Italian, Portuguese, and Spanish. These are benchmark languages included in retrieval training. Full MIRACL covers 18 languages, full MLDR covers 13, and MTEB Code averages 12 retrieval tasks.

We evaluate MIRACL using the dev split of mteb/miracl-hard-negatives and MLDR using the test split of mteb/MultiLongDocRetrieval.

Model	Params.	BEIR-15 ↑	MIRACL tgt ↑	MIRACL ↑	MLDR tgt ↑	MLDR ↑	MTEB Code ↑	FLOPS ↓
Sparse
mSlideOn	339M	56.00	63.21	62.78	75.79	61.68	74.78	44.88
BGE-M3	568M	40.20	46.55	55.48	71.82	62.13	42.03	1.69
gte-multilingual-base	305M	40.27	48.74	57.49	80.19	70.75	51.09	1.70
MILCO-300M	330M	53.33	64.11	70.05	62.74	51.78	41.25	17.43
MILCO-650M	593M	52.12	64.14	70.02	68.63	58.30	33.06	13.96
OpenSearch multilingual	168M	48.26	57.87	59.41	36.47	27.52	41.28	1.77
Dense
mDenseOn	307M	56.70	59.61	58.02	65.62	51.50	71.53	—
pplx-embed-v1-0.6b	596M	56.70	63.18	68.20	57.08	43.98	75.18	—
jina-v5-text-small	677M	56.70	61.03	66.56	53.00	43.83	73.01	—
jina-v5-text-nano	239M	56.08	60.86	65.84	56.71	47.20	70.75	—
arctic-embed-l-v2	568M	55.55	60.78	66.53	57.01	48.03	53.17	—
Qwen3-Embedding-0.6B	596M	55.33	57.64	60.62	59.11	50.06	73.75	—
harrier-oss-v1-0.6b	596M	55.04	57.41	63.68	53.84	44.23	70.97	—
harrier-oss-v1-270m	268M	50.82	48.95	57.11	52.31	42.63	63.79	—
embeddinggemma-300m	308M	53.69	58.72	64.58	49.11	40.89	68.81	—
gte-multilingual-base	305M	51.08	57.39	64.14	66.41	56.65	57.46	—
voyage-4-nano	346M	49.96	49.29	58.65	62.10	51.97	76.43	—
BGE-M3	568M	48.78	62.22	69.62	61.16	52.47	51.49	—
granite-311m-r2	312M	49.24	53.38	59.74	50.53	41.86	63.50	—
Late interaction
mLateOn	312M	57.56	65.61	67.04	87.69	77.92	73.48	—
pplx-embed-v1-late-0.6b	596M	56.11	63.67	69.55	68.69	55.51	63.29	—
LFM2.5-ColBERT-350M	353M	54.50	57.40	42.45	78.39	61.05	51.62	—
jina-colbert-v2	559M	52.96	62.19	65.65	13.97	11.54	49.88	—
GTE-ModernColBERT	149M	54.23	43.96	36.35	67.68	45.95	54.37	—
ColBERT-Zero	149M	55.82	43.89	39.36	71.32	49.50	53.59	—

Retrieval results on BEIR-15, MIRACL, MLDR, and MTEB Code (↑ higher is better), with retrieval FLOPS (↓ lower is better). Blue denotes our models; bold values mark the best result within each family for each metric. OpenSearch multilingual corresponds to opensearch-neural-sparse-encoding-multilingual-v1.

Retrieval FLOPS estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Measurements use the fixed sample of 6,000 documents and 1,271 queries from SciFact, NFCorpus, and FiQA.

Key Findings

Multilingual and long-document retrieval. mSlideOn improves over mDenseOn on full MIRACL and MLDR by 4.76 and 10.19 points, respectively, while remaining behind mLateOn. The per-language results show gains beyond the languages included in retrieval training.

Code retrieval. mSlideOn outperforms every sparse and late-interaction baseline in this comparison on MTEB Code.

English retrieval. mSlideOn reaches 56.00 on BEIR-15, compared with 56.70 for mDenseOn and 57.56 for mLateOn. The gains on multilingual, long-document, and code tasks therefore come with a modest reduction in English benchmark performance.

Per-dataset and per-language differences in nDCG@10 points for mSlideOn versus mDenseOn. Teal bars indicate gains and red bars indicate declines relative to mDenseOn. Blue labels mark languages included in retrieval training.

Related Checkpoints

Model	Description
mSlideOn (this model)	Multilingual sparse retrieval model for natural language, long documents, and code
mSlideOn-SAE	mLateOn-based SAE initialization before retrieval fine-tuning
SlideOn	English sparse retrieval model, after SAE pretraining and retrieval fine-tuning
SlideOn-SAE	DenseOn-based SAE initialization before retrieval fine-tuning
mLateOn	Multilingual late-interaction retriever used to initialize mSlideOn's backbone

The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning.

Model Details

Model Description

Property	Value
Model type	Learned sparse retriever; Sentence Transformers SparseEncoder
Backbone	mLateOn / mmBERT-base
Total parameters	339M
Maximum sequence length	8,192 tokens, including special tokens and markers
Output dimensionality	32,768 learned latent features
Token representation	128 dimensions, L2-normalized before sparse encoding
Sparse encoder	Lift to 768 dimensions, residual MLP, and latent projection
Token sparsity	Top-32 latent selection followed by ReLU
Sequence pooling	Maximum over token activations
Similarity function	Dot product
Prompts	[Q] marker for queries; [D] marker for documents, inserted automatically after the beginning-of-sequence token. No query: / document: text prefixes.
Retrieval training languages	English, French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, Arabic, plus code

Usage

Direct Usage (Sentence Transformers)

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"

Load the model and encode multilingual queries and documents:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/mSlideOn", trust_remote_code=True)

queries = [
    "Qu'est-ce qui provoque une éclipse solaire ?",
    "Was verursacht eine Sonnenfinsternis?",
]
documents = [
    "A solar eclipse occurs when the Moon passes between the Sun and Earth.",
    "Um eclipse solar ocorre quando a Lua passa entre o Sol e a Terra.",
    "Wind turbines convert the kinetic energy of wind into electricity.",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([2, 32768]) torch.Size([3, 32768])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

encode_query() inserts the [Q] query marker, and encode_document() inserts the [D] document marker, immediately after the beginning-of-sequence token. Pass raw text; the model adds these special tokens automatically and does not use query: or document: text prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.

Choosing Query and Document Budgets

Use top_k independently on each encoding call to cap the number of nonzero features after pooling. This is separate from the per-token top-32 operation and does not change the vector dimensionality. For example, to use q-128 / d-1024:

query_embeddings = model.encode_query(queries, top_k=128)
document_embeddings = model.encode_document(documents, top_k=1024)

Omitting top_k, or setting it to 0, uses unpruned vectors, matching the headline results.

Dimensions represent learned latents, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; SparseEncoder.decode() cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.

Citation

If you find this work useful for your research, please consider citing us:

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}