mmarco-mMiniLMv2-en — English-pruned ONNX reranker for Statelet

English-pruned ONNX build of the cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 cross-encoder, packaged for the reranking stage in Statelet.

This is an English-only derivative. Do not use it for multilingual reranking.

What was changed

The multilingual checkpoint carries a 250,002-row XLM-R word-embedding table (roberta.embeddings.word_embeddings.weight, [250002, 384]) — a Gather that dynamic INT8 quantization never touches. That table is pruned to its ~120k Latin / shared / special rows, and the kept rows are stored INT8 on top of the already-matmul-INT8 graph.

Because a cross-encoder emits a relevance score rather than a vector, the export is verified on the relevant-minus-irrelevant score gap (fp32 vs pruned-INT8) rather than on vector cosine.

Files

file	notes
model.onnx	default CPU graph
model_fp16.onnx	fp16, dynamic shapes
model_gpu_b16_l64_fp16.onnx	fp16, static, batch 16 × seq 64
model_gpu_b16_l96_fp16.onnx	fp16, static, batch 16 × seq 96
model_gpu_b16_l128_fp16.onnx	fp16, static, batch 16 × seq 128
tokenizer.json	pruned XLM-R Unigram tokenizer — must be used with these graphs

The tokenizer is remapped alongside the table. Pairing these graphs with the upstream tokenizer produces garbage.

Use in Statelet

STATELET_MODEL_ROOT=/absolute/path/to/models

with this repository at $STATELET_MODEL_ROOT/mmarco-mMiniLMv2-en, or set STATELET_RERANKER_MODEL to point straight at it.

License

Apache-2.0, inherited from the upstream model. Training data is mMARCO, the machine-translated MS MARCO derivative — review MS MARCO's own terms if your use is commercial.