mlboydaisuke/all-MiniLM-L6-v2-ExecuTorch

all-MiniLM-L6-v2 — ExecuTorch

Sentence embeddings: text in, one vector out. For search, clustering and retrieval that never leaves the device.

• Source: sentence-transformers/all-MiniLM-L6-v2 — 22M parameters, 6 BERT layers, 384-dimensional output
• License: Apache-2.0
• Input: input_ids and attention_mask, both [1, 256] int64
• Output: [1, 384]

The pooling is in the graph, on purpose

sentence-transformers keeps the recipe per model in 1_Pooling/config.json and modules.json, and the three small models on this shelf do not agree:

	pooling	normalised
all-MiniLM-L6-v2	mean	yes
bge-small-en-v1.5	CLS	yes
paraphrase-multilingual-L12	mean	no

This one uses mean over the attention mask, then L2 normalise. Getting it wrong does not throw: mean-pooling BGE, or normalising the multilingual one, gives vectors that look fine and rank wrong. So it is baked in rather than left to the caller.

Verification (Mac arm64, 2026-08-23)

build	size	latency	worst cosine vs eager
XNNPACK fp32	90.4 MB	14.4 ms	1.000000
XNNPACK fp16	45.3 MB	26.3 ms	0.999999
Core ML fp32	45.4 MB	2.0 ms	0.999984

Eager fp32 on the same input is 8.5 ms. Cosine is against the model run in eager through its own documented pooling, over eight sentences including one in Japanese — not against random token ids, which would tell you nothing.

And that the vectors are useful, which agreement alone cannot show:

0.588 a paraphrase   vs -0.063 an unrelated sentence

Across languages — "機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the phone", with an unrelated sentence about the weather — this model scores -0.047 against 0.125 — it does not, and should not.

Not shipped

int8 converts and holds — worst cosine 0.998018 against eager over the same eight sentences — but it comes out at 58.6 MB against fp16's 45.3 MB, so nothing would pick it. The reason is arithmetic rather than a defect: dynamic int8 quantizes the linear weights and leaves the token embedding table alone, and that table is 46.9 MB of a 90.4 MB model. Quantizing every linear saves 31.8 MB; fp16 halves the table too. On a model whose weights are mostly a vocabulary, fp16 is the smaller build.

An earlier version of this card said int8 does not export, with an IndexError: tensors used as indices must be long, int, byte or bool tensors. That was true of the build at the time and the cause was misread. The error is not in the embedding lookup: XNNPACKQuantizer's transform_for_annotation lifts every scalar argument of add.Tensor/mul.Tensor into a buffer and writes it as torch.tensor(float(arg)) regardless of the node's dtype, so the arange(n) + 0 that builds a HuggingFace attention mask comes back float32 and can no longer index. It is one line in ExecuTorch (backends/xnnpack/quantizer/xnnpack_quantizer_utils.py, still present on main), and it affects every BERT-family model, not these three.

Worth knowing about the speed

XNNPACK fp32 is slower than PyTorch eager here (14.4 ms against 8.5), and fp16 is slower again while halving the file. Core ML is the one that pays: 2.0 ms, roughly 4.2x eager, at half the size. If this is going on an Apple device, take the Core ML build.

Conversion

python convert/export_embed.py all_minilm_l6
python convert/check_embed.py all_minilm_l6

Sequence length is fixed at 256; the attention mask makes padding harmless for mean pooling, and chunking anything longer is the caller's job.

(conversion scripts: executorch-models)