mlboydaisuke/bge-small-en-v1.5-ExecuTorch

bge-small-en-v1.5 — ExecuTorch

Sentence embeddings: text in, one vector out. For search, clustering and retrieval that never leaves the device.

• Source: BAAI/bge-small-en-v1.5 — 33M parameters, 12 BERT layers, 384-dimensional output
• License: MIT
• Input: input_ids and attention_mask, both [1, 256] int64
• Output: [1, 384]

The pooling is in the graph, on purpose

sentence-transformers keeps the recipe per model in 1_Pooling/config.json and modules.json, and the three small models on this shelf do not agree:

	pooling	normalised
all-MiniLM-L6-v2	mean	yes
bge-small-en-v1.5	CLS	yes
paraphrase-multilingual-L12	mean	no

This one uses the CLS token, then L2 normalise. Getting it wrong does not throw: mean-pooling BGE, or normalising the multilingual one, gives vectors that look fine and rank wrong. So it is baked in rather than left to the caller.

Verification (Mac arm64, 2026-08-23)

build	size	latency	worst cosine vs eager
XNNPACK fp32	133.0 MB	28.7 ms	1.000000
XNNPACK fp16	66.7 MB	50.9 ms	1.000000
Core ML fp32	66.9 MB	3.6 ms	0.999990

Eager fp32 on the same input is 16.3 ms. Cosine is against the model run in eager through its own documented pooling, over eight sentences including one in Japanese — not against random token ids, which would tell you nothing.

And that the vectors are useful, which agreement alone cannot show:

0.793 a paraphrase   vs 0.308 an unrelated sentence

Across languages — "機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the phone", with an unrelated sentence about the weather — this model scores 0.491 against 0.632 — it does not, and should not.

Not shipped

int8 converts and holds — worst cosine 0.999113 against eager over the same eight sentences — but it comes out at 69.5 MB against fp16's 66.7 MB, so nothing would pick it. The reason is arithmetic rather than a defect: dynamic int8 quantizes the linear weights and leaves the token embedding table alone, and that table is 46.9 MB of a 133.0 MB model. fp16 halves the table too. On a model whose weights are largely a vocabulary, fp16 is the smaller build.

An earlier version of this card said int8 does not export, with an IndexError: tensors used as indices must be long, int, byte or bool tensors. That was true of the build at the time and the cause was misread. The error is not in the embedding lookup: XNNPACKQuantizer's transform_for_annotation lifts every scalar argument of add.Tensor/mul.Tensor into a buffer and writes it as torch.tensor(float(arg)) regardless of the node's dtype, so the arange(n) + 0 that builds a HuggingFace attention mask comes back float32 and can no longer index. It is one line in ExecuTorch (backends/xnnpack/quantizer/xnnpack_quantizer_utils.py, still present on main), and it affects every BERT-family model, not these three.

Worth knowing about the speed

XNNPACK fp32 is slower than PyTorch eager here (28.7 ms against 16.3), and fp16 is slower again while halving the file. Core ML is the one that pays: 3.6 ms, roughly 4.5x eager, at half the size. If this is going on an Apple device, take the Core ML build.

Conversion

python convert/export_embed.py bge_small_en
python convert/check_embed.py bge_small_en

Sequence length is fixed at 256; the attention mask makes padding harmless for mean pooling, and chunking anything longer is the caller's job.

(conversion scripts: executorch-models)