iktino/multilingual-e5-large-mlx-int8

multilingual-e5-large · MLX Swift 8-bit

A 100-language sentence embedder for on-device retrieval on Apple Silicon, at 0.63 GB.

Base	intfloat/multilingual-e5-large @ 3d7cfbda
Task	Sentence embeddings (1024-d), multilingual retrieval and similarity
Format	MLX Swift safetensors, 8-bit affine, group size 64 (Linear + Embedding)
Size	0.63 GB (fp32 original ≈ 2.2 GB)
Accuracy	byte-identical downstream results vs fp32 on a multilingual benchmark (below)
License	MIT (unchanged)

Intended use

Semantic search and related-item retrieval across 100 languages, fully on-device: notes, meetings, documents. Follow the E5 convention: prefix inputs with query: / passage: and mean-pool the last hidden state, then L2-normalise.

How it was made

The original fp32 weights were loaded with an MLX Swift BertModel (MLXEmbedders naming, after its sanitize step) and quantized in memory with MLXNN.quantize (affine, 8-bit, group size 64, default Linear and Embedding layers). The quantized module parameters were saved unchanged as model.safetensors; quantization.json records the scheme.

Verification

An item-linking benchmark of four multilingual packs (English, Czech and a multilingual parliament set) uses this model together with two companion models from the same family of repos (embedder, reranker, NLI). It produced byte-identical outputs with these 8-bit files and with the original fp32 weights.

Loading (read this first)

These are MLX Swift weights, not a Transformers checkpoint: transformers and sentence-transformers cannot load them. Use the original repo for those. To load in MLX Swift:

1. Build the MLXEmbedders BertModel from config.json (encoder only; mean pooling).
2. Call quantize(model: model, groupSize: 64, bits: 8, mode: .affine) on it before loading weights. Quantized layers store weight / scales / biases.
3. Load model.safetensors into the module and use the tokenizer files as usual.

quantization.json ({"bits": 8, "groupSize": 64, "mode": "affine"}) is the marker a loader can check to take step 2.

Files

file	content
model.safetensors	8-bit parameters
quantization.json	quantization scheme
config.json	copied from the source revision
tokenizer.json, tokenizer_config.json, special_tokens_map.json	copied from the source revision

License and attribution

MIT, the same as intfloat/multilingual-e5-large. All credit for the model goes to its authors (Wang et al., Multilingual E5 Text Embeddings, 2024). The change here is quantization only.