janihal/embeddinggemma-300m-8bit-mlx

embeddinggemma-300m — MLX 8-bit (compact)

An 8-bit (affine, group size 64) MLX quantization of google/embeddinggemma-300m, produced with mlx-embeddings. Runs the full EmbeddingGemma pipeline (mean-pool → 2× Dense → L2-normalize), 768-dim output, Matryoshka truncation to 512/256/128 supported.

Weights on disk	344 MB (all Linear + Embedding layers int8, scales/norms fp16)
Quantization	affine, bits=8, group_size=64, every Linear + the token-embedding table
Backend	Apple-Silicon GPU via MLX
Sibling	embeddinggemma-300m-8bit-hi-mlx — larger (534 MB) but matches FP32 accuracy exactly

Which one do I want? This model is the smallest option and is fine for most use — it costs about −0.2 pp STS / −0.5 pp nDCG vs full precision (see below). If you want zero measurable accuracy loss, use the -hi-mlx sibling or the llama.cpp ggml-org GGUF instead.

Usage

pip install mlx-embeddings

import mlx.core as mx
from mlx_embeddings.utils import load

model, tokenizer = load("janihal/embeddinggemma-300m-8bit-mlx")

# EmbeddingGemma prompt prefixes matter:
QUERY = "task: search result | query: "
DOC   = "title: none | text: "

texts = [QUERY + "What is the capital of France?",
         DOC   + "Paris is the capital and most populous city of France."]

enc = tokenizer.batch_encode_plus(texts, return_tensors="mlx", padding=True)
out = model(enc["input_ids"], attention_mask=enc["attention_mask"])
emb = out.text_embeds                    # (2, 768), already mean-pooled + L2-normalized
sim = float((emb[0] * emb[1]).sum())     # cosine similarity

Prompt templates for other tasks (from config_sentence_transformers.json): Retrieval-query → task: search result | query: · Retrieval-document → title: none | text: · STS → task: sentence similarity | query: · Classification → task: classification | query: · Clustering → task: clustering | query: .

How it was produced

python -m mlx_embeddings.convert \
  --hf-path google/embeddinggemma-300m \
  --mlx-path embeddinggemma-300m-q8 \
  -q --q-bits 8 --q-group-size 64 --q-mode affine

---

Evaluation

All numbers below are from a controlled comparison on one Apple M5 Max. Reference = the same model converted to MLX float32 (no quantization). Quantized candidates are scored against it on two public benchmarks and on raw embedding agreement.

• STS: STS-Benchmark test, 1379 pairs → Spearman(cosine, gold 0–5).
• Retrieval: NFCorpus (BEIR) test, 323 queries × 3633 docs → nDCG@10.
• mean cos vs FP32: mean cosine between this model's output vectors and the FP32 reference's, over all 6714 benchmark texts.
• Δ confidence intervals are paired bootstrap, 2000 resamples.

Accuracy

Model	Format	Size	STS Spearman	ΔSTS (95% CI)	NFCorpus nDCG@10	ΔnDCG (95% CI)	mean cos vs FP32
google/embeddinggemma-300m (MLX fp32)	fp32	1.2 GB	88.18	—	39.06	—	1.0000
➤ this — MLX 8-bit, all-linears	int8 g64	344 MB	87.98	−0.19 [−0.44, +0.05]	38.57	−0.49 [−1.05, +0.05]	0.9918
MLX 8-bit, blocks-only (hi-mlx)	int8 g32 + fp16 embed	534 MB	88.24	+0.07 [−0.06, +0.19]	39.00	−0.07 [−0.36, +0.24]	0.9942
llama.cpp GGUF Q8_0 — ggml-org	Q8_0	318 MB	88.17	−0.01 [−0.02, +0.01]	39.10	+0.04 [−0.27, +0.37]	0.9987
llama.cpp GGUF Q8_0 — unsloth (QAT)	Q8_0	314 MB	88.63	+0.45 [+0.03, +0.91]	38.53	−0.52 [−1.44, +0.46]	~0 †

† The unsloth GGUF is built from Google's QAT checkpoint and omits the two Dense projection layers (converted without --sentence-transformers-dense-modules). Its output is the pooled backbone in a different vector space — hence ~0 cosine to the reference even though its STS score is fine. Not a like-for-like point.

Read: this compact model's accuracy loss is small but real and consistently negative (STS, nDCG, and embedding cosine all point the same way). The ggml-org GGUF and the -hi-mlx sibling are effectively lossless (Δ CIs cover 0).

Weight-space quantization error

Param-weighted relative RMSE of the de-quantized weights vs the FP32 base:

Source	relRMSE vs FP32
MLX 8-bit, all-linears (this)	0.59 %
MLX 8-bit, blocks-only (hi-mlx)	0.49 %
GGUF Q8_0 (ggml-org)	0.60 %
GGUF Q8_0 (unsloth, QAT)	2.75 % (different checkpoint)

MLX affine-8bit and llama.cpp Q8_0 quantize the weights equally well. The end-to-end accuracy gap of this model comes almost entirely from 8-bit quantizing the token-embedding table (65 % of the parameters); keeping it in float precision — what the -hi-mlx sibling does — removes the gap.

Performance (Apple M5 Max, both stacks on the Metal GPU)

Stack	Corpus throughput (3633 docs, 1.26 M tok)	Query latency, batch 1 (p50)	Model load	Peak RAM
this — MLX 8-bit	253 docs/s · 87 k tok/s	3.2 ms	1.1 s	1.6 GB
MLX 8-bit blocks-only (hi-mlx)	249 docs/s · 86 k tok/s	3.2 ms	1.1 s	1.8 GB
llama.cpp GGUF Q8_0 (ggml-org)	137 docs/s · 47 k tok/s	~5 ms ·	0.4 s	3.3 GB

· llama.cpp latency measured through llama-server and includes a localhost HTTP round-trip. Its model load is ~3× faster than MLX (no Python import).

Read: on this machine MLX is ~1.8× faster for bulk embedding and ~1.3–1.5× faster per query than the GGUF, at roughly half the RAM. MLX throughput peaks around batch 16–32 (batch 64 drops to 220 docs/s from padding waste). Single-run numbers, ±10 %.

Compared to other embedding models

Same benchmark (STS-B test, NFCorpus test) and same Apple M5 Max. MLX rows run on MLX/Metal, GGUF rows on llama.cpp/Metal. docs/s = wall time to embed the 3633-document NFCorpus corpus; query latency is single-text, warm.

Model	Params	STS ρ	nDCG@10	docs/s	Query p50	Peak RAM
EmbeddingGemma-300m — FP32 reference	0.3 B	88.2	39.1	—	—	—
➤ this — MLX 8-bit	0.3 B	88.0	38.6	253	3.2 ms	1.6 GB
EmbeddingGemma-300m — MLX 8-bit-hi (sibling)	0.3 B	88.2	39.0	249	3.2 ms	1.8 GB
EmbeddingGemma-300m — GGUF Q8_0 (llama.cpp)	0.3 B	88.2	39.1	137	~5 ms	3.3 GB
Qwen3-Embedding-0.6B — GGUF Q8_0	0.6 B	91.3	36.7	50	4.0 ms	2.5 GB
Qwen3-Embedding-4B — GGUF Q8_0	4 B	93.6	40.8	10	12.6 ms	6.0 GB
Qwen3-VL-Embedding-2B — GGUF Q8_0	2 B	86.6	35.7	23	6.5 ms	3.7 GB

Read: only Qwen3-Embedding-4B clearly out-retrieves EmbeddingGemma-300m (nDCG 40.8 vs ~39) — at ~25× the embedding time and 4× the RAM. Qwen3-Embedding-0.6B is a stronger pure-similarity model (STS 91.3) but a weaker retriever (nDCG 36.7) and ~5× slower. Qwen3-VL-Embedding-2B (multimodal) is weaker than EmbeddingGemma on text on both axes. For retrieval / RAG at this size, EmbeddingGemma-300m is the best accuracy per byte and per second.

(Nemotron-3-Embed-1B GGUF did not produce usable embeddings through this llama.cpp build and is omitted.)

Caveats

• "FP32 reference" is the MLX implementation; the GGUFs run in llama.cpp, so a small cross-framework gap (~0.1–0.2 %) is folded into their numbers.
• One benchmark pair (STS + one retrieval set). Not a full MTEB run.
• Performance measured on M5 Max; ratios shift with hardware, batch size, and text length. llama.cpp likely has some tuning headroom (-fa, threads, ubatch).
• Loadable only with mlx-embeddings, not vanilla sentence-transformers.

---

License & attribution

Derived from google/embeddinggemma-300m (Google DeepMind) by post-training weight quantization only — no fine-tuning.

Use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy. This is a modified version of EmbeddingGemma; the same terms and use restrictions apply to this model and its outputs.

Quantization tooling: mlx-embeddings · MLX. GGUF baselines: ggml-org/embeddinggemma-300M-GGUF, unsloth/embeddinggemma-300M-GGUF.