janihal/embeddinggemma-300m-8bit-hi-mlx

embeddinggemma-300m — MLX 8-bit (accuracy-matched)

A mixed-precision MLX quantization of google/embeddinggemma-300m: the 24 transformer blocks are 8-bit (affine, group size 32), while the token-embedding table, both Dense projection heads, and all RMSNorms are kept in float precision (fp16). Runs the full EmbeddingGemma pipeline (mean-pool → 2× Dense → L2-normalize), 768-dim output, Matryoshka truncation to 512/256/128 supported.

Weights on disk	534 MB (blocks int8 g32, embedding + Dense + norms fp16)
Quantization	affine, bits=8, group_size=32, transformer blocks only (skip_embed, skip_dense)
Backend	Apple-Silicon GPU via MLX
Sibling	embeddinggemma-300m-8bit-mlx — smaller (344 MB), ~0.2–0.5 pp accuracy cost

Why this variant? Plain 8-bit MLX (the sibling) quantizes the token-embedding table too, which distorts the embedding space enough to cost ~0.2 pp STS / ~0.5 pp retrieval. Keeping that table (65 % of the parameters) in fp16 closes the gap entirely — this model is statistically indistinguishable from full precision on the benchmarks below, matching the llama.cpp ggml-org GGUF. The price is size: it is larger than plain 8-bit, though still smaller than fp16.

Usage

pip install mlx-embeddings

import mlx.core as mx
from mlx_embeddings.utils import load

model, tokenizer = load("janihal/embeddinggemma-300m-8bit-hi-mlx")

QUERY = "task: search result | query: "
DOC   = "title: none | text: "

texts = [QUERY + "What is the capital of France?",
         DOC   + "Paris is the capital and most populous city of France."]

enc = tokenizer.batch_encode_plus(texts, return_tensors="mlx", padding=True)
out = model(enc["input_ids"], attention_mask=enc["attention_mask"])
emb = out.text_embeds                    # (2, 768), mean-pooled + L2-normalized
sim = float((emb[0] * emb[1]).sum())     # cosine similarity

Prompt templates for other tasks (from config_sentence_transformers.json): Retrieval-query → task: search result | query: · Retrieval-document → title: none | text: · STS → task: sentence similarity | query: · Classification → task: classification | query: · Clustering → task: clustering | query: .

How it was produced

mlx_embeddings.convert quantizes every Linear + the Embedding. To quantize the transformer blocks only, a custom class_predicate was passed to mlx.nn.quantize:

import mlx.nn as nn
from mlx.utils import tree_flatten
from mlx_embeddings.utils import fetch_from_hub, save_weights, save_config

model, config, tok = fetch_from_hub(get_model_path("google/embeddinggemma-300m"), lazy=True)
model.load_weights([(k, v.astype(mx.float16)) for k, v in tree_flatten(model.parameters())])

def keep_fp(p, m):                       # True  -> quantize
    if not hasattr(m, "to_quantized"):   return False
    if "embed_tokens" in p:              return False   # keep token table fp16
    if p.startswith("dense"):            return False   # keep Dense heads fp16
    return m.weight.ndim >= 2 and m.weight.shape[-1] % 32 == 0

nn.quantize(model, group_size=32, bits=8, mode="affine", class_predicate=keep_fp)
save_weights("embeddinggemma-300m-q8-blk16", dict(tree_flatten(model.parameters())))

---

Evaluation

Controlled comparison on one Apple M5 Max. Reference = the same model converted to MLX float32. Candidates are scored against it on two public benchmarks and on raw embedding agreement.

• STS: STS-Benchmark test, 1379 pairs → Spearman(cosine, gold 0–5).
• Retrieval: NFCorpus (BEIR) test, 323 queries × 3633 docs → nDCG@10.
• mean cos vs FP32: mean cosine to the FP32 reference's vectors, over all 6714 texts.
• Δ confidence intervals are paired bootstrap, 2000 resamples.

Accuracy

Model	Format	Size	STS Spearman	ΔSTS (95% CI)	NFCorpus nDCG@10	ΔnDCG (95% CI)	mean cos vs FP32
google/embeddinggemma-300m (MLX fp32)	fp32	1.2 GB	88.18	—	39.06	—	1.0000
➤ this — MLX 8-bit, blocks-only	int8 g32 + fp16 embed	534 MB	88.24	+0.07 [−0.06, +0.19]	39.00	−0.07 [−0.36, +0.24]	0.9942
MLX 8-bit, all-linears (8bit-mlx)	int8 g64	344 MB	87.98	−0.19 [−0.44, +0.05]	38.57	−0.49 [−1.05, +0.05]	0.9918
llama.cpp GGUF Q8_0 — ggml-org	Q8_0	318 MB	88.17	−0.01 [−0.02, +0.01]	39.10	+0.04 [−0.27, +0.37]	0.9987
llama.cpp GGUF Q8_0 — unsloth (QAT)	Q8_0	314 MB	88.63	+0.45 [+0.03, +0.91]	38.53	−0.52 [−1.44, +0.46]	~0 †

† The unsloth GGUF is built from Google's QAT checkpoint and omits the two Dense projection layers. Its output lives in a different vector space (hence ~0 cosine to the reference); not a like-for-like point.

Read: both Δ CIs for this model straddle 0 on both benchmarks — it tracks full precision as tightly as the ggml-org GGUF does. A run with the embedding table in fp32 (939 MB) reaches mean cos vs FP32 = 0.99992; storing it fp16 trades a little raw-vector fidelity for half the size with no measurable task cost.

Weight-space quantization error

Param-weighted relative RMSE of the de-quantized block weights vs FP32 (the embedding + Dense are not quantized here, so 0 error there):

Source	relRMSE vs FP32
MLX 8-bit blocks-only (this) — blocks only	0.49 %
MLX 8-bit all-linears (8bit-mlx) — all tensors	0.59 %
GGUF Q8_0 (ggml-org) — all tensors	0.60 %
GGUF Q8_0 (unsloth, QAT) — all tensors	2.75 % (different checkpoint)

Performance (Apple M5 Max, both stacks on the Metal GPU)

Stack	Corpus throughput (3633 docs, 1.26 M tok)	Query latency, batch 1 (p50)	Model load	Peak RAM
this — MLX 8-bit blocks-only	249 docs/s · 86 k tok/s	3.2 ms	1.1 s	1.8 GB
MLX 8-bit all-linears (8bit-mlx)	253 docs/s · 87 k tok/s	3.2 ms	1.1 s	1.6 GB
llama.cpp GGUF Q8_0 (ggml-org)	137 docs/s · 47 k tok/s	~5 ms ·	0.4 s	3.3 GB

· llama.cpp latency measured through llama-server and includes a localhost HTTP round-trip. Its model load is ~3× faster than MLX (no Python import).

Read: keeping the embedding table in fp16 rather than int8 costs essentially nothing at inference — throughput and latency match the compact sibling (an embedding lookup is a gather, not a matmul). On this machine MLX is ~1.8× faster for bulk embedding and ~1.3–1.5× faster per query than the GGUF, at about half the RAM. Single-run numbers, ±10 %.

Compared to other embedding models

Same benchmark (STS-B test, NFCorpus test) and same Apple M5 Max. MLX rows run on MLX/Metal, GGUF rows on llama.cpp/Metal. docs/s = wall time to embed the 3633-document NFCorpus corpus; query latency is single-text, warm.

Model	Params	STS ρ	nDCG@10	docs/s	Query p50	Peak RAM
EmbeddingGemma-300m — FP32 reference	0.3 B	88.2	39.1	—	—	—
➤ this — MLX 8-bit-hi	0.3 B	88.2	39.0	249	3.2 ms	1.8 GB
EmbeddingGemma-300m — MLX 8-bit (sibling)	0.3 B	88.0	38.6	253	3.2 ms	1.6 GB
EmbeddingGemma-300m — GGUF Q8_0 (llama.cpp)	0.3 B	88.2	39.1	137	~5 ms	3.3 GB
Qwen3-Embedding-0.6B — GGUF Q8_0	0.6 B	91.3	36.7	50	4.0 ms	2.5 GB
Qwen3-Embedding-4B — GGUF Q8_0	4 B	93.6	40.8	10	12.6 ms	6.0 GB
Qwen3-VL-Embedding-2B — GGUF Q8_0	2 B	86.6	35.7	23	6.5 ms	3.7 GB

Read: this model matches FP32 / the GGUF on both tasks. Among the others, only Qwen3-Embedding-4B clearly out-retrieves EmbeddingGemma-300m (nDCG 40.8 vs ~39) — at ~25× the embedding time and 4× the RAM. Qwen3-Embedding-0.6B is a stronger pure-similarity model (STS 91.3) but a weaker retriever (nDCG 36.7) and ~5× slower. Qwen3-VL-Embedding-2B (multimodal) is weaker than EmbeddingGemma on text on both axes. For retrieval / RAG at this size, EmbeddingGemma-300m is the best accuracy per byte and per second.

(Nemotron-3-Embed-1B GGUF did not produce usable embeddings through this llama.cpp build and is omitted.)

Caveats

• "FP32 reference" is the MLX implementation; the GGUFs run in llama.cpp, so a small cross-framework gap (~0.1–0.2 %) is folded into their numbers.
• One benchmark pair (STS + one retrieval set). Not a full MTEB run.
• Performance measured on M5 Max; ratios shift with hardware, batch size, and text length. llama.cpp likely has some tuning headroom (-fa, threads, ubatch).
• Loadable only with mlx-embeddings, not vanilla sentence-transformers.

---

License & attribution

Derived from google/embeddinggemma-300m (Google DeepMind) by post-training weight quantization only — no fine-tuning.

Use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy. This is a modified version of EmbeddingGemma; the same terms and use restrictions apply to this model and its outputs.

Quantization tooling: mlx-embeddings · MLX. GGUF baselines: ggml-org/embeddinggemma-300M-GGUF, unsloth/embeddinggemma-300M-GGUF.