digital-creature/embeddinggemma-300m-onnx-int8

EmbeddingGemma 300M — ONNX, dynamic int8 (CPU)

A CPU build of google/embeddinggemma-300m for fast query embedding with ONNX Runtime.

What it is

• Source: the fp32 export onnx-community/embeddinggemma-300m-ONNX (onnx/model.onnx).
• Quantization: onnxruntime.quantization.quantize_dynamic on MatMul only, int8 weights, per-channel. The matrix multiplications run in integer arithmetic. The token embedding table stays fp32: a lookup reads a few rows, so it needs no dequantization pass.
• Inputs input_ids, attention_mask; output sentence_embedding (float32, 768-d, pooled and normalized as in the source graph). No token_type_ids.
• Files: onnx/model.onnx + onnx/model.onnx.data, config.json, tokenizer.json. No safetensors, so loaders that prefer safetensors take the ONNX path.

The model_quantized.onnx in the source repository is a weight-only build: it dequantizes every MatMul weight and the whole embedding table on each run. On CPU it is slower than fp32 and peaks above 1 GB of memory. This build is the one to use for CPU inference.

Measured (Apple M-series, onnxruntime, 2026-10-07)

Build	14-token query, 4 threads / 1 thread	171-token document	Peak RSS
fp32	24.8 / 45.0 ms	211 ms	~495 MB
this int8	9.6 / 14.6 ms	81 ms	~330–370 MB

Cosine similarity to the fp32 vectors: 0.98–0.99. In a hybrid product-search evaluation (WANDS and four e-commerce catalogs), queries embedded with this build against documents embedded in fp32 scored the same NDCG@10 as fp32 queries.

Prompts

EmbeddingGemma expects task prompts; the graph adds none. Prefix queries with task: search result | query: and documents with title: {title} | text: {content}.

Licence

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. This is a modified (quantized) version of google/embeddinggemma-300m; use of it is subject to the same terms and the Gemma Prohibited Use Policy.