wesjos/embeddinggemma-2-openvino-int8

EmbeddingGemma 2 — OpenVINO INT8 (weight-only)

google/embeddinggemma-2 quantized to INT8_SYM weights only, per-channel, data-free (no calibration set). Activations stay fp32. Three towers, 1.98× smaller, 1.5–2.2× faster on CPU, and the embeddings are still interchangeable with the fp16 model.

Tower	Files	Size	Output
Text	text_int8.xml + .bin	275 MB	sentence_embedding [b, 768]
Vision	vision_int8.xml + .bin	170 MB	image_features [n, 512]
Audio	audio_int8.xml + .bin	296 MB	audio_features [n, 512]

The .xml holds the topology and the .bin the weights; keep them side by side with a matching name prefix. Total 741 MB vs 1470 MB fp16.

Usage

pip install openvino tokenizers numpy pillow
python demo.py

text x document:
  set up a faster package source on linux  -> How to configure the apt mirror on Debian  0.740
  quantum bit operations                   -> Quantum computing uses qubits               0.818
  a sleeping animal                        -> The cat sleeps on the windowsill            0.802

dog.jpeg x captions:
  a dog                            0.726
  a panda                          0.576
  a penguin using ubuntu computer  0.505

jfk.wav x queries:
  a president's speech about serving your country  0.772

Those are the same anchors the fp16 and ONNX repos publish (0.725 / 0.576 / 0.505, audio 0.772) — int8 lands on the same numbers within 0.001.

• Outputs are L2-normalized, so a dot product is the cosine similarity.
• Retrieval prefixes: queries take task: search result | query: , documents take title: none | text: .
• Images: vision tower produces [N, 512] soft tokens, which the text tower fuses. Put <|image|> (258880) placeholders in input_ids and pass the features via image_features. Audio works the same way with <|audio|> (258881); video frames each go through the vision tower.
• Image preprocessing (aspect-ratio resize to a multiple of 3*16, 16x16 patchify, pad to 280*3²=2520 patches with pixel_position_ids = -1) is inlined in demo.py and only needs numpy and PIL.

Speed on 8 CPU threads, same host (Core Ultra 5 125H): short query 14 ms, 64-document batch 9.4 ms/doc, 2.7 s for one image, 0.19 s for 11 s of audio.

How it was quantized

import nncf, openvino as ov
model = ov.Core().read_model("model.xml")          # the fp16 IR
q = nncf.compress_weights(model, mode=nncf.CompressWeightsMode.INT8_SYM, ratio=1.0)
ov.save_model(q, "text_int8.xml", compress_to_fp16=True)

Drift against the fp16 IR on the same inputs (acceptance threshold 0.995, all PASS):

Tower	cos min	cos mean	extra
text	0.99899	0.99976	retrieval top-1 identical 23/23
vision	0.99955	0.99966	image-embedding level, 4 images
audio	0.99973	—	one 11 s clip

Do not re-try these two, they are measured, not guessed:

• INT8_ASYM destroys the vision tower (mean per-token cos 0.12; the same graph with INT8_SYM gives 0.9985).
• W8A8 activation quantization (nncf.quantize + calibration images) destroys the vision tower (0.019, not rescued by 32 calibration samples) and makes NNCF fail outright on the audio tower. It works on the text tower but is only ~8 % faster than weight-only.

So: weight-only INT8_SYM for this model. Activations stay fp32.

Per-token caveat

Weight-only int8 rotates a few individual soft-token rows while the pooled embedding is untouched: vision 1 of 256 rows and audio 4 of 275 rows fall below 0.99, while the item embedding stays ≥0.9993. Those rows are not padding — their norms match the rest. The fusion step pools over all tokens, so retrieval and similarity are safe; re-validate before feeding raw image_features / audio_features into a downstream model.

CPU, NPU and GPU

CPU is the recommended target; every number above is CPU. On this Meteor Lake host (Core Ultra 5 125H), OpenVINO exposes ['CPU', 'NPU'] — the iGPU is not enumerable without the Intel compute runtime.

• NPU, text tower — works, but only with static shapes and batch 1. At 1×128: 37.1 ms vs 65.1 ms on CPU, direction still correct (cos 0.9996 vs CPU), but the trailing L2 normalization is dropped, so the output arrives un-normalized (norm ≈ 3.49) and you must renormalize by hand.
• NPU, audio tower — do not use. Compiling with a static reshape "succeeds", but the output comes back as [275, 275] instead of [275, 512]: wrong shape, not the soft tokens. The shipped dynamic graph fails to compile outright (Upper bounds are not specified). The fp16 IR fails on the NPU as well, so this is an NPU-plugin limit, not a quantization artefact.
• NPU, vision tower — fails to compile: ConvertReduceToPooling Pass failed on the padded patch grid. Same for the fp16 IR.

Provenance

Unofficial conversion. The base weights come from wesjos/embeddinggemma-2-onnx-fix, where the upstream export's com.microsoft.RotaryEmbedding / com.microsoft.MultiHeadAttention contrib nodes are replaced with standard-domain ops so OpenVINO does not silently mistranslate them. The fp16 IR is wesjos/embeddinggemma-2-openvino.

Audio note: audio_feats/jfk.npz holds precomputed mel features. Generating them needs transformers >= 5.18 (Gemma4AudioFeatureExtractor); demo.py itself does not import it.

Original model by Google (Apache-2.0).