okayuji/embeddinggemma-2-coreml-ane

EmbeddingGemma 2 for Core ML: text, image and audio embeddings

Turn text, an image, or an audio clip into a vector for local search and similarity. This repository provides FP16 and INT8 variants of Google's EmbeddingGemma 2, with a self-contained Python runner.

For developers building retrieval tools on Apple silicon Macs or integrating the fixed Core ML interfaces into an app. The runner defaults to CPU plus Apple Neural Engine. The export target is macOS 15 / iOS 18; execution was verified on M4 Max with macOS 27. iPhone execution is untested.

Quick start

Use Python 3.11 on macOS. Start with fp16/; choose int8/ for smaller files and review its measured differences below.

python3.11 -m venv .venv
source .venv/bin/activate
pip install "huggingface-hub==2.2.0"
hf download okayuji/embeddinggemma-2-coreml-ane --include "fp16/*" --local-dir embeddinggemma2-coreml
pip install -r embeddinggemma2-coreml/fp16/requirements.txt
python embeddinggemma2-coreml/fp16/embed.py --text "Meeting notes" --role query --output embedding.json

Replace fp16 with int8 in the download pattern and paths to use the quantized variant. Each variant is a complete bundle. Select its folder, rather than the repository root.

For your own media:

python embeddinggemma2-coreml/fp16/embed.py --image photo.png --output image.json
python embeddinggemma2-coreml/fp16/embed.py --audio speech.wav --output audio.json

Recorded example

Input: Meeting notes, role: query, variant: FP16, default cpu_and_ne.

The complete recorded JSON contains a 768-element embedding. Its first eight values are shown here as an excerpt:

[-0.030129466205835342, 0.0026138152461498976, 0.02965630777180195, -0.019094206392765045, -0.02828262373805046, 0.03986736014485359, -0.02609999291598797, 0.08168840408325195]

The command took 14.96 seconds as one fresh Python process on M4 Max with 128 GiB of physical memory / macOS 27.0 (26A428), using Python 3.11, coremltools 9.0, Torch 2.9.0 and Transformers 5.19.0. This is process wall time including imports, processor initialization, model loading and prediction; it is not accelerator-only latency or a throughput benchmark. The exact measurement is retained in the recorded conditions.

Variants

Sizes are approximate uncompressed bundle bytes, including the runner and verification files; they are not RAM measurements. Exact per-file sizes and hashes are in each manifest.

Folder	Weight storage and compute	Bundle size
fp16	FP16 model weights/compute; original BF16 vocabulary table	1.50 GB
int8	Per-channel INT8 Linear/Conv weights with floating norms, biases, output projections and vision position table; FP16 compute	0.81 GB

The INT8 vocabulary is stored separately in blocks of 32 with FP16 scales and decoded on the host for selected tokens. INT8 is post-training weight quantization, not Google's quantization-aware LiteRT-LM checkpoint.

What runs on the Neural Engine

On M4 Max / macOS 27.0 (26A428), with CPU_AND_NE, the main Linear, Conv and MatMul operations in all six encoders were assigned to ANE. The image encoder retains 27 CPU input operations. Model-named ANE Prediction events were recorded for text, vision and audio in both variants. This does not mean all application work, including tokenization and media preprocessing, runs on ANE.

The audio attention uses tensors of rank at most four, scaled RMS normalization and the original FP16 clipping bound of 65504. INT8 uses per-channel weight scales. These changes and their scope are recorded in FP16 verification and INT8 verification.

Input contract

• Text: at most 256 tokens including task prefix and special tokens. The runner rejects overflow. Use --role query for search queries and --role document for indexed text.
• Images: one RGB-converted image; aspect-preserving preprocessing; 630 patches pooled spatially into at most 70 soft tokens.
• Audio: 16 kHz mono PCM16 WAV, at least 161 samples. Each model window is at most 10 seconds. The runner returns separate timestamped vectors for longer recordings; a tail shorter than 161 samples uses an overlapping window.
• Output: a 768-dimensional normalized vector. --dimensions 512, 256 or 128 takes a prefix and re-normalizes it. Keep model variant and dimensions identical across an index and its queries.

The tokenizer adds BOS/EOS. The exact text prefixes are task: search result | query: and title: none | text: , including the trailing space. The runner applies them for you.

Images and audio produce soft tokens that are inserted between the corresponding special tokens and passed through the common text encoder. There is no transcript or generated response.

Files and integration

Each variant contains:

• text.mlpackage, vision.mlpackage, audio.mlpackage: the three Core ML programs.
• processor/: the pinned original tokenizer and processor settings.
• token_embeddings.*: the external vocabulary table and, for INT8, its scales.
• embed.py, requirements.txt, bundle.json, USAGE.txt: a standalone runner, pinned direct dependencies and input specifications.
• verification.json, ranking-comparison.json, stress-reference.json and the ANE event XML: bounded validation evidence.
• manifest.json, LICENSE, NOTICE, SOURCE_MODEL_CARD.md: file identities and licensing/provenance.

Program	Inputs	Output
text	inputs_embeds [1,256,512] float32; attention_mask [1,256] float32	embedding [1,768] float32
vision	pixel_values [1,630,768] float32; image_position_ids [1,630,2] int32; pooling_weights [1,70,630] float32	soft_tokens [1,70,512] float32
audio	input_features [1,1000,128] float32; input_features_mask [1,1000] float32	soft_tokens [1,250,512] float32

For custom integrations, follow the vocabulary scaling, spatial pooling, log-mel features, special-token insertion and masks in embed.py and USAGE.txt. The Python runner uses Transformers for preprocessing and Core ML for learned-model inference. These packages are separate from the Core AI edition.

Operational notes

Models are loaded lazily. Initial preparation can dominate a short command, so do not use process wall time as steady-state inference latency. Peak runtime memory, long-duration thermal behavior and cold/warm comparisons have not been measured for this release.

--compute-units cpu_and_gpu selects the verified alternative backend. cpu_only is selectable, but was not independently validated in these runs.

For Use 16 kHz mono PCM16 WAV audio, convert the recording to that format. For The input exceeds 256 tokens including its prefix and special tokens; split it into chunks, split the text into shorter inputs. Keep the complete variant folder together.

coremltools 9.0 reports that Torch 2.9.0 is newer than its tested Torch 2.7.0 baseline. The checks below were executed with the pinned dependencies; they are not a guarantee for other versions.

Limitations

• Verification covers M4 Max / macOS 27.0, not iPhone execution or sustained production workloads. The iOS 18 / macOS 15 export target is not an execution claim for every device or OS release.
• The fixed profile covers one text, image or audio window at a time. Video, interleaved mixed inputs and the source model's full context window are outside this conversion.
• Google warns against a naive FP16 cast. This conversion changes numerical operations and validates a bounded fixture set; it does not establish fidelity for all inputs.
• INT8 differs from the unquantized source. Silence and a synthetic 10-second chirp miss the original-fidelity gate. Fixed-fixture source top-1 agreement for 12 texts against 8 images is 11/12 for FP16 and 10/12 for INT8; this is ranking agreement, not labeled retrieval accuracy.
• Switching precision or model versions requires rebuilding embeddings for a consistent index. Broad retrieval quality and suitability for a particular business dataset require separate evaluation.

Verification

Measured on 2026-11-02: M4 Max with 128 GiB of physical memory, macOS 27.0 (26A428), coremltools 9.0, Torch 2.9.0, Transformers 5.19.0; batch size 1, sequential execution. The gate is cosine >= 0.999, maximum absolute difference <= 0.01, finite values and norm deviation <= 0.001. Raw Core ML output and host-normalized output are both checked.

CPU + ANE comparison, 26 fixed inputs	Passed	Minimum cosine	Maximum absolute difference
FP16 vs original FP32	26/26	0.999335516	0.003922381
INT8 vs FP32 with the same quantized weights	26/26	0.999096598	0.005787356
INT8 vs original, unquantized FP32	24/26	0.996373577	0.022329122

The set contains 12 texts, 8 images and 6 audio clips. CPU + GPU passed the matching-reference gate for the same 26 inputs per variant. Three additional audio cases per variant—161 samples, full-amplitude noise and a 10-second square wave—also passed on CPU + ANE. These checks measure conversion fidelity, not general search accuracy.

Full measurements: FP16, INT8. File sizes and SHA-256: FP16 manifest, INT8 manifest.

License and attribution

Derived from google/embeddinggemma-2 revision 914f7f89142e33e77833254d9c9b90c3cef7303b, by Google DeepMind. The upstream model is licensed under Apache 2.0. Copies of the license, attribution and modification notices are included with each variant. The upstream model card also directs deployments to the Gemma Prohibited Use Policy.

This is an independent conversion by Yuji Oka, not an official Google or Apple release. Google, Gemma, Apple and Core ML names and marks belong to their respective owners; no endorsement is implied.