mlboydaisuke/EmbeddingGemma-2-Transcribe-Reader

EmbeddingGemma 2 Transcribe — reader v1

A 13.3 M-parameter reader that turns the audio soft tokens of Google's EmbeddingGemma 2 into English words: each 40 ms token is mapped into the model's 768-d text-embedding space and read as the closest of 200,000 words, or as blank, with greedy CTC decoding and no language model. The EmbeddingGemma 2 encoder stays frozen; only the reader is trained.

The idea comes from Shahir's post (2026-10-13) and the Embedding Gemma Transcribe Mac app (eg2.haku.sh), whose code and weights are not public. This reader is an independent reproduction, trained from scratch on LibriSpeech. It is not affiliated with Google.

These weights are framework-neutral (a PyTorch state dict and a NumPy table). The converted builds that run them on a device:

Build	Repository
Core AI (iOS 27 / macOS 27): the EmbeddingGemma 2 audio encoder and this reader with its word table, as .aimodel bundles	mlboydaisuke/EmbeddingGemma-2-Transcribe-CoreAI
LiteRT (Android)	mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT

Files

File	Bytes	SHA-256	What
reader.pt	53,095,605	67a06a857b9aeb066347f8947e4faeed11d2ec3e89e0c23974be0a2c20243d5b	the reader: a PyTorch state_dict, float32
word_table.npy	307,200,128	b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8	float16 [200000, 768], one L2-normalized row per word
vocab.txt	1,737,588	bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68	the 200,000 words, one per line; the line number is the word index
reader_config.json	2,019	68efaabae4c2eaa8ae2a5abf8f85fab115791b13178ee3a22e57a0cec727b94e	the contract for ports: input, output, logits, decoding, the window policy, the WER
RELEASE.md	12,584		the release note: WER details, the reader's output on the 20 fixture utterances, training log, weaknesses

SHA256SUMS lists every file. RELEASE.md is the release note of the reader/ directory of the eg2-transcribe repository; the paths in it (train/, fixtures/) are relative to that directory.

Contract

• Input: EmbeddingGemma 2 audio soft tokens, float32 [T, 512], 25 per second: the embed_audio output of google/embeddinggemma-2 at revision 914f7f89142e33e77833254d9c9b90c3cef7303b (model.get_audio_features(input_features, input_features_mask).pooler_output; transformers 5.19.0 with the default attention, not eager, whose audio mask is inverted in that version).
• Body: LayerNorm → Linear 512→512, 4 pre-LayerNorm transformer layers (width 512, 8 heads, FFN 2048 with GELU, rotary positions with base 10000), LayerNorm → Linear 512→768. Attention covers the whole input window in both directions. 13,268,994 parameters.
• Output per token: a unit vector in the 768-d text space and a blank logit.
• Logits: word w scores scale × cos(vector, word_table[w]), with scale = exp(log_scale) = 91.218 (learned); blank is class 200,000 and scores blank_logit.
• Decoding: per token, blank where blank_logit > top-1 word logit (a tie reads the word), else the top-1 word; merge repeats, drop blanks. No language model, no punctuation, no casing.
• Long audio: 30 s encoder windows (750 tokens), each starting 96 tokens (3.84 s) before the end of the previous one, keeping only its new tokens and dropping its last token: the encoder is causal, so the start of a window lacks its left context and the last token of a 30 s cut misses one input frame. reader_config.json → window.
• Weights file: reader.pt keys body.* (the reader body), blank.weight [1, 768], blank.bias [1], log_scale; the module is Reader(arch, n_vocab, k) of reader/train/model.py in the eg2-transcribe repository (arch a).

WER

Greedy, no language model, every utterance of each set. Reference and hypothesis are lower-cased with runs of spaces collapsed; words outside the vocabulary stay in the references and count as errors.

Set	Utterances	Reference words	WER	Substitutions	Deletions	Insertions
dev-clean (model selection)	2,703	54,402	6.57 %	2,823	375	374
dev-other	2,864	50,948	14.93 %	5,885	1,089	631
test-clean	2,620	52,576	6.81 %	2,789	412	378
test-other	2,939	52,343	15.10 %	6,040	1,141	724
fixtures test-clean-20	20	541	4.81 %	22	1	3

On a 3,257.4 s LibriVox recording (The Adventure of the Speckled Band), read in 125 windows of 30 s as above through the Core AI build on an Apple M4 Max, the WER against the Project Gutenberg text is 5.80 %, next to 5.78 % for the float32 PyTorch path on the same windows (9,889 reference words; the LibriVox intro and outro are in the audio only).

Training

• Data: LibriSpeech train-clean-100 (CC BY 4.0), 28,539 utterances, 100.65 h. The 2,117 utterances with a word outside the vocabulary were left out: 26,422 utterances, 92.8 h, 8,351,376 tokens. Soft tokens computed once in float32, stored as float16.
• dev-clean chose the reader body and the epoch; test-clean and test-other were decoded once, after training.
• Objective: CTC over the 200,000 words and blank (torch.nn.functional.ctc_loss, float32, zero_infinity, mean of loss / target length).
• Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, 500 warmup steps, then cosine to zero at step 16,520; batches of 16 utterances bucketed by length; gradient clipping 1.0; float32 throughout.
• 8 epochs, 13,216 steps; the best dev-clean WER came at epoch 8 (these weights). Training stopped on its active-time budget (2.72 h of 3.0 h).
• Hardware: one Apple M4 Max (128 GB), torch 2.11.0 on MPS, macOS 27.0 (26A428); 3.64 h of wall clock, 1.02 h of it paused while other jobs measured on the same Mac.
• Body chosen after one epoch each of three candidates on dev-clean: a 4-layer transformer from scratch (13.48 %, this one), EmbeddingGemma 2 text layers 0–3 (14.60 %), a per-token MLP (25.21 %).

The command that continued the chosen body, from the reader/train/ directory of the eg2-transcribe repository:

mkdir -p runs/v1 && cp runs/cmp_a/ckpt_epoch1.pt runs/v1/
for e in $(seq 2 10); do  # one process per epoch
  .venv/bin/python -I train.py --arch a --cache cache --vocab ../vocab.txt --table ../word_table.npy \
    --run-dir runs/v1 --resume runs/v1/ckpt_epoch$((e - 1)).pt --epochs $e --schedule-epochs 10 \
    --max-hours 3 --patience 2 || break
  [ "$(jq -r .stop_reason runs/v1/metrics.json)" = "reached $e epochs" ] || break  # early stop or budget
done

Word table

• vocab.txt: every word of the LibriSpeech lexicon (openslr.org/resources/11, public domain), lower case, sorted by code point, apostrophes kept.
• word_table.npy: the EmbeddingGemma 2 text embedding of each word with the prompt task: search result | query: (sentence-transformers, float32, then stored as float16 and L2-normalized). Each of 100 random words, embedded again, comes back at rank 1.
• Out of vocabulary: test-clean 220 of 52,576 words (0.42 %), test-other 283 of 52,343 (0.54 %). These are always errors.

Limits

• By word length: on 2,518 aligned words of 100 test-clean utterances, the word error rate is 2.8 % for words of 1–2 tokens (up to 80 ms), 3.4 % for 3–5 tokens and 9.7 % for 6 or more.
• The most frequent substitutions on test-clean are function words (in → and 41, and → in 39, the → a 20); homophones are 12.4 % of all substitutions, which a reader without a language model cannot tell apart.
• English read speech only (audiobooks); trained on utterances of up to 24.5 s and measured on utterances of up to 35.2 s; longer audio goes through the 30 s windows above.

License and credits

• This repository: Apache-2.0 (LICENSE); see NOTICE.
• LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on train-clean-100.
• LibriSpeech lexicon: https://www.openslr.org/11, public domain.
• EmbeddingGemma 2 (google/embeddinggemma-2, Google DeepMind): Apache 2.0. It produces the soft tokens the reader reads and the word table. Deployments must adhere to the Gemma Prohibited Use Policy.