gemma-4-E4B-it-ONNX (text path, q4)

The text half of Gemma 4 E4B instruction-tuned, in ONNX, as consumed by Stenograf's Windows sidecar for on-device question answering over saved meeting transcripts.

Re-hosted from onnx-community/gemma-4-E4B-it-ONNX so the sidecar fetches from one pinned place alongside the other models it downloads. Use of these weights remains subject to the Gemma Terms of Use.

What is here

Only the two graphs the decode loop needs, at q4:

file	purpose
onnx/embed_tokens_q4.onnx (+ data)	ids to hidden state, plus the per-layer inputs this architecture takes
onnx/decoder_model_merged_q4.onnx (+ data)	one decode step, 24 key/value pairs in and out
tokenizer.json	SentencePiece-style BPE, 262144 pieces
generation_config.json	names the stop ids — [1, 106, 50]

The vision and audio encoders are not included: the transcripts are already text by the time a question is asked.

Why q4 and not q4f16

q4f16 is smaller and looks like the better choice, but it cannot run on CUDA with ONNX Runtime 1.28. Both variants pass an attention_bias into GroupQueryAttention, which the CUDA kernel rejects; in q4 those twenty nodes fall back to the CPU provider and the model still runs, while in fp16 no such fallback exists and session execution fails outright.

That fallback is also where the remaining headroom is — profiling one decode step put 23.3% of the time in GroupQueryAttention on the CPU provider. A runtime that accepts attention_bias on CUDA would recover it for free.

Notes for anyone driving these graphs

• The key/value layers are not uniformly shaped: most carry a 256-wide head, some 512. Build each tensor from its own declared signature.
• Keep the cache on the device between steps. Round-tripping it through host memory measured 2.4x slower on the same GPU.
• Gemma 4 renamed its turn markers to <|turn> and <turn|>. Gemma 3's <start_of_turn> is absent from this vocabulary, so a prompt written the old way encodes as ordinary text and the model never emits a stop id.