visible-cx/gemma-4-E4B-CoreAI

Gemma 4 E4B — Core AI (.aimodel)

google/gemma-4-E4B-it-qat-q4_0-unquantized converted to Core AI .aimodel bundles for Apple silicon by visible-cx. These are derivative artifacts: Google's QAT-trained weights rounded onto the int4 grid they were trained for and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable by PyTorch, GGUF or MLX.

Gemma 4 E4B uses Per-Layer Embeddings, so the working bundles take a large embedding gather table as a static graph input rather than carrying it in the graph. That table ships in ios-frontend/ and the bundles do not load without it; a missing table produces a bare input-arity error naming ple_table/ple_scale.

⚠️ Known issue — memory growth on long generations. The Core AI runtime caches a graph specialization per input-shape signature, and this export makes the sequence length load-bearing: every generated token adds one token to position_ids and thus mints a new signature, retaining roughly 80 MB of GPU allocations per generated token (both engines, guided and unguided alike, at equal context). Workloads whose prompt/output lengths repeat — e.g. short constrained extraction rows — stay bounded, because previously seen lengths replay from cache at no cost. Long free-form generations grow without bound until the process exits; no in-process mitigation exists (token caps only postpone it, and shape-length reuse is the only stable state). A re-export onto a shape-stable contract that removes this growth is available in this repository under stable/ — see its section below. It requires updated host support (in progress); on current hosts, use the bundles above for guided generation only.

Provenance

Base checkpoint	google/gemma-4-E4B-it-qat-q4_0-unquantized (ungated)
Zoo recipe	gemma-4-e4b, status = "verified" — int4lin
Recipe (pf64)	export_gemma4_pf_pipelined.py --pf 64 with --tbl and --raw-dir pointed at the gather table below
Toolchain base	apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 + the coreai-model-zoo python overlay
Toolchain	coreai-torch 0.4.1, coreai-core 1.0.0b2, coreai-opt 0.2.1, torch 2.9.0
Producer fingerprint	coreai-core 1.0.0b2 on every inner asset metadata.json
Weight format	int4, per-block-32 (int4lin, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on
Vocab	262,144
Export functions	main (S=1 decode) and, in _pf64 bundles, prefill (S=64 chunked prefill)

"QAT-unquantized" means QAT-trained, stored full width; the int4 rounding happens at export onto the grid training already targeted.

_tbl = the PLE gather table is bound as a static graph input. _pf64 = a second entrypoint, prefill, with a static query width of 64 (function_map: {"main": ["main", "prefill"]}).

Contents

Bundles

Path	Bytes	Context	Functions
gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64	3,989,986,858	4096	main + prefill
w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64	3,990,690,638	16384	main + prefill

Each folder holds <name>.aimodel/ (main.mlirb ≈ 3.96 GB, main.hash, asset metadata.json), a bundle-level metadata.json, and tokenizer/ (tokenizer.json 32,169,626 B, tokenizer_config.json, generation_config.json, chat_template.jinja 18,569 B). The w4a8 folder additionally ships its calibration_corpus.jsonl (35,045 B).

Stop token: every bundle declares eos_token = "<turn|>" (id 106), which is the turn terminator Gemma 4 emits. generation_config.json independently lists eos_token_id: [1, 106, 50]. A host that stops on the raw upstream <eos> instead will overrun every reply.

The PLE gather-table sidecar — required, not optional

Path	Files	Bytes
ios-frontend/gemma4_e4b_qat_gather_raw/	7	3,601,859,815

File	Bytes
embed_per_layer.i8	2,818,572,288
embed_tokens.i8	671,088,640
proj.f32	110,100,480
embed_per_layer.scale.f32	1,048,576
embed_tokens.scale.f32	1,048,576
proj_norm.f32	1,024
meta.json	231

Every _tbl bundle binds ple_table (from embed_per_layer.i8) and ple_scale (from embed_per_layer.scale.f32) as static graph inputs.

Requirements

• Apple silicon Mac, Core AI runtime.

• Engine contract: 4 inputs — input_ids, position_ids, plus the static ple_table and ple_scale. Two engines accept that:

  • Pipelined engine — binds the static inputs zero-copy over the caller's buffer, but does not support logits, so no grammar-constrained decoding.
  • Sequential engine — the only logits-capable engine, and therefore the only path for guided decoding. It accepts >= 2 inputs and binds everything beyond input_ids/position_ids from EngineOptions.staticInputBuffers; an extra input with no buffer is rejected by name. It pays a one-time copy of every constant input at load — 2.69 GB for E4B, because its submission path otherwise materialises a foreign buffer-backed view on every pass. Binding the table as a raw view instead drives peak footprint to 14.9 GB and faults the runtime mid-prefill.

• States: keyCache / valueCache Float16, 24 × 1 × 2 × ? × 512. Dynamic sequence dim → GrowingKVCache (initial 256, doubling).

• KV cost: 98,304 bytes per token (fp16) — 403 MB at 4096, 805 MB at 8192, 1.61 GB at

• Memory envelope, 16 GB Mac (Metal recommendedMaxWorkingSetSize ≈ 10.7 GB):

  Weights (main.mlirb)	3.96 GB
  PLE gather table (static input)	2.82 GB
  Base	6.78 GB
  Affordable context, fp16 KV	~39,900 tokens

  KV is not the ceiling at this tier; the PLE table is. Measured peak footprint on a guided run: 9.99 GB, max RSS 9.63 GB — right at the working-set edge. Minimum practical machine memory: 16 GB with nothing else resident, for guided decoding only; 24 GB or more is the comfortable tier. Free-form generation does not fit this envelope on any machine — see the known issue above.

• Run one E4B session per process. Wired memory is not reclaimed until the process exits; a second full run in the same process drives wired memory to 13.2 GB on a 16 GB machine.

• Sliding-window note: E4B interleaves 20 sliding and 4 full attention layers across 42 layers collapsed to 24 KV slots. The export models the window in the mask, not in memory — sliding layers ride full-length slots and zero-pad head_dim 256→512, so 83.3% of KV bytes sit in windowed slots and 41.7% is dead padding.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1. Both engines derive the static query width from the graph.

Performance

Measured on a 16 GB Apple silicon Mac (M2 Pro, macOS 27 beta): 10-sample harness, guided JSON-constrained decoding against a fixed schema, greedy, sequential engine with the PLE tables bound as static inputs, reset() between samples, 128-token cap. Load excluded from s/row; sample 1 excluded as a cache-warm outlier.

gemma4_e4b_qat_decode_int4lin_tbl_pf64:

Load	16.7 s
Guided JSON parse	10/10
Enum-clean	10/10
s/row (long samples)	14.70
s/row (short samples)	5.93
Decode	9.5–26.4 tok/s (mean 17.8)
TTFT	1.23–6.51 s
Peak footprint	9.99 GB
Max RSS	9.63 GB
Stop token	<turn|>, clean self-stop on every sample

E4B is memory-bound at the 16 GB tier, and its per-sample cost is sensitive to what else is resident on the machine — expect roughly 2× these figures when the model has to share.

The grammar is close to free; prefill is the cost. Decode throughput is essentially unchanged from unguided pipelined measurements. Guided samples cost ~25% more than unguided ones because the sequential engine's prefill is step-synchronous, not because of the constraint.

Enum conformance is entirely the grammar's doing. Unguided, this model emits an off-schema enum value in all ten samples. Guided, all ten are correct, because an off-enum token is unsamplable.

Published Mac figures of ~55.8 tok/s for E4B decode should be read as larger-machine figures.

Unguided workload — memory-capped

Free-form generation from a fixed prompt. Every generation length below is a memory cap, not a model stop — see the known issue.

depth	prompt tokens	TTFT	decode	generated	wall	peak footprint
3.4k	3314	68.46 s	0.78 tok/s	64 (capped)	151.6 s	16.91 GB
8k	7907	159.1 s	0.52 tok/s	32 (capped)	218.8 s	31.33 GB

Output quality up to the cap is sound; the limit is memory, not capability.

Needle-in-haystack recall at 8k: ≥2/3. Three distinctive facts planted at 10/50/90% of the filler; the 32-token memory cap truncated the answer mid-fact-2, so fact 3 was never reachable. 15k was not attempted.

Status

Artifact	Status
gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64 (ctx 4096)	QUALIFIED FOR GUIDED GENERATION — measured: 10/10 parse, 10/10 enum-clean, 14.70/5.93 s/row, 9.5–26.4 tok/s, 9.99 GB peak footprint. Not qualified for free-form generation on the current reference engine implementation.
w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64	EXPERIMENTAL — built, unmeasured. Gate: a Mac-side oracle/parity check plus a device benchmark.
stable/…_tbl_pf64_stable_c16384	EXPERIMENTAL — shape-stable decode contract; removes the per-token memory growth by design; host support exists; measured E2B: memory fix confirmed but ~1 tok/s decode at capacity 16384 — awaiting a smaller-capacity export before any use.

About the w4a8 bundle

Same QAT int4 per-block-32 weights and the same graph as gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64; the difference is an int8 quantize/dequantize pair on the inputs of every F.linear, calibrated on 128 synthetic samples (the corpus ships in the folder). Manifest context 16384.

The caveat comes from the compression toolchain's own documentation: the runtime fast path is W_INT8 × A_INT8, and a float weight path executes in floating point regardless of activation quantization. These weights use int4 scale-multiply dequant, not an int8 LUT, so the expected TTFT gain may be zero or negative. This bundle exists to be measured, not because a win is predicted.

No numerics gate has been run on any bundle in this repo. The 10/10 results are behavioural (parse rate, enum conformance, clean stop); a decode oracle against an fp32 reference has not been run.

License

The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma model and the Gemma terms apply downstream, which is why this repo declares license: gemma. Use is subject to the Gemma Terms of Use, the Gemma Prohibited Use Policy and the Gemma 4 license. Those obligations travel with any redistribution of these bundles, including the gather-table sidecar, which is derived from the same weights. The contribution here is the conversion, not the weights.

stable/ — shape-stable decode contract

⚠️ EXPERIMENTAL — built, not yet measured on a Mac. A re-export of the same weights and the same quantization onto a decode/prefill contract in which no input shape moves between steps, which removes the per-generated-token memory growth described in the known issue above. It requires a host that feeds position_ids as the absolute positions of the S new tokens only (host support in progress); a host that feeds the growing 0..N prefix will write the KV cache at the wrong offset. Host support exists and was gated on the E2B sibling (memory fix confirmed there: flat footprint, byte-identical guided output, but ~1.0 tok/s decode from the full-capacity cache read). This E4B bundle has not been run on-device; the same capacity cost applies. Treat as a working proof of the contract, not a deployable bundle.

Bundle	Bytes	Context	Functions	Producer	Created
stable/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64_stable_c16384	3,989,939,812	16384	main+prefill	coreai-core 1.0.0b2	20260818T004019Z

Contract. Both entrypoints take four inputs and two states, all statically shaped:

main    IN  input_ids Int32 1x1  | position_ids Int32 1x1  | ple_table Int8 V x (L*ld) | ple_scale Float32 V
prefill IN  input_ids Int32 1x64 | position_ids Int32 1x64 | (same statics)
        ST  keyCache / valueCache Float16 [slots, 1, n_kv, 16384, 512]
        OUT logits Float16 1 x S x 262144

position_ids carries the absolute position of each of the S tokens in the call, and position_ids[0,0] is also the cache slot the K/V for those tokens is written at. The attention mask (causality, unwritten slots, and the 512-token sliding window) is derived inside the graph from those positions, so there is no mask input to bind. The KV state's sequence extent is a literal 16,384 rather than a dynamic dimension, so a host that resolves its cache strategy from the state descriptor will allocate the full cache up front (0.50 GB for E2B, 1.61 GB for E4B) instead of growing it — that fixed cost is the trade for the unbounded growth it replaces.

The context ceiling here is encoded in the graph: a different window needs a different export, not a manifest edit.

Verification. Before conversion, the stable graph was checked against the shipped graph in eager torch on the same weights: greedy argmax identical on all tested decode steps (including positions past the 512-token sliding window), worst logits cosine 0.99999 (fp16), and the prefill path bit-identical in fp32. On-device behavior is unmeasured until host support lands.

ring/ — shape-stable decode, ring sliding caches

⚠️ EXPERIMENTAL — built and gated in torch, on-device measurements pending. A re-export of the same weights and the same quantization as stable/, onto the same host contract, with the KV layout reworked. No throughput or footprint numbers have been taken on a Mac yet; the structural facts below are read off the converted program, not inferred.

Bundle	Bytes	Context	Functions	Producer	Created
ring/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64_ring_c16384	3,989,979,750	16384	main+prefill	coreai-core 1.0.0b2	20260818T085331Z

What this is. stable/ removed the per-generated-token memory growth by making every input shape a literal, and it worked — but it gave every layer a full 16,384-slot KV cache and read all of it every step. ring/ keeps that contract byte-for-byte and changes what sits behind it:

• sliding layers read a ring buffer, 576 slots (the model's own 512-token sliding window plus one 64-token prefill chunk) instead of 16,384. Ring slot r holds the newest written position with pos % 576 == r, and the mask that selects the written, causal, in-window slots is derived inside the graph from the absolute positions.
• the grouped-query head expansion is folded into the query rather than materialised. The stock lowering broadcasts an index tensor to the full [1, heads, K, head_dim] and gathers K and V through it; reshaping the query so the head counts already match produces the same dot products with no index tensor and no gathered copies.
• both regions are packed into the same two states, so the host still binds exactly two KV states, positionally, at whatever literal extent the descriptor declares.

Contract — unchanged from stable/. Both entrypoints take four inputs and two states, all statically shaped:

main    IN  input_ids Int32 1x1  | position_ids Int32 1x1  | ple_table Int8 V x (L*ld) | ple_scale Float32 V
prefill IN  input_ids Int32 1x64 | position_ids Int32 1x64 | (same statics)
        ST  keyCache / valueCache Float16 [1, 1, n_kv, TOTAL, 512]
        OUT logits Float16 1 x S x 262144

position_ids carries the absolute position of each of the S tokens in the call, and position_ids[0,0] is the write position. The mask is derived in the graph, so there is no mask input to bind. The KV states have literal extents, so a host that resolves its cache strategy from the state descriptor allocates them up front rather than growing them.

Structural facts (dumped from the converted program):

	E2B	E4B
KV state shape	[1, 1, 1, 56064, 512] x2	[1, 1, 2, 77056, 512] x2
KV bytes, both states	114.8 MB	315.6 MB
same for stable/	503.3 MB	1.61 GB
cache slots read per decode step	56,064	77,056
same for stable/	245,760	393,216
sliding / full layers	12 x 576 + 3 x 16,384	20 x 576 + 4 x 16,384
dynamic dimensions anywhere	none	none

One new host precondition. A prefill call's first position must be a multiple of 64. The ring write is a fixed-width store at p0 % 576, and 576 is nine 64-token chunks, so a 64-aligned chunk can never straddle the wrap; an unaligned one would write past the end of its own ring region. Feeding whole 64-token chunks from a 64-aligned position — and the remainder one token at a time through main — was already what the contract asked for. stable/ tolerated an unaligned chunk; ring/ does not.

Unlike the ctx*/ folders, the context ceiling is encoded in the graph: a different window needs a different export, not a manifest edit.

ring3/ — per-layer-type SDPA form

EXPERIMENTAL — gated in torch, no on-device numbers yet. Same weights, same quantization, same host contract as ring/. One thing changes inside the graph: the grouped-query expansion is folded into the query only in the full-attention layers, and the sliding layers keep the stock gathered form.

Why the split. Folding the GQA expansion into the query removes the gathered copies and the broadcast index tensor, but it also turns G independent per-head matmuls into a single batch-1 matmul with G times the rows. That trade is worth taking when the key length is long (the full-attention layers read the whole capacity) and not worth taking when it is short (the sliding layers read a fixed 576-slot ring, where the operation is short enough that losing the per-head parallelism costs more than the bytes it saves). ring/ applied the fold everywhere; ring3/ applies it only where the key length is long.

Both forms compute the same dot products in the same order. The equality gate below is run against the shipped graph, not against ring/.

Contract: unchanged. Same four inputs in the same order, same two KV states in the same positional order, position_ids carrying the absolute position of each of the S tokens in the call, the mask derived inside the graph, the same requirement that a prefill call start at a multiple of 64. A host driving ring/ drives these with no change.

Equality. Gated in eager torch against the shipped graph before conversion: 1,600 prompt tokens as 25 chunks of 64 then 32 greedy decode steps, final position 1,631, so the sliding ring wraps about three times. fp32, 57 comparisons per run, 0 argmax mismatches on every bundle below. No dynamic dimension appears in either entrypoint of any of them.

Bundle	Context	KV state shape	KV bytes, both states	Sliding / full slots
ring3/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64_ring_c4096_gqafull	4,096	[1, 1, 2, 27904, 512] x2	114.3 MB	20 x 576 + 4 x 4,096

E4B runs 35 sliding and 7 full attention layers; in this bundle the 35 keep the gathered form and the 7 use the folded one. This is the first E4B bundle in the shape-stable family to be published since ring/; ring/'s E4B was never measured on a Mac.