visible-cx/gemma4-26b-a4b-CoreAI

Gemma 4 26B-A4B (MoE) — Core AI (.aimodel)

google/gemma-4-26B-A4B-it-qat-q4_0-unquantized converted to a Core AI .aimodel bundle for Apple silicon by visible-cx. This is a derivative artifact: Google's QAT-trained weights rounded onto int4 and re-expressed as a Core AI graph, with the 128-expert sparse branch of every layer lowered onto a Metal gather kernel.

It is a sparse mixture-of-experts model — 26.5B total parameters, roughly 4B active per token.

⚠️ UNQUALIFIED. This bundle has never produced a token, on any machine. The graph exports and the producer fingerprint passes; the block wiring is verified against the reference implementation (below). Nothing else is established.

⚠️ The 16 GB tier is excluded. Weights are ~17.6 GB resident against a 16 GB Mac's ~10.7 GB Metal working set. Minimum practical machine memory: 24–32 GB.

Provenance

Base checkpoint	google/gemma-4-26B-A4B-it-qat-q4_0-unquantized @ f1e06dc520982d9b9edd76859fdb7ab209449949
Checkpoint size	51,644,341,801 B, bf16, 1,013 tensors across 2 shards; ungated
Recipe	export_gemma4_moe_decode_pipelined.py int4lin --lin-sym --metal-sdpa --max-ctx 16384
Toolchain base	apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 + the coreai-model-zoo python overlay, plus three new modules (below)
Toolchain	coreai-torch 0.4.1, coreai-core 1.0.0b2, coreai-opt 0.2.1, torch 2.9.0
Producer fingerprint	coreai-core 1.0.0b2 — verified on the inner asset metadata.json
Asset creation date	20260817T215629Z
Vocab	262,144
Export functions	main (S=1 decode) only

"QAT-unquantized" means QAT-trained, stored bf16; the int4 rounding happens at export onto the ggml q4_0 grid the training already targeted. There is no separate pre-quantised int4 artifact to prefer.

To reproduce this conversion you need a ≥128 GB host: the export peaks at 102.25 GB RSS (VmHWM) on Linux x86_64.

Architecture

	26B-A4B	31B	E4B
hidden_size	2816	5376	2560
layers	30 (25 sliding / 5 full)	60	42
dense MLP intermediate_size	2112	21504	10240
num_experts / top_k	128 / 8	—	—
moe_intermediate_size	704	—	—
attention heads	16	32	8
KV heads (sliding / full)	8 / 2	16 / 4	2 / —
head_dim / global_head_dim	256 / 512	256 / 512	256 / 512
sliding_window	1024	1024	512
Per-Layer Embeddings	none	none	256
KV-shared layers	0	0	18

Same attention family as the dense 12B/31B — same dual head_dim, same attention_k_eq_v (full layers carry no v_proj), same dual RoPE, same softcap. There are no Per-Layer Embeddings and therefore no gather-table sidecar in this repo.

Every layer carries a dense MLP and a sparse branch, in parallel:

residual = x                                              # post-attention hidden
h  = mlp(pre_feedforward_layernorm(x))                    # dense branch
h1 = post_feedforward_layernorm_1(h)
w, idx = router(residual)                                 # routes on the RAW residual
h2 = experts(pre_feedforward_layernorm_2(residual), idx)  # sparse branch
h2 = post_feedforward_layernorm_2(sum_k w_k * h2_k)
x  = residual + post_feedforward_layernorm(h1 + h2)
x  = x * layer_scalar

Two details are load-bearing: the router reads the un-normalised residual, and the experts are GELU-gated (gelu_pytorch_tanh), where the SwitchGLU primitive's default is SiLU. The router itself: scale-free RMSNorm → × scale × hidden**-0.5 → linear to 128 → fp32 softmax over all experts → top-8 → renormalise to sum 1 → multiply by a learned per_expert_scale gathered at the selected ids.

Compression recipe

Weights	Scheme	Rationale
Routed experts (128/layer, 22.84 G params)	affine int4 (aff4), per-(output row, K-block-32) scale + bias, MSE-optimal clip over 6 candidates	see below
Attention, dense MLP, untied LM head	linear int4 per-block-32, plain absmax (--lin-sym)	the ggml q4_0 grid the QAT checkpoint was trained on — the 12B/31B recipe verbatim
Router (proj, scale, per_expert_scale)	fp16, excluded by name	routers are the quantization-sensitive part of an MoE and the whole set is 0.02 GB
embed_tokens (in-graph)	fp16	the gather stays exact; the head is untied and quantized separately

sym8 on the experts would put them at 22.8 GB before anything else, so int4 is not optional. aff4 was chosen over a k-means palette (km4) because this checkpoint was QAT-trained on per-block-32 q4_0 (w = (q-8)·d, 16 uniform levels), and an affine int4 block-32 grid represents that exactly (scale = d, bias = -8d). A palette fits one 16-entry codebook per 32 output rows across the whole K axis and structurally cannot follow per-block QAT scales. The price is aux bytes: aff4 stores an fp16 scale and bias per 32-element K block, i.e. 5 bits/param effective against km4's ~4.03 — about 3 GB of the bundle.

The down projection's K of 704 is padded to 768 by the kernel's automatic _kpad256 treatment — 64 zero columns, ~0.5% wasted expert bytes. gate/up need no padding.

The gather kernel is load-bearing. Without MetalSwitchGLU, the sparse branch lowers to a dense matmul that reads all 128 experts every token — a 16× over-read.

New source modules

Three files were authored for this port; nothing in the shared toolchain was modified:

File	Contents
coreai_models/models/macos/gemma4_moe_text.py	Gemma4MoeConfig, Gemma4MoeRouter, GeluGLU, Gemma4MoeDecoderLayer, Gemma4MoeForCausalLM, and a loader that splits the checkpoint's fused gate_up_proj [E, 2*704, 2816] into the SwitchGLU's separate stacks
coreai_models/models/macos/gemma4_moe_pipelined.py	Gemma4MoePipelinedForCausalLM — subclasses the dense pipelined core and overrides one method; the attention half is inherited unchanged
coreai-model-zoo/conversion/export_gemma4_moe_decode_pipelined.py	the recipe

Contents

Path	Bytes	Files	Manifest context	Functions
gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8	17,580,059,414	8	16384	main (decode only)

File	Bytes
…aimodel/main.mlirb	17,547,866,346
…aimodel/main.hash	32
…aimodel/metadata.json	105
metadata.json (bundle manifest)	689
tokenizer/tokenizer.json	32,169,626
tokenizer/chat_template.jinja	18,683
tokenizer/tokenizer_config.json	3,730
tokenizer/generation_config.json	203

Manifest context is 16384. --max-ctx sets language.max_context_length and nothing else; lower it if the machine cannot afford the KV — that is a metadata edit, not a re-export.

Stop token: eos_token = "<turn|>" (id 106), the turn terminator Gemma 4 emits, applied by the export script itself. A host that stops on the raw upstream <eos> will overrun every reply.

Decode only, and a prefill variant is not deliverable by a flag

There is no prefill function. Two independent kernel blockers:

1. --metal-sdpa's flash-decode kernel is structurally q=1 with no causal mask. The full attention layers need that kernel for the same scratch-heap reason the dense 12B/31B do.
2. MetalSwitchGLU.forward is itself decode-only — it asserts a token batch of 1 and expands the single activation row across the routed slots. A BatchedMetalSwitchGLU (sort-by-expert grouped GEMM) exists for q>1, so this half is reachable, but it has never been exported for this model.

A prefill variant therefore needs a new flash-prefill Metal kernel plus the first export of the batched MoE kernel, plus a numerics gate for each. Prompts are prefilled one token at a time.

Requirements

• Apple silicon Mac with ≥24–32 GB unified memory, Core AI runtime.

• Engine contract: 2 inputs (input_ids, position_ids) → logits, one growing KV pair, no static inputs and no per-step mask.

• States: keyCache / valueCache Float16, 30 × 1 × 8 × ? × 512 — one growing pair, 30 slots (no KV sharing). The dynamic sequence dim means the runtime resolves GrowingKVCache, not a static allocation at the manifest maximum.

• KV cost: 491,520 bytes per token of context (fp16):

  Context	KV
  4096	2.01 GB
  8192	4.03 GB
  16384	8.05 GB
  32768	16.1 GB

• Resident weights, ~17.6 GB:

  	bytes
  Routed experts, aff4 (incl. _kpad256 zero columns)	~14.7 GB
  Attention + dense MLP + untied head, int4 block-32	~1.4 GB
  embed_tokens, fp16 in-graph	~1.5 GB
  Router, fp16	~0.02 GB
  Total	~17.6 GB
  + KV at 4096	~19.6 GB
  + KV at 16384	~25.7 GB

  Tier	Verdict
  16 GB	no — weights alone are ~1.6× the entire ~10.7 GB Metal working set
  24–32 GB	yes at ctx ≤ 8192 (~21.6 GB); 16384 is tight
  64 GB	yes, at any context

  This table is arithmetic from state shapes and weight formats — a prediction to check, not a measured tier claim. The residency mechanism for MoE bundles on macOS (mmap'd vs wired expert slabs) has not been measured.

  25 of the 30 layers have head_dim 256 zero-padded to 512, and all 30 slots grow linearly even though 25 of them only ever attend a 1024-token window. A ring-buffered sliding cache would be roughly 5× cheaper per token; that is model authoring, not a flag.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.

Performance

No measurement exists, on any machine. Every figure in this card is either a byte count, an export-host measurement, or derived from state shapes and weight formats. No token has been generated from this bundle.

The intended argument for this model is throughput at comparable size against the dense 31B — 17.6 GB vs 20 GB, with ~4B active parameters per token read through a gather kernel — and quality per token against the 12B at ~2.1× the footprint. The nearest measured analogue is LFM2.5-8B-A1B at 140 tok/s through its gather kernel versus 39 tok/s for the same weights read as a dense over-read. Neither claim is measured for this bundle.

Status

Artifact	Status
gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8	UNQUALIFIED — never executed on target hardware. Established: the checkpoint is the QAT source, the graph exports, the producer fingerprint passes, and the MoE block wiring matches the reference implementation to 3.5e-7 relative error on real layer-0 weights (below).

A qualifying run would need, in priority order:

1. A decode oracle — generate and compare against an fp32 HF reference. This is the gate that catches an int4 or kernel-level error which the authoring gate below cannot see.
2. Isolate the two Metal kernels. gather_qmm (the aff4 variant) and the flash-decode SDPA have never been in the same graph, on any model. If decode produces garbage, re-export without --metal-sdpa first — it is an optimisation, the MoE kernel is not.
3. Measure residency, both dirty footprint and maximum resident including mmap. No MoE bundle has been measured this way.
4. Compare tok/s against the dense 12B and 31B on the same machine.
5. Router sanity at depth. 128 experts / top-8 with a learned per_expert_scale is far sparser routing than the LFM2.5-8B-A1B's 32/top-4. Check expert-utilisation spread on a real prompt set before trusting long-form output.

Verification

MoE block wiring verified against the reference implementation. transformers 4.57.6 (the pinned export environment) has no gemma4, so the gate transcribes Gemma4TextDecoderLayer.forward (the MoE-block half), Gemma4TextRouter.forward and Gemma4TextExperts.forward from upstream and runs both implementations on the real layer-0 weights in fp32:

T=1: max|dFFN|=7.629e-05  rel=3.453e-07  |  same top-8 ids: True  max|dw|=0.000e+00
T=4: max|dFFN|=9.537e-05  rel=2.735e-07  |  same top-8 ids: True  max|dw|=0.000e+00

Identical expert selection, bit-identical routing weights, ~3.5e-7 relative error on the block output. This gates the authoring — the wiring, the fused-weight split, the activation choice, the norm placement, the router maths. It does not gate the exported graph, the int4 rounding, or either Metal kernel. The attention half was not re-gated: it is the dense Attention class already used in the 12B and 31B bundles.

License

The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma model and the Gemma terms apply downstream, which is why this repo declares license: gemma. Use of these weights is subject to the Gemma Terms of Use, the Gemma Prohibited Use Policy and the Gemma 4 license. Those obligations travel with any redistribution of this bundle. The contribution here is the port and the recipe, not the weights.