visible-cx/Gemma-4-31B-CoreAI

Gemma 4 31B — Core AI (.aimodel)

google/gemma-4-31B-it-qat-q4_0-unquantized converted to a Core AI .aimodel bundle for Apple silicon by visible-cx. This is a derivative artifact: Google's QAT-trained weights rounded onto the int4 grid they were trained for and re-expressed as a Core AI graph with a custom Metal flash-decode attention kernel.

⚠️ UNQUALIFIED. This bundle has never been executed. It converted cleanly and its producer fingerprint passes; nothing beyond that is established. Read the Status section before assuming anything about its behaviour.

⚠️ Expected known issue — long unguided generations (unverified on this model). On the sibling E2B/E4B exports, the Core AI runtime retains one compiled specialization per sequence-length signature, costing roughly 80 MB of GPU allocations per generated token until process exit; the decode graphs in this repository share the length-dependent structure that causes it. Until a shape-stable re-export exists for this model, treat these bundles as suitable for guided/constrained and short generations; budget memory for long free-form runs accordingly. Measured details: the gemma-4-E2B-CoreAI card's known-issue section.

Provenance

Base checkpoint	google/gemma-4-31B-it-qat-q4_0-unquantized — 31.3B bf16 parameters, two shards, 62,578,507,874 B, ungated
Recipe	export_gemma4_12b_decode_pipelined.py int4lin --lin-sym --metal-sdpa — zoo recipe gemma-4-31b, status = "verified"
Toolchain base	apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 + the coreai-model-zoo python overlay
Toolchain	coreai-torch 0.4.1, coreai-core 1.0.0b2, coreai-opt 0.2.1, torch 2.9.0
Producer fingerprint	coreai-core 1.0.0b2 — verified on the inner asset metadata.json
Asset creation date	20260817T112551Z
Weight format	int4, per-block-32, symmetric absmax (--lin-sym) — the ggml q4_0 grid the QAT checkpoint was trained on
Vocab	262,144
Export functions	main (S=1 decode) only

"QAT-unquantized" means QAT-trained, stored at full width. The int4 rounding happens at export, onto the grid the training already targeted — this is not post-training quantization of a full-precision model.

To reproduce this conversion you need a ≥192 GB host. The export peaks at 148.9 GB RSS during the in-RAM int4 quantization pass (841 s wall, Linux x86_64), falling back to 60–90 GB for the export itself. A 128 GB box under a 75% kill rule aborts at 96 GB.

--metal-sdpa is mandatory here, not an optimisation

_msdpa_g8 in the bundle name is the custom Metal flash-decode SDPA kernel at its default split factor 8. The 31B's full-attention layers have a 32 heads × 512 fp16 query tensor that overflows MPSGraph's GPU decode scratch heap; with the stock SDPA the model crashes at the first token.

Contents

Path	Bytes	Files	Manifest context	Functions
gpu-pipelined/gemma4_31b_qat_decode_int4linsym_msdpa_g8	20,122,108,533	7	4096	main (decode only)

File	Bytes
…aimodel/main.mlirb	20,089,915,701
…aimodel/main.hash	32
…aimodel/metadata.json	105
metadata.json (bundle manifest)	656
tokenizer/tokenizer.json	32,169,626
tokenizer/tokenizer_config.json	3,730
tokenizer/chat_template.jinja	18,683

Decode only, and it cannot be otherwise on the current stack. There is no prefill function and none can be exported with --metal-sdpa: that kernel opens with b, h, s, hd = query.shape # s == 1 and reshapes the query to [h, hd], which is only valid at S=1, and it computes no causal mask because a decode step's single query attends the whole grown cache. A chunked prefill needs a second Metal kernel — a genuine flash-prefill with a query axis and an intra-chunk causal mask — plus its own numerics gate. Prompts are prefilled one token at a time.

There is no ctx8192/ctx16384 folder. --max-ctx sets language.max_context_length in the manifest and nothing else, so a wider window is a manifest edit rather than a conversion. Whether it is affordable is a separate question; see the envelope below.

Stop token: eos_token = "<turn|>" (id 106), the turn terminator Gemma 4 emits. A host that stops on the raw upstream <eos> instead will overrun every reply.

Requirements

• Apple silicon Mac, Core AI runtime. iPhone and iPad are unsupported for this model.

• Engine contract: 2 inputs (input_ids, position_ids) → logits, with one growing KV pair. No static inputs and no per-step mask input.

• States: keyCache / valueCache Float16, 60 × 1 × 16 × ? × 512. The sequence dim is dynamic, so the runtime resolves a GrowingKVCache (initial 256, doubling), not a static allocation at the manifest maximum.

• KV cost: 1,966,080 bytes per token of context (fp16) — 8.05 GB at 4096, 16.1 GB at 8192, 32.2 GB at 16384. This is the steepest KV curve of any bundle in this org.

• Memory envelope:

  Weights resident (main.mlirb)	20.09 GB
  + KV at 4096	~28.1 GB
  + KV at 8192	~36.2 GB

  On a 16 GB Mac, Metal's recommendedMaxWorkingSetSize is ≈ 10.7 GB, so the weights alone exceed that machine's entire working set by roughly 2× before a single token of KV. Minimum practical machine memory: 64 GB. A 32 GB machine is not a supported target for this bundle at any context.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.

Performance

No measurement has been made on Apple silicon. No token has been generated from this bundle.

The only figures that exist are the model zoo's, on hardware not used here:

Source	Machine	Decode	Prefill
Core AI model zoo (upstream, same recipe)	M4 Max	17.2 tok/s	22.1 tok/s

Read published Gemma 4 Q4 figures of 40–50 tok/s with care: those use Gemma 4's MTP speculative drafters, which Core AI does not implement. Compare plain-decode to plain-decode.

Status

Artifact	Status
gpu-pipelined/gemma4_31b_qat_decode_int4linsym_msdpa_g8	UNQUALIFIED — converted and fingerprinted, never executed. What is established: the checkpoint is the QAT source, the recipe is the zoo's verified 12B recipe applied to a larger checkpoint id, the export completed, and the producer fingerprint passes. No oracle, no parity gate, no throughput, no residency measurement, no guided-decoding check.

A qualifying run would need, in order: a decode oracle against an fp32 reference (the gate that catches int4 or kernel-level error, which no fingerprint can see); a residency measurement covering both dirty footprint and maximum resident including mmap; and grammar-constrained decoding against a real schema, which has never been exercised on any msdpa bundle.

License

The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma model and the Gemma terms apply downstream, which is why this repo declares license: gemma. Use of these weights is subject to the Gemma Terms of Use, the Gemma Prohibited Use Policy and the Gemma 4 license. Those obligations travel with any redistribution of this bundle. The contribution here is the conversion, not the weights.

hybrid-pf64/ bundles

EXPERIMENTAL — not qualified. These bundles have not passed a Mac-side oracle/parity gate or a device benchmark. Do not route production traffic to them until they have.

What this is. The first Gemma 4 31B bundle with a prefill entrypoint. main (S=1) runs the full-attention layers on the custom flash-decode Metal kernel gemma4_dense_full_sdpa_occ8 — the same kernel the zoo's published _msdpa_g8 decode bundles carry, unmodified. prefill (S=64) runs those same layers on the MPSGraph SDPA composite, which has a query axis and applies the intra-chunk causal mask. Sliding layers use the composite in both entrypoints, as they always have. One copy of the weights, one growing KV pair, and the 2-input engine contract (input_ids, position_ids) that CoreAISequentialEngine requires — verified by reading the graph bytecode: the declared symbols are input_ids, position_ids, keyCache, valueCache, logits, prefill, and the kernel. There is no mask input.

Why it is built this way. Two catalog facts said the 31B could have decode or prefill but not both. (1) MetalDenseFullSDPA.forward reshapes [b,h,s,hd] -> [h,hd], valid only at s == 1, and computes no causal mask, so the kernel cannot serve a prefill chunk. (2) Without the kernel the 31B full layers put a [1,32,1,512] fp16 Q (32 KB) through MPSGraph SDPA, whose lowering overflows the ~208 KB decode scratch heap — "crashes at the first token" (apple/coreai-models#27; the 12B report was the same crash at half the size). Both are true, and the dichotomy still does not follow: the SDPA implementation is a property of the entrypoint, not of the bundle. main and prefill are two separate torch.export traces over one set of weights, each at a static query width, so a wrapper that dispatches on query.shape[2] is resolved at trace time and each entrypoint gets the implementation that is correct for it.

Measured on the conversion box (Linux x86_64, no GPU). Export wall 836 s; peak RSS 147.3 GB (the decode-only msdpa build measured 148.9 GB, so the second trace costs no extra peak); bundle 20,123,256,366 B; producer coreai-core 1.0.0b2; function_map {"main": ["main", "prefill"]}; ctx 4096. Trace accounting: 10 full layers took the metal branch at S=1 and the composite branch at S=64, 10/10 both ways — a bundle where either count is 0 fails the export.

Numerics gate (box-side, gate_hybrid_sdpa.json). The kernel's own torch reference vs the composite at this model's exact full-layer decode shapes (32 query heads, 4 global KV heads replicated across 16 cache slots, head_dim 512): max abs 1.98e-2 at S=137 and 1.17e-2 at S=1024, which is the composite's own fp16 error — against an fp32 ground truth the kernel is the more accurate of the two (max abs 9.7e-4, PSNR 75.4 dB, vs the composite's 53-60 dB). The 12B shape (1 global KV head, 8 slots) is the control and gives the same figures. This gates the block-GQA mapping and the scale=1.0 convention, which the 31B exercises for the first time — the 12B single global head makes that mapping degenerate. It does not gate the Metal source (no GPU on the box) and does not replace a Mac oracle.

Tier: 64 GB. Weights ~19.7 GB, KV 1,966,080 B/token (60 slots x 16 KV x 512 x 2 x fp16) = 8.05 GB at the manifest 4096, so ~28 GB resident at full context against a 64 GB Mac ~49 GB working-set budget. Not a 32 GB artifact at this context and not an iPhone/iPad one at any.

EXPERIMENTAL — the three things a Mac must answer, in order.

1. Does main load and decode? This is the bundle whole reason to exist: the full layers no longer touch MPSGraph SDPA, so the scratch-heap crash should not occur. If it still crashes, the crash was never about the SDPA lowering and the catalog account of apple/coreai-models#27 is wrong.
2. Does prefill survive 31B dims? Unknown and unknowable here. The documented crash is on the decode path; an S=64 chunk allocates a different, larger shape through the same MPSGraph lowering. Predicted outcome: it works (the 12B dense2in-pf64 rides the same composite at S=64), but if it crashes, the honest fix is to drop the prefill function, not the kernel — decode is the leg that must survive.
3. Chunk parity. A prompt walked in 64-token chunks must give the same logits as the same prompt walked token by token. The two entrypoints use different attention implementations, so this is a real question here in a way it is not for a single-kernel bundle. The box-side gate above says the two agree to fp16 noise at S=1; nothing has checked S=64.

Nothing already published was touched. The pinned gpu-pipelined/…msdpa_g8 decode bundle is unchanged and remains the only 31B artifact with a Mac measurement behind it.

Bundle	Source checkpoint	Context	Functions	Bytes	Producer	Created
hybrid-pf64/gemma4_31b_qat_decode_int4linsym_msdpa_g8_pf64	google/gemma-4-31B-it-qat-q4_0-unquantized	4096	main+prefill	20,123,256,366	coreai-core 1.0.0b2	20260818T132515Z