visible-cx/Gemma-4-12B-CoreAI

Gemma 4 12B — Core AI (.aimodel)

google/gemma-4-12B-it-qat-q4_0-unquantized converted to Core AI .aimodel bundles for Apple silicon by visible-cx. These are derivative artifacts: Google's QAT-trained weights rounded onto the int4 (or int8) grid and re-expressed as a Core AI graph. Gemma 4 12B is a dense model — no Per-Layer Embeddings and no gather-table sidecar, unlike the E2B and E4B bundles.

Every bundle here is a dense2in* build: exported without --metal-sdpa, so the graph takes 2 inputs (input_ids, position_ids) and no per-step mask. That is the contract both engines accept, and it is what makes an S=64 prefill entrypoint exportable.

⚠️ Expected known issue — long unguided generations (unverified on this model). On the sibling E2B/E4B exports, the Core AI runtime retains one compiled specialization per sequence-length signature, costing roughly 80 MB of GPU allocations per generated token until process exit; the decode graphs in this repository share the length-dependent structure that causes it. Until a shape-stable re-export exists for this model, treat these bundles as suitable for guided/constrained and short generations; budget memory for long free-form runs accordingly. Measured details: the gemma-4-E2B-CoreAI card's known-issue section.

Provenance

Base checkpoint	google/gemma-4-12B-it-qat-q4_0-unquantized (ungated)
Zoo recipe	gemma-4-12b, status = "verified" — the dense recipe
Recipe (dense2in)	that recipe with int4lin --lin-sym and no --metal-sdpa, --max-ctx 4096
Recipe (dense2in-pf64)	as dense2in, multifunction, via export_gemma4_dense_pf.py (supplies the trace_query knob on build_export_spec; no model code changed)
Recipe (dense2in-int8)	int8lin (symmetric-with-clipping, per-block-32), multifunction, same 2-input dense graph
Toolchain base	apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 + the coreai-model-zoo python overlay
Toolchain	coreai-torch 0.4.1, coreai-core 1.0.0b2, coreai-opt 0.2.1, torch 2.9.0
Producer fingerprint	coreai-core 1.0.0b2 on every inner asset metadata.json
Weight format	int4 per-block-32 symmetric absmax (--lin-sym) — the ggml q4_0 grid the QAT checkpoint was trained on; or int8 per-block-32 for dense2in-int8
Vocab	262,144
Export functions	main (S=1 decode); prefill (S=64) additionally in the two pf64 bundles

"QAT-unquantized" means QAT-trained, stored full width; the int4 rounding happens at export onto the grid training already targeted. That is why int4 is the primary recipe rather than a quality compromise — the int8 variant is an anchor, not the default.

Contents

Path	Bytes	Weights	Context	Functions	Engine inputs
dense2in/gemma4_12b_qat_decode_int4linsym	8,746,180,229	int4	4096	main	2
dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64	8,747,078,381	int4	4096	main + prefill	2
dense2in-int8/gemma4_12b_qat_decode_int8lin_pf64	14,700,368,629	int8	16384	main + prefill	2

Each folder holds <name>.aimodel/ (main.mlirb — 8.71 GB int4, 14.67 GB int8 — plus main.hash and asset metadata.json), a bundle-level metadata.json, and tokenizer/ (tokenizer.json 32,169,626 B, tokenizer_config.json 3,090 B, chat_template.jinja 18,683 B, and generation_config.json in the two pf64 folders).

dense2in-pf64's weights are bit-identical to dense2in's; the multifunction build adds an S=64 prefill entrypoint for under a megabyte, because the exporter deduplicates weights across entrypoints.

Manifest context is metadata only. --max-ctx sets language.max_context_length and nothing else, so the dense2in-int8 bundle's 16384 can be lowered and the int4 bundles' 4096 raised without a re-export. Whether the result fits in memory is a separate question; see the envelope below.

Stop token: all three bundles declare eos_token = "<turn|>" (id 106), the turn terminator Gemma 4 emits. A host that stops on the raw upstream <eos> instead will overrun every reply.

Requirements

• Apple silicon Mac, Core AI runtime. Mac-only; this does not fit an iPhone or iPad.

• Engine contract: 2 inputs — input_ids [1,1], position_ids [1,S] → logits, with one growing KV pair and no per-step mask. That is the contract both engines accept, and the logits-capable sequential engine is what grammar-constrained decoding needs.

• States: keyCache / valueCache Float16, 48 × 1 × 8 × ? × 512. Dynamic sequence dim → GrowingKVCache (initial 256, doubling).

• KV cost: 786,432 bytes per token (fp16) — 3.22 GB at 4096, 6.44 GB at 8192, 12.9 GB at 16384. This dominates the envelope.

• Memory envelope:

  	int4	int8
  Weights (main.mlirb)	8.71 GB	14.67 GB
  + KV at 4096	~11.9 GB	~17.9 GB
  + KV at 16384	~21.6 GB	~27.6 GB

  Against a 16 GB Mac's Metal recommendedMaxWorkingSetSize of ≈ 10.7 GB, the int4 bundle's affordable context works out at roughly 2,500 tokens — the 12B does not comfortably fit a 16 GB machine even at the 4096 its manifest declares, and the int8 bundle does not fit it at all. Minimum practical machine memory: 32 GB for int4, and 32 GB for int8 with the manifest context lowered from 16384 unless the machine is larger.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.

Performance

No measurement has been made on Apple silicon for any bundle in this repo. These bundles exceed the memory tier of the machine used for the measurements published on the smaller models in this org, so no comparable figure exists.

The only figure available is upstream's, on hardware not used here: approximately 33 tok/s decode on an M4 Max for the int4 12B. Read published Gemma 4 Q4 figures of 40–50 tok/s with care — those use Gemma 4's MTP speculative drafters, which Core AI does not implement. Compare plain-decode to plain-decode.

Status

Artifact	Status
dense2in/gemma4_12b_qat_decode_int4linsym	EXPERIMENTAL — built and fingerprinted, unmeasured. Gate: a Mac-side decode oracle against an fp32 reference, then a guided-decoding run on a real schema.
dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64	EXPERIMENTAL — built and fingerprinted, unmeasured. Same weights as dense2in plus the S=64 prefill function, which was worth 7.7× on time-to-first-token when measured on the E2B sibling. Same gate.
dense2in-int8/gemma4_12b_qat_decode_int8lin_pf64	EXPERIMENTAL — built and fingerprinted, unmeasured. Same gate, plus a quality comparison against int4; it exists as an anchor for the case where int4 phrasing quality fails qualification, at 1.7× the resident weights.

No numerics gate has been run on any bundle in this repo. The producer fingerprint proves the toolchain, not the numerics. dense2in* in particular changes the attention path relative to the recipe upstream marked verified, so its oracle is not optional before production use.

License

The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma model and the Gemma terms apply downstream, which is why this repo declares license: gemma. Use is subject to the Gemma Terms of Use, the Gemma Prohibited Use Policy and the Gemma 4 license. Those obligations travel with any redistribution of these bundles. The contribution here is the conversion, not the weights.