Model Card for gemma-4-E4B-it (QAT, GGUF)

A GGUF repack of Google's quantization-aware-trained gemma-4-E4B-it build, packaged to run on device. It carries the text, vision and audio towers in one self-contained file: the tokenizer and chat template are embedded, so nothing ships alongside it and nothing calls out to a server at inference time.

This repository holds the repacked weights only, not training data or the original checkpoint.

Model Details

Model Description

gemma-4-E4B is a multimodal decoder with a per-layer-embedding (PLE) design: alongside the usual token embedding, each layer receives its own 256-wide input gathered from a large per-layer table, gated and projected into the hidden stream. Attention alternates five sliding-window layers to one full-attention layer, the two using different head widths and RoPE bases, and the upper eighteen layers share their neighbours' key/value state instead of computing their own. A vision encoder and a Conformer audio encoder project into the same embedding space.

The upstream checkpoint is quantization-aware TRAINED, not post-hoc quantized: Google trained it while simulating low-bit error, which is why its two-bit layers hold up where a post-hoc conversion of the same layers does not. E4B spends its bits differently from E2B: every feed-forward block stays at four bits, while the two embedding tables and the output projection drop to two.

Developed by: Google (base model); GGUF repack by leok7v
Model type: Multimodal (text, image, audio) causal language model
Language(s): the languages of the base model
License: Apache 2.0, inherited from the base model; see the Gemma 4 license
Repacked from model: google/gemma-4-E4B-it-qat-mobile-transformers

Model Sources

Repository (this build): https://huggingface.co/leok7v/gemma-4-e4b-it-qat
Base model: google/gemma-4-E4B-it-qat-mobile-transformers

Uses

Direct Use

On-device assistants and chat in offline or privacy-sensitive settings where prompts must not leave the device.

Out-of-Scope Use

Not for tasks requiring verified factual accuracy, and not a basis for legal, medical, financial, or safety decisions. Not intended for high-stakes automated decisions without human review.

Bias, Risks, and Limitations

A language model predicts plausible text; it does not verify facts and has no understanding of truth. Output can be confident and still be wrong, incomplete, or biased. The base model's biases and limitations carry over; see the base model card.

Recommendations

Treat every response critically and check important output against a trusted source.

How to Get Started with the Model

This file does not load in upstream llama.cpp. 111 of its tensors use Q2_0, a block type that exists in the PrismML/Gadeon ggml lineage and not in upstream ggml, so a tool that does not know Q2_0 will fail to read it. It is published for runtimes that carry that type.

The tokenizer (262144 tokens, 514906 merges, byte fallback and metaspace) and the chat template are embedded in the file, so a runtime reads both from the GGUF rather than from sibling files.

Technical Specifications

Model Architecture and Objective

Property	Value
Objective	Autoregressive next-token prediction
Hidden size	2560
Layers	42 (35 sliding-window + 7 full attention)
Feed-forward width	10240
Attention heads	8 query / 2 key-value (grouped-query)
Head width	256 sliding, 512 full
Sliding window	512
Shared-KV layers	18
Vocabulary	262144
Logit softcap	30
Vision encoder	16 layers, native resolution
Audio encoder	12 Conformer layers
Activations	fp32 / fp16

Weight format

3.795 GB, 1149 tensors, GGUF v3, 16384-byte tensor alignment.

Type	Tensors	Covers
Q4_0	270	attention, every layer's MLP, the audio tower's lconv input
Q2_0	111	both embedding tables, output projection, audio tower
Q8_0	196	vision tower, per-layer gate and projection
BF16	419	norms, layer scalars, the per-layer model projection
F32	153	the checkpoint's own unconverted modules

The integer codes transfer from the QAT checkpoint exactly rather than being re-quantized: its INT4 codebook and offset are Q4_0's, and its INT2 codes are Q2_0's under a negation. The only numerical difference from the upstream weights is a per-block fp32 to fp16 scale cast.

Citation

Please cite the upstream Gemma model. See the base model card at google/gemma-4-E4B-it-qat-mobile-transformers for the canonical reference and license.