gemma4-e4b-w4a16-mtp-vLLM

W4A16 (GPTQ, full-precision lm_head) quantization of gemma-4-E4B-it, calibrated on our own CLI/tool-call logs instead of a generic text corpus, paired with the Gemma-4 MTP drafter for speculative decoding. Serves at ~130 tok/s single-stream decode on one 16 GB consumer GPU (RTX 4060 Ti) with correct tool calls and exact rare-token output — no vocabulary pruning, no exotic patched inference engine.

This is the exact checkpoint + launch config running our own DeckDoctor prod chat deployment. Everything needed to reproduce it — calibration provenance, drafter, launch script, pinned versions — is in this repo.

🧊 What this is

Base model	google/gemma-4-E4B-it
Method	GPTQ W4A16, group size 128, static act-order (via llm-compressor)
lm_head	full precision (not quantized, not pruned) — exact rare tokens
Calibration	our own CLI + tool-call logs (not WikiText/C4) at ctx 8192, 32 samples
Serving engine	stock vLLM 0.26.0 (no custom kernels/patches required)
Speculative decoding	Gemma-4 MTP drafter, num_speculative_tokens=7
Tool calling	✅ --tool-call-parser gemma4, verified correct arguments
Context	up to 131,072 tokens
VRAM	fits + serves on a single 16 GB GPU (RTX 4060 Ti)

⚡ Why this exists

Stock W4A16 gemma-4-E4B decodes at ~68 tok/s on a 4060 Ti — usable, but slow enough that an agentic tool-calling loop feels laggy. The gemma-challenge community demonstrated 500+ tok/s is possible on datacenter GPUs, but their fastest recipe pins a nightly vLLM fork, bakes vocabulary pruning into the weights (breaks exact rare-token output, e.g. "Pineple" instead of "PINEAPPLE"), and has no tool-calling support at all.

This checkpoint takes the middle path: keep the full vocabulary and stock vLLM (so it's a drop-in vllm serve on any machine with the pinned versions below), and get the speed back from a fine-tuned Multi-Token-Prediction drafter instead of kernel/vocab surgery. Net result: ~1.9x the tok/s of the un-accelerated W4A16 checkpoint, keeping exact tokens and tool calls.

📊 Benchmark

Single RTX 4060 Ti (16 GB), --max-model-len 131072, greedy decode, single stream unless noted. Measured 2026-08-09 / 2026-08-27.

Config	Decode tok/s	lm_head	Tool calls	Engine
This checkpoint, no drafter	~68	full	✅	stock vLLM 0.26.0
This checkpoint + MTP drafter (this repo)	~130	full	✅	stock vLLM 0.26.0
Same weights, patched osoi5-v8 engine + fp8 KV cache	~126–134	full	❌ (engine gap)	pinned vLLM nightly + custom kernels
gemma-challenge osoi5-v0 reference (pruned head, A10G)	535.9	pruned to 16,384 rows	❌	pinned vLLM nightly + custom kernels

The patched-engine row is included for context, not because it's what this repo ships: it needs a pinned vLLM nightly and several out-of-tree kernel patches for a ~3% speed difference and no tool-calling support, so we don't consider it worth the portability cost. The MTP-drafter + stock-vLLM config above is what this repo reproduces.

With --max-num-seqs raised and fp8 KV cache, the same checkpoint also serves several concurrent long-context conversations from one GPU (KV cache capacity scales roughly linearly with --max-num-seqs); see deploy/launch.sh for the knobs.

🔬 How it was made

Quantized with our internal tool (quant_tuner.vllm_export.w4a16, uses llm-compressor's GPTQModifier under the hood) against google/gemma-4-E4B-it:

• Scheme: W4A16, group size 128, static activation ordering, 1% dampening, Hessian offload.
• Ignored (kept full precision): lm_head, vision tower, audio tower, vision/ audio/token embeddings, per-layer projections, multimodal projector.
• Calibration corpus: 32 samples, budget 262,144 tokens, drawn from our own CLI and tool-call transcripts at ctx 8192 — not a generic text corpus. This matters for tool-calling fidelity specifically, since GPTQ's Hessian estimate is only as representative as its calibration set, and generic corpora under-represent JSON-shaped tool-call text.
• Provenance: exact recipe (checkpoint/recipe.yaml) and corpus manifest with a sha256 of the calibration file (checkpoint/quant_tuner_ptq.json) are included in this repo for full reproducibility.

The MTP drafter

drafter/ is Gemma-4's own assistant head (Gemma4AssistantForCausalLM) — a small model that predicts several future tokens from the target model's last hidden state. vLLM runs it as a speculative decoding drafter (method: "mtp"); the trunk checkpoint is never modified, and this exact drafter pairs with any W4A16/BF16 build of the same base model. It is the persistent, unquantized Google-released assistant checkpoint — not a vocabulary-remapped drafter fit to a pruned head, which is why it composes cleanly with the full-lm_head trunk above.

🚀 Usage

Download both checkpoint/ and drafter/ from this repo (they're separate directories on purpose — the drafter is reused across quant levels of the base model, so don't merge them).

pip install -r deploy/requirements.txt   # vllm==0.26.0 + pinned deps
MODEL_DIR=./checkpoint DRAFTER_DIR=./drafter ./deploy/launch.sh
# serves an OpenAI-compatible API on :8000

Or invoke vllm serve directly:

vllm serve ./checkpoint \
  --served-model-name gemma-4-e4b-w4a16-mtp \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 1 \
  --speculative-config '{"method":"mtp","model":"./drafter","num_speculative_tokens":7}' \
  --enable-auto-tool-choice --tool-call-parser gemma4 \
  --reasoning-parser gemma4

OpenAI-compatible API (Python)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="gemma-4-e4b-w4a16-mtp",
    messages=[{"role": "user", "content": "What's the boiling point of water at 2000m altitude?"}],
)
print(resp.choices[0].message.content)

Tool calling

resp = client.chat.completions.create(
    model="gemma-4-e4b-w4a16-mtp",
    messages=[{"role": "user", "content": "What's the weather in Tucson?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "parameters": {"type": "object", "properties": {"city": {"type": "string"}}},
        },
    }],
)
print(resp.choices[0].message.tool_calls)

Reproducing the benchmark

Single-stream: send one request at a time and time usage.completion_tokens / wall_clock_seconds on the /v1/chat/completions endpoint above with max_tokens large enough to see steady-state decode (short prompts finish before the pipeline warms up and understate tok/s).

⚠️ Gotchas

• GLOO_SOCKET_IFNAME=lo / VLLM_HOST_IP=127.0.0.1 (baked into deploy/launch.sh) — without this, vLLM's internal process group can bind a VPN/tailnet interface and hang or fail to start on a machine with one configured.
• --speculative-config's model path must be reachable at server start; vLLM does not fetch it from the Hub itself, so download drafter/ alongside checkpoint/ even though they're separate directories in this repo.
• This checkpoint has no output-vocabulary pruning, so if you've read about the gemma-challenge speed numbers and were expecting garbled rare tokens as the price of speed — that trade isn't made here.

🪪 License & attribution

• Base model: google/gemma-4-E4B-it, used under the Gemma Terms of Use.
• MTP drafter: Google's released Gemma-4 assistant head, same license.
• This repo's quantized weights are a derivative work distributed under the same Gemma Terms of Use.