mando2222/gemma-4-26b-a4b-it-QB2

Gemma-4-26B-A4B-it — Tenstorrent Blackhole (vLLM bundle)

A tt-kernel vLLM bundle that runs google/gemma-4-26B-A4B-it — a Mixture-of-Experts model (128 experts, top-8, ~4B active params/token) — on 2× Tenstorrent Blackhole p300c boards (4 ASICs ≈ 4× p150a) via the Tenstorrent vLLM plugin. Ships no kernel cache and no weights — only the vLLM registration, the bounded sliding-window KV adapter, and a launch recipe; weights are pulled from the base repo and kernels JIT-compile at first-run warmup.

Serve

tt-kernel serve mando2222/gemma-4-26b-a4b-it-QB2

OpenAI-compatible server on :8000, model id google/gemma-4-26B-A4B-it. Structured tool calling + reasoning channel enabled; single slot (max_num_seqs=1).

Context length — validated 256K

This bundle serves the model's full native 256K (262144) context, and long-range recall is validated there — not just "it fits in DRAM". The unlock is whole-body chunked prefill (GEMMA4_CHUNKED_PREFILL=1, default on): the prompt is prefilled in bounded 16K-token chunks, which (a) keeps every chunk inside a reliable prefill regime and (b) holds peak prefill memory constant regardless of context length, so 256K completes with no OOM. Combined with the bounded sliding-window serving-KV (only the 5 global layers scale with context; the 25 sliding_window=1024 layers use a fixed 1024-token ring), the full 262K-token KV cache fits at max_num_seqs=1.

Measured prose needle-in-haystack recall (5 depths, 10/25/50/75/90%):

Context	Recall	TTFT
128K	5/5 (all depths correct)	~41 s
256K	4/5 (occasional single miss deep in the middle)	~95 s

At 256K a fact placed deep in the middle is occasionally not retrieved (a long-context "lost-in-the-middle" effect, not corruption — output stays fluent and correct-form). Endpoints and most depths are reliable. If you need strict 5/5 at every depth, cap at 128K.

Configurable knobs (launch env / vllm_bundle/vllm_metadata.json)

Env var	Default	Effect
MAX_MODEL_LEN	262144	Context cap. Lower it (e.g. 131072) for strict all-depth recall; the model's native max is 262144.
GEMMA4_CHUNKED_PREFILL	1	Whole-body chunked prefill. Required for contexts beyond ~64–128K — bounds peak prefill memory (constant in context) and keeps each chunk's recall reliable. 0 = single-chunk prefill (OOMs / degrades past ~128K).
GEMMA4_PREFILL_CHUNK_SIZE	16384	Prefill chunk size (tokens). Snapped to a multiple of the 1024 sliding window and capped at 32768. Smaller = lower peak memory, slightly more overhead.
GEMMA4_PREFILL_GATHERED	1	Per-token gathered expert prefill: 5–9× faster TTFT, no large-prefill device stall. 0 = legacy all-ones (all 128 experts/token; **hangs the device on prefills >32K**).
GEMMA4_PREFILL_GATHER_MIN_SEQ	2048	Prompts below this use all-ones (gather overhead not worth it when short).
GEMMA4_BOUNDED_SLIDING_KV_CACHE	1	Bounds the 25 sliding layers' serving KV to their 1024 window (what makes 256K KV fit).

See vllm_bundle/README.md for the full knob reference and trade-offs.

Exact stack this was built and validated on

Component	Version
tt-metal	autoport branch autoport/qwen3-32b-bringup (models/autoports/google_gemma_4_26b_a4b_it/), including the chunked-prefill driver (per-chunk valid-length + sliding-ring page-table handling)
tt-vllm (fork)	dev branch (ships the gemma4 tool + reasoning parsers + Gemma4 arch registration)
vLLM	0.1.dev1+g61bbb1eae (empty-target build; TT provided at runtime)
Hardware	2× Blackhole p300c boards = 4 ASICs (≈ 4× p150a), 1×4 mesh (MESH_DEVICE=P150x4), firmware 19.12.0, iommu=pt + 1 GB hugepages

Model / precision

MoE Gemma4ForConditionalGeneration (text-only), 30 layers, 128 experts / top-8, GQA 16/8, head-dim 256, vocab 262144, hybrid attention (5:1 sliding:global, sliding_window=1024), dual RoPE, final_logit_softcapping=30. TP=4 + EP=4 (each chip owns 32 of 128 experts). Precision: BFP8 projection/expert weights + LoFi dense-decode fidelity, BFP8 paged KV, BF16 activations/CCL; on-device greedy sampling; FABRIC_1D; metal-trace decode + gathered chunked eager prefill.

Measured (2× p300c / 4 ASICs)

• Decode: ~30–33 tok/s/user (single user), unaffected by context length.
• Prefill (TTFT): gathered + chunked path — 128K ~41 s, 256K ~95 s. Peak prefill memory is constant in context length (chunk-bounded), so long prompts scale in time but not memory. ~12× faster than the old single-chunk floor.
• Accuracy: AIME24 top-5 1.00 / top-100 1.00; GPQA-Diamond 80% (10-sample); gathered prefill PCC 0.9995 vs the all-ones reference.
• Long-context recall: prose needle 128K 5/5, 256K 4/5 (see above).
• Tool calling / reasoning: structured tool_calls via the gemma4 parser; reasoning routed to reasoning_content with clean content.

Consumer prerequisites

1. 2× Blackhole p300c boards (4 ASICs ≈ 4× p150a), 1×4 mesh, iommu=pt + 1 GB hugepages. A literal 2×2 mesh does not work (fabric routing).
2. tt-metal built with the autoport importable at models.autoports.google_gemma_4_26b_a4b_it (the adapter subclasses its Gemma4ForCausalLM, uses tt/experts/prefill_gathered.py, and the chunked-prefill correctness path in the shared tt_transformers generator + the autoport attention). Currently on autoport/qwen3-32b-bringup, not upstream — build from a tree that includes these.
3. Tenstorrent vLLM fork (dev) + TT plugin.
4. Gated weights: accept the Gemma license and provide an HF token (~52 GB).

Known limitations

• 256K recall is ~4/5 deep in the middle — endpoints/most depths reliable; cap at 128K for strict all-depth recall (see "Context length").
• Long-context TTFT is seconds-to-~1.5 min (256K ~95 s). Use generous client timeouts.
• Single slot (max_num_seqs=1) — a 2nd concurrent request queues (looks like a hang); ≥3 concurrent is not supported (a batched-decode device path is still open).
• All-ones prefill (GEMMA4_PREFILL_GATHERED=0) hangs on prefills >~32K — why gathered is default.
• Chunked prefill off (GEMMA4_CHUNKED_PREFILL=0) degrades/OOMs past ~128K — why chunking is default.

Changelog

2026-07-23 — validated 256K via chunked prefill

• Context 72K → full 256K (262144), recall-validated. Adds the whole-body chunked-prefill driver (default on, 16K chunks): bounds peak prefill memory (constant in context) and keeps each chunk's recall reliable across chunk boundaries. Prose needle recall 128K 5/5, 256K 4/5. TTFT 128K ~41 s, 256K ~95 s. No tt-vllm-plugin edits.
• New knobs: GEMMA4_CHUNKED_PREFILL, GEMMA4_PREFILL_CHUNK_SIZE; MAX_MODEL_LEN default → 262144.

2026-07-23 (earlier) — gathered expert prefill, 72K cap

• Per-token gathered expert prefill (default on): ~5–9× faster TTFT, no large-prefill stall. Cap 72K (single-chunk recall ceiling before chunking).

2026-07-21 — bounded sliding-window KV

• Bounded sliding-window serving-KV adapter (Gemma4BoundedForCausalLM): only the 5 global layers scale with context, so the full 256K KV cache fits at max_num_seqs=1.

License

Weights: google/gemma-4-26B-A4B-it under the Gemma Terms of Use. This bundle contains only Tenstorrent integration metadata + adapter + launch config.