mando2222/laguna-s-2.1-QB2

Laguna-S-2.1 — Tenstorrent Blackhole (vLLM bundle)

A tt-kernel vLLM bundle that runs poolside/Laguna-S-2.1 — a 118B Mixture-of-Experts reasoning model (256 experts, top-10 + 1 shared, ~4B active params/token; hybrid attention = 12 global + 36 sliding-window-512 layers) — on 2× Tenstorrent Blackhole p300c boards (4 ASICs ≈ 4× p150a) via the Tenstorrent vLLM plugin. Ships no kernel cache and no weights — only the vLLM registration, the full autoport adapter + tt/ model code, and a launch recipe; weights are pulled from the base repo and kernels JIT-compile at first-run warmup.

Unlike most bundles, this one is self-contained for the model code: the autoport package (models/autoports/poolside_laguna_s_2_1/) ships inside the bundle and is laid into EXTRA_MODELS_DIR on pull, so you do not need the autoport pre-installed in your tt-metal tree (you still need a compatible tt-metal for models.common.* / models.tt_transformers.* + the plugin).

Serve

tt-kernel serve mando2222/laguna-s-2.1-QB2

OpenAI-compatible server on :8000, model id poolside/Laguna-S-2.1 (aliases laguna-s-2.1, laguna-s-2.1-QB2). Tuned/"thinking" model — the laguna_glm_thinking_v8 chat template applies automatically via /v1/chat/completions. Single slot (max_num_seqs=1).

Model / precision

MoE LagunaForCausalLM (text-only), 48 layers, 256 experts / top-10 + 1 shared (~4B active/token), hidden 3072, 72 Q / 8 KV heads (GQA), head-dim 128, vocab 100352 (untied lm_head), hybrid attention (12 global + 36 sliding_window=512), qk-norm, per-layer YaRN partial-rotary, sigmoid+bias router. TP=4 + EP=4 — each ASIC owns 64 of the 256 experts (real expert parallelism: sparse_matmul reads only the active experts). Precision: BFP4 routed-expert weights, BFP8 attention projections + BFP8 paged KV, BF16 activations/CCL/lm-head. FABRIC_1D + Linear CCL; metal-trace decode; on-device greedy sampling (see below).

Decode path — traced + hybrid host-select sampling

Decode runs a traced 48-layer model-forward (LAGUNA_VLLM_TRACE_DECODE=1) to TP-vocab-sharded logits, then greedy-samples via a hybrid host-select (LAGUNA_SAMPLER_HOST_SELECT=1): per-shard argmax/max run on-device (local reductions, no cross-shard collective), and only the 4 (value, id) candidates are read to host where the winning vocab shard is chosen in fp32. This is numerically exact (matches a full-vocab host argmax) while keeping the expensive full-vocab all-gather off the host path — it is what converts the tuned model-forward into a correct served number above 20 tok/s. (The pure on-device ttnn.sampling path compares candidates in bf16 and flips cross-shard near-ties → do not disable host-select; a full-vocab on-device argmax deadlocks in-trace.) The routed-expert sparse_matmul tuning (in0_block_w=8) lifts the model-forward from ~15.5 to ~23.3 tok/s device-level.

Measured (2× p300c / 4 ASICs, single user, greedy)

• Decode: ~21.9 tok/s/user steady-state (256-token generation via /v1/chat/completions); ~21.9 tok/s on a 128-token completion. Device-level tuned model-forward ~23.3 tok/s.
• TTFT: ~0.6–1.7 s on short prompts.
• Correctness: per-layer PCC vs the HF fp32 oracle — decode 0.9995, global-attention prefill 0.9977, sliding prefill ≥0.995; on-device greedy is numerically exact (fp32 host-select). Coherent reasoning output via the thinking template (verified on prose + coding prompts).

Configurable knobs (launch env / vllm_bundle/vllm_metadata.json)

Env / flag	Default	Effect
LAGUNA_VLLM_TRACE_DECODE	1	Traced 48-layer decode model-forward (metal-trace). 0 = eager decode (~7 tok/s, integration only).
LAGUNA_SAMPLER_HOST_SELECT	1	Hybrid exact fp32 greedy select. 0 = pure on-device bf16 ttnn.sampling (corrupts high-entropy output — do not use).
sample_on_device_mode (additional-config)	decode_only	Routes greedy sampling through the on-device candidate reduction (paired with host-select). Omit for pure host sampling (correct but ~14 tok/s).
--max_model_len	8192	Served context cap (see below).
--max_num_seqs	1	Single slot.
--block_size	32	Paged KV block size (validated).
LAGUNA_VLLM_NUM_LAYERS	unset (all 48)	Reduced layer count for a fast integration smoke; NOT for real serving evidence.

Context length

• Model-capable (byte-math): ~1M. Only the 12 global layers grow KV with context; the 36 sliding layers cap at their 512-token window (~25/32 GB per ASIC at 1M).
• Served context is LOWER. The TT vLLM plugin allocates a uniform full-attention KV cache for every layer — it does not honor the 36 sliding layers' 512-token bound, so all 48 layers are costed as full-context and max_model_len fits far fewer tokens than the 1M byte-math allows. The default 8192 is a conservative serving value; the true served ceiling is found by binary search. This is a plugin KV-groups gap, not a model limit — correctness is unaffected (the SDPA op still trims sliding layers to their 512 window on the read side). See vllm_bundle/README.md.

Exact stack this was built and validated on

Component	Version
tt-metal	branch autoport/qwen3-32b-bringup @ e21f2d09f1, autoport models/autoports/poolside_laguna_s_2_1/ (shipped inside this bundle)
tt-vllm (fork)	branch deploy/gemma4-harness-fixes @ ca0be99 + TT plugin
vLLM	0.1.dev1+g61bbb1eae (empty-target build; TT provided at runtime)
Hardware	2× Blackhole p300c = 4 ASICs (≈ 4× p150a), 1×4 mesh (MESH_DEVICE=P150x4), TT_METAL_WATCHER_DISABLE_ETH=1

The precompiled-kernel compatibility key is deliberately omitted (kernels-less bundle) — kernels recompile at first-run warmup, so it is portable across tt-metal builds.

Consumer prerequisites

1. 2× Blackhole p300c (4 ASICs ≈ 4× p150a), opened as a 1×4 mesh (MESH_DEVICE=P150x4), iommu=pt + 1 GB hugepages. A literal 2×2 mesh does not work (fabric routing).
2. A compatible tt-metal build — the bundle ships the autoport model code itself, but its adapter imports models.common.* and models.tt_transformers.*, so a tt-metal tree providing those (and ttnn) must be importable.
3. Tenstorrent vLLM fork + TT plugin (install via the package manager's scripts/install.sh).
4. Weights pulled from poolside/Laguna-S-2.1 at first run — provide an HF token if the repo is gated.

Known limitations

• Served context is capped well below the ~1M byte-math by the plugin's uniform-KV allocation (see "Context length"); default 8192.
• Single slot (max_num_seqs=1) — a 2nd concurrent request queues; batched decode is a follow-up.
• Greedy served path. On-device sampling is greedy (host-select); non-greedy/penalized sampling params are under-honored on this path (omit sample_on_device_mode to route to vLLM's host sampler).
• Prefill/TTFT is not yet the optimized path — decode is the tuned surface; long-context prefill throughput is a follow-up.

License

Weights: poolside/Laguna-S-2.1 under its base-repo license (see the linked repo). This bundle contains only Tenstorrent integration metadata + the autoport adapter/model code + launch config.