qwen3.8-flash-next on Tenstorrent p300x2

Serves the language model of Qwen/Qwen3.8-Flash-Next by the Qwen team at revision f5d08274. See the original model card for training and evaluation details.

Qwen3.8-Flash-Next, a mixture-of-experts model with 125B parameters (about 6B active per token) plus a 51B n-gram embedding table, served with vLLM as a text-only chat and completion endpoint with tool calling and separated reasoning output. Routed experts run in 4-bit block floating point (BFP4), derived once from the BF16 checkpoint.

Runs on Tenstorrent p300x2 (mesh (4, 1)) — 262,144-token context, up to 1 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p300x2
Context	262,144 tokens
License	qwen-community-1.0

Intended use

Chat, reasoning and tool calling on text through an OpenAI-compatible API, one request at a time.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300x2
• About 360 GB of disk for the checkpoint, plus about 64 GiB for the converted expert cache the first serve writes

Quickstart

tt serve tt-hous/qwen3.8-flash-next

tt serve downloads the Docker image and the Qwen/Qwen3.8-Flash-Next at f5d08274bafd880402bd16f5e3e6c514136ec06c weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/qwen3.8-flash-next.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "Qwen/Qwen3.8-Flash-Next" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser qwen3_xml): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Reasoning output is separated (--reasoning_parser qwen3): the thinking text arrives in reasoning, apart from content.

Requests that set no sampling fields get the checkpoint's defaults (temperature 1.0, top_p 0.95, top_k 20). For long reasoning, add a presence_penalty of 1.0 to 1.5. Greedy requests (temperature 0) are sampled on the device. Every request with temperature above 0, penalties, logprobs, a seed, min_p, logit_bias, bad_words, allowed_token_ids or structured output is sampled on the host, about 18 ms per token slower.

Expected performance

Task evals:

eval	this package	reference	prompts	notes
GSM8K, exact match on the final number, thinking off	92% (92 of 100)	—	100	greedy, max_tokens 512; 8 answers cut at the cap

GSM8K ran through the served endpoint. The Qwen model card reports no GSM8K figure, and none of the text benchmarks it does report (GPQA Diamond, LiveCodeBench v6, IFBench, HLE, SWE-bench Pro, SWE-bench Multilingual, DeepSWE 1.1, NL2Repo-Bench, Toolathlon Verified, Agents' Last Exam, CoWorkBench, JobBench) has been run against this package.

Token agreement with a BF16 reference of the same checkpoint. This is a teacher-forced comparison of next-token predictions, not a task benchmark:

measure	this package	reference	prompts	notes
Top-1 token agreement, decode	96.3%	—	14 prompts, 1,386 positions	reference token in this package's top-5: 100.0%
Top-1 token agreement, decode, prompts up to 2,048 tokens	97.0%	—	11 prompts
Top-1 token agreement, decode, 3k to 12k-token documents	93.9%	—	3 prompts, 297 positions
Top-1 token agreement, prefill	88.8%	—	12 prompts, 4,594 positions	reference top-1 in this package's top-5: 98.8%

The reference is a BF16 Hugging Face run (Transformers 5.16) of the same checkpoint revision on the CPU. The reference's tokens are fed in and each next-token prediction is compared. The prompts cover mathematics, code, JSON extraction, Chinese and Japanese text, chat, summarization, tool-call formatting and long documents.

profile	ISL	OSL	concurrency	N	TTFT (ms)	prefill tok/s/u	decode tok/s/u	E2EL (s)
long-context	128	128	1	1	340	379	24.3	5.60
long-context	1,024	256	1	1	1,590	642	24.2	12.10
long-context	4,096	256	1	1	7,990	513	13.9	26.40
long-context	16,384	256	1	1	38,270	428	13.8	56.70
long-context	32,768	256	1	1	79,250	413	13.8	97.70
long-context	65,536	256	1	1	163,000	402	13.8	182.00
long-context	131,072	256	1	1	338,000	388	13.8	357.00

Means over one run per row, measured on 2026 Sep 17 on two P300 boards (four Blackhole chips) with a closed-loop latency client: random-token prompts of exactly ISL tokens, output pinned to OSL tokens with ignore_eos, greedy, one unrecorded warmup request per input length, 4 requests per row up to 4,096 input tokens, 2 at 16,384 and 1 above. Prefill tok/s/u is ISL divided by TTFT.

Limitations

• Text only: requests with image or video content are rejected. Tested on two P300 boards only. No batching above one request: concurrent requests queue.
• No benchmark that the Qwen model card reports has been run against this package; only GSM8K and token agreement with the BF16 reference were measured.
• Not for interactive use with long prompts: TTFT grows by about 2.6 ms per prompt token (about 38 s at 16k, 5.6 min at 131k). Set client timeouts of at least 10 min for 131k prompts and stream the response.
• No prefix caching: every turn of a multi-turn chat re-prefills the whole prompt.
• Decode is about 24 tokens per second per user under 2,048 tokens of context and about 14 above it. Host sampling lowers these to about 17 and 11.
• Chains of thought beyond about 25,000 to 30,000 reasoning tokens can drift and lose the final answer. Cap max_tokens near 32,768 for such prompts and prefer several shorter attempts.
• After prompts above 2,048 tokens, output agrees with the reference less often (93.9% top-1 against 97.0% for short prompts).

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/qwen3.8-flash-next/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
vLLM	v0.24.0
vllm-tt-plugin	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
code/ digest	1b5c8d17b1caee5f (sha256, first 16 hex digits)
built	2026-09-17T16:37:48+00:00 by tt-model 0.1.0