gpt-oss-120b on Tenstorrent p150x4

Serves openai/gpt-oss-120b by OpenAI at revision b5c939de. See the original model card for training and evaluation details.

gpt-oss-120b, OpenAI's 117B-parameter mixture-of-experts reasoning model (5.1B active parameters), served with vLLM for chat, reasoning and function calling.

Runs on Tenstorrent p150x4 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p150x4
Context	131,072 tokens
License	apache-2.0

Intended use

Chat, reasoning, coding and function calling through an OpenAI-compatible API.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150x4
• Four P150 chips: two P300C boards (a QuietBox 2) or four P150 boards

Quickstart

tt serve tt-hous/gpt-oss-120b

tt serve downloads the Docker image and the openai/gpt-oss-120b at b5c939de8f754692c1647ca79fbf85e8c1e70f8a weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/gpt-oss-120b.

Serving profiles

Select a different serving profile using --profile.

profile	hardware	mesh	max_num_seqs	max_model_len	description
default (default)	p150x4	P150x4	32	131072	Interactive and agentic traffic: chat, multi-turn, agents, RAG, mixed prompt lengths. Long prompts are prefilled in 8,192-token steps with 8 decode steps for the running users between the steps of one prompt, so every user keeps receiving tokens; prefix caching on.
throughput	p150x4	P150x4	32	131072	Bulk work where only the time to finish a batch counts (evals at 32 concurrent requests, offline summarization or classification, batch document processing): prefill-first scheduling in 32,768-token steps, no decode steps between the steps of one prompt; prefix caching on. Lower time to finish a burst of long prompts, no streaming during another user's long prefill.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "openai/gpt-oss-120b" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser openai): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

Pick a profile for the traffic. default prefills long prompts in 8,192-token steps and keeps decoding running requests between the steps: use it for chat, agents and mixed prompt lengths. throughput prefills first in 32,768-token steps and pauses decode for every request while a long prompt prefills: use it for batch work. Where the speed table below measures both profiles at the same point, throughput finishes 8 or 32 prompts of 16,384 tokens and 8 prompts of 32,768 tokens 7 to 9% sooner than default, and 32 prompts of 4,096 tokens about 1% sooner.

Reasoning comes back separately from the answer. max_tokens covers both. The server keeps clamp(max_tokens // 8, 128, 1024) tokens of every chat completion's budget for the answer, so a request that runs out of budget while reasoning still returns an answer (finish_reason: "length" with non-null content). reasoning_effort (low, medium, high) sets how much the model reasons, and usage.completion_tokens_details.reasoning_tokens reports what it used.

Sampling: temperature 0, or top_k of 32 or lower, is sampled on the device. The request fields that move a batch to host sampling are listed under Limitations.

Prefix caching is on in both profiles. The prefix-cache table under Expected performance shows what a cache hit saves.

Expected performance

eval	this package	reference	prompts	notes
GPQA Diamond, CoT zero-shot	78.3%	80.1%	198	boxed-letter extraction
AIME 2025	90.0%	92.5%	30
AIME 2024	90.0%	95.8%	30
MMLU, generative	85.3%	90.0%	3,529 (25% of each subject)	reference is the full set
MMMLU, 14 languages	82.2%	81.3%	9,842 (5% of each)	reference is the full set
HLE, text-only	13.2%	14.9%	500	judge: gpt-oss-120b here, o3-mini for the reference
SWE-bench Verified	50.0%	62.4%	50 (first 50 by id)	mini-swe-agent here; reference: 500 instances, OpenAI scaffold
IFEval, prompt-level strict	85.8%	78.2%	541	reference: frozebench run
IFEval, instruction-level strict	90.2%	—	541
IFEval, prompt-level / instruction-level loose	88.9% / 92.2%	—	541
AIME 2025, tt-inference-server release workflow	76.67%	92.5%	30	release-workflow recipe, see below
GPQA Diamond, tt-inference-server release workflow	71.72%	80.1%	198	release-workflow recipe, see below
MMLU, tt-inference-server release workflow	86.06%	85.9%	14,042	release-workflow recipe, see below

The reference figures are from the OpenAI model card at reasoning effort high without tools, except IFEval. Evals run through the chat API: reasoning_effort high, one sample per item, temperature 1.0, top_p 1.0, top_k 32, output budgets of 65,536 tokens for GPQA Diamond and AIME, 32,768 for HLE and 8,192 for MMLU and MMMLU. IFEval: reasoning_effort medium, greedy, 4,096 tokens, 32 concurrent requests. The last three rows use the tt-inference-server release workflow (lm-evaluation-harness over chat completions, temperature 1.0, seed 42; AIME 2025 and GPQA Diamond at reasoning_effort high with 122,880 output tokens, MMLU at low with 65,536). In the release-workflow AIME 2025 and GPQA Diamond rows, a response whose reasoning used the whole 122,880-token budget returned no answer and counts as a miss.

profile	ISL	OSL	concurrency	N	TTFT (ms)	prefill tok/s/u	decode tok/s/u	E2EL (s)
default	128	128	1	1	247	518	65.4	2.19
default	128	128	8	1	559	229	30.7	4.70
default	128	128	32	1	991	129	20.1	7.32
default	1,024	256	1	1	291	3,515	64.6	4.24
default	1,024	256	8	1	1,564	655	30.4	9.95
default	1,024	256	32	1	4,344	236	18.5	18.10
default	2,048	128	1	1	590	3,469	64.2	2.57
default	2,048	128	8	1	2,768	740	29.1	7.13
default	2,048	128	32	1	6,731	304	13.2	16.36
default	4,096	256	1	1	724	5,661	63.8	4.72
default	4,096	256	8	1	4,572	896	26.3	14.25
default	4,096	256	32	1	13,488	304	11.6	35.56
default	16,384	256	1	1	2,668	6,140	62.1	6.78
default	16,384	256	8	1	9,384	1,746	12.2	30.29
default	16,384	256	32	1	58,837	278	4.8	112.07
default	32,768	256	1	1	5,649	5,800	59.9	9.91
default	32,768	256	8	1	28,265	1,159	9.4	55.44
default	32,768	256	32	1	112,215	292	5.3	160.06
default	65,536	256	1	1	12,417	5,278	56.0	16.97
default	65,536	256	8	1	62,762	1,044	7.7	95.79
default	65,536	256	32	1	238,147	275	6.4	277.85
default	130,816	256	1	1	30,010	4,359	49.8	35.13
default	130,816	256	8	1	152,025	860	8.1	183.51
default	130,816	256	32	1	569,790	230	7.5	603.93
throughput	4,096	256	16	1	10,999	372	16.7	26.31
throughput	4,096	256	32	1	18,117	226	15.0	35.15
throughput	16,384	256	1	1	2,810	5,831	62.0	6.92
throughput	16,384	256	8	1	15,432	1,062	20.0	28.16
throughput	16,384	256	16	1	25,921	632	8.6	55.59
throughput	16,384	256	32	1	68,592	239	7.7	101.56
throughput	32,768	256	1	1	5,616	5,834	59.9	9.88
throughput	32,768	256	4	1	16,576	1,977	25.3	26.66
throughput	32,768	256	8	1	27,399	1,196	11.1	50.33
throughput	65,536	256	1	1	11,552	5,673	55.9	16.11
throughput	65,536	256	2	1	19,865	3,299	25.4	29.92
throughput	65,536	256	4	1	32,628	2,009	12.5	52.96
throughput	130,816	256	2	1	46,100	2,838	12.9	65.83

Means over one run per row on a QuietBox 2 (two P300C boards), against this package's image, with a closed-loop latency client: random-token prompts of exactly ISL tokens, output pinned to OSL tokens, 4 requests per user up to 4,096 input tokens, 2 per user at 16,384 and 1 per user above. The default rows are selected from a sweep at 1 to 32 concurrent users. Random prompts never hit the prefix cache. Prefill tok/s/u is ISL divided by TTFT, so above concurrency 1 it includes queueing time: at most 17 sequences of 16k tokens, 8 of 32k, 4 of 64k or 2 of 128k are resident at once, and later requests wait for admission.

Prefix cache, measured with vllm bench serve (a shared random prefix plus a 512-token unique suffix, 64 output tokens, greedy, one user, 12 requests), median TTFT:

shared prefix (tokens)	TTFT without a cache hit (ms)	TTFT with a cache hit (ms)
16,384	3,024	820
32,768	6,059	929
65,536	13,150	1,161

A cache hit still prefills about 2,560 tokens plus the new suffix, at any prefix length. A shared prefix under about 5,000 tokens saves little.

Limitations

• Needs four P150 chips; P150 and P150x2 are not offered. Tested on one QuietBox 2 only.
• Not for workloads that need OpenAI's built-in Python or browser tools: none is provided, so OpenAI's "with tools" results do not apply. A malformed tool-call JSON from the model is passed through as is, and an empty final message ends with finish_reason: "stop", so agent scaffolds need a retry for both.
• Capacity is 296,960 tokens of context across all requests. Further requests wait for admission, and the wait is part of TTFT. Prompts above 2,048 tokens prefill one at a time, and the first prompt of a new length adds a few seconds. 9 to 16 concurrent requests decode at about 50 to 58 ms per token instead of 42.
• One request with min_p, logit_bias, bad_words, allowed_token_ids, min_tokens, structured output, any logprobs, or a non-zero temperature with top_k above 32 or unset, moves the whole batch to host sampling and slows decode for every user until it finishes.
• max_tokens covers reasoning and answer. A hard problem under a small budget gets a short, truncated answer. Requests with n above 1, logprobs, ignore_eos or a required tool call have no answer reserve and can return null content with finish_reason: "length".
• With 8 or more concurrent requests at 16k tokens and up, one request per batch occasionally receives its own prompt streamed back as generated text, in one chunk of up to 725,000 characters. Clients should tolerate an oversized first chunk.
• Under sustained 32-user load with 122,880-token budgets, the server can fail with no free device state slot. Submitting such work in fixed batches of 32 avoids it.

Licensing

Weights: Apache-2.0. Port and serving code: Apache-2.0, from public branches of tt-metal, vLLM and vllm-tt-plugin.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/gpt-oss-120b/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	ee5703cefab26d5360728770800d48d442a048fd
vLLM	vllm-0.26.0.dev4+g2bbd2a6d9-cp310-cp310-linux_x86_64.whl — a wheel the author built
vllm-tt-plugin	30e9b8ea4d38d1a2ca3cf7c2c7da542747bf91b2
code/ digest	325798b898bb4c4a (sha256, first 16 hex digits)
built	2026-09-29T16:02:28+00:00 by tt-model 0.1.0