stisiTT/smollm2-135m-instruct-p150

SmolLM2-135M-Instruct on Tenstorrent Blackhole (p150)

A tt-model vLLM bundle that serves HuggingFaceTB/SmolLM2-135M-Instruct on a single Blackhole chip through the Tenstorrent vLLM plugin, with an OpenAI-compatible API.

This is a v4 bundle: metadata plus a pointer to the upstream weights. Weights are not stored here by design, they come from HuggingFaceTB/SmolLM2-135M-Instruct and tt-model pull fetches them for you.

Prebuilt wheels are attached under wheels/, including a portable ttnn build. See Skip building tt-metal.

What you need

Hardware	One Blackhole chip. Validated on a BH QuietBox 2 (p300x2, 2x P300 cards, 4 chips), using one of its chips
tt-metal	v0.78.0-dev or newer, built, with ttnn importable
vLLM	0.25.1 built with VLLM_TARGET_DEVICE=empty
Plugin	vllm-tt-plugin

If you do not have vLLM and the plugin yet, from an activated tt-metal env:

git clone https://github.com/tenstorrent/vllm-tt-plugin.git && cd vllm-tt-plugin
source docs/install-vllm-tt.sh

That takes about 45 seconds. VLLM_TARGET_DEVICE=empty means no kernels are compiled.

Skip building tt-metal (prebuilt wheels)

Building tt-metal from source is the largest cost in getting started. The wheels under wheels/ let you skip it:

Wheel	Size	Notes
ttnn-0.75.0rc10.dev1031+g46fa19bf6be-cp312-cp312-manylinux_2_35_x86_64.whl	406 MB	auditwheel-repaired: vendors libtracy/libmpi/libnuma/libhwloc, RPATH rewritten to $ORIGIN
vllm-0.25.1+empty-cp312-cp312-linux_x86_64.whl	7.9 MB	vLLM built with VLLM_TARGET_DEVICE=empty
vllm_tt_plugin-0.1.0-py3-none-any.whl	142 KB	the Tenstorrent vLLM plugin

huggingface-cli download stisiTT/smollm2-135m-instruct-p150 \
  --include 'wheels/*' --local-dir .
uv venv --python 3.12 venv && source venv/bin/activate
uv pip install wheels/*.whl
python -c "import ttnn; print(ttnn.get_num_devices(), 'device(s)')"

Verified: in a fresh venv with no tt-metal source tree on the path, import ttnn works and enumerates devices.

Read this before relying on them:

• cp312 only. Built for CPython 3.12. Importing under another minor version segfaults inside nanobind with no usable traceback.
• Built from a branch, not upstream main. Commit 46fa19bf6be, which is upstream plus the MESH_DEVICE fix described under Caveats. Treat them as a convenience build, not a release.
• ttnn alone is not the whole repo. The vLLM adapter lives in tt-metal's models/ tree, so serving through tt-model or the demo still needs a tt-metal checkout on PYTHONPATH. The wheels save you the C++ build, not the clone.
• numpy<2 is required by ttnn. If you install vLLM's full dependency set alongside it, pin opencv-python-headless==4.11.0.86, as vllm-tt-plugin's overrides do. Newer opencv requires numpy>=2, which makes the install unsolvable rather than merely awkward: pip's --constraint cannot override a declared requirement, so you need uv's --override. The v5.1 container path handles this for you.

Use it

tt-model pull  stisiTT/smollm2-135m-instruct-p150
tt-model serve stisiTT/smollm2-135m-instruct-p150

The plugin auto-registers the architecture from the bundle, so no per-model edit is needed.

Without tt-model, the same thing directly:

export MESH_DEVICE=P150 HF_MODEL=HuggingFaceTB/SmolLM2-135M-Instruct
pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"

Bundle format status (read if you are choosing a format)

This bundle is v4, which works today but is not where tt-model is heading. As of 2026-08-27 three proposals are open, all still drafts:

	What it does	State
#37	v5.1: ship the whole platform as an OCI image. Consumer needs only Docker and a card, no tt-metal, no vLLM, no venv, no matching Python or OS	draft
#32	v6 thin bundle, per-model venv from pip pins	draft
#34	drop v3/v4 schemas, keep v5 fat + v6 thin	draft, stacked on #32

v5.1 is explicitly additive and states that v3/v4/v5 are untouched, so nothing here breaks when it lands. The format that would remove v4 is #34, which is a draft stacked on another draft.

This bundle has an expiry date. Its manifest declares "schema": 4. tt-model currently accepts {"3", "4", "5"}, but #34 narrows that to v5 and v6 only, and a bundle on any other version is refused rather than degraded. If #34 merges before this is republished, tt-model pull will stop working here. The wheels and the measurements below are unaffected, since neither depends on the bundle format.

If you are packaging a new model today, v5.1 is the better target. It has already been proven on hardware: qwen3-coder-30B-A3B on a QB2 (p300x2), packaged, served, stopped, reloaded from the OCI layout and served again with byte-identical output, then published as 272 files. A 10.2 GB image publishes as 2.1 GB because layers are content-addressed and deduplicated.

Weights stay a pointer in v5.1 too, so that part of this bundle carries over unchanged.

Measured performance

Single chip, MESH_DEVICE=P150, no workarounds. Weights bfloat8_b (MLP w1/w3 bfloat4_b), paged attention, tracing on. Greedy decode throughout.

What to look at

For anything resembling real use, the rows that matter are the longer prompts:

Shape	Concurrency	TTFT p50 (ms)	TPOT (ms)	Output tok/s
1024 in / 128 out	1	22.2	8.0	122.7
1024 in / 128 out	7	114.7	8.3	770.3
2048 in / 128 out	1	49.3	11.3	86.7
4096 in / 128 out	1	121.3	17.4	55.3
128 in / 1024 out	1	11.5	6.7	148.4

Time to first token scales with prompt length as you would expect (22 ms at 1k, 121 ms at 4k), and per-token decode cost roughly doubles from 1k to 4k context.

Max context: 8192 tokens, the checkpoint's own max_position_embeddings. Not reduced.

Caveat on the 128/128 numbers

The full sweep below includes ISL/OSL 128/128, where this model reaches 6513 tok/s aggregate at concurrency 32. That figure is included for comparability with other posted numbers, not because it means anything. A 128-token prompt with 128 greedy-decoded tokens does not correspond to a workload anyone runs, and quoting its peak throughput as a headline overstates what the hardware will do for you. Judge this model on the table above.

These numbers are also unoptimised and ungraded. No perf targets are configured for this model on p150, nothing was tuned, and no attempt was made to find a better configuration. Treat them as a floor and a baseline for improvement, not as a claim.

Full sweep

Serving (via vllm bench serve, OpenAI chat endpoint):

Concurrency	Requests	ISL	OSL	TTFT mean (ms)	TTFT p50	TPOT (ms)	Output tok/s	Total tok/s
1	8	128	128	273.3	11.1	5.3	135.0	270.0
32	256	128	128	496.3	120.2	6.0	3256.8	6513.5
1	4	128	1024	11.9	11.5	6.7	148.4	167.0
7	28	128	1024	52.5	52.5	6.9	1010.3	1136.5
1	4	1024	128	21.1	22.2	8.0	122.7	1104.0
7	28	1024	128	109.8	114.7	8.3	770.3	6932.8
1	4	2048	128	43.1	49.3	11.3	86.7	1474.2
3	12	2048	128	104.4	116.7	11.4	248.3	4221.1
1	4	4096	128	100.2	121.3	17.4	55.3	1825.0

Single-stream generation (tt-metal demo, batch 1, 30 layers, 200 tokens): time to first token 5.79 ms, decode 3.63 ms/token = 275 tok/s/user. A repeat run gave 5.93 ms and 278 tok/s, so it is stable.

Read the first row carefully

Row 1 shows a 273 ms mean TTFT against an 11 ms p50 and a 1963 ms p99. That is first-request compile and warmup cost dragging the mean, not steady-state latency. The 128/1024 concurrency-1 row, which ran later against a warm server, is the honest single-stream number at 11.9 ms mean against a 13.4 ms p99. This is why the summary table above quotes p50 rather than mean TTFT.

Where the headroom is

Nothing here was tuned, so if you want to improve on it the untried levers are: a trace_region_size override (Llama-3.1-8B's p150 entry uses 900 MB; this ran on dynamic allocation), datatype choices beyond the bfloat8_b default, and max_num_seqs sizing. Whether any of them helps a model this small is unmeasured.

Caveats

• tt-model pull currently fails on every bundle, including this one, with AttributeError: type object '_ActivityTqdm' has no attribute 'get_lock' (issue #35). Fix is in PR #36. Until it merges, use the direct pytest path above. The wheels and the numbers on this page do not depend on it.
• On a single-chip host, stock tt-metal works unpatched. Confirmed by running it: with the system presenting exactly one device, the unmodified conftest.py passes both the smoke and the full 30-layer run with MESH_DEVICE=P150, no fabric timeout. The buggy lookup below falls back to the device count, which is 1, so fabric is correctly disabled.
• On a multi-chip host such as a QuietBox 2, you need a patch. Stock tt-metal fails with a fabric router sync timeout when you ask for fewer chips than the box has. models/tt_transformers/conftest.py resolves MESH_DEVICE through a map missing the Blackhole SKUs (P150, P300, P150x4, BHGLX), falls back to the physical device count, decides the run is multi-device, and starts fabric across the whole cluster for a 1x1 mesh. On a QB2 that breaks both MESH_DEVICE=P150 and MESH_DEVICE=P300. Fix is on branch stisi/smollm2-p150.
• This is the Instruct checkpoint. The base SmolLM2-135M defines no chat template, so it serves /v1/completions but returns HTTP 400 on /v1/chat/completions.
• It is a 135M model. Under greedy decoding it degrades into repetition on longer generations. That is the checkpoint, not the port: HF fp32 on CPU reproduces the same loop from the same context at a near-identical unique-token ratio.
• PAD_MLP_CORES is not needed. hidden_size is 576, but that knob pads intermediate_size (1536), for which it is a measured no-op.

Reproducing the benchmarks

With tt-inference-server:

python3 run.py --model SmolLM2-135M-Instruct --tt-device p150 \
  --workflow benchmarks --local-server --dev-mode --no-auth \
  --tt-metal-home /path/to/tt-metal --skip-system-sw-validation

--no-auth is required. Without it the server answers /health with 200 while every API endpoint returns Unauthorized, and the benchmark client's readiness probe spins for its full 600 second timeout with no indication why.

Provenance

tt-metal	46fa19bf6be (v0.78.0-dev)
ttnn	0.75.0rc10.dev1031+g46fa19bf6be
vLLM	0.25.1+empty
vllm-tt-plugin	0.1.0
Adapter	models.tt_transformers.tt.generator_vllm:LlamaForCausalLM (tt-metal built-in)
Measured	2026-08-27, one chip of a BH QuietBox 2