ndaly/Mistral-Small-4-119B-2603

mistral-small-4-119b-p300x2 on Tenstorrent p300x2

Derived from mistralai/Mistral-Small-4-119B-2603 by Mistral AI, unmodified. See the original model card for training and evaluation details.

Mistral Small 4 (Mistral-Small-4-119B-2603), a 119B-parameter mixture-of-experts transformer with 128 experts, 4 active and 6.5B active parameters per token, and multi-head latent attention, served with vLLM in instruct mode. Text only, tested at one user and at 32 concurrent users.

Runs on Tenstorrent p300x2 (mesh P300x2) — 1,048,576-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p300x2
Context	1,048,576 tokens
License	apache-2.0

Intended use

General chat assistant and coding use through the OpenAI-compatible chat and completions API, in the languages the checkpoint covers (English, French, German, Spanish, Portuguese, Italian, Chinese, Japanese, Korean, Arabic and others).

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300x2
• 121 GB of free space in the Hugging Face cache for the weight shards, and about 100 GB more under ~/.cache/tt-model/ for the converted weights
• Hugepages configured on the host

Quickstart

tt serve ndaly/Mistral-Small-4-119B-2603

tt serve downloads the Docker image and the mistralai/Mistral-Small-4-119B-2603 at a11f36bebf709121056b1dbcc943d1c6afbe494d weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve ndaly/Mistral-Small-4-119B-2603.

The first boot also converts the weights for the four-chip mesh, which takes tens of minutes; later boots load the converted weights from the cache.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "mistralai/Mistral-Small-4-119B-2603" — the weights id, not this package's name.

The chat template adds a default system prompt of about 564 tokens to every chat completion; it is included in the time to first token below.

Expected performance

eval	this package	reference	prompts
IFEval, prompt-strict (1,024-token cap)	72.83 %	—	541
HumanEval-instruct, pass@1	92.68 %	—	164

Evals ran through the served endpoint. The original model card reports its benchmarks as charts, with no IFEval or HumanEval figure.

profile	ISL	OSL	concurrency	N	TTFT (ms)	prefill tok/s/u	decode tok/s/u	E2EL (s)
default	~600 (a short prompt plus the default system prompt)	—	1	—	~1,500	~500	46	—
default	100	—	32	—	—	—	2.1 (66 aggregate)	—

Measured on p300x2 through the served chat completions endpoint on 2026-09-16. Decode is 21 to 22 ms per token for one user; prefill runs at about 2 ms per prompt token up to the full 1,048,576-token context. OSL, the 32-user TTFT and the number of runs behind each row were not recorded.

Limitations

• Validated on p300x2 only (one four-chip mesh, tensor-parallel over 4 chips); up to 32 concurrent sequences.
• Text only: image input, which the checkpoint supports, is not ported.
• No reasoning parser or tool-call parser is configured: the checkpoint's reasoning mode is not validated, and tools in a request are not turned into tool_calls.
• Prefix caching and chunked prefill are off, so every request prefills from scratch and prefill is sequential across users; at 32 users aggregate decode is about 66 tok/s.
• Prompts up to 1,048,576 tokens are accepted, at about 2 ms per prompt token (around 35 minutes for a full-context prompt). Accuracy beyond the checkpoint's 262,144-token training context was not evaluated.
• GPQA was not measured.
• The serve profile sets TT_METAL_PINNED_MEMORY_CACHE_LIMIT_BYTES=0, which disables tt-metal's pinned host-memory upload path. On newer host kernels (seen on 7.0.0-34 with tt-kmd 2.10.0) that path spins forever inside the driver when a weight tensor is uploaded straight from the memory-mapped checkpoint (tt-metal issues #56613 and #57269). The fallback costs about one extra minute on the first boot.
• Weights run at a reduced-precision block-float policy, not bf16; outputs differ from the Hugging Face reference.

Licensing

Weights: Apache-2.0 (Mistral AI). Port and serving code in code/: Apache-2.0.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/ndaly/Mistral-Small-4-119B-2603/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
vLLM	vllm-0.1.dev14195+g8c28fcecb.d20260916.empty-py3-none-any.whl — a wheel the author built
vllm-tt-plugin	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
code/ digest	a4dc23aeb6810dad (sha256, first 16 hex digits)
built	2026-09-16T16:55:50+00:00 by tt-model 0.1.0