voxtral-mini-4b-realtime-2602-p300x2 on Tenstorrent p300x2

Derived from mistralai/Voxtral-Mini-4B-Realtime-2602 by Mistral AI, unmodified. See the original model card for training and evaluation details.

Voxtral-Mini-4B-Realtime-2602, Mistral AI's realtime speech-to-text model: a 970M-parameter causal audio encoder feeding a 3.4B-parameter Ministral text decoder, 13 languages, one text token per 80 ms of audio. Served with vLLM as an OpenAI-compatible /v1/audio/transcriptions endpoint, tested at one request and at 32 concurrent requests.

Runs on Tenstorrent p300x2 (mesh P300x2) — 131,072-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p300x2
Context	131,072 tokens
License	apache-2.0

Intended use

Speech-to-text transcription of mono speech audio (wav, flac, mp3 and other formats soundfile decodes; resampled to 16 kHz) through the OpenAI audio transcriptions API, in the 13 languages the checkpoint covers.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300x2
• 1 GB hugepages configured on the host
• About 9 GB of free space in the Hugging Face cache for the weights

Quickstart

tt serve ndaly/Voxtral-Mini-4B-Realtime-2602

tt serve downloads the Docker image and the mistralai/Voxtral-Mini-4B-Realtime-2602 at 2769294da9567371363522aac9bbcfdd19447add weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve ndaly/Voxtral-Mini-4B-Realtime-2602.

This server answers transcription requests only: tt-model curl sends a chat completion and gets a 404. Send audio as shown under Capabilities.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "mistralai/Voxtral-Mini-4B-Realtime-2602" — the weights id, not this package's name.

The one inference endpoint is POST /v1/audio/transcriptions (multipart form, OpenAI shape); /v1/chat/completions is not mounted.

curl -s http://127.0.0.1:20000/v1/audio/transcriptions \
  -F model=mistralai/Voxtral-Mini-4B-Realtime-2602 -F file=@clip.wav
# streaming (server-sent events, text deltas as they are decoded):
curl -sN http://127.0.0.1:20000/v1/audio/transcriptions \
  -F model=mistralai/Voxtral-Mini-4B-Realtime-2602 -F file=@clip.wav -F stream=true

Any OpenAI client works the same way (client.audio.transcriptions.create(model=..., file=...)). Decoding is greedy by default.

Expected performance

eval	this package	reference	prompts
LibriSpeech test-other, WER (greedy)	5.64 %	—	2,939 utterances
Transcript identical to the Hugging Face reference	8 of 10 clips	—	10 clips
Top-1 token agreement with the bf16 reference	99.5 %	—	—

WER was scored with lmms-eval through the served endpoint; the two clips that differ do so by near-tie words. The original model card reports Fleurs, long-form and short-form English WER at a 480 ms delay (Fleurs average 8.72 %, English 4.90 %) but no LibriSpeech figure; those sets were not run here.

profile	ISL	OSL	concurrency	N	TTFT (ms)	prefill tok/s/u	decode tok/s/u	E2EL (s)
default	5.28 s of audio	78	1	—	37.5	—	49.7	1.59
default	—	—	32	—	—	—	18.9 (606 aggregate)	3.63 (median)

Measured on p300x2 through the served endpoint on 2026-09-23, greedy, streaming; ISL is audio length. Decode is 20.14 ms per step for one user, and 32 of 32 concurrent requests completed. The payload of the 32-request run and the number of runs behind each row were not recorded.

Limitations

• Transcription only: no chat, text completion, translation, speaker diarization or audio question answering; /v1/chat/completions is not mounted.
• Validated on p300x2 only, as one tensor-parallel replica over four chips; no data parallelism, no other board.
• At most 32 concurrent requests. Validated on utterances up to about 35 s; longer audio runs through the same streaming path but was not measured.
• Weights run at reduced precision (bfp4 with LoFi inner projections, bfp8 LM head and KV cache), not bf16. On LibriSpeech test-other this drops one word onset on 1 of 2,939 utterances where the bf16 reference does not, and under concurrency 1 of 2,939 utterances came back empty.
• Sampled decoding is supported but not evaluated for accuracy; use the greedy default.

Licensing

Weights: Apache-2.0 (Mistral AI). Port and serving code in code/, and the vLLM fork and plugin inside the image: Apache-2.0.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/ndaly/Voxtral-Mini-4B-Realtime-2602/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
vLLM	vllm-0.1.dev14195+g8c28fcecb.d20260916.empty-py3-none-any.whl — a wheel the author built
vllm-tt-plugin	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
code/ digest	b26d652115a867ab (sha256, first 16 hex digits)
built	2026-09-25T16:20:42+00:00 by tt-model 0.1.0