flo5k5/Voxtral-Mini-4B-Realtime-2602-MLX-6bit

Voxtral-Mini-4B-Realtime-2602 — MLX 6-bit

6-bit MLX quantization of mistralai/Voxtral-Mini-4B-Realtime-2602, Mistral's streaming speech-to-text model, for Apple Silicon with mlx-audio.

Why this conversion: on a 64 GB M5 Pro, the fp16 MLX weights cannot keep up with live speech (0.47× real time), so a streamed transcript falls further and further behind. This 6-bit version keeps up, with no measurable accuracy loss on real French voices (numbers below).

It is in the mlx-audio format (model_type: voxtral_realtime). Note that mlx-community/Voxtral-Mini-4B-Realtime-6bit uses Mistral's original config layout, which mlx-audio's loader does not read.

Conversion

From mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16, with mlx-audio at commit 94c7716 (0.5.7) and MLX 0.32.3:

python -m mlx_audio.convert \
  --hf-path mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16 \
  --mlx-path Voxtral-Mini-4B-Realtime-2602-MLX-6bit \
  -q --q-bits 6

Affine quantization, 6 bits, group size 64. Size: 3.6 GB (fp16: 8.9 GB).

Measurements

Apple M5 Pro, 64 GB. 120-clip sample of the Common Voice 17 French test set (120 speakers, 12 min, clips with ≥ 2 up-votes and no down-vote), word error rate on lower-cased text without punctuation.

Streaming, served by oMLX over its realtime WebSocket (/v1/audio/transcriptions/realtime, 480 ms transcription delay), audio sent at speaking pace in 80 ms packets:

	fp16	6-bit
First text after audio starts (median, 40 clips)	5.4 s	1.8 s
Final text after speech ends (median)	8.8 s	0.27 s
Lag of the text behind the voice, 79 s stream (median / p90)	130 s / 161 s	0.12 s / 2.6 s
WER of the streamed text (mean per clip)	9.7 %	8.6 %

Accuracy by transcription delay (mlx-audio streaming session, 40 clips):

Delay	240 ms	480 ms	960 ms	2400 ms
fp16 WER	10.4 %	9.2 %	6.3 %	6.0 %
6-bit WER	10.6 %	8.5 %	6.8 %	6.8 %

The sample is small (40 clips for streaming), so differences of about one point are within noise. The lag of the long stream is estimated assuming a steady speaking rate: at time t, with c of the C final characters received, the text covers about c/C of the speech duration. Only French was measured; the conversion does not touch the other languages, but they were not checked.

Usage

mlx-audio:

from mlx_audio.stt.utils import load_model

model = load_model("flo5k5/Voxtral-Mini-4B-Realtime-2602-MLX-6bit")
print(model.generate("speech.wav", language="fr").text)

# streaming
session = model.create_streaming_session(transcription_delay_ms=480)
session.feed(samples_16khz_float32)   # call repeatedly as audio arrives
for delta in session.step(max_decode_tokens=8):
    print(delta, end="")

oMLX: place the folder in your model directory; it is discovered as an audio_stt model and served on /v1/audio/transcriptions and on the realtime WebSocket /v1/audio/transcriptions/realtime.

License

Apache 2.0, as the original model. All credit for the model goes to Mistral AI; this repository only changes the numeric precision of the weights.