Voxtral Mini 4B Realtime GGUF for audio.cpp
This repository contains quantized standalone GGUF checkpoints for running Voxtral Mini 4B Realtime ASR with audio.cpp. The GGUF files embed the audio.cpp model spec and required sidecars, so the model can be used directly from the checkpoint path without a separate local model-spec directory.
What audio.cpp does
audio.cpp is a young C++/GGML audio inference framework focusing on CUDA performance. It runs speech and audio models locally with CLI and server interfaces, including ASR, TTS, voice conversion, source separation, diarization, VAD, and audio generation models. The project currently tracks 40+ model families and is growing quickly. For Voxtral Realtime, audio.cpp provides offline and streaming speech recognition from a single GGUF checkpoint.
Files
| File | Quantization | Size | Recommended use |
|---|---|---|---|
voxtral-mini-4b-realtime-2602-q8_0.gguf |
Q8_0 | 4.8 GiB | Default balanced checkpoint for high-quality ASR with lower memory than BF16. |
voxtral-mini-4b-realtime-2602-q4_k.gguf |
Q4_K | 2.9 GiB | Lower-memory and faster checkpoint for CUDA testing and deployment. Validate output quality for your domain. |
Download
Each checkpoint is a single self-contained file, so download only the quantization you intend to run rather than the whole repository.
Q8_0 (default):
hf download mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF \
voxtral-mini-4b-realtime-2602-q8_0.gguf \
--local-dir ./voxtral-realtime-gguf
Q4_K (smaller and faster):
hf download mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF \
voxtral-mini-4b-realtime-2602-q4_k.gguf \
--local-dir ./voxtral-realtime-gguf
Either lands at ./voxtral-realtime-gguf/<filename>.gguf, which is the path to pass as --model.
The hf command ships with huggingface_hub (pip install -U huggingface_hub). On releases before the CLI was renamed, the same call is huggingface-cli download.
Without Python, fetch the file over HTTP instead:
curl -L -o voxtral-mini-4b-realtime-2602-q8_0.gguf \
https://huggingface.co/mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF/resolve/main/voxtral-mini-4b-realtime-2602-q8_0.gguf
Point --model at the .gguf file itself, not at the folder holding it. audio.cpp accepts a directory only when it can narrow it to a single GGUF, so a folder containing both quantizations is rejected rather than guessed at:
model directory contains 2 GGUF files: ./voxtral-realtime-gguf; found: ...
pass one of them directly with --model, or keep a single GGUF in the directory
Performance
Measurements below are from audio.cpp CUDA validation runs. Results vary by GPU, driver, backend, audio length, and decode settings.
Q8_0 vs BF16 reference
BF16 was used only as a local reference baseline for validation. This repository publishes quantized GGUF checkpoints only.
| Mode | BF16 reference | Q8_0 | Q8_0 improvement |
|---|---|---|---|
| Offline ASR speed | 11.1x-12.5x realtime | 14.7x-16.7x realtime | 1.31x-1.38x faster |
| Offline ASR peak VRAM | 10,909 MiB | 7,754 MiB | 3,155 MiB lower |
| Streaming server TTFT | 207.308 ms | 179.896 ms | 27.412 ms lower |
| Streaming client TTFT | 550.526 ms | 530.558 ms | 19.968 ms lower |
| Streaming speed | 4.7x realtime | 5.4x realtime | 1.15x faster |
| Streaming peak VRAM | 12,616 MiB | 8,972 MiB | 3,644 MiB lower |
Q4_K quick check
| Route | Q8_0 RTF | Q4_K RTF | Q4_K vs Q8_0 |
|---|---|---|---|
| Offline short | 0.0862 | 0.0629 | 1.37x faster |
| Offline medium | 0.0643 | 0.0476 | 1.35x faster |
| Offline longer | 0.0576 | 0.0439 | 1.31x faster |
| Offline sampled | 0.0630 | 0.0500 | 1.26x faster |
| Streaming path | 0.1036 | 0.0904 | 1.15x faster |
In the quick validation set, Q4_K transcripts matched Q8_0 except for one capitalization-only difference.
Use with audio.cpp
Build audio.cpp with the helper script for your platform, then use the generated audiocpp_cli binary. The project provides build paths for Linux, Windows, and macOS; see the audio.cpp README for the current build matrix and detailed requirements.
git clone https://github.com/0xShug0/audio.cpp
cd audio.cpp
# Linux: CUDA, Vulkan, or CPU
scripts/build_linux.sh --backend cuda --target audiocpp_cli
# Windows: CUDA or CPU presets
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli
# macOS: Metal
scripts/build_metal.sh --target audiocpp_cli
The examples below assume audiocpp_cli is on your PATH. You can also replace it with the built binary path for your platform, such as build/linux-cuda-release/bin/audiocpp_cli. They pass --backend cuda; use --backend metal on macOS, or --backend cpu when no GPU backend is built.
Run offline ASR with Q8_0:
MODEL=/path/to/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf
audiocpp_cli \
--task asr \
--family voxtral_realtime \
--model "$MODEL" \
--backend cuda \
--threads 8 \
--audio input.wav \
--text-out transcript.txt
Run the Q4_K checkpoint by changing the model path:
MODEL=/path/to/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q4_k.gguf
Streaming ASR
While a stream runs, the CLI reports each update as the text decoded since the previous one, so the updates concatenate into the transcript. On a terminal they are appended and the transcript scrolls like ordinary output; when stdout is redirected, each update is written as its own flushed partial_text= line so pipes and logs stay parseable. --text-out and the final text_output= line carry the complete transcript either way.
Streaming from an audio file:
audiocpp_cli \
--task asr \
--family voxtral_realtime \
--model "$MODEL" \
--backend cuda \
--threads 8 \
--mode streaming \
--audio input.wav \
--text-out transcript.txt
Streaming raw 16 kHz mono PCM from ffmpeg:
ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le - \
| audiocpp_cli \
--task asr \
--family voxtral_realtime \
--model "$MODEL" \
--backend cuda \
--threads 8 \
--mode streaming \
--audio -
Going faster with stream_batch_tokens
A streaming step always advances 80 ms of audio, so each step has to cost under 80 ms to keep pace with a live source. A step is one frontend pass, one audio-encoder forward, and one text-decoder step. Setting voxtral_realtime.stream_batch_tokens=<n> makes a single encoder forward cover n audio tokens instead of one. The decoder still runs once per token, so batching amortizes the encoder's fixed per-dispatch cost and leaves decode work untouched:
audiocpp_cli \
--task asr \
--family voxtral_realtime \
--model "$MODEL" \
--backend cuda \
--threads 8 \
--mode streaming \
--audio input.wav \
--session-option voxtral_realtime.stream_batch_tokens=4
Measured on an Apple M3 MacBook Air with Metal and Q8_0, over a 14 s clip (169 steps, mean of three runs). These are laptop numbers and are not comparable to the CUDA figures earlier in this card; what carries across hardware is the split between the two middle columns.
stream_batch_tokens |
Encoder ms/step | Decoder ms/step | Total ms/step | Streaming speed |
|---|---|---|---|---|
| 1 (default) | 30.8 | 49.0 | 79.9 | 1.04x realtime |
| 2 | 27.5 | 50.2 | 77.7 | 1.07x realtime |
| 4 | 13.3 | 50.0 | 63.4 | 1.31x realtime |
| 8 | 7.4 | 49.6 | 57.1 | 1.46x realtime |
The decoder column is flat, which is the whole shape of this knob: batching can only remove encoder time, so the decoder sets a floor and the gain flattens once the encoder is no longer the bottleneck. 2 is not worth taking — it barely moves the encoder — while 4 captures most of the available win.
What you pay for it is latency. A chunk is not transcribed until all n of its audio tokens have arrived, so each update is delayed by up to n * 80 ms: 0.32 s at 4, 0.64 s at 8. Transcribing a file, that delay costs nothing perceptible; on a live microphone it is the tradeoff to weigh.
Note also that the decoder runs one step per 80 ms whether the audio holds speech or silence, so a session that does fall behind stays behind — the lag does not recover during pauses. Measure your own hardware before relying on a live source.
Prompting and decoding
audio.cpp exposes Voxtral Realtime as an ASR model. Pass audio to the CLI and audio.cpp builds the model's transcription prompt internally; no free-form text prompt is required for normal transcription.
Decode options can be controlled from the request:
audiocpp_cli \
--task asr \
--family voxtral_realtime \
--model "$MODEL" \
--backend cuda \
--threads 8 \
--audio input.wav \
--text-out transcript.txt \
--request-option max_new_tokens=256 \
--do-sample false \
--temperature 1.0 \
--top-p 1.0 \
--top-k 50 \
--seed 1234
Notes
- Task:
asr - Family:
voxtral_realtime - Supported modes: offline and streaming
- Timestamp output is not currently exposed by audio.cpp for this model
- The GGUF package is intended to be standalone: model weights, sidecars, and audio.cpp model spec are embedded in the checkpoint
- Downloads last month
- 161
8-bit