harshav/Voxtral-Mini-4B-Realtime-CoreAI

Voxtral Mini 4B Realtime — Core AI

Core AI conversions of mistralai/Voxtral-Mini-4B-Realtime-2602, made from source revision 2769294da9567371363522aac9bbcfdd19447add.

These files require Core AI 1.0.0b2 (macOS 27.0+, Xcode 27.0.0-beta.4+). They are not compatible with Core ML, MLX, GGUF/llama.cpp, or ONNX Runtime. Use voxtral_rt_decoder_int4_qkv_sdpa_cl512 + voxtral_rt_encoder_stream_int4_buf3072.aimodel + voxtral_rt_encoder_first_chunk_float16_buf3072.aimodel unless you have a specific size/speed requirement.

What this is

Voxtral Mini 4B Realtime is Mistral's streaming speech-to-text model (Whisper encoder + MLP adapter + 26-layer Ministral-3 LLM decoder). This repository contains a complete native Core AI port: batch encoder (fp16), streaming encoder (int4), batched first-chunk encoder (fp16), and unified prefill+decode decoder (int4 with fused QKV projection and SDPA fused attention).

The port achieves 4.45s end-to-end for a 12s clip (RTF 0.37), 0.148s streaming TTFT (time-to-first-token, compute-only), and 100% token parity with the Hugging Face reference implementation in both offline and streaming modes.

What will and will not load it

Runtime	Compatible?
Core AI 1.0.0b2+ (macOS 27.0+)	✅ Yes
Core ML (.mlmodelc)	❌ No — different format
MLX	❌ No — different runtime
GGUF / llama.cpp / whisper.cpp	❌ No — different format
ONNX Runtime	❌ No — different format
transformers / PyTorch	❌ No — this is a compiled Core AI asset, not a checkpoint

Artifact manifest

File	Bytes	SHA-256	Role	Recommended?
artifacts/voxtral_rt_decoder_int4_qkv_sdpa_cl512/voxtral_rt_decoder_int4_qkv_sdpa_cl512.aimodel/main.mlirb	2,519,512,295	4ed29e2b363682c8c88b4087b1766735f32912b87815b61e0c35ece2d88c530d	Decoder (int4 + fused QKV + SDPA)	Yes
artifacts/voxtral_rt_decoder_int8_qkv_sdpa_cl512/voxtral_rt_decoder_int8_qkv_sdpa_cl512.aimodel/main.mlirb	4,032,607,471	5867ccfd9bb984b3e68d0e3550d28a3019dc0ffb75a332e034f182b2262b6fc1	Decoder (int8 + fused QKV + SDPA)	Higher quality
artifacts/voxtral_rt_decoder_fp16_qkv_sdpa_cl512/voxtral_rt_decoder_fp16_qkv_sdpa_cl512.aimodel/main.mlirb	6,869,584,654	61b88de4a5eaa16ea6bdeb289911283bf0e17c42611b05fa9b9fdebacd842e14	Decoder (fp16 + fused QKV + SDPA)	Reference
artifacts/voxtral_rt_encoder_stream_int4_buf3072.aimodel/main.mlirb	561,800,107	0777b124f14f5d909233e7c5c6d6b89eea2a84eb93790d02f3db4200144d49b6	Streaming encoder (int4, 8-frame chunks)	Yes
artifacts/voxtral_rt_encoder_first_chunk_float16_buf3072.aimodel/main.mlirb	1,991,831,559	e293cbccfd2bd6a401aff633478221df43d106d8f6a9f4884837a0090b746676	Batched first-chunk encoder (fp16, 312 frames in one call)	Yes
artifacts/voxtral_rt_encoder_float16.aimodel/main.mlirb	1,993,153,653	5323ea58dd4184ebc34717b468fdb34ae6070023c885b0079a6804ec136c314c	Batch encoder (fp16, whole-clip)	Reference only
voxtral-coreai-conversion.tar.gz	1,861,612	0142bb77847d59d68f0f1ad4807e614c5566de7026c70cc4de8fb3551e7f07b6	Conversion source snapshot	For reproduction

Why two encoders? The streaming encoder processes 8-mel-frame chunks incrementally (for real-time streaming). The batched first-chunk encoder processes the initial 312-frame chunk in one graph call (for fast TTFT). Both are needed: the batched encoder handles the first chunk, then the streaming encoder handles subsequent chunks.

Quantization comparison

All variants use weight-only quantization with symmetric_with_clipping per-block-32. The oracle is the upstream BF16 model's greedy token sequence on a 12.02s synthetic say audio fixture.

Variant	Size	Batch e2e	Decode ms/tok	Tokens vs oracle	Recommendation
fp16 + fused QKV + SDPA	6.4 GB	7.93s	45.3	161/161	Reference
int8 + fused QKV + SDPA	3.8 GB	6.11s	35.6	161/161	Higher quality
int4 + fused QKV + SDPA	2.3 GB	4.74s	28.1	161/161	Recommended
int4 + fused QKV + SDPA + batched first chunk	2.3 GB	4.45s	26.6	161/161	Best (batch)

Streaming (5-pass median, matched warmup):

Path	TTFT (compute)	chunk+decode	Tokens
Serialized (39 calls)	1.530s	59.6ms	128/128
Batched (1 call)	0.148s	59.6ms	128/128

The batched first chunk is 10.3x faster for TTFT with zero chunk+decode cost.

WER (synthetic say audio references, not human labels): 0.000 (3s/6s), 0.071 (12s), 0.051 (20s). Token parity (161/161 batch, 128/128 streaming) proves port fidelity; WER measures transcription accuracy against the synthetic references.

Reproduce

Source model: mistralai/Voxtral-Mini-4B-Realtime-2602@2769294da9567371363522aac9bbcfdd19447add Converter: apple/coreai-models@af98e4d3bfcc7f797ff1d2c23251bf15f55dd72c Optimization: apple/coreai-optimization@7ad2df3c2766636f8e76e3f9a2b53a5d9db4b33d Torch converter: apple/coreai-torch@c89f6a44713249a12a84beec9f3e0cf2206ecc38 Toolkit SHA-256: 0142bb77847d59d68f0f1ad4807e614c5566de7026c70cc4de8fb3551e7f07b6

# Set up a clean working directory
export WORKDIR=$(pwd)/voxtral-coreai-repro
mkdir -p $WORKDIR && cd $WORKDIR

# Clone and pin the three dependency repos
git clone https://github.com/apple/coreai-models.git && cd coreai-models && git checkout af98e4d3bfcc7f797ff1d2c23251bf15f55dd72c && cd $WORKDIR
git clone https://github.com/apple/coreai-optimization.git && cd coreai-optimization && git checkout 7ad2df3c2766636f8e76e3f9a2b53a5d9db4b33d && cd $WORKDIR
git clone https://github.com/apple/coreai-torch.git && cd coreai-torch && git checkout c89f6a44713249a12a84beec9f3e0cf2206ecc38 && cd $WORKDIR

# Download source model
hf download mistralai/Voxtral-Mini-4B-Realtime-2602 --local-dir $WORKDIR/Voxtral-Mini-4B-Realtime-2602

# Download and extract conversion toolkit
hf download harshav/Voxtral-Mini-4B-Realtime-CoreAI voxtral-coreai-conversion.tar.gz --local-dir $WORKDIR
cd $WORKDIR && tar xzf voxtral-coreai-conversion.tar.gz

# Run oracle (produces golden tokens + embeddings)
uv run make_oracle.py --model $WORKDIR/Voxtral-Mini-4B-Realtime-2602

# Run streaming oracle (produces streaming golden tokens + chunk mels)
uv run make_oracle_stream.py --model $WORKDIR/Voxtral-Mini-4B-Realtime-2602

# Export encoder (batch)
PYTHONPATH=$WORKDIR/coreai-models/python/src:$WORKDIR/coreai-optimization/src uv run export_encoder.py --model $WORKDIR/Voxtral-Mini-4B-Realtime-2602 --dtype float16

# Export encoder (streaming)
PYTHONPATH=$WORKDIR/coreai-models/python/src:$WORKDIR/coreai-optimization/src uv run export_encoder_stream.py --model $WORKDIR/Voxtral-Mini-4B-Realtime-2602 --mode int4 --buf 3072

# Export encoder (batched first chunk)
PYTHONPATH=$WORKDIR/coreai-models/python/src:$WORKDIR/coreai-optimization/src uv run export_encoder_first_chunk.py --model $WORKDIR/Voxtral-Mini-4B-Realtime-2602 --dtype float16 --buf 3072

# Export decoder (int4 + fused QKV + SDPA)
PYTHONPATH=$WORKDIR/coreai-models/python/src:$WORKDIR/coreai-optimization/src uv run export_decoder.py --model $WORKDIR/Voxtral-Mini-4B-Realtime-2602 --mode int4 --fuse-qkv --use-sdpa

# Gate (batch)
PYTHONPATH=$WORKDIR/coreai-models/python/src:$WORKDIR/coreai-optimization/src uv run gate_e2e.py \
  --decoder artifacts/voxtral_rt_decoder_int4_qkv_sdpa_cl512/voxtral_rt_decoder_int4_qkv_sdpa_cl512.aimodel \
  --compute-encoder gpu --compute-decoder ane

# Gate (streaming, 5-pass median)
PYTHONPATH=$WORKDIR/coreai-models/python/src:$WORKDIR/coreai-optimization/src uv run gate_stream.py \
  --decoder artifacts/voxtral_rt_decoder_int4_qkv_sdpa_cl512/voxtral_rt_decoder_int4_qkv_sdpa_cl512.aimodel \
  --oracle golden_stream --buf 3072 --passes 5 \
  --first-chunk-encoder artifacts/voxtral_rt_encoder_first_chunk_float16_buf3072.aimodel

Usage

Download the recommended artifacts:

hf download harshav/Voxtral-Mini-4B-Realtime-CoreAI \
  artifacts/voxtral_rt_decoder_int4_qkv_sdpa_cl512/voxtral_rt_decoder_int4_qkv_sdpa_cl512.aimodel/main.mlirb \
  artifacts/voxtral_rt_encoder_stream_int4_buf3072.aimodel/main.mlirb \
  artifacts/voxtral_rt_encoder_first_chunk_float16_buf3072.aimodel/main.mlirb \
  --local-dir ./Voxtral-Mini-4B-Realtime-CoreAI

Run with Core AI 1.0.0b2+ on macOS 27.0+ (Apple Silicon). The decoder uses a 512-token KV cache; the streaming encoder uses a 3072-frame buffer (~61s append-only capacity). Longer streams require rolling eviction (not implemented).

Requirements:

• macOS 27.0+ (Darwin 27.0.0+)
• Xcode 27.0.0-beta.4+ (for Core AI runtime)
• Apple Silicon (M4 Max validated; other Apple Silicon should work but unvalidated)
• ~4GB free disk space for the three recommended artifacts
• ~8GB unified memory during inference (model + KV cache + encoder buffers)

Known limitations:

• Streaming capacity is fixed at ~61s (buf=3072 append-only). Rolling KV eviction is not implemented.
• The batch encoder processes whole clips up to 12s. Longer clips require the streaming path.
• The decoder's num_delay_tokens is baked at 6 (480ms delay). Other delay values require re-export.
• ANE execution was attempted but the encoder's ANE compile failed (ANECCompile: MLIR MPS to ANEC conversion failed). The decoder with ANE preference runs ~10% faster but xctrace shows zero ANE hardware intervals — the preference changes codegen but doesn't execute on ANE.

Validation evidence

Machine-readable evidence is in evidence/:

• conversion_gate.json — golden token/embedding shapes from the oracle
• benchmark.json — batch and streaming benchmark results with protocol
• quantization_quality.json — WER against synthetic say references (not human labels)
• artifact_hashes.json — exact sizes and SHA-256 for all published artifacts
• environment.json — pinned revisions for model, converter, runtime, and toolchain

License and attribution

Upstream model: mistralai/Voxtral-Mini-4B-Realtime-2602 by Mistral AI, licensed Apache-2.0.

Base model: mistralai/Ministral-3-3B-Base-2512 by Mistral AI.

Conversion: This repository is a Core AI conversion and does not claim authorship of the upstream model. The conversion toolkit (voxtral-coreai-conversion.tar.gz) is provided for reproduction.

Runtime: Apple Core AI (BSD-3-Clause).

Rejected variants

The following were tested and rejected (not uploaded):

• fp16 decoder (6.4GB, 7.66s e2e) — too large, too slow. Reference only.
• int8 decoder (3.8GB, 6.24s e2e) — larger than int4, slower than int4+QKV+SDPA. No advantage.
• int4 without QKV/SDPA (2.3GB, 5.44s e2e) — slower than the fused variant. No advantage.
• K/V state fusion (52→26 writes) — engine segfault. Not viable.
• Baked AdaRMSNorm constants — 39.8 ms/tok (regression from 36.3). Index op dispatch worse than fused matmuls.
• Broadcast GQA — 34.1 ms/tok (marginal regression). Reshape overhead > repeat_interleave copy.
• Fused write+fetch (mutable_cache_update_and_fetch) — no improvement (34.6 vs 34.0 warm).