huybik/hibiki-zero-3b-mlx-q4

Hibiki-Zero 3B — MLX 4-bit

4-bit (MLX) quantization of kyutai/hibiki-zero-3b-pytorch-bf16, a simultaneous speech-to-speech + speech-to-text translation model (FR / ES / PT / DE → EN) with voice transfer.

The language model is quantized to 4-bit (group_size=32), shrinking it from 5.8 GB → 2.2 GB. On an Apple M4 Pro this runs at ~1.3× real-time (16.5 tok/s @ 12.5 Hz), faster than the bf16 PyTorch/MPS path. The Mimi codec is kept separate in bf16.

group_size=32 is intentional: stock moshi-mlx (and the moshi-swift iOS loader) hardcode gs32 for .q4.safetensors, so a larger group size would not load without patching the loader.

Results (Apple M4 Pro, leon.wav, FR→EN)

	Value
LM weights	5.8 GB bf16 → 2.2 GB q4 (578 layers quantized)
Speed	16.5 tok/s ≈ 1.3× real-time — vs 0.7× for the PyTorch/MPS path (1.9× faster)
Quality	Coherent FR→EN — correctly translated the Léon Marchand / Paris 2024 Olympics commentary; minor q4 artifacts (e.g. one "Paris 1024" slip)
Audio out	24 kHz wav ✅

Files

File	Description
hibiki.q4.safetensors	4-bit quantized LM (MLX)
config.json	model config (needed to build the LmConfig)
mimi-pytorch-e351c8d8@125.safetensors	Mimi codec (bf16)
tokenizer_spm_48k_multi6_2.model	SentencePiece tokenizer
mlx_hibiki_patch.py	runtime patches for moshi-mlx (required)
verify_mlx_q4.py	example inference script

Why the patch is required

Stock moshi-mlx (0.3.0) targets moshi / older hibiki and misses three hibiki-zero deltas, so these weights will not load without mlx_hibiki_patch.py:

1. config — honour hidden_scale (feedforward dim) and kv_repeat=2 instead of the hardcoded 4*dim / kv_repeat=1.
2. attention — grouped-query attention in the forward pass (the main transformer uses kv_repeat=2).
3. positional embedding — rope_concat == RoPE with interleave=False (MLX traditional=False).

Usage

pip install moshi-mlx

python - <<'PY'
import mlx_hibiki_patch  # patches moshi_mlx for hibiki-zero — import first
from moshi_mlx import run_inference
import sys
sys.argv = [
    "run_inference",
    "--lm-config", "config.json",
    "--moshi-weights", "hibiki.q4.safetensors",
    "--mimi-weights", "mimi-pytorch-e351c8d8@125.safetensors",
    "--tokenizer", "tokenizer_spm_48k_multi6_2.model",
    "input_fr.wav", "output_en.wav",
]
run_inference.main()
PY

See verify_mlx_q4.py for a ready-to-run example.

License

Inherits CC BY-NC-SA 4.0 (non-commercial, share-alike) from the base model kyutai/hibiki-zero-3b-pytorch-bf16.