Drama-3-MLX-4bit

This is the pure Apple Silicon MLX 4-bit affine quantized release of Fish Audio Drama 3.

Drama 3 is Fish Audio's state-of-the-art 35B parameter (3B active) Mixture-of-Experts (MoE) conversational text-to-speech model featuring a hybrid linear (GatedDeltaNet) + full softmax attention backbone and a 10-codebook Codebook-MTP audio decoder.

This 4-bit MLX release quantizes all 10,240 MoE experts using affine 4-bit quantization (group_size=64, bits=4) while preserving the 44.1kHz Modded DAC neural codec, embeddings, normalization, and MTP decoder tower at full precision.

Performance on Apple Silicon (M3 Max 128GB)

Benchmarked on Apple Silicon M3 Max running pure MLX (zero PyTorch at runtime):

Precision	Model Size	Full Sentence Latency (7.06s Audio)	Step Latency	Real-Time Factor (RTF)	Whisper ASR Accuracy
4-bit (this model)	23.2 GB	24.38 s	160.4 ms/step	3.453	100.0%
8-bit	32.6 GB	28.29 s	210.0 ms/step	4.827	100.0%
BF16	69.0 GB	42.90 s	350.0 ms/step	5.238	100.0%

The 4-bit model runs 1.89x faster than BF16 on Apple Silicon unified memory due to a 75% reduction in memory bandwidth bus traffic.

Features

Zero-Shot Voice Cloning: High-fidelity cloning with reference audio and transcript.
Multi-Speaker Characters: Supports tags <|speaker:0|> to <|speaker:15|> (up to 16 distinct characters in one dialogue).
Point Tags & Emotion: In-text paralinguistic and emotional cues like [sighs], [laughs], [whispers].
44.1kHz High-Fidelity Audio: 10-codebook Modded DAC neural audio codec.
Streaming Output: Yields chunks in real time as frames are generated.

Quick Start

Installation

git clone https://github.com/vanch007/drama-3-mlx.git
cd drama-3-mlx
pip install -e .

CLI Synthesis

# 1. Plain TTS
drama3-mlx tts   --model vanch007/Drama-3-MLX-4bit   --text "你好，世界！这是 Fish Audio Drama 3 的 4-bit MLX 版本。"   --output outputs/sample.wav

# 2. Zero-Shot Voice Cloning
drama3-mlx tts   --model vanch007/Drama-3-MLX-4bit   --ref-audio ref.wav   --ref-text "We asked over twenty different people, and they all said it was his."   --text "Get the trust fund to the bank early, before the storm arrives tonight."   --output outputs/cloned.wav

# 3. Multi-Speaker Dialogue
drama3-mlx tts   --model vanch007/Drama-3-MLX-4bit   --text "<|speaker:0|> 晚上想吃什么？ <|speaker:1|> 随便，听你的就好。"   --output outputs/dialogue.wav

Python API

from drama3_mlx.model import Drama3Model
import soundfile as sf

model = Drama3Model.from_pretrained("vanch007/Drama-3-MLX-4bit")

audio = model.tts(
    "Fish Audio Drama 3 is now running in 4-bit on Apple Silicon.",
    ref_audio="ref.wav",
    ref_text="Transcript of reference audio clip."
)
sf.write("output.wav", audio, 44100)

References & Evaluation

GitHub Repository: https://github.com/vanch007/drama-3-mlx
Upstream Hugging Face: fishaudio/drama-3
Upstream GitHub: fishaudio/fish-speech
Evaluation Methodology: mlx-tts-evaluation

Downloads last month: -; Downloads are not tracked for this model. How to track