fishaudio/drama-3-int4

Drama 3 · int4 (hybrid W4A16)

The smallest working Drama 3. Unofficial int4 quantization of fishaudio/drama-3 — Fish Audio's 35B (3B active) multi-speaker TTS model — with real eval numbers and one-command serving through SGLang Omni.

Thanks so much to AK for the quantization!

• 22.1 GB download (vs 41.5 GB original) — routed experts and selected attention projections use int4; the shared expert and audio decoder stay in bf16.
• Runs on Ampere (A100, sm80) — GPTQ-Marlin requires compute capability 8.0 or newer.
• Published WER and speaker-similarity evaluations, plus an FP8-vs-int4 comparison on Seed-TTS English.
• 5 verified sample types: plain TTS, voice cloning, [sighs]/[laughs] tags, long-form, Chinese.

Samples (all generated by this checkpoint)

prompt	clip
plain TTS	int4_plain.wav
voice clone (reference clip)	int4_clone.wav
clone + [sighs]/[laughs] tags	int4_tags.wav
long-form	int4_long.wav
Chinese	int4_zh.wav

Eval

Seed-TTS English: FP8 vs int4

The two checkpoints were compared on one H200 with the default SGLang Omni server settings: the 1,088 Seed-TTS English prompts, each cloning the voice of its own reference clip, sent as non-streaming requests at concurrency 8. WER is computed on the generated audio transcribed by Qwen3-ASR-1.7B.

Checkpoint	Weights	Latency mean	Latency p95	RTF mean	Throughput (req/s)	WER (corpus)	WER without samples above 50%
fishaudio/drama-3 (FP8 experts)	37.0 GB	1.235 s	1.902 s	0.281	6.47	1.99%	1.32%
akhaliq/drama-3-int4	20.8 GB	1.334 s	2.178 s	0.320	5.99	1.63%	1.42%

Drama 3 FP8 and int4 checkpoints on Seed-TTS English, concurrency 8, one H200.

The int4 checkpoint is about 8% slower per request and its WER stays at the FP8 level: the gap between the two is about as large as the run-to-run spread of one checkpoint (a second FP8 run measured 1.365 s mean latency and 2.02% WER).

Community-quant comparison: Whisper large-v3 WER + resemblyzer speaker similarity

An additional evaluation compared this checkpoint against darioooooo0o/drama-3-w4a16-32gb (int8 dense layers, targeting 32 GB cards), using the same 8 prompts, the same scorer, and one A100.

	mean WER ↓	mean WER excl. zh¹ ↓	speaker sim ↑
this repo (int4 hybrid, 22.1 GB)	0.197	0.082	0.846
comparison checkpoint (int8 dense, 24.5 GB)	0.259	0.154	0.846

¹ The Chinese prompt scored WER 1.0 for both checkpoints despite nearly identical transcripts — a jiwer tokenization artifact on unspaced Chinese, not a model difference. Full per-prompt rows in eval/eval_results.json; clips in eval/.

Notable per-prompt outcomes: the comparison checkpoint hallucinated an ending on the long paragraph (WER 0.667 vs our 0.028); ours truncated the tag line's tail on one take (0.333 vs their 0.25). Speaker similarity is a tie.

This small-prompt evaluation uses a different dataset and ASR scorer from the Seed-TTS evaluation above; the WER values should not be compared across the two evaluations.

Quickstart

Install the current SGLang Omni runtime with Fish Audio support:

git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni

uv venv -p 3.12 .venv
. .venv/bin/activate
uv pip install --prerelease=allow -e ".[fishaudio]"

Download and serve the checkpoint:

hf download akhaliq/drama-3-int4

sgl-omni serve \
  --model-path akhaliq/drama-3-int4 \
  --port 8000

The server reads its quantization from the checkpoint, so it starts with the same command as the original model. No manual loader patch is required with the current runtime.

Then use the standard Drama 3 API: /v1/audio/speech with input, optional references for voice cloning, [sighs]/[laughs] tags, multi-speaker scenes, and previous_takes editing all work as on the original.

Engine weight memory: 20.8 GB instead of 37.0 GB for the original checkpoint, which leaves more of the engine's share for the KV cache. This is the engine's weight footprint, not total serving memory.

An earlier A100-80GB test measured 57.3 GB GPU memory at startup and 57.9 GB peak across the five sample generations, using explicit memory fractions of --tts_engine.gpu_memory_fraction 0.62 --vocoder.gpu_memory_fraction 0.2. Those measurements used different settings from the default-server H200 benchmark above.

What we quantize

component	original	here	notes
text embeddings, output head, router	bf16	bf16	retained in bf16
attention q/k/v/o_proj of full attention layers	bf16	int4	symmetric 4-bit weights, group size 128
linear attention in_proj_qkv, in_proj_z, out_proj	bf16	int4	symmetric 4-bit weights, group size 128
linear attention in_proj_a/b	bf16	bf16	retained in bf16
shared-expert MLP	bf16	bf16	int4 passed numerical checks but produced zero audio frames in the earlier serving test; see shared_expert_verdict.md
routed MoE experts (30,720 matrices)	FP8 (block)	int4	the bulk: 32.2 GB → 16.6 GB
codebook-MTP audio decoder + codec	bf16	bf16	retained in bf16

The routed MoE experts and quantized attention projections hold symmetric 4-bit weights with one scale per 128 input channels, and run on GPTQ-Marlin kernels. GPTQ-Marlin requires compute capability 8.0 or newer.

Quantization method: RTN + per-group clip search (group 128, symmetric), no calibration data. Weights are stored in GPTQ format; GPTQ storage and kernels do not imply that calibration-based GPTQ optimization was used.

Reproduce with quantize.py. Quantization statistics are in quant_stats.json.

The repository also retains apply_patch.py for the earlier patched-loader workflow. That patch fixes fused-name matching (gate_up_proj/qkv_proj/in_proj_qkvz), without which the earlier loader could silently miss int4 weights and return zero audio frames. The current SGLang Omni runtime reads the checkpoint's quantization configuration directly.

Limitations

• RTN, no calibration — a GPTQ/AutoRound pass with real calibration data may improve quality; the building blocks are in quantize.py.
• Tags prompts occasionally truncate a take — regenerate if needed.
• The 8-prompt community-quant comparison is a small sample; use the larger Seed-TTS evaluation for the reported FP8-vs-int4 comparison.
• Serving has been tested on A100-80GB and H200. Serving on 48 GB and 32 GB cards remains unverified; weight size alone does not establish that the complete serving pipeline fits.

License

Derivative Work of Fish Audio Materials under the Fish Audio Community License v1.0 (LICENSE) — same terms as the original model, including: research/non-commercial use free; commercial use only under USD 500k annual revenue; no hosted TTS/voice-cloning service without a separate license; no cloning real people's voices without consent; do not remove watermarks; do not use outputs to train other speech or foundation models.

Credits

• Model: Fish Audio — fishaudio/drama-3 · tech report
• Quantization: AK — akhaliq/drama-3-int4
• Runtime: SGLang Omni and SGLang int4 kernels · GitHub stars
• Companion quants: darioooooo0o/drama-3-w4a16 (experts-only int4, first) · darioooooo0o/drama-3-w4a16-32gb (int8 dense, 32GB-card target)