pentacoxian-dev/drama3-expressive-medium-GGUF

drama3-expressive-medium: Fish Audio Drama 3 that follows acting directions mid-line, in 24 GB

Built From Fish Audio. This is a fine-tuned and quantized derivative of Fish Audio Drama 3. Fish Audio does not sponsor or endorse it.

Drama 3 follows an acting direction at the start of a line, but it mostly ignores one placed in the middle: [calm] It's fine, really. [voice breaking] I just miss her. tends to stay calm, or to go silent at the second direction. drama3-expressive-medium was fine-tuned to change delivery where the direction says. It also speaks the whole line and ends cleanly far more often. It keeps all 256 experts and runs in 24 GB of VRAM.

File	drama3-expressive-medium-IQ4_XS.gguf, 20.97 GiB
VRAM	Peak 21.9 GiB on an RTX 4090 (30 s uncached reference, 32 s long-form output, streaming or offline with the optional patch)
Speed	About 2.6× real time on an RTX 4090, including encoding the 29 s reference
Languages	Japanese, English (directions are written in English for both)
Features	Zero-shot cloning from about 10–30 s of reference audio, or a voice made up per seed with no reference; inline acting directions at the start of a line and mid-line
Runtime	audio.cpp main at 71f42b84 or later (CUDA); the optional drama3-expressive-medium-audiocpp-optional.patch adds the pause fix and low-VRAM offline decoding. Or the standalone PyTorch runtime in torch/ (GGUF only, 23.1 GiB on a 4090)

Samples

No reference audio: each sample's seed makes up its own voice, speaking the line's language (--task tts with no voice_ref). The first table keeps the words fixed and changes only the opening direction; the second changes the delivery mid-line.

• Voices: none is cloned from a recording. By speaker-verification similarity (WavLM-SV), no sample is closer to a fine-tuning speaker than 95% of the fine-tuning speakers are to their nearest other speaker.
• Runtime: audio.cpp 71f42b84 with the optional patch.
• Settings: temperature 0.7, top-p 0.7, top-k 30, pause fix on with this repo's silence_codes.json.
• Processing: loudness-normalized per take, so a whispered take plays at about the level of the others, with 0.1 s of silence added at each end. Nothing else.

Same line, four directions (a different voice in each take; seeds: English 9079, 9035, 9047, 9083; Japanese 9080, 9062, 9070, 9077)

Direction	You knew. You knew the whole time.	うそでしょ…本当にやったの？信じられない。
[overjoyed, laughing]
[furious, shouting]
[heartbroken, crying]
[terrified, whispering]

Mid-line changes

Sample	Prompt	Audio
JA, whisper → shout, seed 9086	[whispering, scared] ねえ、静かにして。誰かいる…[shouting, furious] ちょっと！人の部屋で何してるの！
JA, laughing → crying, seed 9153	[laughing, overjoyed] やった、合格したよ！ずっと頑張ってきたんだ！[crying, heartbroken] …でも、一番に伝えたかったおばあちゃんは、もういないんだ。
JA, bored → excited, seed 9106	[bored, flat monotone] はい、次の方どうぞ。ご用件は…[shocked, screaming with excitement] えっ、うそ、本物！？サインください！
EN, whisper → shout, seed 9135	[whispering, scared] Shh, keep your voice down. Someone's in the house... [shouting, furious] Hey! Get out of my kitchen!
EN, crying → laughing, seed 9122	[sobbing, heartbroken] I studied every night, and I still failed... [laughing, overjoyed] Wait, what? I passed? I actually passed!
EN, bored → excited, seed 9146	[bored, monotone] Next, please. Name and ticket number... [shocked, excited] Wait, is that really you? Oh my gosh, can I get a photo?

Writing directions

• Put a direction in square brackets at the start of the line, and another right before the word where the delivery should change: [cheerful] Good morning! [suddenly worried] Wait, where's my phone?
• Write directions in English for Japanese lines too. 2–8 words work best. Lead with the feeling (angry, heartbroken, amused, nervous, tender), then add how it is voiced if useful (whispering, voice breaking, through laughter, rushed).
• Directions describe delivery only; the model does not speak them.

What changed compared with Drama 3

Part	Drama 3	drama3-expressive-medium
Routed experts	256 per layer, FP8	All 256 kept; IQ4_XS, Q5_K in layers 0/1/2/36, with a speech imatrix (16.4 GiB). Not trained
Attention / Gated DeltaNet, shared experts	BF16	Fine-tuned (rank-32 LoRA, merged), then Q8_0
Embeddings, audio tower (MTP)	BF16	Q8_0
Output head	Full vocabulary	EOS + 4,096 semantic rows (speech only)
Codec	F16	F16 (unchanged)

Training

All stages train the same LoRA adapters while the routed experts stay frozen as the shipped IQ4_XS / Q5_K banks, so what was trained is what ships.

1. Supervised fine-tuning on emotional speech.: Every line was given inline acting directions by Qwen3.8-Omni-Flash from the audio, with a calibrated prompt; they were not reviewed by a person. Directions that only repeat the opening's emotion were removed.
2. Mid-line contrast. 2,322 spliced lines: two takes of the same speaker in different emotions, joined at a short pause and kept only where Qwen heard the change. A loss term makes the audio after the splice more likely with the right mid-line direction than with none or a wrong one.
3. Recovery. 3,000 lines with a natural pause stretched to 1–3 s, or silence appended after the last word. The added silence is not scored; resuming the next word and ending the take are.
4. Preference optimization (DPO) on the model's own takes. 4,214 pairs from 4–6 takes each of about 2,660 lines. Preferred: no long pause, no runaway, and following the mid-line directions (judged by Qwen3.8-Omni-Flash). The words must be right in both takes of a pair.

Throughout, 25% of each batch distills the original FP8 Drama 3 on plain Japanese and English speech, to keep ordinary speech close to the original.

Evaluation

All numbers use one take per line, sampling as above, audio.cpp 71f42b84 with the optional patch and the pause fix on, and Qwen3.8-Omni-Flash as an automatic judge. The judge under-detects audible changes: the repo owner heard mid-line changes in takes where the judge did not.

Mid-line directions, held-out test. 178 two-part lines, mostly from speakers not seen in training. "Requested delivery" = after the second direction, the judge hears a delivery with the same emotion as the direction asked for. Each cell is with the mid-line direction / without it; the gain is the difference.

Model	Requested delivery (gain)	Second half spoken	Runaway (≥ 20 s)	Pause at the change, median / 90th pct.
Drama 3 (Q5_K experts)	21% / 24% (−3)	68% / 79%	45% / 29%
Drama 3, this repo's quantization	28% / 33% (−5)	67% / 83%	40% / 29%	1.54 / 5.86 s
drama3-expressive-medium	46% / 30% (+16)	98% / 100%	5% / 3%	0.38 / 1.89 s
Original recordings				0.64 / 1.53 s

The two Drama 3 rows were rendered with the earlier drama3-small runtime build, without the pause fix.

On a separate dev set (120 lines): requested delivery 50% / 35% (+15); runaways 2% / 2%; takes with any pause over 0.7 s 14%, against 17% for the original recordings.

Plain speech. drama3-small's five listening sets: 172 takes, 7 voices, Japanese and English, tags and long-form. A take fails if it runs on, drops content, or comes out in the wrong voice.

Model	Failures
Drama 3, this repo's quantization	27 / 172
drama3-expressive-medium	7 / 172

Speaker similarity on takes both models pass is unchanged (median within ±0.002).

Listening. On 10 random dev lines, the repo owner preferred this model over the untuned quantization on 9 and rated 1 a tie (blind).

Distance from the original on plain speech (teacher-forced KL over the 4,097 speech candidates, 60 held-out Japanese and English pairs): 0.048, against 0.013 for the untuned quantization.

Usage

audio.cpp (recommended)

Build upstream audio.cpp. The optional patch adds the pause fix and decodes offline takes in small batches, so offline mode fits a 24 GB card:

git clone https://github.com/0xShug0/audio.cpp && cd audio.cpp
git checkout 71f42b84
git apply /path/to/drama3-expressive-medium-audiocpp-optional.patch   # optional, recommended
scripts/build_linux.sh --cuda on --models fish_drama3

Clone a voice from the command line. requests.json holds {"requests": [{"id", "text", "voice_ref", "options": {"reference_text": ...}, "seed", "temperature": 0.7, "top_k": 30, "top_p": 0.7, "max_tokens": 600}]}. Give the sampling settings explicitly: upstream's defaults (temperature 1.0, top-p 0.9, top-k 4097) were not evaluated with this model.

AUDIOCPP_DRAMA3_MAX_PAUSE_FRAMES=10 AUDIOCPP_DRAMA3_SILENCE_CODES=silence_codes.json \
build/linux-cuda-release/bin/audiocpp_cli --task clon --family fish_drama3 --backend cuda --mode offline \
  --model drama3-expressive-medium-IQ4_XS.gguf --request-sequence requests.json --out-dir out \
  --session-option fish_drama3.codec_encoder_chunk_samples=131072 \
  --session-option fish_drama3.cpu_expert_layers=0 --session-option fish_drama3.context_length=4096

• codec_encoder_chunk_samples=131072 encodes the reference in chunks. Without it, a 30 s reference runs out of memory on a 24 GB card.
• cpu_expert_layers=0 keeps all experts on the GPU. Upstream's default of 4 moves some to the CPU.
• Without the optional patch:
  • Use --mode streaming on 24 GB cards; offline mode runs out of memory on takes of about 30 s.
  • The pause fix is unavailable, so expect longer pauses than reported here.
• With audiocpp_server: pass "options": {"max_pause_frames": "10", "silence_codes": "silence_codes.json"} per request.
• Long server sessions on 24 GB cards: over 240 requests in one process the peak reached 22.0 GiB, because upstream's prefill-graph caches grow. Lowering fish_drama3.backbone_prefill_cache_mb and fish_drama3.tower_prefill_cache_mb from 256 adds headroom; this was not measured.

Pause fix (optional patch). After max_pause_frames consecutive silent frames (10 = 0.47 s), the codes in silence_codes.json are masked out of the next draw, so the model has to resume or end. A length guard ends takes that run far past the length expected from the text. This repo's silence_codes.json has 247 codes, refit on this model's renders.

Optional clean-up. The model is about 3.5 dB louder than Drama 3 and leaves a little more room tone between words. postprocess.py in.wav out.wav --gate loudness-matches a take and pulls the gaps toward silence.

PyTorch

torch/ runs the model from the GGUF file alone, with no audio.cpp and without the original checkpoint. It contains the vendored Fish Audio model code (Fish Audio Community License), a GGUF reader, and Triton kernels that dequantize the IQ4_XS / Q5_K experts on the fly. The generation loop follows audio.cpp's, including the pause fix and the length guard. Details: torch/README_torch.md.

cd torch && pip install -r requirements.txt     # Linux x86_64, CUDA
python infer.py --gguf ../drama3-expressive-medium-IQ4_XS.gguf \
  --text "[calm] It's fine, really. [voice breaking] I just miss her." \
  --ref-audio voice.wav --ref-text "Transcript of voice.wav." --out take.wav --seed 1

	RTX 4090 (24 GB)
Peak VRAM	23.1 GiB, about 0.6 GiB free. Dense weights are kept in bf16; close other GPU programs
Speed	Decoding 62.5 frames/s (2.9× real time); 2.0–2.5× real time per take including prefill and the codec
Fidelity	Teacher-forced KL 0.004 against the training model on 4 held-out sequences. Free-running word error rate and take length match audio.cpp over 45 takes

• Not the same takes as audio.cpp: the random number streams differ, so the same seed gives a different take.
• Not supported: CPU inference, streaming output and multiple GPUs. Multiple references and continuation history work from Python (Drama3TTS) but not from the CLI.

Limitations

• Runtime: needs audio.cpp 71f42b84 or later. Not llama.cpp. The pause fix and low-VRAM offline decoding come only with the optional patch, and the numbers above assume it.
• Mid-line directions are followed more often, not always. About half the time on held-out lines, by an automatic judge.
• Long pauses remain on some lines.
  • On held-out lines with a mid-line direction, 28% of takes contain a pause over 0.7 s, against 19% in real recordings.
  • Sad or calm narration can come out slow, with 1–2.5 s pauses.
  • These pauses are quiet breaths or room sound rather than the listed silence codes, so the pause fix does not shorten them.
• Takes often start at full level. In drama3-small's listening sets, 71% of takes reach within 30 dB of their speech level in the first 30 ms (92% for the untuned quantization), so the first syllable can sound clipped.
• Directions were labelled automatically. Qwen3.8-Omni-Flash wrote them; no human reviewed the training labels.
• One take per line in every evaluation above. Treat differences of a few points as noise.

License and acceptable use

Licensed under the Fish Audio Community License v1.0 (see LICENSE).

Do not use this model or its output to:

• imitate the voice of an identifiable person without their consent
• impersonate anyone to deceive, defraud or harass
• produce sexual content involving minors or non-consenting real people
• remove watermarks or provenance signals

If you offer voice cloning built on this model to others, you must bind them to restrictions at least as protective as these.

Acknowledgements

• Fish Audio Drama 3 and its codec.
• audio.cpp for the runtime, including the i-quant and speech-only-head support contributed for drama3-small (#958, #959).