CtrlSpeech
Checkpoints for CtrlSpeech — expressive speech synthesis with coarse-to-fine latent control. Code, CLI and interactive demo: https://github.com/zszheng147/CtrlSpeech
CtrlSpeech is a zero-shot TTS model you can steer after generating. It exposes the pitch contour, loudness contour and phoneme boundaries of any utterance, lets you edit one of them, and resynthesises following the edit while leaving the others alone. Prosody is conditioned per phoneme token, so a single word can be stretched while the rest of the sentence keeps its timing.
Contents
| Path | Params | Prosody control | Notes |
|---|---|---|---|
control-600m/ |
692 M | pitch · loudness · duration | Default |
control-150m/ |
150 M | pitch · loudness · duration | Smaller |
base-600m/ |
689 M | — | Ablation baseline, no control |
base-150m/ |
148 M | — | Ablation baseline, no control |
svae/ |
— | — | SVAE (DAC + BigVGAN) 16 kHz codec, 40 Hz latents |
shared/campplus.onnx |
— | — | CAM++ speaker embedding |
shared/vocab.json |
— | — | Phoneme vocabulary (ARPAbet) |
Each model folder holds model.safetensors, a training-free config.yaml, and
the qwen_config.json describing the AR backbone architecture. The fine-tuned
backbone weights are inside the checkpoint; no Qwen weights are redistributed
and none are downloaded at load time.
Usage
pip install git+https://github.com/zszheng147/CtrlSpeech
from ctrlspeech import CtrlSpeech, shift_pitch_semitones
tts = CtrlSpeech.from_pretrained("control-600m") # downloads on first use
baseline = tts.from_audio("clip.wav", "the transcript of that clip")
tts.regenerate(baseline, pitch=shift_pitch_semitones(baseline.gen_f0, 5)) \
.save("higher.wav")
Duration editing and adopting your own recording require
Montreal Forced Aligner with
the english_us_arpa acoustic model and dictionary.
Control representation
Analysis runs at 100 frames/s (16 kHz, hop 160).
- Pitch — 128 mel-spaced bins; bin 0 is unvoiced.
- Loudness — 64 A-weighted dB bins, about 1.05 bins per dB.
- Duration — per-phoneme frame counts, at most 191 frames (1.91 s) per phoneme; an edited timeline is capped at 2001 frames (20 s).