x-vits Β· Expresso ex02 (English, expressive)

An expressive Piper/VITS voice that can speak in six styles and stress the words you mark, or the words it thinks matter. Made with x-vits, a recipe that adds prosody modelling (intonation, rhythm, emphasis, vocal effort) to small, fast VITS voices. It's about 25M parameters in all, exported to ONNX, and runs on a CPU.

Version 0.2. Needs the x-vits 0.2 runtime (pip install from the GitHub repo, below).

What's different from a plain Piper voice

  • 🎭 Styles: default, happy, sad, confused, enunciated, laughing.
  • πŸ’₯ Emphasis: write *word* to stress a word. Without marks, a small model predicts which words to stress.
  • 🌊 Sampled intonation: a flow-matching model picks a plausible pitch, voicing, energy and vocal-effort contour per phoneme, instead of averaging every sentence into the same melody.
  • πŸ“£ Vocal effort: how bright and pressed the voice is, learned per style: happy and laughing are brighter, sad is darker.
  • 🎚️ Style-aware timing: sad slows down, confused hesitates, enunciated takes its pauses.

What's new in 0.2

  • Vocal effort as a prosody feature, measured on voiced sounds and bridged smoothly across consonants and pauses.
  • A wider pitch range in training: the speaker's high notes (happy, laughing) were clipped before.
  • Voicing follows the phonemes, which removes occasional blips (a voiced k, a vowel that dropped its voice).
  • Louder emphasised words, peaking with the pitch.

The 0.1 voice is still playable below, next to 0.2, for comparison. Its files are in this repo's history.

🎧 Samples

About the comparison. "Piper" below is not a flat, neutral voice. It's the same speaker, fine-tuned with plain Piper/VITS on the same expressive recordings in all six styles, so it already has lively prosody: VITS samples its own timing and intonation, and this one learned an average of the six speaking manners. What it can't do is choose: there is no way to ask it for "sad" or to stress a particular word. x-vits adds that control. All synthesised samples use the same random seed and none were cherry-picked.

One sentence, six styles

Plain Piper has a single way to read "You broke the tie score!". x-vits reads it in whichever style you ask for. The speaker's own recordings of this held-out sentence are on the right.

Piper (any style):
stylex-vits 0.2x-vits 0.1recording
default
happy
sad
confused
enunciated
laughing

Measured on these clips, 0.2 follows the speaker more closely than 0.1 in several styles. Happy has the speaker's wide pitch range (15.0 semitones; recording 14.8, 0.1 12.5), sad is as narrow as the recording (5.1 semitones; 0.1 6.7), confused slows down like the speaker (2.30 s; recording 2.53, 0.1 1.94), and laughing moves much more (7.3 semitones; 0.1 4.2, recording 12.6). The voice still can't produce actual laughter.

Different sentences, each in its recorded style

Held-out sentences, never seen in training, each with the recording of that sentence in that style.

styletextPiperx-vits 0.2x-vits 0.1recording
defaultI'm more of an all terrain vehicle type.
happyWe didn't get the weather right, guys.
sadNo, in Salt Lake City, today, it won't snow.
confusedIs young adult fiction very popular?
enunciatedOne PM to nine PM, it'll be cloudy, and in the low fifties.
laughingTheir moment had come, or so they said.

Emphasis, by moving the *mark* (style default)

Piper has no equivalent: it stresses whatever its training suggests.

textx-vits 0.2x-vits 0.1
I never said she stole my money.
*I* never said she stole my money.
I never said *she* stole my money.
I never said she stole my *money*.

πŸš€ Usage

sudo apt install libespeak-ng1            # phonemizer backend (macOS: brew install espeak-ng)
pip install "x-vits @ git+https://github.com/femelo/x-vits"
import soundfile as sf
from xvits.runtime import XVits

voice = XVits.from_pretrained("fdemelo/x-vits")

audio = voice.synthesize("We didn't get the weather right, guys.", style="happy")
sf.write("happy.wav", audio, voice.sample_rate)

# explicit emphasis, a different style, a fixed seed and a faster CPU setting
audio = voice.synthesize("I never said *she* stole my money.", style="confused", seed=7, steps=4)

# a stronger style and a bigger pitch peak on the stressed word
audio = voice.synthesize("I *really* didn't expect that.", style="happy", style_strength=1.5, emphasis_scale=2.0)
sf.write("confused.wav", audio, voice.sample_rate)

Or from the command line:

xvits-speak --model fdemelo/x-vits --text "Is it *really* you?" --style sad --out sad.wav
option values
style default happy sad confused enunciated laughing none gives a neutral mix
emphasis auto Β· marked Β· none auto: marks if present, otherwise predicted
emphasis_scale default 1.5 multiplies the emphasis pitch peak, lengthening and loudness; 1.0 = average emphasis in the recordings
style_strength default 1.0 style exaggeration: 0 none, 2 double (happy: +2.2, +3.5, +4.8 semitones above the speaker's average at 0, 1, 2)
guidance "timing,pitch,energy,effort" per-channel exaggeration; default "1.0,2.0,4.0,1.0"
temperature "timing,pitch,energy" variety; default "0.8,0.7,1.0", higher is livelier
length_scale default 1.08 larger is slower
steps default 16 4 is about 2.5Γ— faster on CPU and changes pitch by about 0.1 semitone
seed int reproducible output

Speed with onnxruntime on 4 CPU threads: real-time factor about 0.6 (16 steps) or 0.25 (4 steps).

πŸ“¦ Files

file role size
encoder.onnx phonemes β†’ text states 26 MB
predictor.onnx one step of the flow-matching prosody predictor (style-conditioned) 31 MB
durations.onnx style-aware stochastic duration predictor 2.5 MB
acoustic.onnx prosody-conditioned prior, length regulation, flow, HiFi-GAN 36 MB
prominence.onnx predicts which words to stress 4.8 MB
dp_cond.npz, emphasis_template.json style vectors for the duration predictor; the emphasis shape small
runtime.json, config.json runtime settings; Piper phoneme map (eSpeak en)

These graphs are driven by xvits.runtime, so they aren't drop-in files for Piper's own runtime. The export was verified against PyTorch end to end (maximum difference about 3e-5 at several sentence lengths).

πŸ§ͺ Training

  • Base: Piper's LJSpeech medium checkpoint (LJSpeech is public domain).
  • Data: Expresso speaker ex02, read speech in six styles, 2.2 h after leaving out whisper, singing, long-form and <laugh> rows. 245 utterances held out.
  • Recipe (x-vits 0.2, xvits-train):
    1. fine-tune the base checkpoint on ex02;
    2. extract per-phoneme duration, pitch, energy and vocal-effort targets;
    3. fine-tune with a zero-initialised prosody input to the prior, including frame-level interpolation;
    4. train an OT-CFM prosody predictor with a style input;
    5. train a style-conditioned duration predictor;
    6. train a word-prominence student on acoustic labels;
    7. fit an emphasis template (pitch, duration, loudness) anchored on the stressed vowel, from Expresso's *emphasis* marks.

Held-out results (129 sentences, two seeds; pitch and timing match are correlations with the recording of the same sentence):

pitch match timing match WER (Whisper) UTMOS
Piper, fine-tuned 0.37 0.30 1.7 % 4.08
x-vits 0.1 0.37 0.39 2.1 % 4.01
x-vits 0.2 0.40 0.37 3.6 % 3.86
the recordings 4.12

UTMOS and Whisper both prefer calm read speech: UTMOS gives the speaker's own laughing takes 2.49, and 0.2 loses most of its score in happy and laughing, the styles that became more expressive. Listen above and decide for yourself.

The prominence student finds the recording's main stress 42 % of the time, against 34 % for "last word" and 15 % for a random word.

⚠️ License and limitations

  • CC BY-NC 4.0. Expresso is licensed for non-commercial use only, and so is this voice. The x-vits code is MIT.
  • English, one female speaker, read speech. Styles are as recorded in Expresso; there are no other emotions.
  • Laughing is rendered through intonation, timing and brightness; the voice doesn't produce actual laughter.
  • Questions sometimes end with a smaller rise than a natural reading.
  • Don't use this voice to impersonate real people or to mislead listeners.

Citation

If you use this voice, please cite Expresso:

@inproceedings{nguyen2023expresso,
  title     = {EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis},
  author    = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and Gat, Itai and
               Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and Synnaeve, Gabriel and Hassid, Michael and
               Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel},
  booktitle = {Interspeech},
  year      = {2023}
}
Downloads last month
32
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train fdemelo/x-vits