Instructions to use fdemelo/x-vits with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Piper
How to use fdemelo/x-vits with Piper:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
x-vits Β· Expresso ex02 (English, expressive)
An expressive Piper/VITS voice that can speak in six styles and stress the words you mark, or the words it thinks matter. Made with x-vits, a recipe that adds prosody modelling (intonation, rhythm, emphasis, vocal effort) to small, fast VITS voices. It's about 25M parameters in all, exported to ONNX, and runs on a CPU.
Version 0.2. Needs the x-vits 0.2 runtime (pip install from the GitHub repo, below).
What's different from a plain Piper voice
- π Styles:
default,happy,sad,confused,enunciated,laughing. - π₯ Emphasis: write
*word*to stress a word. Without marks, a small model predicts which words to stress. - π Sampled intonation: a flow-matching model picks a plausible pitch, voicing, energy and vocal-effort contour per phoneme, instead of averaging every sentence into the same melody.
- π£ Vocal effort: how bright and pressed the voice is, learned per style: happy and laughing are brighter, sad is darker.
- ποΈ Style-aware timing: sad slows down, confused hesitates, enunciated takes its pauses.
What's new in 0.2
- Vocal effort as a prosody feature, measured on voiced sounds and bridged smoothly across consonants and pauses.
- A wider pitch range in training: the speaker's high notes (happy, laughing) were clipped before.
- Voicing follows the phonemes, which removes occasional blips (a voiced k, a vowel that dropped its voice).
- Louder emphasised words, peaking with the pitch.
The 0.1 voice is still playable below, next to 0.2, for comparison. Its files are in this repo's history.
π§ Samples
About the comparison. "Piper" below is not a flat, neutral voice. It's the same speaker, fine-tuned with plain Piper/VITS on the same expressive recordings in all six styles, so it already has lively prosody: VITS samples its own timing and intonation, and this one learned an average of the six speaking manners. What it can't do is choose: there is no way to ask it for "sad" or to stress a particular word. x-vits adds that control. All synthesised samples use the same random seed and none were cherry-picked.
One sentence, six styles
Plain Piper has a single way to read "You broke the tie score!". x-vits reads it in whichever style you ask for. The speaker's own recordings of this held-out sentence are on the right.
| Piper (any style): | |||
|---|---|---|---|
| style | x-vits 0.2 | x-vits 0.1 | recording |
| default | |||
| happy | |||
| sad | |||
| confused | |||
| enunciated | |||
| laughing | |||
Measured on these clips, 0.2 follows the speaker more closely than 0.1 in several styles. Happy has the speaker's wide pitch range (15.0 semitones; recording 14.8, 0.1 12.5), sad is as narrow as the recording (5.1 semitones; 0.1 6.7), confused slows down like the speaker (2.30 s; recording 2.53, 0.1 1.94), and laughing moves much more (7.3 semitones; 0.1 4.2, recording 12.6). The voice still can't produce actual laughter.
Different sentences, each in its recorded style
Held-out sentences, never seen in training, each with the recording of that sentence in that style.
| style | text | Piper | x-vits 0.2 | x-vits 0.1 | recording |
|---|---|---|---|---|---|
| default | I'm more of an all terrain vehicle type. | ||||
| happy | We didn't get the weather right, guys. | ||||
| sad | No, in Salt Lake City, today, it won't snow. | ||||
| confused | Is young adult fiction very popular? | ||||
| enunciated | One PM to nine PM, it'll be cloudy, and in the low fifties. | ||||
| laughing | Their moment had come, or so they said. |
Emphasis, by moving the *mark* (style default)
Piper has no equivalent: it stresses whatever its training suggests.
| text | x-vits 0.2 | x-vits 0.1 |
|---|---|---|
| I never said she stole my money. | ||
| *I* never said she stole my money. | ||
| I never said *she* stole my money. | ||
| I never said she stole my *money*. |
π Usage
sudo apt install libespeak-ng1 # phonemizer backend (macOS: brew install espeak-ng)
pip install "x-vits @ git+https://github.com/femelo/x-vits"
import soundfile as sf
from xvits.runtime import XVits
voice = XVits.from_pretrained("fdemelo/x-vits")
audio = voice.synthesize("We didn't get the weather right, guys.", style="happy")
sf.write("happy.wav", audio, voice.sample_rate)
# explicit emphasis, a different style, a fixed seed and a faster CPU setting
audio = voice.synthesize("I never said *she* stole my money.", style="confused", seed=7, steps=4)
# a stronger style and a bigger pitch peak on the stressed word
audio = voice.synthesize("I *really* didn't expect that.", style="happy", style_strength=1.5, emphasis_scale=2.0)
sf.write("confused.wav", audio, voice.sample_rate)
Or from the command line:
xvits-speak --model fdemelo/x-vits --text "Is it *really* you?" --style sad --out sad.wav
| option | values | |
|---|---|---|
style |
default happy sad confused enunciated laughing |
none gives a neutral mix |
emphasis |
auto Β· marked Β· none |
auto: marks if present, otherwise predicted |
emphasis_scale |
default 1.5 | multiplies the emphasis pitch peak, lengthening and loudness; 1.0 = average emphasis in the recordings |
style_strength |
default 1.0 | style exaggeration: 0 none, 2 double (happy: +2.2, +3.5, +4.8 semitones above the speaker's average at 0, 1, 2) |
guidance |
"timing,pitch,energy,effort" |
per-channel exaggeration; default "1.0,2.0,4.0,1.0" |
temperature |
"timing,pitch,energy" |
variety; default "0.8,0.7,1.0", higher is livelier |
length_scale |
default 1.08 | larger is slower |
steps |
default 16 | 4 is about 2.5Γ faster on CPU and changes pitch by about 0.1 semitone |
seed |
int | reproducible output |
Speed with onnxruntime on 4 CPU threads: real-time factor about 0.6 (16 steps) or 0.25 (4 steps).
π¦ Files
| file | role | size |
|---|---|---|
encoder.onnx |
phonemes β text states | 26 MB |
predictor.onnx |
one step of the flow-matching prosody predictor (style-conditioned) | 31 MB |
durations.onnx |
style-aware stochastic duration predictor | 2.5 MB |
acoustic.onnx |
prosody-conditioned prior, length regulation, flow, HiFi-GAN | 36 MB |
prominence.onnx |
predicts which words to stress | 4.8 MB |
dp_cond.npz, emphasis_template.json |
style vectors for the duration predictor; the emphasis shape | small |
runtime.json, config.json |
runtime settings; Piper phoneme map (eSpeak en) |
These graphs are driven by xvits.runtime, so they aren't drop-in files for Piper's own runtime.
The export was verified against PyTorch end to end (maximum difference about 3e-5 at several sentence lengths).
π§ͺ Training
- Base: Piper's LJSpeech
mediumcheckpoint (LJSpeech is public domain). - Data: Expresso speaker ex02, read speech in six styles,
2.2 h after leaving out whisper, singing, long-form and
<laugh>rows. 245 utterances held out. - Recipe (x-vits 0.2,
xvits-train):- fine-tune the base checkpoint on ex02;
- extract per-phoneme duration, pitch, energy and vocal-effort targets;
- fine-tune with a zero-initialised prosody input to the prior, including frame-level interpolation;
- train an OT-CFM prosody predictor with a style input;
- train a style-conditioned duration predictor;
- train a word-prominence student on acoustic labels;
- fit an emphasis template (pitch, duration, loudness) anchored on the stressed vowel, from Expresso's
*emphasis*marks.
Held-out results (129 sentences, two seeds; pitch and timing match are correlations with the recording of the same sentence):
| pitch match | timing match | WER (Whisper) | UTMOS | |
|---|---|---|---|---|
| Piper, fine-tuned | 0.37 | 0.30 | 1.7 % | 4.08 |
| x-vits 0.1 | 0.37 | 0.39 | 2.1 % | 4.01 |
| x-vits 0.2 | 0.40 | 0.37 | 3.6 % | 3.86 |
| the recordings | 4.12 |
UTMOS and Whisper both prefer calm read speech: UTMOS gives the speaker's own laughing takes 2.49, and 0.2 loses most of its score in happy and laughing, the styles that became more expressive. Listen above and decide for yourself.
The prominence student finds the recording's main stress 42 % of the time, against 34 % for "last word" and 15 % for a random word.
β οΈ License and limitations
- CC BY-NC 4.0. Expresso is licensed for non-commercial use only, and so is this voice. The x-vits code is MIT.
- English, one female speaker, read speech. Styles are as recorded in Expresso; there are no other emotions.
- Laughing is rendered through intonation, timing and brightness; the voice doesn't produce actual laughter.
- Questions sometimes end with a smaller rise than a natural reading.
- Don't use this voice to impersonate real people or to mislead listeners.
Citation
If you use this voice, please cite Expresso:
@inproceedings{nguyen2023expresso,
title = {EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis},
author = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and Gat, Itai and
Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and Synnaeve, Gabriel and Hassid, Michael and
Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel},
booktitle = {Interspeech},
year = {2023}
}
- Downloads last month
- 32