V oice C raft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024)

Abstract: We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable generation within an existing sequence. On speech editing tasks, VoiceCraft produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SotA models including VALL-E and the popular commercial model XTTS v2. Crucially, the models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings. In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named . We encourage readers to listen to the demos at https://jasonppy.github.io/VoiceCraft_web. Data, code, and model weights are available at https://github.com/jasonppy/VoiceCraft

## introduction we introduce voicecraft, a transformer-based neural codec language model (nclm) that performs infilling generation of neural speech codec tokens autoregressively conditioned on bidirectional context. voicecraft achieves state-of-the-art (sota) performance on both speech editing (shown in fig. 1 ) and zero-shot tts. our method is based on a two-step token rearrangement procedure that consists of a causal masking step and delayed stacking step. the causal masking technique is inspired by the success of causal masked multimodal 1 data, code, and model weights are available at https: //github.com/jasonppy/voicecraft . model in joint text-image modeling @xcite , and our proposed technique works for speech codec sequences, which enables autoregressive generation with bidirectional context. in addition, we further integrate causal masking with delayed stacking @xcite as our proposed token rearrangement procedure, to ensure efficient multi-codebook modeling. to evaluate speech editing, we manually crafted a first-of-its-kind, realistic, and challenging dataset named realedit. realedit consists of 310 real world speech editing examples, with waveforms sourced from audiobooks @xcite , youtube videos @xcite , and spotify podcasts @xcite , and duration ranging from 5 seconds to 12 seconds. to create the target transcripts, the transcripts of the source speech are edited in such a way that the edited transcripts remain grammatically correct and are semantically coherent. the dataset is designed to cover a wide range of editing scenarios, including insertion, deletion, substitution, and multi-span editing, with the length of the edited text ranging from 1 word to 16 words. compared to commonly used speech synthesis evaluation datasets that only contain audiobooks such as vctk @xcite , ljspeech @xcite , and libritts @xcite , realedit is 1 12442 more challenging in that the recordings have diverse content, accents, speaking styles, recording conditions, and background sounds. we believe that the realism and diversity of realedit makes it a reliable indicator of the practicality of speech editing models in the real world. in the subjective human listening tests, voice-craft significantly outperforms prior sota speech editing model on realedit. importantly, the edited speech produced by voicecraft is nearly indistinguishable from the original unedited recording in terms of naturalness. we found that voicecraft generalizes well to zero-shot tts without any finetuning, achieving sota performance on a dataset comprised of audiobooks and youtube videos, outperforming strong baselines including reproduced vall-e @xcite and the popular commercial model xtts v2 @xcite . in summary, our contributions are: ## method voicecraft casts both sequence infilling (for speech editing) and continuation (for zero-shot tts) as a simple left-to-right language modeling by rearranging neural codec's output tokens. the rearrangement involves two steps: (1) causal masking ( §3.1) to enable autoregressive continuation/infilling with bidirectional context and (2) delayed stacking ( §3.2) to ensure efficient multi-codebook modeling. voicecraft employs decoder-only transformers and is trained with an autoregressive sequence prediction ( §3.3). we introduce the inference setup for speech editing and zero-shot tts in §3.4. ## realedit: a realistic and challenging speech editing dataset to support as realistic an evaluation as possible, we constructed a first-of-its-kind dataset of 310 manually-crafted speech editing examples. each example consists of a tuple: (original audio, original transcript, edited transcript). the dataset contains 100 utterances from libritts (dev-clean and dev-other) @xcite , 100 utterances from youtube (from gigaspeech testset) @xcite and 110 utterances from the spotify pod-cast dataset @xcite . we manually checked the utterances for accuracy, then had native english speakers revise them to create edited transcripts. for each utterance, we determine the type of modification using predefined probability distributions of editing type, number of disjoint spans to be edited, and editing span length. specifically, we study the following categories: 1) number of edited spans: 1 or 2; 2) type of edits: insertion, deletion and substitution; 3) editing span length: short (1-2 words), medium (3-6 words), long (7-12 words). crucially, a edited transcript must be grammatically correct and semantically coherent. examples of the dataset are shown in model. encodec @xcite is used as the speech tokenizer, which has 4 rvq codebooks each with vocabulary size of 2048, and a codec framerate of 50hz on 16khz recordings. (see §c for detailed config). to choose the number of spans to mask in training, we use a poison(1) distribution truncated to a minimum of 1 and maximum of 3. span lengths are sampled from uniform(1, 600) i.e. the masked speech can be as long as 12 seconds. at each time step, the embeddings of codes from different codebooks are summed @xcite , then added by sinusoidal positional encoding @xcite , before being fed to the transformer. text transcripts are phonemized based on the ipa phoneset using the toolkit provided by @xcite . our main voicecraft model has 16 transformers layer with hidden/ffn dimensions of 2048/8192, and 12 attention heads. the output of the last layers are fed to four separate 2-layer mlp modules to get prediction logits. our main model has 830m parameters and codebook weight hyperparameters α is set to be (5, 1, 0.5, 0.1). ablations on model sizes and codebook weights are shown in §5.2. training and inference. the training of the encodec model largely follows the setting in @xcite , detailed in §c. to train voice-craft, we used the scaledadam optimizer and eden scheduler proposed in @xcite with a base learning rate of 0.05, batch size of 400k frames (i.e. 133.2 minutes), and total training step of 50k with gradient accumulation. the training of the 830m voicecraft model took about 2 weeks on 4 nvidia a40 gpus. more details can be found in §c. we compare the performance of scaledadam and adamw in §a.1. for inference, we use nucleus sampling @xcite with @xmath0.8 and a temperature of 1 for all experiments. due to the stochasticity of autoregressive generation, via manual inspection we found that while most of the time the model produces natural sounding speech, it sometimes produces excessively long silence or drags out certain sounds. we found that happens when the codec token generation gets stuck in a repeating loop. to resolve it, we use a simple heuristic: for each input utterance we generate several different output utterances and throw away the longest outputs. specifically for speech editing, we run inference 10 times with different margin parameters, stepping ϵ up from 0.05 to 0.14 in 0.01 increments. the 4 longest outputs are discarded, and then we randomly select one sample from the remaining 6 outputs. for zero-shot tts, we reduce the probability of generating the same token in consecutive timesteps in proportion to how many times that token was consecutively generated in the immediately preceding timesteps. in addition, we generate 5 samples with different random seeds, and select the shortest for tts evaluation. the sample selection process is completely automatic and unsupervised (i.e. no human intervention or asr scoring). baselines. for speech editing, we compare voicecraft with the diffusion-based model flu-entspeech @xcite which is the current open-source sota model for speech editing. since the original fluentspeech model is trained on libritts, for a fair comparison, we took the official github repo and trained the model on gigaspeech. please find more details in §c. for zeroshot tts, we compare our voicecraft with vall-e @xcite , xtts v2 (co-qui, 2023), yourtts @xcite , and fluentspeech. since the original vall-e is not open-sourced, we use the code from the popular open-source implementation by @xcite , and also trained the model on gigaspeech. xtts v2 is a popular commercial zero-shot tts model foot_3 trained on a mixture of publicly available data and webcrawled data, although the exact data sources are unknown. yourtts is trained on vctk, libritts, and also french and portugese corpora. metrics. for ablation studies, since ground truth waveform is avaliable, in addition to wer (using whisper medium.en as the asr model), we use mel-ceptral distortion (mcd), f0 distance (f0) and energy distance (energy). these are all objective metrics and their definitions are detailed in §c. for speech editing and zero-shot tts evaluation, we use a combination of objective and subjective metrics. for the objective metrics, we used wer and speaker similarity (sim) following prior works @xcite . sim is calculated using the wavlm-tdcnn @xcite . wer and sim are calculated on all 310 utterances in realedit, and 250 utterances in the zero-shot tts dataset. for our subjective evaluation, we used the amazon mechanical turk platform to conduct human listening tests. for speech editing, the outputs of our model on all 310 utterances from realedit are evaluated by turkers in terms of naturalness and intelligibility, and we use a 5-point likert scale where 1 means poor and 5 means excellent. we also performed side-by-side a/b testing of voice-craft's output against the original (non-edited) speech, as well as the edited speech produced by fluentspeech. in both cases, turkers were asked to determine which utterance sounds more natural. the turkers can choose either one of the two, or indicate that they are equally natural. each evaluation received 5 ratings from 5 different turkers. for zero-shot tts, we randomly sampled 80 utterances (40 from libritts and 40 from youtube) from the original evaluation set, and asked turkers to rate the naturalness, intelligibility, and speaker similarity of the generated speech to the reference prompt on a 5-point likert scale. each evaluation received 10 ratings. for all evaluations except the side-by-side comparison, mean-opinion-score (mos) with 95% confidence interval are reported. for the side-by-side comparison, we report the percentage of the time one model is preferred over the other. 64 and 59 turkers participated in speech editing and tts evaluation respectively. please refer to §e for instructions and participants description. ## conclusion we introduce a neural codec language model voicecraft that achieves state-of-the-art performance on speech editing and zero-shot tts on inthe-wild data. the key lies in an innovative token rearrangement procedure which enables efficient and effective autoregressive codec generation with bidirectional context. in addition, we introduce a first-of-its-kind high quality, challenging, and re-alistic speech editing dataset realedit, which we believe can reliably measure the practicality of speech editing models.