Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023)

Abstract: We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast TTS as a composition of two sequence-to-sequence tasks: from text to high-level semantic tokens (akin to “reading”) and from semantic tokens to low-level acoustic tokens (“speaking”). Decoupling these two tasks enables training of the “speaking” module using abundant audio-only data, and unlocks the highly efficient combination of pretraining and backtranslation to reduce the need for parallel data when training the “reading” component. To control the speaker identity, we adopt example prompting, which allows SPEAR-TTS to generalize to unseen speakers using only a short sample of 3 seconds, without any explicit speaker representation or speaker labels. Our experiments demonstrate that SPEAR-TTS achieves a character error rate that is competitive with state-of-the-art methods using only 15 minutes of parallel data, while matching ground-truth speech in naturalness and acoustic quality.

## introduction training a text-to-speech (tts) system typically requires hundreds of hours of parallel data in the form of transcribed utterances. as a consequence, high-quality tts is only available for a few dozen languages. moreover, the audio generated by these systems is only as diverse as the parallel data that they are trained on, which should contain many speakers, with various accents, of diverse demographics, and in heterogeneous recording conditions. at the same time, extremely diverse audio-only data can be relatively abundant online, present in the forms of podcasts, radio, and tv shows. this makes it attractive to use such audio-only data for building realistically sounding tts systems while minimizing the need for diverse parallel data. in this paper, we investigate how audio-only data can be leveraged to reduce the need for supervision in training tts systems. we introduce spear-tts, 1 a multi-speaker tts system that can be trained with as little as 15 minutes of parallel data from a single speaker. moreover, spear-tts can synthesize a new voice using only 3 seconds of speech, without any speaker labels or explicit speaker representation. spear-tts leverages recent advances in the ''textless'' modeling of spoken language @xcite @xcite @xcite . these methods represent continuous audio waveforms as sequences of tokens from a finite vocabulary, casting speech generation as a language modeling task. in particular, audiolm @xcite combines two types of discrete tokens: high-level semantic tokens and low-level acoustic tokens, which can be mapped to audio. using these representations, we cast the tts problem as a ''translation'' from text transcripts to acoustic tokens with semantic token representations serving as a pivot ''language'' @xcite . this way, tts is a composition of two sequence-to-sequence (seq2seq) tasks: translating text to semantic tokens, and translating semantic to acoustic tokens. the key benefit of such an approach is that the supervision needed to learn how to map text into the intermediate semantic token representation (''reading'') and how to produce speech from it (''speaking'') become decoupled. while the ''reading'' stage relies on parallel text-audio data, the audio tokens used to train the ''speaking'' component are produced by self-supervised audio models and therefore can be extracted from a massive amount of unlabeled speech data. as a result, the quality and diversity of the generated speech become independent from the available parallel data. casting each stage of spear-tts as a seq2seq problem allows us to use standard transformer models @xcite and makes it easy to tap into the vast pool of ideas developed by the machine translation community to reduce the need for supervision. specifically, we combine bart-style pretraining @xcite with backtranslation @xcite to significantly reduce the amount of parallel supervision required to train spear-tts. to control the voice used by spear-tts when producing an utterance, we leverage an example prompting mechanism that is closely related to prompting in textual language models @xcite . here we condition the ''speaking'' model with an audio clip representing the target voice, steering it to use this voice when generating the utterance. this feature can simplify building controllable multi-speaker tts systems for languages where only single-speaker parallel data is available. our experimental study on english speech shows that, by combining pretraining and backtranslation over a large dataset (551 hours) with just 15 minutes of parallel data from a single speaker, spear-tts: ## discrete speech representations below is a brief overview of the two selfsupervised audio representations which we adapt from audiolm. these representations occupy the opposite ends of the reconstruction quality vs. bitrate trade-off: acoustic tokens are typically highbitrate, allowing high-fidelity audio generation, while semantic tokens are low-bitrate, thus making it easier to achieve a long-span coherence. a detailed discussion can be found in @xcite . semantic tokens the role of semantic tokens is to provide a coarse, high-level conditioning to subsequently produce acoustic tokens. thus, they should provide a representation of speech where linguistic content-from phonetics to semanticsis salient, while paralinguistic information such as speaker identity and acoustic details are removed. to obtain such a representation, we train a selfsupervised speech representation model based on w2v-bert @xcite . after its training, we run a k-means clustering on the meanvariance normalized outputs of a specific layer. we use the centroid indices as discrete tokens. acoustic tokens acoustic tokens are discrete audio representations that provide high-fidelity reconstruction of the acoustic details. we train a soundstream @xcite codec to reconstruct speech while compressing it into few discrete units using a residual quantizer. to represent the hierarchy of residual quantizers in a sequence, we flatten the tokens corresponding to the different levels by interleaving them @xcite . we use the soundstream codec to convert audio to acoustic tokens and synthesize audio back from acoustic tokens. ## spear-tts overview spear-tts extends audiolm @xcite by enabling text as a form of conditioning. similarly to audiolm, spear-tts is organized in two main stages, as illustrated in figure 1 . in the first stage (s 1 ), text inputs are translated into a sequence of discrete semantic tokens. the second stage (s 2 ) maps semantic tokens into acoustic tokens, which are decoded to speech by the soundstream decoder @xcite . this way, s 1 learns to map text to the internal representation provided by semantic tokens (''reading''), while s 2 handles the production of speech from this intermediate internal representation @xcite . by using semantic tokens as an intermediate representation, we achieve two goals. first, semantic tokens provide us with a high-level representation of speech. thus, it is easier to learn a mapping from text transcripts to semantic tokens than directly between text and acoustic tokens. second, as both semantic and acoustic tokens are derived from self-supervised models, the second stage s 2 can be trained using audio-only data. this turns out to be extremely beneficial for training s 2 , as the typical scale of available audioonly data is considerably larger than that of parallel data. in turn, separating s 1 from s 2 allows us to pretrain the former with a denoising pretext task operating on succinct semantic tokens, further harnessing audio-only data. figure 1 : spear-tts. the first stage s 1 (''reading'') maps tokenized text to semantic tokens. the second stage s 2 (''speaking'') maps semantic tokens to acoustic tokens. acoustic tokens are decoded to audio waveforms using the soundstream decoder. ## s 1 : improving supervision efficiency the first stage s 1 maps tokenized text into semantic tokens. we use parallel text-semantic tokens data to learn this mapping. we start with a text-audio tts dataset and extract semantic tokens from audio. as a result, s 1 is reduced to a seq2seq task that can be implemented by encoderdecoder or decoder-only transformer architectures @xcite . training transformer seq2seq models can require substantial amounts of parallel data, which can be extremely scarce for some languages. in the following, we discuss two approaches used to alleviate this limitation: target domain pretraining ( § 5.1) and backtranslation ( § 5.2). ## s 2 : controlling the generation process the second stage model s 2 maps semantic tokens into acoustic tokens. to train it, we extract pairs of semantic and acoustic token sequences from each utterance in an audio-only dataset. next, we train a transformer model to perform seq2seq translation between the two token sequences. the second stage generates utterances with randomly varying voice, tempo, and recording conditions, reproducing the distribution of the characteristics observed in the training data. as s 1 and s 2 are trained independently, this diversity of speech is also preserved when s 1 is trained on a single-speaker dataset. to control the characteristics of the speaker's voice, we combine two findings from audiolm @xcite : (a) whenever the speech prefix is represented solely by semantic tokens, audiolm generates continuations by sampling a different random voice each time; (b) however, when conditioning also includes acoustic tokens, audiolm maintains the voice characteristics captured by the acoustic tokens when generating the continuation. in contrast to audiolm, we explicitly incorporate this ability during training, as illustrated in figure 3 . during training, we randomly select two non-overlapping windows of speech from each training example, from which we compute sequences of semantic and acoustic tokens. we consider one of the windows as the prompt and the other as the target output. next, we concatenate the sequences in the following order: (a) semantic tokens from the prompt, (b) semantic tokens from the target, (c) acoustic tokens from the prompt, and (d) acoustic tokens from the target. during training of s 2 , (a)-(c) are used as prefix and the model learns to generate the target acoustic tokens (d), preserving the speaker identity captured by the acoustic tokens from the prompt. at inference time, (a)-(c) are provided as input, and (d) is generated autoregressively. we also add a special separator token to inform the model about the expected discontinuity. this prevents boundary artifacts, which sometimes occur otherwise. note that the text transcript of the prompt is not needed. the speech samples generated by s 2 might contain some background noise, since this is typically present in the training data. we consider two methods to control the noise level in the synthesized speech at inference time. first, in the case of prompted generation, it is possible to select prompts containing cleaner speech. second, we can use a stochastic sampling (e.g., temperature sampling), generate multiple sequences for the same input and then use a no-reference audio quality metric to select the samples containing the least amount of noise. to this end, we use a mos estimator model similar to dnsmos @xcite . for prompted generation, we concatenate token sequences in the following order: semantic tokens from the prompt, semantic tokens from the target, acoustic tokens from the prompt. then, the model generates acoustic tokens corresponding to the semantic tokens from the target, while preserving the voice and speaking conditions in the acoustic tokens from the prompt. squares correspond to the parts of the prompt. ## experiments we evaluate spear-tts along several dimensions. first, we measure the faithfulness of the generated speech to the input transcript, for different training scenarios and amounts of parallel data available ( § 9.1). then, we show that spear-tts is able to successfully control the speaker voice, without any degradation in terms of fidelity to the transcript ( § 9.2). ## subjective evaluation ultimately, we resort to subjective tests with human raters to compare the quality of spear-tts with the baselines and with ground-truth natural speech. we focus on the scenario with minimal supervision and use the s 1 model that is trained with the 15 minute ljspeech @xcite subset. as baselines, we use the fastspeech2-lr models @xcite trained fastspeech2-lr spear-tts ground-truth parallel training data 15 min 1 h 24 h 15 min -mos 1.72 ±0.04 2.08 ±0.04 2.11 ±0.04 4.96 ±0.02 4.92 ±0.04 to ensure that the evaluation sentences are not part of the training set of spear-tts or the fastspeech2-lr models, we extract sentences from an audiobook chapter released in 2022, read by the same voice as in ljspeech. 6 this chapter was published later than any of the datasets we use. we extract 20 sentences from it, each with duration between 3 and 11 seconds, for a total of 133 seconds. we take transcripts for those sentences in the text of the corresponding book. the baselines are tts systems trained to generate a single voice. to ensure a fair comparison, we prompt s 2 with utterances extracted from the ljspeech dataset, so that spear-tts generates speech with the same voice. to this end, we randomly select 3 second speech samples from ljspeech and filter out samples that have more than 1 second of silence, using the remaining as prompts. we recruited raters using an internal crowdsourcing platform. the raters took an english proficiency test. samples are presented to raters one-by-one, and raters are asked to judge the overall quality on a scale from poor (1) to excellent (5). before starting, the raters were provided with example utterances for each grade. those reference samples represent clean natural speech, speech resynthesized by soundstream, and speech synthesized from partially corrupted soundstream token sequences. we additionally included some examples representing speech degradations common in online communication. each audio sample is evaluated by 20 raters. for each treatment, we average all scores to compute the mean opinion score (mos). we also compare spear-tts and vall-e @xcite in a small-scale subjective test using the examples provided on its demo page. 7 these examples are generated by combining 8 transcripts with 3 prompts each, resulting in 24 speech utterances. using the same instance of spear-tts described above (with s 1 trained with 15 minutes of single-speaker ljspeech), we synthesize 24 utterances using the same transcripts and prompts and conduct a subjective test with the same protocol described above. ## limitations and broader impact while our motivation is to enable high-quality, diverse, and controllable tts for under-served languages, we started our investigations with english, which allowed us to address the problem using a collection of well-studied datasets. this leaves some unknowns on whether our proposed approach would be applicable for languages sufficiently different from english, e.g., for tonal languages. for instance, would we be able to get a useful discretized representation of speech for such languages? would they require a dramatically larger token vocabulary size and would it allow effective speech modeling? however, we are hopeful due to an increasing amount of progress in related research. for example, it was demonstrated that the underlying audiolm framework successfully works across many languages and even for music @xcite @xcite . however, this remains circumstantial evidence and a direct investigation on spear-tts is in order. another potential caveat is that while training the s 2 does not require parallel text-audio data, it still needs a considerable amount of data, in the order of a few thousands hours (see appendix a). that can become a limitation for low-resource languages. a similar problem can be faced when training a w2v-bert model. here, a natural potential solution would be leveraging cross-lingual transfer, particularly from phonetically close languages, which was shown to work remarkably well for some self-supervised models @xcite . we also acknowledge that the ability to mimic a voice can have numerous malicious applications, including bypassing biometric identification and for the purpose of impersonation @xcite . thus it is crucial to put in place safeguards against the misuse and, as an initial step, we verify that speech produced by spear-tts can be reliably detected by a classifier with an accuracy of 82.5% on a balanced dataset, using the same protocol and classifier as @xcite . due to the concerns about a potential malicious use, we decided not to release checkpoints and training code for our models publicly. ## conclusions & future work we introduced spear-tts, a multi-speaker tts system that has two features setting it apart. first, it only requires a minimal amount of parallel data to be trained, i.e., it can synthesize speech with high fidelity when trained on as little as 15 minutes of parallel data coming from a single speaker. second, spear-tts is able to synthesize speech maintaining voice characteristics of a previously unseen speaker using a 3-second long voice example. these capabilities originate from harnessing abundant audio-only data. the key component that unlocks the usage of such data is the hierarchical discrete representation of speech that combines high-level semantic tokens with low-level acoustic tokens. using these representations, spear-tts casts the tts problem as a composition of two sequence-to-sequence tasks, ''reading'' (from tokenized text to semantic tokens) and ''speaking'' (from semantic tokens to acoustic tokens). spear-tts uses audio-only data in three ways: (a) to train the ''speaking'' model, such that the hard task of speech generation benefits from large-scale data, (b) as a domain for pretraining, and (c) to generate synthetic parallel data for backtranslation. our experimental study on english data ( § 9) shows that by combining audio-only data from libritts @xcite with 15 minutes of parallel data sampled from ljspeech @xcite , spear-tts achieves intelligibility comparable to that of an adapted fastspeech2-lr @xcite trained on 24 hours of ljspeech. next, our experiments show that spear-tts can maintain voice characteristics of a previously unseen speaker, in a zero-shot manner, with high accuracy. indeed, our measurements indicate that by taking a 3-second-long voice example for a speaker from librispeech test-clean, spear-tts achieves 92.4% accuracy on maintaining the voice when synthesizing held-out text transcripts, according to our speaker classifier. moreover, when measuring speaker similarity between prompts and generated speech, spear-tts obtains a cosine similarity close to the score reported for vall-e @xcite and significantly higher than the score of yourtts @xcite . subjective evaluations of speech naturalness show that spear-tts has significantly higher quality than a strong single-voice baseline even when trained with 96× less parallel data. moreover, the mos score of spear-tts is on par with the natural speech. when comparing quality of the speech synthesized in a zero-shot voice transfer task, spear-tts obtains a mos that is considerably higher than vall-e, with 240,000× less data. we believe that applying our findings to building a tts system for truly low-resource languages is the main direction for further work.