P arrot TTS : Text-to-speech synthesis exploiting disentangled self-supervised representations (2024)

Abstract: We present ParrotTTS, a modularized text-to-speech synthesis model leveraging disentangled self-supervised speech representations. It can train a multi-speaker variant effectively using transcripts from a single speaker. ParrotTTS adapts to a new language in low resource setup and generalizes to languages not seen while training the self-supervised backbone. Moreover, without training on bilingual or parallel examples, ParrotTTS can transfer voices across languages while preserving the speaker-specific characteristics, e.g., synthesizing fluent Hindi speech using a French speaker’s voice and accent. We present extensive results in monolingual and multi-lingual scenarios. ParrotTTS outperforms state-of-the-art multi-lingual text-to-speech (TTS) models using only a fraction of paired data as latter. Speech samples from ParrotTTS and code can be found at https://parrot-tts.github.io/tts/

## introduction vocal learning forms the first phase of infants starting to talk @xcite by simply listening to sounds/speech. it is hypothesized @xcite that infants listening to ambient language store perceptually derived representations of the speech sounds they hear, which in turn serve as targets for the production of speech utterances. interestingly, in this phase, the infant has no conception of text or linguistic rules, and speech is considered sufficient to influence speech production @xcite as can parrots @xcite . our proposed parrottts model follows a similar learning process. our idea mimics the two-step approach, with the first learning to produce sounds capturing the whole gamut of phonetic variations. it is attained by learning quantized representations * authors contributed equally to this work. of sound units in a self-supervised manner using the raw audio data. the second phase builds on top of the first by learning a content mapping from text to quantized speech representations (or embeddings). only the latter step uses paired text-speech data. the two phases are analogous to first learning to talk followed by learning to read. figure 1 illustrates parrottts contrasting it with the traditional mel-based tts. the ssl module includes a speech-to-embedding (ste) encoder trained on masked prediction task to learn an embedding representation of the input raw audio @xcite @xcite . an embedding-to-speech (ets) decoder is independently trained to invert embeddings to synthesize audio waveforms and is additionally conditioned on speaker identity. this learning to talk is the first of the two-step training pipeline. in the subsequent learning to read step, a separate text-to-embedding (tte) encoder is trained to generate embeddings from text (or equivalent phonetic) inputs. this step requires labeled speech with aligned transcriptions. parrottts offer several advantages over the traditional mel-based neural tts models @xcite . for instance, (a) quantized speech embedding has lower variance than that of mel frames reducing the complexity to train tte (b) direct waveform prediction bypasses potential vocoder generalization issues @xcite . (c) reduced complexity helps in stabler training of tte encoder for either autoregressive or non-autoregressive choice. for example, we observe at least eight-fold faster convergence in training iterations of our tte module compared to that of @xcite and @xcite . while our work closely relates with recent works @xcite @xcite utilizing self-supervised representations for tts synthesis, our focus differs by aiming to achieve a unified multi-speaker, multilingual tts system in low-resource scenarios @xcite . in our work, low-resource refers to the scarcity of paired tts data. here are the key distinctions of our model compared to recent efforts: additionally, it's worth mentioning that certain methods @xcite depend partially or entirely on automatic speech recognition (asr) to obtain paired data. it should be noted that these asr models are trained using substantial amounts of supervised data, inaccessible in low resource settings. while architecturally similar to other ssl-based tts @xcite , our primary contribution lies in achieving promising outcomes in the low resource scenario, where minimal paired data from a single speaker per language is accessible for tts training. ## parrottts architecture parrottts has three modules; two encoders that map speech or text inputs to common embedding space (referred to as ste and tte respectively) and a decoder (ets) that renders speech signal from these embeddings. our speech encoderdecoder choices are borrowed from @xcite . our speech decoder ets is a modified version of hifigan @xcite ). text encoder tte is an encoder-decoder architecture and we experiment with both autoregressive (ar) and nonautoregressive (nar) choices for the same. ## experiments we perform experiments in monolingual and multi-lingual scenarios. details of various par-rottts models trained and of those each of them is compared to is covered below. ## conclusion we investigate a data-efficient parrottts model that leverages audio pre-training from selfsupervised models and ties it to separately trained speech decoding and text encoding modules. we evaluate this architecture in various settings. quality of rendered speech with as little as five hours of paired data per language is on par with or superior to competitive baselines. this is the key result from our experiments that we believe will help scale tts training easily to new languages by bringing low-resource ones into the same quality range as the resource-rich ones. moreover, we have released an open-source, multi-lingual tts model