Low-Resource Multilingual and Zero-Shot Multispeaker TTS (2022)

Abstract: While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world’s over 6,000 spoken languages. In this work, we bring together the tasks of zero-shot voice cloning and multilingual low-resource TTS. Using the language agnostic meta learning (LAML) procedure and modifications to a TTS encoder, we show that it is possible for a system to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language. We show the success of our proposed approach in terms of intelligibility, naturalness and similarity to target speaker using objective metrics as well as human studies and provide our code and trained models open source.

## introduction the applications of modern tts systems are omnipresent and bring major benefits in a very diverse range of tasks. for example, low-resource tts can be used to revitalize and conserve languages with diminishing numbers of speakers @xcite . other recent applications go into the direction of protecting the privacy of a speaker, by exchanging their voice for a different voice, while not affecting the content of what is said @xcite . even in literary studies, tts systems can be applied to investigate perceptive aspects of poetry reading @xcite . however, while the first of those examples can be done with just a single speaker, the latter two require the tts system to be able to exchange the voice of the utterance that is produced, which usually requires large amounts of clean multispeaker data. the same requirement exists for many other such applications, which can also be seen in the rise of interest in the research community on voice-cloning technologies @xcite @xcite @xcite . the communities of speakers of low-resourced languages are thus mostly locked out of plenty of the applications that modern tts enables. for many instances of such languages, like the taa language, which is famous for its 83 click sounds or the yoruba language, in which the tones bear so much meaning, that the language can be mostly whistled, it would be extremely difficult to collect the required amounts of data, and transfer learning to such unique languages is very challenging. still, we believe that a single model that speaks many languages with any voice can exhibit strong generalizing properties and is a promising first step towards fixing these inequalities. in this work we ask the following question: can a multilingual tts system be used to achieve zeroshot multispeaker tts in a low-resource scenario? our approach is to use crosslingual knowledgesharing to enable 1) finetuning a tts on just 5 minutes of data in an unseen language in an unseen branch in the phylogenetic tree of languages and 2) transferring zero-shot multispeaker capabilities from the pretraining languages to the unseen language. to achieve this, we propose changes to a tts encoder to better handle multilingual data and disentangle languages from speakers. further, we show that the laml pretraining procedure @xcite can also be used to train general speaker-conditioned models. to verify the effectiveness of our contributions, we train models on just 5 minutes of german and russian while excluding all germanic and slavic languages from the pretraining respectively. we choose a simulated low-resource scenario over an actual lowresource scenario in order to get more reliable evaluations using both objective measures as well as human studies. furthermore, we show that models trained with this approach do not only serve as a basis for low-resource finetuning with greatly reduced data-need, they can also be used without finetuning as strong multispeaker and multilingual models. we train a model on 12 languages simultaneously and show that it can transfer speaker identities across all languages, even the ones where it has only seen a single speaker during training. all of our code, as well as the trained multilingual model are available open source foot_0 . an interactive demo foot_1 and a demo with pre-generated audios foot_2 are available. ## discussion language embedding investigation the accent transfer has interesting implications on how the distribution of realizations of a phoneme shifts with each language, independent of the context, which can be investigated by synthesizing individual phonemes with only the language embedding changed. we find language typical patterns, even in the languages that have only been trained on 5 minutes of data. so it seems that very little data is enough to capture a lot about how a language is usually spoken. implicit morpheme vocabularies although word boundaries are not explicitly denoted as segmental units in speech, they still have considerable influence on the phonetic realization. consider for example the phenomenon of velar softening, i.e. a velar plosive is realized as alveolar fricative when followed by a long or short i ([i] or [ay]) in some contexts, such as in electri[k] → electri[s]ity. this does however not hold across word boundaries as in electri[k] igniter. another example where word boundaries cause changes in the phone sequence is the phenomenon of final devoicing: voiced obstruents become voiceless if they occur in word-final position e.g. the german word hunde (dogs) is pronounced [hund@] in its plural form but in singular hund becomes @xcite . such rules are however highly dependent on the language. final devoicing is for example observed in german, dutch and polish, but not in english or french. while many of these language specific lexical rules are already captured by the phonemizer, the situation is different in cases where word boundaries are not reflected by the phone sequence itself but only in the intonation, such as in @xcite . while in the latter, there is still a morpheme boundary after acid, this is not a word boundary. this highlights the importance of differentiating between actual word boundaries and word-internal morpheme boundaries in order to produce correct intonation which is crucial for generating intelligible speech. monolingual tts models actually seem to learn an implicit vocabulary of morphemes as well as an intuition in which contexts morpheme boundaries can denote a word boundary in the language they are trained in. but in the case of multilinguality, this vocabulary of morphemes is difficult to construct, because every language has different morphemes. thus, since multilingual models face a more difficult task to identify morphemes, they struggle even more distinguishing morpheme from word boundaries. even with the language embedding, it seems like this is a property that the tts can no longer implicitly capture, at least not given small amounts of data. we especially observe this in compound-nouns in our model trained on german in a low-resource setting. a model without explicit word boundaries adds boundary tones in the middle of the word causing an unnatural intonation that reduces the intelligibility of the word. if the model is trained with word boundaries, even though there are no word boundaries within the composite-noun, the pronunciation becomes much more fluent with the intonation being consistent throughout the word. the proper phrasing within this word would be to combine the [dampfsif] into one unit with the pitch being the highest on [i] and a falling pitch towards the end of the word throughout @xcite . this is the case in the model that is aware of the wordboundaries. for illustration purposes, we include the boundary between steamboat and ride in the plot, the model however does not see this boundary as it happens in the middle of one word. the model which is unaware of the word-boundaries lowers its pitch already at the [i] and lengthens the [dampf] part of the word. this makes the second instance sound as if the model was saying "steam boatride" rather than "steamboat ride". we conclude that by simply making word boundaries explicit, the model no longer overestimates intonation phrase boundaries and boundary tones at every possible morpheme boundary. ## limitations and future work while the laml procedure is, as the name suggests, language agnostic, we only include european languages in our training and testing in order to get more reliable results with the resources for testing we have available. the state of the implementation with which the experiments were conducted cannot handle tonal languages, due to the non-segmental nature of tone. this limits the generality of our findings. our open-sourced code has been updated in the meantime to be able to handle tone and lengthening properly. we plan to extend this work to include a much larger and much more diverse set of languages. ## conclusion we show that through a simple encoder design coupled with a mechanism to encode word boundaries and the laml training procedure, a low-resource capable multilingual zero-shot multispeaker tts can be achieved. we are able to train a german and a russian model on just 5 minutes of data each, which perform comparable or even better to single speaker models trained on 29 and 21 hours of data respectively. we further show that the ability to perform zero-shot multispeaker tts is shared across languages, even those which have seen only 5 minutes of single speaker data. an additional side-effect is that the language embedding design in the encoder allows us to vary the accent of speech regardless of language of the input text and speaker.