Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features (2022)

Abstract: While neural text-to-speech systems perform remarkably well in high-resource scenarios, they cannot be applied to the majority of the over 6,000 spoken languages in the world due to a lack of appropriate training data. In this work, we use embeddings derived from articulatory vectors rather than embeddings derived from phoneme identities to learn phoneme representations that hold across languages. In conjunction with language agnostic meta learning, this enables us to fine-tune a high-quality text-to-speech model on just 30 minutes of data in a previously unseen language spoken by a previously unseen speaker.

## introduction the advance of deep learning @xcite has enabled great improvements in the field of text-to-speech (tts). (towards-)end-to-end models, such as tacotron 2 @xcite , trans-formertts @xcite , fastspeech 2 @xcite , fastpitch @xcite and many more famous instances (e.g. @xcite and @xcite ) allow for speech synthesis with unprecedented quality and controllability. the models mentioned here rely on vocoders, such as wavenet @xcite , melgan @xcite , parallel wave-gan @xcite or hifi-gan @xcite to turn the parametric representations that they produce into waveforms. recently proposed models even include some with the ability to go directly to the waveform from a grapheme or phoneme input sequence, such as eats @xcite or vits @xcite . while these methods all perform remarkably well if given enough data, cross-lingual use of data remains a key challenge in tts. most modern methods are limited to languages and domains that are rich in resources, which over 6,000 lan-guages are not. attempts at reducing the required resources in a target language by making use of transfer learning from multilingual data have been made by @xcite ; @xcite ; @xcite . the mismatch of input spaces however requires complex architectural changes, which limits their ability to be used in conjunction with other modern tts architectures. attempts at fixing the issue of having to transfer knowledge from a source to a target by just jointly training on a mixed set of more and less resource rich languages have been made by @xcite ; de @xcite ; @xcite , which requires complex training procedures. in this work, we will also attempt to transfer knowledge from a set of high resource languages to a low resource language. we fix previous shortcomings by 1) using a linguistically motivated representation of the inputs to such a system (articulatory and phonological features of phonemes) that enables cross-lingual knowledge sharing and 2) applying the model agnostic meta learning (maml) framework @xcite to the field of low-resource tts for the first time. using articulatory features as inputs for neural tts has been attempted recently by @xcite and @xcite , following the classical approach of @xcite . both achieved good results when applying this idea to the codeswitching problem, since unseen phonemes in the input space no longer map to nonsensical positions, as it would be the case for the standard embedding-lookup. it has to be noted however, that this only works across languages with similar types of phonemes. also @xcite have applied phonological features to low-resource tts with fair success. they did however rely on supplementary features, such as dependency parsers and morphological analyzers. furthermore all of their data and models are proprietary and can therefore not be used to compare results to. in this work, we extend the use of articulatory inputs with the maml framework to enable very simple yet well working low-resource tts that can be applied to almost all modern tts architectures. we encounter severe instabilities when using maml on tts, which make the standard formulation of maml infeasible to use. thus we also propose a modification to maml, which reduces the procedure's complexity. this allows us to create a set of parameters of a model that can be used to fine-tune to a well working single-language singlespeaker tts model with as little as 30 minutes of paired training data available and even enables zeroshot adaptation to unseen languages. we evaluate the success of our approach with both automatic measures and human evaluation. our contributions are as follows: 1) we show that it is beneficial to train a tts model on articulatory features rather than on phoneme-identities, even in the standard single-language high-resource case; 2) we introduce a training procedure that is closely related to maml which allows training a set of parameters for a tts model that can be fine-tuned in a low resource scenario; 3) we provide insights on how much data and training time are required to fine-tune a model across different languages and speakers simultaneously using said meta-parameters; 4) we show that the metaparameters can generalize to unseen phonemes and rapidly improve their ability to properly pronounce them when fine-tuning. foot_0 2 background and related work ## further analysis and future work what is the ideal amount of training steps for fine-tuning? to investigate the amount of update steps needed to fully adapt to the new speaker with the added difficulty of learning a new language, we show the cosine similarity of a speaker embedding of the fine-tuned model to that of the ground truth throughout the fine-tuning process in figure 4 . the speaker embedding is built according to the ecapa-tdnn architecture @xcite and provided open source by speechbrain @xcite . it is trained on voxceleb 1 and 2 @xcite @xcite which to the best of our knowledge does not overlap with any of the other training and evaluation data we used. we tried to decrease adaptation time further by incorporating said speaker embedding similarity as an additional objective function, similar to @xcite , we did however see only marginal improvements in the amount of steps needed at the expense of greatly increased training time. can this setup handle zero-shot phonemes? we show the model's zero shot capabilities in figure 5. we removed dutch and finnish from the training data of the meta-checkpoint and trained another version of it, to be able to see how it handles all of the now completely unseen phonemes specific to german. while their correct position in plot (a) can be considered given, since it shows the articulatory featurespace, their meaningful positions in plot (b) and (c) show that the meta-checkpoint does not just collapse the vector of the unseen phoneme to the one it is most similar to, but actually generalizes. while their pronunciation when produced does not match the correct pronunciation perfectly, it can be understood in the context of a longer sequence. this is congruent with the results of @xcite . during the adaptation phase, the pronunciation of the unseen phonemes rapidly matches the correct pronunciation after less than 100 steps. does this setup learn the difference between language and speaker? when analyzing the finetuned meta-checkpoint, we observed that it seems to link the language of the input to the voice of the speaker. for example when synthesizing an unseen hungarian text using tacotron 2, the voice of the synthesis resembles that of the hungarian female speaker, even though the model has been fine-tuned on the male german speaker and there are no additional conditioning signals. we hypothesize that the laml procedure induces certain subsets of parameters in the model to be speaker dependent and the encoder of the model priming those parameters purely based on the phoneme sequence. this leads us to believe, that the fine-tuning of all parameters in the model may neither be necessary, nor even the best way of adapting to new data. this also fits the observations of the speaker embedding over time, since the tacotron model adapts to the new speaker very rapidly. further investigations into the interactions between parameter groups could allow cutting down the amount of parameters that need to be trained significantly, further reducing the need for training data. how can we bring down fastspeech 2's data need further? a similar observation regarding language and speaker can be made with fastspeech 2, however as could be seen from the experiment on naturalness and the training time, the fastspeech 2 model can benefit more from additional data and training time. this may come down to its nearly twice as high parameter count. so a more effective fine-tuning strategy, that considers some parameters as constants, could benefit the fine-tuning capabilities of the fastspeech 2 model greatly. does this work across language families? one limitation to our findings is that we investigated only the transfer of languages that share similar phoneme inventories. it is possible that fine-tuning to a language that uses e.g. the lexical tone rather than pitch accents or word accents would require pretraining in more closely related high-resource languages, such as chinese. however, as vu and schultz (2013) find in their analysis of multilingual asr, the fast adaptation of an acoustic model trained on multiple languages to unseen languages works well, even across different language families. we thus believe that the technique and analysis presented in this paper also holds across language families and types. ## conclusion in this paper, we show an approach for training a model in a language for which only 30 minutes of data are available by making use of articulatory features and language agnostic meta learning. the main takeaways from our work are as follows: articulatory features for tts using articulatory features as the input representation to a tts system enables the use of multilingual data without the need for increased architectural complexity, such as language specific projection spaces. it is furthermore beneficial to use even in singlelanguage scenarios, since the knowledge sharing between phonemes makes the tts system converge much earlier to an usable state during training. maml on tts applying maml to tts does not work well. if we however remove the inner loop, we are able to pretrain a low-resource capable checkpoint for tts. this modification not only makes it work, it also simplifies the formulation.