Adversarial Text-to-Speech for low-resource languages (2022)

Abstract: In this paper we propose a new method for training adversarial text-to-speech (TTS) models for low-resource languages using auxiliary data. Specifically, we modify the MelGAN (Kumar et al., 2019) architecture to achieve better performance in Arabic speech generation, exploring multiple additional datasets and architectural choices, which involved extra discriminators designed to exploit high-frequency similarities between languages. In our evaluation, we used subjective human evaluation, MOS-Mean Opinion Score, and a novel quantitative metric, the Fréchet Wav2Vec Distance, which we found to be well correlated with MOS. Both subjectively and quantitatively, our method outperformed the standard MelGAN model.

## introduction text-to-speech (tts) is the task of generating natural speech that corresponds to a given text. tts systems play essential roles in a wide range of applications, ranging from human-computer interaction to assistance for people with vision or speech impairments. in recent years the field of tts has been dominated by the neural auto-regressive models for raw audio waveform such as wavenet @xcite , samplernn @xcite and wa-vernn @xcite . however, inference with these models is inherently slow and inefficient given the high frequency of audio data; because of the auto-regressive behaviour and the sequential generation of the audio samples. thus, auto-regressive models are usually impractical for real-time applications. researchers put much effort into enabling parallelism of the tts models, which resulted in a number of non-auto-regressive ones, such as parallel wavenet @xcite which distils a trained auto-regressive decoder into a flow-based convolutional student model, wave-glow @xcite which is a flow-based generative model based on @xcite as well as the generative adversarial network (gan @xcite )-based models such as melgan @xcite and gan-tts @xcite . they are highly parallelizable and more suitable to run efficiently on modern hardware. however, those recent developments often came at the price of scale, and hence may be impractical for certain applications with limited compute or data budgets. deep neural networks have revolutionized the field of tts achieving human-level performance on particular languages by leveraging massive collections of good-quality datasets, e.g. the lj speech dataset foot_0 . however, these successes came at cost since creating these large datasets typically requires a great deal of human effort to manually record and label individual data samples. this cost can be particularly extreme when recording and labelling requires expert supervision (for example, recording high quality audio requires a professional studio and staff). for many languages we lack resources to create sufficiently large labelled datasets, which limits the widespread adoption of tts techniques. the lack of available resources makes it extremely valuable to study the relationship between the different languages. the high-frequency similarities between languages can be exploited to learn better speech synthesis models for low-resource languages. however, not much work has focused so far on exploring this direction. the notable exceptions include some multi-lingual tts models @xcite . in @xcite they pre-trained a speech synthesis network using datasets from both high-resource and low-resource languages, and fine-tuned the network using only low-resource data. the results showed that the learned phoneme embedding vectors are located closer if their pronunciations are similar across the languages. in this work, we explore raw waveform generation for low-resource languages using auxiliary data, taking arabic as our case study and melgan @xcite as our baseline model. this study examines the arabic language since it has a large global population, it is a complex language to model, foot_1 and there is a scarcity of arabic tts datasets, making it a low-resource language. our main contributions are as follows: ## background the generative adversarial networks (gans) @xcite are a class of implicit generative models trained by adversarial means between two networks: the generator and the discriminator. generators attempt to produce data that resemble reference distributions, while the discriminator tries to distinguish real data from generated data, providing a useful training signal. due to the high temporal resolution of raw waveform, the presence of structure at different time scales, and the short-and long-term interdependencies among these structures, audio synthesis is a challenging task. most approaches simplify the problem by modelling a lower-resolution intermediate representation that can be efficiently computed from the raw temporal signal and preserves enough amount of information to allow a faithful inversion back to audio. it is therefore common to decompose text-to-speech (tts) systems into two stages: the first stage maps text into the intermediate representation, while the second stage transforms it into audio waveform. among the most commonly used intermediate representations are aligned linguistic features @xcite and mel-spectrograms @xcite . in this work, we use mel-spectrogram as an intermediate representation and focus on the second stage. considering the mel-spectrogram inversion stage, the tts systems can be categorized into three distinct families: the pure signal processing techniques, the auto-regressive models and the non-auto-regressive models. the auto-regressive models like the wavenet @xcite produced the state-of-the-art results in text-to-speech synthesis @xcite but inference with these models is inherently slow and inefficient due to the sequential generation of audio. the non-auto-regressive models hence are highly parallelizable and can exploit modern deep learning hardware like gpus and tpus. well known examples are the waveglow @xcite which is a flow-based generative model based on glow (kingma and dhariwal, 2018), and ganbased tts models like melgan @xcite and gan-tts @xcite . melgan generator is a fully convolutional feedforward network which takes mel-spectrogram as input and outputs a raw waveform. the generator is trained adversarially against a multi-scale architecture comprised of three discriminators that have identical network structures but operate on different audio scales. on the other, end-to-end architectures like the tacotron @xcite , eats @xcite and wavegrad 2 @xcite are introduced in the field of tts to reduce the compound error of two-stage tts systems. tacotron is a generative text-to-speech model based on a seq-to-seq model with an attention mechanism @xcite , whereas tacotron 2 @xcite is a follow-up work that eliminates the non-neural network elements used in the original tacotron. many works covered arabic tts synthesis to generate human-like speech, such as @xcite , @xcite and @xcite , but none of them adopted the gan-based tts models for the arabic language. @xcite describes how to use a modified deep architecture from tacotron 2 @xcite to generate mel-spectrograms from arabic diacritic text as an intermediate feature representation followed by a waveglow @xcite architecture acting as a vocoder to produce a high-quality arabic speech. the proposed model is trained using a published pre-trained tacotron 2 english model using a dataset with a total of 2.41 hours of recorded speech 3 . to the best of our knowledge, this is the best arabic tts available. ## methodology in this section, we present the details of the architectures of our models, the datasets, and the evaluation metrics we used. in melgan's official repository foot_4 , generator weights are publicly available, but discriminator weights are not. we use various methods of knowledge transfer between languages, including fine-tuning and co-training. ## results this section summarizes all the results of the experiments described in the experiments section 4. we evaluated the performance on the test set of the arabic speech corpus dataset3 using the mos and the average of the last five conditional fréchet wav2vec distance scores. it is worth noting that the mean of the best and the mean of the last five scores produced almost the same ordering. also, in all tables and figures, the mixing ratio represents the ratio between main and auxiliary segments respectively we feed to the additional discriminator. tables 4, 5, 6 shows the results of the ablation study. according to the study, melgan + extra disc with 1:2 mixing ratio between arabic 3 and english 1 data sets provided the best performance across all models. as well, adding auxiliary datasets as additional grounds truths in the extra discriminator is better than including the auxiliary dataset in the generator itself. last but not least, by using full capacity extra discriminator and reducing segment lengths, we would achieve better results than with the current settings. fwd how auxiliary date introduced 13.57 generator with auxiliary segments 11.16 extra ground truths for discriminator ## ethical considerations this paper aims to advance the field of text-tospeech and hence all considerations related to potential nefarious applications of such technology apply to this work. this includes the potential use of such systems to imitate voice of a certain individual in order to present a message that such person has never uttered. we also acknowledge that tts systems carry a bias towards the dialect/accent of the population whose speech was used as a training data. however, we hypothesise our model might be suitable to counter such effects: as it has been designed for low-resource languages, it might well be used to improve tts systems for underrepresented dialects or accents of otherwise well-modelled languages, in turn reducing geographical bias affecting certain populations. nevertheless, we believe that overall benefits of improved text-to-speech models outweight these and other ethical risks. ## conclusion in this work, we have proposed an extension for melgan that utilizes information of auxiliary highresource languages/dialects to help training of low resource language audio synthesis models. the proposed approach outperformed standard melgan