Towards Zero-Shot Text-To-Speech for A rabic Dialects (2024)

Abstract: Zero-shot multi-speaker text-to-speech (ZS-TTS) systems have advanced for English, however, it still lags behind due to insufficient resources. We address this gap for Arabic, a language of more than 450 million native speakers, by first adapting a sizeable existing dataset to suit the needs of speech synthesis. Additionally, we employ a set of Arabic dialect identification models to explore the impact of pre-defined dialect labels on improving the ZS-TTS model in a multi-dialect setting. Subsequently, we fine-tune the XTTS model, an open-source architecture. We then evaluate our models on a dataset comprising 31 unseen speakers and an in-house dialectal dataset. Our automated and human evaluation results show convincing performance while capable of generating dialectal speech. Our study highlights significant potential for improvements in this emerging area of research in Arabic.

## introduction over the last few years, there have been dramatic breakthroughs in speech synthesis. the core of any text-to-speech (tts) system lies in its capacity to transform textual input into natural, coherent speech. this transformation, however, is not a simple task. it involves the integration of multiple components. understanding and dissecting these components -from text preprocessing and linguistic analysis to acoustic modeling and prosody generation -is essential for enhancing the quality and naturalness of synthesized speech and harnessing the full potential of tts technology in diverse applications. ## xtts model xtts (eren and the coqui tts, 2021) foot_2 is a multilingual multispeaker speech synthesis model that supports arabic (ar) along with english (en), and multiple other languages. overall, xtts can generate natural-sounding audio with as short as three seconds of reference speech across 15 languages. it is based on autoregressive modeling trained on discrete audio representations. the model has a vq-vae to discretize audio into discrete tokens and subsequently uses gpt2 @xcite model to predict these audio tokens based on the input text conditioned upon latent speaker representation. the computation of speaker latent is facilitated by the perceiver model @xcite , which takes mel spectrogram as inputs and produces latent vectors representing speaker information to prefix the gpt decoder. xtts uses the hifigan model to compute the final audio signal from the gpt2 outputs which reduces the inference latency. the empirical findings indicate that the perceiver model outperforms simpler encoders, such as tortoise @xcite , and speech prompting models like vall-e @xcite , in capturing speaker characteristics. notably, the perceiver exhibits a remarkable consistency in model outputs across various runs, mitigating the occurrence of speaker shifting between different model executions. in addition to that, the intrinsic flexibility of the perceiver is highlighted by its ability to accommodate multiple references without imposing any length constraints. this feature facilitates the comprehensive capture of diverse facets of the target speaker, and the potential to amalgamate characteristics from different speakers, thereby enabling the creation of a distinctive and unique voice. in xtts, input text undergoes bpe tokenization, and a language token is inserted at the beginning, resulting in a structured sequence: appending audio tokens to the input sequence: s1, s2, ..., sk, [bots], [lang], t1, t2, ..., tn, [eots], [boas], a1, a2, ..., a_l, [eoas]. the three-stage training process for xtts involves vqvae, gpt, and audio decoder training. the three components of xtts model are demonstrated in appendix, fig.2. xtts is trained on publicly available datasets across different languages. since publicly available data varies for different languages, we believe that arabic has a relatively small amount of data compared to other high-resource languages such as english, german, and spanish that it supports. although the outputs from xtts seem high quality, developers of the model do not conduct any evaluation for any languages. to address this gap, we first train xtts on approximately 2000 hours of highquality arabic speech described in 4.1. we then perform a comprehensive evaluation using both an automatic metric as well human evaluation. ments of 27,977 speakers. transcripts in qasr are aligned with audio segments using light supervision. one of the primary considerations for selecting qasr for our experiments is that it has the speaker identity of audio recordings. libya 2.8% sudan 2.0% egypt 4.5% lebanon 3.2% palestine 3.3% algeria 6.2% oman 5.1% saudi arabia 1.8% yemen 7.1% iraq 8.3% jordan 1.2% syria 8.3% msa 39.6% text-audio processing. because the audios of the qasr dataset is originally crawled from a news channel, we need to remove the background noise to make the dataset suitable for training the tts system. we employ the frcrn @xcite ) system 7 to suppress the background noise of speech audios. moreover, when exploring the dataset, we found that some text-audio pairs are mismatched. the problem occurs when two people are speaking at the same time in the audio segment. to filter out these mismatched samples, we first employ the whisper @xcite model to generate the transcripts of all audio segments. subsequently, we compute the word error rate (wer) between the generated and ground-truth transcripts. we then remove all samples with wer larger than zero. dialect pseudo labelling. we train our multidialect xtts model on qasr, however, the original qasr data does not have a dialect associated with each segment. to overcome this we employ eight dialect identification models to predict the dialect from the text then we take the label with the highest cumulative confidence as the final dialect label. the models are collected from huggingface 891011 . we provide data distribution across dialects in figure 3 7 https://github.com/alibabasglab/frcrn 8 https://huggingface.co/amr-keleg/adi-nadi-2023 9 https://github.com/lafifi-24/arabic-dialect-identification 10 https://huggingface.co/camel-lab/bert-base-arabiccamelbert-mix-did-madar-corpus26 11 https://huggingface.co/camel-lab/bert-base-arabiccamelbert-msa-did-madar-twitter5 train-test splits. after the processing steps, there are 17341 remaining speakers, of which we use 17310 speakers for training and 31 speakers for evaluation. those 31 speakers are excluded from training for zero-shot setting evaluation. in addition to testing the split of qasr, we use a small amount of in-house dialect data to evaluate out-of-distribution generation quality. we collect this data from youtube and transcribe them using native speakers from the respective dialects. our inhouse dialectal data includes country-level dialects from algeria, jordan, palestine, uae, and yemen. ## results and discussion we fine-tune the xtts model with two settings on roughly 472 hours of multi-dialectal arabic speech data. we report automatic evaluation metrics secs and wer on held-out data 4.1 in in addition to the held-out dataset, we probe the performance of all models on the clean, labeled with strict supervision, in-house dialect dataset. we report the wer, secs, and mos score in ## conlusion generating speech from text has many critical applications particularly for individuals with visual and hearing impairments, enabling them to access and interact with digital content through screen readers, spreading across different regions and languages. however, previous research focuses on building text-to-speech systems for high-resource languages and often lacks comprehensive evaluation. in this work, we address this gap by finetuning two models for arabic text-to-speech generation. we then conduct a thorough evaluation and comparative analysis of pre-trained with ours. findings reveal that although our models are inconsistent in mapping phonemes to sounds, they more precisely preserve acoustic features compared to the baseline. in future work, we intend to train our models on extensive high-quality data while ensuring that the training data is representative of the diverse linguistic varieties of arabic. for instance, we would like to extract a subset of the qasr dataset and perform manual labeling to create a small, clean dataset. subsequently, we will employ semisupervised learning techniques to enhance our models' performance. finally, upon acceptance of our work, we intend to release the model weights, inference pipeline, and training protocols. ## limitations despite the models are capable of generating natural-sounding speech, comprehensible by native speakers, which is often indistinguishable from reference speakers, our work has some clear limitations which we discuss below: