A r TST : A rabic Text and Speech Transformer (2023)

Abstract: We present ArTST, a pre-trained Arabic text and speech transformer for supporting open-source speech technologies for the Arabic language. The model architecture follows the unified-modal framework, SpeechT5, that was recently released for English, and is focused on Modern Standard Arabic (MSA), with plans to extend the model for dialectal and code-switched Arabic in future editions. We pre-trained the model from scratch on MSA speech and text data, and fine-tuned it for the following tasks: Automatic Speech Recognition (ASR), Text-To-Speech synthesis (TTS), and spoken dialect identification. In our experiments comparing ArTST with SpeechT5, as well as with previously reported results in these tasks, ArTST performs on a par with or exceeding the current state-of-the-art in all three tasks. Moreover, we find that our pre-training is conducive for generalization, which is particularly evident in the low-resource TTS task. The pre-trained model as well as the fine-tuned ASR and TTS models are released for research use.

## introduction large pre-trained transformer models are currently at the forefront of speech and text technologies, with applications in various text and speech recognition and generation tasks @xcite @xcite . these models share several aspects: (1) they are based on the transformer architecture @xcite , which enables efficient training of larger models and incorporating wider contexts, (2) they are scaled in terms of model size, which has been shown to correlate with performance @xcite , and (3) they generally use a self-supervised training objectives, such as next token prediction @xcite , masked prediction @xcite , and contrastive loss (baevski et al., * these authors contributed equally to this work. 2020), which enable the utilization of large unlabeled datasets for multiple potential downstream tasks. pre-trained self-supervised models like wav2vec2.0 @xcite , and its multilingual variant @xcite , have mostly replaced traditional acoustic features like mfccs and filter banks in the speech domain. these pretrained models implicitly learn robust and generalizable acoustic representations that consistently improve performance in various supervised downstream tasks with acoustic inputs like automatic speech recognition (asr). this is achieved by simply adding a prediction layer and fine-tuning the model using a suitable loss function, such as ctc loss @xcite . this pre-train-then-finetune framework is flexible for a variety of applications, but most pretrained models are uni-modal and therefore are limited to tasks that share the same input modality. for instance, acoustic models like wav2vec2.0 are not typically used in text-to-speech synthesis applications, where the input is text, and the output is typically in the form of mel spectrograms. for this reason, self-supervised pre-training has not been as widely adopted in speech synthesis research. one exception to this trend is the speecht5 model @xcite , which accepts both text and speech as input and output using modal-specific networks in addition to the core encoder-decoder network. the model is first pre-trained using self-supervised objectives in both text and speech modalities, and then fine-tuned on a variety of supervised tasks, including speech transcription, speech synthesis, and speech classification. speecht5 has been trained only on english using more than 900 hours of speech and 400 million sentences of text data. while the model can technically be fine-tuned for other languages, our preliminary evaluations of arabic fine-tuning show poor performance; the pretraining seems to have biased the model severely for recognizing and generating english speech. in this paper, we introduce arabic text and speech transformer, artst 1 , a project aiming to push the boundaries for arabic open-source speech technology by providing various pre-trained speech and text transformers. the arabic language exhibits significant dialectal variation and codeswitching, which introduce a layer of complexity for speech recognition and generation tasks. we believe this can be best addressed via methodical and focused development of self-supervised models that target this linguistic landscape rather than multi-lingual models that may compromise mono-lingual performance for multi-lingual coverage. the first release, as described in this paper, is a direct adaptation of the speecht5 model, but pre-trained from scratch using modern standard arabic data and evaluated on various downstream tasks. future versions will include dialectal arabic, as well as code-switched speech and text, by exploring the best architectural modifications for improving coverage without sacrificing performance for individual variants. we demonstrate the performance of artst in the following tasks: automatic speech recognition (asr), text-to-speech synthesis (tts), and spoken dialect identification (did). the fine-tuned models on each task achieved performance on a par with or exceeding previously reported results on our test sets, establishing a new state-of-the-art for open-source models. for asr, the model additionally outperforms the large pre-trained asr models, whisper @xcite , and mms @xcite , which further demonstrates the advantage of focusing only on arabic. moreover, we report some interesting findings in tts fine-tuning, as the model learns to synthesize speech without explicit text diacritization in a way that generalizes to unseen domains, which we believe is a result of the unsupervised pre-training on large arabic speech data. our main contributions are: ## artst artst is a text and speech transformer optimized for the arabic language. based on observations from previous studies on multilingual and monolingual asr, tts, and self-supervised pre-training, we believe that training a model from scratch with the arabic language in mind would improve the quality of the resulting models. our strategy is to start with a monolingual setting, and explore the optimal settings for modern standard arabic (msa) speech processing. in future iterations of the model, we will explore how best to expand it to handle various dialects as well as other languages that are often mixed with arabic (i.e. english and french). we believe that an incremental approach of this kind is more likely to lead to optimal performance. here, we describe the first stage of this project, which focuses only on msa. artst is adapted from the transformer-based speecht5 model, which we briefly describe in this section. for more details, please refer to @xcite . ## conclusions & future work we demonstrated the potential of artst in speech recognition, synthesis, and classification, where we achieved results on a par with or outperforming previously reported results with relatively straightforward fine-tuning. what we have demonstrated in this paper is only a subset of potential applications of this framework. as the model can handle both text and speech modalities, it can potentially be applied for text-to-text and speech-to-speech applications, in addition to text classification and generation tasks. we will explore these avenues of application in future work. in this initial work, we focused on msa as the main variant of arabic for pre-training. we explored the potential of the model to generalize to dialectal arabic using small test sets that include dialectal arabic, as well as the dialect identification task. future edition will focus on expanding the coverage of the pre-trained model to include various dialects, and potentially code-switched speech, without sacrificing performance on msa. as demonstrated in this paper, our model outperforms larger multi-lingual models like whisper and mms, which we believe is a result of focusing on the arabic language as a basis of our model from its inception. while multi-linguality may be desirable for some applications, and could be beneficial for low-resource languages, monolingual models have a greater potential for highresource languages, and the arabic language currently boasts large volumes of open-source datasets that can be utilized to develop high-quality models across various tasks.