V ox A rabica: A Robust Dialect-Aware A rabic Speech Recognition System (2023)

Abstract: Arabic is a complex language with many varieties and dialects spoken by ~ 450 millions all around the world. Due to the linguistic diversity and vari-ations, it is challenging to build a robust and gen-eralized ASR system for Arabic. In this work, we address this gap by developing and demoing a system, dubbed VoxArabica, for dialect identi-fication (DID) as well as automatic speech recog-nition (ASR) of Arabic. We train a wide range of models such as HuBERT (DID), Whisper, and XLS-R (ASR) in a supervised setting for Arabic DID and ASR tasks. Our DID models are trained to identify 17 different dialects in addition to MSA. We finetune our ASR models on MSA, Egyptian, Moroccan, and mixed data. Additionally, for the re-maining dialects in ASR, we provide the option to choose various models such as Whisper and MMS in a zero-shot setting. We integrate these models into a single web interface with diverse features such as audio recording, file upload, model selec-tion, and the option to raise flags for incorrect out-puts. Overall, we believe VoxArabica will be use-ful for a wide range of audiences concerned with Arabic research. Our system is currently running at https://cdce-206-12-100-168.ngrok.io/.

## introduction the arabic language, with its diverse regional dialects, represents a unique linguistic spectrum with varying degrees of overlap between the different varieties at all linguistic levels (e.g., phonetic, syntactic, and semantic). in addition to modern standard arabic (msa), which is primarily used in education, pan-arab media, and government, there are many local dialects and varieties that are sometimes categorized at regional @xcite @xcite , country (bouamor et al., ⋆ equal contributions 2018; @xcite @xcite , or even province levels @xcite . historically, this wide and rich variation between different arabic varieties has posed a significant challenge for automatic speech recognition (asr) @xcite @xcite . the main focus has largely been on the recognition of msa with very little-to-no focus on its dialects and varieties @xcite @xcite . as such, asr systems have conventionally been built either for msa or individual dialects, thereby restricting their versatility and adaptability. however, the multifaceted nature of arabic demands a robust asr system that caters for its diverse dialects and varieties. in this work, we fill this research gap by introducing and demoing an asr system integrated with a dialect identification model, dubbed voxarabica. voxarabica is an end-to-end dialect-aware asr system with dual functionality: (i) it offers a supervised dialect identification model followed by (ii) a finetuned whisper arabic asr model covering multiple dialects. the dialect identification model works by assigning a country-level dialect, as well as msa, from a set of 18 labels from input speech. this then allows the appropriate asr model to fire. contrary to traditional methodologies that separate dialect identification and speech recognition as two completely different tasks, our proposed pipeline integrates the two components effectively utilizing dialectal information for improved speech recognition. such an integration not only improves the asr output, but also establishes a framework aligned with the linguistic diversities inherent to arabic as well. concretely, our contributions can be summarized as follows: the rest of the paper is organized as follows: in section 2, we overview related works. section 3 introduces our methods. section 4 offers a walkthrough of our demo. we conclude in section 5. for arabic asr specifically, the first e2e model was introduced using recurrent neural networks coupled with connectionist temporal classification (ctc) @xcite . subsequent works have built upon this foundation, including the development of transformer-based models that excel in both msa and dialects @xcite . one challenge for e2e asr models is the substantial requirement for labeled data, particularly for languages with fewer resources such as varieties of arabic. to address this, self-supervised and semi-supervised learning approaches are gaining traction. these models, such as wav2vec2.0 and xls-r, initially learn useful representations from large amounts of unlabeled or weakly labeled data and can later be finetuned for specific tasks @xcite . w2v-bert, another self-supervised model, employs contrastive learning and masked language modeling. it has been adapted for arabic asr by finetuning on the fleurs dataset, which represents dialect-accented standard arabic spoken by egyptians @xcite . unlike whisper, both wav2vec2.0 and w2v-bert necessitate a finetuning stage for effective decoding. arabic did. arabic did has been the subject of a number of studies through recent years, enhanced by collection of spoken arabic did corpora such as adi5 @xcite and adi17 @xcite . and advances in model architecture have mirrored changes in the larger lid research community, from i-vector @xcite based approaches @xcite towards deep learning based approaches: x-vectors @xcite , end-to-end classification using deep neural networks @xcite , and transfer learning @xcite . asr and did. combining asr and did in a single pipeline remains fairly novel for arabic. recent works in this space has employed only limited corpora @xcite , or used asr transcripts only to improve did @xcite . closest to our demonstrated system in this work is farspeech @xcite , since it combines asr and did. however, farspeech is confined to coarse-grain did and only supports msa for asr. in addition, compared to farspeech, our models are modular in that it allows users to run either or both asr or did, depending on their needs. ## walkthrough our demo consists of a web interface with versatile functionality. it allows users to interact with the system in multiple ways, depending on their needs. user audio input. users can either record their own audio through a microphone or upload a prerecorded file. in both cases, we allow different formats such as .wav, .mp3, or .flac, across various audio sampling rates (e.g., 16khz or 48khz). figure 1 demonstrates the different options available to the user upon interacting with voxarabica. model name dialect(s) dataset architecture whisper msa msa cv (6.1, 9.0, 11.0) whisper xls-r msa cv (6.1, 9.0, 11.0) wav2vec 2.0 whisper morroco mor mgb5 whisper whisper egypt egy mgb3 whisper whisper zero-shot --whisper mms --wav2vec 2.0 model selection. users can choose to select an arabic variety for transcription, or have it automatically detected using our 18-way did system. we demonstrate this in figure 2 . once the variety is detected, the corresponding asr model will perform transcription and both did transcription results will be presented on the interface (as shown in figure 3 ). we offer various models: two for the egy and mor, respectively; two for msa; and two generic models that can be used for any variety. we list all models in it is important to note that we do not collect any external user data for any purpose, thus ensuring user privacy. system output. our system conveniently outputs both predicted arabic variety and transcription across two panels as shown in figure 3 . for predicted variety, we show users all top five predictions along with model confidence for each of them. we provide outputs produced by our models in voxarabica when the reference input is egyptian dialect in ## conclusion we present a demonstration of combined did and asr pipeline to illustrate the potential for these systems to improve the usability of dialectal arabic speech technologies. we report example outputs produced by our system for multiple dialects showcasing the effectiveness of integrated did and asr pipelines. we believe that our demo will advance the research to build a robust and generalized ara-bic asr system for a wide range of varieties and dialects and will enable a more holistic assessment of the strengths and weaknesses of these methods. for future work, we intend to add models for more dialects and varieties particularly those which are low resource.