C asablanca: Data and Models for Multidialectal A rabic Speech Recognition (2024)

Abstract: In spite of the recent progress in speech processing, the majority of world languages and dialects remain uncovered. This situation only furthers an already wide technological divide, thereby hindering technological and socioeconomic inclusion. This challenge is largely due to the absence of datasets that can empower diverse speech systems. In this paper, we seek to mitigate this obstacle for a number of Arabic dialects by presenting Casablanca, a large-scale community-driven effort to collect and transcribe a multi-dialectal Arabic dataset. The dataset covers eight dialects: Algerian, Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni, and includes annotations for transcription, gender, dialect, and code-switching. We also develop a number of strong baselines exploiting Casablanca. The project page for Casablanca is accessible at: www.dlnlp.ai/speech/casablanca.

## introduction self-supervised learning (ssl) has significantly advanced the field of speech processing, impacting everything from speech recognition to speech synthesis and speaker verification. however, the success of these methods heavily relies on the availability of large datasets, which are primarily available for a select few languages. this bias towards resource-rich languages leaves behind the majority of the world's languages @xcite @xcite . in this work, we report our efforts to alleviate this challenge for arabic-a collection of languages and dialects spoken by more than 450 million people. we detail a year-long community effort to collect and annotate a novel dataset for eight arabic dialects spanning both africa and asia. this new dataset, dubbed casablanca, is rich with various layers of annotation. in addition to speech transcriptions, we include speaker gender, dialect, and code-switching information. notably, to the best of our knowledge, some of the dialects included in casablanca have not been featured in any prior speech or broader nlp research. in addition to describing our dataset, we develop baseline systems for automatic speech recognition (asr). to summarize, our contributions are as follows: non-arabic asr data. similar efforts exist for collecting diverse speech datasets across various language varieties and dialects. for instance, stt4sg-350 @xcite introduces a swiss german corpus divided into seven dialect regions, annotated with standard german transcriptions. afrispeech @xcite also offers 200 hours of pan-african english speech, featuring 67,577 audio clips from speakers across 13 countries, encompassing 120 indigenous accents for both clinical and general asr applications. the mandi corpus @xcite provides a detailed spoken database of regional mandarin dialects and standard mandarin, with 357 recordings totaling about 9.6 hours from 36 speakers across six major regions. additional information on arabic asr can be found in appendix a.1. casablanca in comparison. casablanca is the largest fully supervised arabic dialects dataset with 48 hours of human-transcribed data, surpassing mgb-3 and mgb-5. although mgb-2 and qasr are larger in size, they utilize light supervision (using asr systems for transcribing and aligning human transcripts) rather than manual transcriptions. this light supervision method accounts for potential inaccuracies in human transcripts, such as omissions, errors, and variations from factors like corrections, spelling errors, foreign language use, and overlapping speech, leading to possible mismatches between the transcriptions and actual spoken content @xcite . casablanca is also the most fine-grained and diverse corpus available: while datasets such as mgb-2 and qasr focus on broad regional dialects like the gulf, the levant, and north africa (including egypt), casablanca targets country-level variation focusing on eight countries belonging to different areas in the arab world. to the best of our knowledge, our dataset is also the first to introduce zeroresourced dialects in addition to the low-resource ones (specifically the emirati, yemeni, and mauritanian dialects), thus filling a significant need in the research landscape. furthermore, casablanca is rich with several layers of annotation: beyond speech transcription, each segment is also labeled with speaker gender and country, which provide valuable demographic information and can be exploited for downstream tasks involving gender and dialect identification. ## dialects description casablanca is a detailed collection of around 48 hours of data covering eight arabic dialects from regions like the levant, gulf, yemen, and north africa, including algerian, egyptian, emirati, jordanian, mauritanian (hassaniya), moroccan, palestinian, and yemeni. casablanca involves subdialects from these countries as well. in addition, to the best of our knowledge, we are among the first to offer annotated data for the less-represented emirati, mauritanian, and yemeni dialects, addressing a gap in linguistic research. ## corpus statistics episode coverage. as spelled out earlier, we annotate approximately 48 hours of content across eight dialects. the average annotation duration per episode is about four minutes, constituting roughly 14.71% of the average episode length. dialects represented by a larger number of episodes typically exhibit lower per-episode annotation durations. this distribution allows annotators to engage with a more diverse range of content. for instance, mauritanian episodes, totaling 247, feature an average of only one minute and 25 seconds (8.23%) of annotation per episode. conversely, the palestinian subset, with 22 episodes, averages 16 minutes and 30 seconds per episode, which is about 53.72% of the total episode length 11 . average duration. as detailed in the average transcript length across all dialects is 8.64 words, with jordanian transcripts being the shortest and palestinian the longest. these differences, even between closely related dialects, stem from episode script lengths and annotator preferences for word separation, including prefixes and suffixes. for instance, in the jordanian dialect, the phrase ("i sent it to her") transcribed by some annotators as a single word: (" "), while others split it into two: (" ") or even three words: ## baseline models we split casablanca into train, dev, and test, keeping the latter two splits each at one hour of the data per country. we perform a number of asr experiments on the dev and test splits of casablanca foot_10 . first, we evaluate general speech models under a zero-shot condition. then, we evaluate models that were finetuned on msa or other dialects. finally, we report experiments on our code-switched data only. we report results in wer and cer, both with and without preprocessing of the data. details of our preprocessing pipeline are in appendix a.7. ## limitations while we believe casablanca will have a significant impact on a wide range of tasks in arabic speech, it is important to acknowledge some limitations. although casablanca includes eight dialects, 21 read-out books also trained on different language (i.e., english). ## ethical considerations in developing casablanca, we adhere to ethical principles to ensure responsible and respectful use of data. our dataset, sourced from publicly available tv series episodes on youtube, is curated with careful consideration for privacy, omitting any personal identifiable information beyond what is publicly accessible. we try our best to ensure diverse representation in terms of gender and dialects to mitigate biases and promote inclusivity in asr systems. all annotations and evaluations were conducted with linguistic and cultural sensitivity. while aiming to share the dataset to advance research, we implement access policies that require responsible use and proper citation. our commitment to ethical standards is ongoing, and we welcome community feedback to continuously improve our practices. speech directly to text, gained popularity, simplifying asr pipelines. @xcite introduced an e2e asr model for arabic, leveraging brnns with ctc for alignment. the introduction of an e2e transformer model addresses the morphological complexity and dialectal variations inherent in arabic using self-attention mechanism and sub-word tokenization. @xcite advanced arabic asr by employing a transformerbased encoder-decoder with a tdnn-lstm language model, using mel filter banks for acoustic features and training on mgb3 and mgb5 corpora, achieving leading performance with wers of 27.5% for mgb3 and 33.8% for mgb5. in the era of large speech models, arabic speech is still in its early stages. the xls-r model @xcite , a large-scale model designed for crosslingual speech representation learning, utilizing the wav2vec 2.0 framework @xcite , was utilized on the mozilla common voice dataset for msa @xcite . the study of @xcite benchmarks foundational models on arabic asr tasks, focusing on the performance of openai's whisper @xcite ), google's usm @xcite , and the kanari asr model. these models were evaluated against a variety of datasets, emphasizing their efficacy across different arabic dialects and speaking styles. notably, usm typically surpassed whisper, while kanari demonstrated exceptional capability, especially in code-switching contexts between msa and egyptian dialect. the performance of whisper across various arabic dialects for asr tasks was explored by @xcite . this evaluation spanned most publicly available datasets, utilizing n-shot (zero-, few-, full) fine-tuning approaches. the study also assessed whisper's adaptability to novel scenarios, including dialect-accented msa and previously unseen dialects. while whisper demonstrated competitive results with msa in zero-shot settings, its ability to adjust to different dialects was limited, showing inadequate performance and random output generation when encountering unfamiliar dialects.