A rabic dialect identification: An in-depth error analysis on the MADAR parallel corpus (2023)

Abstract: This paper provides a systematic analysis and comparison of the performance of state-of-the-art models on the task of fine-grained Arabic dialect identification using the MADAR parallel corpus. We test approaches based on pre-trained transformer language models in addition to Naive Bayes models with a rich set of various features. Through a comprehensive data- and error analysis, we provide valuable insights into the strengths and weaknesses of both approaches. We discuss which dialects are more challenging to differentiate, and identify potential sources of errors. Our analysis reveals an important problem with identical sentences across dialect classes in the test set of the MADAR-26 corpus, which may confuse any classifier. We also show that none of the tested approaches captures the subtle distinctions between closely related dialects.

## introduction dialect identification (did) is a task in natural language processing (nlp) aiming to automatically identify a dialect within a pre-determined language. because dialectal differences tend to be subtle, identifying dialects is considered a more difficult task than language identification @xcite . arabic dialects are considered particularly challenging due to their high level of ambiguity, lack of standardisation, and rich morphology @xcite . most nlp development has focused on modern standard arabic (msa), the formal and standardised version of arabic. however, these tools are not always transferable to dialectal arabic, as dialects differ from each other and msa in terms of lexicon, phonology, orthography, and morphology @xcite . a prominent resource for arabic did is the madar parallel corpus @xcite , targeting dialects on the city-level. madar has been established as an important corpus for the task, serving as a benchmark for multi-task learning @xcite , as well as a shared task corpus @xcite , and as a subject of independent research @xcite . despite several attempts to develop models using deep neural networks @xcite and pre-trained transformer-based language models @xcite , the current state-of-theart approach remains a statistical machine learning model with surface-level feature representation, specifically the multinomial naive bayes (mnb) model introduced by @xcite . the lack of progress on the task, along with the inability of bert models to surpass the mnb model, gives rise to several questions that have not yet been thoroughly explored, and on which we focus in the current work. firstly, do bert models make the same mistakes as the state-of-theart mnb model on the dialect identification task? while @xcite have documented the performance of the mnb model on individual dialects and highlighted the muscat dialect as the most challenging for the model, there is limited research exploring the misclassifications generated by bert models. secondly, if the models make different errors, are these errors centred around the same dialect pairs? thirdly, we explore if a detailed analysis of the misclassified sentences by both the bert models and the mnb model can provide deeper insights into the challenges of the task on madar-26. this paper summarises the findings from a comprehensive project on error-analysis on the madar parallel corpus conducted by @xcite . we release the code for all experiments and analysis on github. foot_0 ## previous work several efforts have focused on building tools and resources to identify arabic dialects. however, the field suffers from fragmented and independent works on different corpora that vary in terms of granularity, size and domain, making it challenging to track the progress of the solutions. early work focused on binary dialect classification by discriminating one dialect from msa @xcite , as well as identifying arabic dialects at both a region-level @xcite @xcite and a country-level @xcite @xcite . in recent years, more efforts have targeted arabic did on a more fine-grained level, particularly through shared tasks. the nuanced arabic dialect identification shared tasks (nadi) @xcite @xcite include sub-tasks on country-and province-level on usergenerated tweets. several corpora of written arabic dialects comprise tweets @xcite @xcite , others consist of user commentaries @xcite , or manually translated sentences @xcite . for the nadi shared tasks @xcite @xcite , all the top performing systems used transformer-based language models pretrained on dialectal arabic. however, these models yielded unsatisfactory results and multiple factors were identified, including imbalanced class distribution (alshenaifi and azmi, 2020), a significant presence of msa content in the training data @xcite , and the inherent challenges associated with distinguishing between arabic dialects. within the madar shared task @xcite , the top five performing systems demonstrate that ensemble techniques, n-gram-based features, and traditional machine learning approaches, such as mnb or support vector machines (svms), yield the highest levels of performance. while the madar corpus proved to be too small for deep learning architectures @xcite , the transfer learning ability of bert-based language models, pre-trained on dialectal arabic, has shown promising results @xcite . however, the mnb model introduced by @xcite is still state-of-the-art with an overall accuracy of 67.9%. ## the madar corpus the madar corpus is a collection of parallel sentences in the travel domain @xcite . the resource contains two corpora with non-overlapping sentences: (1) madar-26: covering 25 cities and msa, and where each dialect is represented with 2000 sentences. (2) madar-6: covering the five selected cities doha, beirut, rabat, cairo, and tunis, in addition to msa, each with 12000 sentences. we use the training, development, and test splits from the madar shared task 1 @xcite shown in ## models we here describe our experimental set-up and the tested models. ## test results we compare the performance of both selected models, mnb and arabert, in terms of f 1 score for each individual dialect in ## error analysis we here provide a systematic analysis of the errors made by the different models. ## conclusion and future work this paper investigates the challenging task of fine-grained dialect identification, focusing on the madar-26 corpus. by fine-tuning three bert models pre-trained on dialectal arabic, we demonstrated that the multinomial naive bayes model introduced by @xcite remains the stateof-the-art model on this data. however, we identified 480 test sentences that were correctly classified by the best performing bert model, but were misclassified by the mnb model. a comprehensive error analysis revealed the bert model exhibits superior performance in predicting sentences in muscat dialect and msa, which may be attributed to the amount of msa content in the pre-training data of the bert model. we also show that some of the challenges of the task can be attributed to dataset limitations. particularly the fact that 10% of the sentences in the test set are identical to one or more parallel sentences in the same set but with different labels. our analysis of different error types confirms that the mnb and bert-based model often make different mistakes, but also that a subset of the test data is challenging for both. notably, we found that the moroccan city dialects rabat and fes are the most confused dialect pair, and show how neither approach is able to capture the subtle distinctions between some of the closely related dialects. although dataset limitations, such as non-arabic proper nouns, short sentences without dialect-specific features, and identical sentences across classes, account for some of these errors, the unique errors generated by each model provide evidence that certain sentences can be correctly classified by one model, but not the other. these findings underscore the need to examine model performance beyond simple metric comparison in order to identify new strategies for enhancing arabic dialect identification. in the future, we would like to address the formulation of the task, by transforming it into a multilabel classification problem. instead of simply removing the duplicate sentences from the data, we can combine the labels of duplicate and nearlyduplicate text, converting the single-label dataset into a multi-label dialect classification format. another avenue for future research is to evaluate models trained or fine-tuned on madar-26 on user-generated data. due to a lack of annotated data matching the city-levels of madar, evaluation on data outside the travel domain has up until recently not been possible. however, the hierarchical mapping schema proposed by @xcite can be leveraged for datasets with comparable or more detailed annotations. more specifically, we want to evaluate the performance of the models on the nadi dataset @xcite by mapping tweets at the province-level to the citylevel.