A rabic Dialect Identification under Scrutiny: Limitations of Single-label Classification (2023)

Abstract: Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects of Arabic. We argue that the currently adopted framing of the ADI task as a single-label classification problem is one of the main reasons for that. We highlight the limitation of the incompleteness of the Dialect labels and demonstrate how it impacts the evaluation of ADI systems. A manual error analysis for the predictions of an ADI, performed by 7 native speakers of different Arabic dialects, revealed that \approx 66% of the validated errors are not true errors. Consequently, we propose framing ADI as a multi-label classification task and give recommendations for designing new ADI datasets.

## introduction adi of text is an nlp task meant to determine the arabic dialect of the text from a predefined set of dialects. arabic dialects can be grouped according to different levels (1) major regional level: levant, nile basin, gulf, gulf of aden, and maghreb (2) country level: more than 20 arab countries, and (3) city level: more than 100 micro-dialects @xcite . different datasets were built curating data from various resources with labels of different degrees of granularities: (1) regional-level @xcite , (2) country-level @xcite @xcite , or (3) city-level @xcite @xcite . despite attracting lots of attention and effort for over a decade, adi is still considered challenging, especially for the fine-grained distinction of micro-arabic dialects on the country and city levels. this ## maximal accuracy of single-label adi datasets for a single-label adi dataset consisting of sentences where each is assigned one dialect label, assume that a percentage perc 2 of those sentences is valid in 2 different dialects. for those sentences, only one of the valid dialects is listed as their label. an effective model trained to predict a single label will randomly assign each of these sentences to one of its two valid labels. thus, the expected maximal accuracy on the dataset e[accuracy max ] that the model can achieve would then be: for example, if 40% of the sentences are valid in two dialects (i.e.: perc 2 = 40%), then the e[accuracy max ] of the dataset would be 80%. this becomes worse when a sentence is valid in more dialects, exceeding ten valid dialects in some cases (as shown in perc @xmath0. the higher the percentages perc n where @xmath1, n dialects ], the lower the maximal accuracy would be. the same pattern would apply to f1 scores. therefore, a model might be achieving low f1 scores as a consequence of framing di as a single-label classification task, which might result in high perc n values. our objective in this paper is to estimate the value of e[accuracy max ] for the existing datasets, which should examine the validity of our hypothesis that modeling adi task as a single-label classification can be highly sub-optimal. ## estimating the maximal accuracy of datasets in our study, we focus on the country-level adi for which multiple shared tasks have been organized since 2019 @xcite @xcite . in order to quantify the percentages p erc n , each sample of a dataset needs to be assessed by native speakers from all the arab countries. given our inability to recruit participants from all the arab countries, we will estimate the percentages using two methods that provide lower bounds perc n for the actual values perc n (i.e.: perc @xmath2. consequently, the estimated maximal accuracy is an upper bound for its true value. ## proposal for framing the adi task given the limitations of using single-label classification for the adi task, elaborated in §4, we propose alternative modeling for adi. zaidan and callison-burch (2014) asked crowdsourced annotators to label dialectal sentences as being egyptian, gulf, iraqi, levantine, maghrebi, 1 we release the judgments through: github.com/ amr-keleg/adi-under-scrutiny/tree/master/data 2 participants are given a third choice maybe / not sure, which we count as no (i.e.: invalid in their dialect). algeria egypt lebanon palestine saudi arabia sudan syria fp's predicted dialect 0 25 50 75 100 25 17 24 69 38 41 26 59 35 97 7 5 10 37 correctness of fps of selected dialects correct fp incorrect fp figure 4: the distribution of the annotations for the validity of the false positives (fps) in 7 arabic dialects. correct fp represents the fp samples for which the model's prediction is invalid. incorrect fp the fp samples for which the model's prediction is valid. algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label 0 50 prediction: algeria correct fp incorrect fp algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label prediction: egypt correct fp incorrect fp algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label 0 50 prediction: lebanon correct fp incorrect fp algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label prediction: palestine correct fp incorrect fp algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label 0 50 prediction: saudi arabia correct fp incorrect fp algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label prediction: sudan correct fp incorrect fp algeria libya morocco tunisia egypt sudan bahrain iraq kuwait oman qatar saudi arabia uae jordan lebanon palestine syria yemen original label 0 50 prediction: syria correct fp incorrect fp that the annotators tend to over-identify their native dialects. annotators might not realize that a sentence valid in their native dialect is also valid in other dialects, and thus can end up choosing their native dialect as the label for this sentence, instead of the general dialect label. @xcite focused on the binary distinction between two varieties of english, portuguese, and spanish. in addition to the two varieties of each language, the annotators are allowed to assign sentences to a third label both or neither. the evaluation results indicate that the both or neither label is harder to model computationally than the other variety labels. the authors noted that there is room for improvement in the treatment and modeling of this third label. consequently, we believe that adding another label such as general or both or neither does not completely solve the limitations of single-label classification datasets. conversely, framing the ## conclusion single-label classification has been the defacto framing for arabic dialect identification (adi). we show that such framing implies that any model would have a maximal accuracy that is less than 100%, since some samples are valid in multiple dialects, and thus their labels are randomly assigned from these dialects in which they are valid. for a set of 490 validated false positives (fps) of an adi model, we found that the model's predicted dialects for 325 of them are also valid. the fact that about 66% of the fps are not true errors hinders the ability to analyze and improve the adi models, and hurts the reliability of the evaluation metrics. given this major limitation of single-label framing, we argue that adi should be framed as a multi-label task. this follows the recommendation of @xcite for french dialect identification. we hope that this paper will spark discussions across the arabic nlp community about the current state of adi, and encourage the creation of new datasets in a multi-label setup, with labels assigned manually by native speakers of the different arabic dialects. for future work, we will investigate the impact of the arabic level of dialectness (aldi) variable introduced by @xcite on identifying the dialect of sentences. intuitively, the dialect of a sentence with a high aldi score is easier to identify since the sentence shows more features of dialectness than those of sentences having low aldi scores. therefore aldi can be used to identify the samples that are more expected to be valid in multiple dialects, facilitating the annotation process of new di datasets.