Estimating the Level of Dialectness Predicts Inter-annotator Agreement in Multi-dialect A rabic Datasets (2024)

Abstract: On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respective dialects to build higher-quality datasets. However, automatically identifying the dialect of samples is hard. Moreover, the pool of annotators who are native speakers of specific Arabic dialects might be scarce. Arabic Level of Dialectness (ALDi) was recently introduced as a quantitative variable that measures how sentences diverge from Standard Arabic. On randomly assigning samples to annotators, we hypothesize that samples of higher ALDi scores are harder to label especially if they are written in dialects that the annotators do not speak. We test this by analyzing the relation between ALDi scores and the annotators’ agreement, on 15 public datasets having raw individual sample annotations for various sentence-classification tasks. We find strong evidence supporting our hypothesis for 11 of them. Consequently, we recommend prioritizing routing samples of high ALDi scores to native speakers of each sample’s dialect, for which the dialect could be automatically identified at higher accuracies.

## introduction arabic is spoken natively by over 420 million people and is an official language of 24 countries @xcite , making it an important language for nlp systems. however, nlp for arabic faces a major challenge in that usergenerated text is typically a mixture of modern standard arabic (msa)-the standardized variant that is taught in schools and used in official communications and newspapers-and regional variants of dialectal arabic (da), which are used in everyday communications, including both speech and social media text @xcite . 1 while msa can be largely understood by most arabic speakers, the different variants of da are not always fully mutually intelligible. despite this mutual unintelligibility, a common practice when developing datasets for multi-dialect arabic nlp is to randomly recruit annotators without regard to their dialect. however, routing dialectal content to speakers of a different dialect for annotation or moderation can present real problems. for example, it has been shown to contribute to unjust online content moderation of da (business for social responsibility, 2022), and racially biased toxicity annotation in american english varieties @xcite . two recent studies of multi-dialect da annotation showed that for annotating hate speech or sarcasm, respectively, annotators were more lenient (for hate speech) and more accurate (for sarcasm) when annotating sentences in their native dialect @xcite . the authors of both studies made the same recommendation for creating new arabic datasets, namely to first identify the dialect of each sample and then route it to appropriate annotators. this recommendation is theoretically appealing, but presents practical difficulties since automatic dialect identification (di) is challenging @xcite , and existing systems assume a single correct label when in fact some texts can be natural in different dialects @xcite . moreover, the representation of native speakers of the different arabic dialects on crowdsourcing sites might be skewed @xcite . therefore, recruiting native speakers of some arabic dialects might be challenging, given the tough conditions of the countries in which these dialects are spoken. in this paper, we address these challenges by building on recent work by @xcite , who presented a system for estimating arabic level of dialectness (aldi)-i.e., the degree to which a sentence diverges from msa, on a scale from 0 to 1. we hypothesize that as sentences with low aldi scores do not diverge much from msa, they can still be understood and accurately annotated by most arabic speakers, while this will be less true for sentences with high aldi scores. if our hypothesis holds, then annotation can be made more efficient while maintaining accuracy, by routing samples with low aldi scores to speakers of any dialect. only high-aldi samples need to be routed to native speakers of the appropriate dialect. we test our hypothesis by investigating the impact of aldi score on interannotator agreement for 15 publicly released datasets annotated for 6 different sentence-classification tasks. 2 we confirm that for most tasks and datasets, higher aldi scores correlate with lower annotator agreement. a notable exception is the dialect identification (di) task, where higher aldi scores correlate with higher agreement, presumably because it is easier to identify a single dialect for sentences that are strongly dialectal. this finding is encouraging for annotation routing, since automatic di systems may also have higher accuracy on these sentences. we conclude that a combination of automatic aldi scoring, followed by di and annotator routing only for high-aldi sentences, is a promising strategy for annotating multi-dialect arabic datasets. ## methodology data we study the impact of aldi scores on the annotators' agreement for publicly released arabic datasets. we analyze datasets satisfying the following criteria: analysis for each dataset, we compute the arabic level of dialectness (aldi) score for each annotated sample (sentence) using the sentence-aldi model @xcite , which returns a score from 0 (msa/non-dialectal) to 1 (strongly dialectal). to investigate the effect of aldi on annotator agreement, we bin the samples by their aldi score into 10 bins of width 0.1. we compute % full agree, the percentage of samples in that bin for which all the annotators agreed on a single label. we employ pearson's correlation coefficient to analyze the relation between aldi (represented by each bin's midpoint aldi score) and % full agree, and also report the slope of the best-fitting line as a measure of the effect size. 4 as aforementioned, our initial hypothesis is that % full agree negatively correlates with high aldi scores. ## results and discussion we use scatter plots to visualize the relation between % full agree and aldi on the studied datasets, as shown in figure 1 . additionally, the histograms of samples across the different bins indicate the dialectal content within the dataset. as per mpold (chowdhury et al., 2020) offensive (2) 27.82% 4000 sentences interacting with news sources, sampled from twitter, facebook, and youtube, annotated three times. youtube cyberbullying offensive (2) 10.24% 15,050 comments and replies to 9 youtube videos labeled by 3 annotators (iraqi, egyptian, libyan). (ytcb) @xcite asad @xcite sentiment (3) 35.63% 95,000 tweets with a skewed representation toward the gulf area and egypt. arsas @xcite sentiment (4) 57.45% 21,064 tweets related to a pre-specified set of entities or events, with confidence scores for the majority votes across three annotations per sample. speech act (6) dialect (5) 57.44% 10,547 tweets, sampled from two different sentiment analysis datasets: atsd @xcite , semeval2017 @xcite , reannotated for sentiment, dialect, and sarcasm. @xcite sarcasm ( 2 ) sentiment (4) mawqif sarcasm (2) 58.04% 4,121 tweets about "covid-19 vaccine", "digital transformation", or "women empowerment" annotated separately for stance and sentiment/sarcasm till the label confidence reaches 0.7 (min. 3 annotators) or 7 annotators label the sample. ( alturayeif et al., 2022) sentiment (3) 58.04% stance (3) 57.99% isarcasm's test set dialect (5) 30.5% 200 sarcastic sentences provided by crowdsourced authors and 1200 non-sarcastic tweets from arsarcasm-v2 @xcite reannotated 5 times. @xcite sarcasm ( 2 ) dart @xcite dialect (5) 0.8% 24,279 tweets with distinctive dialectal terms annotated three times for the dialectal region. samples of complete disagreement are not in the released dataset. for non-di tasks, aldi negatively correlates with agreement. inspecting the trends depicted in figure 1 , strong negative pearson's correlation dataset sample translation of the underlined cue aldi individual labels ytcb hatespeech detection you insect 0.98 hatespeech(3x) you scabies 0.81 hatespeech(3x) gulf's colt 0.98 hatespeech(3x) arsarcasm-v1 sentiment analysis i found him disgusting 0.81 negative(2x) positive(1x) # # hahaha 0.94 positive(2x) negative(1x) for di, agreement is lowest for mid-range aldi scores (if msa is a possible label) or low aldi scores (if it is not). by definition, msa sentences have an aldi of 0, and normally the aldi estimation model assigns them very low scores. for the arsarcasm-v1 and isarcasm datasets, the set of labels for the di task includes msa (i.e., some sentences in these datasets are not dialectal). for both datasets, one notices high percentages of agreement scores for the bin having aldi @xmath1, 0.1] (generally agreeing that the label is msa). the percentages decrease for the few succeeding bins, before rising again for the bins with high aldi scores. sentences of high aldi scores (e.g., ∈ [0.8, 1]) are expected to have multiple dialectal cues, which increases the chance of attributing them to a single dialect. for sentences of intermediate aldi scores, annotators can agree a sentence is not in msa, however, they would struggle to determine the dialect of the sentence, which is manifested as having lower percentages of the full agreement for these bins. 5 refer to appendix §e for a possible explanation of the unexpected trends of the arsas dataset. the authors of the dart dataset do not include msa in the labels since they curated sentences with distinctive dialectal terms. this explains the low percentage of full agreement for the bin of aldi @xmath2, 0.1], unlike the other two di datasets. however, the pattern of having higher full agreement percentages for bins with higher aldi scores still holds. ## analysis of trends by class label a more nuanced analysis of the non-di datasets can be done by splitting the samples according to their majority-vote labels (see appendix §d). the declining trend for agreement as aldi scores increase is consistently salient for the negative class of the different tasks (i.e., not sarcastic, neutral (sentiment), non-offensive). one explanation is that agreeing on one of these labels requires fully understanding the sample, which is expected to be harder for non-native speakers of the dialects of sentences with high aldi scores. moreover, the general usage of msa for formal communications, and da for more personal ones might be biasing the annotators. further controlled analysis is required to investigate these intuitive explanations. conversely, the presence of specific words might be a strong cue for the label of the overall sentence, which might be useful for the positive classes (i.e., sarcastic, positive/negative, obscene/offensive). consequently, annotators might be able to agree on a label for a sentence even if it contains some unintelligible segments. ## conclusion and recommendation factors such as task subjectivity and vague guidelines could cause disagreement between annotators. for arabic, we demonstrate that the arabic level of dialectness of a sentence (aldi), automatically estimated using the sentence-aldi model @xcite , is an additional overlooked factor. analyzing 15 datasets, we find strong evidence of a negative correlation between aldi and the full annotator agreement scores for 8 of the 12 non-dialect identification datasets. moreover, for the 3 dialect identification datasets, we find that annotators have higher agreement scores for samples of higher aldi scores, which by definition would have more dialectal features. the combination of more dialectal features in a sentence is more probable to be distinctive of a specific dialect. previous research recommended routing samples to native speakers of the different arabic dialects for higher annotation quality. our analysis indicates that a large proportion of 6 datasets are samples with aldi @xmath3.1, which are expected to be msa samples that can be routed to speakers of any arabic dialect. moreover, the lower agreement scores for samples with high aldi scores show that extra care should be given to these samples. dataset creators should prioritize routing high-aldi samples to native speakers of the dialects of these samples, for which the dialects can be automatically identified at higher accuracy as these samples have more dialectal cues.