The Effect of A rabic Dialect Familiarity on Data Annotation (2022)

Abstract: Data annotation is the foundation of most natural language processing (NLP) tasks. However, data annotation is complex and there is often no specific correct label, especially in subjective tasks. Data annotation is affected by the annotators’ ability to understand the provided data. In the case of Arabic, this is important due to the large dialectal variety. In this paper, we analyse how Arabic speakers understand other dialects in written text. Also, we analyse the effect of dialect familiarity on the quality of data annotation, focusing on Arabic sarcasm detection. This is done by collecting third-party labels and comparing them to high-quality first-party labels. Our analysis shows that annotators tend to better identify their own dialect and they are prone to confuse dialects they are unfamiliar with. For task labels, annotators tend to perform better on their dialect or dialects they are familiar with. Finally, females tend to perform better than males on the sarcasm detection task. We suggest that to guarantee high-quality labels, researchers should recruit native dialect speakers for annotation.

## introduction many natural language processing (nlp) tasks rely on training machine learning (ml) models on labelled data. the labels are assigned in different approaches, amongst the most common ones is human annotation. these labels are sometimes highly subjective and might be affected by annotators' backgrounds and beliefs. such subjectivity would have minimal effects for objective tasks where people have consensus @xcite . however, these differences can be disruptive when considering subjective tasks such as sentiment analysis @xcite , sarcasm detection @xcite , hate speech @xcite and many others. this applies to all languages, but for arabic, it is more important due to the large dialectal variety amongst arab annotators. arabic has three variants; the first is classical arabic (ca), which is the language of quran and early literature. the second is modern standard arabic (msa), which is standardized and mainly used in news and books. the third is dialectal arabic (da), which is the colloquial language spoken in everyday life and it varies from one region to another. da differs from msa in the sense that these dialects are not standardized. arabic dialects substantially differ from msa and each other in terms of phonology, morphology, lexical choice and syntax @xcite . these variations affect how speakers of different dialects understand each other; where some words or maybe complete sentences can be incomprehensible. previous works on arabic dialects focused on dialect identification either in text or speech such as the works of @xcite @xcite @xcite @xcite . other works focused on higher level tasks exploiting dialectal data such as sentiment analysis @xcite , emotion @xcite , offensive language @xcite , and sarcasm @xcite . most of these datasets are created through manual data annotation. those annotations are collected by either recruiting designated annotators or through crowd-sourcing platforms. especially in the case of crowd-sourced annotations, the annotators are usually from different regions and speak different dialects. in this paper, we argue that dataset creators should take into consideration the effects of annotators' native dialect and dialect familiarity on the annotation process. due to the differences between arabic dialects, annotators might be assigning inaccurate labels to texts written in dialects they do not fully understand. in our work, we aim to analyse how a speaker of one dialect understands another. also, we study the effect of dialect familiarity on the data annotation process, taking arabic sarcasm as a case study of a highly subjective task. in our paper, we investigate the following research questions: ## methodology in this section, we describe our methodology for the analysis of dialects comprehension during data annotation tasks. we initially discuss the dataset we used and its ground-truth labels. then we explain collecting third-party labels from annotators of different dialects, which will be compared later to the ground-truth labels for the analysis process. ## discussion in this section, we provide a discussion of the results mentioned in section 4. we also revisit and answer our research questions as follows: rq1: how do speakers of different dialects understand other dialects? there are some similarities between dialects and, to some extent, people speaking different dialects can understand each other. the confusion between the dialects might be due to the fact that these dialects share many words or the differences are mostly phonological. also, due to the slight differences between dialects' orthography, annotators might confuse sentences in dialects they are unfamiliar with and assign them to a different one. this phenomenon is clear in section 4.3, where levantine speakers had better performance on msa for sarcasm detection, but they assigned an incorrect dialect label. rq2: how do speakers of different dialects perform on the sarcasm detection task? as discussed in sections 4.2 and 4.3, annotators tend to better understand sarcasm expressed in their dialect. this is due to the fact that annotators unfamiliar with a dialect would struggle to grasp the complete meaning of a sentence. also, the fact that sarcasm usually relies on cultural references that can be specific to rq3: is there a correlation between gender and the performance of an annotator on the sarcasm detection task? based on the results in section 4.4, female annotators seem to detect sarcasm better than male annotators. with the small number of annotators and the available data, we cannot provide an explanation for this observation. future works should consider studying this in a better-designed setup that considers other factors such as educational background and personality traits. we hope the findings of our study here will be of large benefits for researchers who work in the field of arabic nlp, especially when applying data annotations. we have shown that dataset creators need to be careful when appointing annotators for labelling arabic data. our findings can act as a guide to appoint annotators with the suitable dialectal background for annotating data in each dialect. ## conclusions in this paper, we analyse how arabic speakers understand and identify other dialects in written text. we also analyse human performance on sarcasm detection and compare it across different dialects. we use semeval's 2022 task 6 dataset, which has first-party sarcasm and dialect labels. our analysis shows that the performance of annotators varies based on the annotators' familiarity with the text's dialect. also, our analysis shows that annotators might not be familiar with the text's dialect and would confuse it with a different one. our results also show that females are more likely to understand sarcasm compared to males. based on the analysis, it is clear that dialect familiarity affects how annotators understand texts and their performance on a specific task. consequently, we recommend that arabic dataset creators should consider collecting annotations from native dialect speakers, which would guarantee higher-quality labels.