Towards Responsible Natural Language Annotation for the Varieties of A rabic (2022)

Abstract: When building NLP models, there is a tendency to aim for broader coverage, often overlooking cultural and (socio)linguistic nuance. In this position paper, we make the case for care and attention to such nuances, particularly in dataset annotation, as well as the inclusion of cultural and linguistic expertise in the process. We present a playbook for responsible dataset creation for polyglossic, multidialectal languages. This work is informed by a study on Arabic annotation of social media content.

## introduction natural language processing (nlp) is the foundation of numerous automated decision-making systems in a growing number of scenarios and languages, including content moderation on platforms with global reach and consequence (gillespie, 2020). thus it is highly pertinent to address how practitioners can build responsible nlp systems, models, and workflows for languages beyond english @xcite @xcite . in doing so, it is essential that these systems are designed with the inclusion of domain experts and stakeholder groups with native fluency and local/regional knowledge @xcite . this will not only ensure the presence of the deep problem understanding necessary to create accurate systems and anticipate potential harms, but foster earned trust and legitimacy in the system @xcite . many performant machine learning/nlp algorithms to date are supervised, relying on large scale annotated training data, thus the veracity and curation of the data labels can have significant impact on model performance @xcite . and further, even unsupervised and semi-supervised systems require labeled data for evaluation as a bare minimum to allow visibility into any blindspots in the system performance. in this position paper, we focus on nlp datasets, highlighting the potential for compounding harms to at-risk populations and calling for greater care and attention to annotation and annotator support @xcite . this is demonstrated via the varieties of arabic. foot_0 ## arabic varieties and nlp arabic is a semitic language, spoken by over 420m people globally with the highest concentration in the middle east and north africa (mena) where it is the dominant or official language of over twenty nations. arabic is better described as a family of languages as the so-called "dialects" are highly variable and often have low inter-dialect comprehensibility @xcite . e.g. moroccan and egyptian varieties are about as mutually intelligible as spanish and romanian. importantly, arabic should not be considered as an analog to english, i.e. one language with closely related dialects (e.g. british, american, and australian english). this comparison buries the polyglossic (when languages co-exist in a community) and heterogeneous characteristics of arabic that are crucial to take into account when building effective arabic nlp systems. we describe several relevant features of arabic in this section. first, while these varieties share the same root in classical arabic, varying historical and cultural experiences across the arab world have led to the divergence in spoken practice, leading to frequent cases of "faux amis," a.k.a. false cognates. this phenomenon refers to words or phrases that appear or sound the same between the varieties but have different meanings. for example, the word zahry foot_1 means my luck in tunisian arabic, however the same word refers to the color pink in levantine. faux amis have implications for detecting objectionable speech, e.g. a common slur in moroccan arabic is zamel, however in yemeni it is a type of singing that has become popular in recent years with the war in yemen. notably, this phenomenon is not only on the lexical level but also on the phrasal level, e.g. yetyk alefyap in levantine dialects means "may you have good health," however in moroccan it translates to "go to hell." second, arabic exemplifies one of the best known cases of diglossia, foot_2 where the formal modern standard arabic (msa) or fushy, and the regional spoken vernaculars co-exist in virtually every speech community. today, msa is used in practice primarily for international news and legal contracts. it is generally the language of education, and thus only fully accessible to those with a certain level of literacy and training. in fact, msa is a formal language akin to shakespearean english and is not spoken colloquially. despite that, msa is considered a relatively high-resource language @xcite due to its status as the shared language in the arab world, as such it is often the variety used in arabic language technologies. moreover, it is the prevalent arabic variety of language-learning courses across the world. notably, l2 language learners of msa often have trouble communicating with native arabic speakers due to considerable divergences between msa and spoken varieties. due to these factors, when nlp systems are built for msa a large portion of arabic speakers cannot benefit fully from the technology, creating an access disparity between those with and without a more advanced education. 4 the third feature to highlight is the fact that the colloquial varieties of arabic greatly differ from one another along a geographic and social continuum, however general consensus splits arabic into six broad groupings. following habash (2010) these are: iraqi (iraq), levantine (syria, jordan, lebanon, palestine), maghrebi (morocco, tunisia, algeria, libya, foot_4 western sahara, mauritania), peninsular/gulf (oman, qatar, saudi arabia, bahrain, united arab emirates, kuwait foot_5 ), yemeni (yemen), egyptian (egypt & sudan foot_6 ). these broad dialect groupings of course do not fully account for the heterogeneous political and cultural aspects of mena or the arabic speaking diaspora. this can be a pitfall in annotation given that contextual knowledge is often necessary for full semantic and pragmatic understanding. as an example, the maghrebi dialect may be more appropriately broken down further, as algerian is often unintelligible to moroccan and tunisian variety speakers due to the infusion of novel, contextuallybased vocabulary that correspond to the country's particular political context. this is a common phenomenon across mena, and of course for other languages. the unique country-level situations for arabic dialects -not to mention contexts and groupings that are sub-national or transcend country borders -are important to take into account as nlp practitioners strive to produce systems that capture how humans naturally use language. a final element to mention is the fact that there are no standard orthographies for arabic dialectal text and the written variants exhibit pervasive code switching. this property is of course particularly important in the context of text annotation. such a language profile pushes the boundaries of nlp, and perceptions away from arabic as a monolith. foot_7 ## motivational case study: arabic on social media to highlight the necessity for a more nuanced treatment of polyglossic, multidialectal languages, we study annotation of arabic content on the facebook platform. the text-based samples in the study are in native arabic script (i.e. non-romanized), created by users in ten arabic nations: algeria, morocco, libya, syria, iraq, lebanon, egypt, saudi arabia, sudan, and yemen. we select these nations to cover all broad dialect groups in the region ( §2), with some inter-dialect comparisons, e.g. syria and lebanon. country-level groupings are employed in the study, rather than the broad dialects, to capture the contextual nature of content in each nation. approximately 3000 pieces of content are sampled per country from a source dataset created for the purpose of hate speech classification evaluation. for content from each country, we compare data labeled by annotators residing in morocco (aim) to reviews by annotators with native fluency and expertise for each country in the study, a.k.a. country experts (ce). for example, for syrian content, annotation by aim is compared to that by ce who are verified syrian annotators. it is worth noting that the ce annotation consistency as measured by intra ce agreement -inter-annotator agreement or iaa -is quite high ranging from 64% for egyptian to 92% for moroccan arabic. 9 the overlap sample sizes for the ce iaa measurement ranged in size from 91 unique items corresponding to 26% of the total saudi arabic dataset to 975 unique items corresponding to 41% of the total iraqi dataset. to achieve this, the content is first sampled and sorted primarily by ip into per-country queues. ce then choose the dominant language variety and country of the sample. from there, the data is filtered to only arabic content from the country. after filtering, the ce label the content (previously labeled by aim) with one of two classes: "delete" (i.e. the content is deemed hateful) or "ignore" (i.e. the content is deemed benign) per a predefined set of content guidelines. the main findings of this study are as follows: (1) for every country dataset, the majority of the content is found to be in the arabic dialect of that nation, rather than msa. this tracks with the understanding that users communicate informally on 9 variations in ce agreement is outside the scope of this investigation. for this, we defer to future research. social media @xcite , (2) the saudi and algerian datasets show a significantly higher presence of msa content (31% and 27%, respectively) than other country datasets. additionally, 74% of the arabic samples in the saudi dataset are identified by ce as non-saudi content, a property of that could be explained by inward migration. 10 thus after filtering the saudi dataset is significantly smaller than that of any other country dataset. (3) signal detection theory (sdt) models decision-making as a mixture of gaussians (one for benign content and another for violating content) and determines the implied threshold of the reviewers in aggregate @xcite . employing sdt, we find the aim reviewers are labeling moroccan content more leniently with statistical significance (figure 1 ) as compared to all other country datasets, with the notable exception of the saudi dataset. of further note, considering the ce labeling for each dataset as ground truth, the morocco dataset had the most accurate labeling by the aim cohort at 87%. this is an intuitive result that was verified in this study. this finding, coupled with the sdt results shown in figure 1 , point to two potential interpretations: when reviewers understand the linguistic and contextual content of the samples, they 1. are more likely to understand the nuances of the content that make it benign (e.g. sarcastic or idiomatic speech); 2. may feel comfortable giving a certain benefitof-the-doubt (a.k.a. leniency) to the content creator, whereas they may not for groups in which they are not a member. this observed leniency, alone, could imply that reviewers should not review content within their own variety, however when coupled with the sociolinguistic knowledge presented in section 2 on the distance between the varieties and the finding that, for the morocco dataset, the the aim cohort and ce had the highest agreement, these findings instead point to the fact that reviewers could be misunderstanding other-variety content or are relatively more strict on out-group member content. this study highlights the impact of arabic variety differences on annotation. accordingly we gather observations and formalize the process rendering it applicable across similar scenarios, namely, dataset creation for polyglossic, multidialectal languages such as indo/malay, varieties of german, etc. ## a playbook for dataset creation for accurate annotation of natural language content, it is important for nlp practitioners to consider the entire flow of content to the training or evaluation datasets. 11 here, we propose general recommended practices for annotation, with further information for arabic nlp dataset creation, informed by the case study in section 3. the guidelines aim to: (1) ensure there is sufficient expertise to -at minimum -understand the variety and context of the annotation samples; (2) route samples to experts who are best equipped to field and process them; and, (3) provide expert support and inclusion in the process of nlp design and implementation. these recommendations could seem obvious to some, however they are worth crisply laying out to set expectations of expert inclusion and support in nlp development. they are as follows: for the case study, we employed a stratified sampling technique to endeavor to capture the linguistic landscape of arabic variety-speakers, and relevant context coverage for arabic content from the countries and communities in focus. evaluations should be designed carefully as many speakers of one variety have passive knowledge (vs. deep/native understanding) of other varieties. arabic speakers may consider themselves fluent enough in the passive variety however it is important that their proficiency level is carefully assessed. 14 furthermore, evaluations per-variety in many of these polyglossic multidialectal language families should include contextual elements relevant to the country or region. at present, many out-of-thebox arabic proficiency tests are specific to msa, which is not recommended unless employed as a supplement. for the case study ( §3), a major difference between the morocco-based annotators and country experts is their verified relevant expertise. the country experts are proficient in the arabic variety and regional contexts relevant to the labeling tasks, with sufficient cultural understanding to accurately interpret content from the community. broad groupings ( §2) could be employed, depending on the granularity of contextual content and goals of the model. setting aside the vast in-homogeneity of cultures and complex contexts across the arabicspeaking world, there can be a misconception that if an individual can speak one arabic dialect/variety they can accurately label content for any or adjacent varieties. this should not generally be assumed @xcite . for arabic varieties, queues and routing at least at the granularity level of each broad grouping ( §2) is generally essential, due to the low inter-variety comprehension. if country context is deemed relevant for the type of samples and application, e.g. political contexts/social value content, further subdivisions (e.g. country, age, gender identity, etc.) are likely to be important for accuracy and consistency in labeling. for systems with automated routing, language and geographic identification could be employed to detect and separate dialects and contexts. a possible starting point is the groupings as described in §2, enhanced with guiding per-variety word lists. these elements bring up the difficult question of granularity: not just how, but how deeply should practitioners divide datasets and annotation, in order to ensure sufficient coverage and understanding of the content? this is a deceptively challenging question as languages most commonly exist on a continuum and practitioners can often divide the natural language groups further with no clear ideal stopping point. this question of granularity presents the need for a careful deliberation of trade-offs, often around resource expenditure vs. annotator expertise and data coverage. data availability or the cost of obtaining high-quality labeling will frequently become a limiting factor. the overarching recommendation in this work is to prioritize high quality annotation over breadth of coverage, indicating that in the cases where researchers do not have the resources for high quality annotation and annotator support they instead reduce coverage by tightening the goals of their nlp technology/application. (6) sample re-routing capabilities are relevant to capture routing errors, especially as errors in language identification can be surprising. it's important to build flexibility into the annotation system to allow annotators to route/skip samples outside their expertise, as well as a mechanism to surface such routing failures to improve the system. for the case study, we provided a mechanism to filter or surface routing errors. (7) data evaluation for faithful and accuratelylabeled evaluation and training (if applicable), such that a resulting model can report real patterns @xcite . this includes label evaluation and auditing systems to measure accuracy, sensitivity, and any potential bias (e.g. sampling biases, statistical biases, or stereotypes) encoded in the datasets @xcite . a multi-evaluation system for the stability of the labels, or uncovering ambiguity, could be significant @xcite . for multidialectal, polyglossic languages such as arabic, it is important that data evaluations account for the language variety and contextual differences, particularly with tricky faux amis. for the case study, we employed a multistage process with adjudication using multiple expert reviewers of content to check annotation quality. (8) reviewer well-being support systems, including but not limited to rest periods and psychological support, are important for any type of annotation, and especially-so when the dataset is composed of disturbing or traumatic content. these means of support are even more essential for the well-being of arabic annotators, given the high rate of ptsd in the arab world @xcite . care is needed when constructing arabic natural language datasets, due to the prevalence of faux amis, modern standard arabic and pervasive linguistic code switching, as well as the complex cultural and political contexts of arabic-dominant countries. this is not to mention the potentially dire consequences of errors for economically disadvantaged and vulnerable groups in mena @xcite @xcite . the guidelines in this section serve the additional purpose of surfacing failure modes in labeling that could scale when training classifiers, or obfuscate issues with model outcomes for evaluation datasets. ## conclusion polyglossic and multidialectal languages present both challenges and opportunities to the nlp community, chief among them are the trade-offs inherent in dataset creation. from developing best practices with arabic, we can apply the guidelines to other polyglossic, dialectal languages -such as chinese, indonesian/malay, and german -with the understanding that with each language comes new challenges.