Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling (2023)

Abstract: Much work in natural language processing (NLP) relies on human annotation. The majority of this implicitly assumes that annotator’s labels are temporally stable, although the reality is that human judgements are rarely consistent over time. As a subjective annotation task, hate speech labels depend on annotator’s emotional and moral reactions to the language used to convey the message. Studies in Cognitive Science reveal a ‘foreign language effect’, whereby people take differing moral positions and perceive offensive phrases to be weaker in their second languages. Does this affect annotations as well? We conduct an experiment to investigate the impacts of (1) time and (2) different language conditions (English and German) on measurements of intra-annotator agreement in a hate speech labelling task. While we do not observe the expected lower stability in the different language condition, we find that overall agreement is significantly lower than is implicitly assumed in annotation tasks, which has important implications for dataset reproducibility in NLP.

## introduction while inter-annotator agreement is commonly used in natural language processing (nlp) research to measure annotation reliability (how well annotators agree with each other) @xcite , intra-annotator agreement (the extent to which individuals provide the same responses for the same prompts when asked repeatedly) is rarely reported @xcite . however, measurements of intra-annotator agreement are essential for nlp datasets as they indicate the consistency of each human annotator and thus the stability of the data they generate @xcite . intra-annotator measures can be used to control the quality of the annotation process (e.g. @xcite @xcite or to assess the difficulty and subjectivity of a particular task @xcite . the field's continuing failure to measure and report intra-annotator agreement, though, suggests that it is implicitly assumed (by omission) that annotators' responses are 100% stable-even when this is intuitively and empirically not the case. in fact, there is widespread evidence from psychology that the same people often make wildly inconsistent judgments depending on seemingly unrelated factors such as mood, time of day, the weather, or even how well their preferred sports team is performing @xcite . here, we consider the following two aspects: time: there is some evidence that annotator inconsistency increases as a function of time @xcite . however, in the majority of cases in which intraannotator agreement is reported, repeat annotations are collected in the same session that annotators label the items in the first instance @xcite . in those circumstances, annotators' responses are likely influenced by the recency effects of priming on memory @xcite . in this study, we therefore re-examine annotators after substantial temporal intervals of between two and eight weeks. language: the kind of language to annotate is likely to affect annotations. annotating or producing abusive language, such as hateful speech, can be understood as "morally motivated behavior[s] grounded in people's moral values and perceptions of moral violations" @xcite . however, there is evidence that people take different moral positions when presented with dilemmas in their first or second languages-the 'foreign language effect' @xcite . furthermore, @xcite shows that bilingual people perceive the emotional force of swearwords and taboo words to be weaker in their second languages, suggesting that they may judge toxic language differently when observed in different languages. we can therefore expect annotators to respond differently to text examples from hateful language datasets that feature moral issues and toxic slurs in their first (l1) or second language (l2). in this work, we investigate the stability of labels produced by bilingual annotators in response to near-identical (i.e., carefully translated) examples presented in both english and german. we ask the following research questions: r1: are annotators' responses stable over time when labelling hateful language? r2: is label stability lower when repeated annotation items are presented in a different language than in the same language? we use the xhate999 corpus (glavaš et al., 2020), the test set of which consists of (999) abusive and non-abusive texts that have been translated from english to five other target languages. translations were made by experts with an emphasis on maintaining the level and nuance of abuse, hatefulness, and aggression present in the texts. we use the english and german language versions of the 'gao' hatefulness subset @xcite (the data is originally sourced from three separate datasets). we chose german as it is the most widely-spoken of the target languages, which we expect to expediate annotator recruitment. we chose the 'gao' subset as we judged the domain or language and topics of the other two to be somewhat esoteric for primarily europe-based annotators: @xcite consists of disputes on the content of wikipedia pages among their authors; and @xcite is comprised of hindi-english political discussions. while many examples from the gao subset concern specific events, we expected the subject matter (e.g. the #blacklivesmatter movement, israel-palestine conflict) to be more well-known internationally. the test set comprises 99 items. foot_0 ## experimental setup we recruited 30 billingual german (l1) and english (l2) speaking annotators from the prolific crowdsourcing platform, foot_1 chosen for its capacity to facilitate longitudinal studies, ethical participant payment policies, and data quality @xcite . we presented the particpants with 96 examples from the test data: 48 in english and 48 in german. with these, we interspersed four items taken from hatecheck @xcite , which we used as attention check questions @xcite , as they were designed to be clearcut examples of hateful language. we use a 'descriptive dataset paradigm' @xcite , with annotators provided with minimal instructions to encourage the emergence of subjective perspectives. as such, annotators were presented with the original definition of hateful language from @xcite : we define hateful speech to be language which explicitly or implicitly threatens or demeans a person or a group based upon a facet of their identity such as gender, ethnicity, or sexual orientation. we also provided one example each of hateful and non-hateful items taken from the three unused test set items. in order to maintain concentration and to regularly provide the option of withdrawing participation, we split the task into 20 pages with five items to be annotated on each. the instructions, definition, and examples were repeated on each page, and are available in appendix a. we made the task available to all germanspeakers on the platform that are also fluent in english, as managed by prolific's in-built participant filtering functions. based on the findings of @xcite and @xcite , we then waited two weeks before opening a second round of the task to the same annotators, filtering by their prolific identification codes. here, the annotators were presented with a further four attention check items and the 96 items from the test set. again, half of these were in english and half in german. 50% were presented in the same language as in round one, and 50% in the alternative language. to control for order-effect bias, we split participants into two groups, and presented the items to each in a different order, also shuffling within-item response options (hateful/not hateful). following the principles of perspectivist data practices ( abercrombie et al. (2022 for reproducibility, we include the question item order and full instructions. we also provide a full data statement and annotator demographic information in appendix b. ## analysis of the 28 participants that completed both rounds of annotation, 22 labelled all attention check items in agreement with the original hatecheck labels, and we report results for these annotators only. foot_2 ## conclusion while attention has been paid to noise in linguistic annotations caused by factors such as subjectivity and ambiguity (e.g. @xcite @xcite , and the level and quality of annotator attention (e.g. @xcite , this study represents an initial foray into a hitherto understudied aspect of the human labeling work that most nlp research and systems are built upon: intra-annotator agreement and label stability. for this hateful language annotation task, we find that label stability is far lower than common practise implicitly implies (r1). in this study, presenting the items for reannotation in a different language does not lead to lower stability overall (r2), with l1 german speakers no less consistent-and often more so-than when re-annotating items in the same language. we suspect that our data sample of 96 items is too small to disentangle any l2 effects from other factors that may affect label stability. however, we see lower agreement overall (inter-and intra-) on the german language items, suggesting that the translation process adds some ambiguity. future work should investigate the linguistic and cultural factors that influence annotators' judgments more closely and-on a larger set of items. despite the limitations of our study, we have shown that annotator stability, along with reliability, is necessary for the repeatability and reproducibility of annotation studies @xcite . we therefore recommend that researchers and practitioners measure and report intra-(as well as inter-) annotator agreement scores for the labeled nlp datasets they create. the fact that this measure is still rarely reported adds to the emerging reproducibility issues in the field @xcite .