Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign (2023)

Abstract: In this paper we report on the complexity of persuasion technique annotation in the context of a large multilingual annotation campaign involving 6 languages and approximately 40 annotators. We highlight the techniques that appear to be difficult for humans to annotate and elaborate on our findings on the causes of this phenomenon. We introduce Holistic IAA, a new word embedding-based annotator agreement metric and we report on various experiments using this metric and its correlation with the traditional Inter Annotator Agreement (IAA) metrics. However, given somewhat limited and loose interaction between annotators, i.e., only a few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for IAA between annotators tasked to annotate different documents and in different languages, for which classical IAA metrics can not be applied.

## introduction in the recent years we have observed an emergence of automated tools for facilitating online media analysis for better understanding of the presented narratives around certain topics across countries, and to identify manipulative, deceptive and propagandistic content. developing such tools requires annotated data of high quality. we report on the complexity of annotating such manipulative devices, i.e., persuasion techniques, in the context of a large annotation campaign involving 6 languages and approximately 40 annotators, whose details are described in @xcite . the persuasion technique taxonomy used in the campaign is an extension of the taxonomies used in different shared tasks, contains 23 techniques, and includes i.a., the techniques appealing to emotions, justifications and some forms of logical fallacies. the resulting dataset has been used in the semeval 2023 task 3: detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup @xcite . the primary objective of the work reported in this paper is threefold, namely: we first highlight the techniques that appear to be difficult for humans to annotate using the classical cohen's κ @xcite , and krippendorff's α @xcite . classical iaa measures impose certain limitations. first, they only capture the coherence of the annotations in texts written in the same language. secondly, considering annotations done for a single language, there were many annotators, but annotating totally different subsets of documents. the classical iaa metrics are computed using a tiny fraction of the whole dataset: the one where the annotators annotated the same articles, despite the fact that the exact same text could be annotated in different articles by different annotators. finally, the classical iaa measures only capture agreement at the time of the annotation, but do not tell us anything about the coherence and quality of the final curated dataset. in order to overcome the aforementioned limitations, we introduce holistic iaa, a new multilingual word embedding-based iaa metric and we report on various experiments using it and its correlation with the traditional iaa metrics. however, given somewhat limited and loose interaction between annotators, i.e., only a few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for iaa between annotators tasked to annotate different documents and in different languages. we present our preliminary results on this research problem with an ultimate goal of establishing a mechanism that allows to compare all annotators no matter which document they annotated, and to detect diverging annotations across languages. our contributions can be summarized as follows: (i) we measure how confusing were the persuasion technique labels for different groups of annotators; (ii) we assess the coherence of the dataset using standard iaa measures; (iii) we introduce a new mutlilingual pancorpus iaa measure based on semantic similarity; (iv) we exploit this new measure on the raw and curated annotations of the annotators, and compare the resulting ranking of annotators to the one obtained by standard iaa measurements; (v) we comment on the self-coherence of the annotators using the new measure, as well as of the dataset language-wise. this paper focuses primarily on the annotation agreement and complexity, whereas the description of the resulting dataset is kept to the minimum necessary for understanding the content. for further details please refer to @xcite . the paper is organized as follows. section 2 reports on the related work. section 3 introduces the persuasion technique taxonomy and describes the annotation process. next, section 4 reports on the annotation coherence computed using traditional iaa metrics and highlights the hard-to-annotate techniques. subsequently, section 5 introduces a new word embedding-based annotator agreement metric and reports on various experiments using it and correlating it with the traditional iaa metrics. we end with some concluding remarks in section 6. ## related work persuasion detection in text is related to work on propaganda detection. the work in the latter area initially focused on document-level analysis and predictions, e.g., @xcite reports on prediction of 4 classes (trusted, satire, hoax, and propaganda) of documents, whereas @xcite presented a corpus of tagged either as propaganda or non-propaganda). in parallel, other efforts focused on the detection of specific persuasion techniques. @xcite presented a corpus annotated with 5 fallacies that directly relate to propaganda techniques. a more fine-grained analysis was done by @xcite , who developed a corpus of english news articles labelled with 18 propaganda techniques at span/sentence level. somewhat related work on detection of use of propaganda techniques in memes is presented in @xcite , the relationship between propaganda and coordination @xcite , and work studying covid-19 related propaganda in social media @xcite . @xcite reported on the creation of annotated text snippet dataset with logical fallacies for covid-19 domain. @xcite presents three-stage evaluation framework of detection, coarse-grained, and fine-grained classification of logical fallacies through adapting existing evaluation datasets, and evaluate various state-of-the-art models using this framework. @xcite proposed the task of logical fallacy detection and a new dataset of logical fallacies found in climate change claims. all the persuasion techniques and logical fallacy taxonomies introduced in the aforementioned works do overlap to a very high degree, but are structured differently, and use different naming conventions. various related shared tasks on the detection of persuasion techniques were organized recently, and various taxonomies were introduced @xcite @xcite @xcite . related work on iaa which explores going beyond the limitation of standard measures was reported in @xcite , proposing an idea similar to our in that they are able to compare all annotators between themselves, however, this comparison is done statistically on label distribution while we look at actual content of the annotate textd. moreover, they are interested in assessing the gold label uncertainty, which is a similar concern to our effort of capturing the label definition difficulty. however, they treat it in a statistical fashion, while we provide simple descriptors. it would be an interesting future work to explore the combination of both approaches. ## conclusions we reported on the complexity of annotating persuasion techniques in a large-scale multilingual annotation campaign. we introduced the holistic iaa paradigm, a new measure to serve as a proxy of the estimation of inter-annotator agreement and actual corpus coherence in settings that are fundamentally outside the scope of usual iaa measures. we demonstrate that annotator ranking computed using this new measure is positive and can highly correlates with ranking computed using cohen's kappa in some settings. using it, we can observe the beneficial impact of the second step of our 2step curation phase, and also identify similarity and divergence between annotators for some subsets of labels. the experiment conducted in this study supports what was informally remarked regarding the estimation of the performance of the annotators and increased our confidence in the coherence of the final corpus. we believe that using holistic iaa as part of the monitoring of multilingual or monolingual large-scale annotation campaigns could help to spot problems by flagging potential incoherence in the labels of semantically similar sentences at an early stage. in future work we envisage exploration of thresholds for finer interpretation and exploring the use of other semantic similarity models.