Establishing Annotation Quality in Multi-label Annotations (2022)

Abstract: In many linguistic fields requiring annotated data, multiple interpretations of a single item are possible. Multi-label annotations more accurately reflect this possibility. However, allowing for multi-label annotations also affects the chance that two coders agree with each other. Calculating inter-coder agreement for multi-label datasets is therefore not trivial. In the current contribution, we evaluate different metrics for calculating agreement on multi-label annotations: agreement on the intersection of annotated labels, an augmented version of Cohen’s Kappa, and precision, recall and F1. We propose a bootstrapping method to obtain chance agreement for each measure, which allows us to obtain an adjusted agreement coefficient that is more interpretable. We demonstrate how various measures affect estimates of agreement on simulated datasets and present a case study of discourse relation annotations. We also show how the proportion of double labels, and the entropy of the label distribution, influences the measures outlined above and how a bootstrapped adjusted agreement can make agreement measures more comparable across datasets in multi-label scenarios.

## introduction annotation efforts have long been characterized by the (implicit or explicit) assumption that there is a single true interpretation for every item, and all other interpretations are incorrect. this has become even more pervasive with the increasing usage of annotations as input for classifiers and other downstream computational tasks. however, the single-truth assumption has been challenged in recent years (e.g., @xcite , based on counterexamples from many different fields showing that two coders can disagree on an annotation and still both be right, due to the subjectivity and complexity of many tasks. we here focus on the field of discourse relation annotation as a case study, although the insights could be applied to different types of linguistic research @xcite . establishing inter-annotator agreement in scenario's where multiple labels are possible is challenging. in cases where multiple interpretations are possible but coders are restricted to annotate only a single label, it is unclear whether disagreement reflects incorrect interpretations (coder error or issues with the coding scheme or training) or item ambiguity. given an instance where two interpretations are equally likely, assuming that at least one coder inferred both possible readings but only a single label can be annotated, the chance that the coders agree would be only 50%. this does not reflect that there is actually high agreement on the labels. allowing for multi-label coding can result in more accurate annotations as well as higher agreement, and with that most likely higher reliability of a final label. although two sets of multi-labels can be compared by calculating the difference between their distributions (e.g. comparing the probability distribution of a classifier to the distribution of crowdsourced labels, @xcite , the distributions and ranks of the labels are not necessarily relevant or accessible (e.g. expert annotations where multiple labels are allowed). we here assume a scenario where unranked multiple labels are allowed, regardless of whether more labels are possible and aim to properly estimate agreement. we will therefore focus on estimating the reliability of a single final label per item, while also discussing other scenario's. traditional agreement statistics, such as cohen's kappa @xcite will not be suitable for evaluating the reliability of data that allow for multi-label annotationsat least not without adjustments. it should be taken into account that multi-label coding also inflates the chance agreement: by providing more labels, there is a higher chance that at least one of those labels overlaps with the annotations from another coder. the current paper will analyze several ways to determine the reliability of multi-label annotations. we argue that chance agreement should always be taken into account for any agreement measure to make it comparable across datasets. the contributions of this paper are: (i) we propose a bootstrapping method to estimate the expected agreement (see section 4), and (ii) we compare agreement on various measures for simulated datasets with different parameters, as well as for a real-life dataset (section 5 and 6). we make available a reproducible script of the implementations of the measures and the calculations. 1 we use the following terminology throughout the paper. an item (i) is an instance which is annotated. it can be annotated with one or more labels, which together make up one annotation. categories (k) refer to the options that coders (c) have to assign an item to. ## multiple labels: ambiguity and uncertainty there are several scenarios in which multiple labels are possibly desirable. a first one is a scenario in which annotators are uncertain which of multiple categories are correct, caused by a lack of knowledge or information @xcite . allowing them to provide multiple labels would then reflect a probabilistic representation of the target label. these probabilistic labels also occur often in crowd-sourced annotations, where workers might lack the knowledge to select the correct label. secondly, multiple labels might reflect true ambiguity. these are cases where more than one interpretation is possible. to illustrate, consider the discourse relation in example (2): both a speci-fication relation (which could be expressed by inserting the cue phrase more specifically between the two sentences), and a manner relation (which could be expressed by to do so,) interpretation are valid. ambiguity has been argued to be an "inherent property of natural language" and an important source of disagreement in language annotation @xcite 1 https://osf.io/f5v4p/?@xmath0. @xcite show that the vast majority of disagreements between annotators arise because multiple labels are valid and recommend allowing for such disagreements, rather than focusing on inter-annotator agreement. the present contribution shows that it is possible to establish inter-annotator agreement while taking ambiguity into account by allowing for multiple labels. more specifically, the focus of the present paper is on establishing a reliability measure for ambiguous cases where multiple labels are valid, but we will also address how reliability measures can reflect these different multi-label scenarios. ## reliability measures reliability is the extent to which different coders arrive at the same interpretations of items. reliability can be measured by calculating the inter-coder agreement using an agreement coefficient: a numerical index of the extent of agreement between the coders. however, the goal of obtaining reliable data is not merely to have data on which two coders agree (coders might be wrong or biased, after all), but to have annotated labels that reflect the true meaning of the items. it is important to note that this validity, despite being the goal of annotation efforts, is not captured by agreement coefficients. agreement coefficients usually consist of two components: observed agreement (a o ) and expected agreement (a e ). together, these can be used to calculate an adjusted agreement, i.e. an agreement coefficient (a c ): observed agreement is taken to be 1 when two coders assign an item to the same category and 0 when the item is assigned to different categories. observed agreement does not take into account chance agreement, which occurs when one or both coders rate an item randomly. in order to get a reliable index of the extent of agreement between coders, observed agreement therefore has to be adjusted for the proportion of agreement that is expected to occur by chance. the crucial difference between various inter-coder agreement measures often lies in the way in which they estimate this expected agreement (see @xcite for a detailed overview). one of the most frequently used inter-coder agreement measures is @xcite . when each data point in a corpus is assigned a single label, calculating chance agreement, and κ, is straightforward. more specifically, cohen's κ calculates the probability of each label being selected by each coder independently: however, traditional kappa is not applicable to multi-label scenarios. one of the simplest solutions would be to treat each multiple-label annotation as a distinct label, but this inflates the number of categories in the coding scheme, which negatively affects κ. the traditional κ measure would therefore need to be adapted to make it suitable for multi-label annotations. ## bootstrapping expected agreement the existing measures are problematic for calculating multi-label agreement, because they do not correct for chance agreement for multiple labels (soft-match and f1) or they penalize multiple labels even if both annotators agree on the double labels (augmented kappa). the main contribution of the current paper is therefore to suggest a bootstrapping method for obtaining chance agreement, that can be used to adjust existing measures. we propose to sample from the provided distribution per coder, in order to estimate the true expected agreement needed to calculate the adjusted agreement. more specifically, for each item we draw from the distribution of labels provided by each coder, following cohen's κ. this distribution is obtained by dividing the number of times each category has been assigned by the total number of labels provided. the amount of sampled labels is likewise sampled from the probability distribution of the amount of labels for each item occurring in the original data set. in other words, if a coder provided a single label in 40% of the items and two labels in 60% of the items, double labels are also sampled in 60% of the items in the simulated dataset. by simulating the sample data, we can bootstrap the expected agreement of several measures. for example, we can calculate the average proportion of intersecting labels across n simulated datasets. using the expected agreement obtained by the simulations, existing agreement measures can be adjusted by taking into account chance agreements, using equation 1. this allows us to obtain a variety of new measures. for calculating agreement on the intersecting labels, similar to soft-match, we refer to this measure as boot-match. its expected agreement is estimated using the bootstrapping method, contrasting it with the soft-match measure where the expected agreement is calculated after removing additional labels that are not an intersecting label (see section 3.1). similarly, bootstrapping the expected agreement allows us to correct precision, recall and f1 for chance agreement. the traditional version of these measures will be referred to as the observed components and the bootstrapping method allows us to also obtain the expected and chance-adjusted agreement components for these measures: we will therefore refer to these measures as boot-f1, boot-precision and boot-recall. ## comparing reliability measures on multi-label annotations the previous sections show that there are various ways to estimate agreement in multi-label annotation tasks. the goal of an agreement measure is to be able to compare how much coders agree across different datasets. to evaluate which measure would be preferred for estimating agreement in multi-label datasets, a good measure should therefore (a) estimate expected agreement proportional to the amount of multi-labels and (b) provide a higher agreement score for tasks with more agreement. we will explore how the measures discussed above behave in different scenarios below. to illustrate how the proportion of double labels as well as intersection agreement influence the scores, we simulate datasets with different char-acteristics for two parameters: the percentage of double labels provided and the percentage of observed intersection agreement. we thus manipulate how often a double label is chosen, ranging from never to always, with 25% intervals. in addition, the datasets vary in how much coders agree on the labels, which is manipulated by simulating data with various degrees of intersection agreement: low equals 60% intersection agreement, medium 75% and high 90%. for each of these parameter combinations, we sample 100 datasets. for each of these, we bootstrapped the expected agreement separately, using 100 simulations. each dataset contained 100 items, which were assigned to one (or two) of five categories with equal probability. we then calculate the average observed, expected and adjusted (i.e. κ) agreement for each measure across these datasets. 3 the agreement statistics for all the different datasets in the simulation analysis are provided in figure 1 . for the scenarios where no double labels are provided, the statistics are the same across the different measures: the adjusted agreement is 0.50, 0.69 and 0.87 for the different levels of intersection agreement. however, when more double-annotated labels are added, the patterns for the observed, expected and adjusted agreement diverge for the different measures. the first row in figure 1 considers oracle agreement, using the soft-match measure. needless to say, the observed agreement (in red) increases when agreement is higher. moreover, observed agreement for the soft-match measure remains constant when the percentage of double labels increases. however, obtaining intersection agreement is easier when more labels are provided. soft-match does not take this into account, because the expected agreement does not change when more labels are provided. across all simulated datasets, expected agreement is 0.21. as a result, the adjusted agreement for this measure is also constant across datasets with varying amounts of double labels. the adjusted agreement for soft-match therefore remains very close to the observed agreement, which is too liberal. augmented kappa corrects for multi-label scenarios. the expected agreement remains constant with this measure, regardless of the number of double labels. the observed agreement, on the other hand, decreases significantly. note that this is partly due to the fact that the second label was sampled randomly. however, even if there would be perfect agreement on a double label for this measure (i.e. both coders assign an item to category a and b), the observed agreement is only 0.5 (see the observed agreement for the boot-match is the same as for soft-match, as it also considers intersection agreement. like soft-match, it thus also remains constant when the dataset contains more double labels. unlike the soft-match measure, however, the boot-match measure takes the percentage of double labels into account when calculating the adjusted agreement, as it estimates the expected agreement by simulating the provided distributions in the dataset 100 times. expected agreement therefore increases when more labels are added, resulting in a lower adjusted agreement. with a fully double-label annotated dataset, expected agreement is 70%. in the low agreement scenario, the adjusted agreement is therefore even negative when all labels receive a double annotation. using this adjusted agreement, we can compare how agreement on double-label datasets relates to datasets without double labels. for example, with five categories, achieving 90% intersection agreement with all double labels is comparable to achieving 75% agreement with 0% double labels (adjusted agreement ≈ 0.67). note that this relationship depends on several parameters, such as the entropy of the label distribution and the number of categories. we will return to this issue below. for the boot-f1 measure, the observed agreement decreases when the percentage of double labels increases foot_2 , similarly to augmented kappa. this is partly due to the fact that the manipulation of agreement was only for the intersection agreement and any additional labels besides the intersecting labels were sampled randomly. as a result, perfect agreement is not always achieved in the simulations, unlike for the soft-match and boot-match agreement. in this scenario, however, a perfect f1 would be obtained if both coders assign an item to the same two categories. boot-f1 is slightly higher than augmented, across the different measures in the various datasets, because it yields higher agreement when both coders provide the same two labels (as in item 3 in more importantly, note that the expected boot-f1 increases when more double labels are added to the dataset. reporting only observed f1 is therefore misleading, because achieving the same observed f1 on a dataset with few double labels compared to one with many double labels is more difficult. the results are therefore not comparable across datasets. calculating the chance-adjusted boot-f1 solves this problem, because it takes this chance agreement into account. ## case study real-world datasets often have very different characteristics than the simulated datasets in the previous section. we therefore explore how the measures behave in a real-world dataset: a case study for discourse relation annotation, which is a notoriously difficult task. it is often difficult to achieve a κ > .7 on single-label annotations @xcite . this is partly due to the fact that discourse relation frameworks often distinguish many different categories (as can be seen below). in addition, coherence is not a feature of the text, but of the mental representations that readers have of the text @xcite . therefore, discourse relation annotations depend on coders' interpretation of the text, which may be subjective. furthermore, ambiguity plays an important role for discourse relation annotation, which can partly explain low agreement with single label annotations. recent studies have therefore turned to crowd-sourcing to source discourse relation annotations (e.g. @xcite @xcite . this allows researchers to capture a larger variety of interpretations per instance. however, in some cases, such an approach is not suitable, or researchers still target a single final label. the case study was part of a psycho-linguistic experiment in which participants were asked to provide a one-sentence continuation to a prompt. two expert coders annotated the continuations with respect to their discourse relation with the prompt. the coding scheme allowed for maximally two labels, when coders believed both senses held. in total, the coders annotated 884 items, using 19 categories. 11 of these categories occurred in the final intersection label. the first coder had provided double labels in 11.4% of the cases, the second coder in 65.5% of the items. as a result, the first coder has a greater influence on the intersection item, because only one of the items by the second coder will be selected when one of them overlaps with the label provided by the first coder. as can be seen in finally, relating to the issue of ambiguity in annotation and selecting a single final label, we note that this is a debatable issue in itself. in correspondence to the assumption that items can express more than one meaning, a soft label -consisting of a probability distribution of all categories per item -more accurately captures an item's ambiguity. for example, crowdtruth @xcite evaluates data quality by capturing the ambiguity inherent in semantic annotation through the use of disagreement-aware metrics. fornaciari et al. (2021) and uma et al. (2021) showed that models trained on soft labels, such as these, outperform those trained on singlelabel data, especially if they are evaluated using soft labels as well. a larger number of coders would be needed to more accurately calculate a probability distribution for an item. this is not the case for tasks that require (or choose to use) expert annotations only, as is the case in psycho-linguistics, or when annotating a gold dataset. ## discussion and conclusion annotating data with multiple labels better reflects the true meaning of the items, as these items can be ambiguous or even have multiple interpretations. after all, the goal of agreement measures is not to establish how strongly the coders agree, but rather how reliable the label is. the label on which annotators agree is more likely to be a true label, regardless of whether all the labels are captured. obtaining a distribution of labels may be helpful in some, but not all, tasks (cf. @xcite . in addition, for many tasks such distributions are not available, either because not enough observations were obtained, or because computational models predict a limited number of labels. the present study therefore explored measures for evaluating various agreement measures on scenario's with more than one label. the augmented kappa has been proposed as a measure of agreement on multi-label annotated datasets @xcite , but it penalizes additional labels heavily and does not consider items assigned to the same multiple categories as full agreement. the underlying assumption is thus that there is only one true label, reflecting uncertainty rather than ambiguity. however, it is not always true that only one true label exists @xcite . precision, recall and f1 should also be corrected for chance agreement, as that varies with respect to how many labels are provided to each item. observed precision, recall and f1 are therefore not comparable across datasets that differ in e.g. the amount of multiple labels. when additional labels are potentially correct, intersection agreement is a good option, but only when it is corrected for chance agreement. rather than taking the intersecting label to calculate adjusted agreement as in a soft-match measure, the data could be simulated to estimate the true chance agreement on the intersection (i.e. boot-match). this measure increases expected agreement when more labels are provided, resulting in a lower adjusted agreement. coders should therefore only provide double labels if they are certain that both labels hold. one limitation that needs to be taken into account when selecting the intersection label, is that the coder who provides fewer labels influences the final label more than the coder who provides more labels. ideally, coders would therefore provide a similar proportion of double annotations. furthermore, in an extreme case where one coder assigns an item to all categories, there would always be agreement. this is corrected for slightly in boot-match, but is more easily detected using boot-f1 and similar measures. finally, when some labels are more frequent than others, achieving intersection agreement on this label is more likely. as a result, the intersection agreement will contain a higher proportion of dominant labels. the distribution of the intersecting labels thus does not necessarily reflect the true distribution of labels in the dataset and researchers should be careful to draw conclusions about the distributions of aggregated labels. to conclude, if only one true label is believed to be possible for each item, augmented kappa can be used to calculate agreement in cases where annotators provide more than one label. however, if items are believed to be ambiguous, with possibly more than one true label per item, boot-match best estimates the reliability of a single final label per item. boot-f1 and related measures reveal more about the structure of the data, such as asymmetries between the coders. which measure is reported therefore depends on the goal of the annotation effort. regardless, for all of these measures, chance agreement should be taken into account to make the measure comparable across datasets with different characteristics. as demonstrated above, our proposed method of bootstrapping the expected agreement can be used for this.