Estimating Agreement by Chance for Sequence Annotation (2024)

Abstract: In the field of natural language processing, correction of performance assessment for chance agreement plays a crucial role in evaluating the reliability of annotations. However, there is a notable dearth of research focusing on chance correction for assessing the reliability of sequence annotation tasks, despite their widespread prevalence in the field. To address this gap, this paper introduces a novel model for generating random annotations, which serves as the foundation for estimating chance agreement in sequence annotation tasks. Utilizing the proposed randomization model and a related comparison approach, we successfully derive the analytical form of the distribution, enabling the computation of the probable location of each annotated text segment and subsequent chance agreement estimation. Through a combination simulation and corpus-based evaluation, we successfully assess its applicability and validate its accuracy and efficacy.

## introduction reliable annotation is a cornerstone of nlp research, enabling both supervised learning methods and evaluation. though not frequently employed for evaluation of model performance in the field of nlp, one of the most widely accepted metrics for evaluation of annotation reliability is cohen's kappa, which offers an assessment of interrater reliability that is adjusted in order to avoid offering credit for the portion of observed agreement that can be attributed to chance. some nlp tasks, such as named entity recognition or other span detection/labeling tasks, lack an appropriate chance corrected metric. this paper addresses this gap by proposing such a measure for these tasks, demonstrating its application in both simulation and conll03 corpus experiments. numerous studies caution against using nonchance-corrected agreement metrics. they can lead to unfair task or system comparisons due to biases introduced due to varying levels of chance agreement across tasks and systems @xcite @xcite @xcite . furthermore, without correction for chance agreement, measurements tend to cluster within a narrow range, making it difficult to discern differences between approaches @xcite . therefore, both estimating and correcting for chance agreement have become critical in annotation evaluation, except in cases where chance agreement is negligible. the main contributions of our work are summarized as follows: in the remainder of the paper, we provide a theoretical foundation for our work through a review of past literature. we then explain our methodology, and evaluate it first through a simulation study, and then through application to real-world corpora. finally, we conclude with discussions of limitations, ethical considerations, and future research. ## theoretical foundation and motivation estimation of chance agreement is a key element in the evaluation of classification tasks. however, though the field of nlp features a wide variety of span detection and labeling tasks, there is a lack of widely adopted chance-corrected metrics for them. in classification tasks, the kappa coefficient is one of the most popular chance-corrected interannotator agreement measures @xcite @xcite @xcite . the kappa coefficient is defined as (a o -a e )/(1-a e ), where a o is the observed agreement without chance agreement correction, and a e is the expected agreement assuming random annotation behavior. to estimate the chance agreement a e , the key problem is how to build a random annotation model with reasonable assumptions. chance-corrected agreement is unarguably desirable for the evaluation of complex text annotation tasks beyond classification. these tasks encompass sequence annotation tasks @xcite @xcite , which involve a wide array of challenges. the complexity arises from the fact that estimating chance agreement is notably more intricate in comparison to straightforward classification tasks. in classification, the decisions to be made and the available options for each decision are uniform among annotators. however, with span prediction tasks, annotators initially identify the spans requiring labeling and subsequently assign a category to each of these spans. discrepancies can arise at either of these stages, resulting from variations in span selection or category assignment. let's consider the named entity recognition (ner) task as an illustrative example. it's important to note that the quantity and size of recognized entities can significantly differ among various annotators working on the same text. in while estimating inter-annotator agreement has become a crucial step in annotation evaluation, the challenge of estimating chance agreement for sequence annotation remains an open problem. as highlighted by numerous prior studies, the sample space for a sequence annotation task is often not well-defined @xcite . for instance, when considering the variability in annotator preferences, some tend to combine adjacent information, while others prefer to label them as distinct spans. additionally, some annotators choose to encompass surrounding text within a segment, whereas others aim for shorter spans. all of these factors contribute to the complexity of estimating chance agreement in the context of sequence annotation tasks. there is very little research on estimating chance agreement for span prediction tasks like ner. to the best of our knowledge, the most comprehensive and in-depth attempts so far have been the family of krippendorff's alpha coefficients. unlike kappa, the alpha coefficient is grounded in the concept of disagreement, represented as 1-d o /d e , where d o stands for observed disagreement, and d e denotes expected disagreement. in 1995, krippendorff first attempted to extend his alpha coefficient for classification tasks to sequence labeling tasks @xcite . the approach involved concatenating all annotations by different annotators for the same text and generating two copies. one copy remained unaltered, while the other undergoes all possible cyclic shifts. krippendorff estimated the expected disagreement by comparing the differences between pairs of segments across these two sets of annotations. however, this shift-based random annotation model lacks a solid theoretical foundation and exhibits sensitivity to the location of relevant segments. in 2016, krippendorff introduced another datadriven approach to estimate expected disagreement @xcite . this technique compares the dissimilarities between pairs of segments annotated by different annotators. it heavily relies on a large-scale annotation dataset. notably, as it combines all annotation data from diverse texts indiscriminately, it cannot differentiate between different chance agreements corresponding to different annotation tasks. in addition, mathet proposed the gamma coefficient as a new metric for sequence labeling in 2015. the gamma coefficient paper @xcite extensively discusses the various applications and characteristics of sequence labeling tasks. although the gamma coefficient has many contributions, such as combining an optimization of alignment in the computation of the measure, its estimation of expected chance agreement is in line with krippendorff's work and differs fundamentally from our approach. it is critical to emphasize that neither of krippendorff's methods are suitable for sequence annotation tasks, especially within the context of information extraction. when calculating disagreement, the alpha coefficient accounts for all disagreements between segment pairs, encompassing both relevant and irrelevant segments. in cases where relevant information is sparse, the alpha coefficient may be disproportionately influenced by disagreements related to irrelevant information, regardless of the consistency of annotations for relevant content. however, in information extraction tasks, our primary concern typically focuses on the consistency of annotations related to portions of text with a high concentration of relevant information. in the experiments section, we will probe further into this issue by exploring the limitations of alpha coefficients within the context of information extraction. while the specific problem of estimating chance agreement for span prediction tasks is an open problem, we must acknowledge that some relevant research has been done in connection with classification and clustering problems that informs our work and provides a continuum that our work extends @xcite @xcite @xcite @xcite . estimating agreement by chance is relatively simple in classification, because the sample space is fixed and the same for each annotator. in contrast, clustering problems present a greater challenge and bear closer resemblance to span prediction issues. from a conceptual standpoint, one could draw a parallel between elements within the same span and elements within the same cluster. the most commonly employed randomization model in clustering is the permutation model (gates and ahn, 2017), where all potential clusters, each with a fixed number of clusters and a fixed cluster size, are randomly generated with equal probability. however, what distinguishes span prediction from clustering is that the permutation model in clustering doesn't impose any restrictions on the placement of elements within the same cluster. elements within the same cluster can be positioned anywhere. this assumption isn't suitable for sequence annotations, where segments are most typically comprise contiguous elements rather than fragmented. in essence, annotators treat each segment as a whole, rather than labeling each token independently. the variation in sample spaces caused by different labeling tendencies and connectivity constraints within each segment makes this problem quite challenging, especially when annotated segments need to be non-overlapping. therefore, considering the characteristics of span prediction tasks and different annotation tendencies, we propose a new random annotation model to fulfill these requirements. our random annotation model independently models each annotator's tasks. specifically, given the observed annotations for each task by each annotator, our random model uniformly randomizes entity positions while preserving the respective number of entities and the length of each entity. to cater to various application requirements, we have designed two sub-models: the overlapping model and the non-overlapping model. these submodels can accommodate situations where tasks necessitate non-overlapping spans and situations where no such requirement is specified. for example, in as another motivating observation, we recognize that many similarity measures are additive. in essence, the comparison between the annotations of different annotators involves accumulating comparisons among all segment pairs annotated by different annotators. for example, one of the most popular metrics, the f1 score for binary classification, can be expressed as 2a/(2a + b + c), where a represents the number of items labeled as positive by both annotators, and b and c indicate the numbers of items rated as positive by one annotator but negative by the other. it's important to note that when the number and length of spans are both observed, the value of 2a + b + c is a constant. the "positive agreement" rating, denoted as a, reflects the cumulative sum of positive agreements for all compared segment pairs. to simplify the modeling of random sequence annotations, we approach each segment individually, even though each labeled segment is still influenced by constraints imposed by other labeled segments within the same text, particularly in situations where segment overlap is not allowed. we have successfully derived the analytical distribution for the location of each individually labeled segment. additionally, we've observed that the probability remains relatively consistent across most segment locations, reducing the need for numerous redundant calculations. further details will be presented in the next section. ## method in this section, we provide the specification of the random annotation model for sequence annotation, also known as span prediction, and present the calculation, approximation, and asymptotic properties of chance agreement through random annotation. taking ner as an example, we begin by introducing random sequence annotation models for both non-overlapping and overlapping scenarios, accompanied by the mathematical definition of chance estimation. leveraging additive similarity measures, we significantly simplify the esti-mation of expected chance agreement in proposition 1, alongside its corresponding analytical formula for the distribution of random annotations in proposition 2. in proposition 3, we emphasize that each randomly annotated segment exhibits the same probability for most locations, with the exception of a few at the extreme ends, thus further reducing computational complexity. moreover, for lengthy texts with sparse annotation information, the expected chance agreement becomes so negligible that it can be safely disregarded. this assertion is substantiated in proposition 4. the preceding conclusions primarily pertain to non-overlapping scenarios, and we briefly encapsulate the outcome for the overlapping model in proposition 5, as its derivation is straightforward. given space constraints, we present only the primary conclusions and concepts within this section. for detailed proofs, please consult the appendix. we adopt the ner as a representative of complex text sequence annotation tasks to demonstrate how to estimate the chance agreement or performance for sequence annotation evaluation. given a text @xmath0. . . ≺ t n } with a sequence of n tokens t i , @xmath1, . . . , n}, and a pre-defined tag set @xmath2, . . . , c m } with m categorical tags; as a typical task in information extraction, named entity recognition aims to locate and classify segments of text t into pre-defined categories c, such as recognizing disease, medication, and symptom information from clinical notes. mathematically, the annotation task for ner can be formulated as a function φ : t ×@xmath3, where ω is the set of all possible annotations. for any ψ ∈ ω, ψ = {ψ 1,1 , . . . , ψ 1,k 1 , . . . , ψ m,1 , . . . , ψ m,km }, where ψ is an annotation of segments for all predefined categories, k i is the number of segments for i-th category. for an annotation segment ψ i,@xmath4,j , a i,j }, st i,j denotes the index of the first token and a i,j denotes the length for the j-th segment with i-th category. to simplify the discussion, in the following we will focus on single-tag text annotation (i.e., @xmath5, ψ = {ψ 1 , . . . , ψ k }, ψ @xmath6, a j }) since it is straightforward to generalize these techniques to multi-tag annotation as shown in the experiments. to gauge chance agreement, we need a precise definition of random annotation. adapting the permutation model, which is commonly used for clustering, to sequence annotation tasks is impractical due to the absence of location constraints within clusters. this conflicts with the usual intra-segment connectivity assumption in a text annotation setting. to overcome this, we propose a novel random annotation model. it accommodates annotator and task variation while upholding the coherence of text segments. ## experiments to demonstrate the accuracy and effectiveness of our approach, we conducted both simulation and corpus-based experiments foot_0 . we designed the simulation experiments to validate our probability distribution estimation for random sequence annotation. additionally, by varying the length of text, entity length, and quantity in the simulation experiments, we demonstrated the effectiveness of chance correction, comparing it with alpha coefficients. ultimately, we illustrated how our chance estimation impacts the evaluation and ranking of model performance in corpus experiment. since the estimation of chance agreement for the overlapping model is considerably simpler than for the non-overlapping model, all experiments in this paper are configured with the non-overlapping constraint. specifically, for the estimation of the probability distribution for random text annotation, we set to label four segments with lengths of 1, 5, 10, and 15 on a sequence of length 100. figure 1 shows the probability distributions of the four segments at all possible locations calculated with the analytical formula in proposition 2. the four distributions are approximately distributed as the inverted trapezoids with high ends and flat middle part, which confirms the conclusions of proposition 2 and 3. 2 the problem of chance estimation and correction is unique in that, to our knowledge, there is no real benchmark data that can be used to evaluate the performance. therefore, most classic works in this field use synthetic data to illustrate and evaluate the effect of chance correction, such as @xcite and @xcite . intuitively, we know that the chance agreement is related to the size of the search space, the number of annotated objects, and the lengths of the annotated objects. we design the corresponding comparison experiments by varying these three factors. we design three sets of comparison experiments by varying the length of text (simulation 1), the number (simulation 2) and length (simulation 3) of entities. in case a of simulation 1 shown in with 20 tokens, annotator 1 labels 3 entities with lengths of 2, 3, and 4. annotator 2 labels 3 entities with lengths of 3, 4, and 5. the annotations of case b for two annotators are the same as in case a, the only difference is that ten 0s are added after the 20 tokens, that is, neither annotator 1 nor annotator 2 have labeled the extra 10 tokens. as reported in the results in we also compared our results with two alpha coefficients, namely alpha and µalpha (see @xcite equation 2 and equation 5a for specific formulas). at first glance, alpha coefficients exhibit a similar trend in simulations 1 and 3, consistent with intuition, while the results in simulation 2 contradict intuition. however, the underlying reasons are different. our results are derived from chance agreement estimations that align with intuition, whereas the results of alpha coefficients are influenced by their measurement metrics. for the critical estimation of expected disagreement (expd and expµd), it should have an inverse trend with expected agreement (chancef1) because the more the agreement, the less the disagreement. however, the actual results are the opposite, primarily because alpha coefficients include agreement for irrelevant segments, which does not align with the needs of most information extraction tasks. the main purpose of chance correction is to use different baselines for different tasks. in addition, chance correction may also change the ranking of model performance for the same task, although this is not common. as shown in the ## conclusion and discussion in this paper, we propose a novel sequence random annotation model that takes into account the different annotation styles of annotators and the characteristics of sequence annotations. for complex cases where labeled objects are required to be disjoint, we investigate the corresponding distribution characteristic and remove redundant calculations. we also derive an analytical formula to calculate the exact distribution. our focus in this work is how to establish a general framework and corresponding fast algorithm for calculating similarity by chance in complex text annotations. the framework and method proposed in this paper are applicable to all additive similarity measures. moreover, our approach can extend to nested spans by iteratively applying the same method layer by layer, ensuring compliance with the nested structure.