Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement (2025)

Abstract: We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks. While inter-annotator agreement is frequently used as an indication of label reliability by measuring consistency between annotators, we argue for the additional use of intra-annotator agreement to measure label stability (and annotator consistency) over time. However, in a systematic review, we find that the latter is rarely reported in this field. Calculating these measures can act as important quality control and could provide insights into why annotators disagree. We conduct exploratory annotation experiments to investigate the relationships between these measures and perceptions of subjectivity and ambiguity in text items, finding that annotators provide inconsistent responses around 25% of the time across four different NLP tasks.

## introduction agreement measures are commonly used to assess the utility of judgements made by human annotators for natural language processing (nlp) tasks. indeed, the reporting of inter-annotator agreement (or inter-rater reliability) has long been the standard to indicate dataset quality @xcite and frequently serves as an upper bound for model performance on a task @xcite . while inter-annotator agreement is frequently used in nlp to determine the reliability of labels or the processes used to produce them @xcite , intra-annotator agreement is rarely, if ever, reported. however, we can use it to measure the temporal consistency of the annotators who chose the labels and, hence, the stability of the labels and data that they generate. 1 consistency and label stability are important because, without them, annotation schemes are unlikely to be repeatable or reproducible @xcite . 2 such measures of intra-rater agreement are frequently reported in areas of medicine such as physiotherapy (e.g. @xcite , and speech pathology (e.g. @xcite . intra-rater measures are also reported in other fields as diverse as economics @xcite , software engineering @xcite , and psychology @xcite . however, reporting intra-annotator agreement is so far extremely uncommon in nlp, as we show in a systematic review in section 2. disagreement and label variation in nlp in addition, we argue that the use of inter-and intraannotator agreement allows us to distinguish and measure different sources of observed label variation @xcite . this is important as nlp researchers have increasingly recognised that, for many tasks, different points of view may be equally valid @xcite @xcite , and that their aggregation can erase minority perspectives @xcite . one of the main challenges in implementing this new paradigm is the interpretation of disagreement. disagreement between annotators may be due to two sources: 1) genuine differences in their subjective beliefs/perspectives, which can be desirable under this paradigm, or 2) task difficulty, ambiguity, or annotator error, all of which are undesirable. while agreement measures between annotators can give us an idea of task subjectivity, they provide little insight as to its difficulty, ambiguity, or the quality and attentiveness of the annotators themselves @xcite . in the following, we propose the use of intraannotator agreement as a measure of subjectivity. ## intra-annotator agreement in the nlp community to get a snapshot of the extent to which intraannotator agreement is reported in the nlp community, we conducted a systematic review of papers published in the anthology of the association for computational linguistics (acl). 4 here, we wish to discover for which tasks and what purposes nlp researchers collect and report on repeat annotations and evidence for how and when repeat items should be presented to annotators. full details of the re- ## exploratory annotation experiments we conduct an exploratory annotation experiment to investigate the relationships between agreement measures and the possible reasons for disagreements and inconsistencies. we also investigate whether, as is commonly believed, specific task types are generally more subjective than others. ## 65 data we use subsets of four english language datasets, see ## results we report agreement for each task, and examine differences between the groups of items labelled 7 https://www.prolific.co/ 8 particluarly in the offensive language dataset. 9 the guardian april 20 2021 @xcite . overall agreement as intra-annotator agreement is typically assumed to be 100% (i.e. by omitting to consider it @xcite )), we measure and raw report percentage agreement as a primary metric to examine whether this holds. for inter-annotator agreement, we calculate these pairwise across annotators and report the means. for completeness, we also report cohen's kappa scores in appendix c. agreement scores are presented in ## discussion and conclusion we have examined the role and use of intraannotator agreement measures in nlp research. calculation of such measures can act as an important quality control and could potentially provide insights into the reasons for disagreements between annotators. however, in a systematic review, we found that they are rarely reported in this field. we have proposed a framework for the interpretation of inter-and intra-annotator agreement, the reliabilty-stability agreement matrix. exploratory annotation experiments failed to validate our theory that this framework can be used to tease apart subjectivity and ambiguity, and it proved to be very hard to recognise or agree on these, even for trained annotators. however, we have shown how comparing both inter-and intra-annotator agreement enables quantification of the difficulty of particular 5 67 tasks and/or annotation items. strikingly, we found that, across four different tasks, crowdsourced annotators were consistently inconsistent, calling into question the implicit assumption that labels provided by individual annotators are stable, and reinforcing the need to collect within-annotator labels for nlp tasks, including those typically considered to be 'objective'.