Another Approach to Agreement Measurement and Prediction with Emotion Annotations (2025)

Abstract: Emotion annotation, as an inherently subjective task, often suffers from significant inter-annotator disagreement when evaluated using traditional metrics like kappa or alpha. These metrics often fall short of capturing the nuanced nature of disagreement, especially in multimodal settings. This study introduces Absolute Annotation Difference (AAD), a novel metric offering a complementary perspective on inter- and intra-annotator agreement across different modalities. Our analysis reveals that AAD not only identifies overall agreement levels but also uncovers fine-grained disagreement patterns across modalities often overlooked by conventional metrics. Furthermore, we propose an AAD-based RMSE variant for predicting annotation disagreement. Through extensive experiments on the large-scale DynaSent corpus, we demonstrate that our approach significantly improves disagreement prediction accuracy, rising from 41.71% to 51.64% and outperforming existing methods. Cross-dataset prediction results suggest good generalization. These findings underscore AAD’s potential to enhance annotation agreement analysis and provide deeper insights into subjective NLP tasks. Future work will investigate its applicability to broader emotion-related tasks and other subjective annotation scenarios.

## introduction despite the significant progress in multi-modal nlp @xcite , such as gpt-4o 1 , accurately recognizing and interpreting human emotions across different modalities @xcite remains a substantial challenge. this difficulty primarily arises from the complexity and variability of emotional expressions @xcite , which often manifest themselves differently across modalities. consequently, there is a growing demand for fine-grained 1 https://openai.com/index/hello-gpt-4o/ and reliable datasets to support the training and evaluation of emotion recognition systems @xcite . as a common and popular practice, the use of evaluation metrics like the kappa/alpha family has almost become a standard step in dataset construction @xcite . however, even with careful dataset design, many annotated (multimodal) emotion datasets exhibit low kappa/alpha scores @xcite @xcite @xcite , and few studies have explored the reason behind these low scores. given that the interpretation of kappa/alpha values can be significantly influenced by factors such as the numbers of annotators and categories @xcite , and considering the inherently subjective nature of emotion annotation @xcite @xcite , we propose the complementary use of the absolute annotation difference (aad) as an intuitive metric to better measure and examine agreement and disagreement patterns, particularly in datasets with low kappa/alpha scores. to validate this proposal, we conducted two experiments. the first is a pilot study on a small multimodal emotion dataset, where (dis)agreement was assessed using both kappa/alpha and aad. the findings suggest that aad provides a distinct perspective on (dis)agreement and effectively uncovers annotation patterns. building on these insights, the second experiment applied aad to (dis)agreement modelling and prediction, achieving an accuracy improvement of nearly 10%. together, these experiments highlight the added value of aad in enhancing the analysis and prediction of (dis)agreement in emotion annotation tasks. by offering a complementary view to conventional metrics, our work contributes to a more nuanced understanding of annotation reliability. we hope this research can inspire further methodological innovation in dataset evaluation and design. ## related work many tasks in natural language processing and computer vision sometimes suffer from disagreement @xcite @xcite , as they involve tasks (e.g. emotion detection, hate speech detection) which are difficult to define and influenced by an annotator's cultural, social, ethnic, and other backgrounds. in addition, annotation differences might also just be caused by attention slips @xcite . in their survey paper, @xcite identified several sources of disagreement, including annotator errors, annotation schemes, ambiguity, subjectivity and item difficulty. although disagreement is sometimes undesirable, there are also scholars embracing disagreement and proposing to preserve disagreement as different perspectives to the same stimuli @xcite @xcite . ## data to thoroughly investigate annotator disagreement within and across modalities and identify factors that make certain data types (textual, audio, silent video, or multimodal) challenging to annotate, we designed a two-session annotation study. in the first session, four annotators independently annotated a small dataset across four modality se-tups: text, audio, silent video, and multimodal, providing distinct sets of annotations for each modality to assess inter-annotator agreement. in the second session, one annotator re-annotated the dataset twice -114 and 290 days later. these additional annotations enabled intra-annotator agreement analysis by comparing the three sets over time. the annotator reported vaguely remembering the content of some instances but stated not to have a recollection of the previous annotations. following du et al. (2025) , we use a subset of their unic dataset, consisting of 94 youtube video clips featuring authentic emotional expressions, unlike the exaggerated portrayals common in movies or tv series. each video clip spans about 10 seconds, which was deemed sufficient in preliminary tests for identifying emotional states across modalities @xcite . four annotators (two male, two female college students proficient in english) participated after training on the annotation method and tools, ensuring consistent and informed annotations. ## annotation difference analysis to evaluate the annotations across annotators and modalities, we performed significance tests using the four sets of annotations from the first annotation session. chi-square test results suggest that both the categorical and dimensional emotion annotations are significantly influenced by the modality (@xmath0.068e -6 , @xmath1.002), and the annotators (@xmath2.669e -25 , @xmath3.660e -42 ). as a common practice in dataset construction, we calculated both fleiss' kappa and (weighted) krippendorff's alpha. for emotion and valence, the kappa results, ranging from 0.19 to 0.33, suggest low agreement in the annotations, and similarly, the krippendorff's alpha results, ranging from 0.22 and 0.64, reflect the same conclusion. this holds true even when considering different levels of measurement (e.g., ordinal and interval, etc.) or using weighted versus unweighted approaches valence annotations. note that in our experiments, valence is scaled as integers from 1 to 5, which can be interpreted as very negative, negative, neutral, positive and very positive, making it a hybrid of multiple data types @xcite . default weights were applied in the calculation across these data types. for arousal, the results indicate less agreement. the results in ## disagreement prediction based on the insights from our agreement analysis, we also explored the potential of using aad to model and predict disagreement, with the goal of identifying instances where annotators exhibit diverse interpretations, which can reveal valuable insights into the data. however, there are only a few studies on disagreement prediction, particularly concerning modalities such as audio or video. one recent research that caught our attention is the work of @xcite who performed dis-agreement prediction on a dataset of over 100,000 textual instances @xcite . given the constraints of data availability and computation cost, we conducted our initial investigation on texts, taking the research of @xcite as a starting point. ## conclusion while traditional iaa measures are favoured for providing a single comprehensive score that summarizes overall agreement across a dataset, they often complicate the interpretation of low scores and fail to capture finer (dis)agreement patterns. prior research (e.g., @xcite ) has highlighted these limitations, but effective solutions remain an open area of research. our study contributes a systematic exploration of aad as a more interpretable measure of annotation variations, particularly in subjective tasks like emotion recognition. rather than presenting aad as a completely novel metric, we demonstrate its potential to complement existing agreement measures by providing richer insights into (dis)agreement. we first applied aad to analyze both inter-and intra-annotator (dis)agreement with a multimodal dataset, which enables us to observe how these (dis)agreements manifest differently depending on the input channel, proving a more comprehensive understanding of (dis)agreement across modalities. furthermore, a nearly 10% increase in accuracy in the disagreement prediction task demonstrates the advantages of our aad-based approach. due to the scarcity of available (multimodal) emotion datasets with sets of annotations for agreement study, we conducted our study on the most suitable dataset currently accessible. while a larger dataset could further validate our findings, our dataset is representative of real-world annotation challenges, and the observed improvements in disagreement prediction align with prior work. we would extend this research when new datasets become available, but the current results already demonstrate the effectiveness and potential impact of aad.