Understanding Disagreement: An Annotation Study of Sentiment and Emotional Language in Environmental Communication (2025)

Abstract: Emotional language is central to how environmental issues are communicated and received by the public. To better understand how such language is interpreted, we conducted an annotation study on sentiment and emotional language in texts from the environmental activist group Extinction Rebellion. The annotation process revealed substantial disagreement among annotators, highlighting the complexity and subjectivity involved in interpreting emotional language. In this paper, we analyze the sources of these disagreements, offering insights into how individual perspectives shape annotation outcomes. Our work contributes to ongoing discussions on perspectivism in NLP and emphasizes the importance of human-centered approaches and citizen science in analyzing environmental communication.

## introduction addressing the escalating environmental crises requires coordinated global action @xcite . emotions play a key role in motivating such action, shaping a range of behaviors from policy support to civil disobedience @xcite @xcite . although there has been limited interdisciplinary research on the role of emotional language in environmental communication, existing studies suggest that such language can play a key role in mobilizing individuals for collective action @xcite @xcite . in this context, we define emotional language as the use of words or expressions that convey affective states. importantly, we use the term emotional language -rather than emotion -to emphasize that our focus is on the strategic use of emotion-related expressions in group communication, rather than on measuring the actual felt emotions of individual speakers or writers. this distinction is particularly relevant when analyzing collective actors such as environmental groups, whose language is often shaped by strategic communication goals. however, the outcome of using emotional language in different socio-political contexts -especially in the discourse of groups with different ideologies, identities and thematic priorities -is still poorly researched and not well understood @xcite @xcite . this paper is part of a broader project examining emotional language in environmental communication by highly visible and polarizing activist groups, and analyzing the emotional reactions such language provokes among the public @xcite . while the larger dataset includes multiple organizations, this study focuses on tweets from extinction rebellion (xr), a global activist group using nonviolent civil disobedience to demand urgent climate action. our overarching goal is to develop a comprehensive, annotated dataset tai-lored to environment-related communication, with applications in both environmental communication research and natural language processing (nlp). for this paper, we annotated sentiment and emotional language in xr's x (formerly twitter) discourse, revealing substantial annotator disagreement. we analyze the factors driving this disagreement and explore how these insights can refine future annotation efforts in nlp and environmental communication research. our findings highlight challenges in creating reliable annotated datasets and contribute to the broader debate on perspectivism in nlp, which recognizes that multiple valid interpretations of a text can coexist due to annotators' diverse backgrounds, experiences, and perspectives-challenging the notion of a single ground truth @xcite @xcite . to guide our investigation of these challenges and the implications of annotator subjectivity, our current work is structured around the following research questions: rq1 what factors may contribute to variation and disagreement in annotator labeling behavior? rq2 what insights can be gained from the observed disagreement, and how can they inform future annotation efforts? the main contributions of this paper are as follows: ## related work this section reviews relevant literature on environmental communication as well as sentiment and emotion analysis. ## data and annotation this section outlines the dataset and annotation process used in our study. ## understanding annotator disagreement to better understand the sources and implications of annotator disagreement in our dataset, we address our two research questions in two parts. first, we conduct a set of quantitative and qualitative analyses to identify factors that may contribute to variation in labeling behavior. then, we reflect on the insights gained from these observations and how they can guide future annotation practices and research design. ## conclusions and future work this study examines disagreement in environmental communication annotation, particularly within activist group discourse. our findings highlight the impact of internal factors, such as sociodemographic backgrounds and emotions, and external factors like the annotation process. these challenges hinder achieving high iaa in subjective language assessment, especially in emotionally charged topics like environmental activism. our results align with previous research questioning the idea of a single ground truth in annotation tasks @xcite @xcite . perspectivism in nlp tasks, such as hate speech detection and emotion recognition, underscores the role of individual annotators' perspectives on labeling outcomes @xcite @xcite @xcite @xcite . this subjectivity is critical in environmental communication, where diverse reactions provide valuable insights into audience perceptions. importantly, disagreements among annotators reveal the varied emotional engagement with environmental issues @xcite . future research should improve annotation methods to better address subjectivity. adopting perspectivist frameworks, using pre-annotation surveys to capture annotators' backgrounds, and integrating llms to complement human labeling are promising approaches. expanding our dataset to include more environmental groups and studying the temporal aspects of annotation subjectivity, such as emotions or external events, could offer further insights. ultimately, applying these findings to tailor environmental communication strategies for diverse audiences will be crucial in bridging nlp and environmental communication.