Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments (2025)

Abstract: Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.

## introduction human values, spanning concepts such as benevolence and self-determination, are the motivations that guide our choice and action, and are ordered by importance to form a system of value priorities @xcite . value-laden arguments are statements grounded in our personal values, which we use to motivate our choices @xcite . the identification of the values that support our arguments can reveal our deepest motivations, and as such has been recently investigated in the nlp community @xcite . supervised nlp methods have been proposed to identify the values that support a text segment @xcite . typically, the ground truth labels are chosen through majority aggregation of the annotations @xcite or the annotators engage 1 our code is publicly available at https://github.com/amirhomayouni/subjectivity-value in discussions to reach an agreement on the annotation @xcite . however, due to the subjective nature of valuing @xcite , disagreement in the interpretation of the values that support an argument is natural. for example, consider the following argument in favor of a multi-party political system: "(it) would bring many new and fresh ideas into the forefront". alice may associate the argument with the value of universalism since a multi-party system can provide all people with equal opportunities. bob, instead, may connect it to the values of achievement and personal security, because a multi-party system can more effectively address issues than a single-party system and thus provide more security to its citizens foot_0 . approaches relying on consensus or majority aggregation might obscure the inherent subjectivity of the identification of the values behind arguments, leading to misinterpretation or, at worst, promotion of biases stemming from the annotation process. identifying subjectivity in value-laden arguments thus has broader implications, particularly in contexts that demand participatory deliberation and collective decision-making. the assumption that annotations should be aggregated into a unique label is being questioned within the nlp community @xcite @xcite , in applications ranging from hate speech detection @xcite to sentiment and emotion detection @xcite . in line with these works, in this paper we propose methods to identify subjective value-laden arguments-that is, detect arguments where annotators may have different interpretations of the values that support them. we envision two primary applications of this work. (1) identifying arguments that might be subjective can support a more nuanced annotation process by prompting the collection of additional annotations. contribution we propose two different approaches to detecting subjectivity in value identification within discourse: three methods for inferring subjectivity from individual value annotations, and three methods for directly identifying subjectivity, which we further enhance with contrastive learning strategies. we validate the approaches on the touché23-valueeval dataset @xcite . our results show that we can identify subjectivity in value-laden arguments, and that directly identifying subjectivity-rather than inferring it through value prediction-greatly improves performance. additionally, leveraging contrastive loss does not improve subjectivity prediction performance but brings other advantages. ## related works we review related works on identifying values in text and on subjectivity in nlp applications. ## methods we propose two approaches to identify subjective value-laden arguments-that is, detect whether we expect annotators to disagree on the value annotation of a piece of text. in the first approach, inferred subjectivity identification (is, section 3.2), we train models to predict value labels for individual annotators and infer subjectivity from the variations in labels predicted across annotators. in the second approach, direct subjectivity identification (ds, section 3.3), we train models to directly classify whether a given argument is subjective. ## experimental setup we describe the models, dataset, and evaluation metrics for our experiments. ## discussion in this work, we explored two distinct approaches to address the challenge of identifying subjectivity in value-laden arguments. we discuss our results across methods and value labels. ## conclusion we introduce multiple approaches to identify subjectivity in value-laden arguments. applying our methods to the touché23-valueeval dataset, we demonstrated that directly identifying subjectivity, as opposed to inferring it through value prediction, significantly enhances performance. implementing a dual-task strategy that combines contrastive loss with bce loss does not directly improve subjectivity identification, but leads to a model that is less dependent on per-label subjectivity. finally, using a state-of-the-art model improves performance for direct subjectivity identification, but not for inferred subjectivity identification. we envision a combination of our proposed approaches as future work, e.g. by combining value and subjectivity prediction to capture individual annotators' perspectives in combination with their annotations, thereby potentially improving the identification of subjectivity in value-laden arguments. datasets encompassing diverse perspectives, including varying demographics, lived experiences, and moral values @xcite , such as the recently introduced d3code dataset @xcite , facilitate the integration of these two aspects. finally, we acknowledge that differences in annotation (i.e., subjectivity) can be confounded with noise in the annotation process. previous work addressed this issue by facilitating deliberation among crowd workers @xcite or by assessing the validity of the explanations provided by annotators for their responses @xcite . our work can be instrumental in identifying potentially subjective annotations to support such approaches.