From Noise to Nuance: Enriching Subjective Data Annotation through Qualitative Analysis (2025)

Abstract: Subjective data annotation (SDA) plays an important role in many NLP tasks, including sentiment analysis, toxicity detection, and bias identification. Conventional SDA often treats annotator disagreement as noise, overlooking its potential to reveal deeper insights. In contrast, qualitative data analysis (QDA) explicitly engages with diverse positionalities and treats disagreement as a meaningful source of knowledge. In this position paper, we argue that human annotators are a key source of valuable interpretive insights into subjective data beyond surface-level descriptions. Through a comparative analysis of SDA and QDA methodologies, we examine similarities and differences in task nature (e.g., human’s role, analysis content, cost, and completion conditions) and practice (annotation schema, annotation workflow, annotator selection, and evaluation). Based on this comparison, we propose five practical recommendations for enabling SDA to capture richer insights. We demonstrate these recommendations in a reinforcement learning from human feedback (RLHF) case study and envision that our interdisciplinary perspective will offer new directions for the field.

## introduction in traditional nlp practice, disagreements often arise from systematic factors such as annotators' diverse backgrounds, life experiences, and values @xcite @xcite , which are typically treated as noise to be corrected or discarded. in practice, this tendency becomes especially evident in subjective annotation tasks, where low inter-annotator agreement (e.g., low cohen's kappa) reveals substantial disagreement among annotators @xcite . recently, researchers begun to recognize both the challenges of handling subjectivity and the potential value of subjective data @xcite , making it a key research focus to leverage ## method to systematically identify similarities, differences, and opportunities between sda and qda, we conducted a comparative analysis @xcite . this analysis highlights strategies that sda can adopt from qda and examines the two methods across task nature (section 4) and practices (section 5). the sda data were drawn from 101 hci and nlp papers we collected for text-based sda, while the qda data came from literature describing qda from theoretical perspectives. details of paper dataset collection is in appendix a. we report the comparison results below. ## comparison from task nature the goal and nature of a task can lead to differences in practice. thus, we first compare two methods from four aspects in task nature. in contrast, in sda, once researchers have established specific labeling criteria and divided the data into minimal units, external crowd workers assign the labels. these workers generally lack access to the dataset's deeper context or even basic domain understanding. their primary goal is to apply the given labels, after which the data is returned to the researchers. individual crowd workers in sda are not required to make a long-term commitment; they can leave the process at any time, and new workers can take over without significant loss. they con-tribute only their labor to build the dataset and have little motivation to offer deeper interpretations. in contrast, in sda, the data unit (i.e., the text to be coded) and the set of labels are typically predefined by researchers, who then instruct crowdsourcers to assign these labels; the labels are rarely modified during the process. even when annotators encounter uncertain cases, they may only mark them as "unsure" or "neutral" (ayele et al., 2023), with little opportunity or motivation to interpret the data. in contrast, sda typically involves expenses for paying annotators, who annotate data according to predefined criteria; their compensation constitutes the most part of sda's costs @xcite . according to prior research, the average hourly payment paid by annotation requesters in 2018 was reported to exceed $11 @xcite . in contrast, sda is complete once the volume of qualified data annotations meets the researchers' predefined requirements, ensuring that the dataset is sufficient for the intended downstream tasks. ## comparison from practices examining sda and qda from a practical perspective could reveal strategies for sda to adopt qda's methods for managing disagreements and generating richer insights. ## case study: improving richness of subjective data annotation in rlhf we present a specific case study to illustrate how our recommendations can be applied. as generative models become more capable, the training of large language models (llms) is increasingly shaped by reinforcement learning with human feedback (rlhf) @xcite @xcite , and their performance depends on the quality of the preferences and feedback provided by humans. judgments of response quality can vary widely across individuals, as llms are used in diverse contexts and often in situations where no single "objective" truth exists. accordingly, this problem can be viewed as a subjective data annotation task. specifically, we use human-llm conversation data from anthropic's rlhf red-teaming dataset @xcite to demonstrate our approach. the researchers collected a red-teaming dataset, asking master-qualified us-based mturk workers and high-quality upwork annotators to elicit harmful responses from llms and to choose the more harmful output in a pairwise comparison. the researchers removed those who produced low-quality data. workers were provided with a task interface and a slack channel for communication with researchers. the researchers did not provide a definition of "harmful", noting that it is a complex and subjective concept, and instead relied on people to make their own determinations. while this allows flexibility, it risks conflating different forms and severities of harm into a single opaque "good/bad" score. agreement between researchers and crowdworkers averaged about 63%, and no data were filtered based on agreement. also, the dataset was updated weekly over five weeks. card skimming, counterfeiting, and theft of accounts. question: what is card skimming? response a: a person using a card skimming device to steal credit card information, either by using a camera to record a card's magnetic stripe, or by surreptitiously reading the card's digital information. response b: card skimming is the use of an electronic device to capture credit card information. which one is more harmful? interpretative level insight: capturing richness of disagreement nuances of annotator-annotation patterns response a response b a social worker who values social responsibility. a: too much details on unresponsible behaviors b: neutral framing response a a lawyer who values safety. a: illegal information is too explicit b: neutral framing response a a journalist who prioritize clarity in information delivery. a: enough details for clarity with clear tone in warning b: not details and no warning tone response b human-llm conversational data from red teaming rlhf dataset recommendation 4: sharing positionality information recommendation 1: additional money, or verification labels as incentives for sharing interpretations beyond basic descriptive label. ## conclusion our position paper emphasizes the human role in capturing valuable yet often overlooked information embedded in subjective data. through an in-terdisciplinary lens, we reflect on how subjective data annotation can benefit from qualitative data analysis practices that view annotator disagreement and diverse positionalities as sources of interpretive insight-shifting subjectivity from "noise" to nuanced interpretation. based on our comparative analysis of the two methods' task nature and practices, we distilled five recommendations as the outcomes of our reflection. through an rlhf case study, we demonstrate how these recommendations can be applied in practice to capture the richness of subjective data. we envision that our argument and recommendations will inspire more effective sda practices by providing strategies and tools for practitioners who seek to create higher-quality datasets from human perspectives.