Just Put a Human in the Loop? Investigating LLM -Assisted Annotation for Subjective Tasks (2025)

Abstract: LLM use in annotation is becoming widespread, and given LLMs’ overall promising performance and speed, putting humans in the loop to simply “review” LLM annotations can be tempting. In subjective tasks with multiple plausible answers, this can impact both evaluation of LLM performance, and analysis using these labels in a social science task downstream. In a pre-registered experiment with 350 unique annotators and 7,000 annotations across 4 conditions, 2 models, and 2 datasets, we find that presenting crowdworkers with LLM-generated annotation suggestions did not make them faster annotators, but did improve their self-reported confidence in the task. More importantly, annotators strongly took the LLM suggestions, significantly changing the label distribution compared to the baseline. We show that when these labels created with LLM assistance are used to evaluate LLM performance, reported model performance significantly increases. We show how changes in label distributions as a result of LLM assistance can affect conclusions drawn by analyzing even “human-approved” LLM-annotated datasets. We believe our work underlines the importance of understanding the impact of LLM-assisted annotation on subjective, qualitative tasks, on the creation of gold data for training and testing, and on the evaluation of NLP systems on subjective tasks.

## introduction large language models (llms) have shown impressive performance in many annotation tasks, including subjective tasks common in content moderation and text analysis in the social sciences. evaluating human annotation of subjective tasks for comparison against llm annotation performance, either for the task of end-to-end qualitative analysis or for the construction of ground truth for nlp tasks, is difficult in the absence of domain experts. accordingly, hiring a large number of crowd an-notators (often in service of creating a crowd decision) becomes attractive in the evaluation of nlp systems on social science tasks. however, managing and paying crowdworkers can be difficult, and crowdworkers often have varied performance. a significant body of research explores how ai suggestions can assist qualitative researchers @xcite @xcite . labeling text according to a complex qualitative "codebook" is a repetitive, timeconsuming task, and advances in llm capabilities have made using llms in annotation attractive @xcite . complex, theory-driven text analysis is increasingly being mediated by llms @xcite or llm-powered tools @xcite . given all this progress, llms have created opportunities to create annotation pipelines that appear to work off the shelf without fine-tuning, making automated annotation accessible to practitioners with less technical skill. llms' reported performance in annotating socially complex topics @xcite , sometimes with greater skill than humans @xcite , potentially opens llm-based annotation to an even wider range of fields and practices compared to past years. this makes understanding the many ways humans may interact with llm annotations more important. llms increasingly power both software interfaces and complex research tools @xcite . generating plausible llm annotations for a variety of tasks is now easy, enabling potentially powerful analysis. some systems suggest that putting a "human in the loop" to check annotations can ensure the model's outputs are "reasonable and reliable" @xcite . but given that we know humans are subject to anchoring bias-the bias towards the first option we are presented, especially if it is in the plausible range @xcite -humans may review and confirm llm annotations that are plausible, but nonetheless signifi-cantly change 1) the annotation evaluation process and 2) the outcome the annotations get used for (such as the decision boundary for classification judgments, or the distribution of annotations used in a text analysis), downstream. in this work, we ask several questions. first, when provided with (different forms of) llm assistance, do crowdworker annotators "produce more" by going faster in a complex, subjective annotation task, and does this result in them "understanding less?" do they find llm suggestions accurate and helpful, and how often do annotators take these suggestions? if annotations influenced by llm suggestions are used to construct ground truth on annotation tasks involving complex social concepts, what effect does that have on evaluated performance of these llms on those annotation tasks? we use two community conversation datasets to address these questions, recruiting crowdworkers to annotate the data according to a list of identified themes. these workers are presented with llmsuggested annotations in a variety of formats. we then study how the crowdworkers use these llm suggestions in a complex annotation process in service of answering these research questions. our findings open up questions regarding the use of ai assistance in qualitative research where annotators act as independent reviewers of ai suggestions and can supposedly still retain full control of the analysis. we believe our work underlines the importance of understanding the impact of llmassisted annotation on subjective, qualitative tasks and the interfaces that mediate them, on the creation of gold data for training and testing, and on the evaluation of nlp systems on subjective tasks. in addition to the findings that answer the aforementioned questions, we release a dataset of human and llm-assisted annotations on two complex qualitative codebooks across a variety of conditions. ## related work llms have been increasingly employed in various "human in the loop" ai assistance setups, e.g., in education @xcite and coding @xcite . llms have also been used for annotation tasks, for reasons like significantly decreasing labeling cost @xcite . @xcite discussed how llms are being widely used in computational social science tasks, including subjective tasks, often with success. @xcite found that the performance of llms trained on synthetically generated ground truth data is negatively associated with the subjectivity of the task. more work has shown the challenges of llm annotation: llm annotation performance can be highly variable to prompts @xcite and can depend on the ordering of choices presented in a prompt @xcite . while using llms to create annotations may have "good enough" performance and likely does decrease cost compared to hiring human crowd workers, many tasks still require or benefit from human involvement "in the loop," especially for subjective annotation and analysis tasks; some methods suggest ways of using humans in the loop to optimize prompts for llm annotation @xcite . many ai-assisted text annotation platforms have also been developed and published within the hci space @xcite @xcite , accelerating this change. research shows ai-suggested labeling platforms increase agreement and convergence on qualitative codes, or text labels @xcite . there is increasing evidence that humans tend to anchor on the suggestions that ai systems give, which can result in changes to communication and even user opinions @xcite . some work observed how humans anchor on llm outputs in text analysis tasks, including topic induction. for instance, @xcite show that in a topic generation task, analysts anchor on llm outputs, resulting in different topic lists depending on whether or not they saw the llm versions. this illustrates the potential risk of homogenization of insight as a result of ai influence on text analysis. this concern is raised by messeri and crockett (2024) regarding ai's influence on science more generally, and the authors discuss llms' potential to reduce diversity in human judgment, creating an environment in science where we "produce more but understand less." many cite time savings as their motivation for using llms as annotators, however, findings have so far been mixed regarding productivity. for example, @xcite shows that while ai can boost coding productivity, there exists a tradeoff between productivity and coding quality, and overney et al. (2024) observe that users with ai support in the sensemate platform spent more time on qualitatively coding data. here, we examine how presenting llmgenerated suggestions to annotators in a complex subjective annotation task affects their self-reported understanding of the task as well as the overlap of crowd decisions on text labels with llm annotations of the same labels, with implications for the evaluation of llm performance on these tasks, even when humans are put "in the loop" to review and confirm annotations. ## data and codebook we source data from the fora corpus @xcite . in 2022, the nyc department of health & mental hygiene, the nyc public health corps (phc), and the non-profit cortico recruited over 100 communities to a series of 28 small group dialogues hosted in new york city to understand community resourcing and vaccine decisions during covid-19. following the conversations, community workers created a codebook of themes of interest to the "nyc" corpus (as we will call it), then labeled quotes from the conversations with the themes denoted by the codebook. similarly, in 2021, a conversation series in boston called "real talk for change" was hosted to understand issues in marginalized communities leading up to the 2021 boston mayoral election. we use the conversation data and codebook created for this "rtfc" corpus as well. the nyc codebook developed by community partners had 7 overarching themes related to health and vaccine decisions, including external motivations: friends & family, intrinsic motivations: not wanting to get the virus, and role of community health organizations: health education & support, and vaccine hesitancy. each of the 7 top-level labels had sublabels, for a total of 20 unique labels related to the nyc corpus identifying phenomena of interest to the community partner. similarly, the rtfc corpus has 9 overarching toplevel labels, and 41 sublabels relevant to the corpus, such as safety: street violence and housing: housing affordability. the full codebook for each corpus is available in the appendix. we sampled 200 quotes foot_0 from the nyc corpus as the main data set for this study, and 200 instances from the real talk for change corpus as a partial replication data set. conversation excerpts had an average length of 592 characters. ## methods the experiment compares the annotation of 200 quotes by 5 unique annotators each according to the codebook for both nyc and rtfc corpora. first, we create a crowdworker "baseline" for the annotation task without llm assistance for both corpora. following typical practices, we construct a ground truth set of labels using a 3/5 majority vote for the inclusion of each label based on a set of annotations given by 5 unique annotators who independently reviewed the quote. we then test how this crowd baseline contrasts to annotations created when presenting llm-suggested labels for annotators to review in three distinct formats in an interface. as such, in addition to our control, we adopt the following three experimental conditions testing how interface mediates llm-assisted annotation suggestions: conditions 1-3 provide a sliding scale of assistance to an annotator. the no-assistance baseline in the control condition provides the basis for crowd truth labels. text-based suggestions (condition 1) provide some assistance, but still require the annotators to read the suggestion, then integrate the suggestions themselves into decisions for each annotation question. condition 2 is the same as condition 1 with the exception of including a disclosure that the suggestions were from "ai". this condition tests whether annotators would change their perception of or behavior towards suggestions if they knew their origin-either in their perceived quality or in their rate of uptake. finally, condition 3 provides annotators with the strongest suggestion, drawing the annotator's attention directly to a colored highlight of the suggested label. these conditions were all shown to annotators through a deployment of the open source annotation interface, potato @xcite . screenshots and interface examples are in the appendix. we ran this control and three experimental conditions on the nyc data with gpt-generated labels. in addition to this main set of experiments, we replicated condition 1, text-based label suggestions, with labels created by by llama 3.1 70b @xcite , an open-source model, using the same prompts. second, we also create a no-assistance baseline control condition for the rtfc data. we use this to compare against a replication of condition 1 (text-based suggestions created by gpt-4) on the rtfc data, testing the generalizability of our findings from a main experimental condition from the nyc data to a different codebook and dataset. we generated label suggestions by prompting one of two llms. for most of our experiments, we used openai's api to prompt @xcite , model version gpt-4-1106-preview, zero shot, and for our replication study of condiiton 1 on the nyc corpus, we used llama 3.1 70b @xcite , accessed through the service llamaapi. we used the same prompt style and instructions to prompt both gpt-4 and llama. we prompted each model once per quote to produce a list of labels for each of the 200 quotes from the nyc and rtfc corpora. the prompt details for each task are available in the appendix. ## results we compare outcomes on these annotation tasks across conditions, including the different assistance conditions, 2 corpora (nyc and rtfc), and 2 models that provided suggestions (gpt-4 and llama). ## discussion while mainstream perceptions suggest llms can help speed up annotation, we find that if humans review individual llm-generated suggestions, annotation time does not decrease. from our own measures, we find annotators who were given assistance self-reported improved task understanding. baseline task understanding, as measured through test questions, remained constant across conditions. follow-up work could examine whether seeing ai suggestions sometimes increased task time because there was more information to process, and if suggestions may teach new annotators to do a complex task, increasing their confidence when assistance is given. future work can also examine whether annotators under a different compensation incentive structure approve llm suggestions more quickly than these workers did. annotation is usually the first step in creating ground truth data for evaluating a model's performance on an nlp task. complex, subjective tasks are common in the subfield of nlp for computational social science and cultural analytics. using llms to annotate data used for training and evaluation of a task is attractive due to time and cost efficiency compared to hiring humans, who may noisily interpret a subjective task like this one. however, our findings provide a cautionary note: humans often take llm suggestions they are given, even when a human individually reviews each label that contributes to the ground truth. as such, in taking llm suggestions, annotators homogenize "ground truth" on these tasks towards llm baselines. in nlp for css tasks and in qualitative coding, using labels to measure prevalence of a concept in text is common, so llm suggestions can change these measurements. for example, imagine using these annotations to analyze cited intrinsic motivations for vaccine decision-making given by participants in the nyc conversations: we could view the list of most commonly cited intrinsic motivation labels to understand the most common motivations participants have for vaccine decisions. in the human crowd baseline, intrinsic motivations: basing decisions on data was the least commonly occurring label, whereas when crowd labels were generated by annotators under condition 1 (gpt-influenced labels presented in a text format), it was nearly five times more common, tying "getting back to normal" as the second most common intrinsic motivation tag. analysts using llm-assisted annotations for a substantive analysis of the conversations could thus come to a different conclusion about the relative importance of basing decisions on data as an intrinsic motivation for vaccine uptake in this community: in the case of the human crowd baseline, it was rare (4 instances), and in the llm-influenced condition 1, it became a heavily assigned intrinsic motivations tag, with 19 occurrences in the data. homogenization towards an llm baseline may not be an inherent problem, but the low performance we observe from both gpt-4 and llama when contrasted with a human baseline raises several possible explanations that deserve attention when planning llm annotation tasks. depending on the level of subjectivity, or even potential polarization of the construct being annotated, human annotators may be noisier in interpreting the annotation task and providing their judgments. however, in situations where humans have high agreement, but llms systematically annotate the same construct differently, the llm may be using a different background concept @xcite for annotating the labels than human crowdworkers do. for example, given a starting f1 score of just .2 comparing llm annotations to the human crowd baseline, gpt-4 may operationalize the identification health education and support differently than our aggregation over human crowdworkers did. when llm annotations are used to identify this construct, annotations may thus be measuring something different than the humans crowd base-line does, and the concept of health education and support being annotated becomes more like the llm's conception of health education and support when annotators have its assistance. llm assistance also inherently increases measures of interrater reliability when the same llm suggested are given to multiple annotators. irr measures are used as a positive signal of reliability when humans annotate social science concepts @xcite . however, when annotators use llm assistance and irr is still used as a positive signal of reliability, researchers would need to be confident that the llm's suggestions-and the way it operationalizes each background concept being identified-is more correct than human judgments, since humans are likely to adopt llm suggestions. in subjective tasks, this can be hard to verify and prove, but error analysis of specific ambiguous labels may help. second, there are many tasks in nlp where varied annotator perspective can be valuable @xcite , both from the perspective of constructing a robust ground truth @xcite , or for representing diverse human perspectives on a complex social construct like hate speech @xcite . in qualitative research, some traditions explicitly embrace divergent annotator perspective in order to widen insight in qualitative annotation, @xcite so in these cases, homogenization of insight as a result of llm involvement in annotation may be seen as a source of concern @xcite . in both the cases of creating ground truth for nlp tasks and qualitative annotation, practitioners should know that using or providing llm assistance to annotators will likely result in lessened variation. given potential consequences for representation and construct validity that vary by task, researchers should only proceed with llm-assisted annotation with a level of caution appropriate to their task and goal. they should recognize that using llm assistance to construct ground truth labels inflates perceptions of model performance on that task, even when humans review them and individual judgments are aggregated into a crowd ground truth. follow-up work can investigate if these findings hold for a similarly complex annotation task, a less complex annotation task, and when employing expert annotators rather than unspecialized crowd workers. homogenization effects may be lessened by presenting information about model confidence, or alternative ideas, and not just a single set of suggestions. ## conclusion in a pre-registered experiment with 410 unique annotators and 7,000+ annotations across 3 experimental conditions, 2 models, and 2 datasets, we find that presenting crowdworkers with llmgenerated annotation suggestions did not make them faster, but did improve their self-reported confidence in the task. annotators strongly uptook suggestions, changing the label distribution to more closely resemble the llm's proposed distribution. we also found that using llm-assisted labels to evaluate model performance resulted in much higher reported f1 scores than when using a human crowd baseline, with increases in f1 scores for model performance on some labels by as much as +x.56. obviously, using labels influenced by the model to evaluate the model is not standard or advisable in classic evaluation paradigms. however, in the many systems being created that "just put a human in the loop" to review llm annotation outputs, this paradigm of reviewing llm outputs to "approve" them is increasingly likely to occur. practitioners should know that, especially in subjective tasks, simply reviewing llm suggestions will nudge the distribution of label outputs towards an llm baseline, even if humans are given a chance to review the outputs.