The LLM Effect: Are Humans Truly Using LLM s, or Are They Being Influenced By Them Instead? (2024)

Abstract: Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses. However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership. The study, conducted in two stages—Topic Discovery and Topic Assignment—integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis. Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics. However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis.

## introduction large language models (llms) like @xcite , llama @xcite etc., have recently dominated the research world by showcasing capabilities that are nearly equivalent to human performance in different analytical tasks. researchers and organizations are increasingly using these models to conduct time-consuming analyses that were previously handled by human experts @xcite . however, this raises a critical question: are llms truly ready to undertake highly specialized tasks? domains such as policy studies are inherently very complex and nuanced, requiring an adept proficiency that may extend beyond the current capabilities of llms. while these models can enhance efficiency and provide substantial support, their ability to match human expertise in specialized fields requires further scrutiny. the advantages of using llms include increased efficiency, consistency in output, and the ability to handle large volumes of data quickly @xcite . on the other hand, using llm suggestions as a helpful-guide for such open ended analysis has the potential to cause experts to rely heavily on the given suggestions, therefore, introducing anchoring bias @xcite for their task. to address these concerns, we designed a user study that integrates experts and llms in a highly structured way. our key contributions are: in summary, we found that with llm suggestions experts performed the topic assignment task much faster than without them. however, a noticeable anchoring bias @xcite was observed in the analysis when experts worked with llm suggestions. the bias introduced by llm suggestions raises an important question: is the trade-off between the increased efficiency worth the potentially biased analysis? we also discovered that during the topic discovery stage, experts with llm suggestions tended to keep the topics as they were, without making significant changes, even though the llm suggestions were mostly very generalized and broad. conversely, experts without llm suggestions often came up with highly tailored topics specific to their given documents. this indicates that while llms are very effective for analyses requiring broad and generalized topics, they struggle with providing the depth needed for more nuanced tasks. ai tools topic modeling @xcite or analysis is the process of identifying patterns of word co-occurrences and using these patterns to group similar documents and infer topics within them. the most well-known algorithm for such topic modeling is latent dirichlet allocation (lda; @xcite , which examines word co-occurrences and groups documents accordingly. however, lda often fails to capture the underlying context of documents, which is necessary for studying contextrich documents like those in our study. in addition, lda yields a specific probability distribution over the words of the vocabulary that need to be interpreted as a "topic", making it difficult to use from a practical perspective. another approach is bertopic @xcite that uses transformer models to understand the context within text and improve topic coherence. however, bertbased models can also struggle with generating interpretable topic labels @xcite . in addition, the underlying model for bertopic (bert) has a very small context window, which leads to cumbersome heuristics needed for topic classification over longer documents. instead of these techniques, we use a slightly modified version of topicgpt @xcite , a prompt-based framework leveraging gpt models to uncover latent topics in a text collection. it produces topics that align better with human categorizations compared to competing methods, while also generating interpretable topic labels and relevant definitions instead of ambiguous bags of words, making it a comprehensive tool for our topic modeling needs. the llm model we use is gpt-4-0125-preview queried via the api. this gpt model has a context window of 128,000 tokens, which makes the feasibility of our study possible, given our 1-hour long interviews. ## study design given the domain of the transcripts, we conducted the analysis focusing on topics relating to ai policy. we consulted four international policy experts to help annotate the transcripts with relevant topics. they were asked to ground their analysis within the realm of ai policy in india. the annotators have extensive background knowledge in policy studies, with one being an expert on indian policies. we conducted our study in two stages (see figure 1 ), each utilizing a research model with two settings. 1. control setting (c), the traditional setting that involves expert annotators conducting their analysis on the given documents without external suggestions from other tools or sources. 2. treatment setting (t), a more custom setting in which we provide the llm-generated suggestions to the expert annotators as a helpful guide. note that, the annotators do not query the llm directly. we designed a user-interface through label studio @xcite to help facilitate this study. the annotators accessed their documents through their own individual label studio interfaces. specifically in the treatment setting, we provided the llm suggestions as highlighted texts with corresponding label names. we instructed our experts to vocalize their thought process while conducting their analysis. this thinking aloud process @xcite during problem-solving requires annotators to continuously talk and verbalize whatever thoughts come to mind while doing the task. unlike other verbal data gathering techniques, this method involves no interruptions or suggestive prompts. annotators are encouraged to provide a concurrent account of their thoughts without interpreting or explaining their actions, focusing solely on the task at hand. two research assistants served as scribes during the user study to document the experts' thought processes. this approach allows us to qualitatively study the strategies employed by the experts, providing insights into how they interpret and tackle the task of analyzing the documents. we also developed pre-and post-analysis surveys to assess how familiar the expert annotators were with llms. the pre-survey aims to understand their initial assumptions regarding the use of llms versus conducting the analysis in the traditional way. with the post-survey, we wanted to gauge their reactions to the llms' suggestions and determine if they would be interested in using such technology in their future workflows. ## stage 1: topic discovery methodology for stage 1, our goal was to have expert annotators build and curate a comprehensive topic list, generated over a set of documents, with and without the llm suggestions. we also generated a similar topic list solely by an llm -which was provided to the annotators in the treatment team -and have analyzed the similarity of both of the topic lists. figure 1 shows the process of forming the final topic list which lays the foundation for subsequent analysis of stage 2. we allotted five hours for expert annotators to complete this stage of the study. we divided our four expert annotators into two teams: annotators 1 (a1) and 2 (a2) conducted the topic discovery task under the treatment setting, while annotators 3 (a3) and 4 (a4) completed the task under the control setting. the annotators were aware of each other's tasks, meaning the control annotators knew that the treatment group would receive llm suggestions generated by the gpt model. we applied topicgpt @xcite ) prompts to generate a llm-provided topic list over the four stage 1 documents. it is a two shot topic modeling prompt that generates a comprehensive topic list over a given document. we prompted the llm four separate times for each of the 4 documents, and then we used a merging prompt to combine the four topics lists and remove any duplicate topics (see c and d). the final llm generated topic list (l) (see control: topic discovery -experts only the annotators were instructed to read over their assigned document and generate a list of latent topics with corresponding definitions that exist within their document. they were also asked to highlight any sentence or paragraph they considered pertinent to a topic within their own generated topic list with the corresponding topic label. ## stage 2: topic assignment methodology in stage 2, we studied how the topic assignments vary for annotators in both control and treatment settings. for this stage, we used 4 documents, different from those used in stage 1. each annotator received 2 documents, and they were instructed to work on these individually sans discussion with other annotators. annotators were also instructed to conduct topic assignments on the two documents in two different settings: one as control and the other as treatment (see to accomplish our research goal of measuring the llm accuracy of topic assignments, we instructed both expert annotators and the llm to assign topics on a per-paragraph basis. this would allow for a granular enough approach to collect a meaningful amount of data points per document, while ensuring enough context for both experts and the llm to comfortably make topic assignment decisions. on average, our stage 2 transcripts contained 44 paragraphs. for the treatment setting, we generated topic assignments over the same set of transcripts by prompting the llm with a topic assignment prompt (see appendix e). the model was provided with the final topic list (h) along with the transcripts at a per paragraph level. multiple topic assignments per paragraph are allowed. control: topic assignment -expert only for the control setting, annotators received a transcript and the final topic list (h) with definitions (see treatment: topic assignment -experts+llm in the treatment setting, we provided the llmgenerated assignments to the experts to annotate each document at a paragraph level with topics from the same topic list as llms, allowing multiple topics per paragraph. annotators received the llm annotations as suggestions and were tasked with cross-checking and, if necessary, correcting the assignments. ## discussion it is apparent there are multiple factors at play when it comes to utilizing llms for open-ended tasks such as topic modeling. in terms of promising impact presented by llms, we put the difficulty of this task fully into perspective. given a document with dozens of paragraphs, the llm must decide on a label or combination of labels out of a possible 20 choices to assign to each paragraph. when we measure the accuracy of these llm label assignments against 4 independent expertly annotated control documents, we get an average recall of 57.7% (see coupled with reasonable accuracy, we observe substantial increases in workflow efficiency. we recorded a 133.5% words per minute annotation speed increase when annotators utilized llm suggestions. this presents one possibility of mas-sive reductions in labor intensive and time consuming workloads. however, if the goal is to obtain gains in workflow efficiency, this will come at significant cost. as mentioned earlier in section 5, we see a significant difference between control and treatment annotation decisions (see additionally, as shown in our stage 1 results, five topics that human annotators decided to add to the final topic list were not generated by the llm. these five topics reflected the effort of a nuanced examination of the transcripts provided to the expert annotators. for example, "policing and surveillance" was not captured by the llm (see ## related work topic modeling the motivation underpinning topic modeling is the notion that concepts or latent "topics" exist within a document. as mentioned in section 2, lda is a popular machine learning methodology for topic modeling. however, implementing lda models for real world applications has proven impractical, because of their inherent lack of interpretability @xcite @xcite . as language models become more powerful and capable, some researchers have begun to develop ways to utilize these ai tools to approach the broad problem of latent topic discovery and assignment. topicgpt @xcite introduced a llm prompting framework which utilizes the power of pretrained gpt models. both collabcoder @xcite and sensemate @xcite propose a human-in-the-loop coding pipeline that is geared towards novice annotators for simple and short coding tasks. collabcoder goes a bit further and also suggests group work as part of its pipeline. while these studies utilized older language models, the potential efficiency gains observed in these studies help reinforce the findings of our own study. human-llm partnership much optimism surrounds the conversation regarding human-llm partnerships and many recent user studies have explored the benefits of integrating llms into human workflows @xcite . microsoft has also published two technical reports regarding employee experiences using "copilot", their gpt-powered ai assistant. in one study they found employees "read 11% fewer individual emails and spent 4% less time interacting with them" @xcite . anchoring bias anchoring bias is a phenomenon of human behavior in which, during the decision making process, a human is introduced to an initial piece of information, and future decisions are heavily influenced by the "anchor" this initial piece of information establishes @xcite . the theory of anchoring bias has been around for many decades and has been observed in many contexts and situations @xcite . in one study with law enforcement agents (leas) and mapping algorithms, @xcite found that leas became easily anchored to the initial algorithmic mapping output. even after numerous suggestions from the researchers to consider options beyond the initial output, leas were still anchored to the first piece of information they saw. in another study, @xcite found that in legal courtroom settings, judges were found to be influenced by some initial information, and passed sentencing judgements while being anchored to that initial information. in a third study, @xcite used a more general knowledge non-expert setting and found subliminal anchoring bias occurring. connecting the theory of anchoring bias with llms, we next mention the well-researched areas describing many of the potential dangers regarding llm usage and outputs. llm hallucinations have been well documented @xcite , and their inherent bias regarding culture, gender, race, etc. has been heavily studied and confirmed @xcite . @xcite states "for all their power and potential, large language models (llms) come with a big catch: they contain harmful biases that can emerge unpredictably in their behavior." along with dangerous content, they have also been found to be overconfident and persuasive @xcite . thus, toxicity, hallucinations, and persuasiveness are a potently dangerous combination. as @xcite state "with the emergence of large language models that produce human-like language, interactions with technology may influence not only behavior but also opinions: when language models produce some views more often than others, they may persuade their users." so, the possibility of toxic and incorrect content combined with persuasive execution of language output can potentially lead to a pernicious influence on end users that is both worrisome and unpredictable. ## conclusion and future work our study highlights the trade-offs of integrating llms into expert topic modeling workflows. llms have made incredible strides in open ended tasks such as discovering and assigning generalized topics over documents. however, as the capabilities of llms continue to improve, safeguards against llm anchoring bias must also be researched and implemented. we are excited for future research that further investigates both the use of llms for such tasks, while also investigating strategies that can mitigate this cognitive bias.