LLM -in-the-loop: Leveraging Large Language Model for Thematic Analysis (2023)

Abstract: Thematic analysis (TA) has been widely used for analyzing qualitative data in many disciplines and fields. To ensure reliable analysis, the same piece of data is typically assigned to at least two human coders. Moreover, to produce meaningful and useful analysis, human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensive and time-consuming. Recently the emerging field of large language models (LLMs) research has shown that LLMs have the potential replicate human-like behavior in various tasks: in particular, LLMs outperform crowd workers on text-annotation tasks, suggesting an opportunity to leverage LLMs on TA. We propose a human–LLM collaboration framework (i.e., LLM-in-the-loop) to conduct TA with in-context learning (ICL). This framework provides the prompt to frame discussions with a LLM (e.g., GPT-3.5) to generate the final codebook for TA. We demonstrate the utility of this framework using survey datasets on the aspects of the music listening experience and the usage of a password manager. Results of the two case studies show that the proposed framework yields similar coding quality to that of human coders but reduces TA’s labor and time demands.

## background and related work given the substantial amount of free-text responses, it seems obvious that nlp techniques can be potential solutions to facilitate ta. existing work tends to rely on topic modeling to extract key topics that represent the data @xcite . another approach is clustering: for instance, @xcite extract phrases, calculate their sentiment scores, and group the phrases into themes based on their sentiment polarity. guetterman et al. (2018) cluster phrases by their wu-palmer similarity scores @xcite . models can be trained to learn how humans code @xcite . human-ai collaboration frameworks for ta have been proposed @xcite @xcite . qr and nlp researchers are now exploring the opportunity to leverage llms for ta. @xcite investigate the use of llms to support deductive coding: they combine gpt-3 with an expert-drafted codebook. @xcite propose a user-friendly interface for collaborative qualitative analysis, leveraging llms for initial code generation and to help in decision-making. paoli (2023) studies the potential usage of chatgpt for ta following the six ta phases proposed by @xcite . in the theme refinement phase, the author generates three versions of themes by modifying the temperature parameter, foot_3 after which the themes are reviewed by a hc (the author). coding can be divided into inductive coding and deductive coding. inductive coding is a bottom-up process of developing a codebook, and deductive coding is a top-down process in which all data is coded using the codebook. we complete both inductive and deductive coding via human-ai collaboration and showcase the feasibility of our approach using two existing open-access datasets. ## human-llm collaboration framework figure 1 illustrates the framework, which consists of four steps. ## conclusion we propose a framework for human-llm collaboration (i.e., llm-in-the-loop) for thematic analysis (ta). the result of two case studies suggests that our proposed collaboration framework performs well for ta. the human coder and machine coder exhibit almost perfect agreement (κ = 0.87 in music shuffle and κ = 0.81 in password manager), where the agreement reported by the dataset authors are κ = 0.66 for music shuffle and κ = 0.77 for password manager. we also address the llm input size limitation by using partial data for codebook development, with results that suggest this is a feasible approach.