Automating Qualitative Data Analysis with Large Language Models (2024)

Abstract: This PhD proposal aims to investigate ways of automating qualitative data analysis, specifically the thematic coding of texts. Despite existing methods vastly covered in literature, they mainly use Topic Modeling and other quantitative approaches which are far from resembling a human’s analysis outcome. This proposal examines the limitations of current research in the field. It proposes a novel methodology based on Large Language Models to tackle automated coding and make it as close as possible to the results of human researchers. This paper covers studies already done in this field and their limitations, existing software, the problem of duplicating the researcher bias, and the proposed methodology.

## introduction qualitative research is an important asset in various fields such as marketing, media studies, social science, psychology, and medical research @xcite @xcite . it stands out from quantitative methods in its ability to go deeper into research questions and capture individual experiences. however, it doesn't have the straightforward statistics or clear answers often found in quantitative research. this makes it harder to draw conclusions and prove hypotheses when dealing with a vast collection of unstructured text documents @xcite . the primary way to analyze data in qualitative research involves open coding, a process that requires meticulously reading through texts to pinpoint significant thoughts, ideas, attitudes, and topics @xcite . following this, axial coding helps identify how these codes interrelate and groups them into broader categories @xcite . this method is time-consuming, often stretching over weeks @xcite , as it demands intensive manual effort and professional expertise to analyze a large number of documents. given these challenges, there's a growing interest in automating or simplifying the text analysis process to make it less labor-intensive. while there has been some progress in automating the analysis of interview data, using techniques like topic modeling @xcite and wordnet hierarchies @xcite , these approaches mainly highlight keywords already present in the text. they don't generate the nuanced "ideas" and "thoughts" that come to mind upon reading it. therefore the main goal of this research is to automate the coding procedure of qualitative data (mainly interviews) to make the result of analysis as close as possible to human researchers' results. in this proposal, we explore existing approaches for analyzing interview data and suggest a new method for automating the full coding procedure. the aim is to develop a model that can analyze interviews minimizing the variability and biases that can be introduced by human researchers. future work will involve producing software that can assist organizations and researchers in managing and interpreting large volumes of textual data efficiently. ## related work before covering existing approaches and software dedicated to qualitative analysis, we need to explore how the coding is done by professional human coders. ## research proposal this phd proposal seeks to explore the ways of automating the analysis of quantitative data, mainly interview transcripts. the main goal is to test the proposed approach on different sets of interviews and compare it with expert coding. next, we outline the main research questions: it is important to note that the concept map described in previous sections is the last step after the ones mentioned above. as its nature lies in relationship extraction and information visualization, it is considered to be the next separate research topic. thus, it will not be covered in the proposed methodology for this proposal, though it was important to describe it as the concluding part of qualitative data analysis. the accomplishment of these research goals will help to systematically organize data, which can be massive and complex, into understandable and manageable themes. this enables researchers to identify patterns and insights that are not immediately apparent, thus adding depth to the research findings. exploring the domain knowledge of social scientists can significantly improve the accuracy of automated models. this knowledge can lead to the creation of algorithms that are more aligned with human cognition, which is especially important when analyzing nuanced human communication. while bias is typically something to be minimized, understanding it can be useful, particularly in developing ai that can replicate human-like understanding. it's important to recognize that complete objectivity is unattainable, and acknowledging bias allows for a more reflexive approach to data analysis. ## proposed approach the proposed methodology follows the cognitive process of a social scientist who typically keeps in mind the study's framework during thematic analysis. as described in section 2.1 on current coding practice, thematic analysis is a two-step process consisting of open and axial coding. the first step involves summarizing each citation's main idea/thought, and the second consists of categorizing all ideas into higher-order categories (fig. 1 ). according to existing manuals, the open coding phase doesn't involve preliminary knowledge of 5 the research topic (glaser and strauss, 2017), while axial coding heavily relies on it @xcite . the next sections describe how each stage of the coding process can be automated. ## dataset for the fine-tuning of the large language model (llm), we propose utilizing a curated dataset comprising coded interview citations collected from social scientists, both academic researchers and students doing qualitative research in social science. an illustrative example, as demonstrated in figure 4 , presents data in a citation-label format. for instance, a citation s i such as "well since it's a smartphone, it's usually always with me," would be associated with a label l i denoted as "mobility". a potential challenge is the limited size of the dataset. however, recent studies, such as those by @xcite , have shown that llms can still perform exceptionally well even when fine-tuned with a minimal dataset. 6 88 6 evaluation the testing of the model will be based on comparison with experts. we will employ krippendorff's alpha to evaluate the reliability and consistency of the coding provided by our model compared to expert coders. this statistical measure is ideal for assessing the agreement among multiple coders on qualitative data. our approach involves constructing a contingency table where each row represents an interview citation and each column a coder (our ai model and human experts). the codes assigned to each citation by different coders are filled in this table. we will then calculate the observed disagreement among coders for each item and sum these to get the total observed disagreement. expected disagreement, which is the disagreement expected by chance, will also be computed based on the distribution of each code. the alpha value is calculated using the formula α = 1 -observeddisagreement expecteddisagreement . a higher alpha value (close to 1) indicates a higher agreement among the coders, suggesting that our model's coding aligns well with human experts. this method will provide a robust quantitative measure of the coding reliability of our ai model in qualitative data analysis. we extend our testing framework by evaluating potential bias in the coding process. this involves comparing the coding consistency of our language model and a golden standard established by expert consensus. the example of the evaluation framework is illustrated in figures 5 6 . the golden standard codes emerge from a discussion process among human coders (c1, c2, c3) to identify the most appropriate codes. to effectively incorporate a bias evaluation, we propose to compute bias scores for each human coder, b(c1), b(c2), b(c3), to figure 6: comparing individual biases and llm ## limitations in this section, we will discuss the limitations of our current research as well as the challenges posed. one of the primary concerns is the issues associated with concept extraction and ontology building. accurately selecting codes is inherently challenging, as the process relies on nuanced human knowledge. it raises a question: how can we develop a model sophisticated enough to replicate these complex human cognitive tasks? another issue is that language is intricate. people have their unique way of speaking, and they often communicate more than what they explicitly say. our model strives to comprehend and code what people express, but it might not be able to do 7 so as accurately as humans. it may miss some of the implicit nuances in language that we naturally understand. one major concern is that our dataset is relatively small. since we are using a limited number of examples to fine-tune our model, there is a possibility that it may not perform as well as we expect. this could result in it being less effective in coding new interviews, as it hasn't had sufficient exposure during training. additionally, the extraction of relationships presents its own set of difficulties. training an algorithm to navigate them poses a significant challenge, further complicated by the diverse strategies individual coders reach a consensus. the question of whether there exists a universal approach or if coders are utilizing various, possibly conflicting, techniques is yet to be answered. furthermore, the unclear nature of large language models (llms) introduces additional complexity. these models often act as "black boxes," making it challenging to discern the rationale behind their outputs. this obscurity necessitates the exploration of explainable ai, a significant area of research aimed at making ai's decision-making processes more transparent. our project might encounter similar difficulties, interfering with our ability to fully understand and explain the model's behavior and decisions. ## ethics statement this research ensures data privacy by anonymizing all interview data and obtaining informed consent, in compliance with data protection regulations. while the goal is to enhance and assist human researchers, potential displacement effects are considered, striving to support rather than replace them. efforts are made to mitigate biases in the llms, maintain fairness, and ensure transparency in the models' decision-making processes. additionally, computational resources are optimized to minimize environmental impact. ## conclusion and future work in conclusion, this proposal outlines a comprehensive framework for automating the extraction of information from qualitative research. by using the advanced capabilities of large language models (llms) and integrating them with the expertise of social scientists, we aim to significantly reduce the time and effort required in the coding process. in this proposal we have addressed the potential for replicating the bias inherent in human coding, recognizing that this aspect of qualitative analysis can be both a challenge and an opportunity. by understanding and potentially simulating these biases, we can approach the human-like analytical capabilities that are currently the domain of experienced researchers. the use of a curated dataset for fine-tuning the llms, along with the development of an algorithmic framework will be the first step in constructing an actual tool that facilitates qualitative analysis. future work will focus on the practical implementation of the proposed methodologies, including the fine-tuning of llms with the constructed dataset and the validation of the coding process against standard qualitative analysis. additionally, we will explore the integration of multiple llms to simulate the collaborative nature of human coding teams. the end goal is the creation of user-friendly software that embodies the strengths of both manual and ai-assisted analysis, involving all stages of qualitative analysis from open coding to the construction of a concept map.