Exploring Large Language Models for Qualitative Data Analysis (2024)

Abstract: This paper explores the potential of Large Language Models (LLMs) to enhance qualitative data analysis (QDA) workflows within the open-source QDA platform developed at our university. We identify several opportunities within a typical QDA workflow where AI assistance can boost researcher productivity and translate these opportunities into corresponding NLP tasks: document classification, information extraction, span classification, and text generation. A benchmark tailored to these QDA activities is constructed, utilizing English and German datasets that align with relevant use cases. Focusing on efficiency and accessibility, we evaluate the performance of three prominent open-source LLMs - Llama 3.1, Gemma 2, and Mistral NeMo - on this benchmark. Our findings reveal the promise of LLM integration for streamlining QDA workflows, particularly for English-language projects. Consequently, we have implemented the LLM Assistant as an opt-in feature within our platform and report the implementation details. With this, we hope to further democratize access to AI capabilities for qualitative data analysis.

## introduction the discourse analysis tool suite @xcite is a platform developed at our university to empower digital humanities (dh) researchers in conducting qualitative data analysis (qda). developed collaboratively and tailored to the specific needs of dh scholars, the platform democratizes access to machine learning methods, enabling nonexperts to manage and analyze large-scale, unstructured, multi-modal data effectively. while the platform's overarching design is rooted in grounded theory-based research @xcite , its versatile features support various disciplines. within the core qda workflow on our platform, researchers engage in a dynamic process of structuring their data and conducting in-depth qualitative analysis. this involves organizing and categorizing documents through metadata assignment and creating a tag taxonomy, which is utilized for classifying documents. simultaneously, they dive into the nuances of the material, developing hierarchical code taxonomies to annotate relevant text passages and capturing their insights through memos. while tasks like metadata extraction and document classification can be repetitive and timeconsuming, qualitative analysis tasks such as annotation, paraphrasing, and summarization are demanding. the potential for ai assistance to streamline and enhance these diverse workflows motivates our exploration of large language models (llms). hence, this work aims to assess how effectively llms can support users in qda tasks and ultimately boost their efficiency and productivity. to this end, we identify four core nlp tasks embedded in our qda platform's core functionalities: 1) document classification, 2) document information extraction, 3) span classification, and 4) text generation. we then curate datasets that closely align with real-world use cases regarding domain and tasks, focusing on english and german. subsequently, we evaluate the performance of three state-of-theart open-source llms, llama 3.1, gemma 2, and mistral nemo, on this benchmark. our findings show the promising potential of llm integration within the discourse analysis tool suite (dats), particularly for english projects. consequently, we implement the llm assistant as an opt-in feature for english projects, paving the way for further enhancements and expansions. contributions of this paper are: catma @xcite is a versatile qda tool focusing on text and image analysis. it currently lacks built-in ai capabilities. known for its comprehensive approach to qualitative and mixed-methods research, maxqda foot_0 has introduced "maxqda ai assist," offering aidriven features like summarization, paraphrasing, and concept explanation. a robust platform for qualitative data analysis, nvivo's foot_1 latest beta version is actively integrating ai functionalities, including thematic coding, sentiment analysis, and text summarization. atlas.ti foot_2 is recognized for its visual and networkbased analysis tools. the platform incorporates ai with existing features like code suggestions, sentiment analysis, summarization, and entity recognition powered by openai's gpt models. notably, ai-powered features within these qda platforms are currently only found in paid versions. they are realized by sending data to third-party providers, potentially leading to data protection issues. in contrast, our open-source qda platform aims to democratize access to state-of-the-art ai capabilities, making advanced functionalities freely available to researchers across disciplines. it can be run in-house if required. llm benchmarks while several prominent general llm benchmarks like mmlu @xcite , superglue @xcite , bigbench @xcite , helm @xcite , and mteb @xcite exist, they often lack a direct connection to specific real-world applications, including qualitative data analysis. they may cover a broad range of tasks but not necessarily those most relevant to qda workflows. in contrast, @xcite explores the potential of llms to transform computational social science (css) by evaluating their zero-shot performance on a range of english css tasks. their extensive evaluation, focused on taxonomic labeling and free-form coding, highlights llms' potential to augment css research as zeroshot data annotators, strongly motivating our work. still, many benchmarks prioritize english data. our benchmark is constructed to be relevant to the tasks and data encountered in qda, incorporating datasets that approximate real-world scenarios within our platform. focusing on document classification, information extraction, span classification, and text generation in english and german, we aim to gather insights that can directly inform the effective integration and utilization of llms in qualitative research. llms and qda @xcite explores the potential of llms to serve as data analysts in qualitative research within software engineering. their approach employs a multi-agent model where each llm agent performs specific researchrelated tasks, such as interpreting textual data and interview transcripts, to automate common qualitative analysis processes. their findings suggest that llms can significantly accelerate data analysis, allowing researchers to handle larger datasets efficiently, which further motivates this work. ## envisioned workflow this section presents an illustrative excerpt of a qualitative data analysis workflow inspired by project partners who actively utilize the discourse analysis tool suite. this scenario highlights potential areas where ai-powered automation enhances productivity. imagine alice, a researcher aiming to analyze local companies across various industries, focusing on their societal impact and challenges. she initiates semi-structured interviews with ceos, stating the current date, introducing herself, and then inquiring about company details (e.g., size, sales volume) and the interviewee's background (e.g., name, age, position) before asking her research questions. she records these interviews with her smartphone. after collecting data, alice starts the qualitative analysis process within our qda platform. she creates a new project, defines document tags for industry categorization, and establishes metadata fields like "interview date," "company size," and "partner name" to capture crucial interview details. upon uploading her recorded interviews, our platform currently utilizes whisper @xcite to generate automatic transcripts. after the automatic pre-processing, alice is presented with interview transcripts, which are now automatically tagged by industry and partially populated with metadata. the ai-powered system suggested tags and metadata values based on the interview content, which alice verifies and completes with the help of an intuitive ui. having an organized document collection, alice starts the qualitative annotation. she constructs a code system aligned with her research questions that incorporates codes like "social impact" and fine-grained sub-codes for "problems". next, she activates the auto-coding feature, and the aipowered system suggests relevant text annotations. an interface allows her to review these suggestions. while reviewing, alice notices occasional disfluencies like repetitions and filler words, typical of verbatim transcriptions. she selects a disfluent passage, activates the paraphrasing feature, and is presented with an ai-generated suggestion. after minor edits, she approves the improved version. similarly, she employs the automatic summarization feature to condense lengthy answers to her interview questions for improved clarity. equipped with such ai-powered tools, alice efficiently processes her remaining transcripts and utilizes our platform's analytical features to answer her research questions. ## integrating llms in our qda platform user feedback during early testing led us to refine our envisioned workflow outlined in section 3. the implementation of summarization and fluency correction aligned with our original plan. however, users wanted to access document-based assistance features (document tagging, metadata extraction, annotation) at any point in their workflow, not just during the initial document import. consequently, we redesign the llm assistant as a standalone feature independent of the pre-processing pipeline. the feature is initiated by selecting the documents for analysis and clicking the llm assistant 2. focus: users specify which tags, metadata fields, or codes the llm should consider. here, the user chooses to keep existing work, overwrite it with the llm suggestions, or merge both per document or in batches. user control and transparency are key requirements throughout the design of the llm assistant. our philosophy is to ensure that any form of automation is a supportive tool, offering suggestions while the user retains decision-making. we firmly believe that ai should augment human expertise, not replace it. thus, we've designed our platform to require explicit user approval for automaticallygenerated suggestions. our implementation achieves user control and transparency in two ways. the prompt review step allows users to inspect and modify the generated prompts, fostering transparency. further, it enables advanced users to exert fine-grained control and provide additional task-specific instructions. in the result view, we prioritize transparency by displaying the llm's reasoning and suggestions. additionally, it allows users to critically evaluate the suggestions and decide how to incorporate them with their existing work, ensuring that the final output aligns with their intentions. the llm assistant is built using react for the frontend and ollama, fastapi, and celery for the backend. celery handles background job processing, ensuring that llm tasks run without interrupting user workflow. ollama hosts the gemma 2 model, which performed best in our benchmark. we reuse the benchmark prompts, as they were intentionally designed with future implementation in mind. we opt for the template-filling prompt style instead of extractive question-answering for metadata extraction. providing metadata descriptions is more intuitive for users than formulating fitting questions for each field. ## conclusion in this paper, we investigated the potential of llms to enhance qualitative data analysis workflows, focusing on common tasks within our opensource discourse analysis tool suite. we designed a benchmark reflecting real-world use cases and evaluated the performance of three prominent open-source llms. our findings demonstrate the promise of llm integration, particularly for english-language projects. consequently, we implemented the llm assistant within our platform, a significant step towards empowering researchers with transparent and user-controlled ai assistance that augments, rather than replaces, human expertise. in future work, we plan to extend the llm assistant to suggest new tags and codes, fostering a more exploratory qda process. furthermore, we aim to incorporate more domain-specific datasets that closely align with dh researchers' real-world use cases. finally, we will explore few-shot learning approaches to enhance performance on nuanced tasks. user activities within our platform (e.g., tagging documents and annotating text passages) generate valuable training data for model fine-tuning, potentially leading to more efficient models tailored to individual user preferences. code for replicating the benchmark foot_5 , the repository of dats foot_6 and a live demo are available foot_7 .