The AI Co-Ethnographer: How Far Can Automation Take Qualitative Research? (2025)

Abstract: Qualitative research often involves labor-intensive processes that are difficult to scale while preserving analytical depth. This paper introduces The AI Co-Ethnographer (AICoE), a novel end-to-end pipeline developed for qualitative research and designed to move beyond the limitations of simply automating code assignments, offering a more integrated approach. AICoE organizes the entire process, encompassing open coding, code consolidation, code application, and even pattern discovery, leading to a comprehensive analysis of qualitative data.

## introduction qualitative data analysis is a crucial research approach in the humanities, cultural studies, and social sciences, focusing on the synchronic and diachronic analysis and interpretation of nonnumerical data such as texts, images, or audio files to gain insights into complex social phenomena, cultural expressions, and individual experiences @xcite . coding is central to this process, structuring and interpreting research materials such as interviews, field notes, or group discussions by systematically assigning analytically relevant concepts to text segments or other data forms @xcite @xcite . although coding offers a formalized structure for data analysis, its application remains contextspecific and flexible, adapting to the nuances of the research question and subject matter @xcite . in many contexts, specifically in ethnographic approaches, coding is inherently iterative and closely tied to an ongoing process of collecting and reflecting on data. codes evolve dynamically through an iterative process where they are merged, adjusted, added, or replaced as researchers engage with the data, identify patterns, and refine their conceptual understanding. this process may involve open or axial coding, deductively or inductively, quantitatively or qualitatively, and can be centered on interpretation or description. @xcite @xcite . however, manual coding faces significant limitations. scalability remains a critical challenge when researchers encounter larger datasets that require extensive time and resources to code effectively @xcite . it also increases the risk of intra-and intercoder unreliability, just to mention a few typical challenges. these constraints have spurred interdisciplinary efforts to automate the coding process over the past decade. automated speech recognition (asr) has emerged as a significant enabler in this landscape, allowing researchers to efficiently transcribe large volumes of interview data and prepare them for further analysis and processing @xcite . related qualitative data processing tasks such as text summarization @xcite @xcite , question answering @xcite , and topic segmentation @xcite have similarly benefited from computational advancements, providing researchers with tools to condense information and identify thematic boundaries. recently, large language models (llms) have demonstrated new epistemic capabilities to annotate research data, yet with certain limitations, such as understanding the broader context of codes @xcite @xcite . in parallel, the concept of agentic llms has emerged, designed to operate autonomously with goal-directed behaviors @xcite . for example, the ai scientist @xcite showcased an end-to-end automated workflow for writing scientific papers, from hypothesis generation, experimental design and manuscript drafting. this work illustrates the potential for autonomous agents to manage complex, multi-stage research processes. inspired by these advances, our approach seeks to explore similar automation in the domain of qualitative research, also as an alternative to ai-assisted data analysis with proprietary systems like maxqda. with the ai co-ethnographer (aicoe), we introduce a novel end-to-end pipeline that extends beyond the conventional focus on code assignments. the aicoe is part of a broader infrastructure for ai-assisted knowledge production, integrating diverse qualitative analysis methods, from open coding to pattern discovery. whereas prior research has largely concentrated on automating the mapping of codes to text segments, our approach encompasses a more comprehensive qualitative analysis process. the pipeline extends the capabilities beyond the deductive application of pre-defined codes. crucially, it also enables inductive code development and application, a process where novel codes are developed directly from the data itself instead of being pre-defined. ## related research llm development has spurred transdisciplinary efforts to automate scholarly work, especially qualitative textual analysis @xcite @xcite @xcite , including ethnographically focused research @xcite . this builds on a rich history of computational methods in qualitative research, from early tools like the general inquirer @xcite ) and salton's vector space model @xcite , to machine learning-based annotation @xcite , and open-source platforms like wordfreak @xcite and @xcite . more recently, @xcite introduced the qualitative code suggestion (qcs) task, which assists in coding by providing a ranked list of predefined codes for a given text passage. to evaluate qcs, the authors present cvdquoding, an annotated dataset of interviews with women at risk of cardiovascular disease. human evaluation shows that their system provides relevant suggestions, highlighting its potential as an assistive tool. however, limitations remain, including a focus on code assignment rather than full codebook development and a lack of evaluation in applied research settings. similarly, @xcite evaluated the potential of llms for automating social science tasks, focusing on their zero-shot capabilities. their find-ings indicate that llms demonstrate proficiency in both classification and explanation, suggesting their ability to augment the social science research pipeline. however, the authors do not recommend llms as a replacement for traditional methods. ## methodology the ai co-ethnographer is composed of a comprehensive pipeline underpinned by llms to automate key qualitative research processes while aiming to preserve the interpretative depth central to ethnography. building on recent advances in llms, the system mirrors several stages of qualitative analysis (see figure 1 ): open coding, code consolidation, code application, and pattern finding. this approach enables scalable and consistent analysis of large volumes of qualitative data while mimicking ethnographic research practices. ## experiments and results the system leverages llama-3.3-70b @xcite as llm, though the modular pipeline design permits integration with any modern llm. we evaluate the model on three interviews each from the cvdquoding and hiaics datasets, the latter being our collection of interviews conducted as part of an ethnographic analysis with ai researchers. the study participants include both researchers who apply ai practically in their scientific disciplines and those who offer theoretical and critical analyses of ai's use in research. the interviews were transcribed using the speaker-attributed asr system by nguyen and waibel (2025). foot_1 ## conclusion the ai co-ethnographer demonstrates both the potential and limitations of ai-supported qualitative research. our evaluation reveals robust codebook development, reasonable code assignments, and the ability to generate meaningful findings. this represents a promising direction for qualitative research, enabling the processing of large volumes of data while maintaining analytical depth. beyond functioning as a mere tool, aicoe serves as an epistemic medium in the research process.