Measuring What Matters: Evaluating Ensemble LLM s with Label Refinement in Inductive Coding (2025)

Abstract: Inductive coding traditionally relies on labor-intensive human efforts, who are prone to inconsistencies and individual biases. Although large language models (LLMs) offer promising automation capabilities, their standalone use often results in inconsistent outputs, limiting their reliability. In this work, we propose a framework that combines ensemble methods with code refinement methodology to address these challenges. Our approach integrates multiple smaller LLMs, fine-tuned via Low-Rank Adaptation (LoRA), and employs a moderator-based mechanism to simulate human consensus. To address the limitations of metrics like ROUGE and BERTScore, we introduce a composite evaluation metric that combines code conciseness and contextual similarity. The validity of this metric is confirmed through correlation analysis with human expert ratings. Results demonstrate that smaller ensemble models with refined outputs consistently outperform other ensembles, individual models, and even large-scale LLMs like GPT-4. Our evidence suggests that smaller ensemble models significantly outperform larger standalone language models, pointing out the risk of relying solely on a single large model for qualitative analysis.

## introduction inductive coding is a fundamental method in social science research, enabling the identification, organization, and analysis of patterns and themes within textual data @xcite @xcite . at its core, this method extracts key ideas from text, transforming unstructured documents into a list of codes that capture the underlying themes. traditionally, this process has relied on human coders who, despite their expertise, are prone to inconsistencies, cognitive biases, and the tendency to overcomplicate labels @xcite @xcite . large language models (llms) offer an opportunity to automate this process, potentially improving efficiency and standardization @xcite . however, ensuring the reliability and interpretability of llm-generated codes remains a challenge. our study addresses two critical gaps in automated inductive coding: (1) the need for structured refinement processes to improve consistency across codes and (2) the limitations of standard evaluation metrics, which fail to capture the nuanced criteria used by human experts, such as label granularity, alignment, and contextual relevance @xcite @xcite @xcite . we introduce a novel ensemble-based framework that integrates refinement processes inspired by retrieval-augmented generation (rag) @xcite , reusing previously generated codes to eliminate their redundancy iteratively. this study makes three key contributions: (1) we develop a pipeline that integrates multiple instruction-finetuned llms with an iterative refinement process to improve coding consistency; the results indicate that the proposed ensemble approach improves the quality of the generated codes, while the refinement process further enhances their consistency, resulting in outputs that are more closely aligned with the human-coded data. additionally, we show that ensemble-based methods reduce variability in code assignments, addressing the subjectivity present in individual llm outputs. these findings provide a strong basis for improving automated qualitative data analysis while maintaining compatibility with human coding standards. ## background qualitative data analysis (qda) plays a critical role in social science research @xcite , enabling researchers to identify, categorize and interpret patterns within textual data. central to this process is the concept of coding, where meaningful segments of text are assigned concise labels, or codes, that capture their core essence. according to @xcite , a code is "often a word or short phrase that symbolically assigns a summative, salient, essence-capturing and/or evocative attribute to a portion of languagebased or visual data." in thematic analysis, one of the most widely used methods in qda, these codes are further grouped into broader categories to reveal hierarchical relationships and underlying themes within the data @xcite . the source material often includes interviews, documents, or other qualitative data formats, and the process involves summarizing the key ideas in each segment and grouping them into overarching themes. recent advances in natural language processing have introduced the use of llms for automating qualitative coding tasks @xcite @xcite . however, we noticed 2 critical points that need to be addressed in this domain. (1) to evaluate codes, some papers noted that traditional summarization metrics like bert and rouge are not good enough for this task @xcite , one research specifically created unsupervised metric to assess codes @xcite . (2) the second critical point in this domain is that although individual llms demonstrate remarkable performance, their output often varies due to differences in training data, architectures, and model parameters @xcite . this variability introduces subjectivity, making a single llm's coding decisions as inconsistent as those of individual human annotators. to address these challenges, ensemble methods, techniques that combine multiple models, have been increasingly explored for their ability to take advantage of the complementary strengths of different models and improve performance @xcite . ensemble learning is a widely used strategy to improve model performance by combining the strengths of multiple models, often referred to as "weaker models" @xcite . there are two common approaches to ensemble learning: one involves weighting individual models based on their performance, while the other focuses on aggregating diverse outputs to produce a unified result. one example is the mixof-experts (moe) framework @xcite , which employs specialized sub-models to make predictions and merges their outputs for improved accuracy. similarly, llm-blender @xcite demonstrates the potential of ensembling by combining ranked outputs from multiple models to achieve superior performance in complex natural language generation tasks. this study builds upon the concept of ensemble methods but diverges from existing approaches by adopting a moderator-based framework. unlike fusion techniques that combine outputs probabilistically, our approach incorporates a final decisionmaking model tasked with selecting the best candidate or proposing a novel output. this design reflects the dynamics of human collaboration, where consensus is sometimes driven by a leader or a final arbiter, rather than by averaging or blending opinions @xcite . using this moderator model, we aim to mimic the style of the hierarchical decision-making process in human qualitative coding and demonstrate its effectiveness. ## dataset our experiments used a dataset of 1,000 code-quote pairs compiled from two main sources: social science research studies and the semeval-2014 task 4 dataset @xcite for augmentation. the social science data includes 600 examples from studies across three universities, featuring topics such as technology interaction, social values, and cultural experiences. the semeval dataset contributes an additional 400 examples, consisting of reviews manually coded by qualitative researchers. to evaluate the ensemble models, we used a second distinct test data set comprising 100 user reviews of chatgpt sourced from kaggle @xcite . this data set, characterized by user-generated unstructured feedback, differs from the structured data used for training, providing an opportunity to test model performance on unseen, real-world content. each review was manually annotated by human coders to create a golden standard reference for evaluation. this dataset was chosen specifically for its diversity in style of language, sentiment, and varied quote lengths to verify the ensemble's ability to handle varied text types. ## pipeline our proposed pipeline consists of two key stages, as illustrated in figure 1 . first, an input x is processed independently by three smaller llms (7b and 8b parameter sizes). the outputs from these models are then evaluated by a set of n moderators (moderator 1 , moderator 2 , . . . , moderator n ), which refine and consolidate the results. finally, code merging is performed to ensure consistency across similar inputs, producing the optimal output. ## metrics to evaluate the performance of individual models that will serve as input to the moderator, two metrics were employed to capture both lexical and semantic similarity: rouge @xcite and bertscore @xcite . bertscore is a metric that computes the similarity between bert token embeddings of two codes, which helps assess the meaning in the generated output compared to the reference. rouge is a lexical similarity measure that calculates the overlap of n-grams (1unigram overlap, 2-bigram overlap, l-longest common subsequence) between the generated text and the reference text. rouge is particularly effective in summarization task @xcite , which is valuable when the exact wording of the output needs to match the reference. ## results the ensemble pipeline with codes merging approach yield significant performance improvements, as detailed in ensembles perform better, with the mixtral8x7b ensemble achieving a significant composite score improvement (0.33→0.91) over its individual counterpart. llama3.3 70b shows a similar trend, improving from 0.38 to 0.50 through ensemble integration. merging of codes provides additional benefits, elevating mixtral8x7b to 0.99 (+9%) and llama3.3 70b to 0.74 (+48% from individual baseline). these improvements demonstrate both the value of model diversity and postprocessing effectiveness in enhancing coding consistency. ## discussion in this section, we discuss the implications of our findings regarding the application of ensemble llms in automated inductive coding. our results highlight key advantages of ensemble-based approaches over individual models, particularly in terms of conciseness and the ability to balance abstraction and specificity in qualitative coding. coder golden standard llama3.3 70b ensemble gpt4o ensemble gpt4 ensemble mixtral 8x7b ensemble quote code assigned "i am not very happy with this because chatgpt answers the questions i ask but it stops midway. then even after continuous efforts it does not give complete answer. this is not very correct. please fix it so that there is no more trouble. this is a very wrong thing, after some use you are asked to upgrade which is very expensive." technical problems while using unsatisfied with chat-gpt user dissatisfaction with incomplete answers unsatisfied with chat-gpt's incomplete responses user: dissatisfied with expensive, incomplete chatgpt answers "this is a great app, the bot is so accurate to anything, it gives me tips in gaming, studies, and life, etc. this app is also helpful if you were having problems in things! also, there are some problems about the server glitch, but don't worry, some glitches are just seconds, overall a great app. note: the gpt-4o is more powerful, and accurate than the regular one." helpful app accurate and helpful app helpful and accurate app experience user satisfaction and app efficiency accurate, helpful tipproviding app "the best so far" best app best so far the best experience yet user satisfaction and app effectiveness top user rating "excellent app for learning" excellent for learning excellent learning app excellent learning tool user satisfaction and app efficiency excellent learning app "thanks for making my life easier" thankfulness life made easier gratitude for increased ease user satisfaction feedback gratitude for ease ## conclusion in this work, we explored the use of ensemblebased llms with refined outputs for automated inductive coding, demonstrating that ensemble approaches consistently outperform individual models in qualitative coding tasks. our results highlight three key findings: (1) ensemble methods improve coding consistency, reducing variance and increasing alignment with human-coded benchmarks; (2) code merging postprocessing enhances coding stability, ensuring greater coherence in label assignments across similar inputs; and (3) smaller ensemble models outperform standalone high-capacity models, offering a more efficient and scalable alternative to traditional llm-based approaches. these findings highlight the potential of ensemble approaches and refinement to advance automated qualitative analysis while maintaining alignment with human coding patterns.