Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents (2025)

Abstract: The deployment of Large Language Models (LLMs) in customer support is constrained by hallucination—generating false information—and the high cost of proprietary models. To address these challenges, we propose a retrieval-augmented question-answering (QA) pipeline and explore how to balance human input and automation. Using a dataset of questions about a Samsung Smart TV user manual, we demonstrate that synthetic data generated by LLMs outperforms crowdsourced data in reducing hallucination in finetuned models. We also compare self-training (fine-tuning models on their own outputs) and knowledge distillation (fine-tuning on stronger models’ outputs, e.g., GPT-4o), and find that self-training achieves comparable hallucination reduction. We conjecture that this surprising finding can be attributed to increased exposure bias issues in the knowledge distillation case and support this conjecture with post hoc analysis. We also improve robustness to unanswerable questions and retrieval failures with contextualized “I don’t know” responses. These findings show that scalable, cost-efficient QA systems can be built using synthetic data and self-training with open-source models, reducing reliance on proprietary tools or costly human annotations.

## introduction while many companies are eager to integrate large language models (llms) into customer service and other applications, widespread deployment remains constrained by hallucination, or the generation of false or unsupported information, and the high financial and computational costs of using proprietary models. this issue is particularly critical in customer support, where unreliable responses can mislead users and erode trust. we develop a cost-effective retrieval-augmented question-answering (qa) pipeline (see figure 1 ) and address critical training data questions: what sources of data are most effective for finetuning open source models, and what preprocessing or filtering mechanisms best mitigate hallucination. to do so, we use a dataset from @xcite comprising crowdsourced questions written by professional annotators about a samsung smart tv user manual (but notably lacking human-written responses). in this work, we address the following research questions: rq1: what is the optimal balance between manual and automated methods for data processing and creation? we explore the trade-offs of using automatic and manual methods in two main situations: data processing and data creation. we use llama-3-8b-instruct (hereafter llama-3) @xcite to generate answers to the crowdsourced questions, followed by two cleaning methods: manual cleaning performed by the first author and automatic cleaning using llms. while many recent studies have shown llm's ability to iteratively evaluate and refine text to reduce hallucination @xcite , these methods are often costly and pose data privacy risks when proprietary models are used at runtime. to address this, we compare the effort of manual cleaning with the effectiveness of closedsource (gpt-4o) and open-source (llama-3) models for data cleaning. we show that while gpt-4o significantly outperforms llama-3 in cleaning quality, it is comparable to manual efforts, suggesting that manual input may not always be necessary. we also explore a realistic scenario in which no training data is available. perhaps surprisingly, we demonstrate that llm-generated synthetic training data leads to lower hallucination rates than crowdsourced data, as measured by factscore and human evaluation, possibly due to increased variability in human-written questions. rq2: how does self-training compare to model distillation in terms of hallucination rates? we examine the benefits of synthetic data by comparing two training approaches: finetuning models on data generated by the same model (self-training with llama-3) versus finetuning models on data generated by a stronger model (knowledge distillation using gpt-4o). @xcite suggest that knowledge distillation reduces hallucination, but their study only tests on synthetic questions. meanwhile, @xcite and @xcite show that self-training can reduce hallucination, though without any human evaluation and with a train/test time mismatch in the case of @xcite . to our knowledge, our work is the first apples-to-apples comparison of these two approaches. surprisingly, we find that selftraining of a small model and distillation of a large one achieve comparably low hallucination rates, as measured by @xcite and human evaluation, when the same data cleaning is used for both methods. to explore this result, we analyze the potential role of exposure bias, which refers to the tendency of a model to perform better in contexts observed during training, leading to errors when faced with unfamiliar contexts during inference. we hypothesize that models trained on their own generated data benefit from greater familiarity with the training examples, compensating for the quality gap between the models. this suggests that self-training can serve as a resource-efficient alternative to model distillation in tasks where minimizing hallucination is critical. rq3: how can retrieval failures and unanswerable questions be anticipated? the dataset includes questions scraped from community forums such as amazon product qa sections, which are noisier, more diverse, and often unanswerable using the user manual. such questions are prone to hallucination as the model relies on pretraining rather than the provided document. since stateof-the-art retrieval models return n-best lists with imperfect accuracy @xcite , it is critical for qa systems to recognize retrieval failures and respond appropriately (e.g., i don't know the answer) while confirming the user's question was understood. while we do not focus on retrieval, we mitigate this issue by inserting negative examples during training, teaching models to provide contextualized "i don't know" responses, which also reduces hallucination rates. in light of these questions, this paper makes the following key contributions, with a focus on customer support systems: ## related work recent studies suggest that finetuning on new, unfamiliar knowledge can lead to hallucination @xcite @xcite . for instance, lin et al. (2024) propose training on self-generated data to reduce hallucination, but introduce a training-test mismatch where models use grounding documents during training but not testing, potentially causing hallucinations. we maintain consistent setups. like lin et al. (2024), zhang et al. ( 2024 ) employ self-training to reduce hallucinations. our approach differs in three ways: first, we use simple supervised finetuning (sft) instead of techniques like reinforcement learning (rl) and direct preference optimization (dpo), which are promising avenues for future work. second, we compare selftraining with knowledge distillation, investigating the value of synthetic data from a model's own outputs and from a more performant model. third, we validate our results with human evaluation in addition to automatic metrics. other works also focus on iterative self-refinement @xcite , though do not specifically focus on the problem of hallucination. in contrast, lewis and white (2023) employ knowledge distillation to reduce hallucination, using chatgpt to generate and clean documentgrounded training data. however, their approach is limited in two ways: they finetune a t5-large model @xcite , which reduces hallucination over gpt-3.5 but limits robustness and fluency, and they evaluate only on synthetic data. farquhar et al. (2024) detect hallucinations during inference using semantic entropy, which clusters generated outputs based on semantic equivalence and measures uncertainty at the level of meaning. while semantic entropy excels at runtime detection in open-domain settings, the entailmentbased clustering method is very expensive. by contrast, our approach reduces hallucinations at their source by improving training processes for rag settings. ## discussion our findings demonstrate that self-training and knowledge distillation can be comparably effective in reducing hallucination, while self-training is much less costly. models trained on self-generated data consistently performed as well or better than those trained on gpt-generated data, supporting the hypothesis that exposure bias plays a key role in finetuning effectiveness. additionally, our best blend vs. worst blend analysis revealed that using high-perplexity examples at training time led to increased hallucination, reinforcing the importance of training on familiar, low-perplexity data. further improvements were observed with synth llama+, where incorporating simple, context-aware negative examples yielded higher factual accuracy, suggesting promising future directions for hallucination mitigation. while our experiments focus on a single domain, the underlying mechanisms behind exposure bias and synthetic data effectiveness are likely to generalize to other qa tasks. applying this approach in domains such as medical or legal qa would provide a valuable test of its robustness and effectiveness in higher-stakes applications. future work should explore scaling synthetic data generation, refining data selection methods based on perplexity differences, and investigating iterative self-training approaches, where models continuously refine their own synthetic data over multiple training cycles. this could further enhance model alignment and factuality while reducing reliance on external supervision. ## conclusion in this work, we explore the trade-offs between cost, manual effort, and performance in building a qa agent for customer service, with a focus on mitigating hallucination. we elucidate the components of this process that can be automated and what models are best for that automation. we find that models finetuned on synthetic datasets can outperform ones from crowdsourced datasets, and that self-training with data validation not only matches the performance of knowledge distillation but can rival the original model being distilled (gpt-4o). our findings suggest that using this approach, scalable and cost-effective qa systems can be rapidly developed for customer service applications, delivering performance comparable to or exceeding that of current state-of-the-art models.