S im RAG : Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains (2025)

Abstract: Retrieval-augmented generation (RAG) enhances the question answering (QA) abilities of large language models (LLMs) by integrating external knowledge. However, adapting general-purpose RAG systems to specialized fields such as science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To tackle this, we propose SimRAG, a self-training approach that equips LLMs with joint capabilities of question answering and question generation for domain adaptation. Our method first fine-tunes LLMs on instruction-following, question-answering, and search-related data. Then, it prompts LLMs to generate diverse domain-relevant questions from unlabeled corpora, with an additional filtering strategy to retain high-quality synthetic examples. By leveraging these synthetic examples, the LLMs can improve their performance on domain-specific RAG tasks. Experiments on 11 datasets across three different domains verify the efficacy of SimRAG over baselines by 1.2%–8.6%.

## introduction retrieval-augmented generation (rag) @xcite @xcite is a powerful technique that enhances large language models (llms) for various knowledge-intensive tasks such as question answering (qa) by incorporating external knowledge sources. this method not only customizes responses to handle long-tail knowledge but also avoids the need for costly model retraining @xcite . additionally, rag helps reduce the issue of llm hallucination by ensuring responses are grounded in relevant evidence @xcite , thereby improving the overall accuracy and reliability of llm outputs. ## related work retrieval-augmented generation. rag has emerged as a powerful tool in knowledge-intensive nlp tasks such as language modeling @xcite and question answering @xcite . the typical approach involves integrating a retriever with the llm generator and designing a fine-tuning process to align the retriever with llm capabilities. to further refine rag, recent research explored various enhancements. these include developing dynamical retrieval processes to refine the relevance of fetched content @xcite @xcite , and filtering out irrelevant contexts to robustify rag @xcite @xcite . additionally, several studies have developed instruction-tuning methods aimed specifically at improving search and rag capabilities of llms @xcite @xcite . self-training. self-training (or pseudo-labeling) is one of the earliest approaches to semi-supervised learning @xcite . the method uses a teacher model to generate new labels on which a student model is fitted. self-training has been widely adopted for various nlp tasks including text classification @xcite , natural language understanding @xcite and ranking @xcite . recently, the idea of selftraining has also been applied to llm instruction fine-tuning @xcite , reasoning @xcite , and alignment @xcite , yet to the best of our knowledge, this pipeline has not been widely explored for rag applications. the major drawback of self-training is that it is vulnerable to label noise @xcite . there are several approaches to stabilize the self-training, with sample selection @xcite and reweighting @xcite strategies. ## conclusion we introduce simrag, an instruction fine-tuning framework designed to enhance llms for domainspecific question-answering tasks. by equipping llms with joint capabilities for both question answering and question generation, simrag enables the generation of diverse, high-quality synthetic questions from unlabeled domain-relevant corpora. this approach facilitates effective adaptation to specialized fields, where distribution shifts and limited domain-specific data typically pose challenges. extensive experiments across 11 datasets in three domains show that simrag consistently outperforms baseline models, demonstrating its effectiveness in tackling the challenges of retrieval-augmented, domain-specific question-answering tasks.