RAG -Studio: Towards In-Domain Adaptation of Retrieval Augmented Generation Through Self-Alignment (2024)

Abstract: Retrieval-Augmented Generation (RAG) has proven to be an effective paradigm for enhancing the quality of text generation by integrating large language models (LLMs) with external knowledge. However, an off-the-shelf RAG system, which relies on generally pre-trained LLMs and retrievers, often falls short in specialized domains and applications. In this paper, we introduce RAG-Studio, an efficient self-aligned training framework to adapt general RAG models to specific domains solely through synthetic data, eliminating the need for expensive human-labeled in-domain data. RAG-Studio accepts a specialized domain corpus, a general LLM, and a general retriever, then autonomously generates contrastive training data for both the LLM and retriever through self-alignment. We fine-tune them to work cohesively as an integrated and effective domain-specific RAG system, where the LLM is adapted to incorporate new domain knowledge and become robust to noisy contexts, and the retriever learns to better align with the LLM’s preferences, providing more useful information and minimizing the risk of misleading the LLM. Extensive experiments across diverse in-domain question-answering datasets spanning the biomedical, finance, law, and computing domains, show that RAG-Studio attains state-of-the-art performance, consistently outperforming the use of human-annotated data for fine-tuning.

## introduction retrieval-augmented generation (rag) is widely received as a popular paradigm for the application of large language models (llms). by integrating a standing-by retrieval component, llms can leverage authoritative and up-to-date information, thus improving the truthfulness and credibility of their generated outputs @xcite . additionally, frameworks like langchain 1 and llamaindex 2 have significantly simplified the development and deployment of rag systems by providing well-packaged components and automatically selecting the optimal combinations of generators and retrievers. however, in specialized domains such as medical, finance, law, or private enterprise databases, existing off-the-shelf rag systems, which utilize general llms and retrievers trained on generic data, often underperform. this underperformance arises because both the llm and the retriever lack the updated, domain-specific knowledge and deep understanding required to handle in-domain data effectively. furthermore, the retriever and llm do not learn to interact optimally as a unified rag system tailored for specialized contexts. as a result, achieving good performance often requires very intricate prompting, which can be cumbersome in practical applications. in contrast, fine-tuning has been shown to be an effective method for unlocking the potential of rag systems in specific in-domain tasks @xcite . however, in many specific domains or tasks, collecting and scaling the necessary training data for rag (e.g., in-domain question-answer pairs and their relevant or irrelevant passages) can be challenging. to address this issue, we propose rag-studio, a self-aligned training framework for the domain adaptation of rag systems based entirely on synthetic data. all training ingredients in rag-studio are autonomously synthesized through selfalignment @xcite without the need for external datasets or models, significantly enhancing the rag system's ability to adapt to specialized domains with minimal manual intervention. our method starts with a target domain corpus, a general retriever, and a general llm generator. initially, the llm is prompted to produce a collection of synthetic question-answer pairs based on sampled passages from the corpus. subsequently, the retriever is employed to gather additional contextual passages relevant to these questions. the llm then engages in a self-curation process, evaluating, refining, and filtering its own outputs to generate a set of high-quality, informative preference training samples. for the retriever, training signals are derived from the llm's self-feedback, which is elicited through a chain-of-thought @xcite prompting designed to assess the helpfulness of the retrieved contexts @xcite . through our fine-tuning process, both the generator and the retriever are optimized to function as a cohesive, domain-specific rag system. the llm is adapted to integrate new domain knowledge effectively and to withstand noisy contexts provided by the retriever. concurrently, the retriever is trained to align more closely with the llm's preferences, ensuring the provision of helpful information and reducing the likelihood of misleading the llm. we conduct extensive experiments across five indomain question answering datasets, encompassing the fields of biomedical, finance, law, computing, and general wikipedia. the results demonstrate that rag-studio consistently outperforms baselines on these benchmarks. notably, it outperforms the use of human-annotated data for fine-tuning by 5.1% and 3.2% in terms of automated metrics and gpt-4 evaluation scores, respectively. in summary, our main contributions are: the work most closely related to ours is raft @xcite , an in-domain rag fine-tuning method. however, raft primarily focuses on fine-tuning the generator using raw llmgenerated pseudo questions and answers within noisy contexts. in contrast, our approach involves unified rag fine-tuning that includes both the generator and the retriever. we implement a holistic self-alignment process, utilizing self-generated raw data and self-curation to produce higher-quality contrastive training data, thereby offering a more comprehensive training framework. self-alignment self-alignment refers to using a model to improve its own performance and align its responses with desired behaviors @xcite @xcite . this emerging approach shows promise in addressing data scarcity issues and achieving higher-level alignment in large model training. current research explores enhancing various aspects of llms with self-alignment, such as coding @xcite , mathematics @xcite , and general conversational helpfulness @xcite . studies on self-evolved data curation encompass instruction tuning @xcite , reward modeling @xcite , self-generation of explanations @xcite , critiques @xcite , and so on. our work explores self-alignment to support fine-tuning a general retriever and llm into a unified rag system, addressing data scarcity issues in domain-specific adaptations of rag. ## conclusion in this paper, we introduced rag-studio, a selfaligned training framework for domain adaptation of rag systems entirely based on synthetic data. our method eliminates the need for external data or models, enabling efficient fine-tuning of both retriever and generator components. through extensive experiments across various specialized do-mains, rag-studio consistently demonstrated superior performance, highlighting its effectiveness and practicality. we believe our work will advance the development and deployment of domainspecific rag systems, making them more accessible and robust for specialized applications.