Open- RAG : Enhanced Retrieval Augmented Reasoning with Open-Source Large Language Models (2024)

Abstract: Retrieval Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs) by providing external evidence, but existing methods often suffer from limited reasoning capabilities (e.g., multi-hop complexities) in effectively using such evidence, particularly when using open-source LLMs. To mitigate this gap, in this paper, we introduce a novel framework, **Open-RAG**, designed to enhance reasoning capabilities in RAG with open-source LLMs. Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG uniquely trains the model to navigate challenging distractors that appear relevant but are misleading. By combining the constructive learning and architectural transformation, Open-RAG leverages latent learning, dynamically selecting relevant experts and integrating external knowledge effectively for more accurate and contextually relevant responses. Additionally, we propose a hybrid adaptive retrieval method to determine retrieval necessity and balance the trade-off between performance gain and inference speed. Experimental results show that Open-RAG outperforms state-of-the-art LLMs and RAG models in various knowledge-intensive tasks. Our method based on Llama2-7B sets new benchmarks, surpassing ChatGPT-RAG and Self-RAG. For example, in multi-hop HotpotQA, it achieves an EM score of 63.3, compared to RAG 2.0’s 54 and Command R+’s 60.

## introduction the rapid advancement of large language models (llms) has significantly improved various nlp tasks @xcite . however, these models often suffer from factual inaccuracies @xcite . retrieval-augmented generation (rag) has emerged as a promising approach to integrate llms with external knowledge, thereby improving generation accuracy @xcite . ## open-rag: enhanced retrieval-augmented reasoning open-rag transforms an arbitrary dense llm into a parameter-efficient sparse moe model capable not only of self-reflection but also of handling complex reasoning tasks. additionally, we devise an adaptive hybrid retrieval schema to balance the retrieval frequency and speed trade-off. below we first present the overview of open-rag and then discuss the training, including dataset and finetuning, and hybrid adaptive inference. ## results and analysis here, we (i) evaluate the rag models (ii) demonstrate the effectiveness of our adaptive retrieval in balancing the performance-speed (iii) present ablation studies and further analysis. ## related work complex factual reasoning requires contextualizing information from multiple documents @xcite . prior works @xcite @xcite proposed decomposing multi-hop queries into single-hop queries, then repeatedly using llms and retrievers. in addition, @xcite retrieved new documents if the tokens within generated sentences have low confidence. however, the performance improvement of these approaches often comes at the cost of resource-intensive techniques such as interleave chain-of-thought @xcite @xcite or tree-of-thought @xcite reasoning with document retrieval; and requiring external models @xcite . in this work, we train a single moe model capable of answering complex questions in one iteration with a minimal increase in model complexity. ## conclusion to enhance reasoning capabilities in rag models with open-source llms, we develop open-rag featuring a peft moe architecture, contrastive learning, and adaptive retrieval. open-rag shows significant performance improvements in complex reasoning tasks, outperforming sota methods. however, there is still a gap in tasks like long-form generation compared to proprietary models, which we aim to address in future.