Retrieval Augmented Generation or Long-Context LLM s? A Comprehensive Study and Hybrid Approach (2024)

Abstract: Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG’s significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

## introduction retrieval augmented generation (rag) has been shown to be a both effective and efficient approach for large language models (llms) to leverage external knowledge. rag retrieves relevant information based on the query and then prompts an llm to generate a response in the context of the retrieved information. this approach significantly expands llm's access to vast amounts of information at a minimal cost. however, recent llms like gemini and gpt-4 have demonstrated exceptional capabilities in understanding long contexts directly. for example, gemini 1.5 can process up to 1 million tokens @xcite . this prompts the need for a systematic comparison between long-context (lc) llms and rag: on one hand, rag conceptually acts as a prior, regularizing the attention of llms onto retrieved segments, thus avoiding the distraction of the irrelevant information and saving unnecessary attention computations; on the other hand, largescale pretraining may enable llms to develop even stronger long-context capabilities. therefore, we are motivated to compare rag and lc, evaluating both their performance and efficiency. in this work, we systematically benchmark rag and lc on various public datasets, gaining a comprehensive understanding of their pros and cons, and ultimately combining them to get the best of both worlds. different from findings in previous work @xcite , we find that lc consistently outperform rag in almost all settings (when resourced sufficiently). this demonstrates the superior progress of recent llms in long-context understanding. despite the suboptimal performance, rag remains relevant due to its significantly lower computational cost. in contrast to lc, rag significantly decreases the input length to llms, leading to re-duced costs, as llm api pricing is typically based on the number of input tokens. @xcite foot_0 . moreover, our analysis reveals that the predictions from lc and rag are identical for over 60% of queries. for these queries, rag can reduce cost without sacrificing performance. based on this observation, we propose self-route, a simple yet effective method that routes various queries to rag or lc based on model selfreflection. with self-route, we significantly reduce the cost while achieving overall performance comparable to lc. for example, the cost is reduced by 65% for gemini-1.5-pro and 39% for gpt-4o. fig. 1 shows the comparisons of lc, rag and self-route using three recent llms: gpt-4o, gpt-3.5-turbo and gemini-1.5-pro. in addition to quantitative evaluation, we provide a comprehensive analysis comparing rag and lc, including common failure patterns of rag, the trade-offs between cost and performance, and the results on additional synthetic datasets. our analysis serves as a starting point, inspiring future improvements of rag, and as a empirical guide for building longcontext applications using rag and lc. ## related work long-context llms. there has long been efforts for enabling llms to handle long contexts @xcite @xcite . while recent llms like gemini-1.5 @xcite , gpt-4 @xcite , claude-3 @xcite achieve significantly larger context window size, long-context prompting is still expensive due to the quadratic computation cost of transformers regarding to the input token numbers. recent work proposes methods to reduce cost by prompt compression @xcite , model distillation @xcite , or llm cascading @xcite . retrieval-augmented generation. augmenting llms with relevant information retrieved from various sources @xcite has been successful in complementing llms with external knowledge. rag achieves good performance on tasks like language modeling @xcite and qa @xcite , with a significantly lower computation cost @xcite . related to but different from our work, recently works augment rag with correction @xcite , critique @xcite , verification @xcite , or adaptive search @xcite @xcite to improve retrieval quality on knowledge-intensive tasks. ## conclusion this paper presents a comprehensive comparison of rag and lc, highlighting the trade-offs between performance and computational cost. while lc demonstrate superior performance in long-context understanding, rag remains a viable option due to its lower cost and advantages when the input considerably exceeds the model's context window size. our proposed method, which dynamically routes queries based on model self-reflection, effectively combines the strengths of both rag and lc, achieving comparable performance to lc at a significantly reduced cost. we believe our findings contribute valuable insights for the practical application of long-context llms and pave the way for future research in optimizing rag techniques.