Grounding Language Model with Chunking-Free In-Context Retrieval (2024)

Abstract: This paper presents a novel Chunking-Free In-Context (CFIC) retrieval approach, specifically tailored for Retrieval-Augmented Generation (RAG) systems. Traditional RAG systems often struggle with grounding responses using precise evidence text due to the challenges of processing lengthy documents and filtering out irrelevant content. Commonly employed solutions, such as document chunking and adapting language models to handle longer contexts, have their limitations. These methods either disrupt the semantic coherence of the text or fail to effectively address the issues of noise and inaccuracy in evidence retrieval.The CFIC approach addresses these challenges by circumventing the conventional chunking process. It utilizes the encoded hidden states of documents for in-context retrieval, employing auto-aggressive decoding to accurately identify the specific evidence text required for user queries, eliminating the need for chunking. CFIC is further enhanced by incorporating two innovative decoding strategies, namely Constrained Sentence Prefix Decoding and Skip Decoding. These strategies not only improve the efficiency of the retrieval process but also ensure that the fidelity of the generated grounding text evidence is maintained.Our evaluations of CFIC on a range of open question answering datasets demonstrate its superiority in retrieving relevant and accurate information, offering a significant improvement over traditional methods. By doing away with the need for document chunking, CFIC presents a more streamlined, effective, and efficient retrieval solution, making it a valuable advancement in the field of RAG systems.

## introduction recently, retrieval-augmented generation (rag) has marked a significant advancement in the field * * corresponding author. the generator's output should be grounded by precise evidence text obtained by the retriever. however, this poses challenges for most retrieval systems, as they often retrieve lengthy documents such as web pages. in practice, we only need specific grounding text from these documents to help answer user queries. using lengthy documents directly in the rag system presents two difficulties. first, generation models may struggle to handle the extensive length of these documents. second, irrelevant or distracting content within the documents can lead the model astray from the main query, resulting in inaccurate response generation @xcite . to address this issue, common approaches involve chunking documents into smaller passages and employing strategies like reranking for relevance @xcite @xcite , or selecting passages based on other measurements @xcite . however, the chunking process is often suboptimal, as determining the granularity of the passage chunking is challenging. improper chunking can disrupt the semantics and result in incomplete and incoherent retrieved information @xcite . another method involves adapting large language models (llms) to process longer contexts by training them on long contexts or implementing a sliding context window @xcite . while these methods enable llms to handle longer texts, they do not fully address the issue of noise in the lengthy documents and cannot output the grounding text for the generated response @xcite . in this paper, we propose a chunking-free in-context (cfic) retrieval approach aimed at helping the rag system mitigate information bias introduced by document chunking and irrelevant noisy text. specifically, given an input query and a long grounding document, instead of refining the long documents with a chunking-based method, we leverage the document's encoded hidden states to perform chunking-free in-context retrieval. it circumvents the traditional chunking process, allowing the retrieval system to auto-aggressively decode and pinpoint the precise evidence text to ground the response generation to a query. figure 1 shows the comparison between the chunking-based method and the chunking-free method for grounding text retrieval. the chunking-free method demonstrates a superior ability to identify optimal evidence text, as it considers the entire document for a comprehensive perspective. concretely, cfic involves encoding a document into transformer hidden states. when a user query is input, cfic continues to encode the query alongside task instructions following the hidden states, subsequently generating grounding text. in practice, we can cache the documents' hidden states to further reduce computation foot_0 . given the expectation for cfic to process lengthy documents, it becomes imperative to adapt cfic for handling long contexts. considering the trade-off between efficiency and effectiveness, in this paper, we adapt cfic to accommodate a 32k context, utilizing llama2-7b-chat as the foundational model. to achieve this, we construct a dataset containing long document, user query and precise text evidence to training the foundation model via supervised fine-tuning (sft). despite its promise, cfic encounters two major challenges: (1) efficiency issue: the autoaggressive generation process involves executing attention interactions for generating each new token, a procedure that becomes particularly timeconsuming with longer contexts due to the management of exponentially larger attention matrices. this process requires substantial computational resources @xcite , and (2) faithfulness issue: it is challenging to ensure the generation model's output remains faithful to the original input context, given its open-ended decision boundary @xcite . to address these, we propose two decoding strategies that accelerate inference and ensure that generated text evidence originates from the corpus. these include: (1) utilizing sentence prefixes as decoding candidates to shift the model's decision boundary from open-ended to document-dependent generation and (2) upon locating the appropriate sentence prefix, bypassing the decoding of intermediate tokens and directly selecting sentence ends with the highest likelihood of the [eos] token, thereby terminating the generation. furthermore, to retrieve multiple text spans as evidence, we sample several sentence prefixes with the best likelihood as candidates and rank them by sequence likelihood. by this means, cfic not only enhances the relevance and accuracy of retrieved evidence text but also preserves the semantic integrity of the information, effectively addressing major drawbacks of current retrieval systems. we tested cfif on the longbench tasks @xcite including: (1) single-document question answering with datasets like narrativeqa, qasper, mulitfieldqa, and (2) multi-document qa with datasets like musqus and hotpotqa. the experiment results verify the effectiveness of our method. in summary, our contributions are as follows: (1) we propose a chunking-free in-context retrieval method dedicated to the rag system, aiding in locating precise text evidence to answer user queries; (2) we propose the cfic model of which the ability to find text evidence from long context is enhanced via supervised fine-tuning with selfconstructed dataset; (3) we design two decoding strategies that significantly improve the efficiency and accuracy of the cfic's decoding process. ## related work the rag framework, initially introduced in the works of @xcite , aimed to enhance language models' capacity for generating knowledgebased responses @xcite . subsequent research primarily focus on refining the rag's two core components. on the retrieval front, significant strides have been made towards more efficient and precise retrieval methods @xcite @xcite @xcite . for example, the arise of dense passage retrieval significantly surpasses traditional sparse dense @xcite . parallel efforts on the generation side have concentrated on fine-tuning generative models to better harmonize with retrieved information, a notable example being the work of @xcite in optimizing external knowledge utilization @xcite @xcite . nevertheless, rag encounters specific challenges, especially in managing lengthy and complex retrieved documents. researchers, including @xcite , have developed chunking and reranking techniques to enhance passage relevance. furthermore, @xcite introduced methods for jointly learning retriever and generator models, thereby improving the coherence and relevance of outputs. addressing the issue of lengthy contexts in rag has involved either refining contexts @xcite or adapting generation models to handle extended contexts @xcite . recent advancements in rag predominantly incorporate large-scale language models (llms), such as gpt-3 and gpt-4, to augment language processing capabilities @xcite @xcite . the integration of llms has paved the way for more contextually rich and nuanced generation, especially in aligning generated responses with human preferences @xcite @xcite . in rag systems employing llms, the accuracy of retrieved textual evidence is crucial for reducing hallucinations and incorporating external knowledge @xcite @xcite . however, the challenge of processing long and noisy contexts persists @xcite @xcite . this paper introduces a chunkingfree in-context retrieval approach that leverages transformer hidden states to generate grounding text evidence, treating evidence retrieval as a generative process. this method represents a more streamlined and efficient retrieval solution for rag systems, marking a significant advancement over previous retrieval methodologies. ## conclusion this study introduces a chunking-free in-context (cfic) retrieval method for the rag system, addressing the challenges of processing lengthy documents and refining evidence retrieval. unlike traditional chunking-based methods that either compromise textual coherence or struggle with noise and inaccuracies, cfic leverages auto-aggressive decoding to pinpoint precise evidence directly, eliminating the reliance on chunking. cfic incorpo-query: what hedge fund's collapse in 1998 highlighted the need for regulation of derivatives? answer: long term capital management (ltcm) cfic-7b: in 1998, a trillion-dollar hedge fund called long term capital management (ltcm) was near collapse. using mathematical models to calculate debt risk, ltcm used derivatives to leverage @xmath01 trillion. the derivative transactions were not regulated, nor were investors able to evaluate ltcm's exposures.