You Only Use Reactive Attention Slice When Retrieving From Long Context (2025)

Abstract: Retrieval-Augmented Generation is a powerful method for enhancing language models (LMs), but existing retrieval techniques are limited.Embedding-based methods are often inaccurate due to their reliance on lexical similarity, while neural retrievers are computationally expensive to train.To overcome these issues, we introduce You Only Use Reactive Attention slice (YOURA), a training-free and fine-tuning-free attention-based retrieval technique. When retrieving, YOURA uses a novel reaction score heuristic, which quantifies how an LM’s self-attention “reacts” to a user query. We also propose a sentence extraction algorithm to efficiently preprocess the context.Evaluations on three open-source LMs using the LongBench and BABILong datasets show YOURA’s effectiveness. Our framework improves QA task accuracy by up to 15% and inference throughput by up to 31% compared to embedding-based retrieval.

## introduction language models (lms) are integral to natural language processing tasks, whereas a limited context window size remains a critical bottleneck for complex tasks. to address the issue, recent studies explored fine-tuning of pre-trained model @xcite , advanced attention @xcite , and retrieval-augmented generation (rag) techniques. however, fine-tuning requires extra computational power for each pre-trained model and is often tailored to individual tasks or datasets. advanced attention mechanisms conceptually extend the context window by selectively choosing a subsequence of tokens for attention, but risk dropping key information. ## related work limited context window size of language models (lms) is a well-acknowledged problem and various approaches such as longer context window sized models, novel attention mechanisms, finetuned models, context compression techniques, incontext learning, test-time learning, and retrieval-augmented generation (rag) have been proposed. in this section, we discuss retrieval related techniques, and list other techniques in appendix. retrieval-augmented generation. retrieval-augmented generation (rag) retrieves relevant context chunks from the preprocessed knowledge base and provides the following benefits: processes inputs longer than the language model's context window size, improves generation quality by excluding distractful information, and reduces computational cost with fewer input tokens to autoregressively generate from. prior works integrate rag with language models for question answering with long documents @xcite and in open-domain @xcite . furthermore, language models adopted retrieval at various stages such as pretraining @xcite @xcite , finetuning @xcite and inference stage @xcite . probabilistic retrieval framework. bm25, a probabilistic retrieval framework, was widely adopted as a relevance measurement of two texts @xcite . two texts yield high relevance if they have a high frequency of identical word sequences, with some adjustment to mitigate false positive scenarios such as a repetitive phrase or common phrase like "is a". ## you only use reactive attention slice (youra) traditional retrieval-augmented generation (rag) systems typically segment input into discrete chunks, assessing the relevance of each chunk to the query in isolation. this segmented approach can overlook inter-chunk dependencies, leading to potential context fragmentation and a loss of nuanced, cross-chunk relationships. in contrast, youra introduces a holistic method that evaluates contextual relevance across the entire input sequence. by computing relevance as a whole, youra preserves cross-chunk context, resulting in more accurate and contextually cohesive retrieval. this approach enables youra to prioritize relevant chunks not only by direct query similarity but also by their broader contextual alignment within the input, enhancing retrieval precision and relevance. design overview. figure 2 illustrates an overview of our design. the left side of the figure depicts how a reaction vector is calculated, while the right side shows how this vector is used for retrieval and how the language model then leverages the retrieved sentences to answer a query. ## experiments we evaluate how youra improves the answer quality of three pre-trained models on qa tasks, including single-document, multi-document, and the needle-in-a-haystack task. results for non-qa task are listed in appendix d.1 datasets. we evaluate youra on the singledocument qa, multi-document qa, summarization, few-shot learning, and synthetic task datasets ( ## conclusion we presented a novel attention-based sentence retrieval framework that leverages a holistic measurement of sentence-to-query relevance to enhance retrieval accuracy, generation quality, and inference throughput. our proposed heuristic combines strong theoretical foundations with practical effectiveness for sentence retrieval in long-context qa tasks. while the framework introduces a preprocessing overhead, we demonstrated how this cost can be amortized across multiple queries, which makes it an efficient and scalable solution.