LCIRC : A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLM s (2025)

Abstract: While large language models (LLMs) excel in generating coherent and contextually rich outputs, their capacity to efficiently handle long-form contexts is limited by fixed-length position embeddings. Additionally, the computational cost of processing long sequences increases quadratically, making it challenging to extend context length. To address these challenges, we propose Long-form Context Injection with Recurrent Compression (LCIRC), a method that enables the efficient processing long-form sequences beyond the model’s length limit through recurrent compression without retraining the entire model. We further introduce query dependent context modeling, which selectively compresses query-relevant information, ensuring that the model retains the most pertinent content. Our empirical results demonstrate that Query Dependent LCIRC (QD-LCIRC) significantly improves LLM’s ability to manage extended contexts, making it well-suited for tasks that require both comprehensive context understanding and query relevance.

## introduction large language models (llms) have demonstrated remarkable capabilities in generating coherent and contextually rich outputs @xcite . however, a major limitation remains in their ability to handle long-form contexts efficiently @xcite . transformer-based architectures, which underlie most llms, are constrained by fixed-length position embeddings, limiting the number of tokens they can process in a single pass @xcite . once this limit is exceeded, crucial information from earlier parts of the input is truncated, leading to reduced performance in tasks † co-corresponding authors. to address these challenges, we propose longform context injection with recurrent compression (lcirc), a method designed to extend llm's ability to process long-form inputs efficiently. lcirc compresses the context beyond the model's length limit into compact representations, which are injected back into the model, allowing it to retain essential information without retraining the entire model. by recurrently compressing the input sequence, our approach maintains a balance between preserving context and minimizing computational overhead, enabling the model to generate outputs grounded in long-span contexts @xcite @xcite . in many real-world applications, llms need to process input in response to specific queries or instructions @xcite . compressing all available information indiscriminately can lead to irrelevant data being retained, reducing the model's effectiveness @xcite . to address this, we introduce query dependent context modeling, which selectively compresses information based on its relevance to the query. this ensures that the model focuses on the most pertinent parts of the input, enhancing its performance in tasks like multiple choice and long-form question answering, where query relevance is crucial. our results show that lcirc, combined with query dependent modeling, significantly improves the ability of llms to handle long-form contexts in a scalable and efficient manner, making it wellsuited for applications that require both extensive context understanding and precise query relevance. in summary, our primary contributions are as follows: (1) we propose lcirc, a method that extends the context window of llms through recurrent compression, allowing efficient handling of long-form inputs without retraining the entire model. (2) we introduce query dependent context modeling, which selectively compresses queryrelevant information, enhancing performance in tasks that require the comprehensive understanding of extended contexts. (3) our approach significantly improves the efficiency of llms in longform contexts, reducing computational costs while demonstrating performance improvements quantitatively through various benchmarks. ## efficient long-form context modeling in this section, we present an efficient approach to handle long-form inputs in pretrained llms. we introduce a method that enables models to process lengthy contexts by compressing and injecting the relevant information back into the model in a computationally efficient manner. additionally, we outline the training strategy used to optimize the proposed components while maintaining the foundational capabilities of the model. ## query dependent context modeling llms frequently process context based on specific instructions or queries. to enhance the ability of llms to handle long-form context, we extend our method to incorporate query dependent context compression. this allows the model to selectively focus on the context most relevant to the given query, thus improving the overall efficiency and relevance of the model's responses. unlike the vanilla recurrent context compression, which merges all the information in the current segment s i into the compressed features h (i) as shown in eq. ( 3 ), query dependent compression selectively injects information that is most relevant to the user query. this selective compression is achieved through the addition of a gated cross attention block to our recurrent context compression method as shown in figure 2 . as illustrated in figure 2 , the model integrates the user query into the compression pipeline. at each compression step, the learnable query vectors or the previously compressed features h (i-1) used as the query features for the perceiver module are transformed into a query dependent representation. this is done through the newly introduced gated cross attention block, which takes h (i-1) as the query features, and the user query embedding e query as the input features. the query dependent ## conclusion we propose long-form context injection with recurrent compression (lcirc) to address challenges llms face with extended inputs. lcirc efficiently compresses long-form contexts, expanding context length while reducing computational overhead. by incorporating query dependent context modeling, it selectively retains relevant information, improving performance in tasks requiring long-context comprehension and query relevance. our experiments demonstrate significant advancements in both areas. future work will focus on extending lcirc to multilingual settings to integrate context across diverse languages and cultures.