S ir LLM : Streaming Infinite Retentive LLM (2024)

Abstract: As Large Language Models (LLMs) become increasingly prevalent in various domains, their ability to process inputs of any length and maintain a degree of memory becomes essential. However, the one-off input of overly long texts is limited, as studies have shown that when input lengths exceed the LLMs’ pre-trained text length, there is a dramatic decline in text generation capabilities. Moreover, simply extending the length of pre-training texts is impractical due to the difficulty in obtaining long text data and the substantial memory consumption costs this would entail for LLMs. Recent efforts have employed streaming inputs to alleviate the pressure of excessively long text inputs, but this approach can significantly impair the model’s long-term memory capabilities.Motivated by this challenge, we introduce Streaming Infinite Retentive LLM (SirLLM), which allows LLMs to maintain longer memory during infinite-length dialogues without the need for fine-tuning. SirLLM utilizes the Token Entropy metric and a memory decay mechanism to filter key phrases, endowing LLMs with both long-lasting and flexible memory. We designed three distinct tasks and constructed three datasets to measure the effectiveness of SirLLM from various angles: (1) DailyDialog; (2) Grocery Shopping; (3) Rock-Paper-Scissors. Our experimental results robustly demonstrate that SirLLM can achieve stable and significant improvements across different LLMs and tasks, compellingly proving its effectiveness. When having a coversation, “A sir could forget himself,” but SirLLM never does! Our code is publicly available at https://github.com/Zoeyyao27/SirLLMhttps://github.com/Zoeyyao27/SirLLM

## introduction the proliferation of large language models (llms) @xcite @xcite has spurred the development of various nlp applications @xcite @xcite @xcite , including widely-used tools like chatbots @xcite , writing assistants @xcite , and programming assistants @xcite . these applications, aiming to enhance user interaction and conversational experience, often require infinite input length and a certain degree of memory capability. however, current llms are usually pre-trained on texts of limited length, and studies have shown that their text generation capabilities dramatically decline when input lengths exceed those of the pre-training texts @xcite . merely extending the length of pre-training texts is impractical, as acquiring infinitely long text data is exceedingly challenging, not to mention that it would result in substantial memory consumption for llms. therefore, researching how to enable llms to handle infinite input lengths while maintaining memory capability is an urgent issue to be addressed. with the emergence of this demand, researchers have gradually shifted their focus towards exploring ways to expand the input context length of llms. a line of these studies has particularly focused on optimizing the attention mechanism of llms. @xcite first proposes the sliding-window attention, as shown in figure 1 (a) . by restricting each token to only attend to a certain number of recent tokens, this method reduces computational complexity. in deployment scenarios, llms utilize a key-value (kv) cache to store the key and value tensors of past tokens at each generation step to effectively reduces the need to recompute past key and value tensors, thereby siglayer 26 head 0 layer 32 head 0 layer 1 head 0 layer 6 head 0 figure 2: attention sink phenomenon @xcite . we visualize the average layer attention logits over 256 sentences, each with a length of 20, in vicuna-7b-v1.3. we can see that in the shallow layers, a significant amount of the attention score is dedicated to the first tokens and in the final layer, the model focuses more on the recent tokens. by remembering only the crucial information, the model might be able to maintain a longer memory span in the context of infinitely long conversations. in response to the aforementioned challenges, we propose the streaming infinite retentive llm (sirllm) in this paper, as illustrated in figure 1 (d). initially, we employ an llm to calculate the token entropy metric for each input token, thereby assessing their significance. subsequently, tokens with higher token entropy values, deemed as key tokens, are preserved within the kv cache. this method enhances the model's memory capabilities in the context of infinitely long streaming dialogues. to validate the effectiveness of sirllm, we conducted experiments across three distinct tasks: (1) dailydialog: we created a multi-turn daily dialogue dataset based on the dailydialog dataset @xcite . (2) grocery shopping: we developed a grocery shopping dataset. users first inform the llm about the groceries they need to purchase. following this, users engage in multiturn dialogues with the llm, culminating in the users asking the llm to recall the required groceries. (3) rock-paper-scissors: we constructed a rock-paper-scissors dataset featuring three types of players, each with a preference for one of the three moves (rock, paper, scissors). players engage in multiple rounds of rock-paper-scissors with the llm, which is tasked with analyzing the user's historical preferences to maximize its winning rate. the results of these experiments effectively demon-strate the enhanced memory capabilities of sirllm in infinite conversation. ## related work many works @xcite @xcite @xcite focused on expanding the input context length of llms by optimizing the attention mechanism. @xcite first proposes the sliding window attention, which let each token to only attend to a certain number of recent tokens. when the kv cache is full sliding window attention would discard the earliest token to preserve a stable decoding speed and performance. child et al. (2019) proposed the fixed sparse transformer. formally, this method initially preserves the key and value states of recent tokens as local context information. subsequently, it employs a column attention mechanism with a specified stride. this mechanism summarizes information from previous locations and propagates it to all future tokens, functioning as a form of global attention. @xcite proposed a logsparse self-attention where each element can only to attend to itself and its previous cells with an exponential step size. @xcite introduced the attention sink phenomenon and proposed streamllm, a model specifically designed to achieve true infinite input length. stream-llm, during its attention calculation, maintains the focus on both the initial tokens and the recent tokens. this approach ensures stable performance in the context of infinite streaming conversations. however, the aforementioned approaches either save tokens with given stride, randomly select, or do not preserve the key-value (kv) cache of history tokens, leading to significant forgetting issues in the model. sirllm addresses this by utilizing the llm itself to calculate token entropy, selectively preserving the kv cache of tokens with the highest entropy. this method effectively conserves memory space, ensuring that only the most crucial information is retained. another line of related work is kv cache optimization @xcite @xcite . @xcite introduced fast-gen, an adaptive kv cache compression method for large language models. fastgen begins by analyzing the behavior of various attention heads to select the most effective compression strategy for each and optimizes kv cache management when generating new tokens by applying the chosen com-pression strategy to each token, instead of merely appending new kv vectors. @xcite proposed h 2 o , a kv cache eviction policy that dynamically balances recent tokens and heavy hitters. the eviction policy is framed as a dynamic submodular problem, using attention scores to retain the most influential tokens in the kv cache. a greedy algorithm provides theoretical guarantees for nearoptimal performance. however, these works focus more on kv cache optimization rather than the streaming scenarios of multi-turn dialogues and enhancing the memory capabilities of llms. another category of work related to our research is context compression. @xcite compress the input context by selecting the lexical units (tokens, phrases, sentences) with higher selfinformation computed by a base language model. @xcite proposed a token filtering method for optimizing retrieved long documents to speed up the decoding process. this method involves using mean cross-attention scores computed at a specific layer across all attention heads to eliminate less critical tokens. then, only the top k% of input tokens with the highest scores are retained and used in predicting subsequent tokens. although retrieval-based methods can identify more accurate contexts based on input, they typically require greater computational and time resources. in contrast, sirllm does not necessitate maintaining an additional vector database and does not disrupt the model's end-to-end computational process. sirllm can significantly enhance the model's memory capabilities efficiently without modifying the model's architecture or requiring fine-tuning. @xcite proposed streamllm. they discovered that the model disproportionately focuses on initial tokens and break when removing initial tokens' kv cache. therefore, based on the slidingwindow attention, instead of throwing away all of the previous kv cache except the recent token's kv cache, they keep the first initial tokens kv cache as shown in figure 1 (c ). figure 1 (c ) illustrates the streamllm process, which can be formulated as follows. we define the indices of the attention sink tokens and the recent tokens as id sink and id recent , respectively: ## conclusion addressing the critical challenges of managing infinite input lengths and maintaining memory capability, sirllm harmonizes long dialogue retention without the necessity of model fine-tuning by selectively fortifies the model's focus on pivotal information. through experiments across three tailor-made tasks: dailydialog, grocery shopping, and rock-paper-scissors, sirllm has demonstrated a consistent and stable improvement over existing models, irrespective of the complexity and length of the dialogues. the experimental outcomes validate the robustness and versatility of sirllm, making it an invaluable asset for future explorations and applications in natural language processing.