F rag R el: Exploiting Fragment-level Relations in the External Memory of Large Language Models (2024)

Abstract: To process contexts with unlimited length using Large Language Models (LLMs), recent studies explore hierarchically managing the long text. Only several text fragments are taken from the external memory and passed into the temporary working memory, i.e., LLM’s context window. However, existing approaches isolatedly handle the text fragments without considering their structural connections, thereby suffering limited capability on texts with intensive inter-relations, e.g., coherent stories and code repositories. This work attempts to resolve this by exploiting the fragment-level relations in external memory. First, we formulate the fragment-level relations and present several instantiations for different text types. Next, we introduce a relation-aware fragment assessment criteria upon previous independent fragment assessment. Finally, we present the fragment-connected Hierarchical Memory based LLM. We validate the benefits of involving these relations on long story understanding, repository-level code generation, and long-term chatting.

## introduction the limited context window length constrains applications of large language models (llms) in some practical scenarios, such as answering questions based on complete books or movie scripts @xcite , writing codes within complete github repositories @xcite , etc. to resolve this problem, some works @xcite @xcite attempt to expand the context length of classical llm inference framework via continual training or sparse attention mechanism. however, existing approaches are either limited to a finite expansion length @xcite , or prone to performance degradation, especially when dealing with very long contexts @xcite . recent studies @xcite @xcite explore to hierarchically process the long text. each time only partial fragments of long text are retrieved from external memory and fed into llm's context window, a.k.a, temporary working memory, thereby eliminating the context length constraint and alleviating the inferior influence of substantial irrelevant content. however, current external memory managers simply split the complete long context into independent fragments, assessing their isolated importance during retrieval. this constrains the inference capability of hierarchical memory based llms, particularly in scenarios (e.g. understanding coherent story or code repository) where there are intensive associations across fragments of long text. to address this issue, we propose to integrate these fragment-level relations into the external memory management by introducing a relationaware fragments assessment score during retrieval. first, we formulate the relations between two fragments as a real number, with higher values corresponding to stronger relation strength. the calculation of relation quantity could have different instantiations for different context types (e.g. narrative stories, code repositories, or historical dialog) and different relation types (e.g. semantic relations or structural relations). next, based on the isolated relevance scores used in past external memory retrievers, we further calculate every fragment's environmental relevance score, which is defined as a normalized relation-weighted summation of other fragments' independent scores. during retrieval, the combination of independent score and environmental score is employed for assessing every fragment's importance. an adjustable coefficient is introduced to control the influence of environmental score. finally, in the same as previous works @xcite , we concatenate the retrieved content and the requested instruction as llm inference input. extensive experiments validate the benefits of incorporating these fragment-level relations during retrieval. the experiments encompass a variety of base llms (llama2 @xcite , chat-gpt, chatglm @xcite , etc. ), different temporary context lengths (1k, 4k, 8k, 20k, etc. ), and multiple long context scenarios (long story understanding @xcite , repositorylevel code completion @xcite , and long-term chatting with human @xcite ). ## related work temporarily long context processing. one line of works explores how to process long context under the typical llm inference framework, in which the complete context is directly stored in the llm context window (i.e. temporary memory). among them, a series of works @xcite @xcite @xcite study extending the context length of transformer-based models via efficient attention mechanism and recurrent inference strategic. in addition, some works @xcite investigate how to compress the length of long text content to mitigate the impact of excessive irrelevant text. although the long text processing capability of the large language models can be enhanced by expanding the context window or compressing the context content, the context length that can be handled remains limited. external memory augmented llms. another line of works introduce an additional external memory, forming a hierarchical memory based inference framework, thereby processing context of any length. in this framework, only partial content fragments are retrieved from external memory for knowledge updating @xcite or answering knowledge-intensive questions @xcite @xcite @xcite . the most popular retrieval method is first calculating the text embedding similarity for isolated fragments of external context using pre-trained embedding models @xcite , and then retrieving the text fragments with higher similarity to the requested question or current temporary context. benefiting from the zero-shot generalization capability of llms, the retrieved fragments can be directly concatenated with instructions as model input @xcite , eliminating the need for additional training. this further facilitates more flexible external memory augmented llm inference frameworks. some studies explore the collaborative use of retrieval and generation @xcite , as well as further multi-round retrieval-generation interleaving framework @xcite @xcite @xcite . @xcite utilizes explicit prompts about the structure of external context for enhanced retrieval. additionally, memgpt @xcite automatically reads and writes the external memory, enabling more flexible external memory reasoning. ## experiment we evaluate the proposed fragment-connected hieramem-llm on three long text understanding tasks: long story understanding (section. 4.1), repository-level code generation (section. 4.2), and memory-enhanced chatting agent (section. 4.3).