Compressing Context to Enhance Inference Efficiency of Large Language Models (2023)

Abstract: Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM’s fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50% reduction in context cost, resulting in a 36% reduction in inference memory usage and a 32% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.

## introduction large language models (llms) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as real-life applications @xcite @xcite . however, a major challenge for existing llms is processing longer context. dealing with longer context with llms is fundamen-context: large languages models (llms) have shown their ability to perform new tasks, resulting in a line of work that focuses on further scaling these models. these efforts are based on the assumption {that more parameters will lead to better performance.} query: what's the assumption behind the efforts to further scale llms? llms: further scaling large language models will lead to better performance on a wide range of tasks. tal in scenarios such as having long conversations, document summarisation, and question answering given long documents. however, it is very computationally expensive, particularly with transformer based llms, due to the quadratic growth of memory and computation associated with the 2-d attention matrix @xcite . this makes llms less accessible and sometimes leads to context truncation during inference. moreover, due to the above limitation, existing llms were usually pre-trained with fixed-context windows, which further constrains their capability in processing longer context. there are active attempts in reducing the computation and memory cost of the transformer architecture with sparse attention @xcite or local dense attention @xcite . there are also efforts to learn soft prompts with further distillation to save context cost during inference @xcite . in contrast to existing approaches that primarily focus on architectures or distillations, we introduce a fresh perspective to tackle the redundancy in the input context itself, thus proposing a complementary, modelagnostic approach that can be potentially combined with other architecture optimisation methods to further enhance inference efficiency. the proposed method is motivated by the potential redundancy and repetition in human language, which has two main sources. the first is the inherent redundancy of natural language. for example, in the conversation "a: did you get the chance to pick up groceries today?", "b: yes, i did get the groceries.", the underlined part can be seen as a common redundancy in communication. linguistic studies suggest redundancy is ubiquitous in language @xcite . the other type of input redundancy is from the overlap with training material. as the example in fig. 1 shows, if some parts of input have already been included in the pre-training stage of llms, then it is safe to delete them and the model can still generate the correct answer. in summary, redundancy in the input context, while beneficial for human comprehension, can be superfluous for llms and might lead to unnecessary computational expense. in this paper, we propose selective context, which prunes redundant content in a given input context, thereby reducing the computational cost and making better use of the fixed context length in llms. selective context evaluates informativeness of lexical units (i.e., tokens, phrases, or sentences) with self-information @xcite computed by a base causal language model. by selectively retaining content with higher self-information, our method provides a more compact and efficient context representation for llms to process without compromising their performance on various applications. we evaluate the effectiveness and different settings of selective context on arxiv papers, bbc news, and real conversation on sharegpt.com with four nlp tasks: summarisation, question answering, original context reconstruction, and conversation. experimental results demonstrate that our proposed method can significantly enhance context efficiency of llms during inference while maintaining comparable performance compared to that achieved when full context is used. ## self-information self-information, also known as surprisal or information content, is a fundamental concept in information theory that quantifies the amount of information conveyed by an event given a distribution @xcite . in the context of language modelling, the event can be regarded as one step of generation (i.e., a token) and the distribution corresponds to its output distribution. so the selfinformation of a token can be defined as the negative log likelihood: in information theory, self-information measures the level of surprise or uncertainty associated with an event; rare events convey more information and thus have higher self-information, while common events convey less information and have lower self-information. in the context of language modelling, self-information can be used to assess the informativeness of lexical units, e.g., words, phrases, or sentences. lexical units with lower self-information are less informative and thus are more likely to be inferred from the context. as a result, we may treat these parts of input as redundant during llm inference. in nlp, self-information has been used to measure surprise in creative language artefacts @xcite . in addition, related concepts of self-information such as entropy and perplexity are widely used in language model optimisation and evaluation. this means we can calculate the self-information of a lexical unit by simply summing the selfinformation of the tokens in it. introduction continual learning ( cl ) , also known as lifelong learning , is a promising learning paradigm to design models that have to learn how to perform multiple tasks across different environments over their lifetime [to uniform the language and enhance the readability of the paper we adopt the unique term continual learning ( cl ) . ] . ideal cl models in the real world should be deal with domain shifts , researchers have recently started to sample tasks from two different datasets . for instance , proposed to train and evaluate a model on imagenet first and then challenge its performance on the places365 dataset . considers more scenarios , starting with imagenet or places365 , and then moving on to the voc/cub/scenes datasets . few works propose more advanced scenarios built on top of more than two datasets. filtered: introduction continual learning ( a promising learning paradigm to design models have to how across overto uniform the language and enhance adopt the unique term continual learning ideal cl models in should deal domain shifts researchers recently started sample tasks two different datasets for instance proposed to train and evaluate on imagenet first challenge places365considers more scenarios starting imagenet or places365 the voc/cub/scenes datasetsfew works propose more advanced scenarios built top more than two datasets we first 1) employ a causal language model such as gpt @xcite , opt @xcite , or llama @xcite , computing self-information for each token. we then 2) merge tokens, along with their corresponding self-information values, into lexical units, which can be phrases or sentences. this step is optional if tokens are being used as the basic units. finally, 3) we eliminate content that is deemed least necessary to render the input more compact. ## experiments the goal of selective context is to reduce the redundancy in the input context without compromising the generation quality of llms. as a result, we are expecting the answers given both selective context and the original context to be as close as possible. we take the generated answer given full context as the reference answer, and compare to the generated answer given the selective context in our experiments. ## results except for §5.5, all results of selective context presented are at the phrase level (the optimal). ## conclusion we introduced selective context to improve the context efficiency of llms in inference by deleting redundant content measured by self-information. our extensive experiments on arxiv papers, bbc news articles, and conversation transcripts showed that our proposed method can significantly reduce gpu memory cost, accelerate generation with minor performance decrease, and potentially enable llms to handle long documents and extended conversations without the risk of context truncation.