Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLM s (2024)

Abstract: The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closed-book level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput.

## introduction the emergence of chatgpt @xcite and gpt4 @xcite , along with other large language models (llms) @xcite has sparked a global sensation. the success of llms is closely tied to the long context capabilities of llms @xcite , especially in the field of multi-document question answering. however, the utilization of long context also introduces challenges such as higher inference cost, longer latency, and inferior perfor-mance caused by redundant information @xcite . many efforts have been made to compress the long context by directly removing a certain percentage of less important words, such as longllmlingua @xcite and selective-context @xcite . another common method is to generate a text summary of the given context @xcite . unlike deleting or reordering the word in the context, autocompressor @xcite compresses long documents into multiple vectors as soft prompts, which are optimized with full parameters of llms. however, our preliminary study shows that these methods have a common flaw: as the compression ratio increases, the compressed context fails to retain key information, resulting in a significant decrease in the performance of llms. the key to solve this problem is query, which defines what key information is. we aim to preserve this query-related key information even at a high compression ratio. specifically, we propose the query-guided compressor (qgc) to fully utilize query information throughout each compression step. we first feed the query and the documents together into a context encoder to learn the queryguide document representations. we then compress these document representations into n-gram representations guided by the importance of each word in relation to the query. subsequently, we propose to augment the n-gram representations by reviewing the query and document, which are finally aligned to the embedding space of the llms. we further propose dynamically adjusting the compression ratio of each document based on its relevance to the query. compared to previous methods, qgc has several advantages: 1) high compression ratios by retaining most query-related information during compression, 2) low training costs by optimizing the compressor only instead of finetuning the entire llm, and 3) better semantic consistency by com-pressing the n-gram structure rather than deleting words. we validate the effectiveness of qgc on the multi-document question answering task, including three datasets: naturalquestions, triviaqa, and hotpotqa. experimental results on the qa task indicate that, compared to longllmlingua, qgc exhibits a 2.75 times higher compression ratio and a 2.42 times higher throughput. additionally, its accuracy has improved by an average of 5 points. we further investigated the loss of key information throughout the compression process. the findings reveal that under high compression ratios and high noise conditions, qgc only incurs a performance loss of about 10%, while longllm-lingua suffers a loss of approximately 47%. this validates the effectiveness of qgc in retaining key information. ## preliminary study in this section, we first briefly formulate the long context compression on the question answering task, and then present an analysis on the key information loss in previous compression methods. ## query-guided compression as shown in figure 2 , we equip the llm with the query-guided compressor to compress long documents into a much shorter sequence of continuous representations, which are then concatenated with the corresponding instruction and query as the input for the llm. in the following, we first introduce the architecture of query-guided compressor and then its training objective. then, we propose a dynamic compression strategy that assigns higher compression ratios for irrelevant documents to further improve the compressed representations. ## experiments in this section, we conduct extensive experiments to investigate the effectiveness of qgc. ## analysis in this section, we conduct in-depth analyses to explore the performance of qgc in terms of key information loss, demonstration compression, detailed throughput and reranker impact. all analyses are conducted on naturalquestions with target llm as longchat-13b. key information loss in qgc as described in section 2.2, previous methods dramatically lose key information as the compression ratio increases. for comparison, we experiment with qgc using the same setting. compared to longllmlingua in figure 4 (a), the performance of qgc only decreases 10% as the compression ratio increases from 1x to 4x, and is even comparable to that of longllmlingua containing the correct answer in the compressed result. as seen in figure 4 (b), we observe that the performance of qgc slightly degrades with more documents, which is only a 12% decrease with 4 documents (27% for autocompressor). these results demonstrate that qgc can effectively retain key information even in much longer context and higher compression ratio scenarios. ## related work long context for llms recently, there have been a lot of studies focusing on expanding the context length of llms @xcite @xcite . existing efforts primarily involve gradually increasing the window size during pre-training @xcite , interpolating position embeddings @xcite , and modifying the attention mechanism @xcite . unlike these works, we do not directly aim to expand the context window of llms. hence, the qgc that we proposed can complement these techniques by enabling llms to access a broader context with reduced cost and shorter latency. retrieval-augmented lms combined with a standalone retriever to augment lms are gaining popularity for benefiting various knowledgeintensive tasks. previous studies have achieved remarkable results in improving perplexity @xcite , factual accuracy @xcite , downstream task performance @xcite , and in-context learning @xcite . besides, many works focus on cooperating llms and retrieved documents, such as reranking retrieved documents @xcite and discarding irrelevant documents @xcite . qgc is also a retrieval augmentation method for llms, which efficiently compresses the retrieved documents into shorter inputs while maintaining no significant performance degradation. ## conclusion and future work in this paper, we have presented a query-guided compressor qgc for llms to solve the loss of key information under high compression ratios. it consists of four essential components: query-guided context encoder, query-guided pooling layer, querydocument reviewing layer, and semantic alignment layer. in addition, we also propose a dynamically compressing strategy during inference. extensive experiments on multi-document qa tasks demonstrate that qgc outperforms previous state-of-theart compression methods in both accuracy and compression ratios. analyses reveal that this is primarily due to our retention of key information throughout the compression process. in the future, we aim to validate our approach on more advanced llms, while also expanding its application to additional tasks like document summarization. besides, we will try to further improve our approach by combining previous studies @xcite @xcite .