Paper: LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (ICLR-2024)

Reviewer comment (Evaluation):
The latency evaluation setup is not well presented. The latency of API calls may have a high variance. Are the experiments conducted several times, and in different hours? If not, the reviewer suggests improving this and update the evaluation setup.

Author response:
1. **"… The latency of API calls may have a high variance. Are the experiments conducted several times, and in different hours? If not, the reviewer suggests improving this and update the evaluation setup."**  
Thank you for your suggestion. We appreciate your concern about the variance in API call latency. We want to assure you that the latency reported in the paper is the result of multiple tests. However, to provide a more comprehensive latency comparison, we have added end-to-end latency results of different methods on Multi-document QA, LongBench, and ZeroSCROLLS in Table 1 and Table 2. Our results show that compared to the original prompts, LongLLMLingua can speedup the end-to-end system latency from 1.4x to 2.3x. We hope that these additional results will address your concerns.

Reviewer comment (Presentation):
The foundational work upon which this research is built, LLMLingua, should be appropriately cited from [https://arxiv.org/abs/2310.05736](https://arxiv.org/abs/2310.05736), where the authors are clearly identified. Citing it with "Anonymous" authors seems unusual.

Author response:
**"… LLMLingua... Citing it with Anonymous authors seems unusual."**  
Thank you for your suggestion, we have fixed this issue in the updated version.

Reviewer comment (Novelty):
It might be beneficial to consolidate the list of contributions into three primary ones. Several contributions, such as document reordering, information reduction loss ... etc (discussed in Sections 4.2-4.4), seem relatively minor individually. They might be more appropriately categorized as a set of techniques or "tricks" that collectively enhance the framework's performance and stability.

Author response:
**"It might be beneficial to consolidate the list of contributions into three primary ones."**  
Thank you for your suggestion, we will rewrite this section in subsequent versions.

Reviewer comment (Evaluation):
Figure 5 alone seems insufficient to establish the full effectiveness of the framework. To provide a more comprehensive understanding, Tables 1 and 3 should include E2E runtime data, particularly to offer perspective on the performance of retrieval-based methods.

Author response:
**"Figure 5 alone seems insufficient to establish the full effectiveness of the framework... Tables 1 and 3 should include E2E runtime data, particularly to offer perspective on the performance of retrieval-based methods."**  
Thanks for the insightful feedback. we have incorporated the end-to-end latency results of different methods on Multi-document QA, LongBench, and ZeroSCROLLS in Table 1 and Table 2. The results show that while LongLLMLingua has a slightly higher latency compared to retrieval-based methods like BM25, it still accelerates the end-to-end system latency by 1.4x-2.3x when compared with the original prompts.

Reviewer comment (Writing):
I believe there might be a typographical error: 'ground-true' should likely be corrected to "ground-truth."

Author response:
**"'Ground-true' should likely be corrected to 'ground-truth.'"**  
Thanks. We have corrected this in the updated version.

Reviewer comment (Experiments):
In the methods, it contains several aspects coarse compression, fine compression, reordering mechanism to reduce information loss in the middle, adaptive compression control and post-compression recovery strategy. It seems that it lacks a detailed ablation study that the influence on the compression rate and performance from each aspect. This is important for us to have a better understanding of this paper.

Author response:
Thanks for the helpful suggestions. We have incorporated your suggestions and added detailed ablation study on different datasets in the updated version (Multi-document QA: Table 3; LongBench: Table 6; MuSiQue: Table 5). In general, removing any module in LongLLMLingua such as question-aware coarse-to-fine compression, dynamic compression ratio, document reordering, and subsequence recovery yield performance improvement for LLMLingua across different benchmarks.

Reviewer comment (Theory):
Based on Figure 3b, it shows that contrastive perplexity is much more stable than the original perplexity metric. Would the authors provide more detailed explanation or mathematical proof on the reasons for this? It is a bit unclear for me to understand this difference.

Author response:
Thank you for the valuable feedback. We added a theoretical derivation of contrastive perplexity in Appendix B. The conclusion is that contrastive perplexity is positively correlated with $p(x^{\text{que}}|x_i, x_{<i})$, signifying the contribution of each token to question $x^{\text{que}}$. The larger the contrastive perplexity, the higher the $p(x^{\text{que}}|x_i, x_{<i})$, the greater the contribution of the token $x_i$, the higher its importance, and the more it should be retained. In addition, we included Figure 5 in Appendix D.1 to illustrate the distribution of document-level contrastive perplexity when the ground-true was placed at different positions (i.e., 1st, 5th, and 15th). It shows that even when key information is not at the beginning of the prompt, using contrastive perplexity can better capture token-level information related to the question.

Reviewer comment (Experiments):
In the benchmark comparison, it would be better to include the GPT-3.5-Turbo-0613 and LongChat-13B-16k for comparison. Although they might be slower and more expensive, it would be helpful to understand the cost from the compression.

Author response:
Thanks for your suggestion. We have added experimental results using GPT-3.5-Turbo-0613(Table 2) and LongChat-13b-16k(Appendix D.5 Table 7) on LongBench. It demonstrates that the compressed prompts achieve higher performance than the original prompts by 4.8 points.

Reviewer comment (Evaluation):
Removing documents and reordering is only relevant in the multi-document scenario where the relevant information is pertained in a small subset of the documents, and the rest are noise (e.g. in open-book question answering). Many use cases (e.g. long-input summarization and question answering, code understanding, multi-hop question answering etc.) will not benefit from it and may even be hurt by this procedure. Namely, in multi-hop question answering where one can determine the relevance of a downstream document only after answering the outer-part of the question, important documents may be filtered out with the procedure proposed in the paper.

Author response:
Thank you for pointing out the potential confusion. LongLLMLingua is designed as a general coarse-to-fine compression framework that allows us to adjust the granularity of compression based on the task at hand. When dealing with tasks such as long-input summarization where important information is evenly distributed throughout the original text, we can perform coarse-grained compression (i.e., “removing” in the question here) at the paragraph or sentence level rather than document level to improve the key information density. Since LLMs tend to be “lost-in-the-middle", reordering can help LLMs better capture the most important information and thus benefit the overall performance. We believe that the experimental results in long-input summarization (Table 6, LongBench summ.), single-/multi-doc question answering (Table 6, LongBench; Table 8, LooGLE), code understanding (Table 6, LongBench Code), multi-hop question answering (Table 5, MuSiQue), and Timeline reorder (Table 8, LooGLE) well demonstrate the effectiveness of LongLLMLingua. We have clarified this in the revised version.

Reviewer comment (Novelty):
Subsequence recovery is only relevant in very specific cases, namely where the task is extractive question answering. However, many real-world use cases that contain naturally occurring tasks over long inputs are not extractive but rather generative.

Author response:
Yes, the benefit of subsequence recovery in certain generative scenarios is a little bit limited than it in extractive tasks. However, in most cases, the generated content may refer a lot to texts in the prompt in terms of entities or phrases, and subsequence recovery can effectively mitigate the information loss here. We conduct ablation study of subsequence recovery on diverse tasks for your information. As shown in Table 6, removing subsequence recovery would lead to a performance drop in Long Bench tasks “SingleDoc (QA)”, “muiltiDoc (QA)”, “Summ. (summarization)”, and “FewShot”.

Reviewer comment (Novelty):
The question-aware compression is only relevant when a question (or more generally, a long and specific instruction) is given. In use cases such as summarization or conditional generation this is not the case.

Author response:
The question-aware module, both coarse-grained and fine-grained, is designed for aware the changing of information density within the prompt. While it is crucial for tasks that involve following specific instructions (e.g., removing question-awareness would lead to a performance drop up to 35.1 points on NaturalQuestions as in Table 3), it can also be beneficial for summarization tasks and other tasks including multi-hop QA task. For example, Table 6 shows that in the LongBench summarization task, dropping question-awareness does lead to performance drop ~1-2points. And Table 5 shows that in the MuSiQue multi-hop QA, dropping question-awareness does lead to performance drop ~9 points. This suggests that question-awareness such as “please summarize this article” also provide valuable global context for the SLM, both coarse-grained and fine-grained, and have an implicit influence on the perplexity distribution during compression.

Reviewer comment (Evaluation):
One of the main pieces of evidence in the paper is the evaluation on ZeroSCROLLS. This evaluation is unsatisfactory because the paper mentions they use the evaluation set of the benchmark, while Shaham et al. 2023 mentions explicitly that the evaluation set contains “a small number of examples (~20 per task) in a ‘validation’ split, meant for eyeballing purposes’’ and does not have enough statistical power to be used as statistical measurement. Additionally, the authors do not provide a breakdown of the results to show that their method is indeed beneficial across different scenarios.

Author response:
Thanks for the suggestions. We use the validate set of ZeroSCROLLS for evaluation because the ground-true test set is inaccessible. We've added the breakdown results of ZeroSCROLLS in Appendix D.2 (Table 4). We further evaluate LongLLMLingua on MuSiQue (multi-hop QA, ~4.8k examples, Table 5) and LooGLE (long dependency QA, ~1.1k examples, Table 8). Experimental results show that compared with the original prompts, LongLLMlingua compressed prompts achieve comparable results on ZeroSCROLLS and outperform the original prompts by a large margin on MuSiQue and LooGLE, well validating its effectiveness in a wide range of scenarios.

Reviewer comment (Experiments):
The paper considers several changes as significant contributions, but there are no ablation studies to show their usefulness. Specifically, in §2 they mention the usage of \( x^{doc} \| x^{que} \| x^{restrict} \) to compute the relevance of each document as an important contribution. An ablation on the ordering of the question and document should be added, as well as for the relevance of the restricting prompt.

Author response:
Thank you for your valuable suggestions. We have incorporated your suggestions and included an ablation study of reordering and restrict prompt on different datasets in the updated version (Multi-document QA: Table 3; LongBench: Table 6; MuSiQue: Table 5). In general, both reordering and the restrict prompt yield improvements for LongLLMLingua across different benchmarks.

Reviewer comment (Experiments):
All ablation studies that were performed were done on the NaturalQuestions dataset, which is explicitly tailored for the proposed contribution. In this dataset, one document contains the relevant information while the others are distractors. The reviewer would like to see the ablation study conducted on multi-document scenarios where this is not the case, such as multi-hop question answering scenarios (e.g. MuSiQue [Trivedi et al., 2021] which also appears in ZeroSCROLLS).

Author response:
Thank you for your valuable suggestions. We have incorporated your suggestions and included an ablation study of reordering and restrict prompt on different datasets in the updated version (Multi-document QA: Table 3; LongBench: Table 6; MuSiQue: Table 5). In general, both reordering and the restrict prompt yield improvements for LongLLMLingua across different benchmarks.

Reviewer comment (Experiments):
The authors mention using a small LM to perform all pre-computation and compressions needed. However, they use LLaMA-2-7B-chat as their small LM, which has significant overhead in itself and may not be widely applicable to many use-cases. An ablation study on the performance of using smaller models should be added.

Author response:
Thanks for your feedback. we've added results with GPT2-small (using GPT2-dolly as the small LM) to Table 3, Table 5, and Table 6. The results show that GPT2-small compressed prompts can derive comparable even higher performance than the original prompts.

Reviewer comment (Presentation):
In Page 8 (§5), the indexing of the two tables is reversed. Namely, Table 3 appears before Table 2.

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Writing):
“Less cost” is used twice instead of “lower cost” (in the abstract and the main results paragraph in §5).

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Writing):
In the abstract, one of the sentences starts with “. and experimental …” where it should have been “. Experimental …”.

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Writing):
“Derive costs” is used several times (including in the abstract) instead of “Drive costs”.

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Writing):
In §2 the phrase “ground-true” is used instead of “ground-truth”.

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Presentation):
The styling guide indicates that third-level headings (namely paragraph titles) should be in small caps, and not capitalized as appearing in the paper. See https://iclr.cc/Conferences/2023/CallForPapers for style information.

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Writing):
In §4.3, the number of documents is denoted with \( N_d \) while it was already denoted as \( K \) beforehand.

Author response:
Thanks for your thorough review. We have fixed these issues in the updated version.

Reviewer comment (Evaluation):
The inference latency can be improved by LongLLMLingua. However, the models need to evaluate the perplexity and compress prompts every time, which leads to non-trivial latency.

Author response:
Thank you for pointing out the potential confusion. Though LongLLMLingua evaluates perplexities every time which may cause some latency, the compression is done by a much smaller language model Llama-7B. Comparing to the latency caused by LLM inferencing on the compressed prompt, LongLLMLingua can still speedup the end-to-end system latency from 1.4x as shown in Table 1 to 2.3x as shown in Table 2 (Figure 5 in previous version).

Reviewer comment (Reproducibility):
How is the thresholding of $s_i$ determined to discard tokens of lower importance? Is it determined by the current budget $\tau_k$ of the documents?

Author response:
Yes, we discard tokens of lower importance, i.e., with $s_i$ smaller than the threshold, and the threshold is determined by the current budget $\tau_k$ of the documents.

Reviewer comment (Reproducibility):
Section 4.3 introduces a dynamic budget scheduler. It is unclear to me how the iteration is defined here. Is there an iterative evaluation of token importance?

Author response:
Thanks for the comments. We evaluate token importance in segment-level by iteratively conditioned on all previous compressed segments and calculate document-level threshold based on Eq. 5. We will add clarification in the final version.

Reviewer comment (Evaluation):
I am interested in the perplexity distribution within sentences. If the perplexity varies dramatically across tokens of one sentence, it can happen that a few tokens are retained sparsely, making the sentence inconsistent.

Author response:
Thanks for the question. Token perplexities do vary a lot within one sentence. Though removing tokens can render the sentence inconsistent and unreadable to human, it doesn’t appear to affect the ability of language models to comprehend the prompt. This phenomenon has been studied in Figure 4 of LLMLingua paper[1], which demonstrates that GPT-4 is capable of recovering the original details from the compressed, non-human-friendly prompt.