Paper: RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation (ICLR-2024)

Reviewer comment (Experiments):
The proposed method utilizes the knowledge of the LMs to train the compressor, while the baselines do not. Some baseline models, such as DRP and contriver, should be finetuned using the dataset employed to train the extractive compressor.

Author response:
[W1] Regarding baseline methods for extractive compressor, we considered off-the-shelf passage rankers such as DPR and contriever. Our method finetunes contriever to optimize the end performance of LM, showing gains over initial, non-fine tuned checkpoint of contriever. We are the first to study compressing retrieved documents for LMs and propose methods to train such compressors leveraging downstream LM performance. We would like to clarify that we do not claim that our proposed method is the best way to train such compressors, and future work can build upon our method to improve performance of the compressors.

Reviewer comment (Experiments):
The paper employs a BM25 retriever, making the results of the proposed method and baselines reliant on the retriever's performance. Therefore, it would be appropriate for the paper to also show performances using another retriever, such as contriver (not merely as a sentence selector). In addition, presenting the distribution of oracle-retrieved documents, similar to what's done in extractive compression, would be advantageous.

Author response:
[W2] We thank the reviewer for the suggestion on testing our method on multiple retrievers. We present new results on applying our compressor (trained with contriever-ms-marco retrieved documents) on (1) contriever retrieved documents (Table 1) and (2) DPR retrieved documents on NQ. (Table 2). 

We see that both our extractive and abstractive compressor generalize robustly to documents retrieved by another retrieval system (contriever-ms-marco to contriever/DPR), with the best compressor outperforming prepending top 1 documents with 4x compression.



Table 1: RECOMP results on NQ with contriever retrieved docs (FLAN-UL2)
| Setting                                | # prepended tokens | EM        |
|----------------------------------------|--------------------|-----------|
| No docs                                | 0                  | 21.99     |
| Top 1 doc                              | 130                | 27.53     |
| Top 5 doc                              | 652                | _33.19_   |
| Top 5 doc (contriever baseline)        | 31                 | 23.55     |
| Top 5 doc (our extractive compressor)  | 35                 | **30.66** |
| Top 5 doc (t5 baseline)                | 10                 | 24.32     |
| Top 5 doc (our abstractive compressor) | 28                 | 30.55     |

Table 2: RECOMP results on NQ with DPR retrieved docs (FLAN-UL2)
| Setting                                | # prepended tokens | EM        |
|----------------------------------------|--------------------|-----------|
| No docs                                | 0                  | 21.99     |
| Top 1 doc                              | 133                | 36.59     |
| Top 5 doc                              | 667                | _42.44_   |
| Top 5 doc (contriever baseline)        | 26                 | 25.60     |
| Top 5 doc (our extractive compressor)  | 36                 | 35.76     |
| Top 5 doc (t5 baseline)                | 10                 | 26.84     |
| Top 5 doc (our abstractive compressor) | 35                 | **37.40** |

Reviewer comment (Experiments):
It is necessary to compare the time taken to generate text by prepending all retrieved documents without compression against the time taken using the proposed method—specifically, compressing the retrieved documents before prepending them.

Author response:
[W3] Regarding inference time, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

Reviewer comment (Evaluation):
During the manual evaluation, the authors assess the outputs of the abstractive compressors. What was the level of annotation agreement among the evaluators?

Author response:
[Q1] Regarding annotation agreement, two of the co-authors annotated faithfulness and comprehensiveness  of summaries generated by GPT-3.5-turbo and our t5 compressor for ten questions. The annotators reach an agreement of 75% for factuality and 85% for the comprehensiveness evaluation (both are three-way classification). We will include more agreement statistics in the final paper.

Reviewer comment (Evaluation):
The performance improvement is limited or nothing in many settings. The authors need to justify it.

Author response:
[W2, Q2] Regarding performance improvement of RECOMP, the reviewer is right that our compressor doesn’t outperform prepending full top 5 documents in most of the settings. However, we would like to highlight that (1) the oracle abstractive & extractive compressor outperform no compression in all of the setting, demonstrating that compressors *can* outperform no compression. As the first work to study compressing retrieved documents, we hope our work inspires future work to propose better methods for training compressors, hence closing the gap. (2) Though not outperforming prepending top 5 documents, our method does outperform prepending top 1 document while prepending less token for both NQ and TQA, this demonstrates that RECOMP allows the model to consider more context, while prepending less tokens, thereby reducing computational cost.

Reviewer comment (Evaluation):
In the current evaluation, the benefit of using summarization for the retrieve augmented language model needs to be clarified. If the inference speed is improved, the authors should report it.

Author response:
[W1, Q1] Regarding inference speed, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.  

[Q3] As the reviewer mentioned, one advantage of RECOMP is that the model can consider more information that does not fit into the context window. Although we didn’t stress test the model to take a large amount of retrieved documents into consideration, our experiments do show that our compressor, which considers top 5 documents, outperforms prepending top 1 document, while prepending fewer tokens.

Reviewer comment (Evaluation):
The paper does not report the inference speedup of their method. Although experimental results show that fewer retrieved tokens are used, the number of tokens does not necessarily reflect the actual efficiency of the system, especially when the context length is not the bottleneck. The tokens of all the retrieved documents are still required to be processed by the compressor (a smaller model though). Therefore, it may be beneficial for authors to report the inference speedup of their method, as this could be the core to support their motivations.

Author response:
[W1, Q1] Regarding inference speed, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

Reviewer comment (Evaluation):
As stated in the weakness above, the inference speedup should be reported.

Author response:
[W1, Q1] Regarding inference speed, our experiments on NQ shows that both our extractive and abstractive compressor enables 2x throughput. Please see the general response for more detail.

Reviewer comment (Evaluation):
The training of the abstractive compressor uses a teacher LM (GPT-3.5), which is expensive. It is unclear if the cost saved in the inference stage is worth the cost in the training stage.

Author response:
[Q2]: The reviewer points out a valid concern about the training cost for such compressors. The training costs consist of (1) generating the training data and (2) training compressors on the generated training data. We describe both below.

For extractive compressors, generating training data incur running base LM inference on all the sentences. For an abstractive compressor, the major cost comes from generating training data using LLM such as GPT-3.5-turbo. It costs ~$130 to generate the abstractive summary for the entire NQ training set and we will report detailed costs for the other tasks in the paper. 

Training the extractive model takes a couple of hours on a single A40 GPU. Training the abstractive models takes a couple of hours on 4 A40 GPUs. 

We would also like to stress that the training cost is a one-time investment, as once the model is trained, it could be deployed for inference for a much larger scale which will make the training costs negligible.