Paper: Retrieval meets Long Context Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The authors of this paper mainly focus on larger LLMs, specifically those with more than 40B parameters. They include two specific LLMs in their study: the 43B GPT and the 70B LLaMA. It is worth noting that most existing efforts in this field only report results on smaller models, such as the 7B model. It would be beneficial if the authors could further investigate the relationship between the size of the LLMs and the tradeoff between retrieval and the context window. The authors propose hypotheses to explain why concurrent work has produced different findings, particularly regarding the usefulness of retrieval for the Llama2-7B-chat-4k model with a 4K context window. It would be better if the authors could provide empirical evidence to support these hypotheses.

Author response:
Following your suggestion, we have conducted the experiments for llama2-7b and the results are presented below. 

|        | Seq len             | Avg.          | QM | QASP | NQA | QLTY | MSQ | HQA | MFQA|
| ------ | --------------- | ------- | -- | - | - | - | - | - | - |
| Llama2-7B            | 4k   | 22.65  | 14.25  |	22.07 |	14.38  | 40.90 | 8.66  |	23.13  | 35.20|
| Llama2-7B (+ret)  | 4k    | 26.04 | 16.45  |	22.97 |	18.18  | 43.25 | 14.68 | 26.62 |40.10 |
| Llama2-7B            | 32k  | 28.20 | 16.09 | 23.66   |19.07   | 44.50 | 15.74 | 31.63 |46.71 |
| Llama2-7B (+ret)  | 32k  | 27.63 | 17.11  | 	23.25  |19.12  | 43.70 | 15.67 | 29.55 | 45.03 | 

For 4k sequence length, we observed retrieval can still have significant improvements while for 32k sequence length, adding retrieval is getting worse, which aligns with the findings in LongBench (Bai et al. (2023) for ChatGLM2-6B-32k. We believe the reason is that for Llama2-7B-chat-32 and ChatGLM2-6B-32k, their retrieval-augmented models have limited zero-shot capability of incorporating retrieved chunks of context, due to the smaller size. As a result, retrieval is not helpful for both Llama2-7B-32k and ChatGLM2-6B-32k, which is different from large LLMs like Llama2-70B-32k.  We add these discussion in the updated draft.

Reviewer comment (Experiments):
In terms of context window extension, this paper only utilizes the method mentioned in reference [2] for both LLMs. It would be interesting to explore whether the findings would be influenced by different methods of extending the context window.

Author response:
We agree with your comment in general. However, making advanced methods (e.g. memory or hierarchical attention) works for existing pretrained large language models e.g. Llama2-70B, is itself non-trivial. Further Pre-training the model with longer sequence requires much more computation. We add those into our future work section.

Reviewer comment (Experiments):
This paper focuses primarily on question answering and summarization tasks, in contrast to Longbench[1], which encompasses a variety of tasks, including few-shot learning, synthetic tasks, and code completion.

Author response:
Thanks for your suggestion. We conduct few-shot experiment on Trec and SAMsum and the results are as follow:

|                                | Trec   | SAMSum |
|:-------------:               |:-------:|:-------------:|
| GPT3.5-turbo-16k   |    68  |    41.7        |
| Llama2-70b-32k     |    73   |    46.48      |
| Llama2-70b-32k-ret |  76    |    47.31      |

Our best model Llama2-70B-32k-ret outperforms its non-retrieval Llama2-70B-32k baseline as well as GPT-3.5-turbo-16k by a large margin. It again confirms the benefits of using retrieval together with long context models. We have included this part in Section 4.5

Reviewer comment (Novelty):
While the idea is straightforward, the novelty is limited.

Author response:
- The focus of this paper is not proposing any novel idea for LLMs research. Intead, it investigates and combines two important trends in LLMs i.e., retrieval-augmented generation (RAG) and long context LLMs. The obtained results are significant; for example, our best model, retrieval-augmented LLaMA2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on seven long context tasks including question answering and query-based summarization (see the updated Table 3 in the updated manuscript). The conclusion and insights are very useful for the practitioners; for example, RAG and long context LLMs should be combined to obtain the best results. As a result, we believe this work is an important, timely and very useful contribution to the field.
  - As your comment on the Strengths section pointed out, “the findings from these extensive empirical experiments are intriguing and have the potential to serve as inspiration for future research in this domain”. We believe the important findings are as important as novel ideas.

Reviewer comment (Evaluation):
The 70B model consistently exhibits improvement when utilizing retrieval or longer context. In contrast, the 43B model shows inconsistent improvement. Consequently, drawing a definitive conclusion regarding when to employ retrieval in relation to specific downstream tasks and model sizes becomes challenging.

Author response:
- Note that both 70B and 43B models have consistent improvement in terms of **average** score across 7 downstream tasks when utilizing retrieval or long context extension, as shown in Table 2.  For 43B LLM, i) We observe improvements from baseline 43B-4k to retrieval-augmented 43B-4k on 6 out of 7 downstream tasks, except on QASP with small difference (baseline 23.66 vs. retrieval-augmented 23.45). ii) We also observe consistent improvement from baseline 43B-4k to long context 43B-16k on all 7 downstream tasks. iii ) The retrieval-augmented 43B-16k still outperforms its baseline 43B-16k in terms of average score (29.65 vs. 29.45), but we see variations on individual downstream tasks here partially due to the inherent variance exists in zero-shot evaluation. It is worth mentioning that we aim to draw conclusions for the average setting instead of specific downstream task across the paper.

Reviewer comment (Experiments):
Given that the conclusion differs from prior work that utilizes smaller models, establishing an apple-to-apple comparison demonstrating instances where the same example could fail in the smaller model but succeed in the larger model would enhance the persuasiveness of the findings.

Author response:
- Bai et al. (2023) find that retrieval is only helpful for Llama2-7B-chat-4k with 4K context window, but not helpful for long context model ChatGLM2-6B-32k.  We discussed this in Related Work Section 2.4 of submission. To highlight it, we move it to the beginning of related work Section 2.1.
  - Per your suggestion, we have added the results using Llama2-7B in the following Table and updated draft. One can actually draw similar conclusions to Bai et al. (2023) for small 7b models. We think the underlying reasons are: i) For Llama2-7B-chat-4k, its short context length is the bottleneck for long context tasks. Thus, retrieval-augmentation largely improves the results. ii) For Llama2-7B-chat-32 and ChatGLM2-6B-32k, the context length bottleneck has been mostly removed. However, their retrieval-augmented models have limited zero-shot capability of incorporating retrieved chunks of context, due to the smaller size. As a result, retrieval is not helpful for both Llama2-7B-32k and ChatGLM2-6B-32k, which is different from large LLMs like Llama2-70B-32k. We also add this analysis and results into our updated manuscript. 


|        | Seq len             | Avg.          | QM | QASP | NQA | QLTY | MSQ | HQA | MFQA|
| ------ | --------------- | ------- | -- | - | - | - | - | - | - |
| Llama2-7B            | 4k   | 22.65  | 14.25  |	22.07 |	14.38  | 40.90 | 8.66  |	23.13  | 35.20|
| Llama2-7B (+ret)  | 4k    | 26.04 | 16.45  |	22.97 |	18.18  | 43.25 | 14.68 | 26.62 |40.10 |
| Llama2-7B            | 32k  | 28.20 | 16.09 | 23.66   |19.07   | 44.50 | 15.74 | 31.63 |46.71 |
| Llama2-7B (+ret)  | 32k  | 27.63 | 17.11  | 	23.25  |19.12  | 43.70 | 15.67 | 29.55 | 45.03 |

  - We also add concrete examples in Appendix where the smaller model fails to incorporate relevant context, while larger models could successfully do that.

Reviewer comment (Experiments):
Do you observe different conclusions when using a smaller LLM (i.e., Llama2-7B and ChatGLM2-6B) as in Bai et al. 2023?

Author response:
- Bai et al. (2023) find that retrieval is only helpful for Llama2-7B-chat-4k with 4K context window, but not helpful for long context model ChatGLM2-6B-32k.  We discussed this in Related Work Section 2.4 of submission. To highlight it, we move it to the beginning of related work Section 2.1.
  - Per your suggestion, we have added the results using Llama2-7B in the following Table and updated draft. One can actually draw similar conclusions to Bai et al. (2023) for small 7b models. We think the underlying reasons are: i) For Llama2-7B-chat-4k, its short context length is the bottleneck for long context tasks. Thus, retrieval-augmentation largely improves the results. ii) For Llama2-7B-chat-32 and ChatGLM2-6B-32k, the context length bottleneck has been mostly removed. However, their retrieval-augmented models have limited zero-shot capability of incorporating retrieved chunks of context, due to the smaller size. As a result, retrieval is not helpful for both Llama2-7B-32k and ChatGLM2-6B-32k, which is different from large LLMs like Llama2-70B-32k. We also add this analysis and results into our updated manuscript. 


|        | Seq len             | Avg.          | QM | QASP | NQA | QLTY | MSQ | HQA | MFQA|
| ------ | --------------- | ------- | -- | - | - | - | - | - | - |
| Llama2-7B            | 4k   | 22.65  | 14.25  |	22.07 |	14.38  | 40.90 | 8.66  |	23.13  | 35.20|
| Llama2-7B (+ret)  | 4k    | 26.04 | 16.45  |	22.97 |	18.18  | 43.25 | 14.68 | 26.62 |40.10 |
| Llama2-7B            | 32k  | 28.20 | 16.09 | 23.66   |19.07   | 44.50 | 15.74 | 31.63 |46.71 |
| Llama2-7B (+ret)  | 32k  | 27.63 | 17.11  | 	23.25  |19.12  | 43.70 | 15.67 | 29.55 | 45.03 |

  - We also add concrete examples in Appendix where the smaller model fails to incorporate relevant context, while larger models could successfully do that.

Reviewer comment (Novelty):
While this paper presents a useful empirical study, it does not introduce any new technique. Combining retrieval augmentation and long-context LLM is a straightforward idea, and the two being complementary to each other, though encouraging, is not very surprising.

Author response:
- We agree this paper does not introduce any new techniques, and the conclusion is “not very surprising”. However, we believe it is an important, timely and very useful contribution to the field. It investigates two important trends in LLMs i.e., retrieval-augmented generation (RAG) and long context LLMs. The obtained results are significant; for example, our best model, retrieval-augmented LLaMA2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on seven long context tasks including question answering and query-based summarization (see the updated Table 3 in the updated draft). The conclusion and insights are very useful for the practitioners; for example, RAG and long context LLMs should be combined to obtain the best results. We update the abstract of the paper to clarify the contribution of this work.

Reviewer comment (Experiments):
If I understand correctly (I find this part of the writing unclear, see Question 1 below and correct me if my assumption is wrong), the retrieval corpus is the collection of the context documents from each evaluation dataset (or maybe even each example). This may be a key reason why retrieval is more helpful in this work. In reality, for most of the tasks, it is impossible to assume to have perfectly relevant retrieval corpora (i.e. gold contexts from the test set), and the helpfulness of retrieval would dramatically decrease if the retrieval corpus mismatches the evaluation data.

Author response:
- Thanks for raising the question. Yes, the long context QA and summarization tasks are mostly similar to reading comprehension of long documents. Note that doing QA and summarization on customers' own documents (can be long for a lot of cases) is one of the major applications of long context LLM e.g., [1, 2]. The retrieval-augmentation certainly works for open-domain QA (e.g., retrieval corpus can be wikipedia). However, the current long context LLM (even up to 128K context length) can’t work in such “open” settings without retrieval.


[1]  Anthropic. Introducing 100k context windows. 2023

[2] OpenAI. Function calling and other API updates (longer context). 2-23

> 1. “What's your retrieval corpus?”
  - The retrieval corpus for each task/dataset is the provided single or multiple documents in that dataset. There is no training set for each task, as all the seven tasks are designed for zero-shot evaluation w/o any task-specific finetuning. 

> 2. “It seems the evaluation tasks are all "closed-type", meaning the gold context is given at inference time. Is this the case? If so, how would retrieval be further helpful if the gold context is already provided? Is it because for certain tasks, the provided gold context can be too long and the relevant parts may have been truncated?”
  - Thanks for raising the question. Yes, the long context tasks are “closed-type”, but the given single or multiple gold documents are quite long e.g. NarrativeQA has average document length as 84k, and maximum document length as multiple hundred of thousand tokens. In many real world applications, a document can have million tokens e.g., detailed technical manuals. In these cases, retrieval is helpful, as i) the current long context LLM won’t be able to handle, ii) even it is able to handle, it is very expensive in terms of computation, iii) the sparsity provided by good retriever can improve accuracy than dense self-attention mechanism on long sequence, as suggestion by our retrieval-augmented Llama2-70B-32k outperforms its  Llama2-70B-32k baseline.

Reviewer comment (Experiments):
As a result, the method considered in this paper is more tailored for one type of task, essentially reading comprehension of long documents, because the assumption that gold contexts, albeit long, are provided at test time. It would be more interesting to also explore the "open" settings where the gold context is not necessarily given.

Author response:
- Thanks for raising the question. Yes, the long context QA and summarization tasks are mostly similar to reading comprehension of long documents. Note that doing QA and summarization on customers' own documents (can be long for a lot of cases) is one of the major applications of long context LLM e.g., [1, 2]. The retrieval-augmentation certainly works for open-domain QA (e.g., retrieval corpus can be wikipedia). However, the current long context LLM (even up to 128K context length) can’t work in such “open” settings without retrieval.


[1]  Anthropic. Introducing 100k context windows. 2023

[2] OpenAI. Function calling and other API updates (longer context). 2-23

Reviewer comment (Experiments):
Most studies primarily focus on improving models during inference with attention given to examining the impact of training methodologies.

Author response:
> 7. “Have you tried training the retriever and LLM from start to finish of just adding augmentation during inference? How does the performance compare in cases?”
  - Thank you for raising the question. We just add augmentation during inference in such retrieval-augmented generation (RAG) setting. Training the retriever and LLM in an end-to-end manner is an interesting direction for future exploration.

Reviewer comment (Experiments):
Experiments have not yet been conducted on contexts, which span 64,000 tokens and this could pose additional challenges.

Author response:
> 2. “Experiments have not yet been conducted on contexts, which span 64,000 tokens and this could pose additional challenges.”
  - Thank you for the valuable comment. Extending the context window to 64k and even longer would be very interesting. One of the major challenges is that training 64k context window LLM with 70B parameters is really expensive. Due to the limit of resources, we leave it for future study.

Reviewer comment (Experiments):
There is a need for evaluations against advanced methods in long context modeling and further benchmarking is necessary.

Author response:
> 3. “There is a need, for evaluations against advanced methods in long context modeling and further benchmarking is necessary. .” and “For example memory or hierarchical attention.”
  - We agree with your comment in general. However, making advanced methods (e.g. memory or hierarchical attention) works for existing pretrained large language models e.g. Llama2-70B, is itself non-trivial. We include it in the future direction section of our updated draft.

Reviewer comment (Experiments):
The experiments section would be more informative if it provided information about characteristics and task formats.

Author response:
> 4. “The experiments section would be more informative if it provided information about characteristics and task formats.”
  - Thank you for the valuable comments. We revised Section 3.2 and added more information about the characteristics and task formats for each task.

Reviewer comment (Novelty):
The related work section should clearly highlight the contributions of this study compared to work like LongBench.

Author response:
> 5. “The related work section should clearly highlight the contributions of this study compared to work like LongBench.”
  - Thank you for the valuable comment. In related work section 2.4 of submission, we describe the difference and contribution of this study compared to the LongBench work. In the updated manuscript, to highlight the discussion, we move this discussion to section 2.1.

Reviewer comment (Novelty):
It's important to note that previous research has explored the concepts of context and retrieval. This study should acknowledge and build upon existing literature.

Author response:
> 6. “The paper mentions another study called LongBench that reached conclusions regarding the impact of retrieval on context models. It would be beneficial if the authors could further analyze the differences that led to these opposing observations.”
  - The LongBench work (Bai et al. 2023) finds that retrieval is only helpful for Llama2-7B-chat-4k with 4K context window, but not helpful for long context model ChatGLM2-6B-32k.
  - We have added the results using Llama2-7B in the following Table and updated draft. One can actually draw similar conclusions to Bai et al. (2023). We think the underlying reasons are: i) For Llama2-7B-chat-4k, its short context length is the bottleneck for long context tasks. Thus, retrieval-augmentation largely improves the results. ii) For Llama2-7B-chat-32k and ChatGLM2-6B-32k, the context length bottleneck has been mostly removed. However, their retrieval-augmented models have limited zero-shot capability of incorporating retrieved chunks of context, due to the smaller size. As a result, retrieval is not helpful for both Llama2-7B-32k and ChatGLM2-6B-32k, which is different from large LLMs like Llama2-70B-32k. We also add this analysis and results into our updated manuscript. 

|        | Seq len             | Avg.          | QM | QASP | NQA | QLTY | MSQ | HQA | MFQA|
| ------ | --------------- | ------- | -- | - | - | - | - | - | - |
| Llama2-7B            | 4k   | 22.65  | 14.25  |	22.07 |	14.38  | 40.90 | 8.66  |	23.13  | 35.20|
| Llama2-7B (+ret)  | 4k    | 26.04 | 16.45  |	22.97 |	18.18  | 43.25 | 14.68 | 26.62 |40.10 |
| Llama2-7B            | 32k  | 28.20 | 16.09 | 23.66   |19.07   | 44.50 | 15.74 | 31.63 |46.71 |
| Llama2-7B (+ret)  | 32k  | 27.63 | 17.11  | 	23.25  |19.12  | 43.70 | 15.67 | 29.55 | 45.03 |

Reviewer comment (Experiments):
There are opportunities to explore approaches in combining retrieval and context that merit investigation.

Author response:
> 1. “There is a lack of analysis when it comes to integrating context and retrieval techniques like determining the ideal number of retrieved passages.”
  - Thank you for the comment. In Table 5 of the submitted draft, we did a comparison with different numbers of retrieved passages from top-5, top-10 to top-20. From the results, there is no optimal predetermined number that works consistently better. It would be interesting to dynamically determine the optimal number of retrieved passages. We leave it for future study.

Reviewer comment (Experiments):
Further validation is required to assess the suitability of these techniques for contexts.

Author response:
> 9. “For context models like those with 64K or 128K tokens do you believe retrieval would still provide benefits?. Does its value decrease after a point?”
  - This is a good question. We believe retrieval would certainly provide benefits, if the document length is longer than 64K or 128K tokens. Indeed, there are a lot of real-world documents having millions of tokens, where the retrieval-augmentation really shines. Another benefit of retrieval is about saving computation by utilizing an efficient retriever to select relevant context, rather than the expensive self-attention mechanism in LLM.

Reviewer comment (Experiments):
The authors attribute the superior performance of retrieval-augmented long context LLMs (e.g., 16K and 32K) over the 4K context LLM in a top-5 setting to the 'lost-in-the-middle' phenomenon. However, since the 'lost-in-the-middle' conclusions have not been widely recorganized, conducting an ablation study would be instrumental in supporting this hypothesis.

Author response:
Thank you for the valuable suggestion. We conduct the “lost-in-the-middle” study as in (Liu et al., 2023) for Llama2-70B-4k and Llama2-70B-32k. See Figure 1 in the updated draft. We confirm that the phenomenon also exists in Llama2 with different context lengths. In particular, the comparison of the curves from Llama2-70B-4k and Llama2-70B-32k suggests that the long context model has better accuracy for incorporating top-5 retrieved context.

Reviewer comment (Evaluation):
The authors use LLM performance metrics to infer the efficacy of different retrieval modules (Table 4). However, it is questionable whether a retrieval system with better LLM performance necessarily correlates with enhanced retrieval capability. The authors should incorporate some form of automated evaluation or a case study on retrieval performance to substantiate this.

Author response:
Thank you for the comment. We don’t intend to claim that one retriever is better than another based on the downstream LLM performance. Indeed, the difference of average accuracy of using Dragon (35.73 for Llama2-70B-4k), Contriever (36.02), and OpenAI-embedding (35.79) are relatively small compared to baseline Llama2-70B-4k w/o retrieval (31.61). So, the takeaway is that retrieval-augmented generation (RAG) with state-of-the-art retrievers can boost the accuracy of  baseline w/o retrieval. We further clarify this in our updated draft.

Reviewer comment (Experiments):
The OpenAI GPT series also offers variants with different context lengths (e.g., GPT-3.5-turbo, GPT-3.5-turbo-16k). Including their results in Table 2 and examining whether the conclusions still hold would be beneficial.

Author response:
Many thanks for your valuable suggestions. In our updated draft, we provide the results of GPT-3.5-turbo (4k context length), GPT-3.5-turbo-16k in Table 3. We add the retrieval-augmentation results of GPT-3.5-turbo and  GPT-3.5-turbo-16k below for MSQ, HQA, MFQA for and also in Table 3 

|                                   | Avg | MSQ | HQA | MFQA|
|--- | - |:-------------:|:-------------:|:-------------:|
|GPT3.5-turbo-4k	| 37.08 | 21.23 | 40.86	| 49.16 |
|GPT3.5-turbo-4k-ret	| 41.15 | 24.41 | 49.54 | 49.50 |
|GPT3.5-turbo-16k	| 43.60 | 26.90 | 51.60 | 52.30 |
|GPT3.5-turbo-16k-ret | 43.27	| 30.40 | 46.60 | 52.80 |
|Llam2-70B-32k-ret     | 44.51	| 26.72 | 53.89 | 52.91 |

For GPT3.5-turbo-4k, retrieval significantly improves the performance . For GPT3.5-turbo-16k, the average scores for retrieval and non-retrieval scores are close to each other which are both lower than our Llam2-70B-32k-ret results. Also note that GPT3.5-turbo-16k is a blackbox API, we don’t know how it is implemented, the model size as well as any preprocessing steps implemented. We add this part to our updated draft.

Reviewer comment (Evaluation):
Regarding the results presented in Table 2, it is unclear how the best retriever was determined. Is a better-performing retriever module always associated with improved LLM performance?

Author response:
Thank you for the question. There is no guarantee that a better-performing retriever (in terms of precision/recall based evaluation metric) could always provide improved LLM performance. In Table 2, we simply pick the retriever based on the best LLM performance. Note that, the differences of LLM’s average performances associated with different SOTA retrievers, e.g. Dragon, Contriver and openai-embedding are relatively small as shown in Table 4.