Paper: Generative Retrieval with Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
Despite the authors' efforts to enhance computational efficiency through the use of shorter prefixes and the Knuth-Morris-Pratt (KMP) algorithm for passage location, the proposed LLM2GR model still necessitates at least three forward computations. This computational burden remains a significant hindrance to its practical applicability.

Author response:
**W1: The trade-off of two-stage method and speed consideration**

The two-stage method does require two forward passes, but by generating short snippets instead of one long passage at a time, we can still greatly reduce the time consumption compared to single-stage methods. Due to the current difficulty for LLMs to directly recall fine-grained passages, we adopted the trade-off of using a two-stage approach.

Reviewer comment (Experiments):
The comparison between LLM2GR and existing methods such as Dense Retriever (DPR), BM25, and Contriever raises questions of fairness. Specifically, LLM2GR employs a considerably more powerful LLM for generative tasks, which inherently skews the comparison with baselines of lesser model capacity. Using an LLM for retrieval is not inherently novel. A more equitable evaluation might involve integrating LLMs with existing retrieval methods—such as DPR—to compare generative capabilities. For instance, one could employ DPR for retrieval and then use an LLM for the generative task based on the retrieved results, thus providing a fairer baseline for comparison. If the proposed two-staged method can still deliver better results, thus demonstrating the effectiveness of the proposed method.

Author response:
**W2: Differences from combining existing LLMs and traditional retrieval like DPR**

There are many studies trying to combine LLM generation with traditional methods like DPR, which is similar to using retrieval tools. Our method aims to explore a completely new paradigm of performing retrieval solely based on the capabilities of LLMs themselves, without needing external tools. We think these two lines of work are actually parallel, and prompting LLMs to mimic human recall itself to accomplish retrieval has unique significance. We also reproduced the query2doc (Wang et al.,2023) method of generating for retrieval enhancement using the same hyperparameters on NQ and WOW datasets.

| Method | Page-level NQ | Page-level WOW | passage-level NQ | passage-level WOW |  
|-|-|-|-|-|
| bm25 | 26.33 | 28.78 | 23.65 | 50.36 |
| query2doc+bm25 | 47.55 | 48.49 | 41.42 | 62.02 |
| LLM2GR | 57.77 | 57.63 | 40.82 | 63.43 |

The query2doc method enhances retrieval by prompting LLM to generate virtual document. We see significant improvements over bm25. However, our method can still achieve better page-level results using LLMs alone, and competitive passage-level results. 

In addition, query2doc takes ~430 minutes on NQ while our LLM2GR only needs 150 minutes.

Reviewer comment (Writing):
Writing Errors: There are typographical errors in the manuscript, such as in Section 2.3 where the phrase "generate related contexts for "retrieval,"" is incorrectly punctuated. Additionally, the usage of quotation marks throughout the paper is inconsistent.

Author response:
**W3: Writing issues**

Thank you for pointing out the writing issues, we will make corresponding modifications!

Reviewer comment (Experiments):
A significant limitation of the proposed method is its dependence on the universal knowledge gained during the pretraining stage. This raises questions about the method's applicability in scenarios that go beyond the scope of Wikipedia or in specialized vertical fields such as medicine or economics. The paper does not address how the method could be adapted or if it remains effective when applied to these more specialized domains. This limitation is a critical factor that constrains my enthusiasm for strongly recommending the paper for acceptance.

Author response:
**W4: Other domain**

Currently our method is only experimented on Wikipedia data, because LLMs are mostly trained on general domain data like Wikipedia during pre-training, while their performance is inferior on some vertical domains which they are not sufficiently trained on. We fully agree on the importance of applying LLM retrieval to various vertical domains, but this is a limitation of our current method. Continued exploration is needed on incorporating new domain documents more effectively, and instruction tuning to activate stronger recall.

Reviewer comment (Theory):
While the paper convincingly demonstrates that Large Language Models (LLMs) can be used for retrieval, it lacks an in-depth discussion on why LLMs can perform this task effectively. Is it because the models have been exposed to the documents during the pre-training phase?

Author response:
**W5: Memorization during pre-training**

We think the main reason our method works is because during pre-training LLMs are exposed to large amounts of documents, so like humans after reading many books, LLMs also remember some document details. This allows us to activate recall of relevant snippets directly through prompting, and ensure factuality via indexed constraints.

Reviewer comment (Reproducibility):
In section 3.3, given a short text prefix $p_s$ of $l_{p_s}$ tokens, the KMP algorithm is used to determine the start position $st$ of $p_s$, then a complete passage is extracted by catching starting from $st$ with the length of $l_p$ tokens. Here a question arises that if the text prefix $p_s$ is equal to the final retrieved passage $p_final = d[st : st + l_p]$?

Author response:
**Q1: Prefix length** 

Here we set the prefix length much smaller than the final length, 16 vs 150. So few cases will have identical prefix and final lengths. When the prefix is at the end, we will still extract from the end back to the prefix, which does not impact performance significantly.

Reviewer comment (Novelty):
Generative retrieval methods have been actively studied, and generating full evidence paragraphs (Yu et al., 2023), URLs (Ziems et al., 2023), Wikipedia titles (De Cao et al., 2020), or substrings (Bevilacqua et al., 2022) have been explored. Bevilacqua et al. (2022) generate substrings, which are similar to short prefix generations, and use the FM-Index to constrain the autoregressive generations. It is not discussed in depth how this work is different from Bevilacqua et al. (2022). One difference might be prompting an LM rather than fine-tuning. Yet, if the novelty is mostly prompting a larger LM rather than fine-tuning a smaller LM, this is not seen as a sufficient contribution for ICLR.

Author response:
**W1: Technical novelty**

Our method explores a completely new retrieval paradigm - generating determined snippets directly by utilizing the capability of large language models to mimic human recall, without any segmentation of long texts or extra models and training. Constrained decoding solves the hallucination problem of directly generating using LLMs in Yu et al. (2023). It also addresses the inferior performance of Ziems et al. (2023) on smaller LLMs. 

The key difference from SEAL(Bevilacqua et al., 2022) is that we rely on the capability of LLMs themselves to determine the start positions of fine-grained snippets, instead of generating part of pre-segmented passages to finish retrieval, which introduces much more difficulty. But we believe this retrieval idea will have greater flexibility in the future.

Reviewer comment (Experiments):
Despite the proposed method being a new generative retrieval method, none of the baselines is generative retrieval. Several prior works including Bevilacqua et al. (2022) or Yu et al. (2023) are tested on KILT or open-domain QA datasets and are open-source, and it is unclear why the authors did not directly compare their method with such prior work.

Author response:
**W2: Comparison with other generative retrieval baseline**

Since all components of Yu et al. (2023) are implemented using closed APIs, direct comparison with our method using open-source LLMs like llama would be very unfair. For llm-url we reproduced results using the same hyperparameters and llama2 13b on NQ and WOW datasets.

| Method | Page-level NQ | Page-level WOW | passage-level NQ | passage-level WOW |
|-|-|-|-|-|  
| bm25 | 26.33 | 28.78 | 23.65 | 50.36 |
| DPR | 54.74 | 26.62 | 47.94 | 45.38 |
| SEAL | 64.15 | 50.49 | 44.94 | 62.08 | 
| llm-url+bm25 | - | - | 19.67 | 43.09 |
| LLM2GR | 57.77 | 57.63 | 40.82 | 63.43 |

The trained SEAL method has been greatly improved compared to dense retrieval methods, showing the effectiveness of generative retrieval through internal model interactions. Note that both SEAL and DPR require extensive training, while our method can achieve competitive results in zero-shot settings by prompting LLMs.

The llm-url method generates Wikipedia URLs using LLMs. Since it is hard to directly map to KILT benchmark documents, we only provided passage-level results. We found llama2 13b can only generate valid urls with efficacy of 39.5% and 45.2% on the two datasets. This makes it hard to generate sufficient accurate URLs to achieve effective results when using smaller LLMs. Our method can achieve competitive results using smaller open-source LLMs with prompting and constrained decoding, which will facilitate the real-world application of generative retrieval.

Reviewer comment (Experiments):
Several advantages mentioned in the introduction or the related work section have not been validated by experimental results. For instance, the claim that generating complete passages can be overly time-consuming and that the proposed SPGL approach addresses this should be supported by benchmarking inference time latency. Given the complex pipeline of the proposed method and active research on fast inference (e.g., speculative decoding, paged attentions), there is uncertainty about whether the method is indeed more practical despite skipping longer generations, as it may take more time due to the expensive search process. The authors should benchmark other generative retrieval methods and quantitatively show the proposed method's effectiveness.

Author response:
**W3: Using faster decoding methods**

We think using faster decoding methods is actually orthogonal to our current approach. Combined with faster LLM inference in the future, our existing method will lead to faster generative retrieval. Since our method focuses more on the retrieval performance itself, we did not incorporate fast model inference. However, designing specialized accelerating algorithms for generative retrieval is an interesting future direction.

**Q1/Q2: Inference latency**

As analyzed in the ablation study in Section 4.3, our method takes about 150 minutes on the NQ dataset, dense retrieval takes about 20 minutes, and the generative retrieval method SEAL needs about 11 minutes, llm-url takes about 1700 minutes to generate URLs. Currently due to the expensive cost of LLMs, the time is still far more than traditional fine-tuning of small models. Further optimizing the speed of generative retrieval is crucial.

Reviewer comment (Experiments):
Did you evaluate other generative retrieval model performance as well as inference latency?

Author response:
**Q1/Q2: Inference latency**

As analyzed in the ablation study in Section 4.3, our method takes about 150 minutes on the NQ dataset, dense retrieval takes about 20 minutes, and the generative retrieval method SEAL needs about 11 minutes, llm-url takes about 1700 minutes to generate URLs. Currently due to the expensive cost of LLMs, the time is still far more than traditional fine-tuning of small models. Further optimizing the speed of generative retrieval is crucial.

Reviewer comment (Evaluation):
Did you compare the inference time latency of non-generative retrieval methods and your method?

Author response:
**Q1/Q2: Inference latency**

As analyzed in the ablation study in Section 4.3, our method takes about 150 minutes on the NQ dataset, dense retrieval takes about 20 minutes, and the generative retrieval method SEAL needs about 11 minutes, llm-url takes about 1700 minutes to generate URLs. Currently due to the expensive cost of LLMs, the time is still far more than traditional fine-tuning of small models. Further optimizing the speed of generative retrieval is crucial.

Reviewer comment (Experiments):
It seems that the proposed two-stage method is technically not very attractive. Under LLM, a simpler design like generating title identifier and prefix in a single stage would be preferred. The kind of simpler method needs to be considered and adopted as the baseline for the comparison.

Author response:
**W1: Single-stage method**

In the ablation experiment in Section 4.3 of our paper, we compared the single-stage generation method of large language model based generative retrieval, i.e. the method without the first stage. In this setting, we directly prompt the large language model to generate the required fine-grained passages.

| Method | Page-level NQ | Page-level TriviaQA | Page-level HotpotQA | Passage-level NQ | Passage-level TriviaQA | Passage-level HotpotQA |  
|-|-|-|-|-|-|-|
| LLM2GR | 57.77 | 54.41 | 48.70 | 40.82 | 68.20 | 30.04 |
| W/o first stage | 32.22 | 24.87 | 23.36 | 36.27 | 63.33 | 24.16 |

We found that in this case the performance of current large language models would drop dramatically. Current large language models have difficulty recalling key segments from a vast space, so we adopted this trade-off of using the two-stage method.

Reviewer comment (Experiments):
It seems that the baselines are weak. In the fully-finetuned case, not only DPR but also the reranking method needs to be provided, as the LLM generates the passages based on a stronger attention-based interaction between a question and a document, than the DRP that uses the inner product between dense vectors.

Author response:
**W2: Re-ranking methods**

Re-ranking methods need to recalculate cross attentions between the retrieved top-k passages and the question, i.e. access each retrieved passage during inference. Our method, like dense retrieval, does not directly access passages during retrieval, but relies on parameters stored in the model, which is different from re-ranking and is a parallel method to dense retrieval. Therefore, we did not compare with further re-ranking methods, similar to previous generative retrieval methods like SEAL(Bevilacqua et al., 2022).

Reviewer comment (Experiments):
The retrieval performance was only compared with previous dense and sparse retrieval methods, lacking comparison with recent generative retrieval methods.

Author response:
**W3/W4: Comparison with other generative retrieval baselines and query2doc**

We conducted experiments with SEAL(Bevilacqua et al., 2022) and reproduced LLM-URL(Ziems et al., 2023) and Query2doc(Wang et al.,2023) using the same hyperparameter settings with Llama2 13b on NQ and WOW datasets.

| Method | Page-level NQ | Page-level WOW | Passage-level NQ | Passage-level WOW | 
|-|-|-|-|-|
| bm25 | 26.33 | 28.78 | 23.65 | 50.36 |
| DPR | 54.74 | 26.62 | 47.94 | 45.38 | 
| SEAL | 64.15 | 50.49 | 44.94 | 62.08 |
| llm-url+bm25 | - | - | 19.67 | 43.09 |
| query2doc+bm25 | 47.55 | 48.49 | 41.42 | 62.02 |
| LLM2GR | 57.77 | 57.63 | 40.82 | 63.43 |

We found that the trained SEAL method has been greatly improved compared to dense retrieval methods, demonstrating the effectiveness of generative retrieval through internal model interactions. Both SEAL and DPR methods have gone through extensive training, while our method can achieve competitive results in zero-shot settings by prompting large language models. 

The llm-url method generates Wikipedia URLs using large language models, but it is difficult to directly map to KILT benchmark documents, so we only provided passage-level experimental results. We found that when using the llama2 13b model, it is difficult to generate enough valid Wikipedia URLs to achieve effective results.

The query2doc method enhances retrieval by prompting LLM to generate virtual document. We noticed a significant improvement over the bm25 method. However, our method can still achieve better results in page-level retrieval by solely utilizing the capabilities of LLMs, and achieve competitive retrieval results at the passage-level.

In addition, the llm-url method took about 1700 minutes on the NQ dataset, while query2doc took about 430 minutes, both exceeding the 150 minutes consumed by our llm2gr method.

Reviewer comment (Experiments):
In the proposed method, only zero-shot settings are presented. Few-shot settings of LLM need to also be provided. Under few-shot prompts, other LLM-based methods such as query2doc could be compared. https://arxiv.org/pdf/2303.07678.pdf

Author response:
**W3/W4: Comparison with other generative retrieval baselines and query2doc**

We conducted experiments with SEAL(Bevilacqua et al., 2022) and reproduced LLM-URL(Ziems et al., 2023) and Query2doc(Wang et al.,2023) using the same hyperparameter settings with Llama2 13b on NQ and WOW datasets.

| Method | Page-level NQ | Page-level WOW | Passage-level NQ | Passage-level WOW | 
|-|-|-|-|-|
| bm25 | 26.33 | 28.78 | 23.65 | 50.36 |
| DPR | 54.74 | 26.62 | 47.94 | 45.38 | 
| SEAL | 64.15 | 50.49 | 44.94 | 62.08 |
| llm-url+bm25 | - | - | 19.67 | 43.09 |
| query2doc+bm25 | 47.55 | 48.49 | 41.42 | 62.02 |
| LLM2GR | 57.77 | 57.63 | 40.82 | 63.43 |

We found that the trained SEAL method has been greatly improved compared to dense retrieval methods, demonstrating the effectiveness of generative retrieval through internal model interactions. Both SEAL and DPR methods have gone through extensive training, while our method can achieve competitive results in zero-shot settings by prompting large language models. 

The llm-url method generates Wikipedia URLs using large language models, but it is difficult to directly map to KILT benchmark documents, so we only provided passage-level experimental results. We found that when using the llama2 13b model, it is difficult to generate enough valid Wikipedia URLs to achieve effective results.

The query2doc method enhances retrieval by prompting LLM to generate virtual document. We noticed a significant improvement over the bm25 method. However, our method can still achieve better results in page-level retrieval by solely utilizing the capabilities of LLMs, and achieve competitive retrieval results at the passage-level.

In addition, the llm-url method took about 1700 minutes on the NQ dataset, while query2doc took about 430 minutes, both exceeding the 150 minutes consumed by our llm2gr method.

Reviewer comment (Experiments):
Other popular datasets are not considered for evaluation, such as MS MARCO, TREC DL 19, and TREC DL 20. To convincingly validate the proposed zero-shot retrieval of LLM, the work needs to provide a full comparison on more IR collections.

Author response:
**W5: Other datasets**

Since our method aims to generate fine-grained snippet information without any pre-segmentation of passages using large language models, while MS MARCO, TREC DL 19 and TREC DL 20 consist of fixed pre-segmented passages without providing original documents and final answers like the KILT dataset, it is difficult to properly evaluate our proposed new retrieval scheme. Therefore, we did not choose these standard IR datasets. The variety of task types and large amount of test data in KILT make it a valid evaluation in our opinion.

Reviewer comment (Experiments):
The construction of FM-index after the first stage seems to be done per query. Its computational overhead is not fully discussed. Why FM-index is not constructed over the full documents, towards a manner of restricting a full FM-index to the only restricted top-k documents.

Author response:
**W6: FM-index pre-building**

In fact, our method uses pre-built FM-index in implementation, which takes about 8h to build the entire Wikipedia corpus. During generation retrieval, we can directly extract the corresponding index sections, so the time consumption is negligible.

Reviewer comment (Experiments):
In section 3.3, the selection method of the unique documents (from document set) and prefix (from multiple prefixes) is naïve. Although in the vast majority of cases the “top k document” was unique in this dataset, the method of “default choice” does not have any ranking or selection capabilities in the case of extending this work to larger corpus or practical applications.

Author response:
**W8: Selecting the first prefix**

Since we select from limited top-k documents, and the short prefixes also have a certain length, this method can produce a unique result in most cases. However, we also noticed that when documents get longer and more top-k documents are obtained, more ambiguity may occur leading to some bias. We think this bias can be solved by combining with re-ranking methods in future work. Since our paper focuses on the retrieval part, we leave this to future work.

Reviewer comment (Experiments):
The proposed two-stage method needs to be compared with a single-stage method that generates a title identifier and prefix of a passage. Why a single-stage method is not considered?

Author response:
**W1: Single-stage method**

In the ablation experiment in Section 4.3 of our paper, we compared the single-stage generation method of large language model based generative retrieval, i.e. the method without the first stage. In this setting, we directly prompt the large language model to generate the required fine-grained passages.

| Method | Page-level NQ | Page-level TriviaQA | Page-level HotpotQA | Passage-level NQ | Passage-level TriviaQA | Passage-level HotpotQA |  
|-|-|-|-|-|-|-|
| LLM2GR | 57.77 | 54.41 | 48.70 | 40.82 | 68.20 | 30.04 |
| W/o first stage | 32.22 | 24.87 | 23.36 | 36.27 | 63.33 | 24.16 |

We found that in this case the performance of current large language models would drop dramatically. Current large language models have difficulty recalling key segments from a vast space, so we adopted this trade-off of using the two-stage method.

Reviewer comment (Experiments):
What is the computational time of constructing the FM-index? Is it not possible to construct the FM-index over the full passages, and to use it in a restricted way on only top-k documents?

Author response:
**W6: FM-index pre-building**

In fact, our method uses pre-built FM-index in implementation, which takes about 8h to build the entire Wikipedia corpus. During generation retrieval, we can directly extract the corresponding index sections, so the time consumption is negligible.

Reviewer comment (Experiments):
In Table 1-2, only DPR is compared. What is the best performance of the existing works on the same collections? The reranking-based method needs to also be compared.

Author response:
**W2: Re-ranking methods**

Re-ranking methods need to recalculate cross attentions between the retrieved top-k passages and the question, i.e. access each retrieved passage during inference. Our method, like dense retrieval, does not directly access passages during retrieval, but relies on parameters stored in the model, which is different from re-ranking and is a parallel method to dense retrieval. Therefore, we did not compare with further re-ranking methods, similar to previous generative retrieval methods like SEAL(Bevilacqua et al., 2022).

Reviewer comment (Theory):
In section 3.2, it was claimed that the “information loss” issue of text chunking for lengthy documents was overcame by the proposed method, however, this work actually generates a short prefix of the document, which is also less informative than the whole document or document chunk. The reviewer would like to see the authors' opinion on this issue.

Author response:
**Q4: The information amount of generated short prefixes**

Regarding the concern that generated short prefixes may lack information amount, we think this information can be obtained from the information stored in the parameters of the large language model. When the language model can accurately recall relevant snippets, a short prefix is enough to locate the target snippet. However, when the recall ability of the language model is not strong enough, we think generating multiple snippet parts or using the multi-stage method like our method can help improve recall.

Reviewer comment (Reproducibility):
What is the memory cost of using prefix tree and FM-index structures? Does this work require significant use of additional memory to store these indexes?

Author response:
**Q5: Memory usage of storage**

As analyzed in Section 4.4 on storage, dense retrieval methods require about 60G memory, sparse retrieval BM25 needs about 17G, while our method only needs 25mb for the title prefix tree, and 8G for the pre-built full Wikipedia FM-index. Our method has more efficient memory utilization compared to previous methods.

Reviewer comment (Experiments):
The length of the short prefix l_(p_s) was not explicitly given, the reviewer would like to know how the authors decide the length of short prefixes and how does the length of the short prefix contribute to the retrieval performance?

Author response:
**Q6: The impact of short prefix length**

In Appendix A.5 we experimented the impact of different prefix lengths on performance. We observed that over-short prefixes may not contain enough information, and shorter lengths make it hard to locate a unique passage, leading to performance drop. When generating longer prefixes, there is also a performance drop, slightly counter-intuitive. We analyzed this may be because current large language models are still more suitable for generating and retrieving shorter snippets, and longer prefixs introduce additional noise, causing performance degradation.