Paper: Synergistic Information Retrieval: Interplay between Search and Large Language Models (ICLR-2024)

Reviewer comment (Theory):
The process depending on an LLM might not offer the same guarantees as trained retrieval models, as the LLM is a black box in this case.

Author response:
Thanks for your great comment. On one hand, in this work we not only utilized the closed-source GPT but also employed the open-source Vicuna, hence it is not entirely a black box as we can get access to the model parameters. On the other hand, most present advanced LLMs are trained on large-scale web data, exhibiting a considerable degree of generalization and impressive performance on knowledge-intensive tasks. And some products even use LLMs for search (New Bing/Perplexity AI). Our experimental results also demonstrate that the LLM aids in the iterative information refinement for RM parts, and the overall framework achieves new state-of-the-art zero-shot retrieval performance.

Reviewer comment (Experiments):
How does the performance vary with different LLMs?

Author response:
The performance of different models depends on various factors such as training compute, model parameters, etc [2]. As depicted in Table-1, the open-source Vicuna-33B-v1.3 yields better results on DL'19 and DL'20 compared to the Vicuna-13B-v1.5 model (despite Vicuna-13B-v1.5 being from LLaMa-2 whereas Vicuna-33B-v1.3 is from LLaMa-1). This to some extent suggests that as the LM within InteR grows larger, the effectiveness of zero-shot retrieval improves.

    - [2] Jared Kaplan et al., 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

Reviewer comment (Evaluation):
What is the overall cost (computational) compared to other approaches?

Author response:
Thank you for your constructive suggestions! As the RM part utilizes BM25 which is much faster, the primary bottleneck lies within the LM. For closed-source ChatGPT models, the latency cannot be accurately ascertained due to the API nature. Hence, we additionally included the latency result of open-source models Vicuna-13B-v1.5 on 1 GPU of A100-40GB for each iteration using vLLM [3] in our revised manuscript. Then we observed that it can handle 20.16 requests per minute, which can be further optimized with more advanced LM inference tools.

    - [3] https://github.com/vllm-project/vllm

Reviewer comment (Experiments):
More experimental details on the effect of hyperparameters is needed given the very empirical nature of the approach.

Author response:
Thank you for your suggestion. We tuned the hyperparameters on validation sets of TREC DL data. We have included these details in our revised manuscript.

Reviewer comment (Writing):
DeepCT is not an unsupervised model.

Author response:
Thank you for your reminder, we have corrected this mistake in our revision.

Reviewer comment (Evaluation):
The method achieves strong accuracy, but its latency must have been severely compromised as it involves multiple rounds of LLM and RM querying. This makes this method unideal for most retrieval applications where latency is prioritized. Would be great to include the latency results so that readers can better understand the tradeoff.

Author response:
Thank you for your constructive suggestions! As the RM part utilizes BM25 which is much faster, the primary bottleneck lies within the LM. For closed-source ChatGPT models, the latency cannot be accurately ascertained due to the API nature. Hence, we additionally included the latency result of open-source models Vicuna-13B-v1.5 on 1 GPU of A100-40GB for each iteration using vLLM [1] in our revised manuscript. Then we observed that it can handle 20.16 requests per minute, which can be further optimized with more advanced LM inference tools.

    - [1] https://github.com/vllm-project/vllm

Reviewer comment (Evaluation):
One part that is missing is an analysis of the side- or negative- impacts of the method. The iterative expansion will likely sometimes introduce information with negative effects. e.g. irrelevant info, contradictory logics, hallucinated content, lexically similar but semantically irrelevant texts, etc. How often does this happen? How does this hurt performance? Any potential mitigation strategy?

Author response:
Thanks for your insightful comments and suggestions!  We acknowledge that iterative information refinement can sometimes lead to negative effects such as the introduction of irrelevant information and hallucinated content. As outlined in the Limitation Section, we have also identified that enhancing the quality control in prompt formulation and query expansion is crucial for further improving the model. Since our study aimed to validate the effectiveness and universality of InteR framework, we did not excessively design this part and retained only the most fundamental components. We are glad to investigate this to future improve our method in the future. Thank you again for your great suggestions.

Reviewer comment (Presentation):
Are the authors from Microsoft? If so, there is a potential **RISK OF IDENTITY LEAK** as they use the Bing logo in Figure 1 but the experiment setting is totally irrelevant to Bing. I am just pointing out the risk here and will let the meta-reviewer decide whether this is an issue or not.

Author response:
Thanks for your kind reminder and sorry for this misunderstanding. We randomly selected a commonly used search engine for daily purposes for the logo. We have rectified this in the revised version. Thank you once again for the kind reminder!

Reviewer comment (Experiments):
On what dataset do you tune the hyperparameters on? If it is the test set, then it largely weakens the reliability of the conclusions.

Author response:
Thank you for the kind reminder! We tuned the hyperparameters on validation sets of TREC DL data. We have also included the missed references in our revision.

Reviewer comment (Novelty):
Another area of related work to cite and discuss is query rewriting in conversational QA (e.g. QReCC), where a NLG model is used to refine the query to augment retrieval.

Author response:
Thank you for the kind reminder! We tuned the hyperparameters on validation sets of TREC DL data. We have also included the missed references in our revision.

Reviewer comment (Novelty):
The idea is not new, as You.com and Bing.com are doing this, and Google Bard undoubtedly has developed InTeR-like capabilities.

Author response:
- Thank you for your valuable comment. To the best of our knowledge, we are not acquainted with the technical details of these two products, as neither product discloses its technical details nor releases technical reports. Combining retrieval models and LLM is a popular direction now, while most existing works only pursue single-step/mono-direction enhancement like “Enhance Retrieval Through LMs” or “Enhance LMs Through Retrieval”. In our work, we are among the earliest to propose iterative information refinement through synergy between retrieval models and LLMs, resulting in more accurate retrieval than state-of-the-art approaches and even outperforming baselines that leverage relevance judgment for supervised learning. We humbly believe that our paper has made a unique contribution and may offer insights for future research in this field.

Reviewer comment (Evaluation):
RMs require well-crafted keywords to deliver accurate results, which might not be user-friendly. Query formulation is a problem to be addressed.

Author response:
- Yes, because the query requires being well-crafted, the retrieval performance is not satisfactory. Therefore, we employ LLM for iterative information refinement of RM queries, aiding in better comprehension of the query. Query formulation, specifically designed in our proposed framework, is indeed adopted to improve retrieval results.