Paper: FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (ICLR-2024)

Reviewer comment (Presentation):
For better visualization, the best results in Table 1 need to be displayed in bold.

Author response:
We have uploaded a new version of our paper where the best results in tables of results are bolded.

Reviewer comment (Evaluation):
Dataset and paper lacks any type of automatic evaluation metric, everything is based on human eval (major)

Author response:
We thank the reviewer for the constructive review. We apologize for the delayed response. We spent the majority of the rebuttal period implementing and evaluating the auto-rater suggested by the reviewer. We are happy to report that it works very reliably and can thus be used in place of human evaluation, and we thank the reviewer for directly improving the accessibility and impact of our FreshQA benchmark.

1. The reviewer inquired about the number of human raters and whether future work will follow the same evaluation protocol. All model responses underwent a rigorous evaluation process conducted by two authors using our two-mode evaluation protocol. We believe that human evaluators possess the expertise and common sense required to detect issues like hallucinations, making them more reliable than automated evaluation metrics for assessing LLMs' factuality. However, researchers have the flexibility to adjust their evaluation methods if human evaluation proves challenging. An easily implemented alternative is to use standard metrics like F1/exact match or recall, which assess the overlap between the model response and the ground truth answer(s) (e.g., see You.com's recent evaluation: https://about.you.com/introducing-the-you-api-web-scale-search-for-llms/). Alternatively, researchers can utilize LLM-based automatic evaluation metrics such as FactScore (https://arxiv.org/abs/2305.14251) or our newly developed FreshEval metric (details below).


2. The reviewer asked about implementing an auto-rater for FreshQA. To facilitate future evaluations, we have developed FreshEval, a simple automatic metric that uses few-shot in-context learning to teach an LLM to assess model responses, achieving an average agreement of 96.5% with human evaluations for Relaxed and 96% for Strict (please refer to Appendix B in the updated version of our paper).  In each evaluation, the model is conditioned on a given question, a list of valid answers for the question, and a model responses, and is then expected to generate a comment on the correctness of the response, followed by a final judgment (please see Figure 5 and Figure 6 for FreshEval’s prompts, and Figure 7 for its sample output). We are in the process of re-evaluating all models with FreshEval.

Reviewer comment (Experiments):
Methodology for retrieval seem to be tailored to Google Search (as the authors acknowledge, minor)

Author response:
5. As for FreshPrompt's compatibility with other search engines, we believe it can readily adapt. FreshPrompt's success depends on the number and order of retrieved evidences, which are typically available from most search engines.

Reviewer comment (Evaluation):
The evaluation in this paper is performed exclusively by human annotation, which raises concerns about usability when providing the dataset to the community for further research.

Author response:
We thank the reviewer for the constructive review. We apologize for the delayed response. We spent the majority of the rebuttal period implementing and evaluating the auto-rater suggested by the reviewer. We are happy to report that it works very reliably and can thus be used in place of human evaluation, and we thank the reviewer for directly improving the accessibility and impact of our FreshQA benchmark.

1. The reviewer inquired about the number of human raters and whether future work will follow the same evaluation protocol. All model responses underwent a rigorous evaluation process conducted by two authors using our two-mode evaluation protocol. We believe that human evaluators possess the expertise and common sense required to detect issues like hallucinations, making them more reliable than automated evaluation metrics for assessing LLMs' factuality. However, researchers have the flexibility to adjust their evaluation methods if human evaluation proves challenging. An easily implemented alternative is to use standard metrics like F1/exact match or recall, which assess the overlap between the model response and the ground truth answer(s) (e.g., see You.com's recent evaluation: https://about.you.com/introducing-the-you-api-web-scale-search-for-llms/). Alternatively, researchers can utilize LLM-based automatic evaluation metrics such as FactScore (https://arxiv.org/abs/2305.14251) or our newly developed FreshEval metric (details below).

Reviewer comment (Evaluation):
It is unclear how many annotators were used to perform each system's evaluation, and although inter-annotator agreement between two evaluators on a subset of 100 questions is reported and shown to be high, it is uncertain whether the authors expect all researchers using the dataset to follow the same evaluation protocol.

Author response:
1. The reviewer inquired about the number of human raters and whether future work will follow the same evaluation protocol. All model responses underwent a rigorous evaluation process conducted by two authors using our two-mode evaluation protocol. We believe that human evaluators possess the expertise and common sense required to detect issues like hallucinations, making them more reliable than automated evaluation metrics for assessing LLMs' factuality. However, researchers have the flexibility to adjust their evaluation methods if human evaluation proves challenging. An easily implemented alternative is to use standard metrics like F1/exact match or recall, which assess the overlap between the model response and the ground truth answer(s) (e.g., see You.com's recent evaluation: https://about.you.com/introducing-the-you-api-web-scale-search-for-llms/). Alternatively, researchers can utilize LLM-based automatic evaluation metrics such as FactScore (https://arxiv.org/abs/2305.14251) or our newly developed FreshEval metric (details below).

Reviewer comment (Evaluation):
The absence of an automatic evaluation metric may limit the dataset’s broad usefulness and make results subject to annotator variance.

Author response:
We thank the reviewer for the constructive review. We apologize for the delayed response. We spent the majority of the rebuttal period implementing and evaluating the auto-rater suggested by the reviewer. We are happy to report that it works very reliably and can thus be used in place of human evaluation, and we thank the reviewer for directly improving the accessibility and impact of our FreshQA benchmark.

2. The reviewer asked about implementing an auto-rater for FreshQA. To facilitate future evaluations, we have developed FreshEval, a simple automatic metric that uses few-shot in-context learning to teach an LLM to assess model responses, achieving an average agreement of 96.5% with human evaluations for Relaxed and 96% for Strict (please refer to Appendix B in the updated version of our paper).  In each evaluation, the model is conditioned on a given question, a list of valid answers for the question, and a model responses, and is then expected to generate a comment on the correctness of the response, followed by a final judgment (please see Figure 5 and Figure 6 for FreshEval’s prompts, and Figure 7 for its sample output). We are in the process of re-evaluating all models with FreshEval.

Reviewer comment (Evaluation):
The authors could potentially have created an automatic evaluation metric by designing questions with multiple-choice answers and carefully chosen hard negative options, similar to strategies already used in benchmarks like MMLU or the original T0 tasks, which would allow computing accuracy. It is unclear why the authors did not choose this option.

Author response:
3. The reviewer suggested the possibility of reformatting FreshQA into a multiple-choice format, and we appreciate this input. Researchers are encouraged to consider reformatting FreshQA into other formats (e.g., multiple-choice QA, timestamped QA), if it aligns with their specific evaluation needs and objectives. Our dataset serves as an open-ended QA benchmark, allowing us to assess not only the correctness of responses but also the presence of any hallucinations. While adopting a multiple-choice format might streamline evaluation, there are several drawbacks to consider: (1) When presented with multiple-choice questions, models can potentially rely on heuristics to deduce the correct answer rather than genuinely understanding the context; (2) Assessing hallucinations becomes challenging in a multiple-choice setup, as models are simply required to select from predetermined options; and (3) Recent models are designed for open-ended responses, and real-time QA often does not involve presenting multiple choices. Thus, retaining FreshQA's open-ended evaluation can be valuable for assessing models' practical utility in real-world applications.

Reviewer comment (Reproducibility):
There is uncertainty about the broad usability of the dataset in its current state without automated evaluation mechanisms.

Author response:
We thank the reviewer for the constructive review. We apologize for the delayed response. We spent the majority of the rebuttal period implementing and evaluating the auto-rater suggested by the reviewer. We are happy to report that it works very reliably and can thus be used in place of human evaluation, and we thank the reviewer for directly improving the accessibility and impact of our FreshQA benchmark.

2. The reviewer asked about implementing an auto-rater for FreshQA. To facilitate future evaluations, we have developed FreshEval, a simple automatic metric that uses few-shot in-context learning to teach an LLM to assess model responses, achieving an average agreement of 96.5% with human evaluations for Relaxed and 96% for Strict (please refer to Appendix B in the updated version of our paper).  In each evaluation, the model is conditioned on a given question, a list of valid answers for the question, and a model responses, and is then expected to generate a comment on the correctness of the response, followed by a final judgment (please see Figure 5 and Figure 6 for FreshEval’s prompts, and Figure 7 for its sample output). We are in the process of re-evaluating all models with FreshEval.

Reviewer comment (Reproducibility):
The evolving nature of FreshQA could pose challenges for researchers aiming for consistent benchmarks over varied time frames.

Author response:
The reviewer is concerned about potential challenges posed by FreshQA’s dynamic nature for consistent benchmarks. While FreshQA may be affected by the ever-evolving nature of information, we believe that it remains a valuable resource for the following reasons:

     * First, FreshQA's dynamic nature mirrors real-life scenarios where information is continually updated. This makes it a suitable benchmark for evaluating LLMs’ ability to handle dynamic and ever-changing information or provide answers based on the most recent updates. These evaluations demonstrate their practical utility in real-world applications, where knowledge updates occur in real-time. Researchers are encouraged to use FreshQA for evaluating models that can continuously learn and adapt to new information.

     * Second, researchers can adapt their evaluation methodologies to account for FreshQA's dynamic nature when needed. One approach is to use various “snapshot” versions of FreshQA that freeze answers at specific time points and include timestamps on questions to indicate when they were asked, similar to prior work such as TimeQA and SituatedQA. Alternatively, researchers can assess performance on a subset of the dataset where the relevant information (e.g., question type) has remained consistent over time. These methods help establish stable benchmarks for different time frames, although they may not fully capture the dynamic nature of real-world information.

Reviewer comment (Novelty):
The FreshQA dataset bears similarities with RealTimeQA and TimeQA, which somewhat dilutes the novelty of this work, although it remains a complementary addition.

Author response:
Regarding the reviewer’s comment on FreshQA’s novelty, below we elaborate on the differences between FreshQA, TimeQA, and RealTime QA, and highlight the unique merits and contributions that FreshQA brings to the field.

     * First, while TimeQA contains mostly synthesized questions, FreshQA exclusively features human-generated natural questions, enhancing the dataset's authenticity and real-world relevance. Additionally, TimeQA is static, with time-stamped questions and fixed answers. Therefore, it does not require a model to access real-time data or browse the Internet for current information. In contrast, FreshQA is a dynamic benchmark where answers may change over time, requiring a model to understand the world's up-to-date knowledge to answer correctly.

     * Second, while RealTime QA emphasizes real-time information updates and how LLMs handle them, it exclusively focuses on news data. It re-evaluates models with approximately 30 new multiple-choice questions about recent events from news websites, annotated weekly by its authors. In contrast, FreshQA features a fixed set of open-ended questions covering a wide range of topics, including arts, music, politics, government, religion, science and technology, environment, transportation, sports, and more. These questions naturally allow for changing answers based on new developments in the world.

     * Third, FreshQA covers a wide variety of question types, including questions that involve old or never-changing world knowledge, questions with false premises that need to be debunked, and single-hop/multihop questions. This diversity of question types is essential for mitigating model bias and preventing overfitting. It guarantees that models' performance remains balanced across different question categories and does not become overly optimized for one category at the expense of others.

     * Lastly, our two-mode evaluation protocol, comprising relaxed and strict evaluations, offers a spectrum for assessing LLMs' factuality. This approach not only provides a deeper and more nuanced understanding of their performance but also quantifies hallucination, a crucial aspect absent in previous QA benchmarks.

Reviewer comment (Novelty):
The proposed method isn't entirely groundbreaking, given precedents like internet-augmented LLM [1] and REPLUG [2]. [1] Internet-augmented language models through few-shot prompting for open-domain question answering [2] Replug: Retrieval-augmented black-box language models.

Author response:
The reviewer raised a question about FreshPrompt’s novelty. While FreshPrompt is not the first few-shot prompting method that integrates current information from a search engine into LLMs, it introduces several innovations. These innovations involve fully leveraging a search engine by extracting all up-to-date and relevant information (including knowledge from related questions), and  efficiently handling an increasing number of retrieved evidences (including casting them into a unified format and reasoning over them to identify the most relevant and up-to-date answer). These contributions have resulted in performance improvements. Our analysis provides insights for future work on search engine-augmented LLMs. We found that both the number and order of retrieved evidences significantly impact the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers has been shown to reduce hallucination.

     Compared to Lazaridou et al. (2022)'s approach, which involves using a single retrieved evidence at a time and requires 50 inference calls to an LLM for each question to generate multiple candidate answers and select the best one, FreshPrompt is more efficient, requiring only a single inference call to an LLM.