F resh LLM s: Refreshing Large Language Models with Search Engine Augmentation (2024)

Abstract: Since most large language models (LLMs) are trained once and never updated, they struggle to dynamically adapt to our ever-changing world. In this work, we present FreshQA, a dynamic QA benchmark that tests a model’s ability to answer questions that may require reasoning over up-to-date world knowledge. We develop a two-mode human evaluation procedure to measure both correctness and hallucination, which we use to benchmark both closed and open-source LLMs by collecting >50K human judgments. We observe that all LLMs struggle to answer questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. In response, we develop FreshPrompt, a few-shot prompting method that curates and organizes relevant information from a search engine into an LLM’s prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. To facilitate future work, we additionally develop FreshEval, a reliable autorater for quick evaluation and comparison on FreshQA. Our latest results with FreshEval suggest that open-source LLMs such as Mixtral (Jiang et al., 2024), when combined with FreshPrompt, are competitive with closed-source and commercial systems on search-augmented QA.

## introduction despite their impressive capabilities, modern llms often "hallucinate" plausible but factually incorrect information @xcite , which reduces their trustworthiness especially in settings where accurate and up-to-date information is critical. this behavior can be partially attributed to the presence of outdated knowledge encoded in their parameters. while additional training using human feedback @xcite or knowledge-enhanced tasks can mitigate this issue, it is not easily scalable for real-time knowledge updates (e.g., stock prices). in-context learning @xcite ) is an appealing alternative by which real-time knowledge can be injected into an llm's prompt. while recent work has begun to explore augmenting llm prompts with web search results @xcite , it is unclear how to take full advantage of search engine outputs to increase llm factuality. in this work, we collect freshqa, a novel benchmark to evaluate the factuality of llm generations. freshqa consists of 600 natural questions that are broadly divided into the four main categories shown in figure 1 . freshqa's questions span a diverse set of topics with diverse difficulty levels (requiring single-hop and multi-hop reasoning), and require a model to "understand" up-to-date world knowledge to be able to answer correctly. additionally, freshqa is dynamic in nature: some of the ground-truth answers may change over time, and a question classified under a specific category may undergo reclassification at some later point in time (e.g., the current false-premise question "how long has elon musk been married to his current spouse?" will fall into the fast-changing category if elon musk gets married again in the future). we benchmark a diverse range of both closed and open-source llms under a two-mode evaluation procedure: relaxed, which measures only whether the main answer is correct; and strict, which measures whether all of the claims in the response are @xcite . through an extensive human evaluation (> 50k judgements), we shed light on limitations of these models and demonstrate significant room for improvement: for example, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. ## freshqa to address llm factuality assessment, we build freshqa, a dynamic benchmark with 600 questions covering diverse question and answer types. we collected freshqa by recruiting both nlp researchers foot_0 and online freelancers foot_1 to write questions of varying difficulty and topics (arts, music, politics, government, religion, science and technology, environment, transportation, sports, etc.), focusing on questions whose answers evolve over time. annotators were shown exemplars of the four broad question types (see figure 1 ) and asked to write questions at two difficulty levels: one-hop, where the question directly states all information needed for the answer (e.g., "who is the ceo of tesla"); and multi-hop, where the question requires additional reasoning steps to find the answer (e.g., "what country does the wimbledon women's champion play for?"). annotators were encouraged to write questions that involve fresh knowledge and appear natural as search engine queries. for falsepremise questions, we requested a brief explanation elucidating why the question is flawed. 3 quality control: upon obtaining the initial dataset, we performed rigorous data cleaning and quality checks. this included manual review for well-formed questions, removal of duplicates and invalid questions (e.g., too easy or controversial), and verification of answers and supporting urls. 4 data splits: freshqa is divided into a test set with 500 examples (125 per question type), a development set with 100 examples (25 per question type), and a 15-example demonstration set for fewshot learning. the development set is reserved for future use and was not used in this paper. 5 ## offline llms struggle on freshqa we first use freshqa to benchmark llms without access to real-time data or web browsing capabilities. 6 we simply feed individual questions into each model and use greedy decoding. while all llms (regardless of size) predictably struggle on questions requiring up-to-date knowledge, they also underperform on false-premise questions. baselines: we evaluate a series of models: t5 @xcite , palm and palmchilla @xcite (varying in size from 770m to 540b parameters), optionally using few-shot prompting @xcite and chain-of-thought (cot) prompting @xcite ; flan-t5 and flan-palm @xcite , gpt-3.5 @xcite , codex @xcite , chatgpt, and gpt-4. 7 see appendix c for more details. ## prompting search engine-augmented language models the low accuracies in section 3 are expected, as none of the evaluated models had access to real-time information. here, we evaluate the impact of search engine augmentation to llms on freshqa. we present freshprompt, a simple fewshot prompting method that substantially boosts llm factuality by incorporating up-to-date information from a search engine into the prompt. ## an updated evaluation in 2024 while our previous experiments were conducted in 2023, we use fresheval to evaluate new llms on an updated version of freshqa (february 5, 2024). we evaluate gpt-4, mixtral 8x7b, llama-2 @xcite , pplx.ai's online llm, and you.com's web llm on the same date of february 5, 2024. see appendix d for more details. as shown in time-sensitive qa: freshqa fits into a growing body of work benchmarking llms' temporal reasoning capabilities @xcite @xcite . @xcite created timeqa by extracting evolving facts from wikidata and then synthesizing timestamped question-answer pairs. zhang and choi (2021) constructed situatedqa by annotating realistic questions from existing open-domain qa datasets with temporal context (i.e., timestamps). streamingqa @xcite consists of both llm-generated and human-written questions, all answerable using a corpus of timestamped news articles. also related is the dynamic realtimeqa benchmark @xcite , which evaluates models weekly on multiple-choice questions about new events extracted from news websites. in contrast, freshqa contains a fixed set of human-written open-ended questions whose answers by nature can change based on new developments in the world and thus offers a complementary generative evaluation of time-sensitive qa. qa over questionable or counterfactual premises: recent work has also introduced qa benchmarks with questionable premises @xcite or counterfactual premises @xcite . crepe @xcite includes reddit questions with false premises annotated by human workers. @xcite constructed (qa) 2 using frequently searched queries annotated by expert annotators and crowdworkers, distinguishing between those with and without questionable premises. consistent with these efforts, we find that current llms struggle with false premise questions; additionally, several llms can debunk a false-premise question if explicitly asked to check for the premise's validity. similar to above, these benchmarks are complementary and combining them is a promising direction for future work. ## conclusion our work offers a fine-grained and exhaustive evaluation of the capabilities of modern llms to adapt to ever-changing world knowledge with and without search engine augmentation. in the process, we develop a new dataset-freshqa-of 600 questions that test a broad range of reasoning abilities, from the incorporation of fast-changing knowledge to identification of questions with false premises. our two-mode evaluation also provides a way to measure both correctness and hallucination. additionally, we propose a simple few-shot in-context learning algorithm called freshprompt that incorporates relevant evidences retrieved from a search engine into an llm's prompt. freshprompt significantly improves performance over competing search engine-augmented approaches on freshqa, and an ablation reveals that factors such as the number of incorporated evidences and their order impact the correctness of llm-generated answers. we release freshqa and commit to updating its answers regularly to facilitate future research. additionally, we develop fresheval, a reliable autorater for quick evaluation and comparison on freshqa.