F lexi QA : Leveraging LLM ’s Evaluation Capabilities for Flexible Knowledge Selection in Open-domain Question Answering (2024)

Abstract: Nowadays, large language models (LLMs) have demonstrated their ability to be a powerful knowledge generator of generate-then-read paradigm for open-domain question answering (ODQA). However this new paradigm mainly suffers from the “hallucination” and struggles to handle time-sensitive issue because of its expensive knowledge update costs. On the other hand, retrieve-then-read, as a traditional paradigm, is more limited by the relevance of acquired knowledge to the given question. In order to combine the strengths of both paradigms, and overcome their respective shortcomings, we design a new pipeline called “FlexiQA”, in which we utilize the diverse evaluation capabilities of LLMs to select knowledge effectively and flexibly. First, given a question, we prompt a LLM as a discriminator to identify whether it is time-sensitive. For time-sensitive questions, we follow the retrieve-then-read paradigm to obtain the answer. For the non time-sensitive questions, we further prompt the LLM as an evaluator to select a better document from two perspectives: factuality and relevance. Based on the selected document, we leverage a reader to get the final answer. We conduct extensive experiments on three widely-used ODQA benchmarks, the experimental results fully confirm the effectiveness of our approach.

## introduction open-domain question answering (odqa) as a knowledge-intensive task, necessitate a substantial amount of world knowledge to be effective @xcite . current methods for handling odqa often share two common paradigms: the retrievethen-read paradigm, which consists of retrieving a small set of relevant contextual documents from sources, and then generating the answer on both the question and the retrieved documents @xcite @xcite ; and the generate-then-read paradigm, which initiates by prompting an llm to generate contextual documents based on the question, then by reading and extracting relevant information from the generated documents to generate the final answer. nevertheless, these two type of paradigms are with their own drawbacks. for the retrieve-then-read paradigm, candidate documents are chunked and fixed for a given question. moreover, the frequently-used two-tower dense retrieval models @xcite often leads to superficial interactions between the document and the question @xcite . these can result in some retrieved documents containing irrelevant or noisy data that is not pertinent to the question. for the generate-then-read paradigm, though there are works show that the generated contextual documents contain the correct answer more often than the top retrieved documents @xcite , there are still some imperative issues to be solved. llms are hard to expand or revise their memory since all the information needs to be stored in the parameters @xcite . moreover, they can't straightforwardly provide insight into their generations, and may produce "hallucinations" @xcite or struggle to address time-sensitive issue. a time-sensitive question is one whose answer will change over time. for example, where will the next olympic games be held? is time-sensitive, while who wrote the book 'the razor's edge'? is not time-sensitive. time-sensitivity becomes a nonnegligible issue when leveraging llms for odqa. there are a few works analyzed it recently @xcite , but didn't try to solve it. meanwhile, retrieval-based models have no such problem because it is easy to replace the external knowledge source and access the timealigned documents. based on the aforementioned observations, we unify these two paradigms and proposed a new odqa pipeline called flexiqa. overall, our contributions are listed as follows: ## method of flexiqa under the zero-shot setting, we will introduce the details of our proposed pipeline as shown in figure 1 which comprises three steps: time-sensitivity discrimination, document selection, answer generation. first, we prompt an llm to discriminate if the given question is time-sensitive. if the answer is yes, we choose the retrieved document as the evidence. otherwise, we further prompt the llm as an evaluator to decide which document (one is from generation, another one is from retrieval) is better from two perspectives: factuality and relevance. and finally we use the picked document as evidence to answer the given question by a reader. ## conclusion in this paper, we unify two classic odqa paradigms and propose a new pipeline called flex-iqa. flexiqa leverages the multi-dimensional evaluation ability of llms flexibly for odqa for the first time, and it tackles three existing drawbacks in the two classic paradigms: the timesensitive issue, the irrelevance issue and the nonfactuality issue. moreover, we release two timesensitivity annotated datasets for widely research on this issue in the future. experimental evaluations show that our model achieves the best performance on three datasets.