Evaluating Open-Domain Question Answering in the Era of Large Language Models (2023)

Abstract: Lexical matching remains the de facto evaluation method for open-domain question answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate answer does not appear in the list of gold answers, which is increasingly the case as we shift from extractive to generative models. The recent success of large language models (LLMs) for QA aggravates lexical matching failures since candidate answers become longer, thereby making matching with the gold answers even more challenging. Without accurate evaluation, the true progress in open-domain QA remains unknown. In this paper, we conduct a thorough analysis of various open-domain QA models, including LLMs, by manually evaluating their answers on a subset of NQ-open, a popular benchmark. Our assessments reveal that while the true performance of all models is significantly underestimated, the performance of the InstructGPT (zero-shot) LLM increases by nearly +60%, making it on par with existing top models, and the InstructGPT (few-shot) model actually achieves a new state-of-the-art on NQ-open. We also find that more than 50% of lexical matching failures are attributed to semantically equivalent answers. We further demonstrate that regex matching ranks QA models consistent with human judgments, although still suffering from unnecessary strictness. Finally, we demonstrate that automated evaluation models are a reasonable surrogate for lexical matching in some circumstances, but not for long-form answers generated by LLMs. The automated models struggle in detecting hallucinations in LLM answers and are thus unable to evaluate LLMs. At this time, there appears to be no substitute for human evaluation.

## introduction reliable benchmarks have been a bedrock to measuring progress in open-domain qa, the task of an-figure 1 : examples of failures in open-domain qa evaluation. top: jicheng is a credible answer although not present in the list of gold answers. existing automated evaluation mechanisms fail to identify it as correct. bottom: a seemingly correct but unattributable answer from instructgpt @xcite for which automatic evaluation goes astray. with the recent success of generative qa systems in the open-domain setting @xcite , it becomes harder for lexical matching to recognize correct answers, and in turn for us, to recognize performance differences between models. the problem is exacerbated by a tendency of large language models(llm)-based systems @xcite @xcite ; inter alia) to occasionally hallucinate plausible but incorrect answers @xcite . for instance, in figure 1 , instructgpt @xcite generates "jack nicholson" in great details to answer who won the oscar for best actor in 1975? but although looks natural, the answer is not factually correct @xcite . therefore, human confirmation of answer correctness demands additional effort and care due to the ability of llms to formulate these answers as complete and seemingly authoritative. while it might be assumed that improved performance under lexical matching would reflect improved performance in an absolute sense, even if some correct answers are missed, we show this assumption does not hold. for this purpose, we manually re-evaluate several open-domain qa models on a random subset of nq-open @xcite , an established benchmark. not only is true performance substantially underestimated by this benchmark, but the relative performance of the models alters after re-evaluation: instructgpt (zero-shot) achieves an accuracy of 12.6% on our nq-open subset, but our human judgment reveals its true performance to be 71.4%, a nearly +60% improvement. our linguistic analysis of the failure cases of lexical matching, an extension of a similar study by @xcite , shows that the mismatches are mostly linguistically shallow and could be captured by simple patterns, such as regular expressions. in contrast, automated evaluation mechanisms such as bem @xcite based on semantic matching between the gold answers and generated answers produce a relative performance that is mostly consistent with human evaluation, although the absolute improvements are lower. however, long-form answers, generated by llms, introduce a new challenge that did not occur on prior models; they are prone to carry unattributable information @xcite . automated evaluation models often deem the hallucinated responses correct, which is why, instructgpt (zero-shot) is overestimated under these models, compared to human judgment. we repeated this experiment with the 20-yearold curatedtrec dataset @xcite that provides its gold answers in the form of regular expressions. we observe that the relative performance of models remains mostly consistent under all three evaluation mechanisms, i.e., regular expressions, human evaluation, and semantic matching, with only slight differences in absolute performance. however, the ranking discrepancy still persists between the two llms, i.e., instructgpt (zero-shot) and instructgpt (few-shot). also, only under human judgment does the absolute performance of llms exceed that of the heavily engineered statistical nlp systems from 20 years ago on this collection. until recently, the best of these classical systems has been substantially superior to even the best of the modern neural models. in light of our observations, we highlight that while semantic matching against exact answers would have been sufficient for qa evaluation prior to llms, they cannot accurately evaluate llms. human judgment in qa. many works @xcite resort to human evaluation to assess qa models. although using humans for evaluation is expensive and not scalable, @xcite find that the performance of qa systems bumps up 23% on average using human judgment. the substantial gap between the true performance and token-based metrics showcases the long known strictness problem of lexical matching. ## open-domain qa evaluation the task of open-domain qa is referred to finding answers for information-seeking questions given a massive knowledge source such as wikipedia @xcite . the questions are typically factoid with short answers and acontextual @xcite . open-domain qa datasets encompass questions with their annotated gold answers that serve as a reference for evaluation. following reading comprehension @xcite , evaluation is carried out via lexical matching using the following two widely used metrics to measure the performance of models: based on the implementation of @xcite , answers are normalized (i.e., case-folded, and punctuation and articles are discarded) to compute these metrics. ## strategies for evaluating open-domain qa models our goal is to shed light on the discrepancies between the actual and the measured accuracy of open-domain qa models. to this end, we adopt three evaluation mechanisms in addition to lexical matching to assess 12 open-domain qa models and draw a comparison between their estimated accuracy and the token-based performance. ## linguistic analysis of correct answers in this section, we aim to examine model answers that are not considered correct based on em, but are in fact acceptable according to our assessment. @xcite conducted a similar analysis on 50 questions for the participating models in the efficientqa competition at neurips 2020. in line with this work, we provide an in-depth analysis on a broader scale using more recent models to emphasize the drawbacks of widely used lexical-based evaluation metrics and semantic similarity methods. we further dissect the categories presented by @xcite into more detailed sub-categories. specifically, we group the 493 question/answer pairs that are deemed correct by humans while cannot be matched with gold answers into hierarchical categories as follows: 6 semantic equivalence: model predictions and gold answers convey the same meaning while not matching verbatim: ## regex matching on curatedtrec an alternative to lexical matching between gold answers and predicted answers during evaluation is to specify gold answers as regular expression patterns. regex matching allows for capturing syntactical answer variations where exact-match falls short. in this section, our main goal is to highlight 7 with only 3 exceptions: instructgpt-eval rejects only 2 actually correct answers matching with gold answers that correspond to list questions where candidate answers appear in the middle of the gold answers. moving the candidate answer to the top of the gold answer list would fix the issue. similarly, bem rejects only 1 exactly matched correct answer, i.e., "p-a-d-a-w-a-n." while the gold answer is "padawan". the advantages and pitfalls of using answer patterns in qa evaluation by comparing its results with our three evaluation mechanisms, described in §3.1. dataset. we make a comparison across opendomain qa models on curatedtrec 2002 @xcite , a dataset whose gold answers are specified by regular expressions. the questions in curatedtrec are derived from the dataset in the qa tracks @xcite of trec 2001 to 2003 after a manual review to discard ambiguous or outdated questions. the knowledge source for trec qa is originally english news text, namely aquaint, from three news sources (ap, nytimes, and xinhua), dating back to the late 90s. here, we opt for the original knowledge source to replicate the same environment as trec qa 2002 so as to quantitatively gauge progress over two decade by comparing recent models with the models that took part in the qa track in 2002. this experiment is an out-of-distribution test for the neural models to check whether they are actually capable of using the knowledge source to answer questions or they answer from memory because the old news articles is less likely to have appeared in the pretraining corpus. however, llms inevitably do not use the knowledge source as they perform the task from their memory in a closed-book fashion. cu-ratedtrec 2002 consists of 444 questions whose answers are looked up in the aqauint corpus, comprising around 1m news articles. we follow @xcite to split the articles into nonoverlapping passages of 100 words, which amounts to over 4m passages in total. models. out of the 12 models, we keep the ones that do not require further training on cu-ratedtrec 2002, leaving us with 7 models. these models produce 1872 unique answers on curat-edtrec 2002. we also obtained the submitted run files of the participants in the trec qa 2002 track from trec organizers to compute their accuracy on curatedtrec 2002. we include top 4 teams as baselines: lccmain2002 (88.1%; pasca and harabagiu 2001), pris2002 (62.4%), exactanswer (54.0%), and uwmtb3 (47.3%). similar to nq-open, we ask two annotators to judge 1872 question/answer pairs, followed by a third annotator who evaluates the diverging cases. the fleiss' kappa score between the two annotators is 83.5%, i.e., 150 disagreements (8.0%), indicating an almost perfect agreement. the results are shown in figure 5 . interestingly, the ranking of models via regex matching is left unchanged by all three evaluation mechanisms, except for instructgpt (zero-shot) and instructgpt (few-shot). consistent with our observation on nq-open, both bem and instructgpt-eval assign a higher accuracy to instructgpt (zero-shot) over in-structgpt (few-shot). however, in contrast to nq-open, they do not overestimate instructgpt (zeroshot). human evaluation shows that instructgpt (few-shot), by scoring 92%, is the best performing model, analogous to nq-open. among the non-llm models, ance+ and contriever consistently surpass other models. similar to em, regex matching is too rigid albeit to a lesser extent. in particular, the accuracy is underestimated by 6.6%, 6.4%, and 9.9% on average via bem, instructgpt-eval, and human evaluation, respectively. we note that lccmain2002, an original trec run, outperforms all models prior to our assessment. human evaluation highlights that both instructgpt models are superior to lccmain2002 by +1.9% (for zero-shot) and +2.9% (for few-shot). however, bem and instructgpt-eval fail to reflect this result. for other non-llm models, ance+ and contriever surpass pris2002 via all three evaluation methods (with the exception of contriever using instructgpt-eval). an interesting finding here is that although neural open-domain qa models are repeatedly proven to be powerful in accomplishing state-of-the-art, lccmain2002, a heavily engineered statistical method from 20 years ago, ruffles their feathers by a substantial margin of 20%. only under human judgment does the absolute performance of llms surpass lccmain2002. ## conclusion despite the simplicity and ubiquity of lexical matching as an evaluation metric in open-domain qa, it is unnecessarily rigid because plausible candidate answers are likely not to appear in the list of gold answers. this flaw has been long known, but the efforts to circumvent it have been mostly artisanal. in this paper, we report a systematic study of lexical matching by manually judging answers generated by several prominent open-domain qa models. we found that llms achieve stateof-the-art on nq-open. the accuracy of models is severely underestimated, with most em failure cases stemming from syntactical variations of answers. moreover, a zero-shot prompting method can be a reasonable substitute for human evaluation although it cannot detect unattributability in long-form answers. our insights and analysis in this paper will hopefully underpin the development of solid evaluation techniques in open-domain qa.