Accurate and Nuanced Open- QA Evaluation Through Textual Entailment (2024)

Abstract: Open-domain question answering (Open-QA) is a common task for evaluating large language models (LLMs). However, current Open-QA evaluations are criticized for the ambiguity in questions and the lack of semantic understanding in evaluators. Complex evaluators, powered by foundation models or LLMs and pertaining to semantic equivalence, still deviate from human judgments by a large margin. We propose to study the entailment relations of answers to identify more informative and more general system answers, offering a much closer evaluation to human judgment on both NaturalQuestions and TriviaQA while being learning-free. The entailment-based evaluation we propose allows the assignment of bonus or partial marks by quantifying the inference gap between answers, enabling a nuanced ranking of answer correctness that has higher AUC than current methods.

## introduction open-domain question answering (open-qa) is a long-established task requiring systems to generate precise answers to factual questions on any topic, from information in a large corpus of text @xcite . a more restricted form of open-domain qa where answers are short is still regarded as challenging and as a reasonable test for the capabilities of recent large language models (llms) @xcite , particularly when it comes to the assessment of llm honesty @xcite , calibration @xcite . open-qa benchmarks @xcite @xcite ; inter alia), consisting of pairs of curated questions and manually-annotated gold answers, have been under intense scrutiny because current automated evaluations have been found primitive, flawed, and insufficient to capture the true capabilities of open-qa systems @xcite @xcite . open-qa evaluators. let s be the set of finite strings. given a question @xmath0, an open-qa system generates a free-text system answer @xmath1, while reference gold answer(s) a * ∈ a * ⊂ s are provided by humans. in the typical setting, an evaluator f : q × a × a * → {0, 1} compares the system answer a with the gold answer(s) a * to provide an evaluator judgment of whether the system correctly answered the question q. while a wide variety of evaluators would be possible, current open-qa benchmarks resort to fairly strict and primitive evaluators, which do a poor job with under-specified questions or when the system provides an answer that is either more general or more specific than the gold standard, and are believed to have hindered the understanding of llm's "emergent" abilities @xcite . ambiguity in open-qa benchmarks. questions from open-qa benchmarks are often ambiguous and under-specified @xcite , leading to multiple possible answers that are not always covered by the gold answers @xcite . figure 1 presents an example from the naturalquestions benchmark @xcite where "oak island" is the sole gold answer to the question "where is the tv show the curse of oak island filmed?". however, due to the lack of specificity, a case can be made that more specific answers such as "on oak island, a small island off the coast of nova scotia, canada", or more general answers such as "nova scotia, canada" should be accepted. the former covers the gold answer and provides more details, while the latter has a lower level of specificity than the gold answer. exact word matching, a commonly used evaluator, fails with both answers. more advanced automated evaluations, including semantic similarity models and llm in-context learning, are also shown to be incapable of capturing such intricacies and deviate from human judgment @xcite . our contributions. we study semantic relations between system answers and gold answers that consider whether system answers cover the gold answer while providing more details, or vice versa. naturally, textual entailment, the task of determining whether a piece of text entails or contradicts another, is a suitable and training-free tool for this categorization. we propose to use textual entailment for the evaluation of open-qa systems, and show that entailment-based evaluation metrics, even when used without finetuning, are consistent with human judgments and are more effective in capturing the true capabilities of a range of open-qa systems when evaluating system answers on both naturalquestions (nq) and trivi-aqa (tq; @xcite . we also propose to use entailment-based evaluation metrics to assign bonus or partial marks to system answers by quantifying the inference gap between system answers and gold answers. we argue that our metric offers a more informative and fairer alternative to current binary evaluation metrics. such a more accurate and nuanced qa evaluation scheme is valuable in solidifying the large body of concurrent studies that build on short-answer qa evaluations. ## related work typical open-qa evaluators rely on exact word match accuracy (lexical match), f 1 score over word matches (formally defined in @xcite ), some semantic similarity model such as bertscore @xcite or bleurt @xcite , or zero-shot or in-context learning using an llm @xcite . these approaches have been under scrutiny, however. @xcite , @xcite and @xcite have looked at quantitatively assessing the correctness of automated evaluators by comparing the judgments they produce against gold judgments produced by human experts. they find that unsupervised automated evaluators, including those powered by pre-trained foundation models and llms, are not consistent with human judgments. @xcite released the evouna dataset for the evaluation of automated evaluators, with 3,020 questions from nq and 1,938 from tq. ques-tions are filtered to exclude those with outdated gold answers, and system answers generated by state-of-the-art open-qa systems, namely dpr @xcite ) + fid @xcite , instructgpt and chatgpt @xcite , @xcite , and bingchat, are annotated with gold judgments. an ideal automated evaluator should produce judgments that are consistent with human judgments, and thus, have higher f 1 scores and accuracies when evaluated against human judgments. we base our work on evouna and study the relation between gold answers and system answers, and subsequently derive an entailment-based evaluator f that is more consistent with human judgments. we also extend the range of f from a {0, 1} binary prediction to r to achieve a more informative and fairer evaluation. to overcome traditional metrics' lack of semantic understanding and, henceforth, underestimation of performance, recent efforts have focused on understanding the semantic equivalence of answers and developing evaluators accordingly @xcite @xcite . we argue that the semantic equivalence is not the only relation between valid system answers and gold answers. a valid answer can range from a vague, less informative one (e.g., a range, time period, or region) to a very specific and detailed answer (e.g., a precise number, time, or location). @xcite proposed to accept all answers that contain at least all relevant content of the gold answer and no misleading content, while making no explicit distinction between semantic equivalence and entailment. we extend this idea and propose to assign a partial order to system answers with regard to how much relevant information the answer contains relative to the gold answer. ## the answer hierarchy let a * denote the set of gold answers for an open-qa benchmark. we define two other sets of answers: a sup is the set of superior answers that provide more information than what is in the gold standard, and a inf is the set of inferior answers that only address the question partially. given a system answer a and the corresponding gold answer a * , we say that @xmath2. (this naturally extends to the case where multiple gold answers are given). similarly, we say that @xmath3. finally, an answer would be figure 1 : qa systems may generate a variety of correct answers that are neither exact matches nor semantic equivalents of the gold answer. judging by the amount of information relevant to the gold answer that the system answers provide, we obtain a partial order of system answers with respect to the gold answer using textual entailment, and group answers into a hierarchy of subsets. before textual entailment, question-answer pairs are rewritten as declarative statements (introduced by @xcite as qa2d) using gpt-3.5, as the question and context are important for assessing answers @xcite . for example, the gold answer in figure 1 is converted into the statement "the tv show the curse of oak island is filmed on oak island". inspired by llm's strong performance in natural language inference tasks @xcite , we use gpt-3.5 to conduct textual entailment tests. in the finalized approach, two steps, converting a question-answer pair to a declarative statement, and performing textual en-tailment test on declarative statements, are perform by a llm in a few-shot manner. we validate that for both steps gpt-3.5 as the llm achieves high statistical reliability (appendix a.2) in terms of the agreement across different seeds, and high validity (appendix a.3) in terms of the alignment with human labels. implementation details and a worked example are provided in appendix a.1. system answers deserve partial credits and bonus credits. the above-described entailmentbased evaluator reveals that a sup ⊕ a inf (the disjoint union) represents a considerable amount of valid system answers that would otherwise be disregarded ( f 1 acc llama-2 (sft) 94.6 92.3 llama-2 + nli (sft) 94.8 92.6 cvi 84.7 73.5 entailment (0-shot) 93.5 90.2 the answer hierarchy is a superior automated evaluator. treating a sup ∪ a inf as correct answers 1 bring the evaluation results of various systems closer to human performance, as shown in although learning-free, entailment is comparable to finetuned evaluators. @xcite and @xcite advocate for learned evaluators to close the gap between automated and human evaluation. for comparison, we partition evouna-nq by questions into 50:50 train/test splits, and finetune a llama-2 @xcite model which only performs slightly better than our method. we show that explicitly including entailment as a feature improves the finetuned model (+nli in ## conclusion in theory, textual entailment is considered ai-complete @xcite -an embodiment of general ai that solves all ai tasks, open-qa evaluation included. in practice, we showed that state-of-the-art textual entailment provides a simple yet powerful replacement for open-qa evaluation, and it offers the prospect of soft and partial marks.