Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022)

Abstract: The predictions of question answering (QA) systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with predefined rules or with the token-level F1 measure.In this paper, we present the first systematic conceptual and data-driven analysis to examine the shortcomings of token-level equivalence measures.To this end, we define the asymmetric notion of answer equivalence (AE), accepting answers that are equivalent to or improve over the reference, and publish over 23k human judgements for candidates produced by multiple QA systems on SQuAD.Through a careful analysis of this data, we reveal and quantify several concrete limitations of the F1 measure, such as a false impression of graduality, or missing dependence on the question.Since collecting AE annotations for each evaluated model is expensive, we learn a BERT matching (BEM) measure to approximate this task. Being a simpler task than QA, we find BEM to provide significantly better AE approximations than F1, and to more accurately reflect the performance of systems.Finally, we demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to X2.6.

## introduction automatically assessing the answers given by question answering (qa) systems for correctness can be nontrivial. this was already recognized in the early large-scale qa evaluation work of voorhees 1 dataset and model information available at: https: //github.com/google-research-datasets/ answer-equivalence-dataset. the bem model can be found at: https://tfhub.dev/ google/answer_equivalence/bem/1 . and tice @xcite , leading them to suggest stringmatching patterns for approximation, yet recognizing their limitations: despite the early recognition of the importance and difficulty of evaluation for question answering, surprisingly little progress has been made. as of today, qa research in the nlp community relies almost exclusively on two token-level metrics, exact match (em) and token f 1 (f 1 ). unfortunately, both fall short of capturing the difference between significant and insignificant span differences. in section 2, we first document these limitations, providing a detailed analysis and examples from the squad dataset. moreover, we extend this analysis with our new findings: the inherent asymmetry of the task and the reliance on the question, also providing examples from the squad dataset. we also identify cases where the context is important. one obvious limitation of token-level measures is their direct dependence on the diversity of the reference answers collected for the dataset @xcite . while this could be addressed by extending the annotations, this is both expensive and has diminishing returns as the true collection of all correct answers might be large. in contrast, we focus on improving the equivalence measure beyond token matching and thereby increase the answer inclusiveness over any reference set. to facilitate research on this issue, we introduce a well-defined answer equivalence (ae) task along with the release of a new dataset (section 3). we collect human annotations on squad examples, comparing gold answers with model predictions. we utilize this data in several ways. first, we use the human judgments to better understand how well f 1 and em capture equivalence in the machine reading setting. we demonstrate that (1) both metrics underestimate the quality of candidate answers, (2) are highly reliant on the number of available references, and (3) that f 1 gives the false impression of a gradual rating, while really there is a similar ratio of equivalent to non-equivalent answers for non-zero f 1 . we further propose to learn a measure that approximates the ae relations (section 4). to study this, we introduce bert matching (bem), that uses a bert model trained on our data to predict equivalence. through multiple experiments with three qa systems, we show that bem better correlates with human judgments, compared to f 1 . finally, we demonstrate the utility of ae and bem on the concrete application of returning small and accurate answers sets (section 5). building on the expanded admission conformal prediction framework @xcite , we show that ae and bem significantly reduce the number of answers in the prediction sets while including a correct answer with arbitrarily high probability. thanks to its simplicity, bem can easily replace f 1 in many other applications that leverage qa predictions (e.g., @xcite @xcite @xcite . our main contributions include: the most popular metrics are token f 1 (f 1 ) and exact match (em), defined as follows: definition 2.1 (exact match). given a set of references a r and a candidate answer c, c is an exact match for a r iff @xmath0. definition 2.2 (token f 1 ). given a set of references a r , a candidate answer c and a tokenization function t, the token f 1 of c with respect to a r and t is the maximum over the token-wise f 1 scores between c and each @xmath1, i.e.: it is also common to remove stop words and punctuation before computing either em or f 1 . ## the ae task definition and dataset to address the issues mentioned in section 2, we require the answer equivalence relation to be asymmetric and to be conditional on both question and context. we want an ideal metric to give credit for a candidate answer that is at least as good as the reference, i.e. it should capture all the important information in the reference and not add irrelevant, or worse, misleading information. a candidate answer that improves over the reference by either removing misleading or irrelevant information, or adding more relevant information, should receive full credit. more formally, we define: definition 3.1 (answer relation). let q be a query and let a 1 , a 2 be answers contained in a context c. then a 2 is a good answer in place of a 1 if both the following are satisfied: note that this approach does not aim to replace the regular qa annotations, but expands on them to create larger, more inclusive sets of acceptable answers. most studies rely on token-level measures to approximate this expansion. however, as detailed in section 2, such measures are inadequate. ## predicting answer equivalence in the previous section we defined the ae task and discussed the value of such annotations. in practice, however, obtaining human ratings is slow and expensive. this is especially true for opendomain or generative qa models, where the space of possible answers is infinite. instead, we train a classifier to predict ae for any answer pair. we find the classifier's score to be more accurate ( §4.1.2), and provide practical gains ( §5). we conjecture that the high performance of the classifier is due to the easier nature of the ae task compared to qa. while, under some assumptions, the question-answering problem is np-complete @xcite , the verification of a candidate answer against an already given true answer is significantly simpler. the classification task. given the four-tuple (context, question, reference answer, candidate answer), predict the equivalence of the answers. ## example application: returning small and accurate prediction sets beyond the value of accurate performance assessment, identifying when the model answers are right can provide immediate practical gains. we demonstrate this on the common setting of constructing prediction sets (e.g., the top-k model's predictions). here, the accuracy is measured by whether a correct answer is included in the predicted set. the size of the set (k) is typically calibrated on an evaluation set, to the desired accuracy. reliably identifying high-ranking correct answers allows us to effectively reduce the size of the sets (smaller k) while keeping them accurate. here, we use the conformal prediction (cp) framework @xcite to construct provably accurate prediction sets. unlike top-k, cp set size is dynamically determined per input with instance-wise hypothesis testing to marginally satisfy the user-specified target accuracy. specifically, given an exchangeable calibration set of pairs of questions and correct answers, we measure their nonconformity scores (here, the negative predicted probability). then, for a new question, cp returns a set of candidate answers by including all answers with nonconformity scores smaller than the inflated α quantile of the calibration scores, where α is the target accuracy. while cp was originally defined for a single label per input, @xcite recently presented expanded admission cp to support multiple answers. this extension allows leveraging an answer equivalence function to reduce the size of the cp sets while preserving the accuracy guarantee. we follow the setting of @xcite and use the collected ae labels to construct accurate cp sets for squad 1.1 with luke's scores. as figure 2 shows the results over the full range of target accuracy (α) averaged over 50 trials, with 16 and 84th percentiles shown in shaded color. additional details are available in appendix c. ## related work answer equivalence. most similar to our work, @xcite annotate 1k answer pairs from the squad gold answers with a similarity score and train a model that classifies the concatenated strings. they focus on the symmetric stringsimilarity problem whereas we aim for asymmetric equivalence conditioned on the answer. another similar effort is the mocha dataset @xcite . the authors also collect annotations on answer candidates and train a learned metric. however, they focus on generative question answering. moreover, their methods for collecting candidate answers, the selection of datasets and the rating task differ considerably from our work. chen et al. ( 2021 ) use natural language inference to verify predictions from qa systems by converting question and answer into a statement. they use this to improve predictions in a setting where no gold answer is known, but one could potentially employ similar methods to compare a predicted answer to a gold answer. text similarity. to our knowledge, @xcite first used token f 1 for automatic evaluation of their "sys called qanda". @xcite find that f 1 is a reasonable metric for extractive qa tasks but fails to account for the higher variability of surface form in generative qa. besides token f 1 and exact match, other popular metrics for text comparison have been tried for question answering, but are not widely in use: bleu @xcite , rouge @xcite and meteor @xcite . @xcite identify the need for methods that go beyond lexical overlap, trying to adapt rouge and bleu to better fit answer comparison, but focusing on just "yes-no" and "entity" questions. using a different approach, @xcite propose to expand entities in gold answers with aliases from freebase @xcite to improve exact match reliability. a good overview of different string distance metrics, in the context of name-matching tasks, can be found in @xcite . metrics based on the wasserstein distance have also been proposed @xcite . in addition to automatic metrics, the neurips 2020 efficientqa competition @xcite uses manual annotations to reward correct answers not contained in the gold answers. in contrast, we focus on only rewarding answers that are equivalent to one of the gold answers. an important topic that is tangentially related to our work is the question of ambiguous questions (cf. @xcite ). our approach to answer equivalence could be useful for detecting the presence of multiple clusters of equivalent answers, suggesting an ambiguous question. learned metrics. recently string based metrics have been replaced by learned metrics for various natural langauge generation (nlg) tasks. examples include bleurt @xcite and comet @xcite for machine translation evaluation, which @xcite find to correlate better with human judgments than e.g. the popular bleu metric @xcite . for question answering evaluation, @xcite propose a variant of bertscore @xcite where the answer tokens are contextualized by adding the question and context but can't show improvements over f 1 for extractive qa. ## conclusion we present a systematic data-driven analysis of the shortcomings of token-level measures, em and f 1 , for evaluating qa systems against a set of reference answers. we design an answer equivalence (ae) task that directly captures the desired relation between candidate and reference answer. also, we collect a large number of annotations for both evaluating and training equivalence models, as well as quantitatively assessing the performance of qa systems. beyond relying on human ae annotations for evaluation, we trained a bert matching (bem) model and showed that it generalized well to new qa models and evaluation questions. specifically, bem allowed a significantly better performance assessment for the qa systems compared to the token-level or other similarity measures. we also demonstrated the value of ae on a practical application beyond qa performance assessment. we hope releasing our data will contribute to further development of better metrics and improve the evaluation and usability of qa systems.