AVA : an Automatic e V aluation Approach for Question Answering Systems (2021)

Abstract: We introduce AVA, an automatic evaluation approach for Question Answering, which given a set of questions associated with Gold Standard answers (references), can estimate system Accuracy. AVA uses Transformer-based language models to encode question, answer, and reference texts. This allows for effectively assessing answer correctness using similarity between the reference and an automatic answer, biased towards the question semantics. To design, train, and test AVA, we built multiple large training, development, and test sets on public and industrial benchmarks. Our innovative solutions achieve up to 74.7% F1 score in predicting human judgment for single answers. Additionally, AVA can be used to evaluate the overall system Accuracy with an error lower than 7% at 95% of confidence when measured on several QA systems.

## introduction accuracy evaluation is essential both to guide system development as well as to estimate its quality, which is important for researchers, developers, and users. this is often conducted using benchmark datasets containing a data sample, possibly representative of the target data distribution, provided with gold standard (gs) labels (typically produced with a human annotation process). the evaluation is done by comparing the system output with the expected labels using some metrics. this approach falls short when the system output spans a large, possibly infinite set of correct items. for example, in retrieval-based question answering (qa) systems, a correct answer can be any string in the referent text database. for example, for the question, when did marlins start?, an answer could be: the miami marlins began play in the 1993 season as the florida marlins; they started in 1993; they firstly played in 1993; in 1993; or any possible natural language text conveying the information that they started in 1993. as annotating all possible system pieces of output is infeasible, the standard approach is to re-evaluate the new output of the system manually. this dramatically limits the experimentation velocity while significantly increases the development costs. a viable solution for specific nlp tasks such as machine translation (mt), automatically estimates an evaluation score between the system and the reference answers, which correlates with human judgment, e.g., the bleu score is one popular measure @xcite . such methods cannot be applied to a standard qa setting, since qa systems, e.g., those developed for trec-qa track @xcite , have the purpose to provide correct answers and are evaluated with accuracy, i.e., the percentage of correct answers. segment overlapping metrics such as bleu, me-teor, or rouge do not provide a binary outcome, i.e., correct or incorrect (as this is not the aim of mt evaluation). hypothetically speaking, we could apply a threshold to their score to obtain a binary outcome. however, it would not be sufficient as the correctness of an answer loosely depends on the match between the reference and candidate answers. two answers can be correct or incorrect independently of their overlap with the reference. for example, for the question, what percentage of water in the body?, associated with a reference, the percentage of water in the body is 60%, a correct answer is most of the human body is water, with an average of roughly 60%. in contrast, an incorrect answer, still very similar to the reference, could be: the percentage of water in the body is variable. the mt metrics above would find the similarity of the reference with the incorrect answer higher than the one of the references with the correct answer. even a powerful model such as bertscore @xcite would not provide a higher score to the correct answer since it is an unsupervised approach, not trained for this task. it should also be noted that simply training models for matching the answer candidate with the reference will again not work. the question semantics would radically influence the correctness of the answer. that is, match(t, r|q 1 ) can be true while match(t, r|q 2 ) can be false, where t and r are a pair of answer candidate and reference, and q 1 and q 2 are two different questions. in this paper, we study the design of models for measuring the accuracy of qa systems, i.e., percentage of correct answers over a test set (to our knowledge this is the first successful and thorough study). in particular, we (i) build several baselines based on pre-trained transformer models @xcite to encode the triple, question q, candidate t, and reference r, in different ways; and (ii) propose a new attention mechanism, peer attention, to model the interaction between t and r, given the semantic bias of q. to develop and test our models, we created (i) a dataset, web-based question answering foot_0 (wqa) for training and testing ava, the point-wise estimation of qa system output, i.e., the evaluation if an answer is correct or not, given a gs answer; and (ii) a system dataset (sd) constituted by a set of outputs from several qa systems, for which ava estimates their accuracy. the results show a high f1 for point-wise models, up to 74.7%. ava can almost always rank systems in terms of accuracy as manual annotation does. finally, the root mean square error (rmse) with respect to human evaluation depends on the datasets, ranging from 2% to 9.5%, with a std. dev. lower than 5%. ## related work automatic evaluation has been an interesting research area for decades @xcite . there are two typical strategies to design an automatic evaluator: supervised and unsupervised. in mt research, for example, bleu @xcite has been a very popular unsupervised evaluation method for the task. other supervised methods have been recently proposed, most notably @xcite . neuralbased automatic evaluators for dialog systems were studied in @xcite @xcite . automatic evaluation for qa was addressed by @xcite and also for multiple sub-domain qa systems @xcite @xcite . however, little progress has been made in the past two decades towards obtaining a standard method. automating qa evaluation is still an open problem, and there is no recent work supporting it. as mentioned in the introduction mt unsupervised metrics, e.g., bleu score or bertscore, are not either a solution or a reasonable baseline for automatic qa evaluation. they could be used as features for our models, but we designed several supervised approaches based on pre-trained transformer models, which subsume these mt features. a remotely related research effort for automatizing answer evaluation concerns student essays. short answer grading (sag), or short answer scoring, involves the automatic grading of students' answers, typically written in free text, for a given prompt or question @xcite . this task has been studied in @xcite for educational applications. neural-based systems have also been recently proposed to improve the models @xcite . despite the conceptual similarity, i.e., evaluating an answer, the problem setting for the task is fundamentally different. specifically, sag is prompt-centric; thus, the learning objective is to score accurately other different answer variants for a particular question by building models trained on previously known variants @xcite . besides, the answers, while written in free text, are not typically complete sentences. therefore, the sag design aims to capture sufficient content covered in the reference responses for a question. on the contrary, ava is designed to operate in an open-domain qa setting, where both the question and answer are arbitrary input and complete sentences. ## problem definition and preliminaries we consider retrieval-based qa systems, which are mainly constituted by (i) a search engine, retrieving top-k documents related to the questions, and (ii) an answer sentence selection (as2) model, which reranks passages/sentences extracted from the documents. we can automatically evaluate the (i) accuracy of the qa system, which is the percentage of correct top sentences, and (ii) complex measures, such as map and mrr, which quantify the quality of the rank produced by the as2 model. ## dataset creation to learn and test our models, we needed to build ava datasets. the interesting aspect is that we can automatically derive them from standard as2 cor-pora if they contain questions with multiple correct answers. for this purpose, we created our dataset wqa for as2 and transformed it into ava-wqa. we describe our approach to transforming as2 to ava datasets in this section. finally, we build another benchmarking dataset for ava constituted by a set of qa systems and their output on target test sets. this is used to measure the end-to-end system performance (system-wise evaluation). ## experiments we study the performance of ava in predicting: (i) the correctness of the individual answers output by a system (point-wise estimation); and (ii) the overall system performance derived on a test set. we consider qa accuracy and passage reranking measures in comparison to human labeling. the first aspect evaluates the quality of our approaches, whereas the second provides evidence on the practical use of ava to develop qa systems. ## conclusion we have presented ava, the first automatic evaluator method for qa systems. we created seven different datasets, classified into three different types, which we used to develop ava. we released those based on public data and plan to release the others. then, we proposed different transformer-based models and a new peer attention approach to capture answer and reference similarity induced by the question semantics. our extensive experimentation has shown the ava effectiveness for different types of evaluation: point-wise and system-wise over accuracy, map and mrr. the results suggest that ava can estimate the measures above, with a max error of 7% at 95% of confidence. ava can also be applied to generate distant supervision data. an example of this future application is given by @xcite .