Semantic Answer Similarity for Evaluating Question Answering Models (2021)

Abstract: The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semantically similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.

## introduction the evaluation of question answering (qa) models relies on human-annotated datasets of questionanswer pairs. given a question, the ground-truth answer is compared to the answer predicted by a model with regard to different similarity metrics. currently, the most prominent metrics for the evaluation of qa models are exact match (em), f1-score, and top-n-accuracy. all these three metrics rely on string-based comparison. em is a binary metric that checks whether the predicted answer string matches exactly the ground-truth answer. while this metric works well for short factual * both authors contributed equally to this research. if a dataset contains multi-way annotations, there can be multiple different ground-truth answers for the same question. the maximum similarity score of a prediction over all ground-truth answers is used in that case, which works with all the metrics above. however, a problem is that sometimes only one correct answer is annotated when in fact there are multiple correct answers in a document. if the multiple correct answers are semantically but not lexically the same, existing metrics require all correct answers within a document to be annotated and cannot be used reliably otherwise. figure 1 gives an example of a context, a question, multiple groundtruth answers, a prediction and different similarity scores. besides em, f1-score, and top-1-accuracy, we also list human judgment. the example shows that the existing metrics cannot capture the semantic similarity of the prediction and the ground-truth answers but are limited to lexical similarity. extractive qa is not the only qa task that requires evaluation metrics that go beyond stringbased matching. abstractive qa requires generative models to synthesize an answer and they need to be evaluated differently, too. as of today, the evaluation of this task reuses metrics from the research field of natural language generation (nlg) but they are mostly string-based and not tailored to qa. given the shortcomings of the existing metrics, a novel metric for qa is needed and we address this challenge by presenting sas, a cross-encoderbased semantic answer similarity metric, and compare it with traditional lexical metrics and two recently proposed metrics for the more general task of semantic textual similarity (sts). to encourage and support research in this area, we release the annotated dataset foot_0 and the trained model foot_1 under the creative commons attribution-sharealike 4.0 international license (cc by-sa 4.0). the remainder of this paper is structured as follows: section 2 describes related work on metrics used for the evaluation of qa and similar tasks. in section 3, we present our new metric sas along with two existing sts metrics. three datasets, two newly created, three-way annotated datasets and one existing dataset from related work, are the basis of the experiments presented in section 4, which compare four lexical and three semantic similarity metrics. we conclude in section 5 with directions for future work. ## related work the related work discussed in this section goes beyond evaluation metrics explicitly meant for qa. the task can be generalized to estimating the semantic similarity of a pair of text inputs, which is often referred to as semantic textual similarity (sts). while there is a benchmark dataset for sts estimation @xcite , to the best of our knowledge, not even a single dataset has been created for the subtask of estimating semantic answer similarity. sts is closely related to paraphrasing, which, strictly speaking, refers to semantic equivalence. as a consequence of this strict definition, @xcite introduce the concept of approximate paraphrases as conveying a similar meaning. measuring to what extent a text is an approximate paraphrase of another text has been addressed in several subfields of research on natural language processing and we list different approaches in the following. a recent tutorial summarizes evaluation metrics used in nlg, including but not limited to the qa task @xcite . the challenge of evaluating nlg has also been recently addressed by @xcite , who introduce the gem benchmark comprising eleven datasets. besides traditional, lexical similarity metrics, such as bleu @xcite , rouge @xcite , and meteor @xcite , the benchmark also includes the two semantic similarity metrics bertscore @xcite and bleurt @xcite . both bertscore and bleurt are bert-based @xcite metrics tailored to evaluating nlg. bleu (bilingual evaluation understudy) @xcite is a metric used to evaluate the quality of machine translations by measuring the n-gram overlap of prediction and ground truth. similarly, there is rouge (recall-oriented understudy for gisting evaluation), which comes in different variations, such as using n-gram overlap (rouge-n) @xcite or the longest common subsequence (rouge-l) @xcite of prediction and ground truth. meteor @xcite addresses the same task but aims to improve on bleu, for example, by using a weighted harmonic mean of precision and recall of uni-gram overlap. it relies on wordnet @xcite and, for this reason, can only be used for english. there are also slightly modified versions of bleu and rouge for yes-no answers and entity answers that introduce a bonus term that gives more weight to correct answers of that type @xcite . still, these modified versions rely on the standard implementations of bleu and rouge, thus inheriting their shortcomings with regard to lexical vs. semantic similarity. bertscore @xcite ) is similar to f1-score in that it performs stop word removal and lowercasing before the comparison. tf-idf is used to lower the influence of stop words on the score. in our work, we argue that stopword removal should not be an extra step of the metric. instead, the metrics should be based on models that have been trained to recognize what words and phrases are more or less important when comparing the meaning of two answers. the main advantage of bertscore over traditional metrics is that it compares contextual embeddings of tokens in the prediction and the ground truth instead of the actual tokens. as future work, @xcite mention that bertscore could be adapted for the evaluation of different tasks and in this paper, we discuss whether bertscore is superior to other metrics for the evaluation of qa tasks. @xcite apply bertscore as an evaluation metric for qa and find that meteor has a stronger correlation with human judgment. while evaluating the correctness of predicted answers is the by far the most popular qa evaluation task, there are also approaches to evaluate the consistency of the predicted answers @xcite or other desirable properties of opendomain qa models, such as efficiency, context awareness, fine granularity of answers, end-to-end trainability or ability to generalize to different input data @xcite . perturbations can serve as a means to evaluate the latter @xcite and, in the same way, perturbations of the training data allow training models that are more robust @xcite . the correctness of answers can be estimated with methods from natural language inference: chen et al. (2021) con-vert answers to declarative statements and check whether the statement can be inferred from the relevant document (context). @xcite consider the task of evaluating the answerability of generated questions. they find that existing string-based evaluation metrics do not correlate well with human judgment and propose modifications of these metrics, which give more weight to relevant content words, named entities, etc. with the metrics and the underlying models presented in this paper, we present end-to-end deep learning approaches, which learn these features automatically if they increase the correlation between the automated metric and human judgment. semantic similarity metrics might also mitigate the influence of an annotator bias on the evaluation, which has been reported to be learned by models and is currently not recognized if the same annotators create both the training and test dataset @xcite . that bias could be, for example, the position of the answer within a document always being in the first few sentences or a specific style of phrasing the questions. with the help of semantic similarity metrics the position of the annotated answer within the context would not make a difference for the evaluation. in line with the evaluation of qa models, the automated evaluation of models for question generation also relies on bleu, rouge, and me-teor, while human evaluation is limited to small datasets @xcite . for the evaluation of conversational qa, @xcite address the problems that arise from teacher forcing, which refers to earlier ground-truth answers being available to a model at each step in the conversation. the authors discuss ideas to mitigate this problem, such as using the model's own predicted answers instead of the ground-truth answers. however, this approach only considers the ground-truth user reaction to the ground-truth answer but not the predicted answer as other reactions are not available in offline training and evaluation. last but not least, there is research on error analyses of qa models, which defines guidelines @xcite or identifies challenges and promising directions for future work @xcite @xcite . these publications present anecdotal evidence of predictions that are evaluated as wrong due to the limitations of lexical similarity metrics but are in fact correct. ## approach we consider four different approaches to estimate the semantic similarity of pairs of answers: a biencoder approach, a cross-encoder approach, a vanilla version of bertscore, and a trained version of bertscore. this section describes each of these four approaches and the pre-trained language models they are based on. bi-encoder the bi-encoder approach is based on the sentence transformers architecture @xcite , which is a siamese neural network architecture comprising two language models that encode the two text inputs and cosine similarity to calculate a similarity score of the two encoded texts. the model that we use is based on xlm-roberta-base, where the training has been continued on an unreleased multi-lingual paraphrase dataset. the resulting model, called paraphrase-xlm-r-multilingual-v1, has then been fine-tuned on the english-language sts benchmark dataset @xcite and a machine-translated german-language version foot_2 of the same data. the final model is called t-systems-onsite/cross-en-deroberta-sentence-transformer and is available on the huggingface model hub. as the model has been trained on english-and german-language data, we use the exact same model for all three datasets in our experiments. an advantage of the bi-encoder architecture is that the embeddings of the two text inputs are calculated separately. as a consequence, the embeddings of the ground-truth answers can be pre-computed and reused when comparing with the predictions of several different models. this pre-computation can almost halve the time needed to run the evaluation. ## experiments to evaluate the ability of the different approaches to estimate semantic answer similarity, we measure their correlation with human judgment of similarity on three datasets. this section describes the dataset creation, experiment setup, and the final results. ## conclusion and future work current evaluation metrics for qa models are limited in that they check for lexical or positional overlap of ground-truth answers and predictions but do not take into account semantic similarity. in this paper, we present sas, a semantic answer similarity metric that overcomes this limitation. it leverages a cross-encoder architecture and transformer-based language models, which are pre-trained on sts bertscore vanilla is pretrained only on wikipedia, whereas bertscore trained is fine-tuned on the sts benchmark dataset @xcite . datasets that have not been used in the context of qa so far. experiments on three datasets demonstrate that sas outperforms four lexical-based and three semantics-based similarity metrics regarding the correlation between the automated metrics and human judgment of the semantic similarity of pairs of answers. a promising path for future work is to analyze pairs of answers where sas differs from human judgment and find types of common errors. based on these findings, a dataset tailored to training models for estimating semantic answer similarity could be created.