Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics (2022)

Abstract: Question answering-based summarization evaluation metrics must automatically determine whether the QA model’s prediction is correct or not, a task known as answer verification. In this work, we benchmark the lexical answer verification methods which have been used by current QA-based metrics as well as two more sophisticated text comparison methods, BERTScore and LERC. We find that LERC out-performs the other methods in some settings while remaining statistically indistinguishable from lexical overlap in others. However, our experiments reveal that improved verification performance does not necessarily translate to overall QA-based metric quality: In some scenarios, using a worse verification method — or using none at all — has comparable performance to using the best verification method, a result that we attribute to properties of the datasets.

## introduction a recent trend in summarization metrics is evaluating the quality of a summary via question answering (qa; @xcite @xcite @xcite . these metrics compare the semantic content of two texts (e.g., the reference and candidate summaries) by generating questions from one and answering those questions against the other. the amount of common semantic content is proportional to the number of questions which are answered correctly. a critical step of qa-based evaluation metrics is to verify whether the qa model's prediction is correct, a task known as answer verification (see fig. 1 ). this helps to both suppress noisy output from the qa model as well as identify inconsistent information across the texts. ## related work & background the majority of summarization evaluation metrics can be viewed as estimating how similar in meaning two pieces of text are. for instance, rouge @xcite does this by calculating the number of overlapping n-grams between the two texts. instead of directly comparing the entire texts, qa-based metrics identify specific phrases within the texts which should be compared, as follows. first, a set of questions is automatically generated from one text. then, those questions are automatically answered against a second text to obtain a set of predicted answers. the final score is proportional to the number of correct predictions, but determining whether those predictions are correct (the task of answer verification) is done by comparing the text of the prediction to the expected answer. therefore, instead of directly comparing the entire contents of the two texts, qa-based metrics instead reduce the scope of the problem to only comparing specific pairs of phrases. current qa-based metrics perform the answer verification step by lexical comparison, either exact match or token f 1 . such metrics include qa-eval @xcite , feqa @xcite , and more @xcite @xcite . however, any such function which calculates the similarity of arbitrary text can be used instead. this includes embedding-based methods such as bertscore @xcite or metrics which have been trained specifically to do this task, such as lerc @xcite . evaluating how these methods perform as answer verification methods for qabased metrics compared to the lexical baselines is the scope of this work. other, related work has also benchmarked various answer verification methods @xcite , but do so as a method for evaluating qa performance rather than as part of a downstream task, as we do in this work. some concurrent work also tries to improve answer verification by expanding the set of possible expected answers via mining additional aliases from knowledge bases @xcite . ## definitions & methods we define the answer verification task as the following: given a question, answer, the source text from which the qa pair was generated, a prediction, and the target text the prediction comes from, score how similar the meanings of the answer and prediction are (see fig. 1 for an example). 2 answer verification is used by qa-based metrics to suppress noisy outputs from the qa model as well as identify when the qa prediction is correct with respect to the target text but incorrect with respect to the expected answer (e.g., unfaithful information). we analyze four different answer verification methods. exact match the exact match (em) method compares the two phrases to see if they are identical (after light normalization). em assigns a score of 1 if the phrases are identical and 0 otherwise. token f 1 the token f 1 comparison calculates an f 1 score based on the number of unigrams the two phrases have in common. this is equivalent to the f 1 variant of rouge-1. bertscore bertscore @xcite compares two pieces of text by aligning the texts' tokens according to which pairs have the highest bert embedding cosine similarity. we adapt bertscore to answer verification by encoding the answer and prediction using their respective contexts, then calculating the bertscore only between the two phrase encodings. since the output of bertscore is often in a narrow range of values, we rescale the scores by defining 0 and 1 as the 2.5th and 97.5th percentiles of the bertscores calculated over the whole dataset. these changes were made to make the score more interpretable as well as prevent outliers from influencing the score rescaling. in practice, we compute the bertscore using embeddings obtained from roberta-large @xcite . @xcite proposed lerc, a learned metric for scoring how similar the expected and predicted answers to a question are conditioned on the question and the target text the prediction comes from. the metric takes as input the target context, question, expected answer, and predicted answer and concatenates them into a single sequence separated by speical tokens. because it was designed for scoring reading comprehension predictions, it does not use the source text. it then encodes the entire sequence with bert and trains a regression layer on top of the encodings to predict a similarity score. the learned metric was fine-tuned on 40k human annotations of how similar the two answers are on a scale from 1 to 5. we rescale the output from lerc to be in the range [0, 1]. ## experiments the answer verification methods are evaluated independently ( §4.1) as well as in combination with two qa-based metrics ( §4.2), qaeval @xcite and feqa @xcite . qaeval measures the content quality of a summary (does the summary contain "summary-worthy" information) by using a reference summary as the source text and candidate summary as the target text. in contrast, feqa estimates the faithfulness of the summary (does the summary contain information consistent with the input) by using the candidate summary as the source text and the input document as the target text. the experiments are run on two datasets, tac'08 @xcite and summ-eval @xcite . these datasets have summaries generated by 58 and 16 models for 48 and 100 inputs, respectively, which are annotated with expert judgments. both qaeval and feqa are evaluated on summeval because it contains annotations for both summary quality and faithfulness, whereas only qaeval is evaluated on tac'08 since it does not have faithfulness judgments. ## conclusion in this work, we benchmarked four different answer verification methods for qa-based summarization evaluation metrics. although we were able to identify that some methods perform better than others at verification, any such improvement does not necessarily translate a better overall metric quality. we hypothesize that several factors, including the quality of the qa model and properties of the datasets, likely explain this result. even though token f 1 may be sufficient in some scenarios, we also recommend that practitioners also use lerc since it is likely to provide additional benefits, even if they are not easily measured.