Q uest E val: Summarization Asks for Fact-based Evaluation (2021)

Abstract: Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in extensive experiments.

## introduction the reliability of automatic evaluation metrics is an important factor for progress in artificial intelligence tasks, enabling the comparison and improvement of the proposed systems. the design of reliable metrics for natural language generation (nlg) systems is very challenging, and still an open research problem @xcite and @xcite showed that current metrics do not correlate well with human judgments, and argued for the development of new evaluation metrics. among nlg tasks, summarization is one of the most difficult to evaluate automatically. for a given document, the number of possible correct outputs is much larger than for other tasks such as machine translation. thus, when only a single reference summary is given -as is typically the case for large-scale summarization datasets, the correlation of standard automatic evaluation metrics with human judgments is low @xcite . furthermore, since a summary must be shorter than the corresponding source document, information selection @xcite is critical so that the summary only contains the salient contents from its source document. for these reasons, n-gram based metrics, such as rouge @xcite , are known to poorly reflect human preference @xcite @xcite . finally, it is crucial for an effective summarization to generate texts that are factually consistent with their source documents. however, this aspect is not measured by n-grams based metrics. notably, while recent stateof-the-art generative models @xcite produce fluent summaries, they frequently contain false or unsupported information @xcite , a phenomenon also known as neural hallucination @xcite . to overcome these limitations, a new approach to evaluate summarization systems has recently emerged, based on question generation (qg) and answering (qa) @xcite @xcite . these metrics measure to which extent a summary provides sufficient information to answer questions posed on its corresponding source document. they can be used to assess the factual consistency (i.e. precision) @xcite or the relevance (i.e. recall) @xcite of the evaluated summary, with respect to its source document. although these works have introduced an interesting and novel method to evaluate summarization, with encouraging preliminary results, none of those metrics is found to perform better than rouge @xcite : automatic evaluation of summarization systems remains an open research problem @xcite . 2020 ). the orange area corresponds to the recall-oriented summaqa @xcite . we extend it with a weighter component for an improved recall (red area). the encompassing area corresponds to our proposed unified approach, qu e s tev a l. inspired by these works, and motivated to take up the challenge of summarization evaluation, we propose qu e s tev a l, a new reference-less metric, which is found to improve the correlation with humans judgments. our contributions are as follows: recent works have leveraged the success of pretrained language models. @xcite proposed bertscore, which uses bert @xcite to compute a similarity score between the reference and the evaluated text. however, its performance is similar to that of @xcite . several works have explored using natural language inference (nli) models to evaluate the factual consistency of summaries @xcite @xcite , finding mixed results in using nli models rather than qa models. content weighting information selection is crucial in summarization: a good summary should contain only relevant facts. however, very few methods have been proposed to tackle this aspect; all the aforementioned metrics only reflect implicitly information selection, trough the gold-reference. @xcite proposed to explicitly model it via a function that weights the source document tokens. as opposed to our proposed weighting method, @xcite function requires the ground truth and thus is not suited to a reference-less scenario. qa-based metrics qa-based approaches for summary evaluation were proposed a decade ago by @xcite for human evaluation. @xcite and @xcite proposed to automate this approach by automatically generating questions from the reference summary. @xcite extended these works by generating the questions from the source document, probing the output summary for information retrieved from the input text (a recall-oriented approach). however, by weighing each question equally, their approach lacks a way to select questions that reflect the most important information of the input. conversely, @xcite al. (2020) and durmus et al. (2020) proposed to generate questions from the evaluated summary. these methods are precision oriented, since they measure the amount of information in the evaluated summary that are supported by the input text. we show in this paper that combining these recall and precision approaches leads to an improved metric. ## a question-answering based framework this paper introduces the qu e s tev a l framework for evaluating summarization systems, which accounts for both factual consistency and relevance of the generated text, without requiring any human reference. qu e s tev a l consists of a qg component q g and a qa component q a , described in this section and depicted in figure 1 . ## the questeval metric in the following, d and s are two sequences of tokens with d denoting the source document and s the corresponding evaluated summary. ## experiments 5.1 summarization datasets to evaluate qu e s tev a l, we measure its correlation with human judgments on different datasets: summeval released by fabbri et al. (2020), it is one of the largest human-annotated datasets for summarization. derived from cnn/daily mail @xcite , it consists of 12,800 summary level annotations. to ensure diversity, the summaries were generated from 16 different summarization models, including extractive and abstractive architectures. to assess quality, three experts annotated four dimensions: i) consistency: the proportion of facts in the summary corresponding to facts in the original text; ii) coherence: how well-structured and well-organized is the summary; iii) fluency: how fluent the summary is to read; and, iv) relevance: the ratio between important and excess information in the summary. 3 qags-xsum wang et al. (2020) released a subset of 239 bart outputs fine-tuned on xsum @xcite . 4 three annotators measured the consistency of each summary. ## conclusion we proposed qu e s tev a l, a new reference-less framework to evaluate summarization models, which unifies previous qa-based approaches and extends them with question weighting, accounting all in one for factual consistency, relevance and information selection. compared to existing metrics, we find that qu e s tev a l correlates dramatically better with human judgments, while at the same time not requiring any reference. this allows for more accurate comparison between systems. moreover, any progress in question answering and generation can directly be applied within our proposed framework, leading to additional improvements. we make the code available 5 with the hope that it will contribute to further progress in the field. we have started to adapt qu e s tev a l to other natural language generation tasks that suffer from the same evaluation limitations, e.g. text simplification @xcite , image captioning @xcite , and data to text @xcite . in future work, we plan to extend qu e s tev a l to multilingual scenarios such as machine translation and multilingual summarization. seq2seq models. we use the t5-base model @xcite implemented in hugging face @xcite . we trained our models on a single nvidia rtx 2080 ti gpu with 11g ram. for selecting entities as the answers candidates, we use spacy 2 foot_4 . for all the experiments, we used the default hyper-parameters. for all the variations of hyper-parameters and models, we tested questeval on qags-xsum data, in order to keep summeval unseen at test time. when we compute questeval precision or recall only , we observe a significant improvement over both qags and summaqa. this improvement could be due to implementation differences: @xcite use a rule based system to generate summaqa's questions, and qags filters the questions via various heuristics. instead, we rely on a t5-base neural generator and use only one simple heuristic. both qags and summaqa use bart/bert for their models, while we use a smaller model, t5-base, coupled with our negative sampling method. each of these changes improve the results, their combination showed that they add up.