Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation (2021)

Abstract: Despite significant progress has been achieved in text summarization, factual inconsistency in generated summaries still severely limits its practical applications. Among the key factors to ensure factual consistency, a reliable automatic evaluation metric is the first and the most crucial one. However, existing metrics either neglect the intrinsic cause of the factual inconsistency or rely on auxiliary tasks, leading to an unsatisfied correlation with human judgments or increasing the inconvenience of usage in practice. In light of these challenges, we propose a novel metric to evaluate the factual consistency in text summarization via counterfactual estimation, which formulates the causal relationship among the source document, the generated summary, and the language prior. We remove the effect of language prior, which can cause factual inconsistency, from the total causal effect on the generated summary, and provides a simple yet effective way to evaluate consistency without relying on other auxiliary tasks. We conduct a series of experiments on three public abstractive text summarization datasets, and demonstrate the advantages of the proposed metric in both improving the correlation with human judgments and the convenience of usage. The source code is available at https://github.com/xieyxclack/factual_coco .

## introduction in recent years, significant progress has been achieved in text summarization, and with the help of deep neural networks, we can generate informative, relevant, and fluent texts @xcite @xcite . however, it still remains a major challenge to ensure the factual consistency of the generated summary with respect to the source document @xcite . for instance, in the * work done at alibaba. † corresponding author. to ensure the factual consistency in text summarization, a reliable automatic evaluation metric is the first and the most crucial factor @xcite . the predominant automatic metrics, e.g., rouge @xcite and meteor @xcite , are mainly based on n-gram lexical-overlap and have been proven to be poorly correlated with human judgments on factual consistency @xcite @xcite . to better evaluate the factual consistency of summarization systems, various types of metrics have been introduced, including computing semantic similarity with pretrained model instead of n-gram based similarity @xcite , and using auxiliary tasks such as textual entailment @xcite and question answering @xcite @xcite . however, none of them tackle this issue from the view of intrinsic cause of the factual inconsistency. besides, some metrics rely on auxiliary tasks (e.g., question answering), which makes these metrics costly and inconvenient. in short, automatic evaluation for factual consistency in text summarization still remains an open research problem. revisiting the sequence-to-sequence (seq2seq) summarization models, a summary is generated according to the encoded source document and the learned decoder. the information in the source document is encoded and used to ensure the factual consistency of generated summary. the decoder, as a language model, learns the language prior from the training corpus to transform the en-coded source document to an informative and fluent summary. however, the side effects also come along with the language prior, hallucinating the inconsistency tokens due to spurious linguistic correlations learned from training corpus. for instance, when the term green leaves occurs frequently in the training corpus and the summarization model has learned such language prior knowledge, it could be with high probability to generate such inconsistency term green leaves, even though the source document is about red maple leaves. similar hallucination phenomena have also been observed in other conditional text generation tasks, including image caption @xcite and data-totext @xcite . shed light by the above challenges and insights, in this paper, we seek to design a simple yet effective evaluation metric, named counterfactual consistency (denoted as coco), for text summarization. different from the existing metrics, coco is proposed to evaluate the factual consistency of summarized texts via counterfactual estimation, and it does not rely on auxiliary tasks, which brings the convenience of usage in practice. to be specific, with the help of causal inference @xcite , we formulate the causal relationship among the source document, the generated summary, and the language prior to build up the causal graph for text summarization. according to the built causal graph and the analysis of causal effect, we point out that the effect of language prior can be blamed to cause factual inconsistency. thus, we propose counterfactual abstractive summarization to estimate the causal effect of language prior on the generated summary, and remove it from the total causal effect. the estimated effect, which is the causal effect of the source document on the generated summary, serves as a factual consistency score of the generated summary. the intuition is that when texts are generated more relying on the source document rather than the language prior, they should be more likely to be factually consistent w.r.t. the source documents. to demonstrate the effectiveness of the proposed metric coco, we conduct a series of experiments on three public datasets, which are derived from widely-used benchmarks cnn/daily mail @xcite or xsum @xcite , and have human annotations on factual consistency. without relying on auxiliary tasks, the proposed metric coco achieves a significant improvement against the existing automatic metrics for text summarization in terms of the correlation with human annotations. ## related work the most popular n-gram based evaluation metrics, e.g., rouge @xcite , bleu @xcite and meteor @xcite , have been proven to perform poorly on measuring factual consistency @xcite . inspired by the success of pretrained contextual word embeddings, bertscore @xcite leverages the pretrained bert @xcite model to compute the similarity between the generated summary and reference. however, these metrics cannot lead to a satisfying correlation to human judgments on factual consistency since they only capture the token-level overlapping or similarity. hence, instead of defining metrics on token level, @xcite proposes to measure the factual consistency by counting the overlap of facts (i.e., relation tuple) extracted from the generated summary and the source document. several works have also explored natural language inference (nli) to evaluate the factual consistency via calculating entailment probability between the document and its abstractive summaries @xcite . they assume that a factually consistent summary is usually entailed by the source document. to address the issue of domain shift in out-of-the-box nli models, synthetic training datasets, e.g., augmented by summarization datasets @xcite and qa datasets @xcite , are created to finetune the bert-based nli models. question answering has also been used as an evaluation method for summarization @xcite . for factual consistency evaluation, the basic intuition is to test whether a summary and its corresponding source have similar answers for the synthetic questions. the differences between various works are mainly in question generation and metric computation. for example, @xcite generates questions from the reference summary; @xcite and @xcite generate questions from the evaluated summary; while @xcite generates questions from both the evaluated summary and its corresponding source document. for metric computation, @xcite averages the percentage of questions answered correctly accord-ing to the generated summaries, while @xcite computes the similarity of the two answers from the summary and its corresponding source with token-level f 1 score. in summary, the existing metrics are proposed to measure the factual consistency via calculating the semantic similarity between the generated summary and the source document or the references, with the help of extrinsic tasks like pretrained language models or other auxiliary tasks. however, few of them explore the intrinsic cause of the factual inconsistency. in this paper, we study this task from the perspective of causal inference @xcite . ## methodology in this section, we introduce the proposed metric for measuring factual consistency in text summarization, named counterfactual consistency (denoted as coco). ## conclusions in this paper, we introduce coco, an effective and convenient metric for measuring the factual consistency of abstractive summarization models. inspired by the intrinsic cause of factual inconsistency in text summarization, coco can evaluate factual consistency via counterfactual estimation without relying on other extrinsic tasks. experiments on human-annotated datasets verify the advantages of our proposed metric in terms of the correlation with human judgments. in the future, several directions can be explored. first, although this paper focuses on evaluation metric, the proposed idea can be incorporated into abstractive summarization models to enhance factual consistency. another interesting direction is to apply this metric in other conditional text generation tasks such as image caption and data-to-text generation. our implementation is based on fairseq and huggingface.