Fine-grained Factual Consistency Assessment for Abstractive Summarization Models (2021)

Abstract: Factual inconsistencies existed in the output of abstractive summarization models with original documents are frequently presented. Fact consistency assessment requires the reasoning capability to find subtle clues to identify whether a model-generated summary is consistent with the original document. This paper proposes a fine-grained two-stage Fact Consistency assessment framework for Summarization models (SumFC). Given a document and a summary sentence, in the first stage, SumFC selects the top-K most relevant sentences with the summary sentence from the document. In the second stage, the model performs fine-grained consistency reasoning at the sentence level, and then aggregates all sentences’ consistency scores to obtain the final assessment result. We get the training data pairs by data synthesis and adopt contrastive loss of data pairs to help the model identify subtle cues. Experiment results show that SumFC has made a significant improvement over the previous state-of-the-art methods. Our experiments also indicate that SumFC distinguishes detailed differences better.

## introduction the goal of summarization models is to rephrase long texts to obtain a short and fluent text containing the original text's main idea. recently, the abstractive summarization models @xcite @xcite have made significant progress and are able to generate fluent and meaningful summaries. however, there are frequent factual errors in the summaries generated by the models, which is presented in most of the commonly used automatic evaluation metrics @xcite @xcite in text generation tasks are based on the overlap of n-gram but not capable of evaluating factual errors, and manual evaluation is time-consuming and costly. therefore, it is vital to detect subtle factual inconsistency automatically for abstractive summarization models. different approaches have been proposed to assess the factual consistency of summarization models, including extracting and checking facts from the source document and the generated text @xcite @xcite , borrowing off-the-shelf natural language inference (nli) datasets for factual consistency checking @xcite , and training pre-trained models through artificial data @xcite . a common drawback of these methods is that they can just deal with some obvious factual errors and ignore the textual details, while differences in the textual details often lead to drastic semantic changes. in this work, we propose a fact consistency assessment framework for summarization models. we split the assessment process into two stages: in the sentence selection stage, top-k pieces of evidence are selected from the original document; in the consistency checking stage, each piece of evidence is reasoned with the summary sentence in detail, then sumfc aggregates results of top-k pieces of evidence. to better distinguish the differences between the positive and synthesized negative sample pairs, the contrastive loss is introduced into the training objectives. the experiments endorse the sumfc's effectiveness over state-ofthe-art methods. document: jerusalem (cnn)a palestinian teenager's name will be removed from an israeli memorial commemorating fallen soldiers and the victims of terrorism after his family and others complained. (...) his father, hussein abu khdeir, said no one asked for his permission to put his son's name on the wall. "i refuse that my son's name will be listed between soldiers of the occupation," he said. almagor, an organization that works on behalf of victims of terror in israel, also opposes abu khdeir's inclusion on the memorial. almagor described the teen's death as a rogue attack and said he's not a terror victim. (...) claim: his father, hussein abu khdeir, said he's not a terror victim. ## related works fact consistency evaluation previous work on assessing factual inconsistency in abstractive summarization can be broadly classified into fact extraction and text classification approaches. the approaches based on fact extraction @xcite @xcite evaluate summaries by comparing the critical facts extracted from the original text and the summary. evaluating metrics based on text classification @xcite @xcite regard the fact consistency assessment as a binary classification task. based on relation extraction, @xcite extract fact triples from both the original text and the modelgenerated summary, then obtains the fact consistency score based on the overlap of fact triples. @xcite and @xcite generate questions on the summary, then adopt question answering accuracy to assess the summary's factual correctness. besides, some researchers consider consistency assessment as the text classification task, yet there are no publicly available large-scale human-annotated datasets. @xcite and @xcite use an out-of-the-box nli dataset to train the model, while @xcite and @xcite obtain the dataset by data synthesis. @xcite generate the training data by a series of rule-based transformations and fine-tune a bert model. @xcite adopt pre-trained sequence-to-sequence (seq2seq) model to make factual error checking and error correction. fact consistency improvement recently, many studies are proposed to improve the consistency between the input and output of the abstractive summarization models, and most of them address this problem by assisting the summarization decoder to be fact-aware. @xcite apply multi-task learning to introduce the text entailment knowledge into the summarization model and adopt text entailment as a reward in the decoding process, which encourages the model to generate summaries entailed by the source text. cao et al. (2018) extract factual descriptions as relation triples from the source document, and propose to force the decoder to attend to both the source text and the extracted facts. gunel et al. (2019) incorporate entity-level knowledge from knowledge graph to sequence-to-sequence model. zhu et al. (2021) extract and represent facts from articles in the form of knowledge graphs, then fuse them with the representation of articles in the transformer-based decoder via attention. dou et al. (2021) propose a general guided framework that can introduce guidance information into the seq2seq summarization model to generate more faithful summaries and enhance the degree of controllability of text generation. in another direction, some studies propose a few pluggable approaches to improve factual consistency. falke et al. (2019) rank the candidate summaries with the consistency assessment score to improve the factuality of the summarization model. matsumaru et al. ( 2020 ) employ a binary classifier to filter out untruthful article-headline pairs from the supervision data, then apply filtered dataset to train the summarization model. @xcite propose to generate candidate summaries by replacing named entities and quantities of generated summaries with that from the source article, then a discriminative model is employed to select the best candidate as the final summary. ## case study several typical examples of the model outputs are listed in