Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking (2022)

Abstract: Despite the recent advances in abstractive summarization systems, it is still difficult to determine whether a generated summary is factual consistent with the source text. To this end, the latest approach is to train a factual consistency classifier on factually consistent and inconsistent summaries. Luckily, the former is readily available as reference summaries in existing summarization datasets. However, generating the latter remains a challenge, as they need to be factually inconsistent, yet closely relevant to the source text to be effective. In this paper, we propose to generate factually inconsistent summaries using source texts and reference summaries with key information masked. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained on summaries generated using our method generally outperform existing models and show a competitive correlation with human judgments. We also analyze the characteristics of the summaries generated using our method. We will release the pre-trained model and the code at https://github.com/hwanheelee1993/MFMA .

## introduction as textual content available on-and offline explodes, automated text summarization is becoming increasingly crucial @xcite ; with the advances in neural text generation methods, abstractive summarization systems that generate paraphrases are quickly replacing extractive ones that simply select essential sentences from the source text @xcite . while abstractive summaries can be more coherent and informative (given the same length) than their extractive counterparts, they frequently contain information inconsistent with the source text. this is a critical article: guus hiddink, the russia and chelsea coach, has had much to smile about in his 22-year managerial career. ,. . . , enjoying success around the world -at different levels with different players in different cultures -has made guus hiddink one of the most admired bosses around. ,. . . , hiddink's resume includes stints in other high-pressure jobs such as fenerbahce, valencia and real madrid. ,. . . , but the straight-speaking dutchman is loyal to the project he has in charge of the russian national side and insists he will leave chelsea at the end of the season regardless. reference summary: born in 1946, hiddink has become one of the best managers in the world . dutchman has enjoyed huge success at club and international level. he's currently coach of russia and is in charge of chelsea until end of the season. ## methods for a given article a and a summary s, we aim to develop a factual consistency checking system that can evaluate whether s is factual consistent with a. in other words, the system is required to discriminate a factual consistent summary s c with the factual inconsistent summary s i that consists of at least one factual error. we consider this problem as a classification task between s c and s i . however, large-scale human-annotated training datasets for this task have not been constructed yet, especially for the inconsistent summaries s i . in this paper, we focus on effective augmentation methods of the inconsistent summaries. in order for that, there are two crucial conditions: 1) guarantee of inconsistency; the generated summaries should be indeed inconsistent with the source article, 2) relevance to the source article; the generated summaries should include contents related to the article. these two factors are in trade-off relations, which means that when the generated summaries are strongly inconsistent they might not be related to the article and vice versa. therefore appropriate negative summary augmentation is required to improve the factual consistency classifier. to generate confusing and hard negative summaries, we propose a summary generation using a masked article and a masked reference summary where some salient information is hidden. by doing so, we let the summarizer model infer hidden information through the masked article to generate plausible negative summaries. note that, previous works such as factcc and docnli generate negative summaries s i by changing positive summaries s c through entity replacements or mask-and-fill methods without referring to the source article. we observe that previous methods can easily guarantee negativeness, but they often generate summaries that are very irrelevant to the source article or unnatural as shown in figure 1 . ## conclusion in this paper, we proposed an effective generation method of factually inconsistent summaries, called mfma. in this method, some proportion of the source text and corresponding reference summaries is hidden, then a summarization model generates plausible but factually inconsistent summaries by inferring the masked contents. experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained using our method generally outperform existing models and show a competitive correlation with human judgment.