Enhancing Factual Consistency of Abstractive Summarization (2021)

Abstract: Automatic abstractive summaries are found to often distort or fabricate facts in the article. This inconsistency between summary and original text has seriously impacted its applicability. We propose a fact-aware summarization model FASum to extract and integrate factual relations into the summary generation process via graph attention. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by existing systems. Empirical results show that the fact-aware summarization can produce abstractive summaries with higher factual consistency compared with existing systems, and the correction model improves the factual consistency of given summaries via modifying only a few keywords.

## introduction text summarization models aim to produce an abridged version of long text while preserving salient information. abstractive summarization is a type of such models that can freely generate summaries, with no constraint on the words or phrases used. this format is closer to human-edited summaries and is both flexible and informative. thus, there are numerous approaches to produce abstractive summaries @xcite @xcite . however, one prominent issue with abstractive summarization is factual inconsistency. it refers to the hallucination phenomenon that the summary sometimes distorts or fabricates the facts in the article. recent studies show that up to 30% of the summaries generated by abstractive models contain such factual inconsistencies @xcite , raising concerns about the credibility and usability of these systems. article real madrid ace gareth bale treated himself to a sunday evening bbq... the welsh wizard was ... scoring twice and assisting another in an impressive victory... cristiano ronaldo scored five goals against granada on sunday ... bottomup ... the real madrid ace scored five goals against granada on sunday. the welsh wizard was in impressive form for... seq2seq ...gareth bale scored five and assisted another in an impressive win in israel... fasum (ours) ...gareth bale scored twice and helped his side to a sensational 9-1 win. cristiano ronaldo scored five goals against granada on sunday... on the other hand, most existing abstractive summarization models apply a conditional language model to focus on the token-level accuracy of summaries, while neglecting semantic-level consistency between the summary and article. therefore, the generated summaries are often high in tokenlevel metrics like rouge @xcite but lack factual consistency. in view of this, we argue that a robust abstractive summarization system must be equipped with factual knowledge to accurately summarize the article. in this paper, we represent facts in the form of knowledge graphs. although there are numerous article the flame of remembrance burns in jerusalem, and a song of memory haunts valerie braham as it never has before. this year, israel's memorial day commemoration is for bereaved family members such as braham. "now i truly understand everyone who has lost a loved one," braham said. her husband, philippe braham, was one of 17 people killed in january's terror attacks in paris... as israel mourns on the nation's remembrance day, french prime minister manuel valls announced after his weekly cabinet meeting that french authorities had foiled a terror plot... bottomup valerie braham was one of 17 people killed in january 's terror attacks in paris. france's memorial day commemoration is for bereaved family members as braham. israel's prime minister says the terror plot has not been done. ## conclusion in this paper, we extract factual information from the article to be represented by a knowledge graph. we then integrate this factual knowledge into the process of producing summaries. the resulting model fasum enhances the ability to preserve facts during summarization, demonstrated by both automatic and human evaluation. we also present a correction model, fc, to rectify factual errors in candidate summaries. furthermore, we propose an easy-to-compute model-free metric, relation matching rate, to measure factual consistency based on the overlapping ratio of relational tuples. for future work, we plan to integrate knowledge graphs into pre-training for more accurate and factually consistent summarization. moreover, we will combine the internally extracted knowledge graph with an external knowledge graph (e.g. con-ceptnet) to enhance the commonsense capability of summarization models.