Factual Relation Discrimination for Factuality-oriented Abstractive Summarization (2023)

Abstract: Most neural abstractive summarization models are capable of producing high-quality summaries. However, they still frequently contain factual errors. Existing factuality-oriented abstractive summarization models only consider the integration of factual information and ignore the causes of factual errors. To address this issue, we propose a factuality-oriented abstractive summarization model DASum, which is based on a new task factual relation discrimination that is able to identify the causes of factual errors. First, we use data augmentation methods to construct counterfactual summaries (i. e., negative samples), and build a factual summarization dataset. Then, we propose the factual relation discrimination task, which determines the factuality of the dependency relations in summaries during summary generation and guides our DASum to generate factual relations, thereby improving the factuality of summaries. Experimental results on the CNN/DM and XSUM datasets show that our DASum outperforms several state-of-the-art benchmarks in terms of the factual metrics.

## introduction using pre-trained models, neural abstractive summarization @xcite can produce fluent and informative summaries. however, the facts in the generated summaries are often inconsistent with the source documents. for example, @xcite and @xcite show that nearly 30% of the summaries generated by recent neural models have factual errors. thus, factuality-oriented abstractive summarization is intended to improve the factuality of summaries and make summarization models applicable in practice. recent work on improving the factuality of generated summaries relies heavily on extracting facts from the source documents and integrating these facts into abstractive summarization models. these approaches can partially deepen the model's perception of facts. however, they still cannot find the cause of factual errors. one of the reasons for this is that the available summarization corpora contain only factual summaries and ignore counterfactual summaries. if a counterfactual summary has the annotated causes, it will help the model to identify the causes of the factual errors and then improve the performance of factuality-oriented abstractive summarization. for example, there is a document whose summary is "tom stole the cheese". if its counterfactual summary is "jerry stole the cheese" and the annotated cause is entity swapping (i.e., replacing "tom" with "jerry"), it is possible to train a model to deal with different factual errors based on their causes. in this paper, we introduce a factual relation discrimination task to factuality-oriented abstractive summarization, which can make the model aware of the causes of factual errors and learn to avoid factual errors during the summary generation process. specifically, we propose a dasum model based on factual relation discrimination to study factuality-oriented abstractive summarization. we first use data augmentation methods (e.g., pronoun swapping, sentence negation, time and date entity swapping, quantifier swapping, and named entity swapping) to construct counterfactual summaries, which, together with the annotated factual summaries, form the factual summarization dataset. experimental results on the cnn/dm and xsum datasets show that our dasum outperforms several state-of-the-art benchmarks on the factual mtrics (i.e., factc, dae and sent). we summarize our contributions as follows: ## related work previous studies on improving the factuality of summaries mainly fall into two categories, i.e., fact integration and reinforcement learning. fact integration is based on integrating factual knowledge into the summarization model. @xcite extract relational triples from the source document as internal facts, and use a dual encoder to encode the source document and internal facts simultaneously. @xcite 2021) recent studies @xcite also improve the factuality of summaries through reinforcement learning (rl). choubey et al. (2021) use two factuality metrics, entity overlap metric and dependency arc entailment metric, to compute the factuality score and then use it as a reward for rl. @xcite focus on external factual errors and annotate the external factual dataset xent, a factual discriminator is trained on xent, and the discriminator score is used as a reward for rl. however, because the accuracy of the factual evaluation metric is not good enough, rl doesn't really make the model understand the facts in the source document. ## methodology fig. 1 shows the overall process of our model da-sum. in the training phase, the factual summarization dataset is first constructed through data augmentation strategies, and then the dasum model is trained together with factual signals; in the testing phase, the correct factual signal "true" is added to all samples to generate summaries. fig. 2 shows the architecture of dasum. da-sum has different training modes for positive and negative samples on the factual summarization dataset. for positive samples, as shown in fig. 2 (b), the correct factual signal (true) is added to the end of the source document. at this point, all factual relations in the summary are factually correct; for negative samples, as shown in fig. 2(a) , the counterfactual signal (false) is added, and there are both factual and counterfactual relations in the summary. in this section, we first introduce the construction process of the factual summarization dataset, then explore the representation methods of core words and dependent words in factual relations, and finally elaborate on the factual relation discrimination. ## experimentation in this section, we first introduce the experimental settings, and then report the experimental results. finally, we give the ablation study and analysis. ## conclusion in this paper, we employ the factual relation discrimination task to deepen the model's understanding of factual errors and guide it to generate factual summaries. specifically, we construct a factual summarization dataset with positive and negative samples through data augmentation. then, we train the model to identify factual samples, discover the locations and reasons for factual errors, and guide the model to generate summaries through factual guidance signals. experimental results on cnn/dm and xsum show that our dasum outperforms the baselines on factual metrics. our future work will focus on how to construct more accurate negative samples to further identify the fine-grained causes of factual errors.