FRSUM : Towards Faithful Abstractive Summarization via Enhancing Factual Robustness (2022)

Abstract: Despite being able to generate fluent and grammatical text, current Seq2Seq summarization models still suffering from the unfaithful generation problem.In this paper, we study the faithfulness of existing systems from a new perspective of factual robustness which is the ability to correctly generate factual information over adversarial unfaithful information.We first measure a model’sfactual robustness by its success rate to defend against adversarial attacks when generating factual information.The factual robustness analysis on a wide range of current systems shows its good consistency with human judgments on faithfulness.Inspired by these findings, we propose to improve the faithfulness of a model by enhancing its factual robustness.Specifically, we propose a novel training strategy, namely FRSUM, which teaches the model to defend against both explicit adversarial samples and implicit factual adversarial perturbations.Extensive automatic and human evaluation results show that FRSUM consistently improves the faithfulness of various Seq2Seq models, such as T5, BART.

## introduction abstractive summarization aims to produce fluent, informative, and faithful summaries for a given document. benefiting from large-scale pre-training techniques, recent abstractive summarization systems are able to generate fluent and coherent summaries @xcite @xcite . however, challenges remain for this task. one of the most urgent requirements is to improve "faithfulness" or "factual consistency" of a model, which requires the generated text to be not only human-like but also faithful to the given document @xcite ). an earlier study observes nearly 30% of summaries suffer from this problem on the gigawords dataset @xcite , while recent largescale human evaluation concludes that 60% of summaries by several popular models contain at least one factual error on xsum @xcite . these findings push the importance of improving faithfulness of summarization to the forefront of research. many recent studies focus on improving the faithfulness of summarization models, which can be mainly divided into three types. the first type modifies the model architecture to introduce preextracted guidance information as additional input @xcite @xcite , while the second type relies on a post-editing module to correct the generated summaries @xcite . the last type takes advantages of auxiliary tasks like entailment @xcite , and qa @xcite on faithfulness. different from previous studies, this work focuses on refining the training strategy of seq2seq models to improve faithfulness universally without involving any extra parameters, post-editing procedures and external auxiliary tasks. in this paper, we study the faithfulness problem of seq2seq models from a new perspective of factual robustness, which is the robustness of generating factual information. we first define factual robustness as the model's ability to correctly generate factual information over adversarial unfaithful information. following this definition, we analyze the factual robustness of a wide range of seq2seq models by measuring their success rate to defend against adversarial attacks when generating factual information. the analysis results (see extensive experiments on several state-of-theart seq2seq models demonstrate the effectiveness of frsum, which improves the faithfulness of various seq2seq models while maintaining their informativeness. besides automatic evaluation, we also conduct fine-grained human evaluation to analyze different types of factual errors. the human evaluation results also show that frsum greatly reduces different types of factual errors. especially, when applying to t5, our method reduces 17.5% and 49.1% of target factual errors on the xsum and cnn/dm datasets, respectively. our contributions can be summarized as the following three points. ## factual robustness on seq2seqs in this section, we introduce the definition and measurement of factual robustness. the factual robustness is defined as the ability of a seq2seq model to correctly generate factual information over adversarial unfaithful information. formally, given a document x, a faithful summary y and a set of unfaithful adversarial samples a s , a model with high factual robustness should satisfy: extending from the conventional adversarial attack framework, we take the generation process of a factual information span as the target for attack. after constructing a set of adversarial samples, we check whether an attack succeeds by comparing the generation probabilities between the span and adversaries. we then define the measurement of factual robustness as the success rate of a model to defend against these attacks in a corpus. following this definition, we measure the factual robustness of current models and analyze their relations with faithfulness. ## frsum in the previous section, we introduce factual robustness and reveal its relation with faithfulness. we also discover that current systems are not robust enough in generating factual spans. based on these findings, it is natural to improve a model's faithfulness by enhancing its factual robustness. thus, we propose frsum, which is a training strategy to improve the faithfulness of seq2seq models by enhancing their factual robustness. frsum is composed of a factual adversarial loss and factual adversarial perturbation. factual adversarial loss encourages the model to defend against explicit adversarial samples. factual adversarial perturbations further apply implicit factual-relevant adversarial permutations to the previous procedure to enhance the factual robustness. we follow the notations in §3 to introduce frsum in detail. frsum can be applied to all kinds of seq2seq models which are composed of an encoder and a decoder. following the common seq2seq architecture, we apply negative log likelihood (nll) in the training process to generate fluent summaries. given a document x and its reference y, the encoder first encodes input document @xmath0, x 2 , . . . , x n ) of length n into hidden representations h. after that, the decoder computes the nll based on h and y: ## results because frsum focuses on faithfulness, we expect improvements on factual metrics without harming the performance of informative metrics. we select the t5 model for an ablation study and human evaluations because it is a widely used model with a relatively moderate factual robustness e%. ## conclusions and future work in this paper, we study the faithfulness of abstractive summarization from the new perspective of factual robustness. we propose a novel adversarial attack method to measure and analyze the factual robustness of current seq2seq models. furthermore, we propose frsum, a faithful improvement training strategy by enhancing the factual robustness of a model to improves its faithfulness.