Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021)

Abstract: Neural abstractive summarization systems have gained significant progress in recent years. However, abstractive summarization often produce inconsisitent statements or false facts. How to automatically generate highly abstract yet factually correct summaries? In this paper, we proposed an efficient weak-supervised adversarial data augmentation approach to form the factual consistency dataset. Based on the artificial dataset, we train an evaluation model that can not only make accurate and robust factual consistency discrimination but is also capable of making interpretable factual errors tracing by backpropagated gradient distribution on token embeddings. Experiments and analysis conduct on public annotated summarization and factual consistency datasets demonstrate our approach effective and reasonable.

## introduction text summarization aims to produce a simplified version of the source document while retaining salient information. abstractive summarization is a branch of methods in which generation text is free from constraint on the tokens that appeared in the source. these methods are extensively studied since its flexibility and generalization ability @xcite @xcite . however, a challenge in abstractive summarization is the trade-off between abstractiveness and factual consistency. recent studies show that about 30% of the summaries generated by abstractive models contain facts errors toward source documents. the proportion will rise further as the data abstractiveness increases @xcite @xcite , causing factual checking an essential process to verify the credibility and usability of models. in previous approaches for detecting or boosting factual consistency can be divided into three kinds. ## methods fig 1 shows the overall architecture of our factual consistency evaluation model. we adopt roberta @xcite as feature extractor f (•). given a sequence concatenated by source document and corresponding summary @xmath0, x 2 , . . . , x lx }, we encode tokens into representations r @xmath1, where e @xmath2. we add a simple linear layer after representation of [cls] token for cal-culating binary cross-entropy loss l ce (f (e; θ, y )), where @xmath3, inconsistent}. in the following, we describe the building methods of artificial datasets, the model's training process, and our proposed error tracing strategy. artificial dataset. we follow the recent methods to generate inconsistent samples through rule-based transformations @xcite . the source document s and the corresponding target summary t is treat as a consistent example x @xmath4, t}. after utilizing corruption on part of the original summary (corresponding to yellow highlighted part in fig. 1 ), the source and the pseudo summary t forms a inconsistent example x @xmath5, t }. three types of strategies are used to corrupt the original summaries. (1) entity swapping foot_0 : replace a random entity in the reference summary by another random entity of the same type in the same source document. to alleviate the bias caused by synonyms, we apply an empirical threshold on the similarity between the original and pseudo entities based on the simple distance algorithm. (2) pronoun swapping, replace a random pronoun with another one of matching syntactic case. (3) negation: transform a random auxiliary verb to its negative form. adversarial augmentation. it is pointed that a classifier trained on artificial datasets only works well on easy examples, thus can hardly generalize well to actual scenarios @xcite . to alleviate this, we propose an adversarial attack mechanism @xcite @xcite @xcite on rule-based pseudo samples as data augmentation. for token embeddings of a sample e, we try to find a worst-case perturbation vector v that maximizes the loss function: where is the norm bound of the perturbation, since the complexity of neural models, it is intractable to compute the perturbation precisely. instead, we apply fast gradient value (fgv) @xcite to approximate a worst-case perturbation: the gradient g is the first-order differential of l ce , representing the direction that rapidly increases the loss. we normalize the gradient and use a small norm to ensure the approximation reasonable. specifically, for inconsistent samples, the perturbations on the corrupted tokens are masked by a filter. we will explain the reason in the following. it is worth noting that fig. 1 only shows an inconsistent example, and for consistent examples, perturbations on all tokens are retained. we name the filtered perturbation as v p and add it to e to obtain new tokens embeddings @xmath6, which can be regarded as an augmented sample. we feed it to the model again to obtain another loss l adv with the same label. finally, we use the weighted sum of two losses to train our model: ## conclusion abstractive summarization models are susceptible to factual inconsistency generation. to optimize the robustness and interpretability of factual consistency evaluation, we proposed an implicit data augmentation method based on the adversarial attack to construct hard factual inconsistent examples and gradient-based fact errors tracing strategy to provide auxiliary information. experiments conduct on public datasets demonstrate the effectiveness of our models. the extensive analysis further reveals the role of the error tracing strategy.