Improving the Robustness of Summarization Systems with Dual Augmentation (2023)

Abstract: A robust summarization system should be able to capture the gist of the document, regardless of the specific word choices or noise in the input. In this work, we first explore the summarization models’ robustness against perturbations including word-level synonym substitution and noise. To create semantic-consistent substitutes, we propose a SummAttacker, which is an efficient approach to generating adversarial samples based on pre-trained language models. Experimental results show that state-of-the-art summarization models have a significant decrease in performance on adversarial and noisy test sets. Next, we analyze the vulnerability of the summarization systems and explore improving the robustness by data augmentation. Specifically, the first vulnerability factor we found is the low diversity of the training inputs. Correspondingly, we expose the encoder to more diverse cases created by SummAttacker in the input space. The second factor is the vulnerability of the decoder, and we propose an augmentation in the latent space of the decoder to improve its robustness. Concretely, we create virtual cases by manifold softmixing two decoder hidden states of similar semantic meanings. Experimental results on Gigaword and CNN/DM datasets demonstrate that our approach achieves significant improvements over strong baselines and exhibits higher robustness on noisy, attacked, and clean datasets

## introduction humans have robust summarization processing systems that can easily understand diverse expressions and various wording, and overcome typos, misspellings, and the complete omission of letters when reading @xcite . however, studies reveal that small changes in the input can lead to significant performance drops and fool state-ofthe-art neural networks @xcite despite its importance, robustness in summarization has been less explored. @xcite and @xcite examined positional bias and layout bias in summarization. @xcite introduced multiple noise signals in self-knowledge distillation to improve the performance of student models on benchmark datasets, but they did not explicitly evaluate the robustness of summarization models against noise. hence, in this work, we first evaluate the robustness of the existing state-of-the-art summarization systems against word-level perturbations including noise and adversarial attacks. the noise consists of natural human errors such as typos and misspellings. to create the adversarial attack test set, we come up with a model named summattacker. the core algorithm of summattacker is to find vulnerable words in a given document for the target model and then apply language models to find substituted words adjacent in the opposite direction of the gradient to maximize perturbations. we validate the effectiveness of summattacker on benchmark datasets with different attributes, i.e., gigaword and cnn/dailymail. experiment results show that by only attacking one word (1% token) in gigaword and 5% tokens in cnn/dailymail, the existing summarization models have drastically lower performance. we next conduct a vulnerability analysis and propose two corresponding solutions to improve robustness. our first conjecture is that worse summaries can be caused by replacing common words with uncommon and infrequently-used words, which the model might not understand well. hence, we employ the outputs from summattacker as inputs for the encoder, so as to improve the diversity in the discrete input space. the second influencing factor is that the attacked inputs introduce more variations in the latent space. correspondingly, we aim to expose the model to more diverse hid-den states in the training process. specifically, we build soft pseudo tokens by multiplying the decoder output probability with target token embeddings. these soft pseudo tokens and original tokens are then manifold softmixed on a randomly selected decoder layer to enlarge the training distribution. the interpolations leveraged in deeper hidden layers help capture higher-level information, improve semantic diversity, and provide additional training signal @xcite . experiments show that our dual augmentation for both encoder and decoder improves the robustness of summarization models on noisy and attacked test datasets. our main contributions are as follows: ## related work we discuss related work on robust abstractive summarization, adversarial examples generation, and data augmentation. ## the proposed summattacker formally, given a trained summarization model with parameters θ, the purpose of an attacking model is to slightly perturb the input x such that the summarization output of the perturbed x deviates away from the target summary y: attacked word selector. since it is intractable to obtain an exact solution for equation 1, we, therefore, resort to a greedy approach to circumvent it. in bart kind of summarization model based on transformer architecture, the sequence representation vector s of input tokens in x is first projected to keys k and values v using different linear mapping functions. at the t-th decoding step, the hidden state of the previous decoder layer is projected to the query vector q t . then q t is multiplied by keys k to obtain an attention score a t and the t-th decoding output: attacking with lm and gradients. next, we aim to find a replacement word that is semantically similar to w i but is adversarial to the summarization model. language models are empowered to generate sentences that are semantically accurate, fluent, and grammatically correct. we take advantage of this characteristic to find a replacement word w ′ i for the target word w i . the general idea is to first identify the top likely candidates that are predicted by the language model for w i , and then select the best candidate with the guidance of prediction gradient. concretely, we first feed the tokenized sequence into the bart model to get a prediction for the attacked word w i . as shown in fig. 1 , for w i with a single token, we use stp (single token prediction) operation to simply obtain the top k predictions that are semantically similar to w i . for w i with multiple tokens, we have mtp (multi-token prediction), which lists c × k possible combinations from the prediction, where c is the token number in the word. then we rank the perplexity of all combinations to get the top-k candidate combinations, denoted as v k . we filter out stop words and antonyms using nltk and synonym dictionaries. following the idea of a gradient-based attack model, we then find the most adversarial word w ′ i that deviates from w i towards a change aligned with the prediction gradient: ## dual augmentation with the proposed attacking model, we first analyze the influences of attacking, and then propose our dasum to counter the negative effects. ## conclusion in this paper, we investigate the robustness problem in the summarization task, which has not been well-studied before. we first come up with a sum-mattacker, which slightly perturb the input documents in benchmark test datasets, and causes a significant performance drop for the recent summarization models. correspondingly, we propose a dual data augmentation method for improving the robustness, which generates discrete and virtual training cases in the same meaning but with various expression formats. experimental results show that our model outperforms strong baselines.