Self-Attention Guided Copy Mechanism for Abstractive Summarization (2020)

Abstract: Copy module has been widely equipped in the recent abstractive summarization models, which facilitates the decoder to extract words from the source into the summary. Generally, the encoder-decoder attention is served as the copy distribution, while how to guarantee that important words in the source are copied remains a challenge. In this work, we propose a Transformer-based model to enhance the copy mechanism. Specifically, we identify the importance of each source word based on the degree centrality with a directed graph built by the self-attention layer in the Transformer. We use the centrality of each source word to guide the copy process explicitly. Experimental results show that the self-attention graph provides useful guidance for the copy distribution. Our proposed models significantly outperform the baseline methods on the CNN/Daily Mail dataset and the Gigaword dataset.

## introduction the explosion of information has expedited the rapid development of text summarization technology, which can help us to grasp the key points from miscellaneous information quickly. there are broadly two types of summarization methods: extractive and abstractive. extractive approaches select the original text segments in the input to form a summary, while abstractive approaches "create" novel sentences based on natural language generation techniques. in the past few years, recurrent neural networks (rnns) based architectures @xcite @xcite @xcite @xcite have obtained state-of-the-art results for text summarization. benefit from longterm dependency and high scalability, transformerbased networks have shown superiority over rnns ## related work neural network based models @xcite @xcite @xcite @xcite @xcite achieve promising results for the abstractive text summarization. copy mechanism @xcite @xcite enables the summarizers with the ability to copy from the source into the target via pointing @xcite . recently, pre-training based methods @xcite have attracted growing attention and achieved state-of-the-art performances in many nlp tasks, and pre-training encoder-decoder transformers @xcite @xcite @xcite show great successes for the summarization task. in this work, we explore the copy module upon the transformer-based summarization model. ## background we first introduce the copy mechanism. in pointer-generator networks (pgnet) @xcite , the source text x are fed into a bidirectional lstm (bilstm) encoder, producing a sequence of encoding hidden state h: on each step t, a unidirectional lstm decoder receives the word embedding of the previous word to produce decoder state s: the vocabulary distribution p vocab over all words in the target vocabulary is calculated as follows: by incorporating a generating-copying switch p @xmath0, 1], the final probability distribution of the ground-truth target word y t is: copy distribution p copy determines where to attend in time step t. in the most previous work, encoder-decoder attention weight α t is serves as the copy distribution @xcite : the loss function l is the average negative log likelihood of the ground-truth target word y t for each timestep t: ## model in this section, we present our approach to enhance the copy mechanism. first, we briefly describe the transformer model with the copy mechanism. then, we introduce two methods to calculate the centrality scores for the source words based on the encoder self-attention layer. finally, we incorporate the centrality score into the copy distribution and the loss function. the framework of our model is shown in figure 1 . ## conclusion in this paper, we propose the sagcopy summarization model that acquires guidance signals for the copy mechanism from the encoder self-attention graph. we first calculate the centrality score for each source word. then, we incorporate the importance score into the copy module. the experimental results show the effectiveness of our model. for future work, we intend to apply our method to other transformer-based summarization models.