Learn to Copy from the Copying History: Correlational Copy Network for Abstractive Summarization (2021)

Abstract: The copying mechanism has had considerable success in abstractive summarization, facilitating models to directly copy words from the input text to the output summary. Existing works mostly employ encoder-decoder attention, which applies copying at each time step independently of the former ones. However, this may sometimes lead to incomplete copying. In this paper, we propose a novel copying scheme named Correlational Copying Network (CoCoNet) that enhances the standard copying mechanism by keeping track of the copying history. It thereby takes advantage of prior copying distributions and, at each time step, explicitly encourages the model to copy the input word that is relevant to the previously copied one. In addition, we strengthen CoCoNet through pre-training with suitable corpora that simulate the copying behaviors. Experimental results show that CoCoNet can copy more accurately and achieves new state-of-the-art performances on summarization benchmarks, including CNN/DailyMail for news summarization and SAMSum for dialogue summarization. The code and checkpoint will be publicly available.

## conclusion we propose coconet that can take advantage of prior copying distributions and encourage the decoder to copy the source word that is relevant to the previously copied one. we further enhance the copying ability through pre-training with the objective of text span generation. our model gains new state-of-the-art results on the news summarization and dialogue summarization tasks.