On Reducing Repetition in Abstractive Summarization (2021)

Abstract: Repetition in natural language generation reduces the informativeness of text and makes it less appealing. Various techniques have been proposed to alleviate it. In this work, we explore and propose techniques to reduce repetition in abstractive summarization. First, we explore the application of unlikelihood training and embedding matrix regularizers from previous work on language modeling to abstractive summarization. Next, we extend the coverage and temporal attention mechanisms to the token level to reduce repetition. In our experiments on the CNN/Daily Mail dataset, we observe that these techniques reduce the amount of repetition and increase the informativeness of the summaries, which we confirm via human evaluation.

## introduction abstractive summarization @xcite @xcite aims at capturing the essence of a document in a limited number of words. despite great performance improvement over the years, many abstractive summarization models still suffer from the repetition problem wherein they tend to repeat the same phrase or ngram over and over again reducing the amount of information conveyed and also reducing the quality of the generated text. for example such a system may generate "he scored five goals. five goals. five goals ...". such kind of repetition is highly uncommon in human written text. a true understanding of the problem is still unknown. some argue that it is because of model architecture @xcite , other suggest that it is because of the nature of sampling performed by the model which is very different from human natural language @xcite . @xcite suggests that the cause of the problem is maximum likelihood training of language generation models. @xcite alleviates this problem to some extend via nucleus sampling and @xcite propose topk sampling to solve the repetition problem. to solve this repetition problem for abstractive summarization we adopt two approaches. all our experiments are based on transformers @xcite . our model, on the cnn/dailymail @xcite dataset outperforms the baseline by a significant margin. in this way we ensure that model does not assign high probabilities to the same token over and over again. in our experiments we see that this additional regularization not only reduces repetition in the generated summaries but also improves the performance in terms of evaluation metric beating the baseline transformer. our key contributions are: abstractive summarization aims at capturing the entire essence of a document within a relatively smaller number of words. there has been great progress made in the recent past where better and better architectures have been proposed to solve the problem. see et al. (2017) used pointer generator networks to copy words from the input document. duan et al. (2019) augmented the transformer architecture with a contrastive attention mechanism to ignore the irrelevant parts of the document. zhang et al. (2020a) pretrained a transformer model for summarization. paulus et al. (2018) proposed to use rewards from policy gradient reinforcement learning to alleviate exposure bias. gehrmann et al. ( 2018 ) developed a bottomup copy attention mechanism to over-determine phrases in the document that should be included in the summary. there approach is build upon the pointer generator network @xcite . our work unlike previous work tries to reduce the repetition in transformer models via techniques that have been successfully applied to language modeling. several methods have been proposed to reduce repetition in language generation. @xcite use different sampling techniques to reduce repetition. @xcite propose the unlikelihood objective to reduce repetition. @xcite propose to combine pairs of words that often follow each other to reduce repetition. our work applies such techniques to abstractive summarization where repetition reduces the amount of information conveyed in the summary. to reduce repetition in abstractive summarization @xcite propose a reinforcement learning based approach. they utilize bertscore @xcite to reward the model for better generations and reduce repetition. @xcite propose a cnn based encoder to perform global encoding of the source side information. their approach also reduces repetition. @xcite propose a reinforcement learning based sentence level neural abstractive summarization model. unlike these approaches both our approaches explicitly try to reduce repetition without using any heuristic based decoding strategies. ## method in this section we provide the training objectives used by us to reduce repetition. all our models and experiments are based on the standard transformer encoder-decoder architecture. the input to the encoder is the input document x represented as @xmath0, x 2 , ..., x n ), and the summary to be generated is @xmath1, y 2 , ..., y m ). we can write the loss function obtained via maximum likelihood estimation as: ## experiments we use the cnn/daily mail dataset for all our experiments. all our models are transformer based, we use the transformer base model with 8 encoder layers, 8 decoder layers, 8 attention heads, 256 is the hidden and embedding dimension and 512 dimensional feed forward layer. we use the linear warm-up followed by square root decay schedule proposed in @xcite . we train our baseline for 200k iterations with a batch size of 32 and we employ early stopping based on validation loss. we use beam search decoding in all our experiments with a beam size of 4, we also employ a length penalty of 1.0. we use label smoothing with α = 0.1. we use sentencepiece tokenizer in all our experiments. all the experiments are performed using pytorch on a single nvidia tesla v100 gpu. ## conclusion in this work we present various techniques to reduce repetitiveness in abstractive summarization. we utilize the unlikelihood objective and the embedding regularizer from previous work to reduce repetition in abstractive summarization. we also propose extensions of the coverage mechanism and the temporal attention mechanism to reduce repetition. our results show that these techniques not only reduce repetition in summaries but also increase performance in terms of rouge scores. our human evaluation results confirm that the summaries generated via our techniques are more informative and have less repetition. the application of our methods on state-of-the-art summarization systems should further improve the results. in future work we would devise newer and better methods to reduce repetition and extend our methods to other sequence to sequence tasks such as machine translation.