Abstractive Document Summarization with Word Embedding Reconstruction (2021)

Abstract: Neural sequence-to-sequence (Seq2Seq) models and BERT have achieved substantial improvements in abstractive document summarization (ADS) without and with pre-training, respectively. However, they sometimes repeatedly attend to unimportant source phrases while mistakenly ignore important ones. We present reconstruction mechanisms on two levels to alleviate this issue. The sequence-level reconstructor reconstructs the whole document from the hidden layer of the target summary, while the word embedding-level one rebuilds the average of word embeddings of the source at the target side to guarantee that as much critical information is included in the summary as possible. Based on the assumption that inverse document frequency (IDF) measures how important a word is, we further leverage the IDF weights in our embedding-level reconstructor. The proposed frameworks lead to promising improvements for ROUGE metrics and human rating on both the CNN/Daily Mail and Newsroom summarization datasets.

## introduction single document summarization is designed to automatically compress a document into its short version without changing the main idea. the summarization task is generally divided into two categories: extractive methods that copy certain sentences or phrases directly from the source text, and abstractive methods that paraphrase the source text by using novel words. abstractive summarization has the potential to produce summaries in the same way that humans do. recent years have witnessed significant progress in the abstractive summarization task performed by seq2seq models, which encode a source text and decode its summary. a hybrid of the extractive and abstractive techniques, called pointer-generator network (pgn) @xcite , has been widely used as a basis for many studies @xcite @xcite thanks to its capability of copying words from the source document and generating new words. with the rich contextual representations, pre-trained encoders @xcite have also improved the state-of-the-art on this task @xcite . however, the conventional pgn and bert face two main problems: 1) when generating summaries, pgn tends to frequently pay attention to some source parts while neglecting other parts, which are regarded as over-and under-attention respectively. therefore, the model requires a mechanism to make sure that as much salient information is transformed from the source to the target as possible. 2) additionally, bert is pre-trained for sentences @xcite , thus it is deficient in distinguishing key points from non-key points within long-range dependencies, which leads to inconsequential phrases or words being copied into the summary. this is mainly due to the lack of ability to recognize topic-signifying words from the document @xcite . similar problems have been encountered in neural machine translation (nmt) @xcite . @xcite developed an encoder-decoderreconstructor framework with a two-step process that first translates a sentence into another language and then reconstructs the translation back to the source language. the newly added reconstructor rewards the model when it correctly reconstructs the input source sentence from the decoder hidden layer, which forces the salient information to be transferred from the source to the target. @xcite applied this concept to unsupervised sentence compression, where the model consists of two encoder-decoder pairs so that no large textsummary dataset is needed. inspired by the above, we propose three recon-struction methods for neural document summarization. the intuitive approach is a reconstructor that rebuilds the source document from the decoder hidden layer in an encoder-decoder architecture that assigns the corresponding likelihood as a reconstruction loss. when the reconstructed text is significantly different from the original one due to the model incorrectly omitting or repeating some parts to generate the summary, the reconstructor penalizes it during training. we refer to this approach as sequence-level reconstruction and choose it as our first implementation of the idea. in contrast to the sequence-level reconstruction, reconstructing the word embeddings of source articles would be a rather simple and effective method for document summarization. the embeddinglevel reconstruction alleviates the length discrepancy between articles and summaries by calculating the distance between the average word embeddings of the source and target sides. we therefore propose the average word embedding reconstructor as our second reconstruction method. we also present the third reconstructor that focuses on the saliency of words. generally, certain words that appear frequently in documents have little importance and should not be kept in the summary. in contrast, words that signify the main idea are supposed to be maintained in the summary. as inverse document frequency (idf) can measure how important a token is and extract salient information @xcite , we reconstruct the idf-weighted word embeddings of the source at the target to keep topic-signifying words in the summary. since the summary has far fewer tokens than the document, it is not appropriate to directly incorporate the term frequency (tf) value into our reconstructor. we therefore consider the idf as the weight of embeddings, rather than the tf-idf. in this work, we mainly adopt the pgn as baseline and incorporate the above three reconstructors upon it. we then assign a loss for each reconstructor and leverage each of them as a complement to the baseline objective. on one hand, the reconstruction objectives facilitate the model to generally focus on the entire text rather than parts of it, thereby avoiding under-attention. on the other hand, the idf-weighted method serves as a selector by giving more weights to topic-signifying words such that it can identify essential parts ( we performed experiments on two datasets and source document (cnn) a mammoth fire broke out friday morning in a kentucky industrial park, sending plumes of thick smoke over the area as authorities worked to contain the damage. the blaze began shortly before 7 a.m. at the general electric appliance park in louisville, according to mike weimer from the city 's emergency management agency. he said that there were no reports of anyone injured or trapped. video showed both smoke and bright orange flames. firefighters took up positions around the affected buildings, spraying water from the periphery. weimer told cnn that authorities didn't know what had caused the fire, which had gone to at least four alarms. according to a ge website, its facility in the louisville appliance park is "revitalizing manufacturing in the united states." the park is large, such that 34 football fields could fit in one of its warehouses in the facility. reference fire breaks out at the general electric appliance park in louisville, kentucky. city official: no is believed to be injured or trapped. pointer-generator the blaze began shortly before 7 a.m. at the general electric appliance park in louisville. authorities didn't know what had caused the fire, which had gone to at least four alarms. pointer-generator with idf-weighted embedding reconstruction a mammoth fire broke out friday morning in a kentucky industrial park. the blaze began shortly before 7 a.m. at the general electric appliance park in louisville. no reports of anyone injured or trapped. firefighters took up positions around the affected buildings. ## preliminaries in the pgn, a document x that consists of a sequence of tokens @xmath0, x 2 , ..., x i } is fed into a bidirectional lstm encoder, producing a sequence of hidden states h i . then a unidirectional lstm decoder generates its corresponding summary @xmath1, y 2 , ..., y t } word by word with the limitation of t i. i and t indicate the lengths of the source article and the summary, respectively. the pgn adopts the attention mechanism to learn the alignment and yield target tokens simultaneously. conditioned on decoder hidden state s t and context vector c t for the t-th decoding step, vocabulary distribution p vocab is as follows: during training, we use the negative log-likelihood as the loss function: we next describe the details of our proposed methods, which can be split into two main components: ## related work compared to the extractive summarization, abstractive methods are more challenging and attract attention because they can generate new words through the source document representation @xcite . with the popularity of deep learning, many neural network-based models, especially seq2seq models @xcite , have been widely applied to natural language processing tasks, such as machine translation @xcite and dialogue systems @xcite . since the work of @xcite , neural seq2seq networks with an attention mechanism have been widely utilized in the abstractive summarization tasks. however, the attention mechanism is sometimes not enough to address different problems. for example, repetitions at the word or phrase level cause grammatical errors and insufficient reflection of the main idea of the source article. therefore, the distraction method @xcite imposes a constraint over the attention that can reduce the probability of repeated content. @xcite and @xcite found that the original attention often leads to over-or under-focus without the memory of past alignment information. thus, they used the coverage concept from statistical machine translation to keep track of the attention history with an additional loss. moreover, the inability to handle out-of-vocabulary (oov) tokens also limits the fluency and readability of generated summaries. to alleviate this problem, hybrid models that combine the extractive and abstractive methods through the copy mechanism account for the vast majority of models used in the summarization task @xcite @xcite . pre-trained language models @xcite @xcite , essentially word embeddings presenting contextual representations, that were learned from large-scale corpora, have recently emerged and achieved stateof-the-art performances in a variety of nlp tasks @xcite @xcite . due to the subword tokenizer, outof-vocabulary words are rarely observed in their output even without the pointer mechanism. however, neither non-pretrained seq2seq models nor bert can measure the proportion of information transmitted from the source to the target. the idea of reconstruction can be implemented in many forms so as to adapt to different types of tasks. for example, @xcite proposed an lstm encoder-decoder model that encodes the video and reconstructs its frame sequence. @xcite proposed a reconstruction model based on nmt that consists of three sequences. if the original sentence can be reconstructed from the target, it proves that the information has been effectively transferred. another work is seq 3 @xcite with a triple sequence structure, which rebuilds the input sentence from the latent representation of the decoder in the unsupervised sentence compression task. ## conclusion in this work, we presented three reconstruction mechanisms for the neural seq2seq abstractive summarization task that reconstruct the essential information from the source document to its target summary. the proposed reconstruction methods are applicable to any attention-based seq2seq summarization architectures. experimental results on both the cnn/daily mail and newsroom datasets showed the improvements from the baselines in terms of rouge metrics and human evaluation. our analysis also indicated that the proposed reconstruction approaches can restrict the underattention to key points and over-attention to redundant parts.