Sentence-level Planning for Especially Abstractive Summarization (2021)

Abstract: Abstractive summarization models heavily rely on copy mechanisms, such as the pointer network or attention, to achieve good performance, measured by textual overlap with reference summaries. As a result, the generated summaries stay close to the formulations in the source document. We propose the *sentence planner* model to generate more abstractive summaries. It includes a hierarchical decoder that first generates a representation for the next summary sentence, and then conditions the word generator on this representation. Our generated summaries are more abstractive and at the same time achieve high ROUGE scores when compared to human reference summaries. We verify the effectiveness of our design decisions with extensive evaluations.

## introduction abstractive summarization has improved drastically in recent years due to more efficient decoder architectures, like the transformer @xcite , and language model pretraining, such as bert @xcite . as a result of these advances, current state-of-the-art models reach the performance of extractive systems, and even surpass them on some datasets @xcite @xcite . part of this success, however, is due to the development of stronger copy mechanisms such as the pointer-generator network @xcite or attention to the source document @xcite . the so-generated summaries copy long sequences from the input document, strung together with filler words. while this achieves better results in the predominant evaluation metric rouge @xcite , it comes at the cost of the summaries' abstractiveness and coherence, two qualities that we expect from human-written summaries. in this paper, we aim to generate more abstractive summaries without sacrificing rouge and coherence. we achieve this by including a planning step at the sentence level before generating the summary word by word. the idea is to plan an outline for the next summary sentence first at a higher level to give the model more capacity for abstraction. as a result, the model has to rely less on copying the input, and thereby generates more abstractive summaries. our model, the sentence planner, is an encoder-decoder architecture. the encoder is initialized from pretrained bert weights. the decoder is hierarchical, and consists of a sentence generator that plans an outline for the summary at the sentence level, and a word generator that is conditioned on this outline when generating the summary's words. both generators attend to the source document in order to condition their predictions on the input. the sentence planner is trained endto-end to predict the words of the target summary, with an additional guidance loss that encourages the sentence generator to produce the encoder's embedding for the target next sentence. this is the first work to propose a hierarchical transformer decoder that generates a summary from latent sentence representations. 1 we extensively evaluate our model on a recently published highly abstractive dataset and an established but more extractive corpus. we show that the sentence planner generates more abstractive summaries while improving the rouge scores of a state-of-the-art model without a hierarchical decoder. we use gradient attribution to quantify the impact of the sentence generator on the model's prediction as well as how much information from the document it captures. moreover, we verify the effectiveness of our model components with an ablation study, and show that simply increasing the baseline's decoder parameters does not bring it up to par with the hierarchical decoder. our automatic evaluations are confirmed in a human evaluation study, where the sentence planner improves upon its strong baseline in each of six quality categories. our contributions are twofold: (a) we are the first to propose a hierarchical transformer decoder that generates summaries from a latent sentencelevel plan, and (b) we perform an extensive evaluation of our model on two summarization datasets and show that it produces more abstractive summaries while retaining high rouge scores, two objectives that are in opposition. ## the hierarchical decoder our approach builds on the bertsumextabs model @xcite . their model consists of an encoder initialized with an extractive summarization model, which in turn was initialized with a bert model, and a randomly initialized transformer decoder. 2 we keep the encoder the same. we replace the decoder with a hierarchical version by introducing a sentence generator that develops a high-level plan for the summary, and a word generator that is conditioned on this plan. a model diagram is shown in figure 1 . section 2.1 describes how the sentence generator develops the outline for the summary, and section 2.2 shows how the word generator makes use of it. ## experimental setup we now describe the datasets ( § 3.1) and metrics ( § 3.2) that we use to evaluate our model, and give implementation details ( § 3.3) to replicate our experiments. dataset statistics are shown in ## results we now turn to evaluation of our method. first, we show the results on curation corpus ( § 4.1). with attribution techniques ( § 4.2) and an ablation study ( § 4.3) we uncover how the model uses the sentence generator component. increasing the number of parameters of bertsumextabs (bsea) does not provide the same improvements as our approach ( § 4.4). on the cnn/dailymail dataset, our model generates more abstractive summaries while retaining high rouge scores ( § 4.5). finally, a human evaluation validates the results from our automatic metrics ( § 4.6). ## related work our proposal is the first hierarchical decoder for a transformer-based text summarization model. we survey previous work on hierarchical attention in summarization on the encoder side, and sentencelevel planning on the decoder side. @xcite first employ a content selector at the word level to decide which words are candidates for copying. they then use a pointergenerator network with just the admissible tokens to generate the summary. @xcite use hierarchical attention networks @xcite to encode the context of previous sentences, which is used to inform the translation of the next word. in contrast to these methods, we employ hierarchy on the decoder side, and generate a sentence representation for the next sentence. ## conclusion we presented the sentence planner, an encoderdecoder model with a hierarchical decoder, consisting of a sentence and a word generator. our sentence generator computes a plan for the next summary sentence. the word generator is then conditioned on this plan when generating the sentence's words. an additional loss term, which guides the sentence planner towards producing the embedding of the target next sentence, improves the sentence generator's plan. when comparing the sentence planner to a state-of-the-art model without a hierarchical decoder, it generates more abstractive and coherent summaries at higher rouge scores. in future work, we aim to apply our model to other generation tasks, such as machine translation or dialogue generation.