Planning with Learned Entity Prompts for Abstractive Summarization (2021)

Abstract: We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains—ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum, and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.

## content planning with entity chains we introduce a new training objective for encoder-decoder generative models to do content planning while summarizing. model formulation. let d be an input document, we aim to teach our model to first generate a content plan c for the summary s as p(c|d), and then generate the summary s as p(s|c, d). we define the ordered chain of entities observed in the summary s as its content plan. instead of modeling p(c|d) and p(s|c, d) separately, we take a simpler approach, we train an encoder-decoder model to encode the document d and generate the concatenated content plan and summary sequences c; s, essentially the decoder first predicts the entity chain c and then continues predicting the summary s using both c and d. we prefix c and s with special markers ''[entitychain]'' and ''[sum-mary]'', respectively, as shown in figure 1 . if s consists of multiple sentences, we use sentence markers ''|||'' to mark them in c. the model is trained with the standard maximum-likelihood objective generating the augmented target c; s. pretraining content plans. we modified pegasus @xcite to pretrain our models for entity-level content planning and summary generation. 2 in particular, we select a maximum of n most important sentences using self-rouge from an input document, the selected sentences work as a proxy for a human-authored abstractive summaries for the rest of the document. we construct a target by prepending the selected sentences dynamically with their entity chain. our model is then trained to generate this target from the rest of the document. 3 modeling entity-level lexical cohesion and coherence. as entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion @xcite and coherence @xcite in the summary. we hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in frost) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded. ## conclusion in this work, we proposed the use of entity chains, both during pretraining and finetuning, to better plan and ground the generation of abstractive summaries. our approach achieves state-of-the-art rouge results in xsum and samsum, and competitive results in cnn/dailymail. compared to other guided summarization models, ctrl-sum and gsum, which perform slightly better in cnn/dailymail, our approach is drastically simpler to implement, trains by augmenting the targets only, utilizes no additional parameters than the baseline pegasus model, and does not rely on any external systems to augment the inputs during inference. we further demonstrate that by modifying the entity chain prompts, we can easily control hallucinations in summaries.