Generating EDU Extracts for Plan-Guided Summary Re-Ranking (2023)

Abstract: Two-step approaches, in which summary candidates are generated-then-reranked to return a single summary, can improve ROUGE scores over the standard single-step approach. Yet, standard decoding methods (i.e., beam search, nucleus sampling, and diverse beam search) produce candidates with redundant, and often low quality, content. In this paper, we design a novel method to generate candidates for re-ranking that addresses these issues. We ground each candidate abstract on its own unique content plan and generate distinct plan-guided abstracts using a model’s top beam. More concretely, a standard language model (a BART LM) auto-regressively generates elemental discourse unit (EDU) content plans with an extractive copy mechanism. The top K beams from the content plan generator are then used to guide a separate LM, which produces a single abstractive candidate for each distinct plan. We apply an existing re-ranker (BRIO) to abstractive candidates generated from our method, as well as baseline decoding methods. We show large relevance improvements over previously published methods on widely used single document news article corpora, with ROUGE-2 F1 gains of 0.88, 2.01, and 0.38 on CNN / Dailymail, NYT, and Xsum, respectively. A human evaluation on CNN / DM validates these results. Similarly, on 1k samples from CNN / DM, we show that prompting GPT-3 to follow EDU plans outperforms sampling-based methods by by 1.05 ROUGE-2 F1 points. Code to generate and realize plans is available at https://github.com/griff4692/edu-sum .

## introduction generating diverse abstracts and then re-ranking can lead to large performance gains (in rouge) @xcite over the standard approach of generating a single summary. typically, diversity is controlled for at the token-level by modifying beam search to introduce sampling (top-k @xcite , nucleus @xcite ) or directly penalize repetition @xcite . <@xmath0,</@xmath1.</@xmath2.</@xmath3.</@xmath4.</@xmath5.</@xmath6, the first part, anyway).</@xmath7, martial-arts star and world's favorite tough-guy joke subject was born march 10, 1940,</@xmath8.</@xmath9.</@xmath10,</@xmath11.</@xmath12. the actor and martial arts master is now known as subject of tough-guy one-liners . ## motivation & analysis elemental discourse units. prior work has shown that reference summary sentences usually combine information from multiple document sentences, while removing non-essential descriptive details @xcite @xcite . as such, an ideal extractive plan would select only the relevant subsentential units to incorporate into the final summary. to achieve this, we rely on discourse level segmentation from rhetorical structure theory @xcite to segment document sentences into elementary discourse units (edus), which are contiguous spans of tokens representing avg. plan rouge f1 0.410 0.413 0.415 0.418 0.420 0.423 2 4 6 8 10 12 14 16 bs dbs nucleus independent clauses. edus are a good approximation @xcite of summary content units (scus) written by human annotators for the pyramid evaluation method @xcite . to extract edus, we use the neural parser @xcite , fine-tuned from xlmroberta-base @xcite on rst treebanks from 6 languages, to segment sentences into non-overlapping, contiguous edu fragments. their model merges short edus (< 5 tokens) to prevent fragmentation. as such, these edu fragments are closer to proposition-level extraction than other possible units of extraction, e.g., entities. text unit # in doc # in oracle rouge-1 f1 sentences 29.2 3.3 57.8 edu 51.6 5.3 61.7 ## plan-guided abstraction (pga) at a high-level, we ensure foot_2 uniqueness by conditioning each candidate on its own unique content plan, and minimize quality degradation by only using the top beam from the abstractive decoder. more specifically, we transform a bart lm into a hierarchical encoder, single-decoder model, which learns to copy extractive content plans at the edu-level ( §4.1). another encoder-decoder model (bart for cnn/dm and nyt, pegasus for xsum) learns to generate the reference given special markers to indicate the content plan ( §4.2). figure 4 4.1 generating edu-level plans tl;dr. inspired by the aredsum-seq model @xcite , which itself is based off the hierarchical encoder from bertsumext @xcite , we adapt a bart conditional language model such that it is able to generate extractive edu fragments left-to-right, in the order in which they appear. the decoder uses a copy mechanism for edus and a special end of extract token. the special token enables edu extractive plans to have variable length. notation. a document d can be expressed as a list of k non-overlapping edu segments: we note that adding edus to an extractive plan in the order in which they appear in the document is nonstandard. most extractive models build summaries in a confidence-first fashion, as in @xcite . we text @xmath13: learning to generate edu plans h' h'' legend edus @xmath14. edu-level mean pooling h_edu 1 h_edu 2 <@xmath15.8 0.2 0.0 0 0.1 0.9 copy distribution special @xmath16. step 2: learning to generate abstracts from edu plans oracle @xmath17. random plan likelihood token-level decoder oracle abstract @xmath18. token-level encoder likelihood step 3: generating plan-guided abstract candidates edu 1 edu 1 edu 2 k beams top beam top beam cand plan cand abstract 1 cand abstract k token-level e-d token-level e-d edu decorated document plan decorated document cand abstract token-edu hierarchical e-d inference for step 1 inference for step 2 figure 4: plan-guided abstraction (pga). in the first step, a token-level encoder processes a document decorated with special edu boundary markers. edu-level hidden states are formed with mean-pooling and serve as the inputs to a shallow edu-level encoder-decoder, which learns to auto-regressively copy oracle edu plans. in the second stage, a plan-guided abstractor learns to generate abstractive reference summaries from inputs decorated with edu boundary markers to indicate the oracle plan, as well as a random distractor plan for unlikelihood training. during inference, the pga generates a single summary for each unique content plan returned by the top k beams of the edu generator. to encode edus, we bracket each edu with @xmath19. we pass the full document: edu markers and tokens through a pre-trained bart encoder, and extract hidden states for each edu with mean pooling over each token within the edu (including the start and stop tokens): {h s 1 ,...,h s 1 }. then, the edu representations are modeled by a newly initialized edu-level bart encoder: e(eoe) represents a learned embedding for the end of extract token. positional embeddings are added to each edu representation (h s i ) to indicate its position in the document, before being passed through the stacked transformer layers in the encoder. at decoder timestep k with hidden state h * k and partial extract s * t , each valid next output (s @xmath20,@xmath21, which can be represented as foot_3 : plan objective. given the above probability distribution, we treat the plan generator as a standard lm and train it with maximum likelihood estimation (mle) of the oracle plan given the source document. oracle labels. as discussed in §3, we use the greedy search algorithm proposed by @xcite to generate oracle edu extractive plans. inference. as a functional lm, we generate distinct edu extractive plans with beam search. ## experimental setup datasets. we use the same datasets as in @xcite , which are cnn / dailymail @xcite , the new york times annotated corpus @xcite , and xsum @xcite . the first two are more extractive while xsum is more abstractive and contains highly noisy references @xcite . we use code from @xcite for data pre-processing and splitting of the corpus, and treat the archival abstract as the ground-truth reference. metrics. we compare summaries to references with rouge 1/2/l f1 @xcite and bertscore f1 @xcite . we use the standard perl rouge script for rouge scoring with ptb tokenization and lowercasing, as in @xcite . for bertscore, we use the default model (roberta-large) and settings from the widely-used bert-score python package foot_4 . baselines. we generate 16 candidates with different decoding methods: beam search, diverse beam search, and nucleus sampling. we use google/pegasus-xsum for xsum, facebook/bart-large-cnn for cnn, and fine-tune a bart-large model on the nyt corpus. for nyt, we fine-tune using a standard mle loss for up to 10 epochs, choosing the best model based on validation rouge score. these are also the checkpoints used to initialize our plan extractor token-level encoder and guided abstractor. we also compare our method to previous work on summary re-ranking. simcls @xcite and brio-ctr @xcite both generate 16 candidates via diverse beam search using the same pre-trained weights as in our work foot_5 . the major difference between the papers is that a roberta @xcite classifier is used for re-ranking simcls, while in brio, the model likelihoods are calibrated to rouge rankings. summareranker (ravaut et al., 2022a) trains a roberta-based mixture of experts classifier on up to 60 candidates ensembled from multiple decoding methods (beam search, diverse beam search, nucleus sampling, and top-k sampling). we report their best ensemble configuration for cnn and nyt, which uses dataset-specific fine-tuned pegasus @xcite checkpoints from the huggingface transformers library @xcite . summafusion (ravaut et al., 2022b) fuses candidate summaries into a single summary. candidates are generated with diverse beam search from the same pegasus checkpoint for xsum (google/pegasus-xsum). training details. for the edu plan generator, we initialize the token-level encoder from fine-tuned summarization checkpoints for each dataset (listed above in baselines paragraph). the edu-level bart encoder and decoder are randomly initialized to have two layers (using a bart-large configuration to determine parameter dimensions). for both edu-extract and guided abstract training, we fine-tune with pytorch lightning @xcite for a maximum of 150,000 steps with 200 warmup steps, a learning rate of 1e-5, batch size of 16, and weight decay of 5e-5. for xsum, we fine-tune plan-guided abstraction from google/pegasus-xsum and use a learning rate of 1e-4 and a batch size of 64. for the edu generator, we select the checkpoint that maximizes the rouge score on the validation set. for the plan-guided abstractor, we select the checkpoint that maximizes the oracle-guided abstract rouge score. we grid-searched λ and β from equation 1 over [0,0.1,1,10] and selected based on top-ranked validation set summaries. for nyt, we set λ=1 and β =0 from equation 1. no regularization is needed. for cnn and xsum, we use more regularization: λ=1 and β =10. for xsum, we enforce the last plan beam to be the null-plan (no edu guidance) foot_6 . ## results please refer to appendix a for an analysis of the beam consistency of pga candidates versus baselines. re-ranked performance. extractive planning may be less useful when reference summaries are shorter and noisier. xsum references have been shown to contain entity-based "hallucinations"-content that is unsupported by the input document @xcite . ## guiding gpt with edu plans background. to date, gpt models @xcite have only been evaluated as summarizers in the conventional single candidate setup @xcite . in zero and few-shot settings, gpt summaries have been shown to underperform fine-tuned models with regards to referencebased metrics, yet over-perform according to human judgments @xcite . ## conclusion in this paper, we demonstrate that offloading content selection to a dedicated extractor, rather than relying on the decoder to perform both content selection and surface realization, can lead to better and more diverse content selection across beams, which ultimately leads to increased rouge scores for top-ranked summaries after applying a re-ranker. edu plan-guided abstraction exhibits other encouraging traits, such as an increased level of fusion and scalability to longer inputs.