Compressive Summarization with Plausibility and Salience Modeling (2020)

Abstract: Compressive summarization systems typically rely on a seed set of syntactic rules to determine under what circumstances deleting a span is permissible, then learn which compressions to actually apply by optimizing for ROUGE. In this work, we propose to relax these explicit syntactic constraints on candidate spans, and instead leave the decision about what to delete to two data-driven criteria: plausibility and salience. Deleting a span is plausible if removing it maintains the grammaticality and factuality of a sentence, and it is salient if it removes important information from the summary. Each of these is judged by a pre-trained Transformer model, and only deletions that are both plausible and not salient can be applied. When integrated into a simple extraction-compression pipeline, our method achieves strong in-domain results on benchmark datasets, and human evaluation shows that the plausibility model generally selects for grammatical and factual deletions. Furthermore, the flexibility of our approach allows it to generalize cross-domain, and we show that our system fine-tuned on only 500 samples from a new domain can match or exceed a strong in-domain extractive model.

## introduction compressive summarization systems offer an appealing tradeoff between the robustness of extractive models and the flexibility of abstractive models. compression has historically been useful in heuristic-driven systems @xcite @xcite or in systems with only certain components being learned @xcite @xcite . end-to-end learning-based compressive methods are not straightforward to train: exact derivations of which compressions should be applied are not available, and deriving oracles based on rouge @xcite @xcite optimizes only for content selection, not grammaticality or factuality of the summary. as a result, past approaches require significant engineering, such as creating a highly specific list of syntactic compression rules to identify permissible deletions @xcite @xcite . such manually specified, hand-curated rules are fundamentally inflexible and hard to generalize to new domains. in this work, we build a summarization system that compresses text in a more data-driven way. first, we create a small set of high-recall constituency-based compression rules that cover the space of legal deletions. critically, these rules are merely used to propose candidate spans, and the ultimate deletion decisions are controlled by two data-driven models capturing different facets of the compression process. specifically, we model plausibility and salience of span deletions. plausibility is a domain-independent requirement that deletions maintain grammaticality and factuality, and salience is a domain-dependent notion that deletions should maximize content selection (from the standpoint of rouge). in order to learn plausibility, we leverage a pre-existing sentence compression dataset @xcite ; our model learned from this data transfers well to the summarization settings we consider. using these two models, we build a pipelined compressive system as follows: (1) an off-the-shelf extractive model highlights important sentences; (2) for each sentence, high-recall compression rules yield span candidates; (3) two pre-trained transformer models @xcite judge the plausibility and salience of spans, respectively, and only spans which are both plausible and not salient are deleted. we evaluate our approach on several summarization benchmarks. on @xcite , wikihow @xcite , xsum @xcite , and reddit @xcite , our compressive system consistently outperforms strong extractive methods by roughly 2 rouge-1, and on cnn/daily mail @xcite , we achieve state-of-the-art rouge-1 by using our compression on top of matchsum @xcite extraction. we also perform additional analysis of each compression component: human evaluation shows plausibility generally yields grammatical and factual deletions, while salience is required to weigh the content relevance of plausible spans according to patterns learned during training. furthermore, we conduct out-of-domain experiments to examine the cross-domain generalizability of our approach. because plausibility is a more domain-independent notion, we can hold our plausibility model constant and adapt the extraction and salience models to a new setting with a small number of examples. our experiments consist of three transfer tasks, which mimic real-world domain shifts (e.g., @xmath0. by finetuning salience with only 500 in-domain samples, we demonstrate our compressive system can match or exceed the rouge of an in-domain extractive model trained on tens of thousands of documentsummary pairs. ## plausible and salient compression our principal goal is to create a compressive summarization system that makes linguistically informed deletions in a way that generalizes crossdomain, without relying on heavily-engineered rules. in this section, we discuss our framework in detail and elaborate on the notions of plausibility and salience, two learnable objectives that underlie our span-based compression. ## summarization system we now describe our compressive summarization system that leverages our notions of plausibility and salience. for an input document, an off-theshelf extractive model first chooses relevant sentences, then for each extracted sentence, our two compression models decide which sub-sentential spans to delete. although the plausibility and salience models have different objectives, they both output a posterior over constituent spans, and thus use the same base model architecture. we structure our model's decisions in terms of separate sentence extraction and compression decisions. let s 1 , . . . , s n denote random variables for sentence extraction where s @xmath1. let c pl 11 , . . . , c pl nm , denote random variables for the plausibility model, where c pl @xmath2. an analogous set of c sal ij is included for the salience model. these variables are modeled independently and fully specify a compressive summary; we describe this process more explicitly in section 4.4. ## training and inference the extraction and compression models in our summarization system are trained separately, but both used in a pipeline during inference. because the summarization datasets we use do not come with labels for extraction and compression, we chiefly rely on structured oracles that provide supervision for our models. in this section, we describe our oracle design decisions, learning objectives, and inference procedures. foot_3 ## experimental setup we benchmark our system first with an automatic evaluation based on rouge-1/2/l f 1 @xcite ). 7 our experiments use the following english datasets: cnn/dailymail @xcite , cnn (subset of cnn/dm), new york times @xcite , wikihow @xcite , xsum @xcite , and reddit @xcite . 8 we seek to answer three questions: (1) how does our compressive system stack up against our own extractive baseline and past extractive approaches? (2) do our plausibility and salience modules successfully model their respective phenomena? (3) how can these pieces be used to improve crossdomain summarization? 6 our pipeline overall requires 3x more parameters than a standard transformer-based extractive model (e.g., bert-sum). however, our compression module (which accounts for 2/3 of these parameters) can be applied on top of any offthe-shelf extractive model, so stronger extractive models with more parameters can be combined with our approach as well. 7 following previous work, we use pyrouge with the default command-line arguments: -c 95 -m -n 2 8 see appendix a for dataset splits. ## out-of-domain experiments additionally, we examine the cross-domain generalizability of our compressive summarization system. we set up three @xmath3: (1) @xmath4, (2) @xmath5, a low-resource domain), and (3) @xmath6. for each transfer task, we experiment with two types of settings: (1) zero-shot transfer, where our system with parameters [θ e ; θ p ; θ s ] is directly evaluated on the target test set; and (2) fine-tuned transfer, where [θ e ; θ s ] are fine-tuned with 500 target samples, then the resulting system with parameters [θ e ; θ p ; θ s ] is evaluated on the target test set. as defined in section 2.1, plausibility is a domainindependent notion, thus we do not fine-tune θ p . more importantly, we see that performance via fine-tuning on 500 samples matches or exceeds in-domain extraction rouge. on @xmath7, our system outperforms in-domain extraction baselines (trained on tens of thousands of examples), and on @xmath8, it comes within 0.3 in-domain average rouge. these results suggest that our system could be applied widely by crowdsourcing a relatively small number of summaries in a new domain. one significant line of prior work in compressive summarization relies on heavily engineered rules for syntactic compression @xcite @xcite . by relying on our data-driven objectives to ultimately perform compression, our approach can rely on a leaner, much more minimal set of constituency rules to extract candidate spans. gehrmann et al. (2018) also extract subsentential spans in a "bottom-up" fashion, but their method does not incorporate grammaticality and only works best with an abstractive model; thus, we do not compare to it in this work. ## conclusion in this work, we present a compressive summarization system that decomposes span-level compression into two learnable objectives, plausibility and salience, on top of a minimal set of rules derived from a constituency tree. experiments across both in-domain and out-of-domain settings demonstrate our approach outperforms strong extractive baselines while creating well-formed summaries.