EASE : Extractive-Abstractive Summarization End-to-End using the Information Bottleneck Principle (2021)

Abstract: Current abstractive summarization systems outperform their extractive counterparts, but their widespread adoption is inhibited by the inherent lack of interpretability. Extractive summarization systems, though interpretable, suffer from redundancy and possible lack of coherence. To achieve the best of both worlds, we propose EASE, an extractive-abstractive framework that generates concise abstractive summaries that can be traced back to an extractive summary. Our framework can be applied to any evidence-based text generation problem and can accommodate various pretrained models in its simple architecture. We use the Information Bottleneck principle to jointly train the extraction and abstraction in an end-to-end fashion. Inspired by previous research that humans use a two-stage framework to summarize long documents (Jing and McKeown, 2000), our framework first extracts a pre-defined amount of evidence spans and then generates a summary using only the evidence. Using automatic and human evaluations, we show that the generated summaries are better than strong extractive and extractive-abstractive baselines.

## introduction pretrained sequence-to-sequence language models such as bart @xcite , t5 @xcite and their variants have achieved state-of-theart results on various tasks such as summarization, machine translation, and data2text tasks @xcite . despite the higher fidelity compared with models without pretraining for tasks such as summarization @xcite , the lack of interpretability in abstractive generation remains an obstacle to their broader adoption. extractive summarization systems, on the other hand, have the advantage of being interpretable but are too restrictive by forcing the output to be spans from the document, reducing their figure 1 : an example of a summary and its evidence (highlighted) as generated by our framework. naturalness, coherence, and conciseness. in this paper, we propose ease, a novel framework that combines the two systems to produce natural summaries that can be traced back to an interpretable extractive summary. our general framework can accommodate different pretrained models and suitable for any evidence-based text generation task. the existing extractive-abstractive systems can be divided into three main categories: 1-relying on attention for interpretability @xcite . due to the probabilistic nature of the attention mechanism, it falls short of providing usable evidence; 2-providing word-level evidence for the generated summaries @xcite . though more useful than attention, this evidence is too granular to be useful for humans; 3-training the content selector separately using pseudo labels or other heuristics @xcite . in contrast, we seek a theoreticallygrounded model that can learn the evidence extraction end-to-end. perhaps the closest work to ours is zhao et al. (2020) focusing on long-document summarization by training a joint extractive-abstractive model via weak supervision. though a complicated and spe-cific framework, it achieves poor results on benchmarks such as cnn/dm. ease on the other hand, is based on the information bottleneck (ib) principle @xcite , which formalizes the trade-off between the size of the extracted evidence and the information provided for the generation of the final output. while this method has been successfully adopted by prior work for a simpler discriminative task @xcite , we extend it to generative tasks where the extracted evidence can be viewed as a coarse version of the final abstractive output. we leverage pretrained language models that first extract the necessary evidence from the source document (extractor) and then, using only the extracted evidence spans, generate the final output (abstractor). fig. 1 shows an example of the summary and evidence generated by our system. our main contributions are as follows: ## extractive-abstractive framework there exists evidence that humans use a twostage extractive-abstractive framework to summarize long documents @xcite by first extracting salient parts and then deciding what to eliminate, reword, and reorganize. inspired by this, we propose ease, a framework that learns extraction and abstraction collectively in an end-toend fashion. this not only provides interpretable evidence for the generated summary, which can be many times smaller than the original document, but also reduces the effective input length used during abstraction. this has been shown to directly correlate with the extent of hallucination in pretrained language models @xcite . in order to formalize the problem, we use the ib principle to learn an optimal model between the original document x and the final summary y through a compressed representation z. the ib objective is to minimize the following: similar to @xcite , we assume that the mask variables over individual words are conditionally independent given the input x. this means that the evidence z can contain redundancies, as the extractor chooses evidence individually without conditioning on prior extractions. since the extracted evidence is not the final summary, the abstractor still has the opportunity to eliminate redundancies. @xcite explore a modeling approach that keeps track of the current state of the summary, but we leave this direction to future work. formally, optimizing the loss in (2) would result in the extractor masking no tokens and hence, maximizing the mutual information between the input and output of the abstractor. therefore, the second loss term is a sparsity constraint to ensure that the extractor's output is a measurable subset of input tokens and can be used as evidence for the abstractor output: ## experimental settings datasets: we primarily experiment with the cnn/dailymail dataset @xcite owing to its extractive-like nature; its summaries are typically closely related to the source sentences. we also present results on the xsum @xcite dataset, a highly abstractive dataset in which summaries can be viewed as a title for the source documents. model hyperparameters and evaluation metrics: we initialize the seq-to-seq abstractor with the bart-large model and initialize the extractor with the bart-base encoder. we use the fairseq codebase foot_0 for our experiments and use the same hyperparameters as used for fine-tuning bart on cnn/dm and xsum by the official codebase. specifically, we fine-tune bart using a polynomial decay learning rate scheduler with the adam optimizer (kingma and ba, 2014). we use a learning rate of 3e-5 with 500 warmup steps and train for 20000 steps. during our initial experiments, we observed similar results for values of β ∈ [1, 10] in (4). we use β = 5 in our reported results. we use rouge f1 scores (r1/r2/rl) for the automatic evaluation. rouge scores were calculated using the files2rouge toolkit foot_1 . ## results in this section, we report the performance of our model from both automatic and human evaluation perspective, along with ablation studies. figure 4 shows example summaries along with evidence highlighted from our system at different sparsity levels. ## conclusion in this paper, we introduced ease, an extractiveabstractive framework for summarization tasks that trains an extractor and an abstractor in an end-toend fashion. the extracted evidence can be viewed as an interpretable extractive summary of the summary from which the final summary is generated by the abstractor. we show that our sentence-level extractive-abstractive summarization systems are better than strong extractive-abstractive baselines and either on-par or only slightly lower in quality compared to strong abstractive baselines.