DYLE : Dynamic Latent Extraction for Abstractive Long-Input Summarization (2022)

Abstract: Transformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this paper, we present DYLE, a novel dynamic latent extraction approach for abstractive long-input summarization. DYLE jointly trains an extractor and a generator and treats the extracted text snippets as the latent variable, allowing dynamic snippet-level attention weights during decoding. To provide adequate supervision, we propose simple yet effective heuristics for oracle extraction as well as a consistency loss term, which encourages the extractor to approximate the averaged dynamic weights predicted by the generator. We evaluate our method on different long-document and long-dialogue summarization tasks: GovReport, QMSum, and arXiv. Experiment results show that DYLE outperforms all existing methods on GovReport and QMSum, with gains up to 6.1 ROUGE, while yielding strong results on arXiv. Further analysis shows that the proposed dynamic weights provide interpretability of our generation process.

## introduction transformer-based @xcite pretrained language models (plms) such as bart @xcite and t5 @xcite , have achieved state-of-the-art performance on short text summarization. however, due to the high memory complexity of the full self-attention @xcite , plms still struggle to handle long inputs @xcite . model efficiency and summary quality present a pair of challenges @xcite : models need to capture information scattered across the long input while maintaining a low computational cost. figure 1: overview of our approach. the input is a document x (each @xmath0, and the output is a summary y. prior models tackled long input summarization mostly in four ways. first, sparse attention @xcite @xcite is used to reduce the memory complexity of the transformers so that they can attend to more tokens. second, extract-then-generate methods extract salient texts from the input and then summarize the extracted texts. extractors are either independently trained with full supervision @xcite or optimized using reinforcement learning @xcite @xcite . third, models are proposed to divide source text into sections @xcite @xcite which are individually summarized and combined to form a full summary. fourth, hierarchical models @xcite improve summarization by capturing sentence or discourse level dependencies. we elaborate on these four directions and their limitations in section 2. we believe that the extract-then-generate approach mimics how a person would handle longinput summarization: first identify important pieces of information in the text and then summarize them @xcite . the extractthen-generate framework is based on the assumption that salient information useful for summarization only occupies a small portion of the input, which is a sensible assumption given the long input length. this approach shortens the source input to a pre-set length, which addresses the main challenge of the model not being able to handle longer input beyond a certain limit. however, previous separately-trained extract-then-generate approaches are limited as they suffer from cascaded errors from the extractor to the generator. though various reinforcement learning techniques are introduced to bridge the two steps, they have noticeable drawbacks (discussed in section 3.3), and we argue that the long input makes this approach suboptimal. in this paper, we propose a new approach for long-input summarization: dynamic latent extraction for abstractive summarization (dyle). dyle jointly trains the extractor and the generator and keeps the extracted text snippets latent. for an output token, dyle compute its probability conditioned on each input snippet separately, and its generation probability is computed by marginalizing over all the input snippets under a learned dynamic weights assigned by the generator conditioned on the previously generated tokens. we optimize the extractor with two surrogate losses. first, we compute the extractive oracle based on the reference summary with a greedy search over the best rouge scores. these oracle snippets are used as targets for the extractor learning signal. moreover, we propose consistency loss to encourage the extractor to approximate its own predicted weights on the snippet to the averaged dynamic weights predicted by the generator. we conducted experiments on three long-input summarization datasets: govreport @xcite and arxiv @xcite for longdocument summarization, and qmsum @xcite for long-dialogue summarization. our method dyle largely outperforms existing methods on govreport and qmsum, while achieving strong results on arxiv. notably, dyle yields gains of 4.2/6.1/4.0 of rouge-1/2/l points over the previous best method on govreport. these experiments demonstrate the generalizability of dyle to multiple long-input summarization tasks. we summarize our contributions as follows: ## related work we introduce in detail the four main categories of methods in recent work to address long-input summarization tasks. ## our approach an overview of our approach is shown in figure 1. in section 3.1, we formulate our task and the extractor-generator framework. in section 3.2, we introduce our parameterization of the extractor for long inputs. in section 3.3, we introduce generator formulation and the novel consistency loss. the extractor module is both optimized with the consistency loss and the oracle loss, which we elaborate on in section 3.4. the overall training objective is summarized in section 3.5. ## analysis and discussion analysis of extracted snippets we are interested in the amount of salient information passed to the generator. to investigate this, we report the decomposed precision and recall of rouge scores in ## conclusions in this paper, we propose the first framework that jointly trains an extract-then-generate model with latent extraction. the first-step extraction picks out salient information from the long input, thereby extending the input length that the model can handle. our novel joint training method addresses the challenge of information loss associated with the prior extract-then-generate approaches. our model largely outperforms the current state-of-the-art on govreport and qmsum, while achieving strong results on arxiv. lastly, dyle has the advantages of being able to process arbitrarily long input with a lower memory cost and interpretable generator weights.