Stepwise Extractive Summarization and Planning with Structured Transformers (2020)

Abstract: We propose encoder-centric stepwise models for extractive summarization using structured transformers – HiBERT and Extended Transformers. We enable stepwise summarization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure. Our models are not only efficient in modeling the structure of long inputs, but they also do not rely on task-specific redundancy-aware modeling, making them a general purpose extractive content planner for different tasks. When evaluated on CNN/DailyMail extractive summarization, stepwise models achieve state-of-the-art performance in terms of Rouge without any redundancy aware modeling or sentence filtering. This also holds true for Rotowire table-to-text generation, where our models surpass previously reported metrics for content selection, planning and ordering, highlighting the strength of stepwise modeling. Amongst the two structured transformers we test, stepwise Extended Transformers provides the best performance across both datasets and sets a new standard for these challenges.

## introduction extractive document summarization is the task of creating a summary by identifying (and subsequently concatenating) the most important sentences in a document @xcite . in recent years this task has matured significantly, mostly thanks to advances in deep neural networks. @xcite conceptualize extractive summarization as a sequence labeling task in which first a hierarchical long short-term memory network (lstm; @xcite is used to encode a document and then another lstm is used to predict for each sentence whether it should be included in the summary. this architecture was later adopted by @xcite , @xcite , @xcite , @xcite and @xcite . following the success of pre-trained transformerbased architectures for many tasks @xcite , the current state-of-theart approach to extractive summarization uses transformers to learn sentence representations and to rank sentences by their saliency @xcite @xcite @xcite . the top scoring sentences are then assembled to produce an extract of the document. summaries built in this fashion @xcite @xcite are prone to contain redundant information. several recent approaches have explored mechanisms to better handle redundancy, such as heuristic-based trigram blocking (triblk; @xcite , handcrafted feature-driven models @xcite and redundancy aware neural sequence models @xcite . one common problem with these models is that their focus is limited to content overlap and to respecting length budgets. however, these are but a small subset of the dimensions necessary to produce informative and coherent summaries. ideally, models would utilize enriched document and summary representations in order to implicitly learn better extractive plans for producing summaries @xcite . one such method is stepwise summarization @xcite , where a summary is constructed incrementally by choosing new content conditioned on previously planned content. in this paper, we propose encoder-centric stepwise models for extractive summarization using structured transformers. structured transformers are transformer-based architectures that have the flexibility to model some form of structure of the input, e.g., hierarchical document structure. in this paper, we specifically study two such architectures -hibert @xcite and extended transformers construction (etc; @xcite . details of these are given in sections 4 and 5. we enable stepwise summarization by injecting the previously planned summary content into the structured transformer as an auxiliary sub-structure. the model then can holistically learn any documentlevel coherence properties, such as saliency, redundancy, and ordering, embodied in the gold summaries. this differs from other methods which are either task specific (e.g., redundancy aware modeling in @xcite or not holistic (e.g., manually curated features in @xcite ). an added advantage of structured encoders is that they break the quadratic attention mechanism of transformers @xcite , making them more efficient and able to process longer inputs, instead of truncating the inputs to 512 tokens @xcite , which is critical for long inputs and outputs which require non-trivial planning. when evaluated on the cnn/dailymail summarization dataset @xcite , we achieve stateof-the-art performance in terms of rouge @xcite without any redundancy @xcite or sentence selection mechanisms @xcite . our model's task-agnostic approach allows it to implicitly learn and leverage content plans directly from the data. moreover, structured transformers form the basis of our model, which are flexible in terms of content type (e.g., text or tables) that can be modeled. we demonstrate this by learning intricate extractive content plan for the rotowire table-to-text generation task @xcite . this task requires the generation of long summaries from large score tables detailing the the specifics of a sports match, which often necessitates dedicated content selection and planning models to generate a high-quality summary @xcite . we show that our stepwise framework achieves higher content selection, planning and ordering scores relative to prior work with task-specific planning mechanisms. the contributions of the paper are as follows: 1) this is first study to use etc @xcite for summarization for its ability and flexibility to better model long and structured inputs; 2) we pro-pose augmentions of two structured transformers, hibert and etc, in order to enable stepwise models for extractive planning; 3) we demonstrate empirically that our models are general purpose and can be adapted as an extractive document summarizer or as a content planner for table-to-text generation; 4) our experiments highlight the effectiveness of stepwise modeling, specifically stepwise etc, which sets a new standard for both tasks. ## problem: stepwise content extraction we define a general paradigm for stepwise content extraction that can be easily tailored to both extractive summarization and table-to-text generation. given an input @xmath0, s 2 , . . . , s n } with n content units, the goal is to learn an extractive content plan, i.e., s @xmath1, s @xmath2, of length m; s m is an empty unit (ø) denoting the end of the plan. we formulate this as an iterative ranking problem @xcite where at each k-th step (1 ≤ @xmath3, we select s @xmath4, d; θ) with model parameters θ. the selected content is then added to s k-1 to construct s k . the best plan ŝ can be defined as: for extractive document summarization, let @xmath5, s 2 , . . . , s n } be a document with n sentences. our goal is to learn an extractive plan (or summary in this case) ŝ which best summarizes d. for table-to-text generation, we represent a table with n records as @xmath6, s 2 , . . . , s n }. we aim to generate a plan s m that can be used by a text generator to generate a meaningful and coherent summary. for exposition, we use the extractive document summarization setup to introduce our stepwise models with hibert @xcite and etc @xcite in the following sections. specifically, we use 'sentence' as a content unit and 'previously' or 'partially generated summary' for a previously selected content plan. ## stepwise hibert hierarchical encodings have been used to model input structure with lstms @xcite @xcite . @xcite proposed hibert with two stacked transformer encoders @xcite for extractive summarization (see the middle diagram in figure 1 ): a sentence encoder that independently builds representations for each sentence in the document; and a document encoder that operates over sentence encodings to build contextual representations for all sentences. these contextual sentence representations are then ingested by a classifier to predict the salience score of each sentence in the document. as in standard transformers, both encoders have multiple layers with each layer composed of a multi-head self-attention layer followed by a feed-forward sub-layer with residual connections @xcite and layer normalizations @xcite . for stepwise hibert, at time step k, we modify the document encoder with the content plan s k-1 , which is the previously selected sentences in the summary. this is depicted in figure 2 (left) and allows the model to implicitly select new sentences relative to the previously generated summary. sentence and document encoders. let @xmath7, s 2 , . . . , s n } be a document, where s @xmath8, w i 2 , . . . , w i |s i | } is a sentence in d and w i j is a token in s i . s i is first mapped to a continuous space e s @xmath9, e i 2 , . . . , e i |s i | } where e i @xmath10. e(w i j ) and p token j are the token ... t t-1 t-2 span nodes input tokens input tokens input tokens layer 1 embeddings layer 2 embeddings segments input tokens block embeddings blocks input tokens special global token ... t t-1 t-2 input tokens layer 1 embeddings layer 2 embeddings segments memory compressed memory standard transformer: hierarchical attention (hibert): transformer xl: bp transformer: star transformer: compressive transformer: long input global input full g2g attention local l2l attention full g2l and l2g attention global-local attention (etc): ... t-2 input tokens input tokens block embeddings blocks input tokens global token ... t-2 mem compressed memory hierarchical attention (hibert): compressive long input global input full g2g attention local l2l attention full g2l and l2g attention global-local attention (etc): stepwise modeling. at step k, let s k-1 = {s 1 , s 2 , . . . , s k-1 } be the partial summary with (k -1) previously extracted sentences. in addition to ĥd , our document encoder takes the sum- while document self-attention learns the contextual hidden representation h @xmath11, summary selfattention learns the contextual hidden representation h @xmath12. we share the parameters of the document and summary selfattention layers. the document-summary attention then builds the contextual hidden representation as key and values @xcite . in addition to the introduction of stepwise mechanism to hibert, our positional embeddings, p token j , p doc ## stepwise etcsum there has been growing interest in addressing the limitation of the transformer architecture used in bert @xcite where memory usage scales quadratically with the size of the input @xcite @xcite @xcite @xcite . hibert alleviates this problem by modeling each sentence independently; the memory usage in hibert scales with the square of the number of sentences, and the square of the maximum length of any sentence. however, the main disadvantage of this approach is that token-level attention across sentences is prohibited and long range attention only happens indirectly at the second-stage encoder (see the middle diagram in figure 1 ). recently, extended transformer construction (etc; ainslie et al., 2020) provides an alternative. it alleviates the quadratic memory growth by introducing sparsity to the attention mechanism via its novel global-local attention mechanism (see the rightmost diagram in figure 1 ). this not only permits encoding of long inputs, foot_1 but also enables a mechanism to model structure directly through nodes in the global attention layer. global-local attention. the etc model architecture receives two inputs: a long input, which in most cases corresponds to the text to be encoded; and an auxiliary global input, which serves as inductive bias features. first, the model builds an attention map, called long-to-long, across the long input with a sparse local attention of fixed length, this bypasses the quadratic memory complexity and allows to scale input lengths to the thousands of tokens, but limits the attention span of tokens to their nearest neighbors. to overcome this limitation, the global-local attention defines three other attention parts: globalto-global, global-to-long and long-to-global, all with unrestricted attention. this allows tokens arbitrarily far apart to attend to each other with at most one hop through the global input tokens. we refer the reader to ainslie et al. (2020) for more details. the right parts of figures 1 and 2 illustrate these four types of attentions and the sparsity diagrams where each cell in a row i and column j is different than white input token w i can attend to input token w j , same relative position embeddings are indicated by using the same color. stepwise modeling. given the document d and its partial summary s k-1 at step k, we construct an input @xmath13, . . . , w |d s k-1 | } by concatenating the document d and the partial summary s k-1 . etc replaces absolute position encodings with relative position encodings @xcite to easily adapt to greater input lengths than seen during pretraining. in addition to modeling relative positions in an input sequence, relative position encodings in etc are also used to model arbitrary pairwise token relations useful for structured inputs. we used the auxiliary global input to represent sentence structure. specifically, following (ainslie et al., 2020), we placed one auxiliary token in the global input per each sentence in the input i. we linked the global tokens with the input tokens by using relative position labels to represent whether each token belongs to that sentence. global-toglobal attention is left unrestricted, allowing all sentences to attend to each other. this result is summary-informed contextualized input token representations via attention through the global nodes. in the rest of the paper we refer to this summarizer by stepwise etcsum. similar to hibert, we take the first token hidden representation h i 1 as the representation for the sentence s i . finally, sentence embeddings are passed to the softmax layer for salience scoring. both hibert and etcsum are then trained with the cross entropy loss. to understand the importance of modeling long documents for extractive summarization, we also trained bertsum, similar to @xcite , with a receptive capacity of 512 tokens initialized with the bert checkpoint. our bert-sum differs slightly from @xcite , in that we don't use segment embeddings. we also report on roberta @xcite initialized version of bertsum (robertasum). ## conclusion the stepwise structured transformer paradigm, exemplified by hibert and etcsum, can be easily adapted both to extractive document summarization or content planning for table-to-text generation. stepwise etcsum, in particular, sets a new standard for both tasks. future work will focus on extending our models to generate extractive plans for better abstractive summarization of long or multiple documents @xcite . of all token embeddings") from the sentence encoder and in additionally non-stepwise models we experimented with positive label weight used to for loss calculation. finally, we used an adam optimizer with learning rate of 0.000025.