Long Document Summarization with Top-down and Bottom-up Inference (2023)

Abstract: Text summarization aims to condense long documents and retain key information. Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents. Most recent models infer the latent representations with a transformer encoder, which is purely bottom-up and thus does not capture long-distance context well. Also, self-attention-based models face the challenge of quadratic complexity with respect to sequence length. We propose a method to improve summarization models on these two aspects. Our method assumes a hierarchical latent structure of a document where the top-level captures the long range dependency at a coarser time scale and the bottom token level preserves the details. Critically, our method enables token representations to be updated in both a bottom-up and top-down manner. In the bottom-up pass, token representations are inferred with local self-attention to leverage its efficiency. Top-down correction is then applied to allow tokens to capture global context. We demonstrate the effectiveness on a diverse set of summarization datasets, including narrative, conversational, scientific documents and news. Our model achieves state-of-the-art performance on a wide range of long document summarization benchmarks, compared to recent efficient transformers. We show that our model can summarize an entire book and achieve competitive performance using 0.27% parameters and much less training data, compared to a recent GPT-3-based model. These results indicate the general applicability and benefits of the framework.

## introduction an abstractive summarization system aims to generate a semantically coherent and linguistically fluent summary by conditioning on the document. the dominant approach for abstractive summarization is to use a seq2seq model @xcite with an encoder-decoder architecture instantiated with either rnns @xcite or transformers @xcite . in such a model, an encoder computes or infers foot_0 latent representations of observed tokens (words or subwords) in a document, conditioning on which a decoder generates a summary. this paper studies the problem of how to compute informative latent representations, which in turn would improve summarization. we propose a method which synergizes bottomup computation with top-down computation while assuming a multi-scale latent structure of a document. in a multi-scale structure, higher-level variables (like those representing sentences, segments) model the document at a coarser time-scale and abstract away details, and are suitable for capturing long range dependency of the document; in contrast, lower-level variables (like those representing tokens) preserve details, and prevent the summary from losing key details (such as the name of an entity). in our method, the summary is generated by conditioning on token representations (low-level variables), similar to recent abstractive summarization models @xcite . there is however a critical difference. in our method, token representations are first bottomup inferred and then top-down updated with high level representations, hence rendering low-level representations aware of global context. see figure 1 for an overview of our method. multi-level models have been widely studied in modeling for images @xcite , speech @xcite , and language @xcite . it is also not new in the summarization literature. prior summarization research has explored hierarchical models @xcite @xcite @xcite . these works focus on the bottom-up computation figure 1 : an overview of the top-down transformer. suppose a document with 7 tokens is the inputs to the model, as shown on the bottom left. the bottom-up inference is achieved with local self-attention (n1 layers) as shown in the left panel. to initialize the top-level representations, we pool bottom-up-inferred token representations with either equal weights or adaptive weights (see section 2.3 for details). top-level representations are then updated with full self-attention (n2 layers) to capture global context. they are then used to update bottom-up-inferred token representations, accounting for the top-down update for token representations, as shown in the middle panel. the final token representations are attended by the decoder to generate a summary. note that inference is used in the sense of statistical inference for latent variables and does not imply no training. in a hierarchical model, computing higher-level representations (e.g., sentences, paragraphs) based on lower-level representations (e.g., words). in contrast, our method emphasizes the combination of bottom-up, as done in prior works, and top-down where lower-level representations are updated and enriched with higher-level representations (see the middle panel in figure 1 ). this design is critical for summarization which requires global context. as shown in our ablations, removing the top-down update undermines the summarization performance. the proposed method is agnostic to the model architecture. due to the dominance of transformer models in nlp @xcite @xcite , we instantiate our method with a transformer-based model. there is a bottleneck of applying transformers to long documents, because its computational and memory cost has a quadratic dependency on the sequence length. this issue is especially critical for summarization since we are more interested in summarizing long documents since short ones can be quickly read through by humans. to address this issue, a large amount of prior works have been devoted to develop efficient transformers with sub-quadratic complexity @xcite @xcite @xcite . our method provides a natural way to diminish this quadratic complexity issue. in the bottomup computation, we use local self-attention where each token only attends the tokens within a local fixed-length window, and thus the complexity does not grow as a function of the input sequence length. the top-down correction for (local) token representations enables them to capture more global context, reducing the limitation of local attention. in prior works like longformer @xcite , bigbird @xcite , local attention is also used. our method is different from these models in terms of how to inject global information to locally computed representations. longformer and big-bird utilize a few global tokens which attend and are attended by all local tokens, whereas we use topdown correction. our approach can better capture global information compared to prior models, as demonstrated by clear performance improvements over these models in our experiments. ## methods figure 1 gives a graphical overview of the topdown transformer. we introduce its details in this section. suppose a document has n tokens, @xmath0. in our method, token representations are computed by combining top-down and bottomup processes. this leads to effective and efficient inference for token representations. they are then attended by a decoder to generate a summary, as in a regular encoder-decoder transformer. ## related work summarization models prior works have proposed extractive models @xcite , abstractive models @xcite , and hybrid models combining extractive and abstractive methods @xcite , for text summarization. although our model mostly follows the abstractive approach, it also has connections to the hybrid models. these models usually first extract salient sentences from the source document and then summarize the extracted sentences with an abstractive model. extracted sentences can be viewed a high level representation of the document, although it is the observed space but not in the latent space as in our framework. a continuous representations in the latent space facilities endto-end learning. moreover, assigning importance weight with the importance tagger in our method resembles an extractive step in a hybrid model, and thus top down transformer with learned importance tagger can be considered a hybrid model. efficient transformers despite the effectiveness of transformers on a variety of tasks, its quadratic complexity with respect to the sequence length has limited its application to problems with long sequences. a large amount of works have attempted to address this limitation. a major line of work focuses on designing various sparse attention mechanisms. these works can be roughly categorized into two groups, depending on whether the sparsity pattern is content-dependent @xcite @xcite or content-independent @xcite @xcite . our work is mostly related to content-independent sparse attention. a main assumption of content-independent sparse attention is that the context temporally and/or spatially proximate to the query token is more important, which is intuitively sensible and supported by empirical attention analysis @xcite . thus, a common sparse attention pattern is local attention, where each query token only attends to a neighborhood within a fixed temporal and/or spatial window. while this reduces the complexity to be linear, a model with only local attention cannot model long-range dependency. prior works combine local attention with other attention patterns with wider or global receptive field such as dilated attention, random attention tokens, and global attention tokens @xcite . our models also use local attention for its efficiency and leverage top-down inference to enable global-context awareness. ## conclusion in this work, we propose a summarization method which combines bottom-up computation with topdown computation to improve token representation inference. in the bottom-up pass, token representations are inferred with local self-attention to exploit its efficiency. top-down correction is then applied to allow tokens to capture global context. our model achieves (1) state-of-the-art performance on a wide range of long document summarization benchmarks, and (2) competitive performance on summarizing whole books using 0.27% parameters and much less training data, compared to a recent gpt-3-based model. these results indicate the general applicability and benefits of the proposed