Sliding Selector Network with Dynamic Memory for Extractive Summarization of Long Documents (2021)

Abstract: Neural-based summarization models suffer from the length limitation of text encoder. Long documents have to been truncated before they are sent to the model, which results in huge loss of summary-relevant contents. To address this issue, we propose the sliding selector network with dynamic memory for extractive summarization of long-form documents, which employs a sliding window to extract summary sentences segment by segment. Moreover, we adopt memory mechanism to preserve and update the history information dynamically, allowing the semantic flow across different windows. Experimental results on two large-scale datasets that consist of scientific papers demonstrate that our model substantially outperforms previous state-of-the-art models. Besides, we perform qualitative and quantitative investigations on how our model works and where the performance gain comes from.

## introduction text summarization is an important task of natural language processing which aims to distil salient contents from a textual document. existing summarization models can be roughly classified into two categories, which are abstractive and extractive. abstractive summarization usually adopts natural language generation technology to produce a wordby-word summary. in general, these approaches are flexible but may yield disfluent summaries @xcite . by comparison, extractive approaches aim to select a subset of the sentences in the source document, thereby enjoying better fluency and efficiency @xcite . although many summarization approaches have demonstrated their success on relatively short documents, such as news articles, they usually fail paragraph 1: medical tourism is illustrated as occurrence in which individuals travel abroad to receive healthcare services. it is a multibillion dollar industry and countries like india, thailand, israel, singapore, … paragraph 2: the prime driving factors in medical tourism are increased medical costs, increased insurance premiums, increasing number of uninsured or partially insured individuals in developed countries, … …… paragraph 5: it is generally presumed in marketing that products with similar characteristics will be equally preferred by the consumers, however, attributes, which make the product similar to other products, will not…. to achieve desired performance when directly applied in long-form documents, such as scientific papers. this inferior performance is partly due to the truncation operation, which inevitably leads to information loss, especially for extractive models because parts of gold sentences would be inaccessible. in addition, the accurate modeling of long texts remains a challenge @xcite . a practical solution for this problem is to use a sliding window to process documents separately. this approach is used in other nlp tasks, such as machine reading comprehension @xcite . however, such a paradigm is not suitable for summarization task because the concatenation of summaries that are independently extracted from local contexts is usually inconsistent with the gold summary of the entire document. figure 1 shows an example to illustrate this problem. the core topic of the source document is "medical tourism," which is discussed in paragraphs 1 and 2. how-ever, the 5-th paragraph is mainly about "consumer and product." as a consequence, the paragraphby-paragraph extraction approach might produce a both repetitive and noisy summary. under this circumstance, the supervised signals will have a negative effect on model behaviors because understanding why paragraph 5 should output an empty result without information conveying from previous texts is confused for the model. in this paper, we propose a novel extractive summarization model for long-form documents. we split the input document into multiple windows and encode them with a sliding encoder sequentially. during this process, we introduce a memory to preserve salient information learned from previous windows, which is used to complete and enrich local texts. intuitively, our model has the following advantages: 1) in each window, the text encoder processes a relatively short segment, thereby yielding more accurate representations. 2) the local text representations can capture beyond-window contextual information via the memory module. 3) the previous selection results are also parameterized in the memory block, allowing the collaboration among summary sentences. to sum up, our contributions are threefold. ## model this section describes our model, namely, the sliding selector network with dynamic memory (ssn-dm), of which figure 2 gives an overall architecture. formally, given a document d of arbitrary length, we first split d into multiple segments according to the pre-defined window length. then, we use a bert encoder to sequentially encode each segment and select salient sentences. during this process, a memory module is applied to achieve the information flow across different windows. finally, the extracted sentences are aggregated to generate the final summary. we elucidate each module in the following subsections. ## conclusion and future work in this study, we propose a novel extractive summarization that can summarize long-form documents without content loss. we conduct extensive experiments on two well-studied datasets that consist of scientific papers. experimental results demonstrate that our model outperforms previous stateof-the-art models. in the future, we will extend our framework (i.e., a sliding encoder combined with long-range memory modeling) to abstractive summarization models.