Toward Unifying Text Segmentation and Long Document Summarization (2022)

Abstract: Text segmentation is important for signaling a document’s structure. Without segmenting a long document into topically coherent sections, it is difficult for readers to comprehend the text, let alone find important information. The problem is only exacerbated by a lack of segmentation in transcripts of audio/video recordings. In this paper, we explore the role that section segmentation plays in extractive summarization of written and spoken documents. Our approach learns robust sentence representations by performing summarization and segmentation simultaneously, which is further enhanced by an optimization-based regularizer to promote selection of diverse summary sentences. We conduct experiments on multiple datasets ranging from scientific articles to spoken transcripts to evaluate the model’s performance. Our findings suggest that the model can not only achieve state-of-the-art performance on publicly available benchmarks, but demonstrate better cross-genre transferability when equipped with text segmentation. We perform a series of analyses to quantify the impact of section segmentation on summarizing written and spoken documents of substantial length and complexity.

## introduction one of the most effective ways to summarize a long document is to extract salient sentences @xcite . while abstractive strategies produce more condensed summaries, they suffer from hallucinations and factual errors, which pose a more difficult generation challenge @xcite . in this study, we focus on extractive summarization of lengthy documents, including both written documents and transcripts of spoken language. extractive summaries have the potential to be highlighted on their source materials to facilitate viewing, e.g., google's browser allows text extracts to be highlighted on the webpage via a shareable link @xcite . as a document grows in length, it becomes crucial to bring structure to it. examples include chapters, sections, paragraphs, headings and bulleted lists @xcite . all of them allow readers to find salient content buried within the document. particularly, having sections is a differentiating factor between a long and a mid-length document. a long document such as a research article contains over 5,000 words @xcite . it is an order of magnitude longer than a mid-length document such as a news article @xcite . writing a long document thus requires the author to meticulously organize the content into sections. in this paper, we equip our summarizer with the ability to predict section boundaries and leverage this ability to improve long document summarization. importantly, sections are essential to both written and spoken documents. a majority of summarization approaches concentrate on written documents, assuming the sections are given. they exploit document structure by hierarchically building representations from words to sentences, then to larger sections and documents @xcite @xcite . it remains an open question as to whether spoken transcripts can be handled in a similar manner. e.g., the transcript for a 1.5-hour video lecture @xmath0,000 words. there are no section boundaries. instead, the lecture content is loosely organized around a series of talking points. discourse cues, e.g., "so next we need to...," have been shown to correlate with the underlying document structure @xcite . we thus aim to leverage such cues to infer section boundaries, which help summarization of both spoken and written documents. our model learns robust sentence representations by performing the two tasks of extractive summarization and section segmentation simultaneously, enhanced by an optimization-based framework to select important and diverse sentences. it mimics what a human would do when identifying salient content from a lengthy document. text segmentation was previously studied as a standalone problem @xcite @xcite . for example, @xcite break wikipedia articles into sections according to tables of contents. in this work, we enhance extractive summarization with a new addition of section segmentation. we train our model on written documents with known section boundaries, then adapt it to transcripts where such information is unavailable to exploit its transferability. we observe that by predicting section boundaries, our model learns to not only encode salient content but also recognize document structure information. ensuring that a summary covers a broad range of important topics is pivotal. a long document may discuss multiple topics. it is inadequate for a summary to have a narrow information focus and miss the important points of the document. crucially, we design a new regularizer drawing on learned sentence representations and determinantal point process @xcite to ensure a set of representative and diverse sentences is selected for the summary. we evaluate our proposed approach against strong summarization baselines and on multiple datasets ranging from scientific articles to lecture transcripts, whose average document length is 3k-8k words. our findings suggest that the approach can achieve state-of-the-art performance and demonstrate better transferability when equipped with a segmentation component. our contributions are summarized as follows. ## related work there is growing interest in generating concise summaries from long documents. most summarizers are enabled by transformer-based models that can process long sequences. e.g., longformer @xcite ) replaces transformer's self-attention mechanism with dilated sliding window attention to reduce computation and memory usage. other methods include content-based and temporal sparse attention @xcite @xcite and hierarchical attention that builds representations from words to sentences and eventually to documents @xcite . our work builds on longformer to process input documents of substantial length while focusing on probing document structure for summarization. while abstractive strategies could produce succinct summaries, they are prone to hallucinations and factual errors that can mislead the reader @xcite @xcite . the problem is exacerbated when the inputs are spoken transcripts, where false starts, repetitions, interjections, ungrammatical sentences are abundant @xcite . they may cause errors to propagate through abstractive systems @xcite @xcite @xcite @xcite . instead, we pursue a more flexible strategy to produce extractive summaries, allowing the reader to grasp the essentials without having to read all materials. our work differs from previous extractive methods in its focus on document segmentation, which holds promise for summarizing lengthy documents. important sentences are often located at the beginning or end of documents @xcite . this simple heuristic gives strong results on news summarization @xcite @xcite @xcite . we take one step further, jointly partitioning a document into multiple sections and estimating sentence salience given their proximity to section boundaries. we then explore segmentation of written and spoken documents to understand the model's transferability. previous studies rely heavily on lexical cohesion to perform text segmentation @xcite @xcite . despite their success, establishing coherence in a text goes beyond repeating keywords. a longformer (16k) @xcite we call it the digital cultural heritage library for europe . figure 1 : an overview of our system named "lodoss." it builds effective sentence representations by combining two essential tasks of section segmentation and sentence extraction. we introduce a new regularizer l dpp drawing on determinantal point processes to collectively measure the quality of a set of extracted sentences, ensuring they are informative and diverse. koshorek et al. (2018) learn sentence representations and label each sentence as ending a segment or not. lukasik et al. (2020) compare three model architectures based on transformers and report results on a wikipedia dataset. in this paper, we focus on unifying section segmentation and summarization into a single optimization framework, exploiting document structure to accurately locate salient content. in what follows, we describe our approach in greater detail. ## our approach let @xmath1, s 2 , ..., s n } be a document containing n sentences. 1 our goal is to create an extractive summary of the document by selecting a subset of k sentences that retains the most important information. the task of long document summarization is significantly more challenging than other summarization tasks @xcite . it has a high compression rate, e.g., >85%, excluding most sentences from the summary and suggesting an extractive summarizer must be able to accurately identify summary-worthy sentences. fig. 1 shows a schematic overview of our system named "lodoss" (long document summarization with segmentation). it learns robust sentence representations by performing both tasks simultaneously. further, it introduces a new regularizer drawing on determinantal point processes @xcite to measure 1 modern speech-to-text services provide automatic punctuation. transcripts are punctuated using commas, periods, question marks and semicolons. it allows us to break down a transcript into a sequence of utterances akin to sentences of a written document. the quality of all summary sentences collectively, ensuring they are informative and have minimum redundancy. we employ the longformer (beltagy et al., 2020) equipped with dilated window attention to produce contextualized token embeddings for an input document. windowed attention allows each token to attend only to its local window to reduce computation and memory usage. it has the added benefit of easing section segmentation. the left and right context of a section break can be captured by the local window, which reveals any words they have in common and new words seen in this context. our longformer model utilizes a large position embeddings matrix, allowing it to process long documents up to 16k tokens. we use dilation, changing window size across layers from 32 (bottom) to 512 (top) to increase its receptive field. our summarizer is built on top of longformer by stacking two layers of inter-sentence transformers to it. we append @xcite to the beginning of each sentence and [sep] to the end, following the convention of @xcite . this modified sequence of tokens is sent to longformer for token encoding. we obtain the vector of the i-th [cls] token as the representation for sentence s i with rich contextual information. these vectors are added to sinusoidal position embeddings, then passed to two layers of inter-sentence transformers to capture document-level context. such global context is especially important for identifying salient sentences, whereas sinusoidal position embeddings indicate the relative position of sentences. the output vectors are denoted by {h i } n @xmath2. summarization and section segmentation. we address both problems simultaneously in a single framework (eq. (1-2)). particularly, y sum,@xmath3; y seg,@xmath4. both tasks are related at their core. a section usually starts or concludes with summaryworthy sentences; predicting section boundaries helps us effectively locate those sentences. moreover, the discourse cues for identifying major section boundaries, e.g., "so next we need to...," are portable across domains and genres. it allows us to perform a series of ablations to adapt our summarizer from written to spoken documents. our base model, "lodoss-base," minimizes the per-sentence empirical cross-entropy of the model w.r.t. gold-standard summary labels (eq. ( 3 )). it learns to identify salient sentences despite that content salience may vary across datasets. further, our joint model, "lodoss-joint," optimizes both tasks through multi-task learning: l(θ) = l sum + l seg . it adds to the robustness of derived sentence representations, because the acquired knowledge for section segmentation is more transferable across domains. here, ŷsum,i and ŷseg,i and are predicted scores for summarization and segmentation; y sum,i and y seg,i are ground-truth sentence labels. a dpp regularizer. it is especially important to collectively measure the quality of a set of extracted sentences, instead of handling sentences individually. we introduce a new regularizer leveraging the determinantal point processes @xcite @xcite to encourage a set of salient and diverse sentences to be selected for the summary. with the dpp regularizer, a ground-truth summary y ′ is expected to achieve the highest probability score compared to alternatives. it provides a summary-level training objective that complements the learning signals of our lodoss-joint summarizer. dpp defines a probabilistic measure for scoring a subset of sentences. let @xmath5, 2, ..., n} be the ground set containing n sentences. the probability of a subset y ⊆ y, corresponding to an extractive summary, is given by eq. ( 5 ), where det(•) is the determinant of a matrix; @xmath6; l ij indicates the similarity between sentences i and j; l y is a principal minor of l indexed by elements in y ; i is an identity matrix of the same dimension as l. we make use of the quality-diversity decomposition for constructing l: @xmath7, where @xmath8; @xmath9. in our model, the sentence quality score q i is given by ŷsum,i (eq. ( 1 )), indicating its importance to the summary. the sentence similarity score is defined by: dpp rewards a summary if it contains a subset of important and diverse sentences. a summary containing two sentences i and j has a high probability score p(@xmath10, j}) if the sentences are of high quality and dissimilar from each other. conversely, if two identical sentences are included in the summary, the determinant det(l y ) is zero. modeling pairwise repulsiveness helps increase the diversity of the selected sentences and eliminate redundancy. as illustrated in eq. ( 6 ), our dpp regularizer is defined as the negative log-probability of the groundtruth extractive summary y our final model, "lodoss-full," is shown in figure 1 . it adds the dpp regularizer to the joint model (eq. ( 7 )); β is a coefficient that balances sentencelevel cross-entropy losses and summary-level dpp regularization. θ are all of our model parameters. in this section, we detail our experimental settings for long document extractive summarization. our datasets include collections of scientific articles and lecture transcripts, their associated summaries and section boundaries. we contrast our approach with strong summarization baseline systems and report results on three standard benchmarks. model ablations and human assessment of summary quality are presented in §5. ## ablations and analyses effect of summary length. we vary the length of output summaries to contain 5-7 sentences and report summarization results on pubmed and arxiv ( ## conclusion we tackle the problem of long document extractive summarization by combining two essential tasks of section segmentation and sentence extraction. we further design a regularizer drawing on determinantal point processes to ensure a set of representative and diverse sentences are selected for the summary. extensive experiments and ablations demonstrate the effectiveness of our proposed approach. our future work includes exploration of various text segmentation techniques to improve our understanding of the latent document structure. another direction would be to extend our study to the realm of neural abstractive summarization with the help of learned document structure.