Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization (2023)

Abstract: For text summarization, the role of discourse structure is pivotal in discerning the core content of a text. Regrettably, prior studies on incorporating Rhetorical Structure Theory (RST) into transformer-based summarization models only consider the nuclearity annotation, thereby overlooking the variety of discourse relation types. This paper introduces the ‘RSTformer’, a novel summarization model that comprehensively incorporates both the types and uncertainty of rhetorical relations. Our RST-attention mechanism, rooted in document-level rhetorical structure, is an extension of the recently devised Longformer framework. Through rigorous evaluation, the model proposed herein exhibits significant superiority over state-of-the-art models, as evidenced by its notable performance on several automatic metrics and human evaluation.

## introduction for writing a good summary of a long document, it is of paramount importance to discern the salient information within the text and to comprehend the intricate interconnections among its various components. contemporary leading-edge systems for abstractive (long) text summarization employ transformer @xcite encoderdecoder architecture @xcite . these sequence-to-sequence (seq2seq) models first transform the source document into a highdimensional content representation and then decode the predicted summary conditioned on the representation @xcite @xcite . it has been demonstrated in the past that such an architecture does a poor job of digging high-level discourse structure during the encoding phase @xcite ; koto et @xcite . however, discourse structure is very important for deciding what to include vs. not to include in the summary @xcite @xcite . given that previous work has indicated that the performance of neural language models can be enhanced through the incorporation of latent structure information @xcite @xcite , we will here explore the integration of discourse relation structure into the longformer model @xcite ; this architecture has been shown to be particularly suitable for encoding long input texts. rhetorical structure theory (rst) serves as a discourse framework designed to articulate the interrelationships among sentences at the document level. this framework distinguishes a plethora of coherence relations delineating the manner in which two text segments are interconnected (e.g., one segment might give a reason for a claim made in another segment, or alternatively, two segments may contrast with one another). moreover, rst distinguishes between paratactic relations, where both segments carry equivalent discourse importance, and hypotactic relations, which classify the segment of greater centrality to the overarching discourse structure as the 'nucleus' and the less central one as the 'satellite'. figure 1 shows a simple example of an rst tree. in this instance, edu1 serves as the nucleus of the elaboration relation, whereas the combination of edus 2 and 3 constitutes the satellite of said relation. furthermore, we can see that edu3 assumes a more central role within the concession relation, hence it is marked as its nucleus, while edu2 holds less important: if edu2 was left out, the elaboration relation between edus 1 and 3 would still hold, but if edu3 was removed, an elaboration relation between edu1 and edu2 would not hold, and the coherence would be lost. as has been recognized early on @xcite , this discourse information can be effectively used in summarization tasks. while there have been some previous attempts at integrating discourse structure into neural text summarization models, as seen in gabriel et al. (2021 ); dong et al. (2021); xiao et al. (2020); xu et al. (2020); cohan et al. ( 2018 ), these approaches do not utilize relation labels and solely consider the 1-best rst tree obtained from preprocessing of a discourse parser. we argue that this leads to two significant issues: firstly, information pertaining to relation type is overlooked, despite its known relevance to the summarization task. secondly, there may be benefits in considering distributions over coherence relation labels, rather than limiting analysis to the 1-best results @xcite . one reason is that external discourse parsers are known to perform poorly on out-of-domain data @xcite @xcite @xcite , and may hence propagate errors into the summarization model. there is a subsequent risk that these errors will be incrementally amplified during back-propagation, thus potentially impairing the model's performance. a second reason is that there might inherently be several coherence relations holding at the same time @xcite , which might be beneficial to represent through the distributions of the discourse structure. hence, we posit that the output of the rst parser holds greater significance when it not only provides the model with the n-best results but also conveys the remaining uncertainty associated with them. in the remainder of the paper, we explore whether incorporating the labeled discourse relation structure with uncertainty, which can be understood as the distributions of discourse structure, into the attention mechanism can effectively augment the performance of neural summarization models. our main contributions are as follows: ## proposed approach in the realm of document discourse parsing, the performance of the rst parser leaves much to be desired @xcite @xcite , with parsing performance deteriorating in conjunction with escalating document complexity. merely passing the 1-best rst tree risks imparting misleading information to the summarization model. inspired by @xcite , the approach to alleviating the aforementioned problems is that we retain uncertainty inside the parser, which can convey the parser's confidence in each discourse relation. furthermore, we contend that discourse relation labels (types) can provide more fine-grained labeled probability distributions that can assist attention heads of the transformer-based model to capture the importance of different discourse units. this in turn would contribute to a more precise estimation of the context vector and can enhance the quality of source document encoding. discourse parsers tend to be more precise (and have more peaked probability distributions) for local coherence relations, which span only a short amount of text, compared to global relations spanning large portions of a text. this aligns well with the dilated (yet still limited) sliding window attention mechanism of the longformer @xcite . we, therefore, integrate the probability distributions over local coherence relations into the attention window w of the longformer. ## conclusion this paper introduces a novel supervised discourse enhanced longformer model. this strategy mainly improves the local attention mechanism in the longformer model by leveraging the rhetorical structure as uncertainty distributions. the experimental findings provide strong evidence that the proposed approach is straightforward, and can effectively employ the discourse structure of source documents to improve the summary performance of longformer. furthermore, this strategy also has a high potential capability for application in other seq2seq natural language tasks.