Predicting Discourse Trees from Transformer-based Neural Summarizers (2021)

Abstract: Previous work indicates that discourse information benefits summarization. In this paper, we explore whether this synergy between discourse and summarization is bidirectional, by inferring document-level discourse trees from pre-trained neural summarizers. In particular, we generate unlabeled RST-style discourse trees from the self-attention matrices of the transformer model. Experiments across models and datasets reveal that the summarizer learns both, dependency- and constituency-style discourse information, which is typically encoded in a single head, covering long- and short-distance discourse dependencies. Overall, the experimental results suggest that the learned discourse information is general and transferable inter-domain.

## introduction extractive summarization is a common and important task within the area of natural language processing (nlp) , which can be useful in a multitude of diverse real-life scenarios. current extractive summarizers typically use exclusively neural approaches, in which the importance of extracted units (i.e., sentences or clauses) and relationship between them are learned by the model from a large amount of data (e.g., @xcite ). inspired by previous work in pre-neural times, indicating that discourse information, especially discourse trees according to the rhetorical structure theory (rst) @xcite , can benefit the summarization task @xcite , several very recent neural summarizers have tried to explicitly encode discourse information to support summarization. overall, it seems that adding these encodings, consistent with pre-neural results, is beneficial. in particular, injecting discourse has been shown to either improve performance on the extractive summarization task itself @xcite , or allow for a substantial reduction in the number of the summarizer's parameters, while keeping competitive performance @xcite . the central hypothesis we are exploring in this paper is whether the synergy between discourse parsing and summarization is bidirectional. in other words, we examine if summarization is a useful auxiliary task to infer discourse structures. @xcite performed a preliminary investigation of this conjecture, showing that structural information can be inferred from attention mechanisms while training a neural model on auxiliary tasks. however, they did not perform any comparison against ground-truth discourse trees. further, recent work showed that discourse trees implicitly induced during training are oftentimes trivial and shallow, not representing valid discourse structures @xcite . in this paper, we address these limitations by explicitly exploring the relationship between summarization and discourse parsing through the inference of document-level discourse trees from pretrained summarization models, comparing the results against ground-truth rst discourse trees. besides @xcite , our idea and approach are inspired by recent works on extracting syntactic trees from pre-trained language models @xcite or machine translation approaches @xcite , as well as previous work on knowledge graph construction from pre-trained language models @xcite . specifically, we generate full rst-style discourse trees from self-attention matrices of a pre-trained transformerbased summarization model. we use three different tree-aggregation approaches (cky @xcite , eisner @xcite and cle @xcite ), generating a set of constituency and dependency trees representing diverse discourse-related attributes. our proposal is thereby addressing one of the key limitations in discourse parsing, namely the lack of large training corpora. we aim to overcome this limitation by generating a large number of reasonable quality discourse trees from a pre-trained summarization model, similar in spirit to what huber and carenini (2020) did with sentiment. admittedly, the discourse information captured with our approach is summarization task-specific, however, our generated discourse treebank can be combined with further task-dependent treebanks (e.g. from sentiment) to train more powerful discourse parsers in a multitask framework. generally speaking, the ability to infer discourse trees as a "by-product" of the summarization task can also be seen as a form of unsupervised discourse parsing, where instead of leveraging pretrained language models like in @xcite , we exploit a pre-trained neural summarizer. we empirically evaluate our method on three datasets with human rst-style annotations, covering different text genres. multiple experiments show that the summarization model learns discourse information implicitly, and that more dependency information are captured, compared to structural (i.e., constituency) signals. interestingly, an additional exploration of the attention matrices of individual heads suggests that, for all models, most of the discourse information is concentrated in a single head, and the best performing head is consistent across all datasets. we further find that the dependency information learned in the attention matrix covers long distance discourse dependencies. overall, the results are consistent across datasets and models, indicating that the discourse information learned by the summarizer is general and transferable inter-domain. nucleus-satellite, satellite-nucleus}. "nucleus" child-nodes thereby generally play a more important role when compared to a "satellite" child-node. although standard rst discourse trees are encoded as constituency trees, they can be converted into dependency trees with near isomorphic transformations. in this work, we infer both, constituency and dependency trees. over the past decades, rst discourse parsing has been mainly focusing on supervised models, typically trained and tested within the same domain using human annotated discourse treebanks, such as rst-dt @xcite , instruction-dt @xcite or gum @xcite . the intra-domain performance of these supervised models has consistently improved, with a mix of traditional models by @xcite and @xcite , and neural models @xcite reaching state-of-the-art (sota) results. yet, these approaches do not generalize well inter-domain @xcite , likely due to the limited amount of available training data. @xcite recently tackled this data-sparsity issue through automatically generated discourse structures from distant supervision, showing that sentiment information can be used to infer discourse trees. improving on their initial results, huber and carenini (2020) published a large-scale, distantly supervised discourse corpus (mega-dt), showing that a parser trained on such treebank delivers sota performance on the more general interdomain discourse parsing task. in this paper, we also tackle the data sparsity problem in discourse parsing, however, using a significantly different approach. first, instead of relying on sentiment, we leverage the task of extractive summarization. second, instead of a method for distant supervision, we propose an unsupervised approach. the area of unsupervised rst-style discourse parsing has been mostly underlooked in the past, with recent neural approaches either taking advantage of pre-trained language models to predict discourse @xcite or using pre-trained syntactic parsers and linguistic knowledge @xcite to infer discourse trees in an unsupervsied manner. similarly. our proposal only relies on a pre-trained neural summarization model to generate discourse trees. recent neural summarization models are typically based on transformers @xcite . one advantage of these mod-els is that they learn the relationship between input units explicitly using the dot-product self-attention, which allows for some degree of exploration of the inner working of these complex and distributed models. here, we investigate if the attention matrices of a transformer-based summarizer effectively capture discourse information (i.e., how strongly edus are related) and therefore can be used to derive discourse trees for arbitrary documents. @xcite pioneered the idea to directly apply rst-style discourse parsing to extractive summarization, and empirically showed that rst discourse information can benefit the summarization task, by simply extracting edus along the nucleus path. this initial success was followed by further work on leveraging discourse parsing in summarization, including @xcite , @xcite , and @xcite . more recently, the benefits of discourse for summarization have also been confirmed for neural summarizers, e.g. in @xcite and @xcite , using the structure of scientific papers (i.e. sections), and in @xcite , successfully incorporating rst-style discourse and co-reference information in the bertsum summarizer @xcite . in contrast to previous approaches demonstrating how discourse can enhance summarization performance, we have recently shown that discourse enables the specification of simpler neural summarizers, without affecting their performance @xcite . in particular, by using a fixed discoursebased attention they achieve competitive results compared to learnable dot-product self-attention mechanisms, as used in the original transformer model. inspired by these findings, suggesting that transformer-based summarization models learn effective discourse representations, we explore if useful discourse structures can be inferred from learnt transformer self-attention weights. admittedly, @xcite and @xcite presented preliminary work on inferring discourse structures from attention mechanisms, while training a neural model on auxiliary tasks, like text classification and summarization. however, they did not perform any comparison against ground-truth discourse trees as we do here. more importantly, we employ a more explicit approach to infer discourse structures, not as part of the learning process, but extracting the structures after the summarization model is completely trained and applied to new documents. while our focus is on discourse, extracting syntactic constituency and dependency trees from transformer-based models has been recently attempted in both, machine translation and language modelling. in machine translation, @xcite and @xcite show that trained translation models can capture syntactic information within their attention heads, using the cky and cle algorithms, respectively. in pre-trained language models, @xcite propose a parameter-free probing method to construct syntactic dependency trees based on a pretrained bert model, only briefly elaborating on possible applications to discourse. in contrast to our work, they do not directly use attention heads, but instead build an impact matrix based on the distance between token representations. furthermore, while their bert-based model cannot deal with long sequences, our two-level encoder can effectively deal with sequences of any length, which is critical in discourse. ## conclusions and future work we present a novel framework to infer discourse trees from the attention matrices learned in a transformer-based summarization model. experiment across models and datsets indicates that both dependency and structural discourse information are learned, that such information is typically concentrated in a single head, and that the attention matrix also covers long distance discourse dependencies. overall, consistent results across datasets and models suggest that the learned discourse information is general and transferable inter-domain. in the future, we want to explore if simpler summarizers like bertsum @xcite can also capture discourse info; specifically studying if the importance of the heads corresponds to the captured discourse info, which may help pruning summarization model by incorporating discourse info, in spirit of @xcite . with respect to dependency tree generation possible improvements could come by looking for additional strategies balancing between guidance and flexibility, as @xcite explore for syntactic dependency parsing. to address the problem of data sparsity in discourse parsing, we want to synergistically leverage other discourse-related tasks, in addition to sentiment and summarization, like topic modeling. 7 more details can be found in appendix c.