MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervision (2020)

Abstract: The lack of large and diverse discourse treebanks hinders the application of data-driven approaches, such as deep-learning, to RST-style discourse parsing. In this work, we present a novel scalable methodology to automatically generate discourse treebanks using distant supervision from sentiment annotated datasets, creating and publishing MEGA-DT, a new large-scale discourse-annotated corpus. Our approach generates discourse trees incorporating structure and nuclearity for documents of arbitrary length by relying on an efficient heuristic beam-search strategy, extended with a stochastic component. Experiments on multiple datasets indicate that a discourse parser trained on our MEGA-DT treebank delivers promising inter-domain performance gains when compared to parsers trained on human-annotated discourse corpora.

## introduction discourse parsing is an important natural language processing (nlp) task, aiming to uncover the hidden structure underlying coherent documents, as described by theories of discourse like rhetorical structure theory (rst) @xcite or pdtb @xcite . not only has discourse parsing been shown to enhance key downstream tasks, such as text classification @xcite , summarization @xcite and sentiment analysis @xcite @xcite , but it also appears to complement contextual embeddings, like bert @xcite , in tasks where discourse information is critical, such as argumentation analysis @xcite . traditionally, rst-style discourse parsing builds a complete, hierarchical constituency tree for a document @xcite , where leaf nodes are clause-like sentence fragments, called elementarydiscourse-units (edus), while internal tree nodes are labelled with discourse relations (e.g., evidence, contrast). in addition, each node is given a nuclearity attribute, which encodes the importance of the node in its local context. a key limitation for further research in rst-style discourse parsing is the scarcity of training data. only a few human annotated discourse treebanks exist, each only containing a few hundred documents. although our recent efforts using distant supervision from sentiment to generate large-scale discourse treebanks have already partly addressed this dire situation @xcite , the previously proposed solution is still limited in: (i) scope, by only building the rst constituency structure without nuclearity and relation labels; and (ii) applicability, by relying on a non-scalable cky solution, which cannot be applied to many real-world datasets with especially long documents. in this work, we propose a significant extension to this line of research by introducing a scalable solution for documents of arbitrary length and further moving beyond just predicting the tree-structure by incorporating the nuclearity attribute, oftentimes critical in informing downstream tasks @xcite @xcite . inspired by the recent success of heuristic search in nlp tasks involving trees (e.g., @xcite ; @xcite al. (2019) ), we develop a beam-search strategy implementing an exploration-exploitation trade-off, as commonly used in reinforcementlearning (rl) @xcite . remarkably, by following this heuristic approach, any large corpus annotated with sentiment can be turned into a discourse treebank on which a domain/genre specific discourse parser can be trained. as a case study for this process, we annotate, evaluate and publicly release a new discourseaugmented yelp '13 corpus @xcite called mega-dt foot_0 (comprising ≈250,000 documents) with nuclearity attributed "silver-standard" discourse trees, solely leveraging the corpus' document-level sentiment annotation. to evaluate the quality of our newly proposed mega-dt corpus, we conduct a series of experiments. we train the top-performing discourse parser by @xcite on mega-dt and compare its performance with the same parser trained on previously proposed treebanks. specifically, we compare our discourse-annotated dataset against a smaller "silver-standard" treebank @xcite containing around ≈100,000 documents @xmath0. results indicate that while training a parser on mega-dt does not yet match the performance of training and testing on the same treebank (intradomain), it does push the boundaries of what is possible with distant supervision. in most cases, training on mega-dt delivers statistically significant improvements on the arguably more difficult and useful task of inter-domain discourse prediction, where a parser is trained on one domain and tested/applied to another one. overall, this suggests that our new approach to distant supervision from sentiment can generate large-scale, high-quality treebanks, with mega-dt being the best publicly available resource for training a discourse parser in domains where no gold-standard discourse annotation is available. ## related work the most closely related line of work is rst-style discourse parsing, with the goal to obtain a complete discourse tree, including structure, nuclearity and relations. based on the observation that these three aspects are correlated, most previous work has explored models to learn them jointly (e.g., joty et al. (2015); ji and eisenstein (2014); yu et al. (2018) ). however, while this strategy seems intuitive, the state-of-the-art (sota) system on structure-prediction by @xcite applies a rather different strategy, first jointly predicting structure and nuclearity and then subsequently pre-dicting relations. the main motivation behind this separation is the large number of possible output classes when predicting these three aspects together. the success of the system by @xcite on the widely used rst-dt corpus inspires us to also learn structure and nuclearity jointly, rather than combining all three aspects. the second line of related work infers finegrained information from coarse-grained supervision signals using machine learning. due to the lack of annotated data in many domains and for many real-world tasks, methods to automatically generate reliable, fine-grained data-labels have been explored for many years. one promising approach in this area is multiple instance learning (mil) @xcite . the general task of mil is to retrieve fine-grained information (called instance-labels) from high-level supervision (called bag-labels), using correlations of discriminative features within and between bags to predict labels for instances. with the recent rise of deep-learning, neural mil approaches have also been proposed @xcite . we previously combined the two lines of related work described above to create discourse structures from large-scale datasets, solely using document-level supervision @xcite . when applied to the auxiliary task of sentiment analysis, we generated a "silver-standard" discourse structure treebank using the neural mil model by @xcite in combination with a sentiment-guided cky-style treeconstruction algorithm, generating near-optimal discourse trees in bottom-up fashion @xcite . although our approach has shown clear benefits, it is inapplicable to many real-world datasets, as it does not scale to long documents and cannot predict nuclearity-and relation-labels. addressing these limitations is a major motivation of this paper. further efforts to automatically generate discourse trees from auxiliary tasks have been mostly focused on latent tree induction, generating trees from text classification @xcite or summarization tasks @xcite . for both approaches, domain dependent discourse trees are induced during the neural training process. while either method has shown to improve the performance on the downstream task itself, subsequent research by @xcite indicates that the induced trees are often trivial and shallow, and do not represent valid discourse structures. the third stream of related work is on leveraging heuristic search algorithms in nlp tasks involving trees. for syntactic parsing, @xcite and @xcite show that a static, small beam size (e.g. 10) already achieves good performance, with @xcite delivering promising results by using greedy decoding. as a recent example for discourse parsing, @xcite successfully combine standard beam-search with shift-reduce parsing using two parallel beams for shift and reduce actions. overall, recent work shows that beam-search approaches and their possible extensions can effectively address scalability issues in multiple parsing scenarios. in this paper, we extend the standard beam-search approach with a stochastic exploration-exploitation trade-off, as used in reinforcement learning, where signals also tend to be sparse and noisy. ## predicting discourse from sentiment previous work has shown that incorporating rststyle discourse trees can help to predict documentlevel sentiment @xcite @xcite . these findings give rise to the assumption that the sentiment of a document can also provide important information on its discourse structure. in the following sub-section, we shortly revisit our previous approach to exploit this assumption by solely relying on document-level sentiment annotations @xcite . afterwards, we will present our new approach to overcome scalability issues and jointly predict structure and nuclearity. ## evaluation in this section, we evaluate our proposed method to generate the mega-dt discourse treebank by assessing the performance of a discourse parser when trained on mega-dt against our previously proposed "silver-standard" treebank @xcite as well as two commonly used, human-annotated, discourse corpora. ## conclusions and future work in this work, we present a novel distant supervision approach to predict the discourse-structure andnuclearity for documents of arbitrary length solely using document-level sentiment information. to deal with the increasing spatial complexity, we apply and compare heuristic beam-search strategies, including a stochastic variant inspired by rl techniques. our results on the challenging inter-domain discourse-structure and -nuclearity prediction task strongly suggests that the heuristic approach taken (1) enhances the structure prediction task through more diversity in the early-stage tree selection, (2) allows us to effectively predict nuclearity and (3) helps to significantly reduce the complexity of the unrestricted cky approach to scale for arbitrary length documents. in conclusion, our new approach allows the nlp community to augment any existing sentimentannotated dataset with discourse trees, enabling the automated generation of large-scale domain/genrespecific discourse treebanks. as a case study for the effectiveness of the approach, we annotate and publish our mega-dt corpus as a high quality rststyle discourse treebank, which has been shown to outperform previously proposed discourse treebanks (namely yelp13-dt, rst-dt and instr-dt) on most tasks of inter-domain discourse parsing. this suggests that parsers trained on our mega-dt corpus (or further domain-specific treebanks generated according to our approach) should be used to derive discourse trees in target domains where no gold-labeled data is available. this work can be extended in several ways: (i) we plan to investigate into further functions for τ to enhance the exploration-exploitation tradeoff. (ii) additional strategies to assign nuclearity should be explored, considering the excessive n-nclassification shown in our evaluation. (iii) we plan to apply our approach to more sentiment datasets (e.g., diao et al. (2014)