Unleashing the Power of Neural Discourse Parsers - A Context and Structure Aware Approach Using Large Scale Pretraining (2020)

Abstract: RST-based discourse parsing is an important NLP task with numerous downstream applications, such as summarization, machine translation and opinion mining. In this paper, we demonstrate a simple, yet highly accurate discourse parser, incorporating recent contextual language models. Our parser establishes the new state-of-the-art (SOTA) performance for predicting structure and nuclearity on two key RST datasets, RST-DT and Instr-DT. We further demonstrate that pretraining our parser on the recently available large-scale “silver-standard” discourse treebank MEGA-DT provides even larger performance benefits, suggesting a novel and promising research direction in the field of discourse analysis.

## introduction discourse parsing is an important upstream task within the area of natural language processing (nlp) which has been an active field of research over the last decades. in this work, we focus on discourse representations for the english language, where most research has been surrounding one of the two main theories behind discourse, the rhetorical structure theory (rst) proposed by @xcite or interpreting discourse according to pdtb @xcite . while both theories have their strengths, the application of the rst theory, encoding documents into complete constituency discourse trees @xcite , has been shown to have many crucial implications on real world problems. a tree is defined on a set of edus (elementary discourse units), approximately aligning with clause-like sentence fragments, acting as the leaves of the tree. adjacent edus or sub-trees are hierarchically aggregated to form larger (possibly non-binary) constituents, with internal nodes containing (1) a nuclearity label, defining the importance of the subtree (rooted at the internal node) in the local context and (2) a relation label, defining the type of semantic connection between the two subtrees (e.g., elaboration, background). in this work, we focus on structure and nuclearity prediction, not taking relations into account. previous research has shown that the use of rst-style discourse parsing as a system component can enhance important tasks, such as sentiment analysis, summarization and text categorization @xcite @xcite @xcite . more recently, it has also been suggested that discourse structures obtained in an rst-style manner can further be complementary to learned contextual embeddings, like the popular bert approach @xcite . combining both approaches has shown to support tasks where linguistic information on complete documents is critical, such as argumentation analysis @xcite . even though discourse parsers appear to enhance the performance on a variety of tasks, the full potential of using more linguistically inspired approaches for downstream applications has not been unleashed yet. the main open challenges of integrating discourse into more nlp downstream tasks and to deliver even greater benefits have been a combination of (1) discourse parsing being a difficult task itself, with an inherently high degree of ambiguity and uncertainty and (2) the lack of large-scale annotated datasets, rendering the initial problem more severe, as data-driven approaches cannot be applied to their full potential. the combination of these two limitations has been one of the main reasons for the limited application of neural discourse parsing for more diverse downstream tasks. while there have been neural discourse parsers proposed @xcite @xcite , they still cannot consistently outperform traditional approaches when applied to the rst-dt dataset, where the amount of training data is arguably insufficient for such data-intensive approaches. in this work, we alleviate the restrictions to the effective and efficient use of discourse as mentioned above by introducing a novel approach combining a newly proposed large-scale discourse treebank with our data-driven neural discourse parsing strategy. more specifically, we employ the novel mega-dt "silver-standard" discourse treebank published by @xcite containing over 250,000 discourse annotated documents from the yelp'13 sentiment dataset @xcite , nearly three orders of magnitude larger than commonly used rst-style annotated discourse treebanks (rst-dt @xcite , instructional-dt @xcite ). given this new dataset with a previously unseen number of full rst-style discourse trees, we revisit the task of neural discourse parsing, which has been previously attempted by @xcite and others with rather limited success. we believe that one reason why previous neural models could not yet consistently outperform more traditional approaches, heavily relying on feature engineering @xcite , is the lack of generalisation when using deep learning approaches on the small rst-dt dataset, containing only 385 discourse annotated documents. this makes us believe that using a more advanced neural discourse parser in combination with a large training dataset can lead to significant performance gains. admittedly, even though mega-dt contains a huge number of datapoints to train on, it has been automatically annotated, potentially introducing noise and biases, which can negatively influence the performance of our newly proposed neural discourse parser when solely trained on this dataset. a natural and intuitive approach to make use of the neural discourse parser and both datasets ("silver-standard" and gold-standard) is to combine them during training, pretraining on the large-scale "silver-standard" corpus and subsequently fine-tuning on rst-dt or further human annotated datasets. this way, general discourse structures could be learned from the large-scale treebank and then enhanced with human-annotated trees. with the results shown in this paper strongly suggesting that our new discourse parser can encode discourse more effectively, we hope that our efforts will prompt researchers to develop more linguistically inspired applications based on our discourse parser. our contributions in this paper are: ## related work the field of discourse parsing has been mainly dominated by traditional machine learning approaches, frequently outperforming initial attempts to apply deep learning and neural networks to the task. independent of the specific approach used, three general methodologies have been followed to learn discourse trees from small datasets, such as rst-dt @xcite or instructional-dt @xcite : (1) top-down discourse parsers, splitting the document into non-overlapping text-constituents starting from the representation of the complete discourse down to individual edus, assigning the two resulting sub-spans a nuclearity attribute and predicting the relation holding between the sub-trees @xcite . (2) bottom-up parsing, starting from the discourse-segmented list of edus and aggregating two adjacent units in every step. this approach is mostly realized using the cky dynamic programming strategy to obtain optimal trees as in @xcite and @xcite or using a greedy method @xcite . (3) a frequently used and more locally inspired approach of bottom-up discourse parsing using the linear shift-reduce framework, adopted from previous work in syntactic parsing. while the current traditional state-of-the-art discourse parser by @xcite uses the bottom-up shift-reduce method predicted by two separate support vector machines (svms) for structure/nuclearity prediction and relation estimation, neural models @xcite utilize multi-layer perceptrons (mlp) for classifying possible actions. in our work we also follow the bottom-up shift-reduce strategy, with the detailed description of our system provided in the following section. besides the active research area on discourse parsing, a second line of work has emerged recently, trying to generate large-scale discourse treebanks through automated annotations from downstream tasks, such as sentiment analysis @xcite ), text classification @xcite , summarization @xcite and fake news detection @xcite . the majority of these approaches follows the intuition that discourse trees can be inferred from downstream tasks by predicting latent representations during the learning process of the task itself @xcite @xcite in an end-to-end manner. however, recent work by @xcite has shown that the trees resulting from this approach are not only poorly aligned with general discourse structures, but furthermore are oftentimes too shallow. a rather different strategy has been employed by @xcite , trying to explicitly generate a discourse augmented treebank through distant supervision from sentiment annotations in combination with multiple-instance learning (mil) and a cky bottom-up tree generation approach. while the resulting mega-dt dataset has only been proposed and released recently, the authors show promising results in their work, reaching the best inter-domain performance when comparing their dataset against rst-dt and the instruction-dt. this leads us to believe that their treebank does not only learn sentiment-related information, but can also be used to infer general discourse structures on a large scale. we will further evaluate this in section 4.7. ## neural discourse parsing we follow the well-established bottom-up shift-reduce aggregation principle, as previously shown effective for traditional discourse parsers such as @xcite as well as neural approaches @xcite . in this section, we will first introduce the general principle of shift-reduce parsing and define the necessary data structures and actions available to our system. based on the general description, we will subsequently describe the approach taken to execute a single step in the linear-time model. ## experiments we first introduce the three datasets we used to train and evaluate our model against strong baselines (section 4.1). further, we describe how to effectively combine diverse treebanks in a neural manner using pretraining and fine-tuning, now possible with our neural discourse parser (section 4.2). our model hyperparameters and the respective search spaces on the development set are presented in section 4.3, followed by the baseline models in section 4.4. the experiments section will then introduce the metrics used in this paper (section 4.5), discuss insights gathered in preliminary evaluations (section 4.6) and finally present results aggregated in section 4.7. ## conclusions in this work, we proposed a rather simple, yet highly effective discourse parser, utilizing recent neural bert-based language models in combination with structural features. the integration of those inputfeatures within a standard shift-reduce framework as well as an unprecedented use of recent large-scale "silver-standard" discourse parsing datasets for pretraining reaches a new state-of-the-art performance on both, the rst-dt and instr-dt treebanks. we show that our new, neural discourse parser already achieves better or similar performance when trained and evaluated on the rst-dt and instr-dt datasets, however, the consistent and significant sota result is reached when incorporating pretraining on the mega-dt corpus. the presented pretraining approach on the silver-standard mega-dt dataset also further validates the usefulness of additional supervision for this task and calls for more work in that area.