RST Discourse Parsing with Second-Stage EDU -Level Pre-training (2022)

Abstract: Pre-trained language models (PLMs) have shown great potentials in natural language processing (NLP) including rhetorical structure theory (RST) discourse parsing. Current PLMs are obtained by sentence-level pre-training, which is different from the basic processing unit, i.e. element discourse unit (EDU).To this end, we propose a second-stage EDU-level pre-training approach in this work, which presents two novel tasks to learn effective EDU representations continually based on well pre-trained language models. Concretely, the two tasks are (1) next EDU prediction (NEP) and (2) discourse marker prediction (DMP).We take a state-of-the-art transition-based neural parser as baseline, and adopt it with a light bi-gram EDU modification to effectively explore the EDU-level pre-trained EDU representation. Experimental results on a benckmark dataset show that our method is highly effective,leading a 2.1-point improvement in F1-score. All codes and pre-trained models will be released publicly to facilitate future studies.

## introduction discourse analysis based on rhetorical structure theory (rst) has received increasing interest in the natural language processing (nlp) community @xcite @xcite @xcite @xcite , which organizes discourse output through a well-defined tree structure. figure 1 shows an example of an rst constituent tree, where the leaf nodes are element discourse units (edus). given an edu sequence, rst discourse parsing aims to automatically construct a hierarchical constituent tree 2 . the shift-reduce transition-based model has been widely adopted in rst discourse parsing @xcite , building the constituent tree incrementally with multiple steps by a sequence of actions. these models take edulevel features as inputs to score transition actions at each step. recently, neural network models have achieved state-of-the-art performance for this task by using sophisticated-designed neural modules @xcite @xcite @xcite . in particular, the contextualized pre-trained language models (plms) such as xlnet @xcite is able to achieve an impressive performance, resulting in f1-score gains of more than 3 points according to previous studies @xcite @xcite and our preliminary findings. although great successes have been observed by the contextualized plms @xcite @xcite , an apparent mismatch in the basic processing units exists between the edu-level rst parsing and the sentence-level contextualized language modeling, which might be unable to fully explore the pre-training paradigm. several previous studies have been investigated to address the mismatch between the target tasks and the standard language model pre-training, e.g., spanbert @xcite for extractive question answering, bart @xcite , and t5 @xcite for sequence-to-sequence (seq2seq) generation, and all these studies achieve improved performances for their target tasks. in this study, we investigate a second-stage edulevel pre-training based on the above observation. concretely, we conduct pre-training from a plm with two edu-level tasks in the second stage. the first task is next edu prediction (nep), which is inspired by next sentence prediction (nsp) in bert @xcite learning, substituting the sentences with edus. the second task is discourse marker prediction (dmp), which is also inspired by the masked language modeling (mlm) in bert @xcite learning, substituting the masked words with the masked discourse markers. to fully utilize contextualized pre-trained representations, we adapt a transition-based neural rst parser that exploits biedu representations with regard to the basic encoding unit instead of the standard single-edu manner. we conduct experiments on rst discourse treebank (rst-dt) @xcite to evaluate the proposed model. first, we derive biedu representations directly from plms, and thus build a very strong transition-based neural rst parser. then, we examine the proposed second-stage edulevel pre-training approach. experimental results show that the two second-stage pre-training tasks improve rst parsing greatly, and their combination leads to further increases. our final model achieves the top performance among all the models reported in the literature. in summary, our contributions are as follows: ## second-stage edu-level pre-training in this section, we introduce the proposed secondstage edu-level pre-training approach. it has two ## transition-based neural rst parser we adopt a transition-based neural rst parser to evaluate the second-stage edu-level pre-training approach. the model has two key components, termed by a transition system and a neural network model, respectively. the transtion system, mainly borrowed from @xcite , formalizes rst parsing into action sequence predictions, and the neural model yields edu representations and outputs action sequences. ## experiments 4.1 settings datasets to show the proposed model is comparable with previous state-of-the-art systems for rst parsing, we conduct experiments on rst-dt foot_2 @xcite . it is a standard benchmark dataset for this task, which is collected from the wall street journal news. it has been divided into training and test sets, which have 347 discourses and 38 discourses, respectively. we randomly select 35 discourses from the training set to develop our model. the original rst-dt contains 78 fine-grained discourse relations. most of previous studies simplify these fine-grained discourse relations to 18 coarse-grained relations. to facilitate comparison with previous studies, we also use 18 simplified coarse-grained relations. to show the domain generalization capability of our proposed rst parser to unseen domain articles, we test it on the georgetown university multilayer (gum) corpus foot_3 . it contains small-scale articles annotated based on rst in several domains, such as news, fiction, conversations, and etc. for more details, one can refer to their paper @xcite . the training corpus for second-stage edu-level pre-training contains unlabeled large-scale collected from a english wikipedia corpus foot_4 . although using a unlabeled news corpus may lead to greater improvements, we find that using a wikipedia corpus is sufficient to provide new sota results. evaluation we use the evaluation recommended by @xcite , which attaches nuclearity and relation labels to non-leaf trees to eliminate redundant evaluations. the evaluation includes four metrics, termd by span, nuclearity, relation, and full, respectively. span evaluates the skeleton of the discourse tree. nuclearity evaluates the discourse tree with nuclearity labels. relation evaluates the discourse tree with relation labels. full evaluates the complete discourse tree with nuclearity and relation labels. hyper-parameters there are several hyperparameters in our proposed second-stage edulevel pre-training approach and rst parser. in nep, the learning rate of plm is set to 5e-6, and the learning rate of the other model parameters is set to 1e-3. the batch size is set to 50. the maximum norm of gradient clipping is set to 1. the maximum tranining epoch number is set to 10. in dmp, the learning rate of plm is set to 1e-6, and the learning rate of the other model parameters is set to 1e-4. the batch size is set to 1. the output hidden size of lstm is set to 200. the settings of maximum training iteration number and the norm of gradient cliping are the same as nep. the hyper-parameters of our rst parser are tuned based on the preliminary results on the development set. the hidden size of all neural layers is set to 200. the dropout is set to 0.25. the learning rate of plm is set to 2e-5, and the learning rate of other model parameters is set to 1e-3. the maximum norm of gradient clipping is set to 1, and the maximum training iteration number is set to 20. we use transformers library @xcite ) to implement plm and use pytorch @xcite to implement other neural network modules. ## related work rst discourse parsing is an important task in the nlp community, which has been studied since early @xcite . early studies adopt statistical models for this task, using humandesigned discrete features @xcite @xcite @xcite . recently, several neural network models show great promising for this task @xcite @xcite @xcite @xcite . with plms such emlo (peters et al., 2018), bert @xcite , xlm-roberta (conneau and lample, 2019), and xlnet @xcite , these neural rst parsers report high competitive performances @xcite @xcite @xcite . we follow the line of these studies, using neural networks to perform rst parsing. recently, several studies aim to alleviate the mismatch between pre-trained language models and target tasks. @xcite use a span masked language modeling to pre-train a language model for extraction question answering. @xcite propose a pre-training approach for text generation tasks, which maps corrupt documents ## conclusion and future work we proposed a second-stage edu-level pretraining approach for plm-based rst discourse parser, reducing the mismatch between the edulevel rst discourse parsing and the pre-training of sentence-level contextualized language modeling. in addition, we extended our rst discourse parser with a light bi-gram edu modification, finding that it is able to exploit plms more effectively. experiments on rst-dt @xcite showed that the proposed approach can bring significantly better performance for rst discourse parsing. we further conducted several experimental analysis to better understand the proposed approach. the results on the rst-dt @xcite and the gum @xcite corpora suggest two possibilities for future research. first, although the xlnet rst parser obtains significantly improvements when the second-stage edu-level pretraining approach is adopted, the relation and the full metrics of our rst parser are still lower than the best system. future research might extend the second-stage edu-level pre-training task, using pesudo rst discourse trees. second, the generalization ability of our proposed rst parser needs to be improved in multi-domain scenarios. so in future we may continue to explore the issue of domain adapation in rst parsing on the basis of the second-stage edu-level pre-training framework.