A Simple and Strong Baseline for End-to-End Neural RST -style Discourse Parsing (2022)

Abstract: To promote and further develop RST-style discourse parsing models, we need a strong baseline that can be regarded as a reference for reporting reliable experimental results. This paper explores a strong baseline by integrating existing simple parsing strategies, top-down and bottom-up, with various transformer-based pre-trained language models.The experimental results obtained from two benchmark datasets demonstrate that the parsing performance strongly relies on the pre-trained language models rather than the parsing strategies.In particular, the bottom-up parser achieves large performance gains compared to the current best parser when employing DeBERTa.We further reveal that language models with a span-masking scheme especially boost the parsing performance through our analysis within intra- and multi-sentential parsing, and nuclearity prediction.

## introduction rhetorical structure theory (rst) @xcite is one of the most influential theories for representing the discourse structure behind a document. according to the theory, a document is represented as a recursive constituent tree that indicates the relation between text spans consisting of a single elementary discourse unit (edu) or contiguous edus. the label of a nonterminal node describes the nuclearity status, either nucleus or satellite, of the text span, and the edge indicates the rhetorical relation between the text spans (figure 1 ). rst-style discourse parsing (hereafter rst parsing) is a fundamental task in nlp and plays an essential role in several downstream tasks, such as text summarization @xcite , question-answering @xcite , and sentiment analysis @xcite . in most cases, the performance of an rst parsing method has been evaluated on the largest english treebank, the rst discourse treebank (rst-dt) @xcite , as the benchmark dataset. the evaluation measures used include the micro-averaged f1 score of unlabeled spans, that of nuclearity-labeled spans, that of rhetorical relation-labeled spans, and that of fully labeled spans, based on standard parseval @xcite , when using gold edu segmentation. there are two major strategies for rst parsing: top-down and bottom-up. the former builds rst trees by splitting a larger text span consisting of edus into smaller ones recursively. the latter builds trees by merging two adjacent text spans. non-neural parsers with classical handcrafted features prefer the bottom-up strategy @xcite @xcite . on the other hand, recent neural parsers prefer the top-down strategy @xcite @xcite , while a few parsers employ the bottom-up strategy @xcite . with advances in neural network models, such neural parsers obtain significant gains over the non-neural parsers. several techniques have been proposed to boost rst parsing performance: @xcite and @xcite introduced beam search and @xcite exploited dynamic oracles in search to improve their parsing algorithms. @xcite used a model ensemble with multiple runs, @xcite introduced adversarial training, and @xcite and @xcite exploited silver data to improve parameter optimization. furthermore, pre-trained language models are playing an important role in improving the parsing performance. shi et al. (2020), nguyen et al. (2021), and zhang et al. (2021) employed xlnet @xcite to obtain better vector representations for arbitrary text spans consisting of edu(s). as a result, the current best top-down parser, @xcite with xlnet, achieved the fully labeled span f1 score of 53.8 with standard parseval. the method has a gain of 4-5 points compared to the best non-neural parser @xcite . however, it is still unclear what contributed the most to the improvement among the models' various factors such as parsing strategies, pre-trained language models, and a model ensemble. therefore, we need a simple but strong baseline for different parsing strategies along with a pre-trained language model to clarify the effects of the parsing strategies and pre-trained models. the baseline will contribute to building more reliable experiments for revealing the effectiveness of newly proposed methods. 1 this paper aims to build strong baselines for rst parsing, based on two simple open-source top-down @xcite and bottomup @xcite ) parsers, employing transformer-based pre-trained language models, without incorporating any of our own mechanisms. 2 the experimental results on rst-dt @xcite and instructional discourse treebank (instr-dt) @xcite with various pre-trained language models demonstrated that the parsing performance strongly relies on the performance of the pre-trained language models rather than the parsing strategies. while the current trend is a top-down parser, a bottom-up parser with deberta @xcite , one of the current state-of-the-art pre-trained language models, achieved the best score, which is higher than those of the current state-of-the-art parsers. further, our analysis based on intra-and multi-sentential parsing, and nuclearity prediction revealed that pre-trained language models with a span-masking scheme improve parsing performance more than those with a token-masking scheme. we will release our code at https://github.com/ nttcslab-nlp/rstparser_emnlp22 . ## related work early studies on rst parsing were based on nonneural supervised learning methods with handcrafted features. as parsing strategies, bottom-up greedy algorithms @xcite , shift-reduce @xcite , crfs (feng and hirst, 2014), and cky-like parsing algorithms @xcite have been employed. in particular, wang et al.'s shiftreduce parser @xcite , based on svms, achieved the best results among the non-neural statistical models on the rst-dt. the method first builds nuclearity-labeled rst trees and then assigns relation labels between two adjacent spans consisting of a single or multiple edus. inspired by the success of neural networks in many nlp tasks, several early neural networkbased models have been proposed for rst parsing @xcite @xcite . however, as reported by @xcite , while some neural approaches outperformed classical approaches, it was not by a large margin. recent end-to-end neural rst parsing models with sophisticated language models, such as glove and elmo, achieved better performance. they used vector representations of text spans based on the lstms whose inputs are word embeddings from the language models. @xcite proposed a bottom-up parser, based on the shift-reduce algorithm, that leverages the information from their neural dependency parsing model within a sentence for rst parsing. the method outperformed traditional non-neural methods and obtained a remarkable relation-labeled span f1 score of 49.2. as another approach, a top-down neural parser based on a sequence-to-sequence (seq2seq) framework was proposed @xcite for use only at the sentence level. the method parses a tree in a depth-first manner with a pointer-generator net- ## end-to-end neural rst parsing we employed the span-based parser @xcite for the top-down parsing strategy and the shift-reduce transition parser @xcite , an end-to-end variant of @xcite , for the bottom-up parsing strategy. these parsers were chosen here for their simple architecture and their open code. overviews of the parsers are shown in figs. 2 and 3 . both parsers basically consist of simple feed-forward networks (ffns) and bertbased embeddings. in this study, we used a twolayer perceptron with the gelu activation function and a dropout layer. ## pre-trained language models since most of the transformer-based pre-trained language models originated in bert, we employed bert and four of its variants as language models to obtain vector representations for text spans. bert: is trained with two tasks: (1) a masked language model (mlm); 15% of the tokens in the training data are randomly masked, and then the model is trained to predict the masked tokens, and (2) a next sentence prediction (nsp) task; the model is trained to correctly predict the following sentence for a given sentence. bert is trained on book corpus and english wikipedia. roberta: is trained with longer and larger batches over more data and longer sequences. it further removes nsp and dynamically changes the masking pattern applied to the training data. in addition to the dataset used for training bert, the training dataset here includes cc-news, openweb-text, and stories as well. xlnet: is a generalized autoregressive pre-trained language model, known as a permuted language model (plm). it is trained by maximizing the expected likelihood over all permutations of the factorization order of the input text to approximately consider bidirectional context. in addition to the dataset used for training bert, the training dataset includes giga5, clueweb, and cc as well. spanbert: is trained with (1) a masked language model with random spans (contiguous tokens) and deberta: is a modified variant of roberta. it uses disentangled attention and an enhanced mask decoder. during training, it masks spans randomly as for spanbert. while the dataset used for train-ing is a subset of that used for roberta, it performs consistently better on various nlp tasks. ## results the results obtained from paired bootstrap resampling tests foot_4 between top-down and bottom-up parsers whilst fixing the language model show that significant differences are found only in span and nuc. for xlnet on rst-dt, and rel. and full for spanbert on instr-dt, respectively. on the other hand, the performance of the parsers varies widely depending on their language model when fixing the parsing strategy. to investigate the significant differences among parsers, we performed multiple comparison tests based on the paired bootstrap resampling tests while controlling the false discovery rate @xcite . the results show that deberta significantly outperformed bert, spanbert, and sometimes significantly outperformed roberta and xlnet in top-down parsing. in contrast, it significantly outperformed bert, roberta, xlnet, and spanbert in bottom-up parsing. roberta and xlnet obtained similar results; they significantly outperformed bert, and sometimes significantly outperformed spanbert. while spanbert only significantly outperformed bert, it sometimes has comparable performance to roberta and xlnet. in particular, the bottom-up parser with deberta outperformed span, nuc., rel., and full scores of the current best parser @xcite by 1.5, 2.5, 1.7, and 1.6 points, respectively. 7 furthermore, most parsers yield a performance comparable to current state-of-the-art parsers. we believe that the results have a significant impact in the rst-parsing community. since we built our baseline parsers based on a simple architecture, as described in section 3, we can conduct more reliable experiments to reveal the effectiveness of newly proposed methods on top of them without any concern regarding the choice of pre-trained language models or parsing strategies. while the evaluation results demonstrate that we successfully built baseline parsers, they also raise the following questions for us: (1) why did de-berta, trained with the half-size dataset (85g), outperform roberta, trained with the most extensive dataset (161g) (2) why did spanbert consistently outperform bert with significant differ-ences, even though they are trained with the same dataset (16gb). it is well known that pre-trained language models trained with large-scale datasets boost the performance @xcite ; however, the above results do not necessarily agree with the finding. we believe that the results may be due to a span-masking scheme, a common feature between spanbert and deberta. with the span-masking scheme, randomly generated spans consisting of a sequence of tokens with the length up to 5 (for spanbert) or 3 (for deberta) are masked, and then the language models are trained to predict the span boundary tokens in the mask. that is, the span-masking scheme is considered more contextsensitive than the token-masking scheme. thus, pre-trained language models with a span-masking scheme are suitable for obtaining vector representations for long text spans consisting of edus. to discuss the impact of the span-masking scheme in more detail, we evaluated our parsers the effectiveness of the span-masking scheme in ## conclusion this paper explored ways to build strong baselines for rst parsing, based on existing top-down and bottom-up parsers, while varying the use of five transformer-based pre-trained language models: bert, roberta, xlnet, spanbert, and deberta. we employed a span-based model as a top-down parser and a shift-reduce model as a bottom-up parser. the experimental results obtained from the rst-dt and instr-dt revealed that the language models, except for bert, boost the performance of rst parsing in both strategies. the deberta-based bottom-up parser achieved the best scores; in particular, the fully labeled span f1 score of 55.4 on the rst-dt. furthermore, our experimental results implied that language models with a span-masking scheme, such as spanbert, deberta, are suitable for rst parsing since they would generate better representations for long text spans than those with a token-masking scheme.