Top-down Discourse Parsing via Sequence Labelling (2021)

Abstract: We introduce a top-down approach to discourse parsing that is conceptually simpler than its predecessors (Kobayashi et al., 2020; Zhang et al., 2020). By framing the task as a sequence labelling problem where the goal is to iteratively segment a document into individual discourse units, we are able to eliminate the decoder and reduce the search space for splitting points. We explore both traditional recurrent models and modern pre-trained transformer models for the task, and additionally introduce a novel dynamic oracle for top-down parsing. Based on the Full metric, our proposed LSTM model sets a new state-of-the-art for RST parsing.

## introduction discourse analysis involves the modelling of the structure of text in a document. it provides a systematic way to understand how texts are segmented hierarchically into discourse units, and the relationships between them. unlike syntax parsing which models the relationship of words in a sentence, discourse parsing operates at the document-level, and aims to explain the flow of writing. studies have found that discourse parsing is beneficial for downstream nlp tasks including document-level sentiment analysis @xcite and abstractive summarization @xcite . rhetorical structure theory (rst; @xcite ) is one of the most widely used discourse theories in nlp @xcite @xcite @xcite . rst organizes text spans into a tree, where the leaves represent the basic unit of discourse, known as elementary discourse units (edus). edus are typically clauses 1 code and trained models: https://github.com/ fajri91/neuralrst-topdown edu-1 edu-4 edu-2 edu-3 elab edu-1: roy e. parrott, the company's president and chief operating officer since sept. 1, was named to its board. edu-2: the appointment increased the number of directors to 10, edu-3: three of whom are company employees. edu-4: simpson is an auto parts maker. of a sentence. non-terminal nodes in the tree represent discourse unit relations. in figure 1 , we present an example rst tree with four edus spanning two sentences. in this discourse tree, edus are hierarchically connected with arrows and the discourse label elab. the direction of arrows indicates the nuclearity of relations, wherein a "satellite" points to its "nucleus". the satellite unit is a supporting sentence for the nucleus unit and contains less prominent information. it is standard practice that the rst tree is trained and evaluated in a right-heavy binarized manner, resulting in three forms of binary nuclearity relationships between edus: nucleus-satellite, satellite-nucleus, and nucleus-nucleus. in this work, eighteen coarse-grained relations are considered as discourse labels, consistent with earlier work @xcite . 2 work on rst parsing has been dominated by the bottom-up paradigm @xcite @xcite @xcite . these methods produce very competitive benchmarks, but in practice it is not a straightforward bigru (single edu) bigru (inter-edu) cnn (@xmath0.... biaffine attention zhang et al., 2020 bilstm (single edu) bilstm (inter-edu) bilstm (seq labelling) queue ours (v1) bilstm (single edu) kobayashi et al., 2020 zhang et al. (2020) and kobayashi et al. (2020). approach (e.g. transition-based parser with actions prediction steps). furthermore, bottom-up parsing limits the tree construction to local information, and macro context such as global structure/topic is prone to be under-utilized. as a result, there has recently been a move towards top-down approaches @xcite . the general idea behind top-down parsing is to find splitting points in each iteration of tree construction. in figure 2 , we illustrate how our architecture differs from @xcite and @xcite . first, @xcite utilize four levels of encoder that comprise 3 bi-grus and 1 cnn layer. the splitting mechanism is applied through a decoder, a stack, and bi-affine attention mechanisms. @xcite use the gold paragraph and sentence boundaries to aggregate a representation for each unit, and generate the tree based on these granularities. two bi-lstms are used, with splitting points determined by exhaustively calculating the bi-affine score of each possible split. the use of paragraph boundaries can explicitly lower the difficulty of the task, as 77% of paragraphs in the english rst discourse treebank ("rst-dt") are actually text spans @xcite . these boundaries are closely related to gold span boundaries in evaluation. in this paper, we propose a conceptually simpler top-down approach for rst parsing. the core idea is to frame the problem as a sequence labelling task, where the goal is to iteratively find a segmentation boundary to split a sequence of discourse units into two sub-sequences of discourse units. this way, we are able to simplify the architecture, in eliminating the decoder as well as reducing the search space for splitting points. specifically, we use an lstm @xcite or pre-trained bert @xcite as the segmenter, enhanced in a number of key ways. our primary contributions are as follows: (1) we propose a novel top-down approach to rst parsing based on sequence labelling; (2) we explore both traditional sequence models such as lstms and also modern pre-trained encoders such as bert; ## related work previous work on rst parsing has been dominated by bottom-up approaches @xcite @xcite @xcite . for example, @xcite introduce dplp, a transition-based parser based on an svm with representation learning, combined with some heuristic features. @xcite propose joint text segment representation learning for predicting rst discourse trees using a hierarchical bi-lstm. elsewhere, @xcite showed that implicit syntax features extracted from a dependency parser @xcite are highly effective for discourse parsing. top-down parsing is well established for constituency parsing and language modelling @xcite @xcite , but relatively new to discourse parsing. @xcite propose a unified framework based on pointer networks for sentence-level discourse parsing, while @xcite employ hierarchical pointer network parsers. @xcite found that most previous studies on parsing rst discourse tree were incorrectly benchmarked, e.g. one study uses macroaveraging while another use micro-averaging. 3 they also advocate for evaluation based on microaveraged f-1 scores over labelled attachment decisions (a la the original parseval). pre-trained language models @xcite have been shown to benefit a multitude of nlp tasks, including discourse analysis. for example, bert models have been used for classifying discourse markers @xcite al., w 2 w 1 s 1 s 2 p 2 p 1 word and pos bi-lstm 1 bi-lstm 2 syntax feature avg pooling avg pooling edu 1 g e1 bi-lstm 3 bi-lstm 4 ỹ em ỹ en mlp layer h e1 h e2 w 2 w 1 s 1 s 2 p 2 p 1 word and pos bi-lstm 1 bi-lstm 2 syntax feature avg pooling avg pooling eduq g eq encoder get sub-sequence h e3 h eq ... h em h en ... segmenter ... ... ... ... t e1 t eq 2019) and discourse relations @xcite . to the best of our knowledge, however, pre-trained models have not been applied in the generation of full discourse trees, which we address here by experimenting with bert for topdown rst parsing. ## top-down rst parsing we frame rst parsing as a sequence labelling task, where given a sequence of input edus, the goal is to find a segmentation boundary to split the sequence into two sub-sequences. this is realized by training a sequence labelling model to predict a binary label for each edu, and select the edu with the highest probability to be the segmentation point. after the sequence is segmented, we repeat the same process for the two sub-sequences in a divide-and-conquer fashion, until all sequences are segmented into individual units, producing the binary rst tree (e.g. figure 1 ). ## analysis in in total, our best model obtains 1,698 out of 2,308 spans of original parseval trees, and correctly predict 1,517 segmentation points (pairs). we further analyze these pairs by presenting the confusion matrices of nuclearity and relation prediction in figure 7 and figure 8 , respectively. first, the model tends to have more errors on nn (nucleus-nucleus) prediction where 53 span in figure 8 we present analysis over top-7 relations and a relation other that represents the rest of 11 classes. similar to the nuclearity prediction, the relation class distribution is also imbalance where elab accounts for 37% of the examples. some relations are related to elab (see ## conclusion we introduce a top-down approach for rst parsing via sequence labelling. our model is conceptually simpler than previous top-down discourse parsers