What’s Hard in E nglish RST Parsing? Predictive Models for Error Analysis (2023)

Abstract: Despite recent advances in Natural Language Processing (NLP), hierarchical discourse parsing in the framework of Rhetorical Structure Theory remains challenging, and our understanding of the reasons for this are as yet limited. In this paper, we examine and model some of the factors associated with parsing difficulties in previous work: the existence of implicit discourse relations, challenges in identifying long-distance relations, out-of-vocabulary items, and more. In order to assess the relative importance of these variables, we also release two annotated English test-sets with explicit correct and distracting discourse markers associated with gold standard RST relations. Our results show that as in shallow discourse parsing, the explicit/implicit distinction plays a role, but that long-distance dependencies are the main challenge, while lack of lexical overlap is less of a problem, at least for in-domain parsing. Our final model is able to predict where errors will occur with an accuracy of 76.3% for the bottom-up parser and 76.6% for the top-down parser.

## introduction powered by pretrained language models, recent advancements in nlp have led to rising scores on a myriad of language understanding tasks, especially at the sentence level. however, at the discourse level, where analyses require reasoning over multiple sentences, progress has been slower, with generalization to unseen domains remaining a persistent problem for tasks such as coreference resolution @xcite and entity linking @xcite . one task which remains particularly challenging is hierarchical discourse parsing, which aims to reveal the structure of documents (e.g. where parts begin and end, which parts are more important than others) and make explicit the relationship between clauses, sentences, and larger parts of the text, by labeling them as expressing a type of e.g. causal, elaboration, etc. more specifically, hierarchical discourse parses identify connections between elementary discourse units (edus, usually equated with propositions) in a text or conversation, classify their functions using a closed tag set, and form a recursive tree structure, which indicates the locally most prominent edu in each tree or subtree. figure 1 shows an example tree in the most popular hierarchical discourse formalism, rhetorical structure theory @xcite , in which the list of units 37-38 is the most prominent (being pointed to by other units directly or indirectly), and discourse relation labels such as cause are identified using edge labels, whose definitions in rst are based on the rhetorical effect which the writer (or speaker) is thought to be conveying to the reader (or hearer). there is by now substantial evidence showing that even for a high resource language like english, state-of-the-art (sota) neural rst discourse parsers, whether employing a top-down or a bottom-up architecture, do not perform well across domains @xcite @xcite , with some crucial tasks, such as predicting the most prominent central discourse unit (cdu) of each document, performing at just 50% @xcite . at the same time, we do not have a good understanding of what exactly prevents good performance-is it the fact that some relations are well-marked (for example, most contingency relations are marked by the discourse marker (dm) if, but most evaluation relations lack a common marker)? conversely, is the presence of distracting markers not associated with the correct relation (e.g. an additional temporal marker such as then inside a unit with a non-temporal function)? alternatively, is it the difficulty in identifying high-level relations, between groups of multiple sentences or paragraphs, compared to less tricky intra-sentential relations between clauses? or is it just the prevalence of out-of-vocabulary (oov) items in test data? in this paper, we would like to systematically evaluate the role of these and other factors contributing to errors in english rst discourse parsing. our contributions include: ## data to examine the role of explicit vs. implicit relations in parsing errors, we first need to know which relations were explicitly signaled. to that end, we use pdtb's methodology to define explicit connectives. note that rst papers often use the term dm without clear inventories; from this point on we will use 'dm' for brevity, but strictly adhere to the pdtb english inventory. specifically, we annotate data from the two largest rst corpora for english, covering the test set of rst-dt foot_0 @xcite and the test and dev sets of gum @xcite , with 1) discourse markers (including 'distractor' dms) and 2) associated relations, thereby attaching dms to each relation they signal, or no relation. these scores indicate a high agreement between annotators for both tasks. ## analysis strictly speaking, the types of errors that topdown and bottom-up parsers make are not identical: while bottom-up, and in particular shift-reduce parsers see analyzed preceding discourse units, grouped in a stack, and remaining discourse units in an upcoming queue, top-down parsers analyze a domain of ungrouped tokens to be split and determine the optimal split point and label for each decision. because we want to analyze what promotes errors both across and for each architecture, we adopt an output-centric view, analyzing edus at which parsers do and do not make errors based on their properties in the completed gold vs. predicted tree. at the same time, we do not want our results to be swayed by coincidental variations in neural models, which can have far-reaching consequences due to cascading errors. instead, we train five models in each architecture, i.e. five training runs, each with a different random seed producing a different initialization for the parser: if only one model fails to predict a relation, it may not be very hard, while 4-5 errors would be indicative of genuinely hard relations. additionally, since models ultimately confront different inputs as a result of such cascaded decisions, we will use a dependency representation of both the gold and predicted rst trees, following the dependency conversion as defined by li et al. (2014), foot_2 as exemplified in figure 2 . although rst uses constituent discourse trees, focusing on each edu and its dependencies will make it possible to make meaningful comparisons across models, and to intuitively understand how challenging edus are at any point in each document, regardless of whether or not they head large constituent structures. in section 4.2 we will also incorporate the spanned domain of each head edu's constituent block as an additional feature to assess the role of block size in predicting errors. ## conclusion this study has several important implications. firstly and unsurprisingly, the explicit/implicit distinction from shallow discourse parsing is mirrored in rst parsing difficulty, and the dataset released in this paper can help study it further. however, explicit marking is clearly less consequential than intra-sentential status, with which explicitness it correlated. secondly, oov rate plays a less important role than we initially suspected, while genre effects remain robust, suggesting that diverse genres may matter more than subject matter. our results also indicate that current architectures do not differ substantially in what they get right or wrong, and with scores being so similar, differences reduce to computational efficiency and personal preference. finally, the study of distractors suggest that rst's tree constraint may mix some cases of multiple concurrent relations with parsing errors, when parsers are actually identifying viable relations. this suggests that we may want to consider ways of allowing and adding concurrent relations to rst parses. we also note that although the error prediction models evaluated in section 4.2 were primarily developed in order to gain a greater understanding of the issues in discourse parsing, they could have some practical applications. 7 predicting regions of low certainty in discourse parses can: 1) assist by highlighting low confidence regions in userfacing downstream applications; 2) flag potential problems during annotation of resources, especially when relying on nlp @xcite or less trained annotators/crowd workers @xcite ; and 3) help guide additional resource acquisition, either automatically using active learning (to prioritize documents predicted to have parsing problems for manual annotation, cf. gessler et al. 2022) or using qualitative evaluation in deciding what data to collect in terms of the relative importance of genres, presence of oov items, etc. amir zeldes, yang janet liu, mikel iruskieta, philippe muller, chloé braud, and sonia badene. 2021. the disrpt 2021 shared task on elementary discourse unit segmentation, connective detection, and relation classification. in proceedings of the 2nd shared task on discourse relation parsing and treebanking (disrpt 2021), pages 1-12, punta cana, dominican republic. association for computational linguistics. zheng zhao and bonnie webber. 2021. revisiting shallow discourse parsing in the pdtb-3: handling intrasentential implicits. in proceedings of the 2nd workshop on computational approaches to discourse, pages 107-121, punta cana, dominican republic and online. association for computational linguistics. yilun zhu, sameer pradhan, and amir zeldes. 2021. ontogum: evaluating contextualized sota coreference resolution on 12 more genres. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), pages 461-467, online. association for computational linguistics.