Why Can’t Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity (2023)

Abstract: Recent advances in discourse parsing performance create the impression that, as in other NLP tasks, performance for high-resource languages such as English is finally becoming reliable. In this paper we demonstrate that this is not the case, and thoroughly investigate the impact of data diversity on RST parsing stability. We show that state-of-the-art architectures trained on the standard English newswire benchmark do not generalize well, even within the news domain. Using the two largest RST corpora of English with text from multiple genres, we quantify the impact of genre diversity in training data for achieving generalization to text types unseen during training. Our results show that a heterogeneous training regime is critical for stable and generalizable models, across parser architectures. We also provide error analyses of model outputs and out-of-domain performance. To our knowledge, this study is the first to fully evaluate cross-corpus RST parsing generalizability on complete trees, examine between-genre degradation within an RST corpus, and investigate the impact of genre diversity in training data composition.

## introduction discourse parsing is the task of identifying and classifying the coherence relations that hold between different parts of a text, such as recognizing that one sentence specifies the cause of events in another, or that a subordinate clause indicates the purpose of a main clause. in hierarchical frameworks, such as the rhetorical structure theory @xcite , parsers construct a labeled constituent tree of discourse units as shown in figure 1 . such trees have numerous applications: retrieving specific relations (e.g. all concessions made in any speech by some politician, e.g. unit 26-27 in figure 1 ), or hierarchically finding the most central unit in an arbitrary span of text (unit 24 in figure 1 ). they can also contribute to downstream tasks such as text generation @xcite . @xcite and summarization (e.g. @xcite @xcite and to qualitative analysis and comparison of texts @xcite . 1 recent advances in nlp have resulted in increasingly accurate systems for tasks such as part-ofspeech tagging @xcite and dependency parsing @xcite , which are now commonly applied with confidence to novel data, at least for high-resource languages such as english. at the same time, rising scores on the main standard english rst benchmark, the rst discourse treebank (rst-dt, @xcite , with data from the 1989 wall street journal (wsj), create the impression that discourse parsing too is becoming reliable, and by proxy, applicable to arbitrary text. this has resulted in use of automatic rst parsing to automatically generate large-scale datasets, such as mega-dt @xcite or amalgum @xcite . however, there are some indications that this picture is too optimistic, with some studies revealing problems in rst-dt-based parsing once the target domain changes @xcite @xcite @xcite @xcite , and qualitative inspection of parser outputs casting doubt on their reliability. a problem in evaluating the stability of rst parsing for english is the scarcity of out-of-domain (ood) data, as well as potential differences between the annotation schemes of different corpora. recently, the availability of increasing amounts of non-newswire data, as well as non-wsj news data, in the english gum corpus @xcite has made some detailed comparisons possible, which we explore below. beyond evaluating ood degradation, the existence of these large datasets for english rst allows us to test different joint training strategies to increase both the accuracy and generalizability of rst parsing for english. 2 our main goals here are: 1) to demonstrate the generalizability limitations of english rst parsing based on rst-dt and quantify them for users; 2) to explore reasons for generalizability issues, with a focus on the genre composition of training sets, pointing the way to the kind of data that robust discourse parsing requires; and 3) to promote multi-genre benchmarks for rst parsing based on our experimental results. overall we find that diverse training data leads to better generalization on unseen genres regardless of model architecture. ## experiments in this section we conduct four experiments. first, we want to find out how compatible the two english rst corpora are, and how well they generalize to each other's domains ( §3.1). second, we want to test whether we can incorporate training data from gum in order to exceed sota performance on rst-dt, with one of three approaches: joint training, model stacking, and pretraining ( §3.2). third, we would like to know whether generalizability issues are mainly due to differences between corpora (details of annotation guidelines or common practices) or the nature of the texts' genres and domains themselves ( §3.3). we test this by leaving out different subsets of gum genres from training, and quantify ood degradation for each genre, which also informs users of sota parsers of what degradation they can expect on genres 'in the wild'. the final experiment ( §3.4) tests whether, given a fixed training set size, having relatively many 'small' genres leads to better ood stability than having few genres with more material each, when the target is a totally disjoint set of genres. the code, trained and finetuned models, and predicted gum parses are available at https:// github.com/janetlauyeung/crossgenre4rst . ## error analysis due to space, we limit our error analysis to examining dependency conversions of gold vs. predicted trees, which allow us to break down ood errors by coarse relation class in errors are also skewed by genre: to further investigate genre-specific phenomena, we analyzed ood parsing errors for how-to guides, which had the worst ood performance in figure 5h shows cases with attachment errors. one major error concerning global structure is seven gold organization instances being confused with joint. two of the errors are unexpected given that there are strong graphical, structural, and adversative attribution causal context contingency elaboration evaluation explanation joint mode organization purpose restatement same-unit pred adversative attribution causal context contingency elaboration evaluation explanation joint mode organization purpose restatement same-unit ## conclusion the analyses in this paper are meant to inform users of current parsers about what to expect from rst parsing in the wild. through dozens of experimental runs we have shown a consistent picture: rst parsing has made impressive progress, but ood degradation is still severe. our results suggest prioritizing genre diversity in training data is crucial, not only to cover more text types as 'in domain', but also to increase performance on unseen text types. rather than focusing only on the development of better models to beat the single genre rst-dt benchmark, robust rst parsing would be promoted most by creating more annotated data. we also hope this paper will motivate researchers to prioritize multi-genre benchmarks and ood settings for rst parsing, and to explore algorithms, representations and features which better capitalize on joint training and foster generalizability, including using data from other sources and theories of discourse analysis (cf. @xcite . we see great challenges for tackling errors which implicate complex pragmatic inference or document level reasoning, but we are also optimistic that as more data becomes available, we will be able to learn more and better representations of discourse structure.