GDTB : Genre Diverse Data for E nglish Shallow Discourse Parsing across Modalities, Text Types, and Domains (2024)

Abstract: Work on shallow discourse parsing in English has focused on the Wall Street Journal corpus, the only large-scale dataset for the language in the PDTB framework. However, the data is not openly available, is restricted to the news domain, and is by now 35 years old. In this paper, we present and evaluate a new open-access, multi-genre benchmark for PDTB-style shallow discourse parsing, based on the existing UD English GUM corpus, for which discourse relation annotations in other frameworks already exist. In a series of experiments on cross-domain relation classification, we show that while our dataset is compatible with PDTB, substantial out-of-domain degradation is observed, which can be alleviated by joint training on both datasets.

## introduction language in discourse is more than an ordered list of sentences or clauses: parts of a text expressing events, states, facts, or propositions are often connected by discourse relations, such as cause (one part of a text specifies the cause for another), con-cession (one part states content which a speaker or author expects recipients to overlook) etc. such relations may be marked explicitly, for example by connectives, which are conjunctions or adverbials such as 'because' or 'nevertheless' in english; or they may be implicit, requiring recipients to interpret the text more actively. given an arbitrary natural language text as input, shallow discourse parsing is the task of identifying pairs of text spans connected by a discourse relation, in some scenarios focusing mainly on explicit, implicit, or other subtypes of relations @xcite , as well as the means by which they are signaled, resulting for example in connective disambiguation (e.g. 'since' is a connective which can express either a cause or temporal relation). most commonly, shallow discourse parsing systems use the inventory of relations defined by the penn discourse treebank (pdtb, currently version 3; see section 2.1). systems and data for shallow discourse parsing can be used for a variety of downstream applications, including relation extraction (identification of relations given two spans, @xcite , instruction fine-tuning or pretraining of language models @xcite , study of argumentation and persuasiveness @xcite , and crosslinguistic lexicography of discourse connectives @xcite @xcite . when finding specific relation types is desired, shallow discourse parsing also forms an end task in itself: for example, finding all con-cession relations in a large corpus of speeches by a politician or political party for computational social science studies. although work on shallow discourse parsing has expanded to a range of languages (e.g. chinese, zhou and xue 2014; czech, @xcite @xcite @xcite , less progress has been made on expanding data to new and diverse domains (see section 2.2). a major cause of this bottleneck is the effort associated with manual construction of high quality data covering a broad range of domains from scratch. in this paper we suggest overcoming this hurdle by not starting from scratch: we target the freely available english gum corpus (which is also available as part of the universal dependencies project, de @xcite , which covers a broad range of 16 spoken and written english genres and for which annotations are available in hierarchical discourse parsing frameworks: rst and erst (see section 2.1). although these frameworks are sub-stantially different from pdtb, they provide sufficient information to obtain a high quality starting point for semi-automatic conversion of data into the pdtb v3 framework. as an added advantage, we also develop a mapping of (e)rst to pdtb relations, allowing for cross-framework comparisons along the lines proposed by @xcite (see @xcite for a similar argument and approach to converting coreference datasets). in the subsequent sections of this paper, we will first briefly survey the discourse relation frameworks involved in this project (section 2), and then we will present our data, its creation process, and an evaluation of its quality (section 3). this will be followed by a set of experiments on cross-corpus and joint-training relation classification to evaluate both the compatibility of our data with pdtb, and the degree of cross-corpus (and by extension, cross-domain) performance degradation. ## experimental setup to test the utility of our corpus and its compatibility (or redundancy) with the existing pdtb v3, we again train the discodisco relation classifier @xcite using the standard disrpt version of pdtb v3 relations, which simply provides the spans of the two arguments including connectives, and their containing sentences, without a separate connective field, and as a result treats explicit, implicit, and other relation types uniformly, foot_3 though we also report separate scores on different relation types and overall. to investigate the effects of both data size and data diversity, we evaluate in three training setups: within-corpus (e.g. train and test on pdtb v3, and the same for gdtb respectively); cross-corpus (train on pdtb v3 and evaluate on the gdtb test set, and vice versa); and joint training (train on both training sets, evaluate on each test set). see implementation details in appendix f. ## results however, the different proportions of explicit relations (which are easier to tag) and implicit ones mean that implicit relations in particular show massive cross-corpus degradation for pdtb, which is again unsurprising: while connectives remain more or less constant across datasets (i.e. 'but' usually signals comparison.contrast or compari-son.concession in both datasets), in implicit settings, the relations between surrounding lexical items must be learned, which vary more substantially by genre. although the gdtb model has seen some news data from gum news, the quantity of the material is insufficient to achieve comparable scores to the pdtb-trained model, which has 1.2m tokens of wsj data to learn from. if we zoom in on genres in gdtb, figure 2 shows scores for subsets of gdtb test from both the within-corpus and the cross-corpus models. 6 overall, the best-performing genres for each model are news and academic respectively, while the worst-performing genre is court for both models. the bottom four places in implicit relations withincorpus are all occupied by spoken genres (podcast, court, conversation, interview). the conversation genre is particularly bad for cross-corpus, i.e. when training on pdtb, with an implicit score of just 11.43%. unsurprisingly, news scores highest on cross-corpus implicit relation prediction (72.22%), though academic scores higher overall (68.60%) (due to frequent explicit relations), and interview scores highest for cross-corpus explicit (84.85%), likely due to the frequent use of easy connectives such as temporal 'when' in questions, and 'then' in sequential narration in answers. one of the major motivations of this work is releasing genre-diverse, complete pdtb v3style data to facilitate cross-framework and crossdomain shallow discourse parsing. while, strictly speaking, there is no other dataset annotated like pdtb v3 to evaluate on, other existing datasets do contain a subset of pdtb v3 annotations. for instance, ## conclusion and outlook in this paper, we present gdtb, a pdtb-style dataset covering 16 english spoken and written genres for open-domain shallow discourse parsing, which we create primarily using a cascade of conversion modules leveraging enhanced rst annotations. the data covers all aspects of pdtb v3 annotation, including explicit and implicit inter-/intra-sentential relations as well as alternative lexicalizations, entity relations, and hypophora. we show that rst relations lead to reliable pdtb-style annotations, particularly for explicit relations. using state-of-the-art fine-tuned sequenceto-sequence models and the (e)rst relations as inputs, we are also able to obtain high quality predictions for implicit relations, which we correct in whole for the test set and in part for the remaining data's most unreliably convertible rst relation types (e.g. context-background and joint-other). our experiments show that there is substantial degradation in cross-corpus pdtb-style relation classification in both directions, demonstrating pdtb's current inadequacy for relation classification in open domain settings. we show that jointly training a relation classification system on both pdtb v3 and gdtb leads to much greater crosscorpus stability without sacrificing much performance on pdtb v3. we are therefore confident that gdtb can be a valuable resource for improving out-of-domain performance of pdtb-style english shallow discourse parsing systems. in future work, we believe the same pipeline presented in this paper can be applied to additional corpora annotated with rst in general and erst in particular. specifically, the recent addition of erst annotations to the gentle corpus (genre tests for linguistic evaluation), an extension corpus applying gum's annotation scheme to 8 more unusual english genres @xcite , should allow for more gdtb-like data to be produced with ease. this data would cover the additional gentle genres, which encompass dictionary entries, esports video commentary, legal documents, medical notes, poetry, mathematical proofs, course syllabuses, and threat letters. finally, beyond discourse parsing, we also see great promise in using gdtb for theoretical studies of discourse relation variation across genres, and for the comparison of alignments between pdtbstyle relations and rst or erst annotations. in particular, we believe gdtb and gum-rst will prove to be informative for future comparisons between theoretical frameworks, along the lines proposed by @xcite .