The DISRPT 2023 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification (2023)

Abstract: In 2023, the third iteration of the DISRPT Shared Task (Discourse Relation Parsing and Treebanking) was held, dedicated to the underlying units used in discourse parsing across formalisms. Following the success of the 2019and 2021 tasks on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification, this iteration has added 10 new corpora, including 2 new languages (Thai and Italian) and 3 discourse treebanks annotated in the discourse dependency representation in addition to the previously included frameworks: RST, SDRT, and PDTB. In this paper, we review the data included in the Shared Task, which covers 26 datasets across 13 languages, survey and compare submitted systems, and report on system performance on each task for both annotated and plain-tokenized versions of the data.

## introduction discourse parsing aims to uncover the underlying structure of monologues or dialogues, where spans of texts are linked together by semantic-pragmatic discourse relations such as explanation, con-trast, temporal-asynchronous, or goal. examples of such structures in different representations are given in figures 1 and 2 . several theoretical frameworks have been proposed for discourse analysis and have subsequently been used in many annotation projects. common ones include the rhetorical structure theory (rst, @xcite , where discourse structures are hierarchical constituent trees (figure 1 ), and relation definitions are based on authors' or speakers' intents; the segmented discourse representation theory (sdrt, @xcite , where structures are graphs with non-terminal nodes, and relations are defined using formal logics; and the penn discourse treebank (pdtb, @xcite with relations between isolated pairs of argument spans, possibly marked by a connective (e.g. but, because) which is then annotated a sense label. in addition, building upon several studies proposing to encode discourse structures as dependencies @xcite , it has been proposed to annotate discourse graphs using pure dependency structures, with no nonterminal nodes (figure 2 ), while keeping relations and segmentation rules from rst-which is abbreviated here as the dep framework @xcite . within each framework, numerous corpora of a variety of languages and domains have been annotated. however, the differences between the annotation projects hinder the evaluation of the progress made and to develop systems that should ideally perform well on the broadest possible range of data. @xcite proposed the first iteration of the shared task of discourse relation parsing and 1 treebanking (disrpt) foot_0 in order to broaden the scope of discourse studies by including datasets and inviting researchers from different discourse theories, to facilitate cross-framework studies. the first edition of the disrpt shared task was limited to task 1: discourse segmentationidentifying the elementary discourse units (edus) that may be linked by discourse relations; and task 2: discourse connective detection-identifying specific lexical items, called connectives, that can signal a discourse relation (e.g. while, because, since, as long as etc.). in 2021, for the second edition, @xcite added a third task, task 3: discourse relation classification-identifying a relation label between a pair of attached discourse units. 2 this year, for the third edition, we maintained the three tasks but expanded the benchmark with 10 new corpora, including datasets from the dep framework: in total, 26 corpora were made available across 4 frameworks and 13 languages in a unified format. in the last phase of the shared task, we released 6 surprise datasets including data for a new language (thai), as well as 4 out-of-domain (ood) corpora for which only dev and test partitions were available. three teams participated in the shared task, with one team including half of the organizers of the shared task. overall, two systems were proposed for tasks 1 and 2, and three systems for task 3. two systems are based on fine-tuning transformer masked language model encoders, while the third one relies on a generative transformer model for relation classification. only one team presented results for all tasks and tracks (melodi), and another team @xcite reported results for all tasks but were limited to the treebanked track (i.e. parsed and gold sentence-split data) for tasks 1 and 2. the third team (discoflan) focused on relation classification only. for the treebanked track, melodi ranked first on the edu segmentation task, and hits ranked first on the connective detection task with very similar mean scores. for relation classification, hits ranked first. ## related work automatic discourse analysis is an active domain of research, with increasing interests for the past few years as tools become increasingly capable of handling such tasks, and discourse information can be helpful for many applications, for example for authorship attribution @xcite , fake news or political bias detection @xcite , sentiment analysis @xcite , or for generation, with uses in machine translation @xcite @xcite or summarization @xcite @xcite @xcite @xcite . discourse parsing is the full task of recovering a discourse structure of a document, either constituent trees in the rst-based framework, dependency trees for dep, or graphs for sdrt. performance is still far from perfect for full discourse parsing, and systems are mostly developed for english, monologues, and the newswire domain, using the largest news corpus available, the rst-dt @xcite , with an f1 score of 55.4 at best for full trees @xcite . recent studies also sometimes report results on other datasets, especially on gum (e.g. @xcite ), the largest rst english corpus to date, which is composed of multiple spoken and written genres @xcite . very recently, @xcite showed the lack of generalization of existing sota rst discourse parsers through a series of experiments, with a significant performance drop when applied to unseen genres, and also demonstrated the importance of heterogeneous training data for robust discourse parsing. a few attempts have also been made to develop systems for dialogues, especially using the sdrt stac corpus @xcite with either supervised methods @xcite @xcite or transfer learning strategies given the small size of the dataset (e.g. @xcite . finally, multilingual rst discourse parsing has been the topic of a few work @xcite @xcite involving transfer to tackle data scarcity. @xcite in particular demonstrated that cross-lingual strategies could even help for english, and also that good segmentation is crucial for full discourse parsing, with a loss of up to 8% when using predicted edus. as a matter of fact, an option to better understand the difficulty and low performance of discourse parsing is to examine its constituent subtasks, such as discourse segmentation, but also relation classification, and attachment ('naked' or unlabeled tree building). many studies have been dedicated to these subtasks, with a specific focus on the first two. the aim of the disrpt shared task is precisely to provide benchmarks for these critical steps toward full discourse parsing, to the extent possible in a formalism-neutral way, allowing participants to demonstrate the generalizability of their systems across languages, domains, and frameworks. discourse segmentation in particular has been seen as a solved task with performance as high as 94% on rst-dt as early as over 10 years ago @xcite . however, systems at the time were trained only on english newswires data with gold information about sentence boundaries and morpho-syntactic features. when facing realistic data in other languages and even in english, with systems based on predicted information, performance drops very substantially @xcite . disparities across languages and datasets were later emphasized within the disrpt shared tasks @xcite under realistic settings (with predicted sentence splits), with performance above 95% for some corpora, but scores in the 80s for the spanish sctb (82.5% at best), the chinese sctb (83.3), or the russian rrt (86.2%). the best-performing system in 2019 @xcite used a single multilingual bert @xcite based model for every corpus, while in the second edition of disrpt @xcite , the winning system @xcite achieved the best performance with an accuracy of around 91.5% on average, which relied on varied language models, either mono-or multilingual, as well as hand-crafted features. a loss of about 2% was observed when gold sentence boundaries are not given. for task 3, discourse relation classification is often further decomposed into different types of relations: explicit relations-ones that are triggered by a discourse connective (e.g. while, because), and implicit relations -ones that do not contain a discourse marker. the latter is considered a harder task, since no explicit cues are present, and has thus been studied more extensively (e.g. @xcite @xcite . for explicit relations, the task generally reduces to identifying the connectives, that is deciding whether a token such as 'and' is being used as a discourse marker, and then identifying the rela-tion, with the connective constraining the possible labels (e.g. 'and' can signal expansion or re-sult, but not purpose). connective detection and explicit discourse relation classification have been considered easy tasks, with high performance @xcite , but it was later shown that performance drops drastically on non-news domains, or in languages with small datasets @xcite @xcite . for the first two editions of disrpt, rather high performance was reported for connective detection: between an f1 score of 92-94 for the english and turkish corpora, and an f1 score of 87 for the chinese one, with only a small drop when gold sentence splits are not provided. however, this may be due to the relatively large and homogeneous datasets used in the evaluation. this year's new edition introduces 6 new corpora for task 2, as well as ood datasets for which no training data is available: this has made the task more challenging, with the mean scores now under 80% (see section 5 below for details). we also report scores for implicit vs explicit relation classification for some corpora, which were not available in disrpt 2021, and demonstrate low scores for implicit relations as well when data is scarce. the disrpt shared task is among the very few studies to report scores for both implicit and explicit (also including other types such as altlex markers, see @xcite ) relation classification, thus making it more practical for models to be able to recognize any types of relations. task 3 was introduced in 2021, and the winning system was transformers-based language-specific models for each target language and a set of hand-crafted features: overall the average performance was nevertheless still rather low (61.8%), showing room for substantial improvement. ## tasks and tracks three tasks were proposed for disrpt 2023: while all corpora have data annotated for task 3, note that they are not all relevant for tasks 1 and 2: the shared task also proposes two tracks for tasks 1 and 2: otherwise, we provided predicted annotations done with stanza @xcite . ## participating systems three teams submitted systems in time for participation: overall there were two systems for tasks 1-2 and three systems for task 3. all scores reported below come from our reproduction of these systems. ## conclusion the disrpt 2023 shared task was very challenging, with the addition of datasets from a new framework, in new languages, and 4 ood surprise datasets without training partitions. the submitted systems still demonstrated rather high performance for edu segmentation, with room for improvement for some corpora / languages / domains. however, further research and error analysis are needed to competitors proposed original and attractive strategies to combine corpora due to data scarcity, but the label set explosion is a major obstacle as well as for analyzing the results. we hope that this work will bring new research and discussion in increasing convergence and cohesion of frameworks and annotation projects. we encourage researchers in the field to use the disrpt data as a benchmark to evaluate their systems in the future in order to provide a realistic view of the robustness and generalization ability of their approaches. hyung won chung, le hou, shayne longpre, barret zoph, yi tay, william fedus, yunxuan li, xuezhi wang, mostafa dehghani, siddhartha brahma, albert webson, shixiang shane gu, zhuyun dai, mirac suzgun, xinyun chen, aakanksha chowdhery, alex castro-ros, marie pellat, kevin robinson, dasha valter, sharan narang, gaurav mishra, adams yu, vincent zhao, yanping huang, andrew dai, hongkun yu, slav petrov, ed h. chi, jeff dean, jacob devlin, adam roberts, denny zhou, quoc v. le, and jason wei. 2022. scaling instruction-finetuned language models. alexis conneau, kartikay khandelwal, naman goyal, vishrav chaudhary, guillaume wenzek, francisco guzmán, edouard grave, myle ott, luke zettlemoyer, and veselin stoyanov. 2020. unsupervised cross-lingual representation learning at scale. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 8440-8451, online. association for computational linguistics. iria da cunha, juan-manuel torres-moreno, and gerardo sierra. 2011. on the development of the rst spanish treebank. in proceedings of the 5th linguistic annotation workshop, pages 1-10, portland, oregon, usa. association for computational linguistics. nicolas devatine, philippe muller, and chloé braud. 2022. predicting political orientation in news with latent discourse structure to improve bias understanding. in proceedings of the 3rd workshop on computational approaches to discourse, pages 77-85, gyeongju, republic of korea and online. international conference on computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. yaxin fan, peifeng li, fang kong, and qiaoming zhu. 2022. a distance-aware multi-task framework for conversational discourse parsing. in proceedings of the 29th international conference on computational linguistics, pages 912-921, gyeongju, republic of korea. international committee on computational linguistics. vanessa wei feng. 2015. rst-style discourse parsing and its applications in discourse analysis. ph.d. thesis, university of toronto. elisa ferracane, su wang, and raymond mooney. 2017. leveraging discourse information effectively for authorship attribution. in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pages 584-593, taipei, taiwan. asian federation of natural language processing. michel généreux, iris hendrickx, and amália mendes. 2012. introducing the reference corpus of contemporary portuguese online. in proceedings of the eighth international conference on language resources and evaluation (lrec'12), pages 2237-2244, istanbul, turkey. european language resources association (elra). luke gessler, shabnam behzad, yang janet liu, siyao peng, yilun zhu, and amir zeldes. 2021. dis-codisco at the disrpt2021 shared task: a system for discourse segmentation, classification, and connective detection. in proceedings of the 2nd shared task on discourse relation parsing and treebanking (disrpt 2021), pages 51-62, punta cana, dominican republic. association for computational linguistics. freya hewett and manfred stede. 2022. extractive summarisation for german-language data: a text-level approach with discourse features. in proceedings of the 29th international conference on computational linguistics, pages 756-765, gyeongju, republic of korea. international committee on computational linguistics. tsutomu hirao, yasuhisa yoshida, masaaki nishino, norihito yasuda, and masaaki nagata. 2013. singledocument summarization as a tree knapsack problem. in proceedings of the 2013 conference on empirical methods in natural language processing, pages 1515-1520, seattle, washington, usa. association for computational linguistics. neil houlsby, andrei giurgiu, stanislaw jastrzebski, bruna morrone, quentin de laroussilhe, andrea gesmundo, mona attariyan, and sylvain gelly. 2019. parameter-efficient transfer learning for nlp. in international conference on machine learning, pages 2790-2799. pmlr. zhiheng huang, wei xu, and kai yu. 2015. bidirectional lstm-crf models for sequence tagging. arxiv preprint arxiv:1508.01991. patrick huber and giuseppe carenini. 2020. from sentiment annotations to sentiment prediction through discourse augmentation. in proceedings of the 28th international conference on computational linguistics, pages 185-197, barcelona, spain (online). international committee on computational linguistics. mikel iruskieta, maría jesús aranzabe, arantza diaz de ilarraza, itziar gonzalez-dios, mikel lersundi, and oier lopez de lacalle. 2013. the rst basque treebank: an online search interface to check rhetorical relations. in 4th workshop on rst and discourse studies, pages 40-49, fortaleza, brasil. mikel iruskieta, iria cunha, and maite taboada. 2015. a qualitative comparison method for rhetorical structures: identifying different discourse structures in multilingual corpora. language resources and evaluation, 49(2):263-309. anders johannsen and anders søgaard. 2013. disambiguating explicit discourse connectives without oracles. in proceedings of the sixth international joint conference on natural language processing, pages 997-1001, nagoya, japan. asian federation of natural language processing. shafiq joty, francisco guzmán, lluís màrquez, and preslav nakov. 2017. discourse structure in machine translation evaluation. computational linguistics, 43(4):683-722. morteza kamaladdini ezzabady, philippe muller, and chloé braud. 2021. multi-lingual discourse segmentation and connective identification: melodi at dis-rpt2021. in proceedings of the 2nd shared task on discourse relation parsing and treebanking (dis-rpt 2021), pages 22-32, punta cana, dominican republic. association for computational linguistics. hamid karimi and jiliang tang. 2019. learning hierarchical discourse-level structure for fake news detection. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3432-3442, minneapolis, minnesota. association for computational linguistics. najoung kim, song feng, chulaka gunasekara, and luis lastras. 2020. implicit discourse relation classification: we need to talk about evaluation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 5404-5414, online. association for computational linguistics. naoki kobayashi, tsutomu hirao, hidetaka kamigaito, manabu okumura, and masaaki nagata. 2022. a simple and strong baseline for end-to-end neural rst-style discourse parsing. in findings of the association for computational linguistics: emnlp 2022, pages 6725-6737, abu dhabi, united arab emirates. association for computational linguistics. li liang, zheng zhao, and bonnie webber. 2020. extending implicit discourse relation recognition to the pdtb-3. in proceedings of the first workshop on computational approaches to discourse, pages 135-147, online. association for computational linguistics. yang liu, ivan titov, and mirella lapata. 2019. single document summarization as tree induction. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1745-1755, minneapolis, minnesota. association for computational linguistics. yang janet liu and amir zeldes. 2023. why can't discourse parsing generalize? a thorough investigation of the impact of data diversity. in proceedings of the 17th conference of the european chapter of the association for computational linguistics, pages 3112-3130, dubrovnik, croatia. association for computational linguistics. zhengyuan liu and nancy chen. 2021. improving multi-party dialogue discourse parsing via domain integration. in proceedings of the 2nd workshop on computational approaches to discourse, pages 122-127, punta cana, dominican republic and online. association for computational linguistics. task on discourse relation parsing and treebanking (disrpt 2021), pages 1-12, punta cana, dominican republic. association for computational linguistics. deniz zeyrek and murathan kurfalı. 2017. tdb 1.1: extensions on turkish discourse bank. in proceedings of the 11th linguistic annotation workshop, pages 76-81, valencia, spain. association for computational linguistics. deniz zeyrek, amália mendes, yulia grishina, murathan kurfalı, samuel gibbon, and maciej ogrodniczuk. 2019. ted multilingual discourse bank (ted-mdb): a parallel corpus annotated in the pdtb style. language resources and evaluation, pages 1-27. deniz zeyrek, amália mendes, and murathan kurfalı. 2018. multilingual extension of pdtb-style annotation: the case of ted multilingual discourse bank. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), miyazaki, japan. european language resources association (elra). deniz zeyrek and bonnie webber. 2008. a discourse resource for turkish: annotating discourse connectives in the metu corpus. in proceedings of the 6th workshop on asian language resources. yuping zhou, jill lu, jennifer zhang, and nianwen xue. 2014. chinese discourse treebank 0.5 ldc2014t21.