The DISRPT 2021 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification (2021)

Abstract: In 2021, we organized the second iteration of a shared task dedicated to the underlying units used in discourse parsing across formalisms: the DISRPT Shared Task (Discourse Relation Parsing and Treebanking). Adding to the 2019 tasks on Elementary Discourse Unit Segmentation and Connective Detection, this iteration of the Shared Task included for the first time a track on discourse relation classification across three formalisms: RST, SDRT, and PDTB. In this paper we review the data included in the Shared Task, which covers nearly 3 million manually annotated tokens from 16 datasets in 11 languages, survey and compare submitted systems and report on system performance on each task for both annotated and plain-tokenized versions of the data.

## introduction building on rapid progress in nlp for discourse parsing in the past decade (e.g. @xcite @xcite @xcite @xcite , the past two years since the disrpt 2019 shared task @xcite have seen unprecedented performance on benchmark datasets for discourse parsing (e.g. @xcite @xcite . following the 2019 shared task, our focus in 2021 remained on the most established frameworks for discourse relation treebanking: rhetorical structure theory @xcite , the penn discourse treebank's framework @xcite , and segmented discourse representation theory @xcite . with the progress achieved on discourse segmentation and connective detection since 2019, this year we decided to extend the competition to a new * discourse relation parsing and treebanking (disrpt 2021) was held in conjunction with codi at emnlp 2021 in the dominican republic and online ( https://sites. google.com/georgetown.edu/disrpt2021 ). in addition to the new relation classification task, we also updated and expanded the two tasks from 2019 and added a new surprise dataset (and language) which became available in the interim: the persian rst corpus @xcite . 1 while it remains impossible to place some data openly online due to licensing constraints, all annotations, as well as text for open access datasets, is available from the shared task repository, foot_1 as well as scripts for reconstructing the underlying text for proprietary datasets. in the next section, we will describe the tasks for this year, starting with the new task of relation classification across frameworks, followed by the existing tasks, followed by section 3, which provides an overview of the shared task data. section 4 briefly discusses our effort addressing the ethical and transparency issues related to nlp shared task, inspired by @xcite . section 5 then describes the shared task system submissions and results, followed by a discussion and conclusion. ## ethical considerations and transparency in disrpt 2021 in the interest of transparency and with inspiration from escartín et al. (2021), we made efforts to address several ethical and transparency matters in the disprt 2021 shared task. we created a faq page foot_2 on our main website that laid out information about the shared task, participation, equity, and the evaluation criteria etc., which are briefly summarized below. participation & organization generally speaking, we allowed anyone to participate, including the organizers. however, in the interest of transparency and fairness, results from teams which overlap some of the organizers or annotators of any of the included datasets are denoted as such. though as escartín et al. (2021) pointed out, there is an understanding that organisers have privileged access to test sets, we pledge that the test data is under strict quarantine until its release and strictly follow this practice. all systems were always rerun and reproduced by evaluators who do not overlap the submission team, and all datasets' original publications had already been made independently of the shared task. there are always scenarios where there is overlap between the shared task organizers, dataset creators, and participants, and excluding organizer teams may discourage people from organizing future shared tasks, while unfairly penalizing students of organizers who are likely to form the next generation of researchers in the field. ## results for edu segmentation and connective detection, we report precision, recall, and f1 scores for systems in the two tasks, each consisting of two scenarios: edu segmentation and connective detec ## analysis of systems edu segmentation and connective detection all four systems tackling the segmentation tasks addressed both edu segmentation and connective detection with a unified architecture, with the partial exception of discodisco, which varied in using a crf layer for connective detection (in order to model b-and i-label sequence probabilities), but not for edu segmentation, where a crf layer was found to be harmful. of the four systems, the three highest scoring ones relied on a transformer-based neural sequence labeling approach backed by large, pretrained language models, while the fourth system, tmvm, focused on grammatical features extracted from gold and predicted trees in order to capture clause boundaries across languages. tmvm's submission is also unique in reporting scores when training on each individual dataset and predicting on other datasets using those features, many of which are general across languages. of the datasets cross-tested by tmvm, training on a different dataset was beneficial for one english case (training on gum and testing on rst-dt, leading to a 5 point gain from 75 to 80), and the two small spanish and chinese datasets (a 3 point gain from 66 to 69 for spanish when training on the larger spanish rst stb, and a more modest but surprising 2 point gain from 72 to 74 when training on basque and testing on chinese, see @xcite . the latter two cases are probably owing to the small size of the target corpora. the discut paper takes a related approach and trains on multiple corpora from the same language or language family, but does not obtain superior results (see @xcite , possibly because the single-corpus training results are already so strong. among the transformer-based systems, although discodisco performs best overall, other systems score highest on particular datasets in different scenarios, with segformers attaining a new sota score on rst-dt from gold trees (97.09) next to discodisco's new sota score on the same data without gold trees (96.35 in the plain text scenario), and discut showing impressively strong performance on german (95.61 from gold trees, and an even better 95.63 in the plain text scenario). overall best scores for edu segmentation in the top 3 systems are within a point of each other, but the consistent slight lead of discodisco may be due to the fact that it was the only segmentation system using per-language transformers and making use of grammatical features, which in the plain text scenario were induced using a sota transformerbased parser and sentence splitter as well. for connective detection, discodisco's lead becomes slightly larger, about 3 points above seg-formers and almost 7 points above discut, perhaps due to a combination of features (which give a minor boost of just a few points across datasets in their paper, see @xcite , the use of a crf layer and the combination of both contextualized and static word embeddings as well as character embeddings, which provided a variety of views on the data and allowed for both fine tuning and fixed generalization in the test set. choice of language models is also an important factor, and here again discut and segformers rely on multilingual lms, while discodisco uses individual per-language models, which may also be a reason for higher scores on chinese and especially turkish, which may be less well-represented by multilingual lms. finally we note that in the plain text scenario, all systems except for discodisco used noncontextualized tools for sentence splitting and/or automatic parsing (segformers: corenlp; discut: stanza; tmvm: spacy); discodisco used the tranformer-based sentence splitter from the amal-gum corpus @xcite and diaparser @xcite , both with language-specific transformers, with the result that plain text numbers are very close to gold numbers for discodisco (and in fact insignificantly better for plain connective detection: 91.49 on average vs. 91.22 for gold treebanked data). this echoes results from the 2019 task (see @xcite ) which showed the crucial importance of high quality preprocessing in general, and sentence splitting in particular. relation classification although both systems tackling the relation classification task use word embeddings, their approaches are rather different: while the best system, discodisco, relies on a similar approach to their segmentation task entry, which encodes each sequence using a transformer, the discrel system uses whole sentence embeddings from the sentence-transformers library (sbert, reimers and gurevych 2020) to compute euclidean distance between discourse units, as part of a two-level feature-based random forest classifier. both systems also use the provided discourse relation direction, which is meant to approach the case of labelling an unlabelled discourse dependency graph. in terms of encoding features, discodisco uses a sentence pair classifier architecture feeding the underlying transformer embeddings in the form of the bert classification ([cls]) token embeddings, while injecting categorical features at two levels: using pseudo tokens, encoded as usual by the transformer, which indicate the relations direction; and using an inserted pseudo token next to the cls token, which encodes all sequence-level categorical features, such as genre, gold speaker information and more (see @xcite ). these features are then read and trained on by the transformer block alongside the word embeddings. the discrel system takes a two level approach to relation classification, using two stacked oneversus-rest random forest classifiers, which first attempt to distinguish the coarse class of discourse relation (for example collapsing relations such as evaluation, interpretation, justify), and then sub-classify the coarse classes into the final target labels using the second classifier (see varachkina and pannach 2021). for large datasets, the first classifier is trained on the train set, while the second is trained using the development data, but for smaller datasets, training data is duplicated along with the development set for the second classifier. discrel's approach performs substantially better for both chinese datasets, the small spanish dataset, and farsi. a commonality in this set is its inclusion of the smallest datasets (the parallel zho.rst.sctb and spa.rst.sctb), which is perhaps unsurprising given the robustness of tree-based classifiers on small data. however, the gains on the farsi data, which is not very small, are also substantial, and the system scores over 2 points better than dis-codisco on cdtb, suggesting either that sbert has better embeddings for these languages, or in the case of cdtb, possibly that the small number of target relations (only 9, the smallest relation set) benefits from the stacked classifier approach. for the remaining 12 datasets, discodisco scores highest, in many cases by a wide margin of over 10 points above discrel (including all english corpora, with +14 on pdtb, +18 on gum and +12 on rst-dt, and +11 on stac), leading to a final mean gain of over +7 points. feature ablations in the discodisco paper suggest that categorical features confer a minor boost in scores, especially for datasets rich in gold metadata and annotations, but that the primary driver of accuracy is the use of contextualized word embeddings. we also note that, as with segmentation, discodisco uses languagespecific lms for each dataset, and not always the same lms as for segmentation, while discrel uses one of two multilingual sbert models for all datasets. for choice of lms, discodisco's paper indicates that transformers pretrained with the next sentence prediction task (nsp) are particularly suited to relation classification, i.e. models such as bert @xcite , but not the roberta family of models @xcite . the gaps between the systems given the reproduction and multiple run averages suggest that the two approaches capture some orthogonal information, indicating that incorporating insights from both systems may be worth pursuing: sentence embeddings and euclidean distance could be incorporated in a neural system, and the features used by discodisco could be used in a two stage approach such as the one taken by discrel. ## conclusion we conducted the second disrpt shared task for discourse relation parsing and treebanking across frameworks, resulting in a number of new state of the art scores on benchmark datasets, as well as the publication of a new multilingual benchmark for discourse parsing sub-tasks. with the introduction of the new classification task in 2021, the disrpt data now covers discourse unit segmentation, connective detection, and relation classification for unit pairs in 16 datasets, coming from 11 languages and containing approx. 3,000,000 tokens. with the exception of the distinction between connective detection and edu segmentation datasets, we were able to bring all datasets to a common denominator by relying on a dependency representation of the underlying discourse annotations. especially in the case of relation classification, this means that we can potentially harness the shared task data for training and evaluation of discourse relation information using multiple datasets for the same language (where available), and learn from multiple frameworks at the same time. publishing uniformly evaluated results using a single scorer, input format, and feature set, with a stable reference gold parse for each dataset, we hope to facilitate the conversation and collaboration between researchers coming from different frameworks, such as rst, sdrt, and pdtb. we are optimistic that the future for computational work on discourse relations is bright, with scores on segmentation and connective detection now looking much higher than in previous years (finally around the low 90s), and relation classification slowly catching up, but still offering a substantial challenge with scores in the low 60s. with some luck, we may soon see automatic discourse analysis methods become close to, or even as reliable, as morphosyntactic tagging and parsing have become in recent years. in order to realize that goal, we are certain that we will need new datasets covering previously unexplored languages and genres, as well as in depth analysis of the kinds of discourse relations we should recognize and the challenges in inducing them automatically and reliably across all types of text.