Multi-lingual Discourse Segmentation and Connective Identification: MELODI at Disrpt2021 (2021)

Abstract: We present an approach for discourse segmentation and discourse connective identification, both at the sentence and document level, within the Disrpt 2021 shared task, a multi-lingual and multi-formalism evaluation campaign. Building on the most successful architecture from the 2019 similar shared task, we leverage datasets in the same or similar languages to augment training data and improve on the best systems from the previous campaign on 3 out of 4 subtasks, with a mean improvement on all 16 datasets of 0.85%. Within the Disrpt 21 campaign the system ranks 3rd overall, very close to the 2nd system, but with a significant gap with respect to the best system, which uses a rich set of additional features. The system is nonetheless the best on languages that benefited from crosslingual training on sentence internal segmentation (German and Spanish).

## introduction discourse segmentation, the separation of a text or conversation in elementary units that make up the arguments of the rhetorical structure of a text, has long been a neglected step in discourse analysis, considered easy and generally assumed as given in discourse parsing studies, where the focus is to predict the rhetorical structure of a document, a labelled relational structure, with properties dependent on the theorerical framework considered, rhetorical structure theory (rst, @xcite , segmented discourse representation theory (sdrt, @xcite , or the penn discourse treebank (pdtb, @xcite . however this important step has generated more interest recently, as illustrated by the 2019 shared task at the discourse relation parsing and treebanking (disrpt) workshop @xcite . this campaign made available several existing corpora in different language in a common format, expressing the task as a sequence tagging problem, where tokens are to be classified as beginning a segment or not, or, in the case of pdtb corpora, being part of a discourse connective signalling a relation between textual arguments. segmentation in itself has also shown a lot of potential as an auxiliary task in machine translation @xcite and summarization @xcite , independently of discourse parsing. disrpt 2021 shared task reproduces the same setting as disrpt 2019, with some additional data and minor modifications of the original datasets, segmentation as task 1, connective identification as task 2, and adds the prediction of relations between segments, assuming those are known, as task 3. the following shows examples from the english datasets, illustrating respectively task 1 and task 2, with intended units to recover marked between brackets: in the first case, tokens "three" and "and" would be marked as segment beginnings. in the second case, where the target connectives are "but" and "in the end", beginning tokens "but" and "in" would be marked "b" (begin) and "the", "end", would be marked as "i" (inside). in both examples, other tokens would be marked "out". since sentences are almost always discourse units in existing frameworks, segmentation can be seen as two sub-problems: detecting sentences, and detecting intra-sentence segment boundaries. to reflect this, the 2019 shared task introduced two sub-tasks for segmentation and connective identification: either sentence-level segmentation, with sentence boundaries given (when annotated, or provided with a sentence splitter otherwise), also informed by syntactic parsing of sentences, or without any of that information. with the exception of systems presented at dis-rpt 2019 @xcite @xcite , existing work on segmentation always assumed gold sentences, e.g. @xcite . one interesting aspect of such a task is the availability of comparable data in different languages. this has been leveraged in the past for segmentation @xcite , but not in the past 2019 campaign, where the best system relied on fine-tuning a contextual language model for each language separately, albeit using the same multilingual embedding model @xcite . here we propose to build on the previous dis-rpt best system and exploit the availability of multiple corpora for the same language, or the same family of languages (romance, germanic) to augment training of dedicated models. combining this approach with a few adjustments to the base model, we manage to improve on many datasets compare to the previous best disrpt systems, with a mean difference in f1 score of 0.46% and 1.24% on segment boundary detection for sentence and document level respectively, and 3.07% on connective detection for sentences (we didn't improve results at document level for discourse connectives). within the disrpt 21 campaign the system ranks 3rd overall, very close to the 2nd system, but with a significant gap with respect to the best system, which uses a rich set of additional features. our code and instructions to reproduce the experiments are available online. foot_1 2 related work discourse segmentation appeared as an nlp task with the creation of the first annotated rst documents in english, and was primarily rule-based @xcite . since then the literature on discourse parsing generally assumed that elementary discourse units (discourse segments) were given, with only a handful of exceptions @xcite @xcite @xcite , until more recent neural-based work @xcite , still at the sentence level, and always on english or mandarin. only the work of @xcite have considered more varied languages, after the creation of a few different datasets in the past ten years (see the data section below). the disrpt 2019 workshop introduced a more general evaluation framework for discourse segmentation with a shared task considering multilingual data, and segmentation both at the sentence and document level @xcite . the best system @xcite at both granularities (sentence and document) used a sequential tagging model fine-tuned on contextual embeddings. multi-lingual discourse parsing is also becoming more popular, see for instance @xcite , in which it is seen as a form of multi-task learning problem, but this was not applied to discourse segmentation. in other nlp subfields, leveraging availability of corpora in different languages for the same tasks is an active area of research, with different strategies for combining tasks and languages, using meta-learning and complex sampling strategies @xcite . a simpler approach that inspired us here, due to @xcite , is to use the relations between close languages to guide the training process on a task: a generic model is trained on groups of languages, further refined with models by subgroups and finally fine-tuned on individual languages. ## data the 2021 shared task provides 16 corpora annotated either with discourse boundaries (13) or discourse connectives in the case of pdtb corpora (3), with the rst farsi corpus as a surprise dataset. this covers 11 different languages, mostly indoeuropean languages, and with a majority of european languages: 3 romance (spanish, french, portuguese), 3 germanic (english, german, dutch), russian, the only non indo-european being turkish, basque, and mandarin. some of the datasets depend on licences for the underlying text corpus, and are not freely available. we had licences for all of them except the mandarin corpus (zho.pdtb.cdtb), provided by the organizers for the evaluation of the task. except farsi, all the datasets were present in the disrpt 2019 shared task, but the russian all tasks are considered as sequence tagging, and annotated as such: for segmentation, each token is marked as being a segment boudary or not, and for connectives, which can span multiple tokens, the annotation follows the bio convention with three labels begin/inside/out for each token. datasets are not homogenous, as they were annotated along different principles based on three competing theoretical frameworks: experimental results cannot be compared to previous multi-lingual segmentation efforts @xcite , because some of the corpora have been revised (gum, rrt) or not taken entirely (cstn), and some have been added (tdb, prstc), but should be quite close to the disrpt 2019 evaluation, as only the russian and the gum corpora have been extended (and there is an additional dataset). more details can be found about all datasets in the following publications: english rstdt @xcite , pdtb @xcite , sdrt-stac @xcite and gum @xcite , spanish rst (2) @xcite mandarin chinese @xcite , german rst @xcite , french sdrt-annodis @xcite , basque rst @xcite , portuguese @xcite , russian rst @xcite , turkish pdtb @xcite dutch rst @xcite and persian rst @xcite . ## approach in this paper we want to leverage combinations of multiple datasets for training, not only with corpora for the same language and task, but also with languages from the same families. ## conclusion we presented an approach for discourse segmentation and discourse connective identification, both at the sentence and document level, in a multi-lingual and multi-formalism context. building on the successful architecture from the 2019 disrpt shared task, we leverage datasets in the same or similar languages to augment training data and improve on the best systems from the previous campaign, on 3 of the 4 sub-tasks. while below the best system which uses a rich set of features over a similar architecture, we still manage to have the best scores on some of the languages where we experimented with cross-lingual training: german and spanish for sentence-internal segmentation. due to time constraints, we could not fully explore all potentially useful language combinations and fine-tuning on specific datasets, that could help improve on the tasks, and give insights on how different languages help each other addressing the discourse segmentation problem. further progress on multi-lingual embeddings or alignments of different embeddings could be a source of future investigations, as well as more elaborate multi-lingual training procedures.