Generalizing across Languages and Domains for Discourse Relation Classification (2024)

Abstract: The availability of corpora annotated for discourse relations is limited and discourse relation classification performance varies greatly depending on both language and domain. This is a problem for downstream applications that are intended for a language (i.e., not English) or a domain (i.e., not financial news) with comparatively low coverage for discourse annotations. In this paper, we experiment with a state-of-the-art model for discourse relation classification, originally developed for English, extend it to a multi-lingual setting (testing on Italian, Portuguese and Turkish), and employ a simple, yet effective method to mark out-of-domain training instances. By doing so, we aim to contribute to better generalization and more robust discourse relation classification performance across both language and domain.

## introduction interpreting discourse relations is an essential part of understanding a text, and has been shown to be beneficial for many down-stream tasks such as argument mining @xcite , summarization @xcite and relation extraction @xcite . however, it is one of the tasks that are not easily solved by modern prompting methods that obliviate the need for training data @xcite : to date, achieving high performance still relies on high quality annotated data for training (or fine-tuning) a model. however, such discourseannotated data is scarce and expensive to obtain. while relatively large resources exist for english newspaper texts, only small datasets (if any) are available for other languages. additionally, recent work has shown that discourse classification performance can also be severely degraded by moving to a different domain @xcite @xcite . in this paper, we work with the penn discourse treebank (pdtb) framework @xcite and aim to systematically explore ways in which existing english data sources can be leveraged to obtain performant models in other languages and domains. we test two different scenarios: the first one is a setting where no discourse-annotated data is available in a language at all. in this setting, one can either translate the pdtb into that language and project the labels onto the translated text, and then treat the resulting data as a dataset for that new language and fine-tune a model on it. alternatively, one could employ a multi-lingual transformer @xcite @xcite that is trained on english discourse relation classification, and then simply use that model to label data in another language. in this paper we compare both of these settings. the second scenario is one where at least a small amount of discourse-labelled data is available in the target language. for this setting, we investigate the benefit of augmenting the small original corpus with english data (or translated data with projected annotations). we specifically consider the common situation where the test data in the other language is also in a different domain than the english pdtb data from which we aim to leverage annotation. we experiment with three other languages: italian, portuguese and turkish. the selection is motivated by the fact that pdtb-annotated resources are available for these languages, thus allowing us to evaluate performance on originally annotated data and to train on target language annotated data for our low-resource setting. our best-performing set-up improves over stateof-the-art results for all three languages. in an attempt to better understand the characteristics of the different languages and domains represented in our data, we analyze the relation distribution and compare corpus sentence similarities. ## background and related work related work on discourse relation classification typically divides into papers focusing on explicit relations only @xcite or end-to-end approaches to discourse parsing @xcite @xcite on the one hand, and implicit relations only on the other @xcite @xcite . because of the relatively strong cues that explicit relations come with (in the form of discourse connectives), the former typically focuses on feature-engineering or makes use of lexical resources, whereas the latter focuses on neural approaches and methods based on contextualized embeddings. see section 3 for more information on the pdtb and its relation types. in this paper, we adopt the state-of-the-art implicit discourse relation classifier from @xcite . because we want to evaluate this in a multilingual and multi-domain setting, we chose the corpora featured in @xcite . in their shared task however, the relation type for pdtb corpora is not explicitly marked in the training and evaluation data, and the aim is to classify the relation between two arguments, regardless of the relation type (implicit, explicit or any other type). this means that we apply a classifier originally intended for implicit relations, to explicit (and other types of) relations as well. with regard to zero-shot transfer learning for discourse relation classification, our work is inspired by @xcite , who experiment with zero-shot transfer learning for implicit relation classification, by taking the model (intended for english and chinese) from @xcite , training it on english and pooling data from different languages, and subsequently testing it on six other languages (german, lithuanian, polish, portuguese, russian and turkish). since 2019, llms with multi-lingual capabilities have become increasingly available, and we follow up on the work of @xcite by investigating the potential of xlm-roberta-base @xcite for generalisation across languages for discourse relation classification. by using corpora featured in the disrpt shared task series @xcite @xcite , we can directly compare our results to the winning systems for the respective corpora. the submissions to the latest iteration include hits by @xcite , who use xlm-roberta (base and large, depending on the training data size), discret by @xcite , who use the multi-lingual bert base cased model (mbert) @xcite , and discoflan by anuranjana (2023), who uses flan-t5 @xcite . for one of our test corpora (the turkish discourse treebank), the system from @xcite , submitted in 2021, was not beat in 2023. @xcite use a transformer-based neural classifier (a language-specific bert model ( devlin et al., 2019) ) which enhances contextualized word embeddings with hand-crafted features. jiang et al. (2023) improved the prior state-ofthe-art in implicit relation classification for english; the penn discourse treebank version 2 and 3 @xcite , by incorporating the hierarchical structure containing all senses, and the hierarchical sense label sequence corresponding to each instance during classification. we adopt their system architecture, and exchange roberta-base for xlm-roberta-base in most of our experiments. our goal is to classify relations between (pre-segmented) arguments according to the pdtb relation sense hierarchy, which first categorizes relations into four top levels (comparison, contingency, expansion and temporal), and further categorizes them into more detailed second-level senses. although the pdtb sense hierarchy actually specifies more unique second-level senses foot_2 , most previous work renders second-level classification an 11-way classification problem, as that is the number of unique senses in the corresponding annotated corpora. the pdtb sense hierarchy includes a third level, but like most related work, we report classification performance on the first and second levels only. we follow the approach of @xcite , by adopting their train/test split and predicting one label for one input sequence (we refer to @xcite for a detailed discussion on evaluation). the scores for 4-way accuracy and f 1 in section 5 thus correspond to classification at the top level of the hierarchy, and the scores for 11-way accuracy and f 1 correspond to distinguishing between the eleven most frequent classes at the second level of the hierarchy. 2023 ) to format the original pdtb data. this results in a train, dev and test split, all with implicit relations only (14,751 in total, see translated discourse data to augment the training data available for other languages (whose corpora are much smaller than the pdtb), we translate the training, development and test sets of the english pdtb into italian, portuguese and turkish using the google translate api foot_4 . in the following sections, the train, dev and test split from @xcite are referred to as pdtb2, while their machine-translated versions are referred to as pdtbit, pdtb-pt and pdtb-tr. italian discourse corpus: luna the luna corpus contains "italian spontaneous speech recorded in the help-desk facility of the consortium for information systems of piedmont regio" @xcite , thus represents italian, and transcribed (but originally spoken) it helpdesk dialogs. luna contains 1,188 relation instances in total (train, dev and test). portuguese discourse corpus: crpc the crpc corpus from @xcite contains a written subset of the reference corpus of contemporary portuguese, which in turn aims to serve as a representative sample for the portuguese language and contains texts from many sources (literature, newspapers, magazines, science, economics, law, parliamentary debates, technical and didactic texts, pamphlets) @xcite . crpc contains 6,274 relation instances in total. turkish discourse corpus: tdb the turkish discourse bank (tdb) corpus @xcite contains written turkish texts from a variety of genres (novels, stories, research surveys, travel and news articles, interviews and memoirs). tdb contains 1,809 relation instances in total. the combination of authentic and synthetic (i.e., machine-translated) corpora enables us to experiment with different set-ups, using in-domain, outof-domain, in-language and out-of-language configurations for training and test sets, to see how well the model generalizes across the different dimensions. statistics of our data sets are included in ## results the following subsections present the results for different base models and different configurations of training and test data. ## discussion overall, we obtain the best results by combining data from different domains, and marking the outof-domain instances at training time. for italian and portuguese, using in-language training data yields better results, whereas for turkish, combining english out-of-domain data with turkish in-domain data yields better results. according to @xcite , the training data for xlm-roberta-base included 20.9 gib of turkish, compared to 30.2 gib for italian, 49.1 gib for portuguese, and 300.8 gib for english. this could explain the better performance when using original, english data, since the model has seen comparatively few turkish at pre-training. however, since the difference between the amount of training data in turkish, italian and portuguese is not that large, we consider more research necessary to draw conclusions on this. for italian and portuguese, it seems that the automatically obtained, synthetic data is good enough to improve classification performance for discourse relation classification when testing on authentic data. the performance difference for the three corpora overall are rather large, but differences in label distributions figure 1 displays the distribution of top-level senses for the four corpora used in our experiments. 8 from this, we can see that luna has a more balanced distribution than the others, possibly explaining its comparatively high scores (for our bestperforming set-up), taking into account that it is by far the smallest corpus. note that pdtb2 has a fairly imbalanced distribution, with a large proportion of expansion relations and relatively few temporal relations. this distribution can be partially attributed to specificities of the newspaper genre, and partially to the fact that for the pdtb2 corpus, we are only working with implicit relations. temporal relations are often expressed explicitly, which may contribute to their low rate in pdtb2. we also note that crpc is most similar to pdtb2 in the context of the number of implicit relations in the dataset, as it has the highest implicit ratio of the three non-english corpora: turning to second-level senses in figure 2 , we see that crpc and tdb have slightly fewer unique senses (6 for both) than the other two (9 for luna, 11 for pdtb2). while having fewer classes often results in higher scores for multi-class classification, this does not seem to be very predictive for our task, as crpc has a relatively high f 1 -score (77.90), but tdb (with only 6 unique senses) scores 53.92, while luna scores 57.13 (with 9 unique senses). in this respect, it is important to point out that there might be corpus-specific biases: for all three systems submitted to the 2023 disrpt shared task, crpc relation classification performs significantly above the corresponding mean of the system, and two of the three systems shows second-best results on this corpus (after the thai corpus) @xcite 5 ). this might indicate that the crpc corpus contains particularly easy relations, and we consider an investigation of what easy means in this context and important direction for future work. our results also showed that including pdtb2 data was detrimental to performance for 11-way classification when no domain flagging is used, and we speculated that this could be due to strong differences in the distributions of second-level senses. in figure 2 , we can indeed observe that in the expansion class, pdtb2 has many more instantiation and restatement relations, and fewer conjunctions than the other corpora. domain differences between corpora in an attempt to assess to what extent the corpora used in our experiments differ with respect to the actual words and phrases used, we include figure 3. this is the result of a pair-wise compari-figure 3: corpus similarity heatmap. ## conclusion in this paper, we adopt a state-of-the-art implicit discourse relation classification model developed for english, and apply it to both implicit and nonimplicit discourse relations from three corpora that differ in language and domain: an italian corpus of transcribed it helpdesk dialogs, a multi-domain portuguese corpus and a multi-domain turkish corpus. by experimenting with different configurations of in-domain, out-of-domain, in-language and out-of-language training data, we explore to what extent the model generalizes across languages and domains. we also demonstrate the importance of using a flag to mark out-of-domain data at training time. overall, our setup improves over prior work by just under 7 points for italian, over 5 points for portuguese, and over 9 points for turkish (all based on 4-way classification accuracy scores). our code is published on github. 10 we attempt to link the classification results to the number of training samples, label distribution and language usage. we find that the number of training samples or sentence similarity between training and test domain is not very indicative of performance, and that instead the label distribution is likely to be a more reliable indicator. in future work, we plan to delve deeper into specific label distributions of the different domains, and potentially continue this line of work by not just looking at different domains, but by also including and testing on annotated data using wholly different label sets (e.g., rhetorical structure theory @xcite . in order to maximize comparability to related work, we tested on implicit relations only in pdtb set-ups, while we combine different relation types in the other set-ups. an interesting direction of future work would be to more systematically investigate the significance of the strict distinction between explicit and implicit relations (as it is often found in the literature), given current, state-of-theart models for discourse relation classification.