Feature-augmented model for multilingual discourse relation classification (2024)

Abstract: Discourse relation classification within a multilingual, cross-framework setting is a challenging task, and the best-performing systems so far have relied on monolingual and mono-framework approaches.In this paper, we introduce transformer-based multilingual models, trained jointly over all datasets—thus covering different languages and discourse frameworks. We demonstrate their ability to outperform single-corpus models and to overcome (to some extent) the disparity among corpora, by relying on linguistic features and generic information about the nature of the datasets. We also compare the performance of different multilingual pretrained models, as well as the encoding of the relation direction, a key component for the task. Our results on the 16 datasets of the DISRPT 2021 benchmark show improvements in accuracy in (almost) all datasets compared to the monolingual models, with at best 65.91% in average accuracy, thus corresponding to a 4% improvement over the state-of-the-art.

## introduction discourse relation classification is the process of identifying the semantic-pragmatic relations between clauses or sentences, forming the discourse structure of a document. it is considered a crucial step in building knowledge graphs @xcite and nlp downstream tasks requiring textual coherence and additional context, for example, text generation @xcite or summarization @xcite ), text categorization @xcite , and question answering @xcite . these relations, also called rhetorical relations, may be considered explicit, when the connection is denoted by the presence of distinct words called connectives, or implicit, i.e. relations expressed without a discourse connective. for example, the concession relation between the two arguments is expressed with the connective however in the first example below, while in the second example, the relation manner is implicit. most previous studies focused on implicit discourse relation classification, which is considered a harder task than the prediction of explicit relations. however, our setting requires that the system identifies both explicit and implicit relations simultaneously, a configuration that is more realistic and includes corpora with and without annotations of explicit markers. varied typologies of discourse relations have been presented in the literature and applied to annotate several corpora in different languages. in this paper, we are presenting an approach to address multilingual, multi-framework discourse relation classification. we use as a take-off point the dis-rpt shared task on discourse relation classification across formalisms and its datasets covering various languages and frameworks @xcite , and compare our results to the current stateof-the-art system on the disrpt data, which is composed of monolingual models, discodisco @xcite . our multilingual approach is based on joint training across all available corpora, covering varied languages and discourse frameworks. we conduct experiments over 16 corpora, covering 11 languages and 3 discourse frameworks. we jointly train a classifier with all the datasets of the shared task, and we compare different transformer-based multilingual pretrained models. we extend the featurebased approach proposed by @xcite and @xcite , to investigate its effect within a multilingual, cross-domain setting. each discodisco monolingual model used different features, hence we evaluate which features are more informative in our joint setting. we also enhance our models with features targeted to our multilingual, cross-framework setting. moreover, we test the effect of relation direction to the classification process. we examine two methods of expressing the direction of the relation between two units, either by annotating it with new tokens @xcite or by switching the position to unify it across relations @xcite . we adhere to the use of pretrained models of base size and fine-tune them for the discourse relation classification task. this ensures reproducibility, and shorter training times and computational power required. overall, we observe that xlm-roberta models perform better than bert models and that, contrary to the monolingual models presented in @xcite , for the multilingual, crossframework settings, using all available features is the most beneficial for all models. for the encoding of the relation position, we observe that encoding with additional tokens is more beneficial than switching the argument position, and both approaches are better than none. finally, we report state-of-the-art performance on discourse relation classification with a maximum of 65.91% in average accuracy over all the datasets, thus outperforming previous results by about 4%. the code for fine-tuning the classifiers can be found on gitlab 1 . ## previous work most of the existing literature on discourse relation classification has focused on implicit relations, since explicit ones are considered easier to predict, with already accuracy above 90% with simple models and features @xcite . however, it has been shown that the task can be more difficult for different domains or languages associated with small datasets @xcite @xcite . approaches for implicit relation classification have either made use of linguistic features @xcite or the least ambiguous connectives 1 gitlab.irit.fr/melodi/andiamo/discret_feat as implicit connectives @xcite , or even explicit connectives @xcite . more recently, several approaches have been proposed relying on transformer-based architectures and pre-trained language models, demonstrating their effectiveness for domain transfer @xcite , or for learning effective representation of sentences for the task @xcite , with also attempts relying on additional pre-training of language models @xcite . the disrpt shared tasks were created to motivate research on challenging discourse analysis tasks, within a multilingual, cross-framework setting, by providing unified file formats for multiple discourse datasets. there have been two editions including the task on discourse relation classification @xcite : since not all datasets have annotations distinguishing between explicit and implicit relations, the focus is on predicting simultaneously all types of relations. this makes for a more realistic scenario, where the nature of the relation is not assumed to be known, and it corresponds to the task performed by a discourse parser. in disrpt 2021 @xcite , there were two submitted systems, for 16 datasets and 11 languages. discodisco @xcite is a system based on monolingual and corpus-specific classifiers based on pretrained bert language models. the inputs were enriched with handcrafted features and direction annotations (described in detail in section 3.2). it was the most successful system, with an average 61.82% accuracy. meanwhile, dis-crel @xcite aimed for a hierarchical and multilingual approach. they used sentence-level embeddings made with sentence-bert @xcite and stacked random forest classifiers, to predict coarse-grained relations first and then fine-grained ones. they achieved 54.23% averaged accuracy. in disrpt 2023 @xcite , three systems were submitted. some datasets were updated from 2021, a new framework was added (dep, @xcite , and 10 new datasets and 2 new languages were added, for a total of 26 datasets and 13 languages. hits @xcite was the system with the best performance for 2023. it employed a combination of framework-based, multilingual, and monolingual classifiers, based on large pretrained language models. to enhance performance, they also employed bootstrap aggregat-ing techniques and adversarial training. the average accuracy score was 62.36% overall. when the score is calculated by including only the corpora available in 2021, the average accuracy is 58.18% @xcite , thus a lower score than dis-codisco. 2 in discret, we @xcite created multilingual classifiers trained jointly on all languages and corpora. we used pretrained multilingual bert language models @xcite and adapters @xcite . we also incorporated modifications on the label distribution to reduce the total number of labels across all corpora (see section 3.3); however, there were problems with fully reverting the labels for the evaluation process. the average accuracy was 54.44%. discoflan @xcite used the flan-t5 generative language model @xcite and trained monolingual models. the prompts queried the model for the relation between the two units. they post-process the model's output to match the labels of each corpus label set (see section 3.3). accuracy was 31.2% on average. ## results in overall, our models outperform the state-of-theart system discodisco in several settings, when linguistic features (i.e. "common/all discodisco features") and/or dataset information ("lcf", language, corpus, framework) are used, with at best 65.91% in average accuracy, against 61.82% for discodisco. this demonstrates that single multilingual, cross-framework models are able to leverage correlations between the different datasets, and thus take advantage of a larger amount of data if fed with additional information. for the models most similar to discodisco, i.e. using mbert with annotations of direction ("add. tokens"), our results are very close to theirs: 60.41% vs 59.54% ("no features") and 61.82% vs 62.56% ("common features"). xlm-roberta models performed better than the mbert-based models and also surpassed the lightweight distilm-bert models. they were the ones that steadily surpassed the discodisco system, with 62.09% and 64.86% respectively for the same configurations. moreover, the mbert models also performed better than the discodisco baseline, when they were provided with the lcf tokens, either when limited to "common features" (63.46%), or when using "all features" (63.67%). observing the different sets of features that we used, the addition of any features improves the accuracy of multilingual classification, and the best configuration was, in most cases, the features used by @xcite , with the addition of corpusspecific features. when we used additional tokens to encode the direction of the relation, the most beneficial set of features for all the models was the "common" features, i.e. only the features used by at least one model in the discodisco 2021 system. we notice that the model with the highest accuracy of all is the xlm-roberta model, using this encoding of the direction and the common discodisco and lcf features. however, using all features in this setting leads to very similar results (-0.4%). when the direction is encoded by switching the units, the situation is reversed: results are better when using all the features rather than only the common ones. the addition of the lcf tokens, alongside the discodisco features, showed an increase in accuracy as well. for the xlm-roberta models, the presence of all features was also most beneficial, but not necessarily the presence of the lcf features. looking at individual datasets, our models outperformed the discodisco 2021 system in all but one dataset, the basque eus.rst.ert (by 0.15%, comparing the performance between the two ways of handling the direction of the relation, the direction annotation based on additional tokens was the better option for most datasets when the discodisco features were used. however, for the dutch nld.rst.nldt and portuguese por.rst.cstn datasets, the performance was identical with either setting. observing the effect of the direction handling without the addition of features, we note that, while the method based on additional tokens performed better overall, there were instances where switching the arguments was better (english eng.rst.rstdt, eng.sdrt.stac, spanish spa.rst.rststb), and one dataset for which no change was marginally better (portuguese por.rst.cstn), see tables 4 , 5 and 6 in appendix. ## discussion our multilingual approach outperformed the monolingual approach of discodisco @xcite in all but one dataset, the basque one. our initial assumption was that the use of the multilingual setting would be beneficial since the use of more data is favorable to the models and the instances of less frequent labels would be higher. indeed, for the very small datasets spa.rst.sctb (spanish, 326 train sentences) and zho.rst.sctb (chinese, 361 train sentences), the improvement was elevated with all models. for the largest datasets (eng.pdtb.pdtb, english, 44.5k train sentences; rus.rst.rrt, russian, 19k train sentences; tur.pdtb.tdb, turkish, 25k train sentences) there was also an improvement of 3 -4%. in the case of eus.rst.ert (basque) with 1.6k train sentences, we observed the label distribution; it has 25 unique labels and similar distributions to spa.rst.rststb, spa.rst.sctb, and zho.rst.sctb. we observe the classification report results @xcite for the most successful model in overall, the addition of features, even as simple as additional tokens in the input sequence, improved classification accuracy significantly. in the monolingual setting, it was possible to test different feature sets to configure which was the best, but for the multilingual setting, selecting features is not straightforward, as different corpora contain different annotations (e.g. the gum and stac corpora are the only ones with the speaker information). the small differences in accuracy between using all features and only the ones used for the discodisco 2021 system are produced because, in a multilingual setting with all the datasets used jointly, some features that are informative for some corpora will not be for others, if the annotation does not exist. the addition of the language, framework, and corpus name was also beneficial, in order to annotate the presence of corpus-specific features, even if the information of language is not directly accessible to the model. finally, regarding the relation direction, human intuition is different than the way models process input. the proposal to unify all relation directions by switching the arguments @xcite sounds beneficial in theory, especially when the same relation can be initiated in either unit. however, transformer-based models are not necessarily sensitive to word order; even though positional information is injected in them, some research suggests that they are not sensitive to permutations @xcite . however, other research supports that not all permutations are processed equally @xcite and that the models learn structural information @xcite . it is, therefore, understandable that the presence of additional tokens noting the direction as in @xcite may communicate more information about the relation direction to the models, than switching unit positions. however, there was also a smaller improvement with the unification of the direction; this points to the models either being capable of constructing a rudimentary structure of the two arguments or the models not being completely insensible to word order. ## conclusion in this paper, we reprised the disrpt 2021 shared task on relation classification across formalisms and revisited the most successful model of the last two editions, discodisco @xcite . we adapted discodisco methodologies to multilingual relation classification models, with the addition of techniques and suggestions from other participating teams of the 2023 edition @xcite . we found that xlm-roberta models outperform bert models, in the multilingual setting, especially with the presence of discodisco's handcrafted features. the most successful model was trained only with the features used by discodisco models, as opposed to all features created in the preprocessing stage-but this success was only marginal to the use of all features, and was not true for all architectures. the addition of corpusspecific tokens (language, corpus name, framework) was also beneficial in the multilingual setting. finally, annotating the relation direction with additional tokens was more successful than unifying the position of the two arguments, due to the make of transformer-based models. it should be noted that this information proved crucial and that further studies are needed on this aspect, in particular on the possibility of predicting direction and on the heterogeneity of existing corpora with regard to its encoding. as a future direction, we are considering using our approach on the updated disrpt 2023 benchmark, which includes modified corpora, additional corpora in more languages, and some small validation datasets that allow for testing out-of-domain performance.