A Unified Approach to Discourse Relation Classification in nine Languages (2021)

Abstract: This paper presents efforts to solve the shared task on discourse relation classification (disrpt task 3). The intricate prediction task aims to predict a large number of classes from the Rhetorical Structure Theory (RST) framework for nine target languages. Labels include discourse relations such as background, condition, contrast and elaboration. We present an approach using euclidean distance between sentence embeddings that were extracted using multlingual sentence BERT (sBERT) and directionality as features. The data was combined into five classes which were used for initial prediction. The second classification step predicts the target classes. We observe a substantial difference in results depending on the number of occurrences of the target label in the training data. We achieve the best results on Chinese, where our system achieves 70 % accuracy on 20 labels.

## introduction discourse relations are an integral part of natural language understanding. they provide information on the interaction between aspects of utterances, such as result "i had already seen the movie, so the ending of the book was spoiled!" or contrast "i did not like her first book, but her latest novel is great!". in natural language processing, understanding discourse relations is beneficial for many tasks, e.g. the classification of sentiment @xcite . due to different frameworks and the complexity of the subject, the classification of discourse relations is a challenging task that has been addressed by the shared task on discourse relation classification across formalisms (task 3 of the workshop on discourse relation parsing and treebanking (disrpt)) 1 . the proposed shared task is to our 1 https://sites.google.com/georgetown.edu/disrpt2021 knowledge the first of its kind on discourse relation classification across formalisms. in the past, a number of classification efforts for the data created within the pdtb framework and rst @xcite @xcite , mostly for english have been attempted. this paper focuses on the data based on the rhetorical structure theory (rst) of nine languages: basque, chinese, dutch, english, farsi, german, portuguese, russian and spanish in a cross-lingual approach. ## data the data sets created on the basis of rst were used for the task of discourse relation classification, since the text data for most other data sets was hidden by underscores and we did not have resources to restore it. out of two data sets provided for spanish the rst spanish treebank (spa.rst.rststb) was chosen. due to the lack of the original text data, the rows with units hidden by underscores were removed from the english data set (georgetown university multilayer corpus -eng.rst.gum). this reduced english model was trained for comparison with other models. the rst corpora were analyzed and compared. we found that they have different size, e.g. the chinese data set is the smallest and the russian data set exceeds the size of all the other data sets. the corpora for different languages differ in their number of labels. only the labels background, condition, contrast and elaboration appear in all languages. german english basque farsi dutch portuguese russian spanish chinese unique labels class alternative alternative 0 antithesis antithesis antithesis antithesis antithesis antithesis antithesis antithesis antithesis 0 attribution attribution attribution attribution attribution attribution 4 background background background background background background background backgroun(d) background background 4 cause cause cause cause cause cause cause cause 2 cause-effect cause-effect 2 circumstance circumstance circumstance circumstance circumstance circumstance circumstance circumstance 4 comparison comparison comparison comparison 0 concession concession concession concession concession concession concession concession concession 0 conclusion conclusion conclusion 2 condition condition condition condition condition condition condition condition condition condition 0 conjunction conjunction contrast contrast contrast contrast 0 disjunction disjunction disjunction disjunction disjunction disjunction 3 effect effect 2 e-elaboration e-elaboration 4 elaboration elaboration elaboration elaboration elaboration elaboration elaboration elaboration elaboration elaboration 4 enablement enablement enablement enablement enablement enablement enablement 4 evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation evaluation 1 evaluation-n evaluation-n 1 evaluation-s evaluation-s 1 evidence evidence evidence evidence evidence evidence evidence evidence evidence 4 explanation explanation explanation 4 interpretation interpretation interpretation interpretation interpretation interpretation interpretation 1 interpretation-evaluation interpretation-evaluation 1 joint joint joint joint joint joint joint joint joint 3 justify justify justify justify justify justify justify 1 list list list list list list list 3 manner manner 2 manner-means manner-means 2 means means means means means means means means 2 motivation motibation motivation motivation motivation motivation motivation motivation 2 nonvolitional-cause non-volitional-cause non-volitional-cause 2 non-volitional-cause-e non-volitional-cause-e 2 nonvolitional-result non-volitional-result non-volitional-result 2 non-volitional-result-e non-volitional-result-e 2 otherwise otherwise otherwise otherwise 0 parenthetical parenthetical 4 preparation preparation preparation preparation preparation preparation preparation preparation 4 purpose purpose purpose purpose purpose purpose purpose purpose purpose 2 question question 4 reason reason 2 restatement restatement restatement restatement restatement restatement restatement restatement restatement 4 restatement-mn restatement-mn 4 result result result result result result 2 sequence sequence sequence sequence sequence sequence sequence sequence sequence 3 solutionhood solutionhood solution-hood solutionhood solutionhood solutionhood solutionhood solutionhood solutionhood 4 span span 3 summary summary summary summary summary summary summary 4 temporal temporal 3 topichange topichange 4 topicomment topicomment 4 topidrift topidrift 4 unconditional unconditional unconditional 0 unless unless unless unless 0 volitional-cause volitional-cause volitional-cause 2 volitional-result volitional-result volitional-result 2 the number of labels in each data set is large (between 16 and 29), so that it is quite difficult for any classifier to distinguish between them. inspired by other discourse relation schemes such as pdtb @xcite that suggests hierarchical labels, unique labels were mapped to five large classes (last column in ## method the subject of discourse relation classification are not text units, but rather relations between them. one of the methods for comparing two text units is to calculate euclidean distance between their embeddings. using multilingual sbert (reimers and gurevych, 2020) models, embeddings for two discourse units were obtained foot_0 . the first version of the model created for 15 languages (yang et al., 2020) was used for the corresponding languages in the shared task data set. its second version for a larger number of languages was used for the remaining languages in the data sets, since the second model works slightly worse on the languages from the first model. then euclidean distance between the embeddings of these two units was calculated. the euclidean distance is the first feature for the model and the second is a categorical one taken from the column that indicate the directionality information. initially, we experimented with several embeddings for the german corpus, but the results were similar, including those for the multilingual embeddings. therefore, we used the same multilingual model for several languages for simplicity reasons. using the features mentioned above, a stacked random forest model was trained with one-versusall approach and balanced weights for classes. the model parameters were the same for all the layers of the stacked model. since the number of original classes is very large, the labels were grouped into larger classes as the last column in ## results poorly predicted large classes set restrictions on further label classification. some of the classes have very few instances so that models cannot learn and recognize them properly. in general, the smaller the number of classes and the larger the data set (and the number of instances for each label), the better is the accuracy score. as can be seen in the organizers of the shared task evaluated our model on all data sets, including non-rst data that we did not have access to due to accessibility restrictions. additional labels from the non-rst data sets should be regarded separately from the labels in ## conclusion although the data taken for this shared task is based on the rhetorical structure theory, each language (and corpus) proposes a different set of labels for classification. however, in many cases there are substantial overlaps which makes a unified approach to discourse relation classification possible. this paper suggests such an approach by grouping all unique labels, some of which are shared between languages, to large classes and using a stacked random forest model in each case. however, a different division into larger groups, for example using automatic approaches such as clustering could be beneficial and reduce the amount of manual work. further fine-tuning of the embeddings with the shared task data will potentially improve the results. as for the random forest method, different hyperparameters for different languages can be tried out in additional experiments. we show that our approach achieves good results on labels with a sufficient amount of instances per class across languages. albeit the simplicity of the model and small number features, we achieve 70 % accuracy on chinese labels and at least 35 % for german.