Exploring Soft-Label Training for Implicit Discourse Relation Recognition (2024)

Abstract: This paper proposes a classification model for single label implicit discourse relation recognition trained on soft-label distributions. It follows the PDTB 3.0 framework and it was trained and tested on the DiscoGeM corpus, where it achieves an F1-score of 51.38 on third-level sense classification of implicit discourse relations. We argue that training on soft-label distributions allows the model to better discern between more ambiguous discourse relations.

## introduction the penn discourse treebank (pdtb) framework @xcite defines 36 discourse relation senses organized hierarchically according to three levels of sense granularity @xcite . being able to correctly recognize these discourse relations in a text is of great importance for many downstream nlp tasks. while current explicit discourse relation recognition (edrr) models can already obtain f1-scores of 90.22 @xcite when considering the second-level sense, the task of implicit discourse relation recognition (idrr) remains arguably the hardest task in discourse analysis with state-of-theart models reaching f1-scores of 55.26 @xcite at the second-level sense. the gap in performance between the two tasks stems from the inherently subjective nature of idrr, where even trained expert human annotators find it difficult to agree on the sense annotation of implicit discourse relations @xcite . the difficulty in idrr is evidenced by the interannotator agreement on different corpora. while we do not have access to the inter-annotator agreement of the last version of the pdtb 3.0 corpus @xcite , the agreement at the thirdlevel sense of pdtb 2.0 @xcite was of 80% -which also includes the easier to annotate explicit relations (45.6% of the entire corpus). moreover, 1,075 (4.93%) of the 21,827 implicit discourse relations on the pdtb 3.0 corpus were annotated with two senses since the annotators could not agree on a single sense. this difficulty is also highlighted in the discogem corpus @xcite , where the inter-annotator agreement at the implicit third-level sense was 60%. however, if we allow implicit relations to convey multiple senses depending on the interpretation of the reader, disagreements do not necessarily indicate inaccuracies in labeling @xcite @xcite . in fact, it might be helpful in downstream nlp applications to have a distribution of multiple interpretations for ambiguous texts @xcite . in this paper, we propose a single label implicit discourse relation recognition model trained on soft-label distributions. the model follows the annotation guidelines of pdtb 3.0 @xcite and was trained and tested on the discogem corpus @xcite . we argue that training on soft-label distributions allows the idrr model to better generalize and discern between the possible multiple interpretations of more ambiguous texts. our model reaches an f1-score of 51.38 on third-level sense classification of implicit discourse relations in the discogem corpus (scholman et al., 2022a) while state-of-the-art idrr models @xcite achieve an f1-score of 55.26 on second-level sense classification in the pdtb 3.0 corpus @xcite . ## previous work in recent years, different models have tried to leverage the power of language models either through fine-tuning @xcite or prompt-tuning @xcite to face the challenging task of idrr. so far, these efforts have relied on the prin-ciple that there should be a single sense in the interpretation of implicit discourse relations. however, idrr is an inherently ambiguous task even for expert human annotators @xcite . acknowledging the importance of including sources of ambiguity in human inference in the evaluation of natural language processing tasks led to a recent paradigm shift in discourse annotation. rather than relying on expert annotators to find a single label for each implicit relation, recent annotation efforts @xcite @xcite have crowdsourced this task to multiple workers in order to capture the possible multiple interpretations of more ambiguous relations. the idea that discourse annotation can often be ambiguous is not new @xcite and had already been highlighted by @xcite at the nuclear level of the rst framework @xcite . in their work, @xcite proposed a weighted approach to the annotation of nuclearity in discourse relations following the rst framework where, similarly to the pdtb framework, a consensual annotation is hard to obtain @xcite . ## dataset in this work we used the discogem corpus @xcite to train and test our idrr classification model. the corpus contains 6,505 intersentential implicit discourse relations following the pdtb 3.0 annotation guidelines distributed across three different genres: 2,800 implicit discourse relations in political texts, 3,060 in literary texts and 645 in encyclopedic texts. rather than relying on a few trained annotators to find a sense label for each implicit discourse relation, the discogem corpus crowdsourced the annotation of each relation to multiple participants which allowed to capture a distribution of labels for each relation. participants were asked to insert a discourse connective between the two arguments of each relation and the authors then inferred the associated sense label from the third-level senses in the pdtb 3.0 @xcite . through this method, @xcite were able to collect 65,863 annotations from 199 participants for a total of 6,505 implicit discourse relations. ## classification model similarly to the current state-of-the-art model in idrr @xcite , we based our classification model on the bidirectional roberta-base @xcite language model. we fine-tuned the sequence classification model from hugging face 1 with a single classification layer using a soft cross-entropy loss with a mean reduction over batches to allow training with soft-label distributions and we optimized our model using the adam method @xcite . we then inferred the single label sense of each discourse relation at the evaluation stage from the element with the highest score at the output of the model. all of the 1 https://huggingface.co/docs/transformers/ model_doc/roberta code used in this paper can be found on github foot_0 . ## results and analysis having selected the optimal hyperparameter configuration (@xmath0, @xmath1, γ = 1e -5 and λ = 0), we applied our classification model to the task of idrr under two different settings. in the first setting we considered only the arguments of the discourse relations as input to our model, while in the second setting we also took into consideration their adjacent textual context. in both settings, the model outputs a soft-label distribution over the possible third-level senses in the pdtb 3.0 @xcite , from which the sense with the highest score is selected and evaluated against the respective majority-sense from the discogem corpus @xcite . as indicated in section 3.1, our model is based on the roberta @xcite language model, whose maximum input length size is 512. however, the average length of the input with context is 531 characters, while the average length of the input without context is 245 characters (see figure 1 ). the results in although we did not test our model directly on the pdtb 3.0 corpus @xcite , our results suggest the benefits of training idrr classification models on soft-label distributions. our model obtained an f1-score of 51.38 on a subset of the discogem corpus @xcite , while the current best model in idrr @xcite obtained an f1-score of 55.26 on a subset of the pdtb 3.0 corpus @xcite . in their work, @xcite obtained an accuracy of 41% on a subset of the pdtb 3.0 corpus when training their model on the union of the discogem and the qadiscourse @xcite corpora. ## conclusion in this paper we proposed a single label implicit discourse relation recognition model trained on softlabel distributions from the discogem corpus and evaluated it on single label classification to allow an easier comparison against existing state-of-the-art idrr models. we obtained an f1-score of 51.38 on third-level sense classification of implicit discourse relations on the discogem corpus following the pdtb 3.0 annotation guidelines. our results hint at the possible benefits of training idrr classification models on soft-label distributions to help generalize and discern between possible multiple interpretations of ambiguous texts.