Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations (2022)

Abstract: Implicit discourse relation recognition is a challenging task that involves identifying the sense or senses that hold between two adjacent spans of text, in the absense of an explicit connective between them. In both PDTB-2 (prasad et al., 2008) and PDTB-3 (Webber et al., 2019), discourse relational senses are organized into a three-level hierarchy ranging from four broad top-level senses, to more specific senses below them. Most previous work on implicitf discourse relation recognition have used the sense hierarchy simply to indicate what sense labels were available. Here we do more — incorporating the sense hierarchy into the recognition process itself and using it to select the negative examples used in contrastive learning. With no additional effort, the approach achieves state-of-the-art performance on the task. Our code is released inhttps://github.com/wanqiulong 0923/Contrastive_IDRR.

## introduction discourse relations are an important aspect of textual coherence. in some cases, a speaker or writer signals the sense or senses that hold between clauses and/or sentences in a text using an explicit connective. recognizing the sense or senses that hold can be more difficult, in the absense of an explicit connective. automatically identifying the sense or senses that hold between sentences and/or clauses can be useful for downstream nlp tasks such as text summarization @xcite , machine translation @xcite and event relation extraction @xcite . recent studies on implicit discourse relation recognition have shown great success. especially, pre-trained neural language models @xcite @xcite have been used and dramatically improved the performances of models @xcite @xcite . the senses available for labelling recently, @xcite have tried to leverage the dependence between the level-1 and level-2 labels (cf. section 2). the current work goes further, using the whole three-level sense hierarchy to select the negative examples for contrastive learning. contrastive learning, which aims to minimize the distance between similar instances (defined as positive examples) and widen the difference with dissimilar instances (negative examples), has been considered as effective in constructing meaningful representations @xcite @xcite . previous work on contrastive learning indicates that it is critical to select good negative samples @xcite @xcite . the insight underlying the current work is that the hierarchy of sense labels can enable the selection of good negative examples for contrastive learning. to see this, consider examples 1-3 below from the pdtb-3. on the surface, they look somewhat similar, but in examples 1 and 2, the annotators took the second sentence (arg2) as providing more detail about the first sentence ( arg1 (1) "valley national ""isn't out of the woods yet ". the key will be whether arizona real estate turns around or at least stabilizes.. a key is whether house republicans are willing to acquiesce to their senate colleagues' decision to drop many pet provisions.. in this work, we use a multi-task learning framework, which consists of classification tasks and a contrastive learning task. unlike most previous work using one benchmark dataset (usually pdtb-2 or pdtb-3), we evaluate our systems on both pdtb-2 and pdtb-3. besides, @xcite have shown that data augmentation can make representations be more robust, thereby enriching the data used in training. we thus follow @xcite and @xcite in identifying a relevant form of data augmentation for our contrastive learning approach to implicit relation recognition. the main contributions of our work are as follows: implicit discourse relation recognition for this task, dai and huang (2018) considered paragraphlevel context and inter-paragraph dependency. recently, @xcite showed that using the bidirectional encoder representation from bert @xcite is more accurately to recognize temporal.synchrony, comparison.contrast, expansion.conjunction and expansion.alternative. @xcite showed that different levels of representation learning are all important to implicit relation recognition, and they combined three modules to better integrate context information, the interaction between two arguments and to understand the text in depth. however, only two existing works leveraged the hierarchy in implicit relation recognition. @xcite and @xcite first attempted to assign a level-1 sense that holds between arguments, and then only considered as possible level-2 senses, those that are daughters of the level-1 sense. contrastive learning recently, there has been a growing interest in applying contrastive learning in both the pre-training and fine-tuning objectives of pre-trained language models. @xcite used a contrastive objective to fine-tune pretrained language models to obtain sentence embeddings, and greatly improves state-of-the-art sentence embeddings on semantic textual similarity tasks. @xcite proposed label-aware contrastive loss in the presence of larger number and/or more confusable classes, and helps models to produce more differentiated output distributions. besides, many works have demonstrated that selecting good negative examples are very important for using contrastive learning @xcite @xcite . in our work, we integrate contrastive learning loss with supervised losses and we use the structure of the sense hierarchy to guide the selection of negative 10705 examples. ## our approach figure 2 shows the overall architecture of our method. as figure 2 illustrates, we firstly use a simple multi-task model based on roberta-base @xcite , and then we develop a contrastive learning algorithm where the sense hierarchy is used to select positive and negative examples. detailed descriptions of our framework and our data augmentation method are given below. ## results and analysis the results on pdtb-2 and pdtb-3 for level-1 and level-2 are presented in these results demonstrate better performance than previous systems for both level-1 and level-2 classification on both pdtb-2 and pdtb-3. in particular, the results clearly demonstrate benefits to be gained from contrastive learning. but there is more to be said: in section 6.1, we discuss different ways of defining negative examples with respect to the sense hierarchy, and in section 6.2, we discuss the relative value of the particular form of data augmentation we have used (cf. section 4.2) as compared with our method of contrastive learning. ## conclusions in this paper, we leverage the sense hierarchy to select the negative examples needed for contrastive learning for the task of implicit discourse relation recognition. our method has better overall performance than achieved by previous systems, and compared with previous work, our method is better at learning minority labels. moreover, we compared different methods of selecting the negative examples based on the hierarchical structures, which shows some potential negative impacts might be produced when negative examples include those from other level-1 types. moreover, we conduct ablation studies to investigate the effects of our data augmentation method and our contrastive learning method. besides, the limitations and the future work are discussed. huibin ruan, yu hong, yang xu, zhen huang, guodong zhou, and min zhang. 2020. interactivelypropagative attention learning for implicit discourse relation recognition. in proceedings of the 28th international conference on computational linguistics, pages 3168-3178, barcelona, spain (online). international committee on computational linguistics. attapol rutherford and nianwen xue. 2015. improving the inference of implicit discourse relations via classifying explicit discourse connectives. in proceedings of the 2015 conference of the north american chapter of the association for computational