Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation (2023)

Abstract: Implicit discourse relation recognition (IDRR) remains a challenging task in discourse analysis due to the absence of connectives. Most existing methods utilize one-hot labels as the sole optimization target, ignoring the internal association among connectives. Besides, these approaches spend lots of effort on template construction, negatively affecting the generalization capability. To address these problems,we propose a novel Connective Prediction via Knowledge Distillation (CP-KD) approach to instruct large-scale pre-trained language models (PLMs) mining the latent correlations between connectives and discourse relations, which is meaningful for IDRR. Experimental results on the PDTB 2.0/3.0 and CoNLL2016 datasets show that our method significantly outperforms the state-of-the-art models on coarse-grained and fine-grained discourse relations. Moreover, our approach can be transferred to explicit discourse relation recognition(EDRR) and achieve acceptable performance.

## introduction discourse relation recognition (drr) aims at detecting semantic relations between two arguments (sentences or clauses, they are denoted as arg1 and arg2, respectively). as illustrated in figure 1 , the discourse relation contingency (denoted as sense) is held between arg1 and arg2, and the explicit connective so is drawn from the raw text while the implicit connective because is manually inserted by annotators. drr is significant to many natural language processing (nlp) downstream tasks such as causal reasoning @xcite and question answering @xcite . however, compared with explicit discourse relation recognition (edrr), implicit discourse relation recognition (idrr) is still less accurate and practical due to the lack of connectives, which is a major challenge in current discourse analysis research. public preference is important, product names should match up. arg1 arg2 connective (explicit) it could be a long wait. [ [ ] ] some investors will tiptoe back in. arg1 arg2 connective (implicit) sense: contingency.cause.reason sense: contingency.cause.result the connectives (e.g., because, so, etc.) are critical linguistic cues for identifying discourse relations. on the one hand, with the aid of explicit connectives, a simple frequency-based mapping is sufficient to achieve over 85% classification accuracy on edrr @xcite . on the other hand, human annotators utilized connectives to aid relation annotation in the most popular pdtb benchmark datasets @xcite . for instance, annotators first manually inserted a connective expression, and then determined the abstract relation in consideration of both the implicit connective and argument pairs. therefore, several studies recognize implicit discourse relations by incorporating connective information. several studies incorporate connective information to recognize implicit discourse relations. one method uses the probability distribution of connectives among sense labels in the corpus (asr and demberg, 2020), but this requires a consistent label distribution, which is not always the case. for instance, the connective since is more likely to represent the relation contingency in the training data but temporal in the test data. other methods predict implicit connectives before recognizing relations @xcite , or project connectives and relations into the same latent space and transfer knowledge @xcite . however, these methods perform poorly because of introducing additional parameters that require training with large amounts of labeled data. inspired by @xcite , several studies exploited the advantage of prompt learning @xcite to guide plms to predict connectives between argument pairs and then map them to corresponding discourse relations @xcite . however, this paradigm predicted connectives by fitting the outputs of models to one-hot hard labels, regardless of the internal association among connectives. as we all know, a discourse relation corresponds to multiple connectives, but previous studies only selected one of them as the positive sample, while other connectives with similar meanings under the same sense labels were treated as negative samples. besides, the correlation between connectives and discourse relations utilized in these studies is a direct mapping, which is vulnerable and inaccurate. finally, both of them spend lots of effort on template construction, which negatively affects the generalization capability. to address above-mentioned problems, we propose a novel connective prediction via knowledge distillation (cp-kd) approach for identifying implicit discourse relations. as suggested in @xcite , knowledge distillation is a popular technique for training the student model to emulate the well-informed teacher model. specifically, we first design a knowledgeable teacher model to generate meaningful soft labels that capture more associations among connectives than one-hot hard labels to guide the optimization of the student model. secondly, we add answer hints representing the relations of arguments as input to the teacher model, which exploits the implicit knowledge between connectives and sense labels, rather than using the direct mapping relationships in the previous studies. this approach mitigates issues of connective ambiguity and the possibility of multiple similar connectives mapping to the same discourse relation. finally, we design a simple but effective template matching the pattern of implicit discourse data, and demonstrate that simple templates can achieve acceptable performance as well. in addition, the method we propose alleviates the dependence of prompt learning on templates and has good generalization across different templates. extensive experiments show that our proposed model outperforms prior state-of-the-art systems on the pdtb dataset by around 3%. our contributions are summarized as follows: ## method in this section, we introduce our proposed connective prediction via knowledge distillation (cp-kd) method in detail. we first present the promptguided connective prediction model in section 3.1 and then describe the overall framework of our cp-kd approach in section 3.2. but however because since ... and first ... then 2% 1% 75% 6% ... 2% 1% ... 1% mlmhead classifier roberta model it could be a long wait. [mask] some investors will tiptoe back in. template construction connective prediction answer mapping answer: contingency arg1 arg2 hint (optional) comp. cont. exp. temp. softmax ## conclusion in this paper, we propose a novel connective prediction via knowledge distillation approach for coarsegrained and fine-grained implicit discourse relation recognition. experimental results demonstrate that our method achieves state-of-the-art performance on the pdtb 2.0/3.0 datasets and the conll-2016 shared task. furthermore, our proposed method fully uses the correlation between connectives and sense labels and achieves good generalization on different templates. finally, we experimentally prove that our approach can be transferred from idrr to edrr and still performs well for edrr. we will later explore the applicability of our approach to some chinese discourse relations datasets for coarse-grained and fine-grained drr.