Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition (2022)

Abstract: Due to the absence of connectives, implicit discourse relation recognition (IDRR) is still a challenging and crucial task in discourse analysis. Most of the current work adopted multitask learning to aid IDRR through explicit discourse relation recognition (EDRR) or utilized dependencies between discourse relation labels to constrain model predictions. But these methods still performed poorly on fine-grained IDRR and even utterly misidentified on most of the few-shot discourse relation classes. To address these problems, we propose a novel Prompt-based Connective Prediction (PCP) method for IDRR. Our method instructs large-scale pre-trained models to use knowledge relevant to discourse relation and utilizes the strong correlation between connectives and discourse relation to help the model recognize implicit discourse relations. Experimental results show that our method surpasses the current state-of-the-art model and achieves significant improvements on those fine-grained few-shot discourse relation. Moreover, our approach is able to be transferred to EDRR and obtain acceptable results. Our code is released in https://github.com/zh-i9/PCP-for-IDRR.

## introduction discourse relations recognition (drr) aims to identify the semantic relation between two discourse units (e.g., sentences or clauses, they are denoted as arg1 and arg2 respectively). drr is essential to many natural language processing (nlp) downstream tasks involving more context, such as document-level machine translation @xcite and machine reading comprehension @xcite , and text summarization @xcite . compared with explicit discourse relation recognition (edrr), due to the absence of explicit connectives, implicit discourse relation recognition (idrr) is more challenging and attractive to researchers. figure 1 shows an idrr sample in the penn discourse treebank 2.0 (pdtb 2.0) @xcite . traditional machine learning based methods primarily relied on human-designed and shallow linguistic features @xcite . with the rapid development of deep learning, a large number of methods prefer to utilize deep neural networks to recognize discourse relation between two arguments @xcite @xcite . recently, pre-trained language models (plms), such as bert @xcite and roberta @xcite , play a dominative role in many nlp tasks through contextual representation learning. leveraging the success of plms, extensive research work design a wide variety of post-processing neural networks @xcite to extract the semantic information by fine-tuning them on the task-specific dataset. on the other side, previous work showed the accuracy of explicit discourse relation recognition achieved more than 93% using the discourse connective alone @xcite , which reveals strong correlations between connectives and discourse relations. therefore, more and more studies focus on implicit discourse relation recognition through connective prediction. @xcite applied an auxiliary pre-training task, which utilized the representation of the @xcite token to predict explicit connectives, to bert using domain text containing explicit connectives. since the idea of their method is contradictory with the original next sentence prediction (nsp) task of bert and does not make use of the knowledge that bert had learned through the masked language model (mlm) task. the experimental results showed that their method does not perform well on coarsegrained and fine-grained discourse relations. kurfalı and östling (2021) came up with a pipeline approach, where a connective selected from candidates and two implicit arguments were first integrated as "arg1 connective arg2" sequence. then this integrated sequence was input into an explicit discourse relation classifier trained in explicit data to identify implicit discourse relations in a distant supervision manner. since there are a lot of connective candidates (about 100 in pdtb 2.0), and it requires a large number of model inferences for each sample. obviously, the simple sequence does not make full use of the semantic knowledge embedded in the plms. therefore, how to make better connective prediction becomes a crucial step for implicit discourse relation recognition. inspired by @xcite , we propose a novel prompt-based connective prediction (pcp) method for implicit discourse relation recognition. we exploit the advantage of prompt learning @xcite to bridge the gap between connective prediction in the pre-training and fine-tuning stage and make better use of the knowledge of plms. specifically, we first manually design templates matching natural language patterns, which elicit large-scale pre-trained language models for connective prediction using specific knowledge (in section 3.2). secondly, we select the less ambiguous connectives corresponding to each discourse relation based on the frequency of the connectives in the dataset to implement answer mapping (mapping predicted connectives to discourse relaiton labels) (in section 3.3). experimental results prove that the simple but proper template can outperform the current state-of-the-art (sota) model based on fine-tuning and achieve zero breakthroughs on few-shot fine-grained discourse relation. furthermore, our approach based on connective frequency effectively avoids the trouble of manually selecting answer words and takes full advantage of the relevance between discourse relations and connectives. our contributions are summarized as follows: ## the prompt-based connective prediction approach in this section, we will introduce our prompt-based connective prediction (pcp) approach to recognize implicit discourse relations in detail. figure 2 shows the overall architecture of our model. ## conclusion in this paper, we propose a novel prompt-based connective prediction method for coarse-grained and fine-grained implicit discourse relation recognition. experimental results demonstrate that our method achieves state-of-the-art performance on the pdtb 2.0 dataset and the conll-2016 shared task. furthermore, our proposed method breaks the bottleneck of previous work in the few-shot fine-grained discourse relation of pdtb 2.0. finally, we experimentally prove that our approach can be transferred from idrr to edrr and still performs well for edrr. we will later explore the applicability of our approach to some chinese discourse relations datasets for fine-grained drr.