C onn P rompt: Connective-cloze Prompt Learning for Implicit Discourse Relation Recognition (2022)

Abstract: Implicit Discourse Relation Recognition (IDRR) is to detect and classify relation sense between two text segments without an explicit connective. Vanilla pre-train and fine-tuning paradigm builds upon a Pre-trained Language Model (PLM) with a task-specific neural network. However, the task objective functions are often not in accordance with that of the PLM. Furthermore, this paradigm cannot well exploit some linguistic evidence embedded in the pre-training process. The recent pre-train, prompt, and predict paradigm selects appropriate prompts to reformulate downstream tasks, so as to utilizing the PLM itself for prediction. However, for its success applications, prompts, verbalizer as well as model training should still be carefully designed for different tasks. As the first trial of using this new paradigm for IDRR, this paper develops a Connective-cloze Prompt (ConnPrompt) to transform the relation prediction task as a connective-cloze task. Specifically, we design two styles of ConnPrompt template: Insert-cloze Prompt (ICP) and Prefix-cloze Prompt (PCP) and construct an answer space mapping to the relation senses based on the hierarchy sense tags and implicit connectives. Furthermore, we use a multi-prompt ensemble to fuse predictions from different prompting results. Experiments on the PDTB corpus show that our method significantly outperforms the state-of-the-art algorithms, even with fewer training data.

## introduction implicit discourse relation recognition (idrr) aims at detecting and classifying some latent relation in between a pair of text segments (called arguments) without an explicit connective word. as illustrated in fig. 1 , an implicit discourse relation of "contingency" is held between argument-1 and argument-2, and the implicit connective 'so' * corresponding author: bang wang is inserted by annotators in the pdtb corpus. it is of great importance for many downstream natural language processing (nlp) applications, such as question answering @xcite , machine translation evaluation @xcite , information extraction @xcite , sentiment analysis @xcite , and etc. however, due to the absence of an explicit connective word, inferring discourse relations from the contextual semantics of arguments is still a challenging task. existing pre-train and fine-tuning paradigm @xcite builds upon a pre-trained language model (plm) with a well-designed sophisticated neural network to encode the semantic content and interactive evidence of argument pairs @xcite @xcite @xcite @xcite . although the plms are adapted to these task-specific neural networks that can effectively learn a kind of contextual semantics of arguments, they introduce some additional parameters that need to be trained by a large amount of labelled data. moreover, the task objective function is often not in accordance with that of the plm. as such, the plm needs to be fine-tuned for solving downstream tasks, resulting in poor utilization of the encyclopedic linguistic evidence embedded in the pre-training process. on the one hand, we notice that the pre-training process of a plm often uses a kind of cloze task, called masked language model (mlm), to predict a piece of masked text from context. on the other hand, it has been reported that an explicit connective of an argument pair can greatly improve the relation classification performance in the explicit discourse relation recognition task @xcite . although explicit connectives are not available in the idrr task, it is of great interests to explore whether we can transform the relation prediction task as a connective-cloze task, such that we can exploit a pre-trained masked language model to predict a missing (yet possibly latent) connective for implicit relation classification. this is actually in accordance with the philosophy of the prompt learning paradigm, that is, predicting a connective as an answer word in some predefined template and then mapping the answer word to one relation sense. the recent pre-train, prompt, and predict paradigm models the probability of text directly based on plms to perform prediction task. specifically, it selects appropriate prompts to reformulate downstream task, so as to utilizing the plm itself to predict the desired output @xcite . moreover, the prompt paradigm is capable of performing few shot even zero-shot learning, as the plm is sufficiently pre-trained and no external parameters need to be trained. however, for its successful applications in many downstream nlp tasks @xcite @xcite , prompts engineering, verbalizer as well as training strategies should still be carefully designed for different tasks. in this paper, we explore how to transform the idrr task against the prompt learning paradigm. to the best of our knowledge, this is the first paper for such explorations. in this paper, we develop a connective-cloze prompt (connprompt) framework to transform the relation prediction task as a connective-cloze task for the idrr task. specifically, we design two styles of connprompt template: insert-cloze prompt (icp) and prefix-cloze prompt (pcp), in which the [mask] token is added for connective answer prediction. the icp template concatenates two arguments as an entire word sequence, and the @xcite token is inserted in between two arguments; the pcp template uses a @xcite token to mark the boundary of two arguments, and the [mask] token is added at the beginning of argument-1 or argument-2 as a prefix. besides, we construct an answer space mapping an answer word to relation senses according to the hierarchy sense tags and implicit connectives in the training dataset. furthermore, in order to leverage the com-plementary advantages of different prompt templates, we use a multi-prompt ensemble to fuse predictions from different prompting results. we conduct the experiments on the pdtb corpus with four advanced masked language models: bert, roberta, ernie and deberta. experiment results show that our connprompt significantly outperforms the state-of-the-art algorithms with full training data. furthermore, our con-nprompt can also achieve comparable performance even with fewer training data. ## method in this section, we first introduce the overall framework of our connprompt, then explain the details of connective-cloze prompt templates, verbalizer construction, multi-prompt ensembling, and model training strategies. ## experiment settings in this section, we present our experimental settings, including dataset, plms, parameter settings and competitor models. ## conclusion this is the first paper on examining the pretrained, prompt, predict paradigm for the idrr task. we have developed a connective-cloze prompt (connprompt) to transform the idrr task as a connective-cloze prediction task based on a pre-trained language model (plm). two styles of manually designed prompt template: insertion connective prompt and prefix connective prompt, have been designed to convert input argument pairs into the prompt formulation, and a discrete answer space is constructed with sixteen answer words for verbalizer. experiments on the pdtb corpus have validated that our connprompt can significantly outperform the state-of-the-art algorithms, even with fewer training data. this paper has applied the basic techniques of prompt learning for the idrr task. in the last year, the prompt paradigm has achieved some new interesting advances, covering the techniques for choosing pre-trained models, designing continuous prompt templates, constructing answer space as well as training and tuning strategies. we note that some of these new techniques shall also be examined and improved for the idrr task. besides, the excellent performance of the prompt learning in this paper also motivates us to further investigate its applications in other nlp tasks.