TEP rompt: Task Enlightenment Prompt Learning for Implicit Discourse Relation Recognition (2023)

Abstract: Implicit Discourse Relation Recognition (IDRR) aims at classifying the relation sense between two arguments without an explicit connective. Recently, the ConnPrompt (Xiang et al., 2022) has leveraged the powerful prompt learning for IDRR based on the fusion of multi-prompt decisions from three different yet much similar connective prediction templates. Instead of multi-prompt ensembling, we propose to design auxiliary tasks with enlightened prompt learning for the IDRR task. Although an auxiliary task is not used to directly output final prediction, we argue that during the joint training some of its learned features can be useful to boost the main task. In light of such motivations, we propose a task enlightenment prompt learning model, called TEPrompt, to fuse learned features from three related tasks for IDRR. In particular, the TEPrompt contains three tasks, viz., Discourse Relation Recognition (DRR), Sense Semantics Classification (SSC) and Annotated Connective Prediction (ACP), each with a unique prompt template and an answer space. In the training phase, we jointly train three prompt learning tasks with shared argument representation. In the testing phase, we only take the DRR output with fused features as the final IDRR decision. Experiments with the same conditions have shown that the proposed TEPrompt outperforms the ConnPrompt. This can be attributed to the promoted decision features and language models benefited from joint-training of auxiliary tasks.

## introduction implicit discourse relation recognition (idrr) is to detect and classify some latent relation in between a pair of text segments (called arguments) without an explicit connective @xcite . fig. 1 illustrates an argument pair example with a contingency relation in the penn discourse treebank (pdtb) corpus, and the implicit connective 'so' is inserted by annotators. idrr is of great importance for many downstream natural language processing (nlp) applications, such as question answering @xcite , machine translation @xcite , summarization (huang and kurohashi, 2021), and etc. however, due to the absence of an explicit connective, inferring discourse relations from the contextual semantics of arguments is still a challenging task. conventional pre-train and fine-tuning paradigm @xcite designs sophisticated neural networks to encode the representation of argument pairs upon a pre-trained language model (plm) for relation classification @xcite @xcite @xcite ). on the one hand, these task-specific neural networks introduce some additional parameters that need to be trained by a large amount of labelled data. on the other hand, the task objective function is often not in accordance with that of the plm, so that the plm needs to be fine-tuned for solving downstream tasks, resulting in poor utilization of the encyclopedic linguistic knowledge embedded in the pre-training process. the recent connprompt model @xcite has successfully applied the pre-train, prompt, and predict paradigm, i.e. the so-called prompt learning, in the idrr task by transforming the idrr as a connective-cloze task to predict an answer word and map it to a relation sense. the connprompt has achieved the new state-of-the-art performance on the commonly used pdtb corpus @xcite , however it designs three different yet much similar connective prediction templates which inserts the @xcite token in between two arguments or at the beginning of one argument for answer prediction. moreover, to fuse different prompt predictions, the connprompt employs a simple majority voting decision fusing as for final relation sense prediction. instead of simple multi-prompt ensemble, we argue that some auxiliary prompt tasks can be designed to enlighten the main prompt task with promoted decision features. for example, as the top relation labels in the pdtb corpus are those plain vocabulary words, we can design an auxiliary task to directly predict such label words from the plm vocabulary. furthermore, as the pdtb corpus also contains manually annotated implicit connectives, we can design another auxiliary task to directly predict an annotated connective. although such auxiliary tasks are not necessarily used to output the final idrr prediction, they can be jointly trained with the main task on a shared plm, by which some features learned from the auxiliary tasks can be fused into the main task to promote its decision features for the final prediction. motivated from such considerations, we propose a task enlightenment prompt learning (teprompt) model, where the main idrr task can be enlightened from some auxiliary prompt tasks in terms of its promoted decision features via fusing auxiliary task features. specifically, the teprompt contains a main prompt task: discourse relation recognition (drr), and two auxiliary prompt tasks: sense semantics classification (ssc) and annotated connective prediction (acp). we design each prompt task with a unique template and an answer space. we concatenate three prompt templates as an entire word sequence with two newly added special tokens [arg 1 ] and [arg 2 ] for shared argument representation, as the input of a plm. in the training phase, we jointly train three prompt tasks upon one plm model but with three different answer predictions as objective functions. in the testing phase, we only take the main prompt decision features yet promoted by fusing the features from the two auxiliary prompts to output the final idrr decision. experiment results have shown that our proposed teprompt outperforms the connprompt with the same conditions and achieves the new state-of-the-art performance on the latest pdtb 3.0 corpus. ## the proposed teprompt model fig. 2 presents our teprompt model, including three modules of prompt templatize, answer prediction and verbalizer for the main prompt task (drr) and two auxiliary prompt tasks (ssc and acp). the main drr prompt task uses a kind of connective-cloze prompt to predict a manually selected answer words between two arguments, and map it to a relation sense; the ssc auxiliary prompt task describes and classifies the sense semantic between two arguments; while the acp describes and predicts the implicit connective words. ## experiment setting in this section, we present our experiment settings, including the dataset, plms, competitors, and parameter settings. the pdtb 3.0 dataset: our experiments are conducted on the penn discourse treebank (pdtb) 3.0 corpus foot_0 @xcite , which contains more than one million words of english texts from the wall street journal. following the conventional data splitting, we use sections 2-20 as the full training set, sections 21-22 as the testing set and 0-1 as the development set @xcite . our experiments are conducted on the four top-level classes of relation sense, including comparison, contingency, expansion, temporal. pre-trained language models: we use two of the most representative masked pre-trained language models (plm) for comparison: bert (devlin et al., 2019) is the first transformer-based large-scale pre-trained plm proposed by google foot_1 , which is pre-trained using a cloze task and a next sentence prediction task; roberta @xcite is a bert-enhanced plm proposed by facebook foot_2 , which removes the next sentence prediction objective and is pre-trained on a much larger dataset with some modified key hyper-parameters. competitors: we compare our teprompt with the following advanced models: parameter setting: we implement the plm models with 768-dimension provided by hugging-face transformers foot_3 @xcite , and run pytorch foot_4 framework with cuda on nvidia gtx 3090 ti gpus. the maximum length of our teprompt template is set to 150 tokens, in which the maximum length of arguments are 70 tokens. we set the mini-batch size to 32, the learning rate to 1e-5, the weight coefficients β and γ to 0.3 and 0.4 respectively, and all trainable parameters are randomly initialized from normal distributions. we release the code at: https://github.com/hustminslab/teprompt .