D isco P rompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition (2023)

Abstract: Implicit Discourse Relation Recognition (IDRR) is a sophisticated and challenging task to recognize the discourse relations between the arguments with the absence of discourse connectives. The sense labels for each discourse relation follow a hierarchical classification scheme in the annotation process (Prasad et al., 2008), forming a hierarchy structure. Most existing works do not well incorporate the hierarchy structure but focus on the syntax features and the prior knowledge of connectives in the manner of pure text classification. We argue that it is more effective to predict the paths inside the hierarchical tree (e.g., “Comparison -> Contrast -> however”) rather than flat labels (e.g., Contrast) or connectives (e.g., however). We propose a prompt-based path prediction method to utilize the interactive information and intrinsic senses among the hierarchy in IDRR. This is the first work that injects such structure information into pre-trained language models via prompt tuning, and the performance of our solution shows significant and consistent improvement against competitive baselines.

## introduction discourse parsing is the task of automatically parsing discourse structure in a text, including the identification of discourse structure and the annotation of discourse relations @xcite . discourse relation recognition (drr) is a crucial task in discourse parsing, recognizing relations between two arguments (i.e., sentences or clauses). it is vital for textual coherence and is considered as the essential step for many downstream tasks involving more context, such as question answering @xcite ), text generation @xcite , and argument mining @xcite . explicit discourse relation recognition (edrr) has already been demonstrated that utilizing explicit * equal contribution. connectives information can effectively determine the discourse relation types @xcite . on the other hand, implicit discourse relation recognition (idrr) is still challenging with the absence of connectives @xcite . traditional works on idrr focus on syntax features, including word pairs @xcite and other surface features @xcite . with deep neural networks and large language models (llms), different approaches pay much attention to text representations via attention @xcite , pre-training @xcite , multi-task learning @xcite , and prior knowledge @xcite . but one important piece of information, i.e., the inherent discourse label hierarchy, is not fully investigated. the sense labels for each discourse relation follow a hierarchical classification scheme in the annotation process of pdtb 2.0 framework @xcite , forming a hierarchy structure. figure 1 shows an example from pdtb 2.0 dataset @xcite . it consists of two arguments (i.e., arg1 and arg2) and is annotated with relation senses, where the semantics of the top-level comparison is further refined by the second-level contrast. besides, we list representative connectives (e.g., however) to help better understand the definitions and semantics of labels. ldsgm @xcite uses graph convolutional networks to encode the label dependencies into text representations, illustrating the importance of label structures on text representation learning and label prediction. however, such usage is not compatible with pre-training because it may significantly affect the representations from language models. prompt tuning has shown its power in text classification without altering the representations from pre-trained language models, especially for low-resource scenarios @xcite . in this paper, we propose a prompt-based path prediction method, discourse relation path prediction prompt tuning model (discoprompt foot_0 ), to utilize the hierarchy and intrinsic senses of labels in idrr. specifically, we transform the hierarchy in figure 1 to "comparison -> concession -> if; • • • ; temporal -> synchrony -> when" as the hierarchical prompt and add it as the prefix of arguments to be classified. the dependencies of top and second-level relation senses are explicitly provided as the context. on the other hand, connectives are provided as the natural language explanations of labels to help the language models better adapt to the prior knowledge. we ask the llms to predict the label's hierarchical path instead of the leaf label for iddr, and we show such a way of providing the label hierarchy ahead of arguments significantly improves the idrr performance. our contributions are summarized as follows: the results support our claims and the success of our proposed discoprompt model. ## related work prompt tuning with llms, such as t5 @xcite and @xcite , prompt-based methods have attracted much attention in the field of natural language understanding @xcite @xcite . compared with fine-tuning, prompt tuning may have a better generalization on various tasks due to the aligned nature of language descriptions and answer semantics, e.g., classification problems @xcite . at the same time, there are some efforts to leverage prompts with structural inputs for knowledge customization @xcite . injecting hierarchy information into prompts is also promising. for example, using top-level predictions to refine prompts of bottom levels can surpass soft prompts and hard prompts @xcite . nevertheless, how to employ llms to better involve hierarchy knowledge is still under investigation. implicit discourse relation recognition it has been discovered that connectives can provide necessary clues in predicting discourse relations to achieve around 95% accuracy @xcite . however, the absence of connectives makes the prediction more challenging. many efforts have been paid to explore the syntax through linguistic features @xcite @xcite @xcite , attention @xcite , pre-training @xcite , knowledge transfer @xcite @xcite , etc. with the power of language models, connective prediction also illustrates its effectiveness in implicit relation prediction @xcite @xcite . in addition, pcp @xcite shows the feasibility of combining label prediction and connective prediction under the manner of prompts. the latest methods reveal the significance of the label hierarchy of discourse relations. ldsgm @xcite utilizes the graph convolutional networks to incorporate label dependencies into text representations, while contrastiveidrr @xcite leverages the sense hierarchy to obtain contrastive learning representation. however, these methods are incompatible with pre-training as they modify the representations from pre-trained language models. therefore, this work investigates injecting the label dependencies information and connectives into pre-trained language models via prompt tuning with aligning the representations. 3 method ## conclusion in this paper, we introduce a path prediction method for tackling the idrr task by utilizing the hierarchical structural information and prior knowledge of connectives. combining label structures in natural language with prompt tuning successfully takes a step further in this task as well as other generalized settings, e.g., prompt adaptation and explicit relation detection. our model achieves new sota performance on pdtb 2.0 and conll-2016 data, and we hope our detailed discussions can help communities in discourse fields.