Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition (2023)

Abstract: Implicit Discourse Relation Recognition (IDRR), which infers discourse relations without the help of explicit connectives, is still a crucial and challenging task for discourse parsing. Recent works tend to exploit the hierarchical structure information from the annotated senses, which demonstrate enhanced discourse relation representations can be obtained by integrating sense hierarchy. Nevertheless, the performance and robustness for IDRR are significantly constrained by the availability of annotated data. Fortunately, there is a wealth of unannotated utterances with explicit connectives, that can be utilized to acquire enriched discourse relation features. In light of such motivation, we propose a \textbf{P} rompt-based \textbf{L} ogical \textbf{S} emantics \textbf{E} nhancement (PLSE) method for IDRR. Essentially, our method seamlessly injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction. Furthermore, considering the prompt-based connective prediction exhibits local dependencies due to the deficiency of masked language model (MLM) in capturing global semantics, we design a novel self-supervised learning objective based on mutual information maximization to derive enhanced representations of logical semantics for IDRR. Experimental results on PDTB 2.0 and CoNLL16 datasets demonstrate that our method achieves outstanding and consistent performance against the current state-of-the-art models.

## introduction implicit discourse relation recognition (idrr) focuses on identifying semantic relations between two arguments (sentences or clauses), with the absence of explicit connectives (e.g., however, be-figure 1 : an unannotated instance with an explicit connective and an idrr instance from gigaword @xcite and pdtb 2.0 @xcite respectively. for idrr, the implicit connective is absent in the raw text, which are assigned by the annotator based on the semantic relations between argument1 and argu-ment2. early works based on traditional machine learning are dedicated to the extraction of intricate syntax features @xcite . with the rapid development of deep learning, pre-trained language models (plms), such as bert @xcite and roberta @xcite , have remarkably improved the performance of idrr. recently, there has been a growing inclination among various work to exploit the multi-level hierarchical information (an instance introduced in appendix a.1) derived from annotated senses @xcite @xcite . nevertheless, such usage exhibits an excessive reliance on the annotated data, thereby restricting the potential of idrr for further expansion to some degree. fortunately, there exists a plethora of unannotated utterances containing explicit connectives that can be effectively utilized to enhance discourse relation representations. as depicted in figure 1 , even in the absence of annotated sense, the explicit data can proficiently convey a contingency.cause.result relation, elucidating a strong correlation between the connective and the discourse relation. in expectation of knowledge transfer from explicit discourse relations to implicit discourse relations, @xcite has undertaken preliminary investigations into leveraging explicit data and connective prediction. this work incorporated an auxiliary pre-training task named explicit connective prediction, which constructed an explicit connective classifier to output connectives according to the representation of the [cls]. nonetheless, this method falls short in terms of intuitive connective prediction through the [cls] and fails to fully harness the valuable knowledge gleaned from the masked language model (mlm) task. besides, zhou et al. (2022) proposed a prompt-based connective prediction (pcp) method, but their approach relies on prefix-prompt, which is not inherently intuitive and introduces additional complexity of inference. in this paper, we propose a novel prompt-based logical semantics enhancement (plse) method for implicit discourse relation recognition. specifically, we manually design a cloze-prompt template for explicit connective prediction (pre-training) and implicit connective prediction (prompt-tuning), which achieves a seamless and intuitive integration between the pre-training and the downstream task. on the other hand, previous work @xcite has revealed that the contextualized representations learned by mlm lack the ability to capture global semantics. in order to facilitate connective prediction, it is important to have a comprehensive understanding of the text's global logical semantics. inspired by recent unsupervised representation learning with mutual information @xcite , we propose a novel selfsupervised learning objective that maximizes the mutual information (mi) between the connectiverelated representation and the global logic-related semantic representation (detailed in section 3.2). this learning procedure fosters the connectiverelated representations to effectively capture global logical semantic information, thereby leading to enhanced representations of logical semantics. our work focuses on identifying implicit intersentential relations, thus evaluating our model on pdtb 2.0 @xcite and @xcite datasets. since the latest pdtb 3.0 @xcite introduces numerous intrasentential relations, there is a striking difference in the distribution between inter-sentential and intrasentential relations reported by @xcite . given this issue, we don't conduct additional experiments on pdtb 3.0. the main contributions of this paper are summarized as follows: representation learning with mi methods based on mutual information have a long history in unsupervised representation learning, such as ... ... ... ... + { { q k v but ... so then and also ... ... ... -a ... ... ... ... but ... so then and also ... ... ... comp cont. temp. exp. the infomax principle @xcite and ica algorithms @xcite . @xcite was the first to integrate mi-related optimization into deep learning models. from then on, numerous studies have demonstrated the efficacy of mi-maximization for unsupervised representation learning @xcite @xcite . inspired by these superior works, we explore how to derive enhanced discourse relation representations by incorporating global logical semantic information through mi-maximization. implicit discourse relation recognition for this task, many prior works based on deep neural network have proposed various approaches to exploiting better semantic representation of arguments, such as shallow @xcite and lstm with multi-level attention @xcite . recently, plms have substantially improved the performance of idrr by leveraging their robust contextualized representations @xcite . furthermore, @xcite proposed an auxiliary pretraining task through unannotated explicit data and connective prediction. zhou et al. ( 2022 ) proposed a connective prediction (pcp) method based on prefix-prompt. regrettably, their approaches fail to capitalize on the potential of data and plms. on the other hand, the multi-level hierarchical information has been explored by @xcite @xcite . however, the efficacy of these methods remains significantly constrained by the availability of annotated data. therefore, our work is dedicated to the proficient utilization of unannotated explicit data via prompt-based connective prediction and mi-maximization. ## methodology figure 2 shows the overall architecture of our framework. 2 in the following sections, we will elaborate on our methodologies including a cloze-prompt template based on connective prediction, a robust pre-training approach to incorporating explicit data, and an effective prompt-tuning method by leveraging implicit data.