Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition (2023)

Abstract: Multi-level implicit discourse relation recognition (MIDRR) aims at identifying hierarchical discourse relations among arguments. Previous methods achieve the promotion through fine-tuning PLMs. However, due to the data scarcity and the task gap, the pre-trained feature space cannot be accurately tuned to the task-specific space, which even aggravates the collapse of the vanilla space. Besides, the comprehension of hierarchical semantics for MIDRR makes the conversion much harder. In this paper, we propose a prompt-based Parameter-Efficient Multi-level IDRR (PEMI) framework to solve the above problems. First, we leverage parameter-efficient prompt tuning to drive the inputted arguments to match the pre-trained space and realize the approximation with few parameters. Furthermore, we propose a hierarchical label refining (HLR) method for the prompt verbalizer to deeply integrate hierarchical guidance into the prompt tuning. Finally, our model achieves comparable results on PDTB 2.0 and 3.0 using about 0.1% trainable parameters compared with baselines and the visualization demonstrates the effectiveness of our HLR method.

## introduction implicit discourse relation recognition (idrr) @xcite is one of the most vital sub-tasks in discourse analysis, which proposes to discover the discourse relation between two discourse arguments without the guidance of explicit connectives. due to the lack of connectives, the model can only recognize the relations through semantic clues and entity anaphora between arguments, which makes idrr a challenging task. through a deeper understanding of this task, it is beneficial to a series of downstream tasks such as text summarization @xcite , dialogue summarization @xcite and event relation extraction @xcite . meanwhile, the discourse relation is arg1: after the race, fortune 500 executives drooled like schoolboys over the cars and drivers. arg2: no dummies, the drivers pointed out they still had space on their machines for another sponsor's name or two. root level 1: contingency level 2: cause level 3: result connective: consequently ... ... annotated as multi-level labels. as shown in figure 1 , the top-level label of this argument pair is comparison, while the sub-label contrast is the fine-grained semantic expression of comparison. beyond that, when annotating the implicit relation, the annotators simulate adding a connective consequently. we regard these connectives as the bottom level of discourse relations. since pre-trained language models (plms) are widely applied, idrr has also achieved considerable improvement. however, previous work @xcite @xcite has mentioned the data scarcity of the idrr, in which data is insufficient to support deep neural networks to depict the high-dimensional task-specific feature space accurately. simultaneously, the hierarchical division of discourse relations is complex, and the extraction of hierarchical semantics relies on a large scale of data to sustain. previous studies @xcite @xcite @xcite alleviate this problem by data augmentation or additional knowledge. however, there are several deficiencies: 1) the difficulty of annotating sufficient data and introducing appropriate knowledge is considerable; 2) noisy data drive models to deviate from the target feature dis-tribution, and unreasonable knowledge injection exacerbates the collapse of feature space of plms. recently, some prompt tuning (pt) methods @xcite @xcite @xcite have shown remarkable results in low resource scenarios (i.e., parameter-efficient prompt tuning, pept). they freeze most or all parameters of plms and leverage a few additional parameters to restrict the approximation in a small manifold, thus reducing the dependency on the scale of data. inspired by the above, we leverage the pept to drive the input to match the pre-trained feature space and further present a parameter-efficient multi-level idrr framework (pemi), which alleviates the under-training problem caused by data scarcity and infuses hierarchical guidance into the prompt verbalizer. thus we can mine better context patterns guided by hierarchical label signals for the idrr. generally, prompt-based framework mostly consists of two parts: template engineering and verbalizer engineering. for the template formulation, instead of manually designed templates, we inject soft prompts into the template and regard them as learnable global context vectors to mine the unique pattern of arguments and adjust input features to align the target distribution under the pre-trained semantic space. however, this alignment is marginal, so it is crucial to adopt the verbalizer for the masked language model (mlm), which maps several label words in vocab to a specific category. but these verbalizer does not have access to learn the hierarchical connection of discourse relations. besides, existing methods @xcite @xcite require feature alignment or extra structure (e.g., gcn, crf), which conflicts with the hypothesis of pept. therefore, we propose a novel method called hierarchical label refining (hlr) to incorporate hierarchical information into the verbalizer. in our method, only the bottom-level label words are parameterized. others are refined from the bottom up according to the hierarchical division. and the dispersed label semantics are continuously aggregated to more generalized ones in each iteration, thus realizing the dynamic updating of the verbalizer. finally, our framework carries out joint learning at all levels, thus combining the intra-level label discrimination process and the inter-level hierarchical information integration process. our contributions are summarized as follows: ## overall framework let @xmath0, x 2 ) ∈ x be an argument pair and @xmath1, l 2 , .., l z be the set of total labels, where l z is the level-z label set. the goal of the midrr is to predict the relation sequence @xmath2, . . . , l z , . . . , l z , where l @xmath3. the overview of our framework is shown in figure 2 . in this section, we explain our framework in three parts. first, we analyze the theory of pept for single-level idrr and infer the association with our idea. next, we describe how to expand the pept to midrr through our proposed hierarchical label refining method. finally, we conduct joint learning with multiple levels so as to fuse the label information of inter and intra-levels. ## conclusion in this paper, we tackle the problem of data scarcity for idrr from a parameter perspective and have presented a novel parameter-efficient multi-level idrr framework, which leverages pept to adjust the input to match the pre-trained space with fewer parameters and infuse hierarchical label guidance into the verbalizer. experiments show that our model adopts parameter-efficient methods while it is comparable with recent sota models. besides, it indicates that our framework can effectively stimulate the potential of plms without any intervention of additional data or knowledge. in the future, we will further explore the linguistic features of labels and enhance the discrimination against connectives.