Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding (2022)

Abstract: Knowledge-enhanced Pre-trained Language Model (PLM) has recently received significant attention, which aims to incorporate factual knowledge into PLMs. However, most existing methods modify the internal structures of fixed types of PLMs by stacking complicated modules, and introduce redundant and irrelevant factual knowledge from knowledge bases (KBs). In this paper, to address these problems, we introduce a seminal knowledge prompting paradigm and further propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be flexibly combined with existing mainstream PLMs. Specifically, we first construct a knowledge sub-graph from KBs for each context. Then we design multiple continuous prompts rules and transform the knowledge sub-graph into natural language prompts. To further leverage the factual knowledge from these prompts, we propose two novel knowledge-aware self-supervised tasks including prompt relevance inspection and masked prompt modeling. Extensive experiments on multiple natural language understanding (NLU) tasks show the superiority of KP-PLM over other state-of-the-art methods in both full-resource and low-resource settings. Our source codes will be released upon the acceptance of the paper.

## introduction pre-trained language models (plms) have become the dominant infrastructure for a majority of downstream natural language understanding (nlu) tasks, which generally adopt a two-stage training strategy, i.e., pre-training and fine-tuning. recent notable plms include bert @xcite , masked mention deep transformer encoder mlm head mention deep transformer encoder fusion module knowledge encoder kg entity word-knowledge graph graph neural networks mlm head [sep] deep transformer encoder mlm head sub-graph mention1 mention2 continuous prompting prompt1 prompt2 knowledge head (a) knowledge masking (b) knowledge fusion (c) graph learning (d) our proposal: knowledge prompting plm module new module special token masked token entity mention token [cls] [sep] [cls] [sep] [cls] [cls] [sep] [sep] roberta @xcite , albert @xcite , xlnet @xcite , gpt-2 @xcite and deberta @xcite . although many existing models learn useful inherent linguistic knowledge from large-scale corpora, it is hard for them to understand the explicit factual knowledge @xcite . to further leverage factual knowledge, a branch of knowledge-enhanced methods @xcite @xcite @xcite have been proposed for plms to capture rich semantic knowledge from knowledge bases (kbs). figure 1 summarizes the paradigms of existing approaches, which can be mainly divided into three categories. first, knowledge-maskingbased methods perform alignment, mask mentions and learn explicit knowledge of entities based on masked language modeling (mlm). second, knowledge-fusion-based methods separately learn embeddings of entities in sentences and that of nodes in kbs, which are further aggregated. third, graph-learning-based methods construct a graph based on contextualized sentences and kbs, and adopt a graph-based encoder to learn entity embeddings, such as graph neural networks @xcite . despite the success, there could still be two problems in these methods. on the one hand, some approaches modify internal structures of existing plms by stacking complicated modules, which adversely affects the computational cost of the model. further, these methods are generally based on fixed types of plms, which leads to the inflexibility and restricts their wide applicability @xcite . on the other hand, some methods introduce redundant and irrelevant knowledge from kbs, which are the knowledge noises and could degrade the model performance @xcite : 1) redundant knowledge. previous methods @xcite @xcite inject the corresponding triples or pre-trained knowledge embeddings into each entity in the context. however, the entity may appear multiple times in one sentence, these methods could introduce duplicate information, which can be viewed as redundant knowledge. 2) irrelevant knowledge. some entities or the corresponding sub-graphs are irrelevant with the whole sentence semantics, which are useless and have less contributions to the performance improvement. recently, the paradigm of prompt-based finetuning (i.e., prompt-tuning) has been proposed to explore the inherent knowledge of plms by introducing a task-specific template with the [mask] token @xcite 2 . inspired by prompt-tuning, in this paper, we introduce a novel knowledge prompting paradigm for knowledge-enhanced plms and propose an effective knowledge-prompting-based plm framework, namely, kp-plm. as shown in figure 1(d) , we aim to construct the related knowledge subgraph for each sentence. to alleviate introducing too much knowledge noises, we restrict all the relation paths start from the topic entity (the entry of each sentence) and end at the tail entity which is mentioned in this sentence. to bridge the structure gap between the context and kb, we can transform 2 for example, in sentiment analysis, a prompt template (e.g., "it was @xcite .") is added to the review text (e.g., "the film is very attractive."). we can obtain the result tokens of masked position for label prediction (e.g., "great" for the positive label and "boring" for the negative label). ## the kp-plm framework in this section, we first describe the basic notations, and then formally present the techniques of the kp-plm in detail. ## experiments in this section, we comprehensively evaluate the effectiveness of kp-plm. we also conduct the hyperparameter analysis in appendix b.3. ## conclusion in this paper, we presented a seminal knowledge prompting paradigm, based on which a novel knowledge-prompting-based plm framework kp-plm was proposed. we constructed contextual knowledge sub-graphs for contexts and employed continuous prompting mapping to generate knowledge prompts. after that, we designed two selfsupervised pre-training tasks to learn semantic knowledge from prompts. finally, we conducted extensive experiments to evaluate the model performance. experimental results validate the effectiveness of knowledge prompting in boosting the performance of plms.