Dipping PLM s Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting (2023)

Abstract: Knowledge Graph Completion (KGC) often requires both KG structural and textual information to be effective. Pre-trained Language Models (PLMs) have been used to learn the textual information, usually under the fine-tune paradigm for the KGC task. However, the fine-tuned PLMs often overwhelmingly focus on the textual information and overlook structural knowledge. To tackle this issue, this paper proposes CSProm-KG (Conditional Soft Prompts for KGC) which maintains a balance between structural information and textual knowledge. CSProm-KG only tunes the parameters of Conditional Soft Prompts that are generated by the entities and relations representations. We verify the effectiveness of CSProm-KG on three popular static KGC benchmarks WN18RR, FB15K-237 and Wikidata5M, and two temporal KGC benchmarks ICEWS14 and ICEWS05-15. CSProm-KG outperforms competitive baseline models and sets new state-of-the-art on these benchmarks. We conduct further analysis to show (i) the effectiveness of our proposed components, (ii) the efficiency of CSProm-KG, and (iii) the flexibility of CSProm-KG.

## introduction knowledge graphs (kgs) have both complicated graph structures and rich textual information over the facts. despite being large, many facts are still missing. knowledge graph completion (kgc) is a fundamental task to infer the missing facts from the existing kg information. graph-based kgc models @xcite @xcite represent entities and relations using trainable embeddings. these models are trained to keep the connections between entities and relations over structural paths, and tail entities are inferred via various transitional relations. despite being effective in modelling kg structural information, these methods are unable to incorporate linguistic context. recently, pretrained language models (plms) are applied to fill up this gap @xcite @xcite . the proposed solutions often directly fine-tune the plms to choose the correct entities either relying on pure textual context or using structural add-ons as a complementary @xcite . however, plms are normally equipped with large-scale parameters and linguistic inherence obtained from their pre-training stage. as a result, these plm-based models remain overwhelmingly focusing on the textual information in kgs and tend to overlook the graph structure. for example, given an incompleted fact (mona lisa, painted by, ?), the plm-based models may con-fuse between leonardo dicaprio and leonardo da vinci simply because they are textually similar. thus, in this paper, we focus on the research question: can we effectively fuse the kg structural information into the plm-based kgc models? to this end, we propose a novel csprom-kg model (conditional soft prompts for kgc) which is a structure-aware frozen plms that could effectively complete the kgc task. the core of csprom-kg is conditional soft prompt that is an structure-aware version of soft prompt @xcite . previously, soft prompt is a sequence of unconditional trainable vectors that are prepended to the inputs of frozen plms. such design could effectively avoid the issue of over-fitting towards textual information caused by fine-tuning and allow the frozen plms to learn the downstream tasks @xcite . however, such naive soft prompts cannot represent any structural information in kg. to remedy this, as shown in figure 1 (c ), we propose the prompt vectors conditioned on the kg entities and relations embeddings. specifically, we use the entity and relation embeddings to generate conditional soft prompts which are then fed into the frozen plms to fuse the textual and structural knowledge together. the fused conditional soft prompts are used as inputs to the graph-based kgc model that produces the final entity ranking results. we further propose local adversarial regularization to improve csprom-kg to distinguish textually similar entities in kg. we evaluate csprom-kg on various kgc tasks and conduct experiments on wn18rr, fb15k-237 and wikidata5m for static kgc (skgc), and on icews14 and icews05-15 for temporal kgc (tkgc). csprom-kg outperforms a number of competitive baseline models, including both graphbased and plm-based models. we conduct ablation studies to show the strength of prompt-based methods against the fine-tuning counterparts and the effectiveness of each proposed components. we also demonstrate the flexibility of csprom-kg with different graph-based models, and the training and inference efficiency of csprom-kg. ## related work graph-based methods graph-based methods represent each entity and relation with a continuous vector by learning the kg spatial structures. they use these embeddings to calculate the dis-tance between the entities and kg query to determine the correct entities. the training objective is to assign higher scores to true facts than invalid ones. in static kgc (skgc) task, there are two types of methods: 1) translational distance methods measure the plausibility of a fact as the distance between the two entities, @xcite @xcite ; 2) semantic matching methods calculate the latent semantics of entities and relations @xcite @xcite . in temporal kgc (tkgc) task, the systems are usually based on skgc methods, with additional module to handle kg factual tuples timestamps @xcite @xcite . plm-based methods plm-based methods represent entities and relations using their corresponding text. these methods introduce plm to encode the text and use the plm output to evaluate the plausibility of the given fact. on skgc, @xcite encode the combined texts of a fact, then a binary classifier is employed to determine the plausibility. to reduce the inference cost in yao et al. (2019), wang et al. (2021a) exploit siamese network to encode (h, r) and t separately. unlike previous encode-only model, xie et al. (2022); saxena et al. (2022) explore the seq2seq plm models to directly generate target entity text on kgc task. prompt tuning brown et al. (2020) first find the usefulness of prompts, which are manually designed textual templates, in the gpt3 model. wallace et al. (2019); shin et al. (2020) extend this paradigm and propose hard prompt methods to automatically search for optimal task-specific templates. however, the selection of discrete prompts involves human efforts and difficult to be optimized together with the downstream tasks in an end-toend manner. @xcite relax the constraint of the discrete template with trainable continuous vectors (soft prompt) in the frozen plm. as shown in @xcite ; @xcite ; @xcite , frozen plm with soft prompt could achieve comparative performance on various nlp tasks, despite having much less parameters than fully trainable plm models. to the best of our knowledge, we are the first to apply soft prompt to plm-based kgc model. plm entity embeddings relation embeddings steve jobs [sep] place of birth [sep] 1955-02-24 <san @xmath0. to improve csprom-kg's ability in distinguishing textually similar entities, we further add lar examples that are similar to the tail entities during training. csprom-kg effectively learns both structural and textual knowledge in kg. ## method in this section, we first formulate knowledge graph completion in sec. 3.1. we then introduce csprom-kg in sec. 3.2 to sec. 3.7. ## experiments in this section, we first compare csprom-kg with other competitive baselines in the skgc and tkgc benchmarks in sec. 4.1. we then conduct ablation studies to verify the effectiveness of our propose components in csprom-kg in sec. 4.2. we further show the efficiency and flexibility of csprom-kg in sec. 4.3 and 4.4, respectively. dataset wn18rr @xcite and fb15k-237 @xcite are the most popular skgc benchmarks where all inverse relations are removed to avoid data leakage. wiki-data5m @xcite ) is a recently proposed large-scale skgc benchmark. for tkgc, we use icews14 @xcite and @xcite which include political facts from the integrated crisis early warning system @xcite . more dataset details are shown in ## conclusion and future work in this paper, we propose csprom-kg, a plmbased kgc model that effectively fuses the kg structural knowledge and avoids over-fitting towards textual information. the key innovation of csprom-kg is the conditional soft prompt that connects between a graph-based kgc models and a frozen plm avoiding the textual over-fitting issue. we conduct experiments on five popular kgc benchmarks in skgc and tkgc settings and the results show that csprom-kg outperforms several strong graph-based and plm-based kgc models. we also show the efficiency and flexibility of csprom-kg. for future work, we plan to adapt our method to other relevant knowledge-intensive downstream tasks, such as fact checking and openended question answering.