Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach (2022)

Abstract: In recent years, pre-trained language models (PLMs) have been shown to capture factual knowledge from massive texts, which encourages the proposal of PLM-based knowledge graph completion (KGC) models. However, these models are still quite behind the SOTA KGC models in terms of performance. In this work, we find two main reasons for the weak performance: (1) Inaccurate evaluation setting. The evaluation setting under the closed-world assumption (CWA) may underestimate the PLM-based KGC models since they introduce more external knowledge; (2) Inappropriate utilization of PLMs. Most PLM-based KGC models simply splice the labels of entities and relations as inputs, leading to incoherent sentences that do not take full advantage of the implicit knowledge in PLMs. To alleviate these problems, we highlight a more accurate evaluation setting under the open-world assumption (OWA), which manual checks the correctness of knowledge that is not in KGs. Moreover, motivated by prompt tuning, we propose a novel PLM-based KGC model named PKGC. The basic idea is to convert each triple and its support information into natural prompt sentences, which is further fed into PLMs for classification. Experiment results on two KGC datasets demonstrate OWA is more reliable for evaluating KGC, especially on the link prediction, and the effectiveness of our PKCG model on both CWA and OWA settings.

## introduction knowledge graph (kg) has gradually become the cornerstone of many natural language processing (nlp) tasks @xcite , as one of the most effective ways to represent world knowledge. to improve the coverage, researchers have automated knowledge extraction techniques or relied on collaborative editing, while these kgs still hardly cover the massive emerging knowledge in the real world. this problem motivates knowledge graph completion (kgc), the task of predicting missing links through understanding existing structures in kgs. soon sweeping across the entire nlp field, the potential of pre-trained language models (plms) for kgc has attracted much attention. @xcite ; @xcite reveal that plms have captured factual knowledge implicitly from massive unlabeled texts. this could be helpful to complete missing knowledge. kg-bert @xcite first introduces plms into kgc. it splices the labels of entities and relation in the triple as the input of plms to verify its correctness. @xcite further introduces multi-task learning on the basis of kg-bert. however, the above plm-based kgc models do not present promising results, or are even quite behind conventional knowledge graph embedding (kge) models (about 20.8% lower than the sota model in hits@10). this raises a question: why the learned factual knowledge in plms cannot be beneficial for kgc? in this work, we find two main reasons for the weak performance of plm-based kgc models: (1) inaccurate evaluation setting. most existing kgc models are evaluated under the closed-world assumption (cwa), which assumes that any knowledge unseen in given kgs is incorrect. such a setting can benefit the automatical dataset construction without manual annotation. however, the introduction of plms brings in much unseen knowledge, which is considered to be incorrect under cwa, wrongly lowering the performance. as shown in figure 1 , for a triple query (england, contains, ?), the plm-based kgc model gives many correct tail entities (highlight with boldface), but only pontefract is considered correct under cwa since it exists in kgs. (2) inappropriate utilization of plms. existing plm-based kgc models simply splice the labels of the entities and relations in the triples as the input of plms. this results in incoherent sentences, which gaps with the pretrained task and thus cannot take full advantage of the knowledge in plms. to alleviate the above two problems, we propose a new benchmark setting for rectification of this line of research and a novel plm-based model. to make the kgc evaluation more credible, we highlight a new evaluation setting based on the open-world assumption (owa) -the knowledge not in kgs is not false, but unknown. thus, false positives under cwa shall be removed, as long as we recognize exact true and false triples from unknown. for these unknown triples, we conduct human annotation to check if they are valid. we further propose a novel plm-based kgc model, pkgc, to better induce the implicit knowledge hidden in the plm's parameters. motivated by the prompt-based models @xcite , the basic idea is to convert each triple into natural prompt sentences instead of simply splicing their labels. in specific, we manually define the prompt template for each relation type and further introduce soft prompts to better express the semantics of triples. moreover, benefiting from prompt tuning, pkgc can flexibly consider the contexts of triples, such as definition and attributes by inserting them as the support prompt at the end of the triple prompt. we conduct experiments on two kgc datasets sampled from wikidata and freebase, and re-evaluate the kge-based and plm-based kgc models under owa instead of cwa. according to our experimental results, we find that: (1) owa provides a more accurate evaluation for kgc, especially for the more knowledgeable plm-based kgc model and the more open link prediction task. ## preliminary knowledge graph is a network composed of entities and relations. it can be defined as @xmath0, r, t }, where e is the set of entities and r is the set of relations. @xmath1, r, t)} ⊆ e × r × e is the triple set, where h and t are the head and tail entities, and r is the relation between them. knowledge graph completion task aims at completing missing triples (h, r, t) / ∈ t for the knowledge graph. there are two main methods to do this task, namely link prediction and triple classification, where the former mainly predicts missing entities for triple queries (h, r, ?) or (?, r, t), and the latter aims to determine whether a given triple (h, r, t) is correct or not. closed-world assumption (cwa) believes that the triples that do not appear in a given knowledge graph are wrong. this means that if the dataset con-sists of training/validation/test set, and the model is tested on the test set, only the triples that have appeared in the entire dataset are considered to be correct. we can easily evaluate the performance of models without annotation under cwa. however, cwa is essentially an approximation and cannot guarantee the accuracy of the evaluation results. open-world assumption (owa) believes that the triples contained in the knowledge graph are not complete. therefore, the evaluation under the open-world assumption is more accurate and closer to the real scenario, but requires additional human annotations to carefully verify whether the completed triples that are not in the knowledge graph are correct or not. specifically, on the one hand, we convert a triple into prompt sentences to use the knowledge in plms. as shown in figure 2 , given a triple, our model transforms it into triple prompts p t and support prompts p s , which are jointly fed into a pre-trained language model. formally, the final input texts t to the plm can be defined as @xmath2. on the other hand, we feed positive/negative triples to our model for triple classification and use cross-entropy loss for training. in this way, our model can exploit the structural information in kgs. in the following sections, we will introduce the design strategy of triple prompts(section 4.2) and the production method of support prompts (section 4.3) in detail. in addition, we will also explain the training method of our model in section 4.4. ## experiments in experiments, we give the results of models under cwa and owa. specifically, the results under cwa are for reference, and the results under owa can better reflect the real performance of the model. ## conclusion and future work with the rapid development of pre-trained language models, some plm-based kgc models are proposed. however, there is still a performance gap between these models and sota kge models. in this work, we find two main reasons for the weak performance: (1) inaccurate evaluation setting. (2) inappropriate utilization of plms. to alleviate these problems, we highlight a more accurate eval-uation setting owa and propose a novel plmbased kgc model. in our experiments, we verify that cwa cannot bring accurate evaluation results. moreover, the experimental results show that our model can achieve better results than the previous method. in our future work, we plan to comprehensively and systematically re-evaluate the existing kgc models to reveal their real performance. to alleviate this problem, in this work, we use a recall and re-ranking framework. specifically, for a triplet query (h, r, ?), we first use a kge model (tucker is used in experiments) to get the ranking of the tail entities. after that, we select the top x ranked entities and use a plm-based kgc model to recalculate the scores. based on these scores, we can re-rank the top x entities.