S im KGC : Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models (2022)

Abstract: Knowledge graph completion (KGC) aims to reason over known facts and infer the missing links. Text-based methods such as KGBERT (Yao et al., 2019) learn entity representations from natural language descriptions, and have the potential for inductive KGC. However, the performance of text-based methods still largely lag behind graph embedding-based methods like TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019b). In this paper, we identify that the key issue is efficient contrastive learning. To improve the learning efficiency, we introduce three types of negatives: in-batch negatives, pre-batch negatives, and self-negatives which act as a simple form of hard negatives. Combined with InfoNCE loss, our proposed model SimKGC can substantially outperform embedding-based methods on several benchmark datasets. In terms of mean reciprocal rank (MRR), we advance the state-of-the-art by +19% on WN18RR, +6.8% on the Wikidata5M transductive setting, and +22% on the Wikidata5M inductive setting. Thorough analyses are conducted to gain insights into each component. Our code is available at https://github.com/intfloat/SimKGC .

## introduction large-scale knowledge graphs (kgs) are important components for knowledge-intensive applications, such as question answering @xcite , recommender systems @xcite , and intelligent conversational agents @xcite etc. kgs usually consist of a set of triples (h, r, t), where h is the head entity, r is the relation, and t is the tail entity. popular public kgs include freebase @xcite , wikidata @xcite , yago @xcite , conceptnet @xcite , and word-net @xcite etc. despite their usefulness in practice, they are often incomplete. knowledge graph completion (kgc) techniques are necessary for the automatic construction and verification of knowledge graphs. existing kgc methods can be categorized into two families: embedding-based and text-based methods. embedding-based methods map each entity and relation into a low-dimensional vector, without using any side information such as entity descriptions. this family includes transe @xcite , transh @xcite , ro-tate @xcite , and tucker @xcite ) etc. by comparison, text-based methods @xcite @xcite incorporate available texts for entity representation learning, as shown in figure 1 . intuitively, text-based methods should outperform embedding-based counterparts since they have access to additional input signals. however, results on popular benchmarks (e.g., wn18rr, fb15k-237, wikidata5m) tell a different story: text-based methods still lag behind even with pre-trained language models. we hypothesize that the key issue for such performance degradation is the inefficiency in contrastive learning. embedding-based methods do not involve the expensive computation of text en-coders and thus can be extremely efficient to train with a large negative sample size. for example, the default configuration of rotate foot_0 trains 1000 epochs with a negative sample size of 64 on the wikidata5m dataset. while the text-based method kepler @xcite can only train 30 epochs with a negative sample size of 1 due to the high computational cost incurred by roberta. in this paper, inspired by the recent progress on contrastive learning, we introduce three types of negatives to improve the text-based kgc method: in-batch negatives, pre-batch negatives, and selfnegatives. by adopting bi-encoder instead of crossencoder @xcite architecture, the number of in-batch negatives can be increased by using a larger batch size. vectors from previous batches are cached and act as pre-batch negatives @xcite . additionally, mining hard negatives can be beneficial for improving contrastive learning. we find that the head entity itself can serve as hard negatives, which we call "self-negatives". as a result, the negative sample size can be increased to the scale of thousands. we also propose to change the loss function from margin-based ranking loss to infonce, which can make the model focus on hard negatives. one advantage of text-based methods is that they enable inductive entity representation learning. entities that are not seen during training can still be appropriately modeled, while embeddingbased methods like transe can only reason under the transductive setting foot_1 . inductive knowledge graph completion is important in the real world as new entities are coming out every day. moreover, text-based methods can leverage state-of-the-art pre-trained language models to learn better representations. a line of recent work @xcite attempts to elicit the implicitly stored knowledge from bert. the task of kgc can also be regarded as a way to retrieve such knowledge. two entities are more likely to be related if connected by a short path in the graph. empirically, we find that text-based models heavily rely on the semantic match and ignore such topological bias to some degree. we propose a simple re-ranking strategy by boosting the scores of the head entity's k-hop neighbors. we evaluate our proposed model simkgc by conducting experiments on three popular benchmarks: wn18rr, fb15k-237, and wikidata5m (both transductive and inductive settings). according to the automatic evaluation metrics (mrr, hits@{1,3,10}), simkgc outperforms state-ofthe-art methods by a large margin on the wn18rr (mrr 47.6 → 66.6), wikidata5m transductive setting (mrr 29.0 → 35.8), and inductive setting (mrr 49.3 → 71.4). on the fb15k-237 dataset, our results are also competitive. to help better understand our proposed method, we carry out a series of analyses and report human evaluation results. hopefully, simkgc will facilitate the future development of better kgc systems. pre-trained language models including bert @xcite , gpt @xcite , and t5 @xcite have led to a learning paradigm shift in nlp. models are first pre-trained on large amounts of unlabeled text corpora with language modeling objectives, and then fine-tuned on downstream tasks. considering their good performance in few-shot and even zero-shot scenarios @xcite , one interesting question is: "can pre-trained language models be used as knowledge bases?" @xcite proposed to probe language models with manually designed prompts. a series of following work @xcite @xcite focus on finding better prompts to elicit the knowledge implicitly stored in the model parameters. another line of work @xcite @xcite injects symbolic knowledge into language model pre-training, and shows some performance boost on several knowledge-intensive tasks. contrastive learning learns useful representations by contrasting between positives and negatives @xcite . the definitions of positives and negatives are task-specific. in selfsupervised vision representation learning @xcite @xcite , a positive pair is two augmented views of the same image, while a negative pair is two augmented views of different images. recently, contrastive learning paradigm has witnessed great successes in many different fields, including multi-modal pre-training @xcite , video-text retrieval @xcite , and natural language understanding @xcite etc. in the nlp community, by leveraging the supervision signals from natural language inference data @xcite , qa pairs @xcite , and parallel corpora @xcite , these methods have surpassed non-contrastive methods @xcite on semantic similarity benchmarks. @xcite ; @xcite ; @xcite adopt contrastive learning to improve dense passage retrieval for open-domain question answering, where the positive passages are the ones containing the correct answer. ## methodology 3.1 notations a knowledge graph g is a directed graph, where the vertices are entities e, and each edge can be represented as a triple (h,r,t), where h, r, and t correspond to head entity, relation, and tail entity, respectively. the link prediction task of kgc is to infer the missing triples given an incomplete g. under the widely adopted entity ranking evaluation protocol, tail entity prediction (h, r, ?) requires ranking all entities given h and r, similarly for head entity prediction (?, r, t). in this paper, for each triple (h,r,t), we add an inverse triple (t,r -1 ,h), where r -1 is the inverse relation of r. based on such reformulation, we only need to deal with the tail entity prediction problem @xcite . ## analysis we conduct a series of analyses to gain further insights into our proposed model and the kgc task. ## conclusion this paper proposes a simple method simkgc to improve text-based knowledge graph completion. we identify that the key issue is how to perform efficient contrastive learning. leveraging the recent progress in the field of contrastive learning, simkgc adopts a bi-encoder architecture and combines three types of negatives. experiments on the wn18rr, fb15k-237, and wikidata5m datasets show that simkgc substantially outperforms stateof-the-art methods. for future work, one direction is to improve the interpretability of simkgc. in methods like ro-tate @xcite and transe @xcite , a triple can be modeled as rotation in complex space or relational translation, while simkgc does not enable such easy-to-understand interpretations. another direction is to explore effective ways to deal with false negatives @xcite resulting from the incompleteness of knowledge graphs.