Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? (2024)

Abstract: Knowledge graphs (KGs) consist of links that describe relationships between entities. Due to the difficulty of manually enumerating all relationships between entities, automatically completing them is essential for KGs. Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities in a KG. Traditional embedding-based KGC methods (e.g. RESCAL, TransE, DistMult, ComplEx, RotatE, HAKE, HousE, etc.) infer missing links using only the knowledge from training data. In contrast, the recent Pre-trained Language Model (PLM)-based KGC utilizes knowledge obtained during pre-training, which means it can estimate missing links between entities by reusing memorized knowledge from pre-training without inference. This part is problematic because building KGC models aims to infer unseen links between entities. However, conventional evaluations in KGC do not consider inference and memorization abilities separately. Thus, a PLM-based KGC method, which achieves high performance in current KGC evaluations, may be ineffective in practical applications. To address this issue, we analyze whether PLM-based KGC methods make inferences or merely access memorized knowledge. For this purpose, we propose a method for constructing synthetic datasets specified in this analysis and conclude that PLMs acquire the inference abilities required for KGC through pre-training, even though the performance improvements mostly come from textual information of entities and relations.

## introduction a knowledge graph (kg) is graph-structured data that includes relationships between entities as links. kgs are useful resources to inject external knowledge into nlp models. since manually considering all possible links between entities is difficult, it is important to use a task such as kg completion (kgc), which automatically completes unseen links from seen ones in a kg. as a basic method for kgc, kg embedding (kge) is a popular chioce for this task. kge embeds entities and their relationships as continuous vectors and then calculates the plausibility of unseen links. traditional kge methods learn these embeddings only from a target kg @xcite @xcite @xcite @xcite . thus, they purely infer unseen links to complete kgs. similar to other nlp fields, kgc also utilizes pre-trained language models (plms) @xcite @xcite @xcite @xcite @xcite @xcite @xcite . unlike traditional kge methods, plm-based kge methods can access knowledge obtained through pre-training. this characteristic makes plm-based kge methods achieve higher kgc performance than the traditional kge methods. however, since the purpose of kgc is to infer unseen links from seen links in kgs, we should separately consider the performance gain from reusing the information of the unseen links obtained in pre-training and inferring unseen links from the seen links in kgs. figure 1 shows an example of plm-based kgc. as we can see, plm-based kgc methods can estimate unseen links without inferring them from seen links in the target kg. this characteristic is problematic because we cannot estimate the inference ability of plm-based kgc methods for truly unseen relationships between entities in kgs. to address this issue, we propose a method to create synthetic datasets for kgc tasks intended to separately evaluate kgc performance by reusing the knowledge from pre-training corresponding to target unseen links and inferring from seen links in kgs. more specifically, we change the textual information of entities and relations while maintaining the graph structure of kgs, thereby creating an environment different from the plms' knowledge corresponding to unseen links in kgs. due to this change, plms cannot rely on their pre-trained knowledge and must rely on their pure inference abilities. we conducted experiments on various pretrained models under our controlled synthetic dataset constructed from wn18rr @xcite , fb15k-237 @xcite , and wikidata5m @xcite . the results showed that plms acquire the inference abilities required for kgc in pre-training but rely more on textual information of entities and relations in kgs. we also observed that the kgc performance of plm-based kgc without pre-trained information is comparable to or lower than that of transe, the traditional kgc. this finding indicates the importance of both traditional and plm-based kgc methods. ## synthetic dataset construction to analyze the behavior of plm-based kgc methods, we create synthetic data corresponding to each setting in ## conclusion in this study, we proposed a method for evaluating the inference ability of plm-based kgc methods by separately considering the information related to unseen links in kgs. using this method as a basis, we developed synthetic datasets that focused on the structure of kgs and changed only textual information, maintaining graph structure. then, we compared plm-based kgc methods using these datasets. the comparison results show that plms acquire the inference abilities for kgc in pre-training, whereas in kgs, they rely more on the textual information of entities and relations. further, we observed that the kgc performance of plm-based kgc without pre-trained knowledge is comparable to or lower than that of transe, the traditional kgc. this highlights the importance of using both traditional and plm-based kgc methods. please see appendix c for more detailed information on improving the current kgc evaluation based on the insights from our work.