Knowledge Context Modeling with Pre-trained Language Models for Contrastive Knowledge Graph Completion (2024)

Abstract: Text-based knowledge graph completion (KGC) methods utilize pre-trained language models for triple encoding and further fine-tune the model to achieve completion. Despite their excellent performance, they neglect the knowledge context in inferring process. Intuitively, knowledge contexts, which refer to the neighboring triples around the target triples, are important information for triple inferring, since they provide additional detailed information about the entities. To this end, we propose a novel framework named KnowC, which models the knowledge context as additional prompts with pre-trained language models for knowledge graph completion. Given the substantial number of neighbors typically associated with entities, along with the constrained input token capacity of language models, we further devise several strategies to sample the neighbors. We conduct extensive experiments on common datasets FB15k-237, WN18RR and Wikidata5M, experiments show that KnowC achieves state-of-the-art performance.

## introduction knowledge graphs are collections of facts, which are represented as sets of triples (h, r, t), denoting that the head entity h has a r relation with the tail entity t. knowledge graphs play important roles in practical applications of many fields, such as question answering @xcite , recommendation system @xcite , and so on. in real world scenarios, large-scale knowledge graphs are usually incomplete, which hinders their performance in downstream tasks. to solve this problem, researchers propose many knowledge graph completion methods to infer the missing facts. to complete the missing links in knowledge graph, many researchers propose to embed entities and relations into vectors using representation * corresponding author. however, existing text-based kgc methods mainly focus on the triple text itself for inferring, while neglect the knowledge context of the target triple, where knowledge context refers to the neighboring triples around the target entities. here the target entity denotes the given entity for relation prediction, and the neighboring triples denote triples directly connected to the target entity. intuitively, these knowledge contexts are important information for the inference of language model, as they describe detailed supplemental information about the target entities. for example, suppose we need to infer the missing triple (james harden, nationality, ?). we could only use the knowledge implicitly inherited in the language model if we have no additional information. in contrast, with access of knowledge context (james harden, borns in, los angeles) as additional information, it will be much easier for language models to get the right answer. to this end, we propose a novel framework named knowc, which models the knowledge context as additional prompts with pre-trained language models for knowledge graph completion. specifically, for a target triple to be predicted, we sample the neighbors around the target entities as knowledge context. we then use these knowledge contexts as supplemental prompts, which are concatenated together with targets and fed into the language models for encoding. consequently, the entities and relationships used for prediction are incorporated with more supplementary information related to them, thereby enhancing the feasibility of arriving at accurate answers. despite this, there might be hundreds of neighbors for a target entity in the knowledge graph, whereas the language model has a limit on the number of input tokens. to solve this problem, we introduce several strategies for neighbor sampling: random sampling, knn sampling, and dynamic sampling. random sampling randomly samples neighbors connected to the target entity, which consumes least computation resource. knn sampling uses the target relationship as query and neighbors relationships as keys to model the relevance between query-key pairs and select the most relevant k neighbors @xcite . dynamic sampling dynamically samples k neighbors from the neighbor set in each epoch to mitigate the overfitting risk to specific neighbor configurations. after getting the text embeddings of entities and relations, we fine-tune the language model for inference. we optimize the model with infonce contrastive loss, following recent state-of-the-art researches @xcite . to evaluate the effectiveness of knowc, we conduct extensive experiments on common datasets: wn18rr, fb15k-237, and wikidata5m, and compare its mrr and hits@k metrics with several benchmarks. experimental results show that knowc gets excellent performance. our main contributions can be summarized as follows: ## conclusion in this paper, we propose a novel text-based knowledge graph completion method. we use neighboring triples as supplementary prompts as additional detailed information about the entities to benefit the completion task. we also design several sampling strategies to limit the input token as well as to help the model improve generalization ability. we conduct extensive experiments to evaluate the excellent performance of our model on knowledge graph completion task.