M o C o KGC : Momentum Contrast Entity Encoding for Knowledge Graph Completion (2024)

Abstract: In recent years, numerous studies have sought to enhance the capabilities of pretrained language models (PLMs) for Knowledge Graph Completion (KGC) tasks by integrating structural information from knowledge graphs. However, existing approaches have not effectively combined the structural attributes of knowledge graphs with the textual descriptions of entities to generate robust entity encodings.To address this issue, this paper proposes MoCoKGC (Momentum Contrast Entity Encoding for Knowledge Graph Completion), which incorporates three primary encoders: the entity-relation encoder, the entity encoder, and the momentum entity encoder. Momentum contrastive learning not only provides more negative samples but also allows for the gradual updating of entity encodings. Consequently, we reintroduce the generated entity encodings into the encoder to incorporate the graph’s structural information.Additionally, MoCoKGC enhances the inferential capabilities of the entity-relation encoder through deep prompts of relations. On the standard evaluation metric, Mean Reciprocal Rank (MRR), the MoCoKGC model demonstrates superior performance, achieving a 7.1% improvement on the WN18RR dataset and an 11% improvement on the Wikidata5M dataset, while also surpassing the current best model on the FB15k-237 dataset. Through a series of experiments, this paper thoroughly examines the role and contribution of each component and parameter of the model.

## introduction as an important method of knowledge representation, the fundamental building blocks of a knowledge graph are factual triples, such as (steve jobs, founded, apple inc.). however, the process of constructing knowledge graphs, whether manually or through automation, inevitably leads to the presence of many missing triplets within the knowledge * corresponding authors graph. therefore, the knowledge graph completion task (kgc) aims to complete the missing triples. among the numerous methods for kgc, knowledge graph embedding (kge) techniques are the most classical and widely adopted. the core idea behind these methods is to generate embedding vectors for entities and relationships and use different scoring functions to predict the missing triplets @xcite @xcite . building upon this, some studies introduce the structure of knowledge graphs as supplementary information in the reasoning process to improve prediction accuracy @xcite @xcite . in recent years, researchers have begun exploring the integration of pre-trained language models (plms) into the task of kgc, aiming to improve accuracy through the textual descriptions of entities and relationships. models based on pre-trained language encoders can be broadly categorized into three types: (1) cross-encoder models @xcite ; (2) bi-encoder models @xcite ;(3) single-encoder models @xcite . although cross-encoder models fully utilize the semantic information of triplets, their performance is often not as good as other methods due to the high cost of obtaining negative samples during training. bi-encoder models, by separating the tail entity, not only preserve the textual information of the tail entity but also effectively increase the number of negative samples. single-encoder models, despite removing the textual information of the tail entity, demonstrate unique advantages by acquiring a large number of negative samples for tail entities and introducing graph information through entity embeddings. in order to preserve the textual information of entities while flexibly integrating entity encoding into the training and prediction phases of the model, this study adopts the momentum contrastive learning mechanism @xcite , thereby proposing the mocokgc model. this model draws inspiration from the bi-encoder architecture, equipped with a head entity-relation encoder and an entity encoder. the distinction lies in that, within the mocokgc model, the update of entity encoding relies on a momentum entity encoder, rather than directly utilizing the entity encoder, thus achieving a smoothing of the entity encoding update process. consequently, the mocokgc model exhibits two significant features compared to other methods: (1) entity queue. this queue is maintained in the order of entity encodings generated by the momentum entity encoder, providing the model with a rich source of negative tail entity samples; (2) reutilization of entity encoding. under this mechanism, entity encoding, as a part of integrating the structural information of the knowledge graph, is re-imported into the encoder. in terms of experimental validation, the mo-cokgc model not only successfully preserved the textual information of entities but also flexibly integrated entity encoding into the model's training and prediction processes. its performance on standard datasets wn18rr, fb15k-237, and wikidata5m demonstrates the model's superior capabilities: on the wn18rr dataset, in mean reciprocal rank (mrr), the model achieved a 7.1% improvement, an 11% increase on the wikidata5m dataset, and surpassed previous models on the fb15k-237 dataset. furthermore, comparative experimental data reveal that mocokgc effectively overcomes the inconsistency issues exhibited by plms when dealing with sparse and dense knowledge graphs @xcite . methods based on pre-trained language models (plms), such as kg-bert @xcite , concatenate the descriptions of head entities, relations, and tail entities, and directly obtain the triplet score by inputting it into the bert @xcite model. star @xcite adopts a dual-encoder architecture, which significantly reduces the inference time overhead of language models. simkgc @xcite introduces a contrastive learning approach. prompt tuning has emerged as a strategy aimed at significantly improving the performance of plms by adding prompt tokens to the input. this approach was initially developed to address the challenge of fine-tuning large language models (llms) for downstream tasks @xcite . recently, in kgc tasks, researchers have improved the performance of plms through immediate learning, lv et al. ( 2022 ) adding prompt templates and soft prompts to the input, while @xcite and @xcite specified different soft prompt tokens for different types of relations. this paper views relations as deep prompt parameters and introduces entity neighborhood prompts, achieving the objective of leveraging both textual descriptions and knowledge graph structural information. contrastive learning by differentiating positive and negative sample features, learns distinctive feature representations and has been successfully applied in multiple domains, including computer vision. moco @xcite proposed a momentum-based contrastive learning method, effectively solving the problem of sample pair construction in unsupervised learning by building a dynamically changing encoder queue. simclr @xcite , as a method of visual representation learning, significantly improved the performance of image recognition tasks through largescale unsupervised contrastive learning. in kgc tasks, simkgc @xcite treats tail entities as positive and negative samples, implementing efficient training through three different simple negative sampling strategies. this paper combines the momentum contrast method of moco, utilizing it while dynamically updating entity encodings. ## conclusion this study proposes mocokgc, a novel kgc model that leverages momentum contrastive learning in conjunction with pmls. by expanding the pool of negative samples, it further enhances kgc through the aggregation of entity textual descriptions and their structural information. the mo-cokgc model demonstrated superior performance across multiple datasets. furthermore, we further validated the critical role of its constituent components and parameter configurations. future work will focus on adapting mocokgc for open knowledge graphs to better manage the emergence of new entities.