Bilateral Masking with prompt for Knowledge Graph Completion (2024)

Abstract: The pre-trained language model (PLM) has achieved significant success in the field of knowledge graph completion (KGC) by effectively modeling entity and relation descriptions. In recent studies, the research in this field has been categorized into methods based on word matching and sentence matching, with the former significantly lags behind. However, there is a critical issue in word matching methods, which is that these methods fail to obtain satisfactory single embedding representations for entities.To address this issue and enhance entity representation, we propose the Bilateral Masking with prompt for Knowledge Graph Completion (BMKGC) approach.Our methodology employs prompts to narrow the distance between the predicted entity and the known entity. Additionally, the BMKGC model incorporates a bi-encoder architecture, enabling simultaneous predictions at both the head and tail. Furthermore, we propose a straightforward technique to augment positive samples, mitigating the problem of degree bias present in knowledge graphs and thereby improving the model’s robustness. Experimental results conclusively demonstrate that BMKGC achieves state-of-the-art performance on the WN18RR dataset.

## introduction knowledge graphs (kgs) are graph-structured knowledge bases, typically consisting of triples, denoted as (h, r, t), where h represents the head entity, r represents the relation, and t represents the tail entity. prominent examples of kgs include freebase , wikidata @xcite , yago @xcite ,concept-net @xcite , and wordnet @xcite . kgs find applications in various domains, including information retrieval @xcite , recommendation systems @xcite , and * corresponding authors question answering @xcite . however, kgs encounter the challenge of incompleteness as real-world information continuously evolves figure1. hence, the tasks of knowledge graph completion (kgc) @xcite hold significant importance. in recent years, to enhance kg completion and utilization, significant research efforts have been devoted to the field of knowledge embedding (ke), which aims to map kgs into low-dimensional vector spaces. existing knowledge embedding methods fall into two categories: structure-based methods and descriptionbased methods. structure-based methods, such as transe @xcite , rotate @xcite and tucker @xcite , leverage the topological information within the knowledge graph to represent entities and relations. description-based methods can be further categorized into sentence matching and word matching approaches.the methods utilized in this paper draw inspiration from sentence matching techniques while falling under the category of word matching methods. these methods enrich entity representations by incorporating descriptive information related to entities and relations. they transform entity-relation pairs and their descriptions into natural language-style sequences and uti-lize pre-trained language models (plm) such as bert @xcite ) to encode them. this encoding process generates comprehensive representations of entities and relations. the distinction between the two approaches lies in their treatment of triplets.sentence matching transforms kgc into a text matching task, as seen in simkgc @xcite , enabling description-based methods to surpass structurebased methods for the first time.in contrast, word matching involves predicting the masked positions in the input data and matching them with all candidate entities. prior word matching techniques, like c-lmke @xcite , have commonly employed a specific token, frequently denoted as 'ent_id', as a replacement for the original entity while handling candidate entities. as a consequence, the suboptimal representation of candidate entities significantly affects the overall performance.furthermore, they position the description of the known entity immediately following the entity, inevitably widening the gap between the predicted entity and the known entity. consequently, this introduces additional challenges for the model in predicting unknown entities.these two factors have led to a significant disparity between word matching methods and sentence matching methods. in this paper, inspired by simkgc @xcite , we introduce a new fuzzy operation to handle relations of candidate entities. we achieve this operation by simultaneously making predictions in both the head and tail encoders. moreover,we design prompts to narrow the distance between the known entity and the predicted entity, with the aim of strengthening their association. the objective of this fuzzy operation is to improve the word matching method based on descriptive embedding, resulting in enhanced single entity embedding representations. furthermore, graph-related tasks commonly encounter the issue of degree bias @xcite , where nodes with lower degrees tend to exhibit weaker representations and poorer downstream performance.we take inspiration from simcse @xcite and utilize the dropout mechanism of pre-trained language models to acquire additional positive samples, addressing the degree bias issue in knowledge graphs. we conduct experiments on three popular benchmark datasets, including wn18rr and fb15k-237. according to automated evaluation metrics mrr and hits@k (@xmath0, 3, 10}), our bi-lateral masking with prompt for knowledge graph completion (bmkgc) method achieves state-ofthe-art performance on the wn18rr dataset and competitive results on the fb15k-237 dataset. ## related work the knowledge graph is composed of triples, denoted as (h, r, t), where each triple consists of a head entity, @xmath1, connected to a tail entity, @xmath2, through a relationship, @xmath3. presently, knowledge graph completion approaches involve representing entities and relationships as vectors in a lower-dimensional space, known as knowledge graph embedding. these knowledge graph embedding are further categorized into structure-based and description-based embedding. structure-based knowledge embedding we categorize structural-based knowledge embedding into three types. firstly, the first type relies on translation-based techniques, such as transe @xcite , transh @xcite , and transr @xcite , which employ distance-based scoring functions. these methods generate embeddings for the head entity (h), relationship (r), and tail entity (t) based on specific translations, allowing assessment of the plausibility of triples (h, r, t) using distance scoring functions.secondly, the complex @xcite model utilizes factorization and complex embeddings to enhance the representation of entities and relationships. tucker @xcite considers knowledge graph completion as a 3-d binary tensor decomposition problem and explores the effectiveness of various factorization techniques.thirdly, knowledge graph embedding is approached as a deep learning task, utilizing various neural network architectures. conve @xcite adopts convolutional neural networks (cnns) to capture interactions between entities and relations, while compgcn @xcite improves knowledge graph representations by incorporating multi-layered information through graph convolutional networks (gcns) @xcite . simultaneously, hitter @xcite and coke @xcite leverage transformers to process information within the knowledge graph. these models demonstrate innovative methodologies based on different neural network architectures aimed at advancing the performance and precision of knowledge graph completion. description-based knowledge embedding recent studies have utilized pre-trained language models (plm) such as bert @xcite to improve the completion of knowledge graphs (kgs) by converting incomplete triples into natural language inputs. in this paper, we categorize descriptive-based embedding into two types: sentence matching and word matching. sentence matching involves dividing the triple (h, r, t) into the head entity h, connected by a relationship r, and the tail entity t. the goal is to determine the most plausible triple by evaluating the similarity of the semantic meanings between these two components. simkgc @xcite significantly enhances performance by introducing efficient contrastive learning. on the other hand, word matching methods such as mem-kgc @xcite mask the tail entity and consider the head entity and relationship as context for predicting the masked entity. c-lmke @xcite utilizes predictions for unknown tail entities from the input, together with representations of all entities within the same batch, to obtain the most probable tail entity. degree bias the c-lmke @xcite enhances the representations of long-tail entities in kgs by incorporating degrees and leveraging text information, effectively improving the performance of plm on kgs. while kg-mixup @xcite validates the presence of degree bias in embedding-based knowledge graph completion (kgc) and identifies its main factors, it is worth noting that kg-mixup only focuses on enhancing structural-based embedding methods. consequently, there is a lack of data augmentation techniques for descriptive-based embedding methods to enhance kg completion performance for longtail entities. our proposed bmkgc model aims to address this gap. ## methods this section provides a comprehensive introduction to bmkgc, as depicted in figure2. firstly, we present a concise definition of knowledge graphs and their relevance to link prediction. subsequently, we elucidate the principles and implementation specifics of our bidirectional masking technique. finally, a succinct overview of degree compensation is provided. ## analysis prompt in contrast to previous methods that rely on descriptions, our prompt aims to reduce the distance between the predicted tail entity, the head entity, and their relationship. we hypothesize that this approach strengthens the association between the entities, thereby improving the model's ability to make description-based predictions. to valieiffel tower: an iconic symbol of paris, stands tall as a testament to architectural... w/ token_id ent_id_i:the enchanting capital of france,known for its rich history...,attraction, @xcite ent_id_j: an iconic symbol of paris, stands tall as a testament to architectural.. degree bias in figure3, we analyze the performance of bmkgc and bmkgc without additional positive samples on wn18rr. the x-axis represents the degree of the tail entities that require completion, which is derived from the training set. we categorize them into four groups since entities with a degree exceeding one hundred are not within the scope of the degree bias problem in our research. the y-axis represents the mean reciprocal rank (mrr) of each group. by conducting a comparison, it is evident that the inclusion of extra positive samples enhances the mrr for entities with lower degrees, corroborating the effectiveness of our proposed method in mitigating the degree bias issue to some extent. ## conclusion this paper proposes bmkgc, a method that effectively enhances entity representation by predicting candidate entities during training to obtain improved single embedding representations. furthermore, we propose a simple method to increase positive samples, thus alleviating the issue of degree bias in the knowledge graph. extensive experimental results convincingly demonstrate that our approach achieves state-of-the-art performance. in future research, we will concentrate on exploring entity-related information within the plm, reducing the impact of noise generated during pretraining, and further advancing entity representation.