SM i LE : Schema-augmented Multi-level Contrastive Learning for Knowledge Graph Link Prediction (2022)

Abstract: Link prediction is the task of inferring missing links between entities in knowledge graphs. Embedding-based methods have shown effectiveness in addressing this problem by modeling relational patterns in triples. However, the link prediction task often requires contextual information in entity neighborhoods, while most existing embedding-based methods fail to capture it. Additionally, little attention is paid to the diversity of entity representations in different contexts, which often leads to false prediction results. In this situation, we consider that the schema of knowledge graph contains the specific contextual information, and it is beneficial for preserving the consistency of entities across contexts. In this paper, we propose a novel Schema-augmented Multi-level contrastive LEarning framework (SMiLE) to conduct knowledge graph link prediction. Specifically, we first exploit network schema as the prior constraint to sample negatives and pre-train our model by employing a multi-level contrastive learning method to yield both prior schema and contextual information. Then we fine-tune our model under the supervision of individual triples to learn subtler representations for link prediction. Extensive experimental results on four knowledge graph datasets with thorough analysis of each component demonstrate the effectiveness of our proposed framework against state-of-the-art baselines. The implementation of SMiLE is available at https://github.com/GKNL/SMiLE.

## introduction knowledge graph (kg), as a well-structured representation of knowledge, stores a vast number of human knowledge in the format of triples- @xcite . kgs are essential components for various artificial intelligence applications, including question answering @xcite , recommendation systems @xcite , etc. in real world, kgs always suffer from the incompleteness problem, meaning that there are a large number of valid links in kg are missing. in this situation, link prediction techniques, which aim to automatically predict whether a relationship exists between a head entity and a tail entity, are essential for triple construction and verification. to address the link prediction problem in kg, a variety of methods have been proposed. traditional rule-based methods like markov logic networks @xcite and reinforcement learning-based method @xcite learn logic rules from kgs to conduct link prediction. the other mainstream methods are based on knowledge graph embeddings, including translational models like transe @xcite , transr @xcite and semantic matching models like rescal @xcite , distmult @xcite . besides, embedding-based methods leverage graph neural networks to explore graph topology @xcite and utilize type information @xcite to enhance representations in kg. nevertheless, the aforementioned methods fail to model the contextual information in entity neighbors. in fact, the context of an entity preserves specific structural and semantic information, and link prediction task is essentially dependent on the contexts related to specific entities and triples. furthermore, not much attention is paid to the diversity of entity representations in different contexts, which may often result in false predictions. quantitatively, dataset fb15k has 14579 entities and 154916 triples, and the number of entities with types is 14417 (98.89%). there are 13853 entities (95.02%) that have more than two types, and each entity has 10.02 types on average. for example, entity nicole kidman in figure 1 has two different types (actress and citizen), expressing different semantics in two different contexts. specifically, the upper left in the figure describes the contextual information in type level about "awards and works of nicole kidman as an actress". in this case, it is well-founded that there exists a relation between nicole kidman and 66 th cannes, and intuitively the prediction of (nicole kidman, ?, lane cove public school) does not make sense, since there is no direct relationship between type actress and type school. but considering that nicole kidman is also an australian citizen, it is hence reasonable to conduct such a prediction. we argue that the key challenge of preserving contextual information in embeddings is how to encapsulate complex contexts of entity neighborhoods. simply considering all information in the subgraph of entities as the context may bring in redundant and noisy information. schema, as a high-order meta pattern of kg, contains the type constraint between entities and relations, and it can naturally be used to capture the structural and semantic information in context. as for the problem of inconsistent entity representations, the diverse representations of an entity are indispensable to be considered in different contexts. as different schema defines diverse type restrictions between entities, it is able to preserve subtle and precise semantic information in a specific context. additionally, to yield consistent and robust entity representations for each contextual semantics, entities in contexts of the same schema are supposed to contain similar features but disparate in different contexts. to tackle the aforementioned issues, inspired by the advanced contrastive learning techniques, we proposed a novel schema-augmented multilevel contrastive learning framework to allow efficient link prediction in kgs. to tackle the incompleteness problem of kg schema, we first extract and build @xmath0, relation, @xmath1. then, we design a multi-level contrastive learning method under the guidance of schema. specifically, we optimize the contrastive learning objective in contextual-level and global-level of our model separately. in the contextual-level, contrasting entities within subgraphs of the same schema can learn semantic and structural characteristics in a specific context. in the global-level, differences and global connections between contexts of an entity can be captured via a cross-view contrast. overall, we exploit the aforementioned contrastive strategy to obtain entity representations with structural and high-order semantic information in the pre-train phase and then fine-tune representations of entities and relations to learn subtler knowledge of kg. to summarize, we make three major contributions in this work as follows: ## the proposed smile framework in this section, we first present notations related to this work. then we introduce the detail and training strategy of our proposed framework. the overall architecture of smile is shown in figure 2 . ## conclusion in this paper, we propose smile, a schemaaugmented multi-level contrastive learning framework for knowledge graph link prediction. we identify the critical issue of conducting effective link prediction is how to model precise and consistent contextual information of entities in different contexts. we propose an approach to automatically extract the complete schema from a kg. to fully capture contextual information of entities, we first sample the two-level negatives and perform contrast estimation in contextual-level and global-level, and then fine-tune the representations of entities and relations to learn subtler knowledge. empirical experiments on four benchmark datasets demonstrate that our proposed model effectively captures specific contextual information and correlations between different contexts of an entity.