Contrastive Learning with Generated Representations for Inductive Knowledge Graph Embedding (2023)

Abstract: With the evolution of Knowledge Graphs (KGs), new entities emerge which are not seen before. Representation learning of KGs in such an inductive setting aims to capture and transfer the structural patterns from existing entities to new entities. However, the performance of existing methods in inductive KGs are limited by sparsity and implicit transfer. In this paper, we propose VMCL, a Contrastive Learning (CL) framework with graph guided Variational autoencoder on Meta-KGs in the inductive setting. We first propose representation generation to capture the encoded and generated representations of entities, where the generated variations can densify representations with complementary features. Then, we design two CL objectives that work across entities and meta-KGs to simulate the transfer mode. With extensive experiments we demonstrate that our proposed VMCL can significantly outperform previous state-of-the-art baselines.

## introduction knowledge graphs (kgs) structure objective knowledge in the form of ("head entity", "relation", "tail entity") triples to express factual connections. representation learning of kgs aims to learn implicit representations (embeddings) of entities and then applies the learned representations to knowledge-intensive tasks such as link prediction @xcite and question answering @xcite . conventionally, kg are embedded in a transductive setting with a fixed predefined set of entities under the assumption that entities to be tested are seen during training. in this transductive setting, the learned entity representations from a source kg can easily be applied to a target kg @xcite . however, it is well admitted that kgs are not static, rather they evolve over time where new kgs figure 1 : example of inductive kgs with a source kg and a target kg. entities in the target kg are different from entities in the source kg. however, there are similarities in relational patterns, e.g., entity facebook in the source kg has relations: headquartered-in, economicsector, member-of, and entity 1 in the target kg has similar relation patterns. so it can be deduced that entity 1 may be a company like facebook. with a novel set of entities emerge. for kgs in such an inductive setting, the learned representations of existing (source) entities are not applicable to the new (target) entities @xcite . thus, the problem of capturing structural patterns from existing source entities and transferring them to a new set of target entities is of practical importance and posits a new set of interesting research challenges to tackle. we provide empirical insights about the representation learning problem in the inductive setting with an example in fig. 1 . entities in the target kg are new which are different from entities in the source kg. while there are similarities in structural patterns between the two kgs, e.g., entity 1 in the target kg may be a company like facebook in the source kg. representation learning of inductive kgs aims to capture the structural patterns from the source kg, then transfer them to help the learning for the target kg. note that what are transferred are structural patterns, not the learned representations because entities in the target and source kgs are different. several efforts devoted to representation learning of inductive kgs. amie @xcite , rulen @xcite , neural-lp @xcite and drum @xcite learn probabilistic logical and entity-independent rules from the source kg and apply such rules to the target kg. grail @xcite and com-pile @xcite learn the ability of relation prediction by subgraph extraction and graph neural networks independent of any specific entities, and generalize such ability to the target kg. recently, morse @xcite learns transferable and entity-independent meta knowledge by meta learning @xcite . although these methods have shown promising results for representation learning of inductive kgs, their performance is affected by at least two factors. first, their capability is limited by the sparsity of kgs caused by missing links. in four inductive kgs: n1-n4 @xcite , the degree of 65%, 47%, 46%, 42% entities is less than 3. second, effective transfer is often limited by the implicitness, e.g., morse uses meta learning to learn transfer patterns implicitly between source and target kgs, lacking explicit joint comparison across the entities and kgs. in this work, we propose vmcl, a contrastive learning (cl) framework with graph guided variational autoencoder on meta-kgs 1 in the inductive setting. fig. 2 gives the framework. we first propose representation generation to capture the encoded and generated representations of entities, where the generated variations can augment the representation space with complementary features (fig. 2 (1)). then, we design two cl objectives that work across entities and meta-kgs to 1 each meta-kg is a subgraph sampled randomly from the source kg and re-labeled to imitate a new kg. simulate the transfer mode (fig. 2 (2)). we perform extensive evaluation and theoretical derivation to understand the performance of vmcl. empirical results and analysis show the superiority of vmcl over state-of-the-art baselines. in summary, our key contributions are: ## method fig. 2 depicts the overall framework of our vmcl. in the following, we first introduce representation generation to capture the encoded and generated representations of entities in each meta-kg. then, we design two cl objectives that work across entities and meta-kgs to simulate the transfer mode. training procedure is described at the end.