Knowledge Router: Learning Disentangled Representations for Knowledge Graphs (2021)

Abstract: The design of expressive representations of entities and relations in a knowledge graph is an important endeavor. While many of the existing approaches have primarily focused on learning from relational patterns and structural information, the intrinsic complexity of KG entities has been more or less overlooked. More concretely, we hypothesize KG entities may be more complex than we think, i.e., an entity may wear many hats and relational triplets may form due to more than a single reason. To this end, this paper proposes to learn disentangled representations of KG entities - a new method that disentangles the inner latent properties of KG entities. Our disentangled process operates at the graph level and a neighborhood mechanism is leveraged to disentangle the hidden properties of each entity. This disentangled representation learning approach is model agnostic and compatible with canonical KG embedding approaches. We conduct extensive experiments on several benchmark datasets, equipping a variety of models (DistMult, SimplE, and QuatE) with our proposed disentangling mechanism. Experimental results demonstrate that our proposed approach substantially improves performance on key metrics.

## introduction knowledge graphs (kg) have emerged as a compelling abstraction for organizing structured knowledge. they have been playing crucial roles in many machine learning tasks. a knowledge graph represents a collection of linked data, describing entities of interest and relationships between them. to incorporate kgs into other machine learning systems, a prevalent way is mapping entities and relations of knowledge graphs into expressive representations in a low-dimensional space that preserves the relationships among objects, also known as knowledge graph embeddings. representative work such as @xcite @xcite @xcite has gained intensive attention across the recent years. the substantial effectiveness of recent work can be attributed to relational pattern modeling in which a suitable relational inductive bias is used to fit the structural information in data. nevertheless, these methods ignore the fact that the origination and formation of kgs can be rather complex @xcite . they may be collected, mined, handcrafted or merged in a complicated or convoluted process @xcite @xcite . to this end, entities in a knowledge graph may be highly entangled and relational triplets may form and be constructed for various reasons under a plethora of different circumstances or contexts. contextual reasons and/or domains may be taken into account at the same time. as such, it is only natural that kg embedding methods trained in this fashion would result in highly entangled latent factors. moreover, the existing holistic approaches fail to disentangle such factors and may result in sub-optimal solutions. recently, disentangled representation learning has achieved state-of-the-art performance and attracts much attention in the field of visual representation learning. a disentangled representation should separate the distinct, informative factors of variations in the data @xcite . disentangling the latent factors hidden in the observed data can not only increase the robustness, making the model less sensitive to misleading correlations but also enhance the model explainability. disentanglement can be achieved using either supervised signals or unsupervised approaches. zhu et al. @xcite propose to untangle the identity and view features in a supervised face recognition task. a bilinear model is adopted in @xcite to separate content from styles. there is also a large body of work on unsupervised disentangled representation learning @xcite @xcite . generally, the disentanglement mechanism is integrated into unsupervised learning frameworks such as variational autoencoders @xcite and generative adversarial networks @xcite . the quality of unsupervised disentangled representation can even match that learned from supervised label signals. inspired by the success of disentangled representation learning, we seek to enhance the disentanglement capability of entities representation in knowledge graphs. our hope is that this idea can address the aforementioned challenge in learning entity embeddings, that is, enabling the entities embeddings to better reflect the their inner properties. unlike learning disentangled representations in visual data, it is more challenging to disentangle the discrete relational data. most kgs embedding approaches operate at the triplet level, which is uninformative for disentanglement. intuitively, information about the entities resides largely within the graph encoded through neighborhood structures. our assumption is that an entity connects with a certain group of entities for a certain reason. for example, tim robbins, as an actor, starred in films such as the shawshank redemption; as a musician, is a member of the folk music group the highwaymen. we believe that relational triplets form because of different factors and this can be disentangled when looking it at the graph level. to summarize, our key contributions are: (1) we propose knowledge router (kr), an approach that learns disentangled representations for entities in knowledge graphs. specifically, a neighbourhood routing mechanism disentangles the hidden factors of entities from interactions with their neighbors. (2) knowledge router is model agnostic, which means that it can play with different canonical knowledge graph embedding approaches. it enables those models to have the capability in learning disentangled entity representations without incurring additional free parameters. (3) we conduct extensive experiments on four publicly available datasets to demonstrate the effectiveness of knowledge router. we apply knowledge router to models such as distmult, simple, and quate and observe a notable performance enhancement. we also conduct model analysis to inspect the inner workings of knowledge router. ## experiments in this section, we conduct experiments on several benchmark datasets to verify the effectiveness of the proposed approach. we target at answering: rq i: whether the disentanglement method can enhance the traditional knowledge graph embedding methods? rq ii: model-agnosticism: can it effectively work with different baseline models? rq iii: how do certain important hyper-parameters impact the model performance and what has the disentanglement algorithm learned? are they meaningful? ## conclusion in this paper, we present knowledge router, an algorithm for learning disentangled entity representations in knowledge graphs. our method is model agnostic and can be applied to many canonical knowledge graph embedding methods. extensive experiments on four benchmarking datasets demonstrate that equipping popular embedding models with the proposed knowledge router can outperform a number of recent strong baselines. via qualitative model analysis, we discover that knowledge router can effectively learns the hidden factors connecting entities, thus leading to disentanglement. we also showcase the impact of certain important hyper-parameters and give suggestions on hyperparameters tuning.