Edge: Enriching Knowledge Graph Embeddings with External Text (2021)

Abstract: Knowledge graphs suffer from sparsity which degrades the quality of representations generated by various methods. While there is an abundance of textual information throughout the web and many existing knowledge bases, aligning information across these diverse data sources remains a challenge in the literature. Previous work has partially addressed this issue by enriching knowledge graph entities based on “hard” co-occurrence of words present in the entities of the knowledge graphs and external text, while we achieve “soft” augmentation by proposing a knowledge graph enrichment and embedding framework named Edge. Given an original knowledge graph, we first generate a rich but noisy augmented graph using external texts in semantic and structural level. To distill the relevant knowledge and suppress the introduced noise, we design a graph alignment term in a shared embedding space between the original graph and augmented graph. To enhance the embedding learning on the augmented graph, we further regularize the locality relationship of target entity based on negative sampling. Experimental results on four benchmark datasets demonstrate the robustness and effectiveness of Edge in link prediction and node classification.

## introduction knowledge graph (kg) 1 embedding learning has been an emerging research topic in natural language processing, which aims to learn a low dimensional latent vector for every node. one major challenge is sparsity. knowledge graphs are often incomplete, and it is a challenge to generate low-dimensional representations from a graph with many missing edges. to mitigate this issue, auxil-1 knowledge graph usually represents a heterogeneous multigraph whose nodes and relations can have different types. however in the work, we follow @xcite , consider knowledge graph enrichment problem where only one relation type (connected or not) appears. iary texts that are easily accessible have been popularly exploited for enhancing the kg (as illustrated in figure 1 ). more specifically, given that kg entities contain textual features, we can link them to an auxiliary source of knowledge, e.g., wordnet, and therefore enhance the existing feature space. with notable exceptions, the use of external textual properties for kg embedding has not been extensively explored before. recently, @xcite used entities of the kg to query babelnet @xcite , added new nodes to the original kg based on co-occurrence of entities, and produced more meaningful embeddings using the enriched graph. however, this hard-coded, cooccurrence based kg enrichment strategy fails to make connections to other semantically related entities. as motivated in figure 1 , the newly added entities "wound", "arthropod" and "protective body", are semantically close to some input kg entity nodes (marked in red). however, they cannot be directly retrieved from babelnet using co-occurrence matching. in this paper, we aim to address the sparsity issue by integrating a learning component into the process. we propose a novel framework, edge, for kg enrichment and embedding. edge first constructs a graph using the external text based on similarity and aligns the enriched graph with the original kg in the same embedding space. it infuses learning in the knowledge distillation process by graph alignment, ensuring that similar entities remain close, and dissimilar entities get as far from each other. consuming information from an auxiliary textual source helps improve the quality of final products, i.e., low dimensional embeddings, by introducing new features. this new feature space is effective because it is obtained from a distinct knowledge source and established based on affinity captured by the learning component of our model. more specifically, our framework takes kg, and an external source of texts, t , as inputs, and generates an augmented knowledge graph, akg. in generating akg we are mindful of semantic and structural similarities among kg entities, and we make sure it contains all the original entities of kg. this ensures that there are common nodes in two graphs which facilitates the alignment process. to align kg and akg in the embedding space, a novel multi-criteria objective function is devised. in particular, we design a cost function that minimizes the distance between the embeddings of the two graphs. as a result, textual nodes (e.g., blue nodes in figure 1 ) related to each target entity are rewarded while unrelated ones get penalized in a negative sampling setting. extensive experimental results on four benchmark datasets demonstrate that edge outperforms state-of-the-art models in different tasks and scenarios, including link prediction and node classification. evaluation results also confirm the generalizability of our model. we summarize our contributions as follows: (i) we propose edge, a general framework to enrich knowledge graphs and node embeddings by exploiting auxiliary knowledge sources. (ii) we introduce a procedure to generate an augmented knowledge graph from external texts, which is linked with the original knowledge graph. (iii) we propose a novel knowledge graph embedding approach that optimizes a multi-criteria objective function in an end-to-end fashion and aligns two knowledge graphs in a joint embedding space. (iv) we demonstrate the effectiveness and generalizability of edge by evaluating it on two tasks, namely link prediction and node classification, on four graph datasets. the rest of the paper is organized as follows. in the next section, we try to identify the gap in the existing literature and motivate our work. next, in section 3, we set up the problem definition and describe how we approach the problem by in-depth explanation of our model. we evaluate our proposed model by experimenting link prediction and node classification on four benchmark datasets and present the results and ablation study in section 4. finally, we conclude our work and give the future direction in section 5. ## related work knowledge graph embedding learning has been studied extensively in the literature @xcite @xcite @xcite @xcite . a large number of them deal with the heterogeneous knowledge graph, where it appears different types of edges. while in this work we consider the type of knowledge graph with only one type (i.e. connected or not) of relation, and only focus on entity embedding learning. our work is related to graph neural networks, such as the graph convolutional networks (gcn) (kipf and welling, 2017) and its variants @xcite @xcite , which learn node embeddings by feature propagation. in the following, we mainly review the most relevant works in two aspects, i.e., graph embedding learning with external text and knowledge graph construction. ## experiment we design our experiments to investigate effectiveness of different components of edge as well as its overall performance. to this end, we aim to answer the following three questions 4 . q1 how well does edge perform compared to state-of-the-art in the task of link prediction? (section 4.1) q2 how is the quality of embeddings generated by edge compared to similar methods? (sections 4.2 and 4.3) q3 what is the contribution of each component (augmentation and alignment) in the overall performance? (section 4.4) 4 we plan to release our code upon publication. ## conclusion sparsity is a major challenge in kg embedding, and many studies failed to properly address this issue. we proposed edge, a novel framework to enrich kg and align the enriched version with the original one with the help of auxiliary text. using external source of information introduces new sets of features that enhance the quality of embeddings. we applied our model on three citation networks and one large scale medical knowledge graph. experimental results show that our approach outperforms existing graph embedding methods on link prediction and node classification.