Entity Prediction in Knowledge Graphs with Joint Embeddings (2021)

Abstract: Knowledge Graphs (KGs) have become increasingly popular in the recent years. However, as knowledge constantly grows and changes, it is inevitable to extend existing KGs with entities that emerged or became relevant to the scope of the KG after its creation. Research on updating KGs typically relies on extracting named entities and relations from text. However, these approaches cannot infer entities or relations that were not explicitly stated. Alternatively, embedding models exploit implicit structural regularities to predict missing relations, but cannot predict missing entities. In this article, we introduce a novel method to enrich a KG with new entities given their textual description. Our method leverages joint embedding models, hence does not require entities or relations to be named explicitly. We show that our approach can identify new concepts in a document corpus and transfer them into the KG, and we find that the performance of our method improves substantially when extended with techniques from association rule mining, text mining, and active learning.

## introduction knowledge graphs (kgs) have gained popularity as a versatile, general-purpose, and domainindependent model to represent information and are the major backbone for many applications on the web @xcite . kgs express knowledge as collections of head-relation-tail statements, named triples, e.g. (:cheney, :vice-of, :bush) expresses that cheney is the vice president of bush. since kgs are mostly built through automatic processes @xcite they are often incomplete, e.g. a kg may contain the fact that cheney was the vice-president of bush, but not that cheney is a us citizen. in addition, kgs evolve and require maintenance: they grow and change as the knowledge they describe expands and adapts to the real world. the problem of deriving missing portions of knowledge is known as kg completion. so far, the problem has been tackled by link prediction, i.e. finding relationships between previously known entities in the graph. in this paper, we focus on the problem of adding and integrating new entities into the kg-a task we call entity prediction. this is different from link prediction, where entities are ex-ante partially described in the kg. in entity prediction, we discover the existence of an entity from an external source and the kg neither contains the entity nor any information about how it relates to the other entities in the kg. as external source, we target document corpora, which describe the entities and the relations between them. for example, a document corpus like wikipedia contains a description of joe biden and his relations with obama (vice president) and @xcite . this lead to our core research question: given a kg g and a document corpus d, how can we complete g with entities (textually) described in d but not yet contained in g? as a solution, we represent the kg and document corpus in a common metric space and exploit this space in conjunction with user feedback and graph features to derive statements describing the new entities. specifically, we leverage joint embedding models for creating a numerical space to represent the kg and the background source. whereas kg embedding models give good performance on the link prediction task @xcite , joint embedding models combine a kg with a document corpus to draw conclusions in terms of similarity between documents and kg entities. our experiments determine that joint embedding models are well-founded methods to propose new entities to a kg. we also discuss how the prediction performance can be improved by integrating user feedback and explicit graph features. the next section discusses related literature and introduces relevant notations. section 4 outlines our solution based on joint embedding models, user feedback, and graph features. section 5 describes the experimental setup and evaluates our hypotheses. finally, section 6 presents overall conclusions and outlines future work. ## related work like in our scenario, ontology population and ontology enrichment foot_0 extract information from documents @xcite . ontology population adds instances to an existing ontology, whereas the structure of the ontology remains unchanged. in contrast to our problem it does not need to learn relations between instances and assumes an ontology to guide the information extraction process @xcite @xcite . ontology enrichment inserts new concepts or relations into an ontology. it differs from our setting in that it extends the schema of an ontology, using its concepts, instances, and schema constraints, while we solely rely on relationships between entities @xcite @xcite . kg enrichment aims at completing a given kg with new statements, or identifying erroneous ones @xcite , by predicting entity types @xcite or links @xcite . in contrast, our goal is to complete a kg by adding new entities and statements related to them. @xcite states that no such approach was known until 2015 and to the best of our knowledge, this has not changed meanwhile. automatic knowledge base construction (akbc) methods such as nell or openie approach a similar problem by means of text processing. they extract named entities and relations from a document, then arrange them as a kg @xcite @xcite . similarly, entity linking extracts named entities from text, then disambiguates and links them with a background database @xcite . these approaches assume that all entities and all relations are explicitly mentioned in the text under their canonical name. in contrast, we consider the scenario where entities are not stated in the text but described implicitly. ## background this section presents the key concepts and notation used throughout the paper. knowledge graphs. we define a knowledge graph as g := (v, r, e), with v a set of vertices (entities), r a set of relations, and e a set of directed edges, also known as statements. a statement is a triple (h, r, t) ∈ e, with h, @xmath0, and @xmath1. for example, the sentence "joe biden is the vice president of barack obama" is represented by the triple (:biden, :vice-of, :obama). let u be the the universe of the entities which can be described. v identifies the entities described in g, and v ⊂ u, i.e., g does not include all possible entities that may be described, which is the case in real kgs. ## approach as inputs, our approach receives a kg g, a document corpus d, and correspondences m between the two modalities. while every entity has at least one corresponding document, we assume a number of surplus documents in d that are not associated with any entity in g. a surplus document can either describe a novel entity we want to add to g or its association to an existing entity in g is unknown. the problem of entity prediction can then be divided into two subproblems: bush obama biden cheney usa preceded senior preceded senior c i t i z e n c i t i z e n senior senior vice-of vice-of s e n io r c i t i z e n c i t i z e n se ni or (a) the goal of entity prediction is to add the entity :biden and all red edges to the kg. (b) the entity embedding is inferred from the document embedding in a joint embedding space. :obama :u sa :bush :cheney :citizen 0.8 0.4 0.75 0.78 :senior 0.5 0.9 0.6 0.7 :preceded 0.6 0.98 0.7 0.65 :vice-of 0.1 0.8 0.3 0.5 (c) triple plausibility is estimated via the kg embedding loss on joint embeddings (lower is better). 1. identify whether a surplus document describes a novel entity. e.g. a document that describes the entity :biden which is not part of the kg in figure 1a. 2. add an entity e * to g and propose edges between e * and the graph's current entities v. e.g. in figure 1a, we would ideally add :biden and all red colored edges. we address both problems in the next sections by describing how our method adds one entity to the kg. for multiple entities, we repeat the procedure for each one independently, and leave an approach that updates the kg and its embeddings incrementally as future work to study. ## results in this section, we first describe the experimental setup, then discuss the novelty detection, and finally show the triple reconstruction results. ## conclusion and future work in this paper, we studied the problem of integrating new entities into a kg given their textual description. we exploited joint embeddings to identify entity candidates, and combined information from the joint embedding model with user feedback and graph features to improve the triple reconstruction. our method solely relies on structural patterns in the data and does not need explicit mentions of entities or relations in the text. our experiments suggest that joint embeddings are viable methods for entity prediction, and confirm that user feedback and graph features have a substantial impact on the triple reconstruction. in particular, experiments indicate that user feedback, features on relations (co), and features on entities (bm ) treat different aspects of the problem, making their combination more successful than using only one of them. comparing the results with the upper baselines shows that there is room for improvement. a possible way to fill this gap is to integrate explicit information into the process, e.g. considering the schema or the semantics of the relations or entities. another approach is to study the incremental addition of new entities and triples: we restore entities and triples independently of each other, however, the restoration provides new information that can be exploited subsequently. finally, our method could be extended in a straight-forward manner to other external data sources such as images, or to predict novel relations instead of entities.