Relation Specific Transformations for Open World Knowledge Graph Completion (2020)

Abstract: We propose an open-world knowledge graph completion model that can be combined with common closed-world approaches (such as ComplEx) and enhance them to exploit text-based representations for entities unseen in training. Our model learns relation-specific transformation functions from text-based to graph-based embedding space, where the closed-world link prediction model can be applied. We demonstrate state-of-the-art results on common open-world benchmarks and show that our approach benefits from relation-specific transformation functions (RST), giving substantial improvements over a relation-agnostic approach.

## introduction knowledge graphs are an interesting source of information that can be exploited by retrieval @xcite and question answering systems @xcite . they are, however, known to be inherently sparse @xcite . to overcome this problem, knowledge graph completion (kgc) enriches graphs with new triples. while most existing approaches require all entities to be part of the training graph, for many applications it is of interest to infer knowledge about entities not present in the graph i.e. open-world entities. here, approaches usually assume some text describing the target entity to be given, from which an entity representations can be inferred, for example via text embeddings @xcite . to the best of our knowledge, only a few such open-world kgc approaches have been proposed so far @xcite @xcite . we suggest a simple yet effective approach towards open-world kgc foot_0 : similar to @xcite 's owe model, our approach enables existing kgc models to perform open-world prediction: given an open-world entity, its name and description are aggregated into a text-based entity representation, and a transformation from text-based embedding space to graph-based embedding space is learned, where the closed-world kgc model can be applied. however, while owe's transformation only takes the open-world entity into account, our approach also utilizes the target triple's relation, such that specific mappings are learned for different relations such as birthplace, spouse, or located in (see figure 1 ). we demonstrate that this extension comes with strong improvements, yielding state-of-the-art results on common open-world datasets. ## related work interest in kgc has increased recently, with most of the work focusing on embedding-based approaches. earlier approaches @xcite have recently been complemented by other models such as distmult @xcite , transr @xcite , complex @xcite , proje @xcite and rotate @xcite . the above models estimate the probability of triples (head, rel, tail) using a scoring function φ(u head , u rel , u tail ), where u x denotes the embedding of entity/relation x and is a real-valued or complex-valued vector. φ depends on the model and varies from simple translation @xcite over bilinear forms @xcite to complex-valued word embeddings text-based knowledge graph average pooling train kgc model graph-based sheila walsh scottish contemporary vocalist and songwriter. open-world entity 1. build embedding spaces 2. learn relation-specific transformations 3. apply kgc model compute score figure 1: our approach first trains a kgc model on the graph without using textual information (bottom left). for every annotated entity, we extract a text-based embedding v by aggregating the word embeddings for tokens in the entity's name and description (top left). a transformation ψ is trained to map v to the space of graph-based embeddings (center). the learned mapping can then be applied to unknown entities, thus allowing the trained kgc model to be applied (right). while the knowledge graph completion models described above leverage only the structure of the graph, some approaches combine text with graph information, typically using embeddings that represent terms, sentences or documents @xcite . embeddings are usually derived from language models, either in static form @xcite or by contextualized models @xcite . kgc models can employ such textual information for entities scarcely linked in the graph @xcite . most approaches combine a textual embedding with structural kgc approaches, either by initializing structural embeddings from text @xcite , interpolating between textual and structural embeddings @xcite , sometimes with a joined loss @xcite and gating mechanisms @xcite . others perform a fine-tuning for kgc based on textual labels of the entities and relations @xcite . only few other works have addressed open-world kgc so far. @xcite proposed dkrl with a joint training of graph-based embeddings (transe) and text-based embeddings while regularizing both types of embeddings to be aligned using an additional loss. conmask @xcite is a text-centric approach where text-based embeddings for entities and relations are derived by an attention model over names and descriptions. closest to our work is owe @xcite , which trains graph and text embeddings independently and then learns a mapping between the two embedding spaces (more details are provided in section 3). while owe's mapping only take entities into account, our extended model's mapping is learned given both the entity and relation when predicting a triple. orthogonal to our work, wowe @xcite extends the owe approach by replacing the averaging aggregator with a weighted attention mechanism and can be combined with our approach. ## approach given a knowledge graph g ⊂ e×r×e containing triples (h, r, t) (e and r denote finite sets of entities and relations), kgc models can perform tail prediction as follows: given a pair of head and relation (h, r), the tail is estimated as we address an open-world setting, where the triple's head is not a part of the knowledge graph, i.e. test entity v h relation r nearest neighbors to ψ r (v h ) sheila walsh (scottish contemporary vocalist and songwriter) birthplace 1. nigel mcguiness (british) 2. darren burridge (british) 3. kim newman (british) recordlabel 1. picture this live (album) 2. thick as a brick (album) 3. whiplash smile (album) test entity v h relation r nearest neighbors to ψ r (v h ) ferlyn wong (singaporean singer, dancer and actress) occupation 1. park ji-yeon (singer) 2. kim sae-ron (actress) 3. park shin-hye (actress) genre 1. tony an (singer) 2. exodus (album) 3. s.e.s. (band) the key step of our approach is to learn transformation functions ψ r that align the text-based and graph-based embedding spaces such that ψ r (v h )≈u h . when applying this mapping, the open-world entity's text-based embedding v h is transformed into a graph-based proxy embedding ψ r (v h ). triples with h are scored by applying the kgc model from equation 1 to ψ r (v h ): like owe @xcite , we use an affine transformation ψ r (v) = a r •v + b r . our focus of this paper is to deal with relation specificity, for which we propose the following two strategies: relation specific transformation (rst) while the owe model consists of a global transformation function (ψ), our proposed rst approach trains a separate transformation function per relation (ψ r ): our hypothesis is that when predicting a tail (h, r, ?), including information on the relation r may be beneficial. have a look at relation clustering transformation (rct) our second approach -relation clustering transformation (rct) -aggregates relations to clusters and then learns a separate transformation ψ c for each cluster c. we use an agglomerative clustering approach (pseudo-code in figure 2 ), which first initializes each cluster with a single relation @xmath1, each joining "similar" clusters: let t(c) denote the set of tails attached to any relation r in c. clusters c, c are fused if the number of shared tails, divided by size of the smaller cluster, exceeds a threshold s: ## evaluation we evaluate our model on fb20k @xcite , dbpedia50k @xcite and fb15k-237-owe @xcite and compare our model with the state of the art on the task in open-world tail prediction. results are illustrated in ## conclusion we have proposed a simple approach to incorporate relation-specific information into open-world knowledge graph completion. our approach achieves state-of-the-art results on common open-source benchmarks and offers strong improvements over relation-agnostic state-of-the-art methods. an interesting direction for future work will be to adapt our model to longer textual inputs, e.g. by using attention to enable the model to select relevant passages (similar to conmask @xcite ).