S t ATIK : Structure and Text for Inductive Knowledge Graph Completion (2022)

Abstract: Knowledge graphs (KGs) often represent knowledge bases that are incomplete. Machine learning models can alleviate this by helping automate graph completion. Recently, there has been growing interest in completing knowledge bases that are dynamic, where previously unseen entities may be added to the KG with many missing links. In this paper, we present StATIK – St ructure A nd T ext for I nductive K nowledge Completion. StATIK uses Language Models to extract the semantic information from text descriptions, while using Message Passing Neural Networks to capture the structural information. StATIK achieves state of the art results on three challenging inductive baselines. We further analyze our hybrid model through detailed ablation studies.

## introduction knowledge graphs (kgs) are appropriate representations of knowledge bases across many domains. these domains include commonsense reasoning @xcite @xcite , question answering @xcite @xcite @xcite , recommendation systems @xcite @xcite , and many others @xcite . these graphs are extremely large and often incomplete. as a result, there is significant interest in training machine learning models that can help complete these knowledge bases. in knowledge graphs, the nodes, called entities, often possess textual descriptions, while edges are typically labeled with one of many relation types, which may also possess textual descriptions. effective kg completion models should learn to leverage this textual information in order to correctly complete the knowledge base. additionally, such knowledge graphs are usually dynamic @xcite as fc barcelona a football team... instance of: -sports team league -la liga country -spain la liga spain sports team cádiz fc a football team... figure 1: depiction of the problem addressed by inductive learning. during training, only the blue portion of the graph exists, including the entities fc barcelona, sports team, la liga, and spain. later, the entity cádiz fc is added to the graph. when added, an entity contains a description and some number of edges (possibly zero). since statik is inductive, it requires no retraining or retroactive processing in any way to make predictions about cádiz fc. this could include predicting cádiz fc's country i.e. the query (cádiz fc, country, ?). the correct prediction, (cádiz fc, country, spain), is displayed in dashed red. a result of the underlying knowledge base being dynamic. typically, this manifests as nodes and edges being added and removed from the knowledge graph while the set of relation types remains more static. thus, another quality we desire of knowledge graph completion models, is that they be inductive and generalize to unseen entities. we propose a completely inductive, hybrid model, called statik, that effectively leverages both the structure of a knowledge graph as well as the underlying textual descriptions of the entities and relations. structure is incorporated through a message passing neural network (mpnn) @xcite that aggregates information from a neighborhood defined around each entity, while textual information is incorporated through a pretrained language model such as bert @xcite . our main contributions are summarized as follows: ## inductive representation learning on knowledge graphs we can define a knowledge graph with textual information as @xmath0, r, t , d) where e is the set of entities, r is the set of relation types, t is the set of triples (h, r, t) ∈ e × r × e, and d is the set of entity and relation descriptions. the inductive graph completion task is defined as follows. let the training graph be g @xmath1, r, t train , d train ) where e train is a subset of e, d train is the corresponding subset of d, and t train is the subset of t containing triples only involving entities in e train . the inference task is then to predict the triples in t /t train , only having trained the model on g train . specifically, for a given evaluation triple, t @xmath2, r, t), do head and tail prediction on the graph g -t i . this means that given a query of the form (h, r, ?) or (?, r, t), rank all possible tail or head candidates (targets) such that the real triple is ranked as highly as possible. figure 1 demonstrates a motivating example. similar to the transductive setting, such a task can be solved by learning a model that scores triples through minimizing some objective. we use a margin ranking loss as our objective where given a set of real triples, t , a corresponding set of negative triples, t ′ , and a scoring function f (higher scores imply more likely triple), we compute the loss as in order to learn the inductive objective, the model should avoid entity specific parameters such as an entity embedding tables as those parameters will not translate to the new entities at test time. to tackle this challenge statik uses text features instead of embeddings and extends prior work by also incorporating structural information through message passing. ## related work much of the work in the area of knowledge graph completion has focused on the transductive setting i.e. performing link prediction between entities seen at training time. generally, these methods learn embeddings in a geometric space such as transe @xcite , complex @xcite , distmult @xcite , rotate @xcite , and simple (kazemi and poole, 2018), or through a machine learning decoder such as conve and hyper @xcite . there has also been effort in using graph neural networks for knowledge graph completion. r-gcn @xcite brings the original gcn @xcite to the multi-relational knowledge graph setting. @xcite looked at using a modified version of gat @xcite to get strong results in the transductive setting. ## statik architecture we can conceptualize kg completion models as having an encoder and a decoder. whereas most kg models use an embedding table for the encoder, our model utilizes a hybrid language model and mpnn based network. our decoder is a relatively simple scoring function. ## experiments in this section we describe our thorough evaluation protocol. we describe the two settings in which we evaluate our inductive model and the infrastructure on which we implemented our model before demonstrating its effectiveness on three benchmark datasets. to analyze the importance of the various components of statik, we present the results of some detailed ablation studies. foot_1 ## conclusion and future work our work introduces and explores a new model for inductive knowledge graph completion that utilizes structural and textual information, and sets stateof-the-art on multiple benchmarks. ideas for future work include (i) testing this model with other types of graph neural networks that have already proved effective on kgs; (ii) testing alternative neighborhood sampling techniques beyond uniform random sampling; and (iii) extending this work to domains beyond text-based kgs.