Message Passing for Hyper-Relational Knowledge Graphs (2020)

Abstract: Hyper-relational knowledge graphs (KGs) (e.g., Wikidata) enable associating additional key-value pairs along with the main triple to disambiguate, or restrict the validity of a fact. In this work, we propose a message passing based graph encoder - StarE capable of modeling such hyper-relational KGs. Unlike existing approaches, StarE can encode an arbitrary number of additional information (qualifiers) along with the main triple while keeping the semantic roles of qualifiers and triples intact. We also demonstrate that existing benchmarks for evaluating link prediction (LP) performance on hyper-relational KGs suffer from fundamental flaws and thus develop a new Wikidata-based dataset - WD50K. Our experiments demonstrate that StarE based LP model outperforms existing approaches across multiple benchmarks. We also confirm that leveraging qualifiers is vital for link prediction with gains up to 25 MRR points compared to triple-based representations.

## introduction the task of link prediction over knowledge graphs (kgs) has seen a wide variety of advances over the years @xcite . the objective of this task is to predict new links between entities in the graph based on the existing ones. a majority of these approaches are designed to work over triplebased kgs, where facts are represented as binary relations between entities. this data model, however, doesn't allow for an intuitive representation of facts with additional information. for instance, in fig. 1 .a, it is non-trivial to add information which can help disambiguate whether the two universities attended by albert einstein awarded him with the same degree. this additional information can be provided in the form of key-value restrictions over instances of binary relations between entities in recent knowledge graph models @xcite @xcite . such restrictions are known as qualifiers in the wikidata statement model @xcite or triple metadata in rdf* @xcite and rdf reification approaches @xcite . these complex facts with qualifiers can be represented as hyper-relational facts (see sec. 3). in our example (fig. 1 .b), hyper-relational facts allow to observe that albert einstein obtained different degrees at those universities. existing representation learning approaches for such graphs largely treat a hyper-relational fact as an n-ary (@xmath0.g., educatedat academicdegree) @xcite losing entity-relation attribution; ignoring the semantic difference between a triple relation (educatedat) and qualifier relation (academicdegree) @xcite , or decomposing a hyper-relational instance into multiple quintuples comprised of a triple and one qualifier key-value pair @xcite . in this work, we propose an alternate graph representation learning mechanism capable of encoding hyper-relational kgs with arbitrary number of qualifiers, while keeping the semantic roles of qualifiers and triples intact. to accomplish this, we leverage the advances in graph neural networks (gnns), many of which are instances of the message passing @xcite framework, to learn latent representations of nodes and edges of a given graph. recently, gnns have been demonstrated @xcite to be capable of encoding mutli-relational (tripled based) knowledge graphs. inspired by them, we further extend this framework to incorporate hyperrelational kgs, and propose stare foot_0 , which to the best of our knowledge is the first gnn-based approach capable of doing so (see sec. 4). furthermore, we show that wikipeople @xcite , and jf17k @xcite -two commonly used benchmarking datasets for lp over hyper-relational kgs exhibit some design flaws, which render them as ineffective benchmarks for the hyper-relational link prediction task (see sec. 5). jf17k suffers from significant test leakage, while most of the qualifier values in wikipeople are literals which are conventionally ignored in kg embedding approaches, rendering the dataset largely devoid of qualifiers. instead, we propose a new hyper-relational link prediction dataset -wd50k extracted from wikidata @xcite ) that contains statements with varying amounts of qualifiers, and use it to benchmark our approach. through our experiments (sec. 6), we find that stare based model generally outperforms other approaches on the task of link prediction (lp) over hyper-relational knowledge graphs. we provide further evidence of the fact, independent of stare, that triples enriched with qualifier pairs provide additional signal beneficial for the lp task. ## related work early approaches for modelling hyper-relational graphs stem from conventional triple-based kg embedding algorithms, which often simplify complex property attributes (qualifiers). for instance, m-transh @xcite requires star-to-clique conversion which results in a permanent loss of entity-relation attribution. later models, e.g., rae @xcite , hype and hsimple introduced in @xcite , converted hyperrelational facts into n-ary facts with one abstract relation which is supposed to loosely represent a combination of all relations of the original fact. recently, getd @xcite extended tucker @xcite tensor factorization approach for n-ary relational facts. however, the model still expects only one relation in a fact and is not able to process facts of different arity in one dataset, e.g., 3-ary and 4-ary facts have to be split and trained separately. nalp @xcite ) is a convolutional model that supports multiple entities and relations in one fact. however, every complex fact with k qualifiers has to be broken down into k + 2 keyvalue pairs with an artificial split of the main (s,p,o) triple into (p s : s) and (p o : o) pairs. consequently, all key-value pairs are treated equally thus the model does not distinguish between the main triple and relation-specific qualifiers. hinge @xcite ) also adopts a convolutional framework for modeling hyper-relational facts. a main triple is iteratively convolved with every qualifier pair as a quintuple followed by min pooling over quintuple representations. although it retains the hyper-relational nature of facts, hinge operates on a triple-quintuple level that lacks granularity of representing a certain relation instance with its qualifiers. additionally, hinge has to be trained sequentially in a curriculum learning @xcite fashion requiring sorting all facts in a kg in an ascending order of the amount of qualifiers per fact which might be prohibitively expensive for large-scale graphs. instead, our approach directly augments a relation representation with any number of attached qualifiers properly separating auxiliary entities and relations from those in the main triple. additionally, we do not force any restrictions on input order of facts nor on the amount of qualifiers per fact. parallel to our approach are the methods that work over hypergraphs, e.g., dhne @xcite , hyper-sagnn @xcite , and knowledge hypergraphs like hype @xcite . we deem hyper-relational graphs and hypergraphs are conceptually different. as hyperedges contain multiple nodes, such hyperedges are closer to n-ary relations r(e 1 , . . . , e n ) with one abstract relation. the attribution of entities to the main triple or qualifiers is lost, and qualifying relations are not defined. combining a certain set of main and qualifying relations into one abstract r k () would lead to a combinatorial explosion of typed hyperedges since, in principle, any relation could be used in a qualifier, and there the amount of qualifiers per fact is not limited. therefore, modeling qualifiers in hypergraphs becomes non-trivial, and we leave such a study for future work. ## preliminaries gnns on undirected graphs: consider an undirected graph @xmath1, e), where v represents the set of nodes and e denotes the set of edges. each node @xmath2. in the message passing framework @xcite , the node representations are learned iteratively via aggregating representations (messages) from their neighbors: different gnn architectures implement their own aggregation and update strategy. for example, in case of graph convolutional networks (gcns) @xcite the representations of neighbours are first transformed via a weight matrix w and then combined and passed through a non-linearity f (•) such as relu. a gcn layer k can be represented as: gcn and other seminal architectures such as gat @xcite and gin @xcite do not model relation embeddings explicitly and require further modifications to support multirelational kgs. gnn on directed multi-relational graphs: in case of a multi-relational graph @xmath3, r, e) where r represents the set of relations r, and e denotes set of directed edges (s, r, o) where nodes @xmath4. the gcn formulation by @xcite assumes that the information in a directed edge flows in both directions. thus for each edge (s, r, o), an inverse edge (o, r -1 , s) is added to e. further, self-looping relations (v, r self , v), for each node @xmath5, enabling an update of a node state based on its previous one, and further improving normalization. for directed multi-relational graphs, equation 2 can be extended by introducing relation specific weights w r @xcite ) however, such networks are known to be overparameterized. instead, compgcn @xcite proposes to learn specific edge type vectors: ## stare in this section, we introduce our main contribution -stare, and show how we use it for link prediction (lp). stare (cf. fig. 2 for the intuition) incorporates statement qualifiers {(qr i , qv i )}, along with the main triple (s, r, o) into a message passing process. to do this, we extend equation 4 by combining the edge-type embedding h r with a fixed-length vector h q representing qualifiers associated with a particular relation r between nodes u and v. the resultant equation is thus: finally, the qualifier vector h q is obtained through a composition φ q of a qualifier relation h qr and qualifier entity h qv . the composition function φ q may be any entity-relation function akin to φ (equation 4 ). the representations of different this formalisation allows to (i) incorporate an arbitrary number of qualifier pairs and (ii) can take into account whether entities/relations occur in the main triple or the qualifier pairs. stare is the first gnn model for representation learning of hyperrelational kgs that has these characteristics. stare for link prediction. stare is a general representation learning framework for capturing the structure of hyper-relational graphs, and thus can be applied to multiple downstream tasks. in this work, we focus on lp and leave other tasks such as node classification for future work. in lp, given a query (s, r, q), the task is to predict an entity corresponding to the object position o. our link prediction model (see fig. 3 ) is composed of two parts namely (i) a stare based encoder, and (b) a transformer @xcite based decoder similar to coke @xcite , which are jointly trained. we initialize two embedding matrices r, v corresponding to relations (r), and entities (v) present in the dataset 2 . in every iteration, stare updates the embeddings ( r, v) by message passing across every edge in the training set. in the decoding step, we first linearize the given query, and use the updated embeddings ( r, v) to encode the entities and relations within it. then, this linearized sequence is passed through the transformer block, whose output is averaged to get a fixed-dimensional vector representation of the query. the vector is then passed through a fully-connected layer, multiplied with v and then passed through a sigmoid, to obtain a probability distribution over all entities. thereafter, it is trivial to retrieve the top n candidate entities for the o position in the query. note that we can use different decoders in this architecture. an explanation and evaluation of few decoders is provided in appendix d. ## wd50k dataset recent approaches @xcite @xcite for embedding hyperrelational kgs often use wikipeople and jf17k as benchmarking datasets. we advocate that those datasets can not fully capture the task complexity. in wikipeople, about 13% of statements contain at least one literal. literals (e.g. numeric values, date-time instances or other strings, etc) in kgs are conventionally ignored @xcite ) by embedding approaches, or are incorporated through specific means @xcite . however, after removing statements with literals, less than 3% of the remaining statements contain any qualifier pairs. out of those, about 80% possess only one 2 as mentioned in section 3, while pre-processing, we add inverse and self-loop relations to the graph. note, we retain the same set of qualifiers as in the original fact while generating inverse hyper-relational facts. the authors of jf17k reported foot_2 the dataset to contain redundant entries. in our own analysis, we detected that about 44.5% of the test statements share the same main (s, r, o) triple as the train statements. we consider this fact as a major data leakage which allows triple-based models to memorize subjects and objects appearing in the test set. to alleviate the above problems, we propose a new dataset, wd50k, extracted from wikidata statements. the following steps are used to sample our dataset from the wikidata rdf dump of august 2019 foot_3 . we begin with a set of seed nodes corresponding to entities from fb15k-237 having a direct mapping in wikidata (p646 "freebase id"). then, for each seed node, all statements whose main object and qualifier values correspond to wikibase:item are extracted. this step results in the removal of all literals in object position. similarly, all literals are filtered out from the qualifiers of the obtained statements. to increase the connectivity in the statements graph, all the entities mentioned less than twice are dropped. all the statements of wd50k are randomly split into the train, test, and validation sets. to eliminate test set leakages we remove all statements from train and validation sets that share the same main triple (s,p,o) with test statements. finally, we remove statements from the test set that contain entities and relations not present in the train or validation sets. wd50k contains 236,507 statements describing 47,156 entities with 532 relations where about 14% of statements have at least one qualifier pair. see ## experiments in this section, we discuss the setup and results of multiple experiments conducted towards (i) assessing the performance of our proposed approach on the link prediction task, and (ii) analyzing the effects of including hyper-relational information during link prediction. ## conclusion we presented stare, an instance of the message passing framework for representation learning over hyper-relational kgs. experimental results suggest that stare performs competitively on link prediction tasks over existing hyper-relational approaches and greatly outperforms triple-only baselines. in the future, we aim at applying stare for node and graph classification tasks as well as extend our approach to large-scale kgs. we also identified significant flaws in existing link prediction datasets and proposed wd50k, a novel, wikidata-based hyper-relational dataset that is closer to real-world graphs and better captures the complexity of the link prediction task. in the future, we plan to enrich wd50k entities with class labels and probe it against node classification tasks.