Inductive Logical Query Answering in Knowledge Graphs
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Formulating and answering logical queries is a standard communication interfacefor knowledge graphs (KGs) and their representations.
Alleviating the notoriousincompleteness of real-world KGs, neural methods achieved impressive resultsin link prediction and complex query answering tasks by learning representationsof entities, relations, and queries.
Still, most existing query answering methodsare inherently transductive and cannot be generalized to KGs containing new entities without retraining entity embeddings.
In this work, we study the inductivequery answering task where inference is performed on a graph containing newentities with queries over both seen and unseen entities.
To this end, we devisetwo mechanisms leveraging inductive node and relational structure representationspowered by graph neural networks (GNNs).
Experimentally, we show that inductive models are able to perform logical reasoning at inference time over unseennodes generalizing to graphs up to 500% larger than training ones.
Exploring theefﬁciency–effectiveness trade-off, we ﬁnd the inductive relational structure methodgenerally achieves higher performance, while the inductive node representationmethod is able to answer complex queries in the inference-only regime without anytraining on queries and scale to graphs of millions of nodes.
Introduction
Traditionally, querying knowledge graphs (KGs) is performed via databases using structured querylanguages like SPARQL.
Databases can answer complex queries relatively fast under the assumptionof completeness , i e., there is no missing information in the graph.
In practice, however, KGs arenotoriously incomplete [29].
Embedding-based methods that learn vector representations of entitiesand relations are known to be effective in simple link prediction predicting heads or tails of querypatterns (head, relation, ?)
, e.g., (Einstein, graduate, ? ), forming the ﬁeld of KG completion [1, 14].
Complex queries are graph patterns expressed in a subset of ﬁrst-order logic (FOL) with operatorssuch as intersection ( ^ ), union ( _ ), negation ( ¬ ) and existentially quantiﬁed ( 9 ) variables 1 , e.g., 26 ?
U. 9 V : Win ( NobelPrize , V ) ^ Citizen ( USA , V ) ^
Graduate ( V, U ) (Fig. 1).
Complex queriesdeﬁne a superset of link prediction on KGs.
The conventional link prediction task can be viewed as acomplex query with a single triplet pattern without logic operators, e.g., Citizen ( USA , V ) , whichwe also denote as a projection query.
To tackle complex queries on incomplete knowledge graphs, query embedding methods are proposedto execute logic operations in the latent space, including variants that employ geometric [12, 20, 35],probabilistic [21, 7], neural-symbolic [23, 6, 4], neural [18, 3], and GNN [9, 2] approaches forlearning entity, relation, and query representations.
Where did US citizens with Nobel Prize graduate?
(cid:1869) =
(cid:1874) .
(cid:1484)(cid:1873) : (cid:1849)(cid:1861)(cid:1866) (cid:1840)(cid:1867)(cid:1854)(cid:1857)(cid:1864) (cid:1842)(cid:1870)(cid:1861)(cid:1878)(cid:1857) , (cid:1873) (cid:1512) (cid:1829)(cid:1861)(cid:1872)(cid:1861)(cid:1878)(cid:1857)(cid:1866) (cid:1847)(cid:1845)(cid:1827) , (cid:1873) (cid:1512) (cid:1833)(cid:1870)(cid:1853)(cid:1856)(cid:1873)(cid:1853)(cid:1872)(cid:1857) ( (cid:1873) , (cid:1874) ) (cid:2273) (cid:3047)(cid:3045)(cid:3028)(cid:3036)(cid:3041)
Nobel Prize
USA
Einstein
University of Zurich
ETH Zurich (cid:2273) (cid:3049)(cid:3028)(cid:3039)(cid:3036)(cid:3031) ( (cid:2273) (cid:3047)(cid:3032)(cid:3046)(cid:3047) ) ( new nodes and edges )
Nobel Prize
USA
Einstein
Feynman
Princeton
University of Zurich
ETH Zurich win citizen graduate answers
Feynman and Princeton and edges such that the same query now has more answers.
However, this very fact of learning a separate embedding for each entity makes those methodsinherently transductive i.e., they are bound to the space of learned entities and can not be generalizedto unseen entities without retraining the whole embedding matrix which can be prohibitively expensivein large graphs.
The problem is illustrated in Fig.
1: given a graph about Einstein and a logicalquery Where did US citizens with Nobel Prize graduate? , transductive QE methods learn to executelogical operators and return the answer set { University of Zurich, ETH Zurich } .
Then, the 40 graph is updated with new nodes and edges about Feynman and Princeton , and the same query nowhas more correct answers { University of Zurich, ETH Zurich, Princeton } as new unseen 42 entities satisfy the query as well.
Such inductive inference is not possible for transductive models as they do not have representations fornew Feynman and Princeton nodes.
In the extreme case, inference graphs might be disconnectedfrom the training one and only share the set of relations.
Therefore, inductive capabilities are a keyfactor to enable transferring trained query answering models onto updated or entirely new KGs.
In this work, we study answering complex queries in the inductive setting, where the model has to dealwith unseen entities at inference time.
Inspired by recent advancement in inductive link prediction on
KGs [36, 10], we devise two solutions for learning inductive representations for complex query: 1)
The ﬁrst solution, NodePiece-QE, extends the inductive node representation model
NodePiece [10]for complex query answering.
NodePiece-QE learns inductive representations of each entity as afunction of tokens from a ﬁxed-size vocabulary, and answers complex query with a non-parametriclogical query executor [4].
The advantages of NodePiece-QE are that it only needs to be trainedon simple link prediction data, answers complex queries in the inference-only mode, and that it canscale to large KGs.
The second solution, NBFNet-QE, extends the inductive link prediction model
NBFNet [36] for complex query answering.
NBFNet-QE learns inductive representations of therelational structure without entity embeddings, and uses the relational structure between the queryconstants and the answers to make the prediction.
NBFNet-QE can be trained end-to-end on complexqueries, achieves much better performance than NodePiece-QE, but struggles to scale to large KGs.
To the best of our knowledge, this is the ﬁrst work to study complex logical query answering in theinductive setting.
Conducting experiments on a novel benchmarking suite of 10 datasets, we ﬁnd that(i) both inductive solutions exhibit non-trivial performance answering logical queries over unseenentities and query patterns; (ii) inductive models demonstrate out-of-distribution generalizationcapabilities to graphs up to 500% larger than training ones; (iii) akin to updatable databases, inductivemethods can successfully ﬁnd new correct answers to known training queries after adding newnodes and edges; (iv) the inductive node representation method scales to answering logical queriesover a graph of 2M nodes with 500k new, unseen nodes; (v) GNN-based models still exhibitdifﬁculties [17, 32] generalizing to larger graphs than they were originally trained on.
Related Work
Knowledge Graph Completion.
Knowledge graph completion, a.k.a.
simple link prediction, hasbeen widely studied in the transductive paradigm [5, 30, 24, 34], i.e., when inference is performedon the same training graph with a ﬁxed set of entities.
Generally, these methods learn a shallowembedding vector for each entity.
We refer the audience to respective surveys [1, 14] covering dozens of transductive embedding methods.
The emergence of message passing [11] and Graph
Neural Networks (GNNs) has led to more advanced, inductive representation learning approachesthat model entity or triplet representations as a function of the graph structure in its neighborhood.
GraIL [25] learns triplet representations based on the subgraph structure surrounding the two entities.
NeuralLP [31], DRUM [22] and NBFNet [36] learn the pairwise entity representations based onthe set of relation paths between two entities.
NodePiece [10] learns entity representations from aﬁxed-size vocabulary of tokens that can be anchor nodes in a graph or relation types.
Complex Query Answering.
In the complex (multi-hop) query answering setup with logicaloperators, existing models employ different approaches, e.g., geometric [12, 20, 35], probabilistic [21,7], neural-symbolic [23, 6, 4], neural [18, 3], and GNN [9, 2].
Still, all the approaches are created andevaluated exclusively in the transductive mode where the set of entities does not change at inferencetime.
To the best of our knowledge, there is no related work in inductive logical query answeringwhen an inference graph contains new entities.
With our work, we aim to bridge this gap and extendinductive representation learning algorithms to logical query answering.
In particular, we focus onthe inductive setup where an inference graph is a superset of a training graph 2 such that (i) inferencequeries require reasoning over both seen and new entities; (ii) original training queries might havemore correct answers at inference time with the addition of new entities.
Preliminaries and Problem Deﬁnition
Knowledge Graph and Inductive Setup.
Given a ﬁnite set of entities E , a ﬁnite set of relations R ,and a set of triples (edges) T =
( E ⇥ R ⇥ E ) , a knowledge graph G is deﬁned as G =
( E , R , T ) .
94 Accounting for the inductive setup, we deﬁne a training graph G train =
( E train , R , T train ) and inference 95 graph G inf =
( E inf , R , T inf ) such that E train ⇢ E inf and T train ⇢ T inf .
That is, the inference graph extends 96 the training graph with new entities and edges 3 .The inference graph G inf is an incomplete part of thenot observable complete graph ˆ G inf =
( E inf , R , ˆ T inf ) with ˆ T inf =
T inf [ T pred whose missing triples
T pred have to be predicted at inference time.
First-Order Logic Queries.
Applied to KGs, a ﬁrst-order logic (FOL) query Q is a formula thatconsists of constants
C ( C ✓ E ), variables V ( V ✓ E , existentially quantiﬁed), relation projections 101 R ( a, b ) denoting a binary function over constants or variables, and logic symbols ( 9 , ^ , _ , ¬ ). The 102 answers A
G ( Q ) to the query Q are assignments of variables in a formula such that the instantiatedquery formula is a subgraph of the complete graph ˆ G .
Fig.
1 illustrates the logical form of a query Where did US citizens with Nobel Prize graduate?
as?
U. 9 V : Win ( NobelPrize , V ) ^ Citizen ( USA , V ) ^
Graduate ( V, U ) where NobelPrize and USAare constants ; Win , Citizen , Graduate are relation projections (labeled edges); V, U - variablessuch that V is an existentially quantiﬁed free variable and U is the projected bound target of thequery.
Common for the literature, we aim at predicting assignments of the query target whereasassignments of intermediate variables might not always be explicitly interpreted depending on themodel architecture.
In the example, the answer set A G ( Q ) is a binding of a target variable U toconstants University of Zurich and ETH Zurich .
Inductive FOL Queries.
In the standard transductive query answering setup, query constants andvariables at both training and inference time belong to the same set of entities, i.e., C train =
C inf ✓
E , V train =
V inf ✓ E .
In the inductive setup covered in this work, query constants and variables at 115 inference time belong to a different and larger set of entities E inf from the inference graph G inf , i.e.,
C train ✓ E train , V train ✓ E train but C inf ✓ E inf , V inf ✓ E inf .
This also leads to the fact that training queries 117 executed over the inference graph might have more correct answers, i.e., A
G train ( Q ) ✓ A G inf ( Q ) .
For 118 example (cf.
Fig. 1), the inference graph is updated with new nodes Feynman , Princeton and theirnew respective edges.
The same query now has a larger set of intermediate variables satisfying theformula ( Feynman ) and an additional correct answer Princeton .
Therefore, inductive generalizationis essential for obtaining representations of such new nodes and enabling logical reasoning over bothseen and new nodes, i.e., ﬁnding more answers to known queries in larger graphs or answering newqueries with new constants.
In the following section, we describe two approaches for achievinginductive generalization with different parameterization strategies.
2 The set of relation types is ﬁxed.
[Figure]
Method
Inductive Representations of Complex Queries.
Given a complex query Q =
( C , R Q , G )) , the 127 goal is to rank all possible entities according to the query.
From a representation learning perspective,this requires us to learn a conditional representation function f
( e |C , R Q , G ) for each entity e 2 E .
129 Transductive methods learn a shallow embedding for each answer entity e 2 E , and, therefore, cannot 130 generalize to unseen entities.
For inductive methods, the function f ( e |C , R Q , G ) should generalize to 131 some unseen answer entity e 0 (or unseen constant entity c2 C 0 ) at inference time.
Here, we discuss 132 two solutions for devising such an inductive function.
The ﬁrst solution is to parameterize the representation of each entity e as a function of aninvariant vocabulary of tokens that does not change at training and inference.
Particularly, thevocabulary might consist of unique relation types R that are always the same for G train and G inf , and 136 we are able to infer the representation of an unseen answer entity (or an unseen constant entity) as afunction of its incident relations (cf.
Fig.
2 left).
The idea has been studied in NodePiece [10] forsimple link prediction.
Here, we adopt a similar idea to learn inductive entity representations forcomplex query answering.
Once we obtain the representations for unseen entities, we can use anyoff-the-shelf decoding method (e.g., CQD-Beam [4]) for predicting the answer to the complex query.
We denote this strategy as NodePiece-QE.
The second solution is to parameterize f ( e |C , R Q , G ) as a function of the relational structure .
143 Intuitively, an answer of a complex query can be decided solely based on the relational structurebetween the query constants and the answer (Fig. 1).
Even after anonymizing entity names (and,hence, not learning any explicit entity embedding), we are still able to infer Princeton as an answersince it forms a distinctive relational structure with the query constants and conforms to the querystructure.
Similarly, intermediate nodes will be deemed correct if they follow a relational structure.
In other words, we do not need to know the answer node is Princeton , but only need to knowthe relative position of Princeton w.r.t.
the constants like Nobel Prize and USA .
Based on thisidea, we design f ( e |C , R Q , G ) to be a relational structure search function.
Such an idea has been 151 studied in Neural Bellman-Ford Networks (NBFNet) [36] to search for a single relation in simplelink prediction.
Here, we chain several NBFNet instances with differentiable logic operations to learninductive complex query in an end-to-end fashion.
We denote this strategy as NBFNet-QE.
4.1 NodePiece-QE: Inductive Node Representation
Here, we aim at reconstructing node representations for seen and new entities without learning shallownode embedding vectors.
To this end, we employ NodePiece [10], a compositional tokenizationapproach that learns an invariant vocabulary of tokens shared between training and inference graphs.
Formally, given a vocabulary of tokens t i 2 T , each entity e i is deterministically hashed into a set of representative tokens e i =
[ t 1 , . .
t k ] .
An entity vector e i is then obtained as a function of tokenembeddings e i = f ✓ ([ t i , . . .
, t k ]) , t
i 2 T | T | ⇥ d where the encoder function f ✓ : R k ⇥ d !
R d isparameterized with a neural network ✓ .
Since the set of relation types R is invariant for training and inference graphs, we can learn relationembeddings R |R| ⇥ d and our vocabulary of learnable tokens T is comprised of distinct relation typessuch that entities are hashed into a set of unique incident relation types.
For example (cf.
Fig.
2 left),a middle node from a training graph G train is hashed with a set of relations e i =
[ ] that stands fortwo unique incoming relations and one unique outgoing relation .
Passing the hashes throughf ✓ , we can reconstruct the whole entity embedding matrix E |E train | ⇥ d .
Additionally, it is possible toenrich entity and relation embeddings by passing them through a relational GNN encoder [28] over atarget graph G : E 0 , R 0 =
GNN ( E , R , G ) .
In both ways, the entity embedding matrix E encodes ajoint probability distribution p ( h, r, t ) for all triples in a graph.
Having a uniform featurization mechanism for both seen and unseen entities, it is now possible toapply any previously-transductive complex query answering model with learnable entity embeddingsand logical operators [20, 9, 21, 6].
Moreover, it was recently shown [4] that a combination ofsimple link prediction pre-training and a non-parametric logical executor allows to effectively answercomplex FOL queries in the inference-only regime without training on any complex query sample.
We adopt this Continuous Query Decomposition algorithm with beam search (CQD-Beam) as themain query answering decoder.
CQD-Beam relies only on entity and relation embeddings E , Rpre-trained on a simple 1p link prediction task.
Then, given a complex query, CQD-Beam appliest-norms and t-conorms [16] that execute conjunctions ( ^ ) and disjunctions ( _ ) as non-parametricalgebraic operations in the embedding space, respectively.
In our inductive setup (Fig. 2), we train a NodePiece encoder f ✓ and relation embeddings R (andoptionally a GNN) on the 1p link prediction task over the training graph G train .
We then apply thelearned encoder to materialize entity representations of the inference graph E |E inf | ⇥ d and send themto CQD-Beam that performs a non-parametric decoding of complex FOL queries over new inferenceentities.
The inference-only nature of NodePiece-QE is designed to be challenging and probing theabilities for zero-shot generalization in performing complex logical reasoning over larger graphs.
4.2 NBFNet-QE: Inductive Relational Structure Representation
The second strategy relies on learning inductive relational structure representations instead of explicitnode representations.
Having the same set of relation types R at training and inference time, we canparameterize each entity based on the relative relational structure between it and the anchor nodesin a given query.
For instance (Fig. 2 right), given a query with a particular relational structureand a set of anchor nodes, the representation of each node captures its relational structure relativeto the anchor nodes.
Each neighborhood expansion step is equivalent to the projection step.
In ourexample, immediate neighboring nodes will capture the intersection pattern , and further nodes, inturn, capture the extended intersection-projection structure .
Therefore, a node is likely to be an answer if its captured (or predicted) relational structure conformswith the query relational structure.
As long as the set of relations is ﬁxed, relation projection isperformed in the same way for training or new unseen nodes.
The idea of a one-hop ( 1p ) projectionfor simple link prediction has been proposed by Neural Bellman-Ford Networks (NBFNet) [36].
In particular, given a relation projection query ( h, r, ?) , NBFNet assigns unique initial states h (0) toall nodes in a graph by applying an indicator function h (0) e
= I NDICATOR
( h, v, r ) , i.e., a head node his initialized with a learnable relation embedding r and all other nodes are initialized with zeros.
Then,
NBFNet applies L relational message passing GNN layers where each layer l has its own learnablerelation embedding matrix R l obtained as a projection of the initial relation R l = W l r + b l .
Finallayer representations h ( L ) are passed through an MLP and activation function  to get a probabilitydistribution over all nodes in a graph p ( t | h, r )
=
( MLP ( h ( L ) )) .
As each query spawns a uniquelyinitialized graph and message passing procedure, NBFNet is seen to be applying a labeling trick [33]to model a conditional probability distribution p ( t | h, r ) which is provably more expressive than ajoint distribution p ( h, r, t ) produced by standard graph encoders.
Applied to complex queries, chaining k
NBFNet instances allows to answer k -hop projection queries,e.g., two instances for 2p queries.
NBFNet-QE employs NBFNet as a trainable projection operator and endows it with differentiable, non-parametric product logic for modeling conjunction ( ^ ),disjunction ( _ ), and negation ( ¬ ) over the fuzzy sets of all entities x 2 [0 , 1] E , i.e., after applying alogical operator (discussed in Appendix A), each entity’s degree of truth is associated with a scalarin range [0 , 1] .
For the next hop projection, the indicator function initializes a node state with arelation vector r i weighted by a scalar probability predicted in the previous hop x e : h (0)
e =
x e r i .
Differentiable logical operators allow training NBFNet-QE end-to-end on complex queries.
Experiments
We designed the experimental agenda to demonstrate that inductive representation strategies are ableto: (1) answer complex logical queries over new, unseen entities at inference time, i.e., when queryanchors are new nodes (Section 5.2); (2) predict new correct answers for known training queries whenexecuted over larger inference graphs, i.e., when query anchors come from the training graph butvariables and answers belong to the larger inference graph (Section 5.3); (3) generalize to inferencegraphs of up to 500% larger than training graphs; (4) scale to inductive query answering over graphsof millions of nodes when updated with 500k new nodes and 5M new edges (Section 5.4).
5.1 Setup & Dataset
Dataset.
Due to the absence of inductive logical query benchmarks, we create a novel suite ofdatasets 4 based on FB15k-237 [26] (open license) and following the BetaE [21] query samplingmethodology.
Given a source graph with E entities, we sample |E train | = r · |E| , r 2 [0 . 1 , 0 . 9] nodes 230 to induce a training graph G train .
For validation and test graphs, we split the remaining set of entitiesinto two non-overlapping sets each with 1  r 2 |E| nodes.
We then merge training and unseen nodesinto the inference set of nodes E inf and induce inference graphs for validation and test from those sets,respectively, i.e., E valinf =
E train [ E val and E testinf =
E train [ E test .
That is, validation and test inferencegraphs both extend the training graph but their sets of new entities are disjoint.
Finally, we sample andremove 15% of edges T pred in the inference graphs as missing edges for link prediction pre-trainingand query sampling.
Overall, we sample 9 such datasets varying r to obtain ratios of inference graphsize to the training graph E inf / E train from 105% to 550%.
For each dataset, we employ the query sampler from BetaE [21] to extract 14 typical query types1p/2p/3p/2i/3i/ip/pi/2u/up/2in/3in/inp/pin/pni .
Training queries are sampled from the training graph
G train , validation and test queries are sampled from their respective inference graphs G inf where at 241 least one edge belongs to T pred and has to be predicted at inference time.
As inference graphs extend training graphs, training queries are very likely to have new answers beingexecuted over G inf with simple graph traversal and without any link prediction.
We create an additionalset of true answers for all training queries executed over the test inference graph G testinf to measure thegeneralization capabilities of query answering models.
This is designed to be an inference task andextends the faithfullness experiment [23].
Dataset statistics can be found in Appendix B.
Evaluation Protocol.
Following the literature [21], query answers are separated into two sets: easyanswers that only require graph traversal over existing edges, and hard answers that require inferringmissing links to achieve the answer node.
For the main experiment, evaluation involves ranking ofhard answers against all entities having easy ones ﬁltered out.
For evaluating training queries oninference graphs, we only have easy answers and rank them against all entities.
We report Hits@10as the main performance metric on different query types.
Implementation Details.
All NodePiece [10]-based models were pre-trained until convergence ona simple 1p link prediction task with the relations-only vocabulary and entity tokenization, MLPencoder, and ComplEx [27] scoring function.
We used a 2-layer CompGCN [28] as an optionalmessage passing encoder on top of NodePiece features.
The non-parametric CQD-Beam [4] decoder for answering complex queries is tuned for each query type based on the validation set ofqueries, most of the setups employ a product t-norm , sigmoid entity score normalization, and beamsize of 32.
Following the literature, the NBFNet-QE models were trained on 10 query patterns( 1p/2p/3p/2i/3i/2in/3in/inp/pin/pni ) where ip/pi/2u/up are only seen at inference time.
Each modelemploys a 4-layer NBFNet [36] as a trainable projection operator with DistMult [30] composi-
[Table]
tion function and PNA [8] aggregation.
Other logical operators ( ^ , _ , ¬ ) are executed with thenon-parametric product t-norm .
Both NodePiece-QE and NBFNet-QE are implemented 5 with PyTorch [19] and trained with the Adam [15] optimizer.
NodePiece-QE models were pre-trainedand evaluated on a single Tesla V100 32 GB GPU whereas NBFNet-QE models were trained andevaluated on 4 Tesla V100 16GB.
All hyperparameters are listed in Appendix D.
5.2 Complex Query Answering over Unseen Entities on Differently Sized Inference Graphs
First, we probe inference-only NodePiece-based embedding models and trainable NBFNet-QE inthe inductive setup, i.e., query answering over unseen nodes requiring link prediction over unseennodes.
Table 1 summarizes the results on a reference dataset with ratio E inf / E train of 175% while
Fig.
3 illustrates a bigger picture on all datasets (we provide a detailed breakdown by query type forall splits in Appendix C).
We observe that even inference-only models pre-trained solely on simple 1plink prediction exhibit non-trivial performance in answering queries with unseen entities.
Paired withan additional GNN encoder, the inference-only baseline exhibits signiﬁcantly better performanceover all query types and inference graphs up to 300% larger than training graphs.
The trainable NBFNet-QE models expectedly outperform non-trainable baselines and can tacklequeries with negation ( ¬ ).
Here, we conﬁrm that the labeling trick [33] and conditional p ( t | h, r )modeling better capture the relation projection problem than joint p ( h, r, t ) encoding approaches.
Still, all evaluated models with message passing, both inference-only NodePiece-QE with GNNand trainable NBFNet-QE, suffer from increasing the size of the inference graph and having moreunseen entities.
Reaching best results on E inf / E train ratios around 130%, both approaches steadilydeteriorate up until ﬁnal 550% by 20 absolute Hits@10 points on EPFO queries and negation queries.
We attribute this deterioration to the known generalization issues [17, 32] of message passing GNNswhen performing inference over much larger graph than the network has seen during training.
Onthe other hand, a simple NodePiece-QE model without message passing retains similar performanceindependently of the inference graph size.
Lastly, we observe that lower performance of inference-only NodePiece models can be also attributedto underﬁtting (cf.
train graph charts in Fig. 4).
Although 1p link predictors were trained untilconvergence (on the inductive validation set of missing triples), the performance of training queries on training graphs with easy answers that require only relation traversal without predicting missingedges is not yet saturated.
This fact suggests that better ﬁtting entity featurization (obtained by
NodePiece or other strategies) could further improve the test performance in the inference-onlyregime.
We leave the search of such strategies for future work.
5.3 Predicting New Answers for Training Queries on Larger Inference Graphs
Simulating the incremental addition of new edges in graph databases, we evaluate the performanceof our inference-only and trainable QE models on training queries on the original training graphand extended inference graph (with added test edges).
As databases are able to immediately retrievenew answers to known queries after updating the graph, we aim at exploring and quantifying thisbehaviour of neural reasoning models.
In this experiment, we probe training queries and their easyanswers that require performing only graph traversal without predicting missing links in the inferencegraph.
While execution of training queries over the training graph indicates how well the modelcould ﬁt training data, executing training queries over the bigger inference graph with new entitiesaims to capture basic reasoning capabilities of QE models in the inductive regime.
Particular challenges arising when executing training queries over a bigger graph are: (1) the samequeries can have more correct answers as more new nodes and edges satisfying the query patternmight have been added (as in Fig. 1); (2) more new entities create a “distractor” setting with morefalse positives.
Generally, evaluation of training queries on the inference graph can be consideredas an extended version of the faithfullness [23] evaluation that captures how well a trained modelcan answer original training queries, i.e., memorization capacity.
In all 9 datasets, most of trainingqueries have at least one new correct answer in the inference graph (more details in Appendix B).
Fig.
4 illustrates the performance of the inference-only NodePiece-QE (without and with GNN) andtrainable NBFNet-QE.
Generally, NBFNet-QE ﬁts the training query data almost perfectly conﬁrmingthe original ﬁnding [36] that NBFNet can perform graph traversal akin to symbolic rule-based models.
NBFNet-QE can also ﬁnd new correct answers on graphs up to 300% larger than training ones.
Then,the performance quickly deteriorates which we attribute to the distractor factor with more unseenentities and the already mentioned generalization issue on larger inference graphs.
The inference-only NodePiece-QE models, as expected, do not fully ﬁt the training data as they werenever trained on complex queries.
Still, the inference-only models exhibit non-trivial performancein ﬁnding more answers on graphs up to 200% larger than training ones with relatively smallperformance margins compared to training queries.
The most surprising observation is that GNN-free
NodePiece-QE models improve the performance on both training and inference graphs as the graphs(and the E inf / E train ratio) grow larger while GNN-enriched models steadily deteriorate.
We attributethis growth to the relation-based NodePiece tokenization and its learned features that tend to be morediscriminative in larger inference graphs where new nodes have smaller degree and thus can be betteridentiﬁed by their incident relation types.
We provide more experimental results for each dataset ratiowith breakdown by query type in Appendix C.
5.4 Scaling to Millions of Nodes on WikiKG-QE
Finally, we perform a scalability experiment evaluating complex query answering in the inductivemode on a new large dataset WikiKG-QE constructed from OGB WikiKG 2 [13] (CC0 license).
While the original task is transductive link prediction, we split the graph into a training graph of1.5M entities (5.8M edges, 512 unique relation types) and validation (test) graphs of 500k unseennodes (5M known and 600k missing edges) each.
The resulting validation (test) inference graphs aretherefore of 2M entities and 11M edges with the E inf / E train ratio of 133% (details are in Appendix B).
None of GNN-enabled models can scale to such sizes, so we use a basic inference-only NodePieceQE.
Due to the problem size, we only sample 10k EPFO queries of each type from the test inferencegraph to run in the inference-only regime.
Each query has at least one missing edge to be predicted atinference.
The answers are ranked against all 2M entities in the ﬁltered setting (in contrast to the
OGB task that ranks against 1000 pre-computed negative samples) and Hits@100 as the target metric.
We pre-train a NodePiece encoder (in addition to relation types, we tokenize nodes with a vocabularyof 20k anchor nodes, total 3M parameters in the encoder) with the ComplEx decoder on 1p linkprediction over the training graph for 1.5M steps (see Appendix D for hyperparameters).
Then, thegraph is extended with 500k new nodes and 5M new edges forming the inference graph.
Then, usingthe pre-trained encoder, we materialize representations of entities (both seen and new) and relationsfrom this inference graph.
Finally, CQD-Beam executes the queries against the bigger inferencegraph extended with 500k new nodes and 5M new edges.
[Table]
As shown in Table 2, we ﬁnd a non-trivial performance of the inference-only model on EPFO queriesdemonstrating that inductive node representation QE models are able to scale to graphs with hundredsof thousands of new nodes and millions of new edges in the zero-shot fashion.
That is, answeringcomplex queries over unseen entities is available right upon updating the graph without the needto retrain a model.
This fact paves the way for the concept of neural graph databases capable ofperforming zero-shot inference over updatable graphs without expensive retraining.
Limitations and Future Work
Limitations.
With the two proposed inductive query answering strategies, we observe a commontrade-off between the performance and computational complexity.
That is, inductive node representation models like NodePiece-QE are fast, scalable, and can be executed in the inference-onlyregime but underperform compared to the inductive relational structure representation models like
NBFNet-QE.
On the other hand, NBFNet-QE incurs high computational costs due to executing eachquery on a uniquely initialized graph instance.
Alleviating this issue is a key to scalability.
Societal Impact.
The inductive setup assumes running inference on (partly) unseen data, that is, thenature of this unseen data might be out-of-distrbution, unknown and potentially malicious.
This facthas to be taken into account when evaluating predictions and overall system trustworthiness.
Conclusion and Future Work.
In this work, we deﬁned the problem of inductive complex logicalquery answering and proposed two possible parameterization strategies based on node and relationalstructure representations to deal with new, unseen entities at inference time.
Experiments demonstrated that both strategies are able to answer complex logical queries over unseen entities as well asidentify new answers on larger inference graphs.
In the future work, we plan to extend the inductivesetup to completely disjoint training and inference graphs, expand the set of supported logical querypatterns aligned with popular queries over real-world KGs, enable reasoning over continuous featureslike texts and numbers, support more KG modalities like hypergraphs and hyper-relational graphs,and further explore the concept of neural graph databases.