Inductive Logical Query Answering in Knowledge Graphs
Abstract
Formulating and answering logical queries is a standard communication interface for knowledge graphs (KGs).
Alleviating the notorious incompleteness of realworld KGs, neural methods achieved impressive results in link prediction and complex query answering tasks by learning representations of entities, relations, and queries.
Still, most existing query answering methods rely on transductive entity embeddings and cannot generalize to KGs containing new entities without retraining the entity embeddings.
In this work, we study the inductive query answering task where inference is performed on a graph containing new entities with queries over both seen and unseen entities.
To this end, we devise two mechanisms leveraging inductive node and relational structure representations powered by graph neural networks (GNNs).
Experimentally, we show that inductive models are able to perform logical reasoning at inference time over unseen nodes generalizing to graphs up to 500% larger than training ones.
Exploring the efficiency–effectiveness trade-off, we find the inductive relational structure representation method generally achieves higher performance, while the inductive node representation method is able to answer complex queries in the inference-only regime without any training on queries and scales to graphs of millions of nodes. Code is available at https://github.com/DeepGraphLearning/InductiveQE .
Introduction
Traditionally, querying knowledge graphs (KGs) is performed via databases using structured query languages like SPARQL.
Databases can answer complex queries relatively fast under the assumption of completeness , i.e., there is no missing information in the graph.
In practice, however, KGs are notoriously incomplete [32].
Embedding-based methods that learn vector representations of entities and relations are known to be effective in simple link prediction predicting heads or tails of query patterns (head, relation, ?)
, e.g., (Einstein, graduate, ? ), as common in KG completion [1, 17].
Complex queries are graph patterns expressed in a subset of first-order logic (FOL) with operators such as intersection ( ∧ ), union ( ∨ ), negation ( ¬ ) and existentially quantified ( ∃ ) variables 1 , e.g., ?
U. ∃ V : Win ( NobelPrize , V ) ∧ Citizen ( USA , V )
∧ Graduate ( V, U ) (Fig. 1).
Complex queries define a superset of KG completion.
The conventional KG completion (link prediction) task can be viewed as a complex query with a single triplet pattern without logical operators, e.g., Citizen ( USA , V ) , which we also denote as a projection query.
Where did US citizens with Nobel Prize graduate?
𝑞𝑞 = 𝑣𝑣 . ∃𝑢𝑢 : 𝑊𝑊𝑊𝑊𝑊𝑊 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑃𝑃𝑃𝑃𝑊𝑊𝑃𝑃𝑁𝑁 , 𝑢𝑢 ∧ 𝐶𝐶𝑊𝑊𝐶𝐶𝑊𝑊𝑃𝑃𝑁𝑁𝑊𝑊 𝑈𝑈𝑈𝑈𝑈𝑈 , 𝑢𝑢 ∧ 𝐺𝐺𝑃𝑃𝐺𝐺𝐺𝐺𝑢𝑢𝐺𝐺𝐶𝐶𝑁𝑁 ( 𝑢𝑢 , 𝑣𝑣 ) 𝒢𝒢 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 Nobel
Prize
USA
Einstein
University of Zurich
ETH Zurich 𝒢𝒢 𝑣𝑣𝑡𝑡𝑣𝑣𝑡𝑡𝑣𝑣 ( 𝒢𝒢 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 ) ( new nodes and edges )
Nobel Prize
USA
Einstein
Feynman
Princeton
University of Zurich
ETH Zurich win citizen graduate answers
To tackle complex queries on incomplete knowledge graphs, query embedding methods are proposed to execute logic operations in the latent space, including variants that employ geometric [15, 23, 38], probabilistic [24, 9], neural-symbolic [26, 8, 5], neural [21, 4], and GNN [11, 3] approaches for learning entity, relation, and query representations.
However, this very fact of learning a separate embedding for each entity makes those methods inherently transductive i.e., they are bound to the space of learned entities and cannot generalize to unseen entities without retraining the whole embedding matrix which can be prohibitively expensive in large graphs.
The problem is illustrated in Fig.
1: given a graph about Einstein and a logical query Where did US citizens with Nobel Prize graduate? , transductive QE methods learn to execute logical operators and return the answer set { University of Zurich, ETH Zurich } .
Then, the graph is updated with new nodes and edges about Feynman and Princeton , and the same query now has more correct answers { University of Zurich, ETH Zurich, Princeton } as new unseen entities satisfy the query as well.
Such inductive inference is not possible for transductive models as they do not have representations for new Feynman and Princeton nodes.
In the extreme case, inference graphs might be disconnected from the training one and only share the set of relations.
Therefore, inductive capabilities are a key factor to transferring trained query answering models onto updated or entirely new KGs.
In this work, we study answering complex queries in the inductive setting, where the model has to deal with unseen entities at inference time.
Inspired by recent advancement in inductive Kg completion [42, 13], we devise two solutions for learning inductive representations for complex query: 1) The first solution, NodePiece-QE, extends the inductive node representation model NodePiece [13] to complex query answering.
NodePiece-QE learns inductive representations of each entity as a function of tokens from a fixed-size vocabulary, and answers complex query with a non-parametric logical query executor [5].
The advantages of NodePiece-QE are that it only needs to be trained on simple link prediction data, answers complex queries in the inference-only mode, and can scale to large KGs.
The second solution, GNN-QE [40], extends the inductive KG completion model
NBFNet [42] for complex query answering.
Originally, GNN-QE was studied only in the transductive setting.
Here, we analyze its inductive capabilities.
GNN-QE learns inductive representations of the relational structure without entity embeddings, and uses the relational structure between the query constants and the answers to make the prediction.
GNN-QE can be trained end-to-end on complex queries, achieves much better performance than NodePiece-QE, but struggles to scale to large KGs.
To the best of our knowledge, this is the first work to study complex logical query answering in the inductive setting without any additional features like entity types or textual descriptions .
Conducting experiments on a novel benchmarking suite of 10 datasets, we find that 1) both inductive solutions exhibit non-trivial performance answering logical queries over unseen entities and query patterns; 2) inductive models demonstrate out-of-distribution generalization capabilities to graphs up to 500% larger than training ones; 3) akin to updatable databases, inductive methods can successfully find new correct answers to known training queries after adding new nodes and edges; 4) the inductive node representation method scales to answering logical queries over a graph of 2M nodes with 500k new unseen nodes; 5) GNN-based models still exhibit some difficulties [20, 35] generalizing to graphs larger than those they were originally trained on.
Related Work
Knowledge Graph Completion.
Knowledge graph completion, a.k.a.
simple link prediction, has been widely studied in the transductive paradigm [6, 33, 27, 37], i.e., when training and inference are performed on the same graph with a fixed set of entities.
Generally, these methods learn a shallow embedding vector for each entity.
We refer the audience to respective surveys [1, 17] covering dozens of transductive embedding methods.
The emergence of message passing [14] and graph neural networks (GNNs) has led to more advanced, inductive representation learning approaches that model entity or triplet representations as a function of the graph structure in its neighborhood.
GraIL [28] learns triplet representations based on the subgraph structure surrounding the two entities.
NeuralLP [34], DRUM [25] and NBFNet [42] learn the pairwise entity representations based on the set of relation paths between two entities.
NodePiece [13] learns entity representations from a fixed-size vocabulary of tokens that can be anchor nodes in a graph or relation types.
Complex Query Answering.
In the complex (multi-hop) query answering setup with logical operators, existing models employ different approaches, e.g., geometric [15, 23, 38], probabilistic [24, 9], neural-symbolic [26, 8, 5], neural [21, 4], and GNN [11, 3].
Still, all the approaches are created and evaluated exclusively in the transductive mode where the set of entities does not change at inference time.
To the best of our knowledge, there is no related work in inductive logical query answering when the inference graph contains new entities.
With our work, we aim to bridge this gap and extend inductive representation learning algorithms to logical query answering.
In particular, we focus on the inductive setup where an inference graph is a superset of a training graph 2 such that 1) inference queries require reasoning over both seen and new entities; 2) original training queries might have more correct answers at inference time with the addition of new entities.
Preliminaries and Problem Definition
Knowledge Graph and Inductive Setup.
Given a finite set of entities E , a finite set of relations R , and a set of triples (edges) T =
( E × R × E ) , a knowledge graph G is defined as G =
( E , R , T ) .
Accounting for the inductive setup, we define a training graph G train =
( E train , R , T train ) and an inference graph G inf =
( E inf , R , T inf ) such that E train ⊂
E inf and T train ⊂ T inf .
That is, the inference graph extends the training graph with new entities and edges 3 .The inference graph G inf is an incomplete part of the not observable complete graph ˆ G inf =
( E inf , R , ˆ T inf ) with ˆ T inf =
T inf ∪ T pred whose missing triples T pred have to be predicted at inference time.
First-Order Logic Queries.
Applied to KGs, a first-order logic (FOL) query Q is a formula that consists of constants C ( C ⊆ E ), variables V ( V ⊆ E , existentially quantified), relation projections
R ( a, b ) denoting a binary function over constants or variables, and logic symbols ( ∃ , ∧ , ∨ , ¬ ).
The answers A G ( Q ) to the query Q are assignments of variables in a formula such that the instantiated query formula is a subgraph of the complete graph ˆ G .
Fig.
1 illustrates the logical form of a query Where did US citizens with Nobel Prize graduate?
as ?
U. ∃ V : Win ( NobelPrize , V ) ∧ Citizen ( USA , V )
∧ Graduate ( V, U ) where NobelPrize and USA are constants ; Win , Citizen , Graduate are relation projections (labeled edges); V, U - variables such that V is an existentially quantified free variable and U is the projected bound target variable of the query.
Common for the literature, we aim at predicting assignments of the query target whereas assignments of intermediate variables might not always be explicitly interpreted depending on the model architecture.
In the example, the answer set A G ( Q ) is a binding of a target variable U to constants University of Zurich and ETH Zurich .
Inductive FOL Queries.
In the standard transductive query answering setup, query constants and variables at both training and inference time belong to the same set of entities, i.e., C train =
C inf ⊆
E , V train =
V inf ⊆ E .
In the inductive setup covered in this work, query constants and variables at inference time belong to a different and larger set of entities E inf from the inference graph G inf , i.e.,
C train ⊆
E train , V train ⊆
E train but C inf
⊆
E inf , V inf ⊆
E inf .
This also leads to the fact that training queries executed over the inference graph might have more correct answers, i.e., A G train ( Q ) ⊆
A G inf ( Q ) .
For example (cf.
Fig. 1), the inference graph is updated with new nodes Feynman , Princeton and their new respective edges.
The same query now has a larger set of intermediate variables satisfying the formula ( Feynman ) and an additional correct answer Princeton .
Therefore, inductive generalization is essential for obtaining representations of such new nodes and enabling logical reasoning over both seen and new nodes, i.e., finding more answers to known queries in larger graphs or answering new queries with new constants.
In the following section, we describe two approaches for achieving inductive generalization with different parameterization strategies.
Method
Inductive Representations of Complex Queries.
Given a complex query Q =
( C , R Q , G ) , the goal is to rank all possible entities according to the query.
From a representation learning perspective, this requires us to learn a conditional representation function f
( e |C , R Q , G ) for each entity e ∈ E .
Transductive methods learn a shallow embedding for each answer entity e ∈ E , and, therefore, cannot generalize to unseen entities.
For inductive methods, the function f ( e |C , R Q , G ) should generalize to some unseen answer entity e ′ (or unseen constant entity c ′ ∈ C ′ ) at inference time.
Here, we discuss two solutions for devising such an inductive function.
The first solution is to parameterize the representation of each entity e as a function of an invariant vocabulary of tokens that does not change at training and inference.
Particularly, the vocabulary might consist of unique relation types R that are always the same for G train and G inf , and we are able to infer the representation of an unseen answer entity (or an unseen constant entity) as a function of its incident relations (cf.
Fig.
2 left).
The idea has been studied in NodePiece [13] for simple link prediction.
Here, we adopt a similar idea to learn inductive entity representations for complex query answering.
Once we obtain the representations for unseen entities, we can use any off-the-shelf decoding method (e.g., CQD-Beam [5]) for predicting the answer to the complex query.
We denote this strategy as NodePiece-QE.
The second solution is to parameterize f ( e |C , R Q , G ) as a function of the relational structure .
Intuitively, an answer of a complex query can be decided solely based on the relational structure between the query constants and the answer (Fig. 1).
Even after anonymizing entity names (and, hence, not learning any explicit entity embedding), we can still infer Princeton as an answer since it forms a distinctive relational structure with the query constants and conforms to the query structure.
Similarly, intermediate nodes will be deemed correct if they follow a relational structure .
In other words, we do not need to know the answer node is Princeton , but only need to know the relative position of Princeton w.r.t.
the constants like Nobel Prize and USA .
Based on this idea, we design f ( e |C , R Q , G ) to be a relational structure search function.
Such an idea has been studied in Neural Bellman-Ford Networks (NBFNet) [42] to search for a single relation in simple link prediction.
Applied to complex queries, GNN-QE [40] chains several NBFNet instances with differentiable logic operations to learn inductive complex query in an end-to-end fashion. So far,
GNN-QE was evaluated solely on transductive tasks. Here we extend it to the inductive setup.
4.1 NodePiece-QE: Inductive Node Representation
Here we aim at reconstructing node representations for seen and unseen entities without learning shallow node embedding vectors.
To this end, we employ NodePiece [13], a compositional tokenization approach that learns an invariant vocabulary of tokens shared between training and inference graphs.
Formally, given a vocabulary of tokens t i ∈ T , each entity e i is deterministically hashed into a set of representative tokens e i =
[ t 1 , . .
t k ] .
An entity vector e i is then obtained as a function of token embeddings e i =
f θ ([ t i , . . .
, t k ]) , t i ∈ R d where the encoder function f θ : R k ×
d
→ R d is parameterized with a neural network θ .
Since the set of relation types R is invariant for training and inference graphs, we can learn relation embeddings R ∈ R |R|× d and our vocabulary of learnable tokens T is comprised of distinct relation types such that entities are hashed into a set of unique incident relation types.
For example (cf. Fig.left), a middle node from a training graph G train is hashed with a set of relations e i =
[ ] that stands for two unique incoming relations and one unique outgoing relation .
Passing the hashes through f θ , we can reconstruct the whole entity embedding matrix E ∈ R |E train |× d .
Additionally, it is possible to enrich entity and relation embeddings by passing them through a relational GNN encoder [31] over a target graph G : E ′ , R ′ =
GNN ( E , R , G ) .
In both ways, the entity embedding matrix E encodes a joint probability distribution p ( h, r, t ) for all triples in a graph.
[Figure] Having a uniform featurization mechanism for both seen and unseen entities, it is now possible to apply any previously-transductive complex query answering model with learnable entity embeddings and logical operators [23, 11, 24, 8].
Moreover, it was recently shown [5] that a combination of simple link prediction pre-training and a non-parametric logical executor allows to effectively answer complex FOL queries in the inference-only regime without training on any complex query sample.
We adopt this Continuous Query Decomposition algorithm with beam search (CQD-Beam) as the main query answering decoder.
CQD-Beam relies only on entity and relation embeddings E , R pre-trained on a simple 1p link prediction task.
Then, given a complex query, CQD-Beam applies t-norms and t-conorms [19] that execute conjunctions ( ∧ ) and disjunctions ( ∨ ) as non-parametric algebraic operations in the embedding space, respectively.
In our inductive setup (Fig. 2), we train a NodePiece encoder f θ and relation embeddings R (and optionally a GNN) on the 1p link prediction task over the training graph G train .
We then apply the learned encoder to materialize entity representations of the inference graph E ∈ R |E inf |× d and send them to CQD-Beam that performs a non-parametric decoding of complex FOL queries over new unseen entities.
The inference-only nature of NodePiece-QE is designed to probe the abilities for zero-shot generalization in performing complex logical reasoning over larger graphs.
4.2 GNN-QE: Inductive Relational Structure Representation
The second strategy relies on learning inductive relational structure representations instead of explicit node representations.
Having the same set of relation types R at training and inference time, we can parameterize each entity based on the relative relational structure between it and the anchor nodes in a given query.
For instance (Fig. 2 right), given a query with a particular relational structure and a set of anchor nodes, the representation of each node captures its relational structure relative to the anchor nodes.
Each neighborhood expansion step is equivalent to a projection step in a complex query.
In our example, immediate neighboring nodes will capture the intersection pattern , and further nodes, in turn, capture the extended intersection-projection structure .
Therefore, a node is likely to be an answer if its captured (or predicted) relational structure conforms with the query relational structure.
As long as the set of relations is fixed, relation projection is performed in the same way for training or new unseen nodes.
The idea of a one-hop ( 1p ) projection for simple link prediction has been proposed by Neural Bellman-Ford Networks (NBFNet) [42].
In particular, given a relation projection query ( h, r, ?) , NBFNet assigns unique initial states h (0) to all nodes in a graph by applying an indicator function h (0)
=
I NDICATOR ( h, v, r ) , i.e., a head node h is initialized with a learnable relation embedding r and all other nodes are initialized with zeros.
Then,
NBFNet applies L relational message passing GNN layers where each layer l has its own learnable relation embedding matrix R ( l ) obtained as a projection (and reshaping) of thein initial relation R ( l ) = W ( l ) r + b ( l ) .
Final layer representations h ( L ) are passed through an MLP and the sigmoid function σ to get a probability distribution over all nodes in a graph p ( t
| h, r ) =
σ ( MLP ( h ( L ) )) .
As each projection query spawns a uniquely initialized graph and message passing procedure, NBFNet is seen to be applying a labeling trick [36] to model a conditional probability distribution p ( t | h, r ) that is provably more expressive than a joint distribution p ( h, r, t ) produced by standard graph encoders.
Applied to complex queries, chaining k
NBFNet instances allows us to answer k -hop projection queries, e.g., two instances for 2p queries.
GNN-QE employs NBFNet as a trainable projection operator and endows it with differentiable, non-parametric product logic for modeling conjunction ( ∧ ), disjunction ( ∨ ), and negation ( ¬ ) over the fuzzy sets of all entities x ∈ [0 , 1] E , i.e., after applying a logical operator (discussed in Appendix A), each entity’s degree of truth is associated with a scalar in range [0 , 1] .
For the i -th hop projection, the indicator function initializes a node state h (0) e with a relation vector r i weighted by a scalar probability predicted in the previous hop x e : h (0)
e =
x e r i .
Differentiable logical operators allow training GNN-QE end-to-end on complex queries.
Experiments
We designed the experimental agenda to demonstrate that inductive representation strategies are able to: 1) answer complex logical queries over new, unseen entities at inference time, i.e., when query anchors are new nodes (Section 5.2); 2) predict new correct answers for known training queries when executed over larger inference graphs, i.e., when query anchors come from the training graph but variables and answers belong to the larger inference graph (Section 5.3); 3) generalize to inference graphs of up to 500% larger than training graphs; 4) scale to inductive query answering over graphs of millions of nodes when updated with 500k new nodes and 5M new edges (Section 5.5).
5.1 Setup & Dataset
Dataset.
Due to the absence of inductive logical query benchmarks, we create a novel suite of datasets based on FB15k-237 [29] (open license) and following the query generation process of BetaE [24].
Given a source graph with E entities, we sample |E train | = r · |E| , r ∈ [0 . 1 , 0 . 9] nodes to induce a training graph G train .
For validation and test graphs, we split the remaining set of entities into two non-overlapping sets each with 1 − r 2 |E| nodes.
We then merge training and unseen nodes into the inference set of nodes E inf and induce inference graphs for validation and test from those sets, respectively, i.e., E valinf =
E train ∪ E val and E testinf =
E train ∪ E test .
That is, validation and test inference graphs both extend the training graph but their sets of new entities are disjoint.
Finally, we sample and remove 15% of edges T pred in the inference graphs as missing edges for sampling queries with those missing edges.
Overall, we sample 9 such datasets based on different choices of r , which result in the ratios of inference graph size to the training graph E inf / E train from 105% to 550%.
For each dataset, we employ the query sampler from BetaE [24] to extract 14 typical query types 1p/2p/3p/2i/3i/ip/pi/2u/up/2in/3in/inp/pin/pni .
Training queries are sampled from the training graph
G train , validation and test queries are sampled from their respective inference graphs G inf where at least one edge belongs to T pred and has to be predicted at inference time.
As inference graphs extend training graphs, training queries are very likely to have new answers when executed over G inf with simple graph traversal and without any link prediction.
We create an additional set of true answers for all training queries executed over the test inference graph G testinf to measure the entailment capabilities of query answering models.
This is designed to be an inference task and extends the faithfullness evaluation of [26].
Dataset statistics can be found in Appendix B.
Evaluation Protocol.
Following the literature [24], query answers are separated into two sets: easy answers that only require graph traversal over existing edges, and hard answers that require inferring missing links to achieve the answer node.
For the main experiment, evaluation involves ranking of hard answers against all entities having easy ones filtered out.
For evaluating training queries on inference graphs, we only have easy answers and rank them against all entities.
We report Hits@ as the main performance metric on different query types.
Implementation Details.
All NodePiece-based models [13] were pre-trained until convergence on a simple 1p link prediction task with the relations-only vocabulary and entity tokenization, MLP encoder, and ComplEx [30] scoring function. We used a 2-layer CompGCN [31] as an [Figure]
[Figure] [Figure] [Table] optional message passing encoder on top of NodePiece features.
The non-parametric CQD-Beam [5] decoder for answering complex queries is tuned for each query type based on the validation set of queries, most of the setups employ a product t-norm , sigmoid entity score normalization, and beam size of 32.
Following the literature, the GNN-QE models [40] were trained on 10 query patterns ( 1p/2p/3p/2i/3i/2in/3in/inp/pin/pni ) where ip/pi/2u/up are only seen at inference time.
Each model employs a 4-layer NBFNet [42] as a trainable projection operator with DistMult [33] composition function and PNA [10] aggregation.
Other logical operators ( ∧ , ∨ , ¬ ) are executed with the nonparametric product t-norm and t-conorm .
Both NodePiece-QE and GNN-QE are implemented 4 with PyTorch [22] and trained with the Adam [18] optimizer.
NodePiece-QE models were pre-trained and evaluated on a single Tesla V100 32 GB GPU whereas GNN-QE models were trained and evaluated on 4 Tesla V100 16GB.
All hyperparameters are listed in Appendix D. To show that the proposed models are non-trivial, we compare them with an Edge-type Heuristic baseline (Appendix E), which selects all entities that satisfy the relations in the last hop of the query in G inf .
5.2 Complex Query Answering over Unseen Entities on Differently Sized Inference Graphs
First, we probe inference-only NodePiece-based embedding models and trainable GNN-QE in the inductive setup, i.e., query answering over unseen nodes requiring link prediction over unseen nodes.
As a sanity check, we compare them to the Edge-type Heuristic and a transductive BetaE model [24] trained with standard hyperparameters (Appendix D) on the reference dataset (with ratio E inf / E train of 175%) with randomly initialized embeddings for unseen nodes at inference time.
Table 1 summarizes the results on the reference dataset while Fig.
3 illustrates a bigger picture on all datasets (we provide a detailed breakdown by query type for all splits in Appendix C).
The experiment on the tranductive
BetaE confirms that pure transductive models can not generalize to graphs with unseen nodes.
With inductive models, however, we observe that even inference-only models pre-trained solely on simple 1p link prediction exhibit non-trivial performance in answering queries with unseen entities.
Particularly, the inference-only NodePiece with GNN baseline exhibits better performance over all query types and inference graphs up to 300% larger than training graphs. [Figure]
The trainable GNN-QE models expectedly outperform non-trainable baselines and can tackle queries with negation ( ¬ ).
Here, we confirm that the labeling trick [36] and conditional p ( t | h, r ) modeling better capture the relation projection problem than joint p ( h, r, t ) encoding approaches.
Still, we notice that models with GNNs, i.e., inference-only NodePiece-QE with GNN and trainable GNN-QE, suffer from increasing the size of the inference graph and having more unseen entities.
Reaching best results on E inf / E train ratios around 130%, both approaches steadily deteriorate up until final 550% by 20 absolute Hits@10 points on EPFO queries and negation queries.
We attribute this deterioration to the known generalization issues [20, 35] of message passing GNNs when performing inference over a larger graph than the network has seen during training.
Recently, a few strategies have been proposed [7, 39] to alleviate this issue and we see it as a promising avenue for future work.
On the other hand, a simple NodePiece-QE model without message passing retains similar performance independently of the inference graph size.
Lastly, we observe that lower performance of inference-only NodePiece models can be also attributed to underfitting (cf.
train graph charts in Fig. 4).
Although 1p link predictors were trained until convergence (on the inductive validation set of missing triples), the performance of training queries on training graphs with easy answers that require only relation traversal without predicting missing edges is not yet saturated.
This fact suggests that better fitting entity featurization (obtained by
NodePiece or other strategies) could further improve the test performance in the inference-only regime.
We leave the search of such strategies for future work.
5.3 Predicting New Answers for Training Queries on Larger Inference Graphs
Simulating the incremental addition of new edges in graph databases, we evaluate the performance of our inference-only and trainable QE models on training queries on the original training graph and extended inference graph (with added test edges).
As databases are able to immediately retrieve new answers to known queries after updating the graph, we aim at exploring and quantifying this behaviour of neural reasoning models.
In this experiment, we probe training queries and their easy answers that require performing only graph traversal without predicting missing links in the inference graph.
While execution of training queries over the training graph indicates how well the model could fit training data, executing training queries over the bigger inference graph with new entities aims to capture basic reasoning capabilities of QE models in the inductive regime.
Particular challenges arising when executing training queries over a bigger graph are: (1) the same queries can have more correct answers as more new nodes and edges satisfying the query pattern might have been added (as in Fig. 1); (2) more new entities create a “distractor” setting with more false positives.
Generally, evaluation of training queries on the inference graph can be considered as an extended version of the faithfullness [26] evaluation that captures how well a trained model can answer original training queries, i.e., memorization capacity.
In all 9 datasets, most of training queries have at least one new correct answer in the inference graph (more details in Appendix B).
Fig.
4 illustrates the performance of the Edge-type Heuristic baseline, inference-only NodePiece-QE (without and with GNN) and trainable GNN-QE models.
Generally, GNN-QE fits the training query data almost perfectly confirming the original finding [42] that NBFNet performs graph traversal akin to symbolic models.
GNN-QE can also find new correct answers on graphs up to 300% larger than training ones.
Then, the performance deteriorates which we attribute to the distractor factor with more unseen entities and the mentioned generalization issue on larger inference graphs.
The inference-only NodePiece-QE models, as expected, do not fully fit the training data as they were never trained on complex queries.
Still, the inference-only models exhibit non-trivial performance (compared to the Edge-type Heuristic) in finding more answers on graphs up to 200% larger than training ones with relatively small performance margins compared to training queries.
The most surprising observation is that GNN-free NodePiece-QE models improve the performance on both training and inference graphs as the graphs (and the E inf / E train ratio) grow larger while GNN-enriched models steadily deteriorate.
We attribute this growth to the relation-based NodePiece tokenization and its learned features that tend to be more discriminative in larger inference graphs where new nodes have smaller degree and thus can be better identified by their incident relation types.
We provide more experimental results for each dataset ratio with breakdown by query type in Appendix C.
5.4 Ranking of Easy and Hard Answers
In addition to evaluating faithfullness that measures whether a model could recover easy answers, it is also insightful to measure whether all easy answers can be ranked higher than hard answers. That is, a reliable query answering model would first recover all possible easy answers and would enrich the answer set with highly-probable hard answers. To this end, we apply a ROC AUC metric over original unfiltered scores.
The ROC AUC score measures how many hard answers are ranked after easy answers.
Note that the score does not depend on actual values of ranks, that is, the metric will be high when easy answers are ranked, e.g., in between 100-1000 as long as hard answers are ranked 1001 and lower.
Therefore, ROC AUC still needs to be paired with MRR to see how easy and hard answers are ranked absolutely.
[Table]
We compute ROC AUC for each query and average them over each query type thus making it macro-averaged ROC AUC .
Our experimental results on all query types using the models reported in Table 1 on the reference 175% dataset are compiled in Table
[Table]
GNN-QE has nearly perfect ROC AUC scores as it was trained on complex queries.
NodePiece-QE models are acceptable for inference-only models that were only trained only on 1p simple link prediction and have never seen any complex query at training time.
5.5 Scaling to Millions of Nodes on WikiKG-QE
Finally, we perform a scalability experiment evaluating complex query answering in the inductive mode on a new large dataset WikiKG-QE constructed from OGB WikiKG 2 [16] (CC0 license).
While the original task is transductive link prediction, we split the graph into a training graph of 1.5M entities (5.8M edges, 512 unique relation types) and validation (test) graphs of 500k unseen nodes (5M known and 600k missing edges) each.
The resulting validation (test) inference graphs are therefore of 2M entities and 11M edges with the E inf / E train ratio of 133% (details are in Appendix B).
GNN-QE cannot scale to such sizes, so we only probe NodePiece-QE models.
Due to the problem size, we only sample 10k EPFO queries of each type from the test inference graph to run in the inference-only regime.
Each query has at least one missing edge to be predicted at inference.
The answers are ranked against all 2M entities in the filtered setting (in contrast to the OGB task that ranks against 1000 pre-computed negative samples) and Hits@100 as the target metric.
We pre-train a NodePiece encoder (in addition to relation types, we tokenize nodes with a vocabulary of 20k nodes, total 3M parameters in the encoder) with the ComplEx decoder on 1p link prediction over the training graph for 1M steps (see Appendix D for hyperparameters).
Then, the graph is extended with 500k new nodes and 5M new edges forming the inference graph.
Then, using the pre-trained encoder, we materialize representations of entities (both seen and new) and relations from this inference graph.
Finally, CQD-Beam executes the queries against the bigger inference graph extended with 500k new nodes and 5M new edges.
[Table]
As shown in Table 3, we find a non-trivial performance of the inference-only model on EPFO queries demonstrating that inductive node representation QE models are able to scale to graphs with hundreds of thousands of new nodes and millions of new edges in the zero-shot fashion.
That is, answering complex queries over unseen entities is available right upon updating the graph without the need to retrain a model.
This fact paves the way for the concept of neural graph databases capable of performing zero-shot inference over updatable graphs without expensive retraining.
Limitations and Future Work
Limitations.
With the two proposed inductive query answering strategies, we observe a common trade-off between the performance and computational complexity.
That is, inductive node representation models like NodePiece-QE are fast, scalable, and can be executed in the inference-only regime but underperform compared to the inductive relational structure representation models like
GNN-QE.
On the other hand, GNN-QE incurs high computational costs due to executing each query on a uniquely initialized graph instance.
Alleviating this issue is a key to scalability.
Societal Impact.
The inductive setup assumes running inference on (partly) unseen data, that is, the nature of this unseen data might be out-of-distrbution, unknown and potentially malicious.
This fact has to be taken into account when evaluating predictions and overall system trustworthiness.
Conclusion and Future Work.
In this work, we defined the problem of inductive complex logical query answering and proposed two possible parameterization strategies based on node and relational structure representations to deal with new, unseen entities at inference time.
Experiments demonstrated that both strategies are able to answer complex logical queries over unseen entities as well as identify new answers on larger inference graphs.
In the future work, we plan to extend the inductive setup to completely disjoint training and inference graphs, expand the set of supported logical query patterns aligned with popular queries over real-world KGs, enable reasoning over continuous features like texts and numbers, support more KG modalities like hypergraphs and hyper-relational graphs, and further explore the concept of neural graph databases.
Acknowledgments and Disclosure of Funding
This project is supported by the Natural Sciences and Engineering Research Council (NSERC)
Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft
Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI
Lab Rhino-Bird Gift Fund and a NRC Collaborative R&D Project (AI4D-CORE-06).
This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-3583139727.
The computation resource of this project is supported by Calcul Québec 5 and Compute Canada 6 .
We would like to
A Differentiable Logical Operators
T-norms ( ⊤ ) and t-conorms ( ⊥ ) are fuzzy versions of conjunction ( ∧ ) and disjunction ( ∨ ), respectively.
Fuzzy operators can be applied to vectors of continuous values within a certain range, e.g., [0 , 1] d , depending on the chosen fuzzy logic , and are executed as algebraic operations which makes them differentiable.
Different fuzzy logics implement different t-norms and t-conorms.
In this work, we experiment with two such logics: product logic and Gödel (min) logic .
In the product logic, conjunction C , disjunction D , and negation N are modeled as follows:
[Equation]
where inputs x , y ∈ [0 , 1] d are d -dimensional vectors with values in the range [0 , 1] , ⊙ is the element-wise multiplication, and 1 is the universe vector of all ones.
In the Gödel logic, conjunction C and disjunction D are modeled as min and max , respectively:
[Equation]
For GNN-QE we employ solely the product logic for end-to-end training on all types of complex queries.
For NodePiece-QE and its inference-only mechanism based on CQD-Beam, we may select the best performing logic for each query type based on the validation set.
The chosen operators for
NodePiece-QE are reported in Table 13 in Appedix D.
B Benchmarking Datasets Details
We sampled 9 datasets (used in Section 5.2 and Section 5.3) from the original FB15k-237 [29] with already added inverse edges for ensuring reachability and connectedness of the underlying graph for the subsequent query sampling.
Creation details are provided in the Section 5.1 and statistics on the sampled graphs are presented in Table 4.
Varying the ratio of entities in the inference graph to the training graph E inf / E train , we aim at measuring inductive capabilities of proposed strategies in the out-of-distribution size generalization scenario.
To measure scalability of inductive query answering approaches, we create WikiKG-QE, an inductive split of the originally transductive OGB WikiKG 2 [16], following the same sampling strategy as for 9 Freebase datasets [Table]
[Table]
In all datasets, entities and relations are anonymized and only have an integer ID.
Furthermode, inference graphs at validation and test time are supersets of the respective training graph with new nodes and edges.
The amount of new unique nodes is simply the difference E inf − E train between entities in those graphs, e.g., for the dataset of ratio 175% , the validation inference graph contains 4 , 278 new nodes and test inference graph contains 4 , 253 news nodes.
Note that those 4 , 278 and 4 , 253 nodes are unique for each graph and do not overlap.
That is, validation inference and test inference graphs are disconnected except sharing the same core training graph.
Then, for each created inductive dataset, we sample queries of 14 query patterns following the
BetaE [24] procedure.
That is, training queries are sampled from the training graph G train and have only easy answers reachable by simple edge traversal.
Validation and test queries are sampled from the respective splits, e.g., validation queries are sampled from the validation graph G val using entities from the validation inference graph E valinf (which, in turn, are a union of training nodes and new, unseen validation nodes E train ∪ E val ), and at least one edge in each query belongs to T valpred and has to be predicted during query execution.
Queries might have easy answers that are directly reachable by traversing edges T valinf in the validation inference graph, whereas hard answers are only reachable after predicting missing edges from the set T valpred .
Final evaluation metrics are computed only based on the hard answers.
Following the literature [24], we only retain queries that have less than 1000 answers.
Table 5 summarizes the statistics on the sampled queries for each dataset ratio, each graph, and query type that we use in Section 5.2 for evaluating inductive query answering performance.
In graphs with smaller inference graphs and smaller number of missing triples, we sample fewer queries with negation ( 2in, 3in, inp, pin, pni ) for validation and test splits.
For WikiKG-QE, due to its size, we only sample 10k EPFO queries to be executed in the inference-only regime without training (at the moment, CQD-Beam does not support queries with negation).
We use those queries in Section 5.5 to evaluate scalability of NodePiece-QE and prediction quality in the inference-only mode
[Table]
[Table] [Figure]
Furthermore, for the experiment in Section 5.3 to measure the abilities of inductive models to find new answers of known queries, we take the created training queries and find their easy answers in the validation inference G valinf =
( E valinf , T valinf ) and test inference G testinf
=
( E testinf , T testinf ) graphs.
That is, those new answers do not require predicting missing edges in the inference graphs and only require a model to execute edge traversal to find (if any) new correct answers involving new, unseen entities and edges.
For the validation (test) split, we only count such training queries q whose answer set in
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Table]
In val 9, 93.
In test 9, 91.
this split is different from the answer set in the training graph, e.g., A val q ̸ =
A train q .
We summarize the statistics of identified new answer sets in all datasets in Table 6 (for EPFO queries) and Table 7 (for queries with negations).
We find that in most query patterns across all dataset ratios, training queries indeed have new answer sets when executed against validation or test inference graphs
[Figure]
C More Experimental Results
Here [Figure] we present a detailed breakdown of query answering performance measured in Sections 5. and 5.3 by query type.
Fig.
5 and Table 8 contain detailed results from Section 5.2 of executing test queries with new, unseen entities over inference graphs of various ratios of new entities
[Figure]
[Table]
Fig.
6 and Table 9 contain detailed results from the experiment in Section 5.3 about executing training queries over the original training and extended test inference graphs.
[Figure]
H i t s @query =
EPFO avg 0.
query =
neg avg 0.
query = 1p 0.
H i t s @query = 2p 0.
query = 3p 0. query = 2i 0. H i t s @query = 3i 0. query =
pi 0.
query = ip 0.
H i t s @query = 2u 0.
[Figure]
query = up 0.
[Figure]
= 2in
[Figure]
[Table]
D Hyperparameters
Both NodePiece-QE and GNN-QE models are implemented with PyTorch [22] (MIT License).
In particular, NodePiece-QE models employ PyG [12] (MIT License) and PyKEEN [2] (MIT License) for training link prediction models.
GNN-QE is implemented based on the official NBFNet repository 7 (MIT License) and TorchDrug [41] library (Apache 2.0).
For all inductive experiments in Sections 5.2 and 5.3, Table 10 lists best hyperparameters for NodePiece-QE models without GNN encoder, Table 11 contains hyperparameters for GNN-enabled NodePiece-QE models.
The GNN-free models use only relation-based tokenization where each entity e is represented with two fixed-size sets: a set of k unique incoming r i and a set of k unique outgoing r o relation types.
Looking up their d -dimensional vectors, we obtain:
[Equation]
If, for some entity, the number of unique relations of a certain kind is less than k , we pad the set with auxiliary [PAD] tokens.
Entity representations are built as a function of the two sets f ( e ) : R 2 × k × d
→ R d :
[Equation]
Particularly, we first sum up tokens of the same direction, pass them through a random projection layer
R ANDOM P ROJ (we found that making this projection learnable does not improve results), sum up representations of incoming and outgoing parts, and pass the resulting vector through a learnable MLP.
This way, the number of learnable encoder parameters does not depend on the sequence length k , i.e., the number of chosen tokens per node.
The GNN-enabled models employ a slightly different Concat + MLP encoder where each node is tokenized with a sample of k incident relations.
Then, we concatenate d -dimensional embeddings of those tokens t i into a single long vector R kd , and then use a 2-layer MLP to project it to a model dimension d , i.e., f ( e ) : R kd [Equation] d :
[Equation]
For the large-scale experiment on WikiKG-QE in Section 5.5, we employ the Concat + MLP encoder.
Instead of separating incoming and outgoing relation types, we first tokenize each node withnearest anchors (pre-selected in advance using the default NodePiece strategy [13]) and add a sample of k unique incident relations.
NodePiece-QE w/ GNN employs a 3-layer CompGCN with the
RotatE interaction function during message computation and sum aggregator.
Due to high memory consumption, we train the 50d model for 2 epochs on 2 x RTX 8000 (48 GB) GPUs.
The overall tokens vocabulary consists of 20,000 anchor nodes, 1,024 relation types (including inverse relations) and one [PAD] token.
All hyperparameters for this experiments are listed in Table 12.
Having trained the link predictors, we tune CQD-Beam hyperparameters on the validation set varying the t-norms, t-conorms, and scores normalization.
Table 13 lists best options for each EPFO query type.
For all experiments, we used a beam size k = 32 except for queries on WikiKG-QE where we used k = 8 due to the memory-expensive need of maintaining a beam over 2M entities.
Table 14 lists hyperparameters for GNN-QE models for all inductive splits.
We found this architecture is quite stable under various configurations and eventually employed the same set of hyperparameters across all datasets.
BetaE (as a transductive baseline for the reference 175% dataset) was configured with 400 d embedding dimension, batch size 512, 32 negative samples, learning rate 0.0005, margin γ 60, and trained onquery patterns {1p,2p,3p,2i,3i,2in,3in,inp,pin,pni} for 200 k steps
[Table]
[Table]
[Table]
ComplEx [
[Table] Concat [Table]
MLP
[Table] [Table] [Table]
encoder
CompGCN [31] + RotatE [27] message function [Table] [Table] [Table]
[Table]
[Table] [Table]
We consider Edge-type Heuristic as a trivial baseline for inductive complex query. Given a query
Q =
( C , R Q , G ) , Edge-type Heuristic finds all entities e ∈ E that satisfy the relations in the last hop of R Q on the inference graph G inf . In other words, this baseline filters out entities that are not consistent with the query according to the edge types, which is a necessary condition for the answers when the inference graph is reasonably dense. Since Edge-type Heuristic only distinguishes the entities into two classes, we randomly shuffle the entities in each class to create a ranking. Edge-type Heuristic can be easily implemented as GNN-QE with a special relation projection .
Given an inference graph G inf , we first preprocess a relation-to-entity mapping M , where M r,t means there exists a head entity h and an edge ( h, r, t ) for tail entity t . Then the relation projection of relation r can be implemented by looking up the row M r . Note that the Edge-type Heuristic only filters entities according to the edge type, and hence the head entity (or the distribution of head entity) is ignored in the relation projection.