Dynamically Pruned Message Passing Networks for Large-Scale Knowledge Graph Reasoning
Abstract
We propose Dynamically Pruned Message Passing Networks (DPMPN) for largescale knowledge graph reasoning.
In contrast to existing models, embedding-based or path-based, we learn an input-dependent subgraph to explicitly model a sequential reasoning process.
Each subgraph is dynamically constructed, expanding itself selectively under a ﬂow-style attention mechanism.
In this way, we can not only construct graphical explanations to interpret prediction, but also prune message passing in Graph Neural Networks (GNNs) to scale with the size of graphs.
We take the inspiration from the consciousness prior proposed by Bengio [1] to design a two-GNN framework to encode global input-invariant graph-structured representation and learn local input-dependent one coordinated by an attention module.
Experiments show the reasoning capability of our model which is to provide clear graphical explanations as well as to predict results accurately, outperforming most state-of-the-art methods in knowledge base completion tasks.
Introduction
Modern deep learning systems need to acquire the reasoning capability beyond their black-box nature to produce interpretable predictions [2, 3]. In what form we can better model a reasoning process should be given more thought than just caring about a ﬁnal correct prediction.
Intuitively, a reasoning process can be regarded as a sequence of using existing facts to establish new knowledge, step by step, and ﬁnally drawing conclusions in the form of constructing explanations as well as making predictions. Therefore, it needs an explicit modeling to identify and organize reasoning steps to form a clear interpretable representation during predicting. A natural idea is to use graph-structured representation where a semantic unit or pairwise relation can be explicitly represented by a node or an edge as building blocks to support graph-based reasoning, a more ﬂexible form in contrast to rigid deductive logical reasoning [4, 5]. Graph-based reasoning can be applied to a wide variety of real-world scenarios.
Here, we choose knowledge graph-related tasks to explore due to its representativeness.
In knowledge base completion (KBC) tasks, embedding-based models [6, 7, 8, 9, 10, 11] can easily obtain a very competitive score by ﬁtting data using various neural network techniques, but lacking an explicit modeling on constructing explanations prevents it from being interpretable, a critical property of reasoning, since
Euclidean embedding space will not produce a clearly stated and human-readable representation. Recent work on knowledge graph (KG) reasoning focuses on path-based [12, 13, 14, 15, 16, 17] or logic-like models [18, 19].
Most of them construct an explicit path to model an iterative decision- [Figure] making process using reinforcement learning and recurrent networks.
However, a question is: do we have a better form, more ﬂexible and interpretable, to express reasoning in the graph context rather than one or several paths.
To this end, we propose to learn a subgraph starting from a head node and expanding itself conditionally and selectively according to a query relation, where a tail node is predicted after the ﬁnal expansion.
To better explain how the tail is determined through a series of expansions, we weigh, prune and save intermediate nodes selected, step by step, to capture long-range dependence and yield a concise and compact subgraph explanation for tail prediction as shown in Figure 1.
Graph reasoning can be powered by Graph Neural Networks.
Graph reasoning demands a way to effectively learn about entities, relations, and rules of composing them, that is, an ability to achieve combinatorial generalization by manipulating structured knowledge and producing structured explanations.
Graph Neural Networks (GNNs) provide such structured representation and structured computation, and also inherit powerful data-ﬁtting capacity from deep neural networks [20, 4].
Speciﬁcally, GNNs follow a neighborhood aggregation scheme, recursively aggregating and transforming neighboring nodes’ representations to update each node’s representations.
Therefore, after T iterations of aggregation, each node can carry the structured information within the node’s T -hop neighborhood [21, 22].
GNNs need graphical attention expression to interpret.
Neighborhood attention operation is a popular way to implement attention mechanism on graphs [23, 24] by using multi-head self-attention to focus on speciﬁc interactions with neighbors when aggregating messages.
However, we argue that a way of graphical attention expression should be considered not only to facilitate more ﬂexible structured computation but also to construct dynamically pruned structured explanations.
We present three considerations: (1) selecting nodes based on currently operated subgraphs, that is, ﬁrst attending over nodes inside a subgraph to pick a smaller core set and then attending over the neighborhood of this picked core set to expand the subgraph, (2) breaking isolation of attention operations used each step and propagating attention across steps, a way to perform a discrete version of normalizing ﬂow [25] over nodes, to capture long-range statistical dependence, and (3) that such ﬂow-style attention mechanism should model a changing node probability distribution, that is, a Markov process driven by step-varying transition matrices. Besides, we should use an attention module disentangled from the pipeline of aggregating and transforming representations in GNNs to explicitly model a reasoning process on graphs out of low-level representation computation.
GNNs need input-dependent pruning to scale.
GNNs are notorious for its poor scalability due to its heavy computation complexity.
Consider, for example, one message passing iteration performed over a graph with | V | nodes and | E | edges.
It has quadratic complexity in the number of nodes, O ( | V | 2 ) , if the graph is fully connected. Even if the graph is sparse so that the complexity can be reduced to O ( | E | ) by exploiting structural sparsity, it is still problematic when meeting large graphs with millions of nodes and edges.
Besides, mini-batch based training with batch size B and high dimensions D would make things worse, leading to the complexity of O ( BD | E | ) .
We argue that this situation can be avoided by learning input-dependent pruning, as in most cases an input example only uses a small fraction of the entire graph, and it is wasteful to perform homogeneous graph computation on the entire graph for each input.
Therefore, we propose to prune message passing depending on speciﬁc inputs so that message passing runs on dynamical computation graphs instead of the static one.
Cognitive intuition of the consciousness prior.
The notion of attentive awareness has been shared by cognitive science communities in several theories [26, 27]. Bengio [1] brought this notion into deep learning models in his consciousness prior proposal.
He pointed out a process of disentangling high-level abstract factors from full underlying representation to form a low-dimensional combination of a few selected factors to constitute a conscious thought, and emphasized the role of attention in expressing awareness during this process.
Bengio proposed to use two recurrent neural networks (RNNs) to encode two types of state: the unconscious state represented by a full high-dimensional vector before applying attention, and the conscious state by a derived low-dimensional vector after applying attention.
In our work, we use two GNNs to encode such two types of state into node representation vectors.
Note that standard message passing runs globally so that messages gathered at one node may come from everywhere and get further entangled after applying aggregation operations. Therefore, we consider drawing an input-dependent or context-aware local subgraph to constrain message passing. Besides, we hope to be able to access certain global information about the graph structure to get a boarder view even though we will focus on a local subgraph per input.
Inspired by the consciousness prior [1], we apply a novel ﬂow-style graphical attention mechanism to connect and organize these two GNNs. The bottom GNN performs input-invariant standard message passing globally, called Inattentive GNN (IGNN) , and the above GNN performs input-dependent pruned message passing locally, called Attentive GNN (AGNN) .
The intuition is that IGNN offers a pool of raw representations for AGNN to select from, entangled but rich, while AGNN focuses on multiple input-dependent subgraphs consisting of a few selected nodes and their edges, cohesive with sharp semantics, disentangled from the full graph. Nodes inside such a subgraph should be more densely connected to form a small community to further exchange information and make decisions collectively on how to grow the subgraph next. In experiments, we ﬁnd our model can run on very large graphs with millions of edges, such as the YAGO3-10 dataset, even just using a laptop without causing the out-of-memory error.
Our prediction results on KBC tasks attain highly competitive scores on HITS@1,3 and the mean reciprocal rank (MRR) compared to the best embedding-based method so far 1 .
Besides, we provide interpretations which they do not have.
Problem Formulation
Notation.
We use a supervised setting with training data { ( x i , y i ) } N i =1 where x i is an input and y i is a target.
We denote a full graph by G with node set V and edge set E , and denote an input-dependent subgraph by G ( x ) with node set V G ( x ) and edge set E G ( x ) . We also denote the set of edge types (or relation types) by R .
We require each subgraph G to hold E G =
{ ( v, u ) ∈ E : v, u ∈ V G } , so that we can deﬁne G = G 1 ∪ G 2 if V G
= V G 1 ∪ V G 2 and deﬁne G 1 ⊆ G 2 if V G 1 ⊆ V G 2 . We deﬁne the boundary of a subgraph as ∂G if V ∂G = N ( V G ) − V G where N ( V G ) means the union of neighbors of all the nodes in V G .
We also deﬁne high-order boundaries such as ∂ 2 G if
V ∂ 2 G = N
( N ( V G )) ∪ N ( V G ) − V G .
Trainable parameters include node embeddings { e v } v ∈V , relation type embeddings { e r } r ∈R , and neural network weights used in two GNNs and an attention module.
When performing standard or pruned message passing, node embeddings and relation type embeddings will be indexed according to the operated graph, and thus we denote them by θ G or θ G ( x ) .
We denote batch size by B and dimensions by D .
For IGNN, we use H t of size |V| × D to denote node hidden states at step t ; for AGNN, we use H t ( x ) of size | V G ( x ) | × D to denote.
We deﬁne the objective based on our two GNNs as (cid:80) Ni
=1 l ( x i , y i ; θ G ( x i ) , θ G ) , where G ( x i ) is dynamically constructed.
First, we write the standard message passing in IGNN as
[Equation]
where f IGNN represents all involved operations in one message passing iteration over G , including: (1) computing messages along each edge with the complexity 2 of O ( BD |E| ) , (2) aggregating messages received at each node with the complexity of O ( BD |E| ) , and (3) updating node hidden states with the complexity of O ( BD |V| ) .
For a T -step propagation, we get the per-batch complexity of
O ( BDT ( |E| + |V| )) .
Considering that backpropagation requires intermediate computation results to be saved during one pass, this complexity counts for both time and space.
However, since IGNN is input-invariant, its node representations can be shared across input examples in one batch so that B can be removed to get O ( DT ( |E| + |V| )) .
If we sample a smaller set ˆ E from E to run such that | ˆ E| ≈ k |V| , we can further reduce the complexity to O ( DT |V| ) .
The pruned message passing in AGNN can be written as
[Equation]
Its complexity can be computed similarly as above.
However, we cannot remove B .
Fortunately, subgraph G ( x ) is not G .
If we let x be a node v , G ( x ) grows from a single node, i.e., G 0 ( x ) = { v } , and expands itself each step, leading to a sequence of ( G 0 ( x ) , G 1 ( x ) , . .
, G T ( x )) .
Here, we describe the expansion behavior as consecutive expansion , which means no jumping across neighborhood allowed, so that we can ensure that
[Equation]
Many real-world graphs follow the small-world pattern, and the six degrees of separation implies
G 0 ( x ) ∪ ∂ 6 G 0 ( x ) ≈ G .
The upper bound of G t ( x ) can grow exponentially in t , and there is no guarantee that G t ( x ) will not explode.
Proposition.
Given a graph G (undirected or directed in both directions), we assume the probability of the degree of an arbitrary node being less than or equal to d is larger than p , i.e., P (deg( v ) ≤ d ) > p, ∀ v ∈ V .
Considering a sequence of consecutively expanding subgraphs ( G 0 , G 1 , . .
, G T ) , starting with G 0 = { v } , for all t ≥ 1 , we can ensure
[Equation]
The proposition implies the guarantee of upper-bounding | V G t ( x ) | becomes exponentially looser and weaker as t gets larger even if the given assumption has a small d and a large p (close to 1).
We deﬁne graph increment at step t as ∆ G t ( x ) such that G t ( x ) =
G t − 1 ( x ) ∪ ∆ G t ( x ) .
To prevent G t ( x ) from explosion, we need to constrain ∆ G t ( x ) .
We propose several sampling strategies:
∆ G t ( x ) =
ˆ ∂G t − 1 ( x ) , which means we sample nodes from the boundary of G t − 1 ( x ) .
∆ G t ( x ) = ∂ (cid:92) G t − 1 ( x ) , which means we take the boundary of sampled nodes from G t − 1 ( x ) . 3. ∆ G t ( x ) = ˆ ∂ (cid:92) G t − 1 ( x ) , which means we sample nodes from the boundary of (cid:92) G t − 1 ( x ) .
∆ G t ( x ) = ˆ ∂ (cid:92) G t − 1 ( x ) (cid:98) , which means we samples nodes from ˆ ∂ (cid:92) G t − 1 ( x ) .
Obviously, we have ˆ ∂ (cid:92) G t − 1 ( x ) (cid:98) ⊆
ˆ ∂ (cid:92) G t − 1 ( x ) ⊆
∂ (cid:92) G t − 1 ( x ) and G t − 1 ( x ) ∪ ∂ (cid:92) G t − 1 ( x ) ⊆
G t − 1 ( x ) ∪ ∂G t − 1 ( x ) .
Further, we let N [-
-]g
and N s be the maximum number of sampled nodes in ∂ (cid:92) G t − 1 ( x ) and the last sampling of ˆ ∂ (cid:92) G t − 1 ( x ) (cid:98) respectively and let N b be per-node maximum sampled neighbors in ˆ ∂G t − 1 ( x ) , and then we can obtain much tighter guarantee as follow: P ( | V ∆ G t ( x ) | ≤ N g ( d − 1)) > p N g for ∂ (cid:92) G t − 1 ( x ) . P ( | V ∆ G t ( x ) | ≤ N g N b ) = 1 and P ( | V ∆ G t ( x ) | ≤ N g · min( d − 1 , N b )) > p N g for ˆ ∂ (cid:92) G t − 1 ( x ) .
AggrOp ... AggrOp ...
AttdOp ...
[Figure]
OneBatch
[Figure]
By ˆ ∂ (cid:92) G t − 1 ( x ) (cid:98) , we can guarantee | V G T ( x ) | ≤ 1 + T min( N g N b , N s ) .
To constrain the growth of
G t − 1 ( x ) , we can decrease either N g N b or N s . However, smaller sample size means less area explored and less chance to hit target nodes.
We thus use attention operations to do the top- K selection instead of random sampling when K has to be small.
We change ˆ ∂ (cid:92) G t − 1 ( x ) (cid:98) to ˆ ∂G t − 1 ( x ) (cid:101)(cid:101) where ∼ represents the operation of attending over nodes and picking the top- K .
There are two types of attention operations, one applied to G t − 1 ( x ) and the other applied to ˆ ∂G t − 1 ( x ) (cid:101) .
Note that the size of ˆ ∂G t − 1 ( x ) (cid:101) might be much larger than G t − 1 ( x ) if we want to sample more nodes with larger N b to sufﬁciently explore the boundary, ∂G t − 1 ( x ) (cid:101) .
Nevertheless, we can address this problem by using small dimensions to compute attention scores, since attention carried by each node is just a scalar, much smaller than a node representation vector computed during message passing over G t − 1 ( x ) .
Model Implementation 3.1 Architecture design for knowledge graph reasoning
Our model architecture as shown in Figure 2 consists of:
• IGNN module : performs standard message passing to compute full-graph node representations.
• AGNN module : performs a batch of pruned message passing to compute input-dependent node representations which also make use of low-level representations from IGNN.
• Attention Module : performs a ﬂow-style attention transition process, conditioned on node representations from both IGNN and AGNN but only affecting AGNN.
We let (cid:104)V , E(cid:105) denote a knowledge graph where V is a set of entities and E is a set of relations.
Each edge or relation is represented by a triple (cid:104) head, rel, tail (cid:105) , where head is the head entity, tail is the tail entity, and rel is their relation type.
The goal is to predict potential unknown links, i.e., which entity is likely to be the tail given a query (cid:104) head, rel, ?
(cid:105) with the head and the relation type speciﬁed.
IGNN module.
We implement it using standard message passing mechanism [21].
If the full graph has an extremely large number of edges, we sample a subset of edges, ˆ E τ ⊂ E , randomly each step.
For a batch of input queries, we let node representations from IGNN be shared across queries, containing no batch dimension.
Thus, its complexity does not scale with batch size and the saved computation resources can be allocated to sampling more edges.
Each node v has a state H τv, : at step τ , where the initial H 0 v, : =
e v .
Each edge (cid:104) v (cid:48) , r, v (cid:105) produces a message, denoted by M τ (cid:104) v (cid:48) ,r,v (cid:105) , : at step τ .
The computation components include:
• Message function: M τ (cid:104) v (cid:48) ,r,v (cid:105) , : = ψ IGNN ( H τv (cid:48) , : , e r , H τv, : ) , where (cid:104) v (cid:48) ,
r, v (cid:105) ∈
ˆ E τ .
• Message aggregation: M τv, : = 1 √ N τ ( v ) (cid:80) v (cid:48) ,r M τ (cid:104) v (cid:48) ,r,v (cid:105) , : , where (cid:104) v (cid:48) ,
r, v (cid:105) ∈
ˆ E τ .
• Node state update function:
H τ +1 v, :
= H τv, : + δ IGNN ( H τv, : , M τv, : , e v ) , where v ∈ V .
We compute messages only for the sampled edges, (cid:104) v (cid:48) , r, v (cid:105) ∈
ˆ E τ , each step.
Functions ψ IGNN and δ IGNN are implemented by a two-layer MLP (using leakyReLu for the ﬁrst layer and tanh for the second) with input arguments concatenated respectively.
Messages are aggregated by dividing the sum by the square root of N τ ( v ) , the number of sampled neighbors that send messages to v , preserving the scale of variance.
We use a residual adding to update each node state instead of a GRU or a LSTM.
After running for T steps, we output a pooling result or simply the last, denoted by H = H T , to feed into downstream modules.
AGNN module.
AGNN is input-dependent, which means node states depend on input query x = (cid:104) head, rel, ?
(cid:105) , denoted by H tv, : ( x ) .
We implement pruned message passing, running on small subgraphs each conditioned on an input query.
We leverage the sparsity and only save H tv, : ( x ) for visited nodes v ∈ V G t ( x ) .
When t = 0 , we start from node head with V G 0 ( x ) = { v head } .
When computing messages, denoted by M t (cid:104) v (cid:48) ,r,v (cid:105) , : ( x ) , we use a sampling-attending procedure, explained in Section 3.2, to constrain the number of computed edges.
The computation components include:
• Message function: M t (cid:104) v (cid:48) ,r,v (cid:105) , : ( x ) = ψ AGNN ( H tv (cid:48) , : ( x ) , c
r ( x ) , H tv, : ( x )) , where (cid:104) v (cid:48) , r, v (cid:105) ∈
E G t ( x ) 3 , and c r ( x ) =
[ e r , q head , q rel ] represents a context vector.
• Message aggregation: M tv, : ( x ) = 1 √ N t ( v ) (cid:80) v (cid:48) ,r M t (cid:104) v (cid:48) ,r,v (cid:105) , : ( x ) , where (cid:104) v (cid:48) , r, v
(cid:105) ∈
E G t ( x ) .
• Node state attending function: (cid:102) H t +1 v, : ( x ) = a t +1 v W H v, : , where a t +1 v is an attention score.
• Node state update function: H t +1 v, : ( x ) =
H tv, : ( x ) + δ AGNN ( H tv, : ( x ) , M tv, : ( x ) , c t +1 v ( x )) , where c t +1 v ( x ) =
[ (cid:102)
H t +1 v, : ( x ) , q head , q rel ] also represents a context vector.
Query context is deﬁned by its head and relation type embeddings, i.e., q head =
e head and q rel =
e rel .
We introduce a node state attending function to pass node representation information from IGNN to AGNN weighted by a scalar attention score a t +1 v and projected by a learnable matrix W .
We initialize H 0 v, : ( x ) =
H v, : for node v ∈ V G 0 ( x ) , treating the rest as zero states.
Attention module.
Attention over T steps is represented by a sequence of node probability distributions, denoted by a t ( t = 1 , 2 . . .
, T ) .
The initial distribution a 0 is a one-hot vector with a
[ v head ] = 1 .
To spread attention, we need to compute transition matrices T t each step.
Since it is conditioned on both IGNN and AGNN, we capture two types of interaction between v (cid:48) and v :
H tv (cid:48) , : ( x ) ∼ H tv, : ( x ) , and H tv (cid:48) , : ( x ) ∼ H v, : .
The former favors visited nodes, while the latter is used to attend to unseen nodes.
[Equation]
(5) where W 1 and W 2 are two learnable matrices.
Each MLP uses one single layer with the leakyReLu activation.
To reduce the complexity for computing T t , we use nodes v [-(cid:
-](cid:48) ∈ VG
[Equation]
( x ) (cid:101) , which contains nodes with the k -largest attention scores at step t , and use nodes v sampled from v (cid:48) ’s neighbors to compute attention transition for the next step.
Due to the fact that nodes v (cid:48) result from the top- k pruning, the loss of attention may occur to diminish the total amount.
Therefore, we use a renormalized version, a t +
=
T t a t / (cid:107) T t a t (cid:107) , to compute new attention scores.
We use attention scores at the ﬁnal step as the probability to predict the tail node.
[Figure]
3.2 Complexity reduction by iterative sampling and attending
AGNN deals with local subgraphs for each input x so that only a few selected nodes are kept in V G t ( x ) , called visited nodes , and | V G t ( x ) | is much smaller than |V| .
The initial V G 0 ( x ) contains only one node v head , and then V G t ( x ) is enlarged each step by adding new nodes.
When propagating messages, we can just consider the one-step neighborhood each step.
However, the expansion goes so rapidly that it covers almost all nodes after a few steps.
The key to address the problem is to constrain the scope of nodes we can expand the boundary from, i.e., the core nodes which determine where we can go next.
We call it the attending-from horizon , G t ( x ) (cid:101) , selected according to attention scores a t .
Given this horizon, we still need edge sampling over its neighborhood instead of using the whole N ( G t ( x ) (cid:101) ) in case of a hub node of extremely high degree. Here, we face a trade-off between coverage and complexity when sampling over the neighborhood. Also, we need node representations within each subgraph to keep their information coherent and avoid possible noises caused by randomly sampling.
Therefore, we introduce an attending-to horizon inside the sampling horizon . We denote the sampling horizon by (cid:98) N ( G t ( x ) (cid:101) ) and the attending-to horizon by (cid:98) N ( G t ( x ) (cid:101) ) (cid:101) .
The attention module runs within the sampling horizon with smaller dimensions in order to sample more neighbors for a larger coverage.
Then, we prune the sampling horizon to obtain the attending-to horizon, which contains a subset of nodes selected according to newly computed attention scores a t +1 .
Current message passing iteration at step t in AGNN can be further constrained on edges between G t ( x ) (cid:101) and (cid:98) N ( G t ( x ) (cid:101) ) (cid:101) , a smaller set than E G t ( x ) .
We illustrate this procedure in Figure 3.
Experiments
Datasets.
We use six large KG datasets: FB15K, FB15K-237, WN18, WN18RR, NELL995, and
YAGO3-10.
{+FB15K-
 [28]+} is sampled from FB15K [6] with redundant relations removed, and WN18RR [8] is a subset of WN18 [6] removing triples that cause test leakage.
Thus, they are both considered more challenging.
NELL995 [13] has separate datasets for 12 query relations each corresponding to a single-query-relation KBC task.
YAGO3-10 [29] contains the largest KG with millions of edges.
Their statistics are shown in Table 1.
We ﬁnd some statistical differences between train and validation (or test).
In a KG with all training triples as its edges, a triple ( head, rel, tail ) is considered as a multi-edge triple if the KG contains other triples that also connect head and tail ignoring the direction.
We notice that FB15K-237 is a special case compared to the others, as there are no edges in its KG directly linking any pair of head and tail in validation (or test).
Therefore, when using training triples as queries to train our model, given a batch, for FB15K-237, we cut off from the KG all triples connecting the head-tail pairs in the given batch, ignoring relation types and edge directions, forcing the model to learn a composite reasoning pattern rather than a single-hop pattern, and for the rest datasets, we only remove the triples of this batch and their inverse from the KG to avoid information leakage before training on this batch.
This can be regarded as a hyperparameter tuning whether to force a multi-hop reasoning or not, leading to a performance boost of about 2% in HITS@1 on FB15-237.
Experimental settings.
We use the same data split protocol as in many papers [8, 13, 14].
We create a KG, a directed graph, consisting of all train triples and their inverse added for each dataset except NELL995, since it already includes reciprocal relations.
Besides, every node in KGs has a self-loop edge to itself.
We also add inverse relations into the validation and test set to evaluate the two directions.
For evaluation metrics, we use HITS@1,3,10 and the mean reciprocal rank (MRR) in the ﬁltered setting for FB15K-237, WN18RR, FB15K, WN18, and YAGO3-10, and use the mean average precision (MAP) for NELL995’s single-query-relation KBC tasks. For NELL995, we follow
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
ComplEx [Table]
N3[
[Table] [Table]
NeuralLP [
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
MINERVA [ ♥ ] 14.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
1) MINERVA [ (cid:52) ] - - [Table] [Table] M-Walk [ ♥ ] 16.5 (.3) 24.3 (.2) - 23.2 (.2) 41.
[Table] the same evaluation procedure as in [13, 14, 15], ranking the answer entities against the negative examples given in their experiments.
We run our experiments using a 12G-memory GPU, TITAN X (Pascal), with Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz.
Our code is written in Python based on TensorFlow 2.0 and NumPy 1.16 and can be found by the link 4 below.
We run three times for each hyperparameter setting per dataset to report the means and standard deviations.
See hyperparameter details in the appendix.
Baselines.
We compare our model against embedding-based approaches, including TransE [6],
TransR [31], DistMult [7], ConvE [8], ComplE [9], HolE [32], RotatE [10], and ComplEx-N3 [11], and path-based approaches that use RL methods, including DeepPath [13], MINERVA [14], and
M-Walk [15], and also that uses learned neural logic, NeuralLP [19].
Comparison results and analysis.
We report comparison on FB15K-23 and WN18RR in Table 2.
Our model DPMPN signiﬁcantly outperforms all the baselines in HITS@1,3 and MRR.
Compared to the best baseline, we only lose a few points in HITS@10 but gain a lot in HITS@1,3.
We speculate that it is the reasoning capability that helps DPMPN make a sharp prediction by exploiting graph-structured composition locally and conditionally.
When a target becomes too vague to predict, reasoning may lose its advantage against embedding-based models.
However, path-based baselines, with a certain ability to do reasoning, perform worse than we expect.
We argue that it might be inappropriate to think of reasoning, a sequential decision process, equivalent to a sequence of nodes.
The average lengths of the shortest paths between heads and tails as shown in Table 1 suggests a very short path, which makes the motivation of using a path almost useless.
The reasoning pattern should be modeled in the form of dynamical local graph-structured pattern with nodes densely connected with each other to produce a decision collectively. We also run our model on FB15K,
WN18, and YAGO3-10, and the comparison results in the appendix show that DPMPN achieves a very competitive position against the best state of the art.
We summarize the comparison on NELL995’s tasks in the appendix.
DPMPN performs the best on ﬁve tasks, also being competitive on the rest.
[Figure]
[Figure]
AthletePlaysForTeamAthletePlaysInLeagueAthleteHomeStadiumAthletePlaysSportTeamPlaysSportOrgHeadQuarteredInCityWorksForPersonBornInLocationPersonLeadsOrgOrgHiredPersonAgentBelongsToOrgTeamPlaysInLeague
[Figure]
AthletePlaysForTeamAthletePlaysInLeagueAthleteHomeStadiumAthletePlaysSportTeamPlaysSportOrgHeadQuarteredInCityWorksForPersonBornInLocationPersonLeadsOrgOrgHiredPersonAgentBelongsToOrgTeamPlaysInLeague
[Figure]
AthletePlaysForTeamAthletePlaysInLeagueAthleteHomeStadiumAthletePlaysSportTeamPlaysSportOrgHeadQuarteredInCityWorksForPersonBornInLocationPersonLeadsOrgOrgHiredPersonAgentBelongsToOrgTeamPlaysInLeague
Convergence analysis.
Our model converges very fast during training.
We may use half of training queries to train model to generalize as shown in Figure 4(A).
Compared to less expensive embeddingbased models, our model need to traverse a number of edges when training on one input, consuming much time per batch, but it does not need to pass a second epoch, thus saving a lot of training time.
The reason may be that training queries also belong to the KG’s edges and some might be exploited to construct subgraphs during training on other queries.
Module analysis.
If we do not run message passing in IGNN, H v, : is just the initial embedding of node v , and we can still run pruned message passing in AGNN as usual.
We want to know whether IGNN is actually useful.
Considering that long-range propagated messages might bring in noisy features, we compare running IGNN for two steps against totally shutting it down.
The result in Figure 4(B) shows that IGNN brings a small gain in each metric on WN18RR.
Horizon analysis.
The sampling, attending-to, attending-from and searching (i.e., propagation steps) horizons determine how large area a subgraph can expand over.
These factors affect computation complexity as well as prediction performance.
Intuitively, enlarging the exploring area by sampling more, attending more, and searching longer, may increase the chance of hitting a target to gain some performance.
However, the experimental results in Figure 4(C)(D) show that it is not always the case.
In Figure 4(E), we can see that increasing the maximum number of attending-from nodes per step is useful, but normal GPUs with a limited memory do not allow for an arbitrarily large number due to heavy intermediate data produced during feedforward computing.
Figure 4(F) suggests that the propagation steps of AGNN should not go below four.
Attention ﬂow analysis.
If the ﬂow-style attention really captures the way we reason about the world, its process should be conducted in a diverging-converging thinking pattern.
Intuitively, ﬁrst, for the diverging thinking phase, we search and collect ideas as much as we can; then, for the converging thinking phase, we try to concentrate our thoughts on one point.
To check whether the attention ﬂow has such a pattern, we measure the average entropy of attention distributions changing along steps and also the proportion of attention concentrated at the top-1,3,5 nodes.
As we expect, attention is more focused at the ﬁnal step and the beginning.
Time cost analysis.
The time cost is affected not only by the scale of a dataset but also by the horizon setting.
For each dataset, we list the training time for one epoch corresponding to our standard hyperparameter settings in the appendix.
Note that there is always a trade-off between complexity and performance.
We thus study whether we can reduce time cost a lot at the price of sacriﬁcing a little performance.
We plot the one-epoch training time in Figure 6(A)-(D), using the same settings as [Figure]
to Figure 4(C)-(F); (E) measures on different batch sizes using horizon setting Max-sampling-per-node =20, Max-attending-to-per-step =20, Max-attending-from-perstep =20, and #Steps-in-AGNN =8.
The charts on FB15K-237 can be found in the appendix.
we do in the horizon analysis.
We can see that Max-attending-from-per-step and #Steps-in-AGNN affect the training time signiﬁcantly while Max-sampling-per-node and Max-attending-to-per-step affect very slightly.
Therefore, we can use smaller Max-sampling-per-node and Max-attending-to-per-step in order to gain a larger batch size, making the computation more efﬁciency as shown in Figure 6(E).
Visualization.
To further demonstrate the reasoning capability, we show visualization results of some pruned subgraphs on NELL995’s test data for 12 separate tasks.
We avoid using the training data in order to show generalization of the learned reasoning capability.
We show the visualization results in Figure 1.
See the appendix for detailed analysis and more visualization results.
Related Work
Knowledge graph reasoning.
Early work, including TransE [6] and its analogues [33, 31, 34], DistMult [7], ConvE [8] and ComplEx [9], focuses on learning embeddings of entities and relations.
Some recent works of this line [10, 11] achieve high accuracy.
Another line aims to learn inference paths [35, 36, 37, 38, 39, 40] for knowledge graph reasoning, especially DeepPath [13], MINERVA [14], and M-Walk [15], which use RL to learn multi-hop relational paths.
However, these approaches, based on policy gradients or Monte Carlo tree search, often suffer from low sample efﬁciency and sparse rewards, requiring a large number of rollouts and sophisticated reward function design.
Other efforts include learning soft logical rules [18, 19] or compostional programs [41].
Relational reasoning in Graph Neural Networks.
Relational reasoning is regarded as the key for combinatorial generalization, taking the form of entity- and relation-centric organization to reason about the composition structure of the world [42, 43].
A multitude of recent implementations [4] encode relational inductive biases into neural networks to exploit graph-structured representation, including graph convolution networks (GCNs) [44, 45, 46, 47, 48, 49, 50, 51] and graph neural networks [20, 52, 53, 54, 21].
Variants of GNN architectures have been developed.
Relation networks [53] use a simple but effective neural module to model relational reasoning, and its recurrent versions [55, 56] do multi-step relational inference for long periods; Interaction networks [54] provide a general-purpose learnable physics engine, and two of its variants are visual interaction networks [57] and vertex attention interaction networks [24]; Message passing neural networks [21] unify various GCNs and GNNs into a general message passing formalism by analogy to the one in graphical models.
Attention mechanism on graphs.
Neighborhood attention operation can enhance GNNs’ representation power [23, 24, 58, 59].
These approaches often use multi-head self-attention to focus on speciﬁc interactions with neighbors when aggregating messages, inspired by [60, 61, 62].
Most graph-based attention mechanisms attend over neighborhood in a single-hop fashion, and [24] claims that the multi-hop architecture does not help to model high-order interaction in experiments.
However, a ﬂow-style design of attention in [63] shows a way to model long-range attention, stringing isolated attention operations by transition matrices.
Conclusion
We introduce Dynamically Pruned Message Passing Networks (DPMPN) and apply it to large-scale knowledge graph reasoning tasks.
We propose to learn an input-dependent local subgraph which is progressively and selectively constructed to model a sequential reasoning process in knowledge graphs.
We use graphical attention expression, a ﬂow-style attention mechanism, to guide and prune the underlying message passing, making it scalable for large-scale graphs and also providing clear graphical interpretations.
We also take the inspiration from the consciousness prior to develop a two-GNN framework to boost experimental performances.
Appendix Proof
Proposition.
Given a graph G (undirected or directed in both directions), we assume the probability of the degree of an arbitrary node being less than or equal to d is larger than p , i.e., P (deg( v ) ≤ d ) > p, ∀ v ∈ V .
Considering a sequence of consecutively expanding subgraphs ( G 0 , G 1 , . .
, G T ) , starting with G 0 = { v } , for all t ≥ 1 , we can ensure
[Equation]
Proof.
We consider the extreme case of greedy consecutive expansion, where G t =
G t − 1 ∪ ∆ G t =
G t − 1 ∪ ∂G t − 1 , since if this case satisﬁes the inequality, any case of consecutive expansion can also satisfy it.
By deﬁnition, all the subgraphs G t are a connected graph.
Here, we use ∆ V t to denote
V ∆ G t for short.
In the extreme case, we can ensure that the newly added nodes ∆ V t at step t only belong to the neighborhood of the last added nodes ∆ V t − 1 .
Since for t ≥ 2 each node in ∆ V t −already has at least one edge within G t − 1 due to the deﬁnition of connected graphs, we can have
[Equation]
For t = 1 , we have P ( | ∆ V 1 | ≤ d ) > p and thus
[Equation]
For t ≥ 2 , based on | V G t | = 1 + | ∆ V 1 | + . . .
+ | ∆ V t | , we obtain
P (cid:0) | V G t | ≤ 1 + d + d ( d [Equation] 1) + . . . + d ( d − 1) t − 1 (cid:1) > p
[Equation]
which is
[Equation]
We can ﬁnd that t = 1 also satisﬁes this inequality.
Hyperparameter Settings
[Table]
The hyperparameters can be categorized into three groups:
• Normal hyperparameters, including batch_size , n_dims_att , n_dims , learning_rate , grad_clipnorm , and n_epochs .
We set smaller dimensions,
n
_dims_att , for computation in the attention module, as it uses more edges than the message passing uses in AGNN, and also intuitively
, it does not need to propagate high-dimensional messages but only compute scalar scores over a sampled neighborhood, in concert with the idea in the key-value mechanism [1].
We set n _ epochs =
in most cases, indicating that our model can be trained well by one epoch only due to its fast convergence.
• The hyperparameters in charge of the sampling-attending horizon, including max_sampling_per_step that controls the maximum number to sample edges per step in IGNN, and max_sampling_per_node , max_attending_from_per_step and max_attending_to_per_step that control the maximum number to sample neighbors of each selected node per step per input, the maximum number of selected nodes for attending-from per step per input, and the maximum number of selected nodes in a sampled neighborhood for attending-to per step per input in AGNN.
• The hyperparameters in charge of the searching horizon, including n_steps_in_IGNN representing the number of propagation steps to run standard message passing in IGNN, and n_steps_in_AGNN representing the number of propagation steps to run pruned message passing in AGNN.
Note that we tune these hyperparameters according to not only their performances but also the computation resources available to us.
In some cases, to deal with a very large knowledge graph with limited resources, we need to make a trade-off between efﬁciency and effectiveness.
For example, each of NELL995’s single-query-relation tasks has a small training set, though still with a large graph, so we can reduce the batch size in favor of affording larger dimensions and a larger sampling-attending horizon without any concern for waiting too long to ﬁnish one epoch.
More Experimental Results
[Table]
and
[Table] [Table] [Figure] [Figure]
[Figure] [Figure] T r [Figure]
i n i n g T i
[Figure]
e
[Figure]
n e
E p
[Figure] c [Figure]  More Visualization Resutls 10.1 Case study on the AthletePlaysForTeam task
In the case shown in Figure 9, the query is ( concept_personnorthamerica_michael_turner , concept:athleteplays-forteam , ?) and a true answer is concept_sportsteam_falcons .
From Figure 9, we can see our model learns that ( concept_personnorthamerica_michael_turner , concept:athletehomestadium , concept_stadiumoreventvenue_georgia_dome ) and ( concept_stadiumoreventvenue_georgia_dome , concept:teamhomestadium_inv , concept_sportsteam_falcons ) are two important facts to support the answer of concept_sportsteam_falcons .
Besides, other facts, such as ( concept_athlete_joey_harrington , concept:athletehomestadium , concept_stadiumoreventvenue_georgia_dome ) and ( concept_athlete_joey_harrington , concept:athleteplaysforteam , concept_sportsteam_falcons ), provide a vivid example that a person or an athlete with concept_stadiumoreventvenue_georgia_dome as his or her home stadium might play for the team concept_sportsteam_falcons .
We have such examples more than one, like concept_athlete_roddy_white ’s and The entity concept_sportsleague_nﬂ cannot help us differentiate the true answer from other NFL teams, but it can at least exclude those non-NFL teams.
In a word, our subgraph-structured representation can well capture the relational and compositional reasoning pattern
[Figure]
For the AthletePlaysForTeam task
[Figure]
[Table] [Table]
10.2 More results
[Table]
[Figure]
_pelfreyconcept
[Figure]
[Figure]
For the AthletePlaysInLeague task
Query:
[Equation]
[Table]
concept_coach_j_j__hardy , concept : athleteplayssport , concept_sport_baseball concept_athlete_hunter_pence , concept : athleteplaysinleague , concept_sportsleague_mlb concept_athlete_hunter_pence , concept : athleteplayssport ,
concept_sport_baseball concept_sportsleague_mlb , concept : coachesinleague_inv , concept_athlete_ben_hendrickson concept_sportsleague_mlb , concept : coachesinleague_inv , concept_coach_j_j__hardy
[Figure]
For the AthleteHomeStadium task
[Table]
For the AthletePlaysSport task
[Table]
[Figure] [Figure]
[Table]
[Figure]
concept_sportsteam_
[Figure]
_sportsteam_
[Figure]
[Figure]
[Figure]
[Figure]
For the TeamPlaysSport task
[Table]
For the OrganizationHeadQuarteredInCity task
[Table]
[Figure]
[Table]
For the WorksFor task
[Table]
[Table]
balmer
[Table]
[Figure] [Figure] paul_allen [Figure]
[Figure]
[Figure]
[Figure]
concept_date_bill
[Figure]
[Figure]
concept_product_powerpoint [Figure] [Figure] concept_product_word_
[Figure]
_mlalgorithm_
[Figure]
[Figure]
[Figure]
[Figure]
concept_
[Figure]
[Figure]
[Figure]
concept
[Figure]
concept_politicianus_rodham
[Figure]
concept_geopoliticallocation
[Figure]
concept
[Figure]
concept
[Figure]
concept
[Figure]
politicalparty_
[Figure]
concept
[Figure]
concept_
[Figure]
concept
[Figure]
[Figure]
concept
[Figure]
concept_
[Figure]
geopoliticallocation_
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] concept_university_
[Table]
, concept : personleadsorganization_inv , concept_person_bill concept_university_microsoft , concept : organizationterminatedperson , concept_scientist_balmer concept_university_microsoft , concept : personleadsorganization_inv , concept_person_robbie_bach concept_personus_steve_ballmer , concept : topmemberoforganization , concept_company_microsoft concept_personus_steve_ballmer , concept : agentcollaborateswithagent_inv , concept_university_microsoft concept_personus_steve_ballmer , concept : personleadsorganization , concept_university_microsoft concept_personus_steve_ballmer , concept : worksfor , concept_
[Table]
For the PersonBornInLocation task
[Table]
[Figure]
[Figure]
[Table]
For the PersonLeadsOrganization task
[Table]
concept_journalist_bill_plante concept_televisionnetwork_cbs concept_person_edward_r__murrow concept_personus_scott_pelley concept_actor_daniel_schorr concept_journalist_wyatt_andrews concept_athlete_anthony_mason concept_journalist_connie_chung concept_journalist_andy_rooney concept_journalist_ed_bradley concept_writer_bill_geist concept_person_byron_pitts concept_personaustralia_harry_smith concept_journalist_morley_safer concept_journalist_gloria_borger concept_writer_bernard_goldberg concept_person_eric_sevareid concept_journalist_lesley_stahl concept_personmexico_david_martin concept_journalist_walter_cronkite concept_personaustralia_phil_jones concept_company_cnn__pbs concept_nonprofitorganization_cbs_evening concept_musicartist_television concept_crustacean_tv concept_website_cbs_evening_news concept_city_new_york concept_personeurope_william_paley concept_coach_douglas_edwards concept_academicfield_media concept_person_sean_mcmanus
concept_televisionstation_wcsc
concept_comedian_leslie_moonves concept_person_nina_tassler concept_televisionstation_kion_tv concept_televisionstation_wowk_tv concept_televisionstation_kbci_tv
concept_televisionstation_wjtv concept_politicsissue_entertainment concept_person_les_moonves concept_televisionstation_kfda concept_televisionstation_wdtv concept_televisionstation_khqa_tv concept_televisionstation_wrbl concept_televisionstation_wltx_tv concept_website_cbs concept_company_cbs_corp_ concept_recordlabel_universal concept_blog_mtv concept_journalist_bill_plante concept_televisionnetwork_cbs concept_person_edward_r__murrow concept_personus_scott_pelley concept_actor_daniel_schorr concept_journalist_wyatt_andrews concept_athlete_anthony_mason concept_company
[Figure]
[Table]
For the OrganizationHiredPerson task
[Table]
[Figure]
[Figure]
[Table]
concept_tradeunion_congress , concept : worksfor_inv , concept_personmexico_ryan_whitney concept_tradeunion_congress , concept : organizationhiredperson , concept_personmexico_ryan_whitney concept_country_left_parties , concept : organizationhiredperson , concept_personus_party concept_person_mark
concept_sportsteam_state_university concept_male_world concept_politicalparty_college
concept_sportsgame_series concept_university_state_university concept_charactertrait_world concept_university_college concept_person_diane concept_university_high_school concept_televisionshow_passion concept_musicsong_gospel concept_university_syracuse_university concept_city_hampshire concept_person_louise concept_jobposition_king concept_county_york_city concept_stateorprovince_california concept_stateorprovince_maine concept_stateorprovince_georgia concept_person_greg001concept_person_michael concept_stateorprovince_author
concept_person_bill concept_person_kevin concept_person_john
concept_politician_jobs
concept_person_mike concept_person_stephen concept_person_fred concept_person_karen concept_person_william001 concept_person_tom concept_journalist_dan concept_person_joe concept_person_adam concept_politician_james
concept_eventoutcome_result concept_geopoliticallocation_world concept_recordlabel_friends concept_politicalparty_house concept_company_apple concept_televisionnetwork_cbs concept_company_apple  concept_automobilemaker_announcement concept_personmexico_ryan_whitney concept_museum_steve concept_city_downtown_manhattanconcept_attraction_nycconcept_city_nyc_ concept_personasia_number concept_scientist_no_ concept_country_united_states concept_geopoliticallocation_agencies concept_company_nbc concept_city_team concept_terroristorganization_state concept_automobilemaker_jeff_bezos concept_ceo_richard concept_lake_new concept_city_service
concept_governmentorganization_u_s__
[Figure]
For the AgentBelongsToOrganization task
[Table]
[Table]
[Figure]
For the TeamPlaysInLeague task
[Table]