E FFICIENT T RAINING AND I NFERENCE OF H YPERGRAPH R EASONING N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of hypergraph reasoning in large domains, e.g., predicting the relationship between several entities based on the input facts.
We observe that in logical reasoning, logical rules (e.g., my parent’s parent is my grandparent) usually apply locally (e.g., only three people are involved in a grandparent rule), and sparsely (e.g., the grandparent relationship is sparse across all pairs of people in the world).
Inspired by these observations, we propose Sparse and Local Neural
Logic Machines (SpaLoc), a structured neural network for hypergraph reasoning.
To leverage the sparsity in hypergraph neural networks, SpaLoc represents the grounding of relationships such as parent and grandparent as sparse tensors and uses neural networks and ﬁnite-domain quantiﬁcation operations to infer new facts based on the input.
We further introduce a sparsiﬁcation loss to regularize the number of hyperedges in intermediate layers of a SpaLoc model.
To enable training on large-scale graphs such as real-world knowledge graphs, SpaLoc makes training and inference-time sub-sampling of the input graphs.
To remedy the information loss in sampled sub-graphs, we propose a novel sampling and label calibration paradigm based on an information-theoretic measure information sufﬁciency .
Our
SpaLoc shows superior accuracy and efﬁciency on synthetic datasets compared with prior art and achieves state-of-the-art performance on several real-world knowledge graph reasoning benchmarks.
I NTRODUCTION
Performing graph reasoning in large domains, such as predicting the relationship between two entities based on the input facts, is an important practical problem that arises in reasoning about molecular modeling, knowledge networks, and collections of objects in the physical world (Schlichtkrull et al., 2018;
Veli ˇ ckovi ´ c et al., 2020; Battaglia et al., 2016).
The necessary inference rules are generally unknown and must be inferred from data, which in general are large.
In this paper, we focus on learning neural networks for graph reasoning tasks.
Consider the problem of learning a rule that explains the grandparent relationship.
Given a dataset of labeled family relationship graphs, we aim to build machine-learning algorithms that can learn to predict a speciﬁc relationship (e.g., grandparent ) based on other relationships, such as father ( x, y ) and mother ( x, y ) .
Neural Logic Machines (NLM; Dong et al., 2019) present a method for solving graph reasoning tasks with a structured neural network.
NLMs keep track of hyperedge representations for all tuples consisting of up to B entities.
Thus, they can infer more complex ﬁnitely-quantiﬁed logical relations than standard graph neural networks that only consider binary relationships between entities (Morris et al., 2019; Barcel´o et al., 2020).
However, there are two disadvantages of such a dense hypergraph representation.
First, the training and inference of NLMs requires simultaneously considering all entities in a domain, such as all of the N people in a family relationship database.
Second, they scale polynomially with respect to the number of entities considered in a single inference.
Even inferring a single relation like grandparent requires O ( N 3 ) time and space complexity.
In practice, for large graphs, these limitations make the training and inference intractable and hinder the application of
NLMs in large-scale real-world domains.
To address these two challenges, we present a novel framework, called Sparse and Local Neural
Logic Machines (SpaLoc), for inducing sparse relational rules from data in large domains.
Our key idea is to exploit locality and sparsity in data: determining a relationship between entities
I
[Figure]
Unary: p(x)
Binary: r(x, y)
Tenary: r(x, y, z) …
Expand
Reduce
Expand
Reduce
Expand
Reduce
Expand …
Concat + PermuteNN
NN NN
NN … … … … …
NN
NN NN
NN … …
NN
NN NN
NN NN
NN NN
[Figure]
usually only requires consideration of a small number of additional entities, and the resulting relation usually only holds for a small number of tuples of entities.
Our contribution is three-fold.
First, we develop a sparse tensor-based representation for representing hyperedge relationships among entities and making inferences about them.
Second, during both training and inference, SpaLoc employs a sub-graph sampling technique based on an information-theoretic measure, information sufﬁciency , which quantiﬁes the amount of information contained in a sub-graph with respect to whether a predicate is true for some tuple of entities.
Since the sub-graph sampling may violate the closed-world assumption (i.e., the information in a sub-sampled graph may be insufﬁcient in predicting the relationship between a pair of entities), we also use the information sufﬁciency measure to calibrate training labels.
Third, to further speed up inference on large graphs, we encourage neural networks recover sparse relationships among objects by using a regularization term based on graph sparsity measurements.
We evaluate SpaLoc on two benchmarks: relational reasoning in synthetic datasets (family trees and general graph reasoning) and real-world knowledge-graph reasoning.
First, we show that, with our sparsity regularization, the computation complexity for inference can be reduced to the same order as the optimal complexity, which signiﬁcantly outperforms the base model NLM.
Second, we show that training via sub-graph sampling and label calibration enables us to learn relational rules in real-world knowledge graphs with more than 10K nodes, whereas the original NLM can be barely applied to graphs with more than 100 nodes.
Finally, SpaLoc achieves state-of-the-art performance on several real-world knowledge graph reasoning benchmarks.
S PARSE AND L OCAL N EURAL L OGIC M ACHINES
In this section, we develop a structured neural network that applies to hypergraphs.
The fundamental structures used for both training and inference, will be hypergraphs H =
( V , E ) , where V is a set of vertices and E is a set of hyperedges.
Each hyperedge e = ( x 1 , x 2 , · · · , x r ) is an ordered tuple of r elements ( r is called the arity of the edge), where x i ∈ V .
We use f : E → S to denote a hyperedge representation function , which maps each hyperedge e to a feature in domain S .
Domain S can be in various forms: discrete labels, numbers, and vectors.
For simplicity, we describe features associated with arity-1 edges as “node features” and features associated with the whole graph as “nullary” or “global” features.
A graph-reasoning task can be formulated as follows: given H and the input hyperedge representation functions f associated with all hyperedges in E , such as node types and pairwise relationships (e.g., parent ), our goal is to infer a target representation function f (cid:48) for one or more hyperedges, i.e. f (cid:48) ( e ) for some e ∈ E , such as predicting a new relationship (e.g., grandparent ).
2. P RELIMINARY : N EURAL L OGIC M ACHINES
We provide an informal overview of the global architecture of Neural Logic Machines (NLMs); for a formal description, see the original paper by Dong et al.
In contrast to typical graph neural networks that only associate representations with input edges in E , NLMs operate on a fully-
[Figure]
connected hypergraph.
Thus, NLMs use dense tensors to represent the input, output and intermediate representations of relations on nodes, edges, and hyperedges.
For example, the input relation parent ( x, y ) in a given hypergraph H can be represented as a 2D tensor parent H of size N 2 , where N = |V| is the number of entities in the hypergraph, and parent H
[ i, j ] =
if and only if person i is a parent of j .
In general, a vector representation of length D associated with hyperedges of arity r can be represented using a tensor of size N r × D .
An NLM is a multi-layer neural network that operates on this dense hypergraph representation.
At each layer, it computes a new set of features associated with the hyperedges based on the input.
To fuse the information across hyperedges of different arities, NLMs use two special tensor operations,
EXPAND and REDUCE , which add and reduce dimensions of the representation tensors.
Remark .
Even when the inputs have only unary and binary relations, allowing intermediate tensor representations of higher arity to be associated with hyperedges increases the expressiveness of
NLMs (Dong et al., 2019).
An intuitive example is that, in order to determine the grandparent relationship, we need to consider all 3-tuples of entities, even though the input relations are only binary.
Despite their expressiveness, NLMs based on hyperedges cannot be directly applied to large-scale graphs.
For a graph with more than 10,000 nodes (e.g., Freebase (Bollacker et al., 2008)), it is almost impossible to store vector representations for all of the N 3 tuples of arity 3.
Our key observation to improve the training and inference efﬁciency of NLMs is that relational rules are usually applied sparsely (Section 2.3) and locally (Section 2.4).
2. A RCHITECTURE OF S PARSE AND L OCAL N EURAL L OGIC M ACHINES
Fig.
1 shows the overall architecture of SpaLoc, which can be seen as a version of NLMs based on sparse tensors.
A SpaLoc model is composed of multiple relational reasoning layers ( RRL s) that operate on hyperedge representations.
It also shows the detailed computation graph at a single layer in a NLM with 3-ary relations.
The EXPAND operation propagates representations from lower-arity tensors to a higher-arity form (e.g., from each node to the edges connected to it).
The
REDUCE operation aggregates higher-arity representations into a lower-arity form (e.g., aggregate the information from all edges connected to a node into that node).
The PERMUTE operation fuses the representations of hyperedges that share the same set of entities but in different order, such as ( A, B ) and ( B, A ) .
Finally, NN is a linear layer with nonlinear activation that computes the representation for the next layer.
The neural network module NN is applied independently and identically to all tuples of the same arity.
This ensures that a NLM learned on a small universe can be applied to a larger universe of entities.
Our RRL s are based on sparse tensor representations.
In the following, we let N = |V| be the number of entities (nodes) in the hypergraph, V = { o 1 , o 2 , · · · o N } be the entities, x 1 , x 2 , · · · be variables, and f ( x 1 , x 2 , · · · , x r ) be a mapping from a tuple ( x 1 , x 2 , · · · , x r ) to a vector representation associated with that tuple.
We will denote the maximum arity of tensors considered in a SpaLoc model as R .
Sparse tensor representation of hyperedge representations.
In SpaLoc, the vector representations f ( x 1 , x 2 , · · · , x r ) associated with all hyperedges of arity r are represented as a coordinate-list (COO) format sparse tensor.
That is, each tensor is represented as two tensors F =
( I , V ) , each with
M entries.
The ﬁrst tensor I is an index tensor, of shape M × r , in which each row denotes a tuple ( x 1 , x 2 , · · · , x r ) .
The second tensor V is a value tensor, of shape M × D , where D is the length of f ( x 1 , x 2 , · · · , x r ) .
Each row V [ i ] denotes the vector representation associated with the tuple I [ i ] .
For all tuples that are not recorded in I , their representations are treated as all-zero vectors.
As an example, in Fig. 2, the input relationship parent ( x, y ) is represented as a sparse tensor with three entries.
Sparse relational reasoning layers.
Now, we show how each of the critical operations of an NLM can be performed on this sparse representation.
The EXPAND operation takes a sparse tensor F of arity r and creates a new sparse tensor F (cid:48) with arity r + 1 : it propagates the representation from arity r to arity ( r + 1) , such as from nodes to edges.
This is implemented by duplicating each entry f
( x 1 , · · · , x r ) in F by N times, creating the N new vector representations for ( x 1 , · · · , x r , o i ) for all i ∈ { 1 , 2 , · · · , N } .
Fig.
2 gives an example.
The REDUCE operation takes a sparse tensor F
=
( I , V ) of arity r and creates a new sparse tensor F (cid:48) with arity r − 1 : it aggregates all information associated with all r -tuples: ( x 1 , x 2 , · · · , x r − 1 , ?)
with the same r − 1 preﬁx.
In SpaLoc, the aggregation function is chosen to be max .
Thus, f
[Equation]
Fig.
2 gives an example.
The PERMUTE operation takes a sparse tensor F of arity r and creates a new sparse tensor F (cid:48) of the same arity.
However, the length of the vector representation will grow from D to D (cid:48)
=
r !
× D .
It fuses the representation of hyperedges that share the same set of entities.
Mathematically,
[Equation]
If a permutation of ( x 1 , · · · , x r ) does not exist in F , it will be treated as an all-zero vector. Thus, the number of entries M may increase or remain unchanged.
Fig.
2 gives an example.
The i -th sparse relational reasoning layer has R + 1 linear layers L ( i, 0) , L ( i, 1) , · · · , L ( i,R ) with nonlinear activations (e.g., ReLU) as submodules with arities 0 through R .
For each arity r , we will concatenate the feature tensors expanded from arity r − 1 , reduced from arity r + 1 , and the output from the previous layer, apply a permutation, and apply L ( i,r ) on the derived tensor.
In the concatenation step, since tensors from different sources may have different sets of entries, we will match the entries for different tensors with the necessary all-zero tensors.
We have seen how a sparse representation for NLMs can support efﬁcient inference.
In the next sections, we show how to learn sparse NLMs from graphs, and then how to use subsampling to learn efﬁciently from very large training graphs.
2. S PARSIFICATION THROUGH H OYER R EGULARIZATION
SpaLoc achieves sparsity during training by adding regularization terms to entries that are not useful in making the target prediction.
At each layer, for each hyperedge ( x 1 , · · · , x r ) , our model jointly predicts a vector representation f
( x 1 , · · · , x r ) as well as a hyperedge gate g ( x 1 , · · · , x r ) ∈ [0 , 1]
During training, we modulate the input representation f with the gate value g ; during inference, we prune all hyperedges whose gate value is smaller than a scalar hyperparameter (cid:15)
= 5% .
We augment the task loss with a sparsity regularization loss that encourages the network to prune as many edges as possible.
Gated hyperedge representation.
Recall that each sparse relational reasoning layer is composed of R + 1 linear layers.
In SpaLoc, for each linear layer L ( i,r ) , we add linear gating layer, L ( i,r ) g , which has sigmoid activation and outputs a scalar value in range [0 , 1] that can be interpreted as the importance score for each hyperedge.
During training, we modulate the output of L ( i,r ) with this importance value.
Speciﬁcally, the output of layer i arity r is L ( i,r ) ( F ) (cid:12) L ( i,r ) g ( F ) , where F is the input sparse tensor, and (cid:12) is the element-wise multiplication operation.
During inference, we prune out edges with a small importance score.
Hoyer-Square measure.
We use a metric inspired by the Hoyer measure (2004) for quantifying the sparsity of the hyperedges and as the regularization loss to encourage hyperedge sparsity.
It is based on the L 1 norm and the L 2 norm of a vector.
Let x be an arbitrary vector of length n .
Then
[Equation]
The Hoyer measure takes values from 0 to 1.
The larger the Hoyer measure of a tensor, the denser the tensor is.
In order to assign weights to different tensors based on their size, we use the Hoyer-Square measure (Yang et al., 2020),
[Equation]
which ranges from 1 (sparsest) to n (densest).
Intuitively, the Hoyer-Square measure is more suitable than L 1 or L 2 regularizers for graph sparsiﬁcation since it encourages large values to be close toand others to be zero, i.e., extremity.
It has been widely used in sparse neural network training and empirically proved to have better performance than other sparse measures Hurley & Rickard (2009). We also empirically compare H S with other sparsity measures in Appendix C.
The overall training objective of SpaLoc is the task objective plus the sparsiﬁcation loss, L =
L task + λ L density , where L density is the sum of the H S for all gate tensors across all layers, divided by the sum of the size of these tensors:
[Equation]
[Figure] ) where i sums over all layers of the SpaLoc and r sums over all arities and # counts the number of elements in a tensor.
2. S UBGRAPH T RAINING
[Figure]
Regularization enables us to learn a sparse model that will be efﬁcient at inference, but does not address the problem of learning efﬁciently from large training graphs.
In this section we describe a novel strategy for substantially reducing training complexity, based on the observation that an inferred relation among a set of entities generally depends only on a small set of other entities that are “between” the target entities in the hypergraph, in the sense that they are connected via short paths of relevant relations.
To exploit this observation, we employ a sub-graph sampling procedure during both training and inference.
In the following, we ﬁrst present a information-theoretic measure to quantify the sufﬁciency of information in a sub-sampled graph for determining the relationship between two entities, namely, information sufﬁciency .
Next, we present a sub-graph sampling procedure that is designed to maximize the information sufﬁciency for training and inference.
Finally, since it is inevitable that we will lose some information during the sampling procedure, we further propose a training paradigm that calibrates the labels for training examples based on the information sufﬁciency.
Information sufﬁciency.
Let H S =
( V S , E S ) be a sub-hypergraph of hypergraph H =
( V , E ) , and e ∗ =
( y 1 , . . .
, y r ) be a target hyperedge in H S , where y 1 , · · · , y r ∈ V S ⊂ V .
Intuitively, in order to determine the label for this hyperedge, we need to consider all “paths” that connect the nodes { y 1 , . . .
, y r } .
More formally, we say a sequence of K hyperedges
[Equation]
is a hyperedge path for nodes { y 1 , · · · y r } if and only if { y 1 , · · · y r } ⊂ (cid:83)
Kj =1 e j and e j ∩ e j +1 (cid:54)
= ∅ for all j .
In a graph with only binary edges, this is equivalent to the existence of a path from one node y 1 to another node y 2 .
We deﬁne the information sufﬁciency measure for a hyperedge e ∗ in subgraph H S as the fraction of the total number of paths connecting { y i , · · · , y r } that are retained in the subgraph:
[Equation]
In practice, we approximate IS by only counting the number of paths whose length is less than a task-dependent threshold τ for efﬁciency.
The number of paths in a large graph can be pre-computed and cached before training, and the overhead of counting paths in sampled graph is small, so this computation does not add much overhead to training and inference.
In the case that the input graphs have maximum arity 2, paths can be counted efﬁciently by taking powers of the graph adjacency matrix.
Subgraph sampling.
We use different sub-graph sampling strategies for training and inference.
During training, each data point is a tuple ( H , f, f (cid:48) ) where H is the input graph, f is the input representation, and f (cid:48) is the desired output representation.
We will sample a subgraph H (cid:48)
⊂ H , and train models to predict the value of f (cid:48) on H (cid:48) given f .
For example, we will train models to predict the grandparent relationship between all pairs of entities in H (cid:48) based on the parent relationship between entities in H (cid:48) .
Thus, our goal is to ﬁnd a subgraph that retains most of the paths connecting nodes in this subgraph.
We achieve this using a neighbor expansion sampler that uniformly samples a few nodes from V as the seed nodes.
It then samples new nodes connected with one of the nodes in the graph into the sampled graph and runs this “expansion” procedure for multiple iterations to get V S .
Finally, we include all edges that connect nodes in V S to form the ﬁnal subsampled hypergraph.
During inference, usually our goal is to just infer the relationship between one pair of entities f (cid:48)
( y 1 , y 2 ) on a graph H given the input representation f .
In this case, we will use a path sampler when dealing with large domains, which samples paths connecting y 1 and y 2 and induce a subgraph from these paths.
We provide ablation studies comparing different sampling techniques in Section 3.4.
The implementation details of our information sufﬁciency computation and subgraph samplers can be found in Appendix D and Appendix B.
Training label calibration with IS.
Due to the information loss caused by graph subsampling, the closed-world assumption (CWA) no longer holds in the subgraph.
That is, the information contained in the subgraph may not be sufﬁcient to make predictions about a target relationship.
For example, in a family relationship graph, removing a subset of nodes may cause the system unable to conclude whether a speciﬁc person x has a sibling.
Thus, we propose to calibrate the model training by assigning each example f (cid:48)
( y 1 , · · · , y r ) with a soft label.
Speciﬁcally, consider a binary classiﬁcation task f (cid:48) .
That is, function f (cid:48) is a mapping from a hyperedge tuple of arity r to { 0 , 1 } .
Denote the model prediction as ˆ f (cid:48) .
Typically, we will train the
SpaLoc model with a binary cross-entropy loss between ˆ f (cid:48) and the groundtruth f (cid:48) .
In our subgraph training, we will instead compute a binary cross-entropy loss between ˆ f (cid:48) and f (cid:48) H S (cid:12) IS , where H S is the sub-sampled graph.
Mathematically,
[Equation]
Fig.
3 shows a concrete example of the label calibration process.
[Table]
E XPERIMENTS
In this section
[Table]
we compare SpaLoc with other methods in two aspects: accuracy and efﬁciency on large domains.
We compare SpaLoc with other approaches on relational reasoning and real-world knowledge graph reasoning benchmarks.
All experiments are under the inductive, supervised learning setup, i
[Table]
e., testing objects and relations have not been seen during training [Table] and the sizes of domains used for training and evaluation are also very different [Table]
3. B ASELINES
For the family tree reasoning tasks, we compare SpaLoc against four baselines.
The ﬁrst three are Memory Networks (MemNN) (Sukhbaatar et al., 2015), ∂ ILP (Evans & Grefenstette, 2018) and Neural Logic Machines (NLMs; Dong et al., 2019), which are state-of-the-art models for relational rule learning tasks.
For these models, we follow the conﬁguration and setup in Dong et al.
The fourth baseline is an inductive link prediction method based on graph neural networks, GraIL (Teru et al., 2020).
Since GraIL can be only used for link prediction, we use the full-batch R-GCN (Schlichtkrull et al., 2018), the backbone network of GraIL, for node property predictions.
For inductive knowledge graph reasoning datasets, we compare SpaLoc with state-ofthe-art models, including Neural LP
(Yang et al.), DRUM (Sadeghian et al.), RuleN (Meilicke et al., 2018), GraIL and TACT (Chen et al., 2021). For transductive learning tasks, we compare SpaLoc with four representative knowledge graph embedding methods, including TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2017), and RotatE (Sun et al., 2019).
3. F AMILY T REE R EASONING
[Figure]
We evaluate SpaLoc on the family-tree reasoning benchmark, a synthetic benchmark for inductive logic programming.
The goal of family-tree tasks is to induce target family relationships or member properties given the following binary relations: Son , Daughter , Father , and Mother .
We give concrete logical deﬁnitions of the family tree benchmark in the Appendix E.
Accuracy & Efﬁciency .
From Table 1 we observe the superiority of SpaLoc in terms of accuracy and efﬁciency.
SpaLoc achieves perfect accuracy on test set.
And when handling large input domains [Table] ( N train = 2000 ), SpaLoc works well because of the sampling techniques, while most baselines cause out-of-memory errors [Table]
Sparsity .
In Fig.
[Table] we show the number of non-zero elements in SpaLoc’s intermediate groundings versus the size of input domains on HasSister , Grandparent and Uncle .
We ﬁt a cubic polynomial equation ˆ f to the data points to illustrate the learned inference complexity of SpaLoc.
The SpaLoc we use in these three datasets has an arity of 3 so the maximum capacity of the model is Θ( N 3 ) .
However, the memory complexity of SpaLoc converges to the optimal algorithm complexity of these tasks , which is O ( N 2 ) .
In comparison, the memory complexity of original dense NLMs will be ﬁxed to Θ( N 3 ) when the model architecture is ﬁxed.
3. R EAL - WORLD K NOWLEDGE G RAPH R EASONING
[Table] We evaluate SpaLoc on the real-world knowledge-graph inductive reasoning benchmarks proposed in GraIL, whose training and evaluation sets are disjoint sub-graphs extracted from WN18RR (Dettmers et al., 2018),
FB15k-237 (Toutanova et al., 2015), and NELL-995 (Xiong et al., 2017).
For each knowledge graph, there are four versions of inductive datasets with increasing sizes.
To further demonstrate the scalability of SpaLoc, we also apply SpaLoc to the complete real knowledge graphs.
SpaLoc still uses the inductive learning setting.
That is, it does not use any entity information learned from training, only using the structural information for inference.
We use the standard WN18RR, FB15K-237, and NELL-995 benchmarks.
We follow the original data splitL.
Table 2 and Table 3 show the results.
SpaLoc signiﬁcantly outperforms all baselines on all datasets and achieves a new state-of-the-art.
This demonstrates SpaLoc’s excellent scalability and generalization ability on noisy large-scale real-world data.
3. A BLATION S TUDY
[Table]
[Table]
Subgraph sampling.
Beside the neighbor expansion sampler , there are also some intuitive and efﬁcient sub-graph samplers, such as the random node ( Node ) and random walk ( Walk ) samplers proposed by Zeng et al.
We compare these samplers in two ways.
Empirically, we test the models trained with the samplers.
Theoretically, we evaluate the overall information sufﬁciency of a sub-graph by averaging the information sufﬁciency of all target edges, introducing a metric called
Mean Information Sufﬁciency (MIS).
In Table [Table] we show the results on the Grandparent dataset.
The Node sampler does not leverage locality, so the performance of models and MIS drops with the size of whole graph increases.
SpaLocs trained with Walk and Neighbor samplers achieve perfect accuracies, and the MIS of sub-graphs provided by these sampers are also comparable.
Also, the MIS of Walk ’s and Neighbor ’s subgraphs do not drop signiﬁcantly with the increasing size of the whole graph, which ensures the application of these samplers on arbitrarily large graphs.
Label calibration.
We compare our information sufﬁciency based label calibration method against a simple constant label smoothing.
We show our results in Table 5.
In the table, “LS” means constant label smoothing by multiplying positive labels with a constant α = 0 .
9 , and “IS” means label calibration with the information sufﬁciency.
The impact of sampling and the failure of the closedworld assumption is particularly signiﬁcant on the two datasets HasFather and HasSister .
Without label calibration, the performance of the model trained on the sub-graphs is close to random prediction.
Our information-based calibration method improves the performance by a large margin, and is signiﬁcantly superior to naive smoothing techniques.
R ELATED W ORK
(Hyper-)Graph representation learning .
(Hyper-)Graph representation learning methods, including message passing neural networks (Shervashidze et al., 2011; Kipf & Welling, 2017; Velickovic et al., 2018; Hamilton et al., 2017) and embedding based methods (Dettmers et al., 2018; Toutanova et al., 2015), have been widely used for knowledge discovery.
Since these methods treat relations (edges) as ﬁxed indices for node feature propagation, their computational complexity is usually small (e.g., O ( NE ) ), and they can be applied to large datasets.
However, the ﬁxed relation representation and low complexity restrict the expressive power of these methods (Xu et al., 2019; 2020) so they cannot be used to solve some general complex problems, such as inducing tenary relational rules.
Moreover, some of the widely used methods such as knowledge embeddings are inherently transductive, and cannot learn lifted rules that generalize to unseen domains.
By contrast, relations in SpaLoc are evolving for reasoning, and the architecture of SpaLoc can be adjusted for problems of different complexities.
Therefore, SpaLoc can learn lifted and complex rules.
Inductive rule learning .
In addition to graph learning frameworks, many previous approaches have studied how to learn generalized rules from data (Muggleton, 1991; Friedman et al., 1999), i.e., inductive logic programming (ILP), with recent work integrating neural networks into ILP systems to combat noisy and ambiguous inputs (Dong et al., 2019; Evans & Grefenstette, 2018; Sukhbaatar et al., 2015).
However, due to the large search space of target rules, computational and memory complexities of these models are too high to scale up to many large real-world domains.
SpaLoc addresses this scalability problem by leveraging the sparsity and locality of real-world rules, and thus can induce knowledge with local computations and optimal complexities.
Efﬁcient training and inference methods .
There is a rich literature on efﬁcient training and inference of neural networks.
Two directions that are relevant to us are model sparsiﬁcation and sampling training.
Han et al.
(2016) propose to prune and compress the weights of neural networks for efﬁciency, and Yang et al.
(2020) adopt Hoyer-Square regularization to sparsify models.
SpaLoc extend this sparsiﬁcation idea by adding regularization at intermediate sparse tensor groundings to encourage concise induction.
Chiang et al.
and Zeng et al.
(2020) propose to sample sub-graphs for GNN training, and Teru et al.
(2020) propose to construct sub-graphs for link prediction.
SpaLoc generalizes these sampling methods to hypergraphs, and proposes the information sufﬁciency based calibration method to remedy the information loss introduced by sub-sampling.
C ONCLUSION
We present SpaLoc, a framework for efﬁcient training and inference of hypergraph reasoning networks.
SpaLoc uses sparse tensors to represent high-order relationships (hyperedges), and implements ﬁnitedomain quantiﬁcation to infer new knowledge.
SpaLoc leverages sparsity and locality to train and infer efﬁciently.
Through regularizing intermediate representation by a sparsiﬁcation loss, SpaLoc’s inference complexities on family tree tasks are the same as those of algorithms designed by human experts.
To be applied to large datasets, SpaLoc sample sub-graphs for training and inference, and calibrate training labels with information sufﬁciency measure to alleviate information loss.
SpaLoc generalizes well on large-scale relational reasoning benchmarks.
R EPRODUCIBILITY S TATEMENT
To maximize reproducibility, we described our methodology in detail in Section 2 and our experimental setup in Section 3.
All the data and codes in this paper will be released upon publication to facilitate future research. S UPPLEMENTARY M ATERIAL A E XPERIMENTAL C ONFIGURATION
[Table]
We optimize all models with Adam (Kingma & Ba, 2015) and use an initial learning rate of 0.005.
All experiments are under the supervised learning setup, we use Softmax-Cross-Entropy as loss function.
Table A shows hyper-parameters used by SpaLoc.
For all MLP inside SpaLoc, we use no hidden layer and the sigmoid activation.
Across all experiments in this paper, the maximum arity of intermediate predicates (i.e., the “breadth”) is set to 3 as a hyperparameter, which allows SpaLoc to realize all ﬁrst-order logic (FOL) formulas with at most three variables, such as a “transitive relation rule.”
We set the speciﬁcation threshold (cid:15) to 0 .
in all our experiments.
This value is chosen based on the validation accuracy of our model.
In practice, we observe that any values between 0.01 and 0.1 do not signiﬁcantly impact the performance of our method and the inference complexity.
We also set the multiplier of the sparsiﬁcation loss λ to 0 .
01 in all SpaLoc’s experiments.
B I MPLEMENTATION OF S UBGRAPH S AMPLERS
Both the neighbor expansion and the path sampler sample subgraphs by inducing from selected node sets.
To deal with input hypergraphs with any arities, the samplers simplify the input hypergraphs into binary graphs.
We deﬁne two nodes in the hypergraph are connected if they are covered by a hyperedge.
Therefore, the neighbor expansion and path ﬁnding algorithms used by the samplers can be applied to any hypergraphs for ﬁnding node sets.
After enough nodes are collected, the sampler will induce a sub-hypergraph from the original hypergraph by preserving all of the hyperedges lying in the set.
C A BLATION S TUDY ON S PARSIFICATION L OSS
[Table]
We compare our Hoyer-Square sparsiﬁcation loss against the L 1 and L 2 regularizers on the family tree datasets.
In Table 7, we show the performance of SpaLoc trained with different sparsiﬁcation losses.
All models are tested on domains with 100 objects.
“Density” is the percentage of non-zero elements (NNZ) in the intermediate groundings.
The lower the density, the better the sparsiﬁcation and the lower the memory complexity.
We can see that, compared with L 1 and L 2 regularizers, the Hoyer-Square loss yields a higher or comparable sparsity while maintaining the nearly perfect accuracy.
D C ALCULATION OF THE I NFORMATION S UFFICIENCY
The crucial part in the computation of the information sufﬁciency is to count the number of k -hop hyperpaths connecting a given set of nodes { v 1 , . .
, v r } in a hypergraph.
We use the incidence matrix to calculate this.
Firstly, we use a n × m incidence matrix B to represent the hypergraph
H =
( V , E ) , where n = |V| and m = |E| , such that B ij = 1 if the vertex v i and edge e j are incident and 0 otherwise.
Next, B ( k ) :
= ( BB T ) k − 1 B is the k -hop incidence matrix of the hypergraph, i.e.,
B ( k ) ij is the number of k -hop paths that the vertex v i and edge e j are incident.
For example, when r = 2 , there are B ( k ) i B Tj k -hop paths connecting vertex v i and v j .
When r = 1 , there are (cid:80) j B ( k ) ij k -hop paths connecting to vertex v i .
E D EFINITION OF R ELATIONS IN F AMILY T REE
The inputs predicates are: Father ( x, y ) , Mother ( x, y ) , Son ( x, y ) , Daughter ( x, y ) .
The target predicates are:
• HasFather ( x ) :
= ∃ a Father ( x, a ) • HasSister ( x ) :
= ∃ a ∃ b Father ( x, a ) ∧ Daughter ( a, b ) ∧ ¬ ( b = x ) • Grandparent ( x, y ) :
= ∃ a parent ( x, a ) ∧ parent ( a, y ) parent ( x, y ) :
=
Father ( x, y ) ∨ Mother ( x, y ) • Uncle ( x, y ) :
= ∃ a Grandparent ( x, a ) ∧ Son ( a, y )
∧ ¬ Father ( x, y ) • MGUncle ( x, y ) :
= ∃ a ∃ b Grandmother ( x, a ) ∧ Mother ( a, b ) ∧ Son ( b, y )
Grandmother ( x, y )
=
∃ a Parent ( x, a ) ∧ Mother ( a, y ) • Family-of-three ( x, y, z ) =
Father ( x, y )
∧ Mother ( x, z ) • Three-generations ( x, y, z ) =
Parent ( x, y ) ∧ Parent ( y, z )
We follow the dataset generation algorithm presented in Dong et al.
In detail, we simulate the growth of families to generate examples.
For each new member of the family, we randomly assign a gender and a pair of parents (can be none, which means it the oldest person in the family tree) for it.
After the family tree is generated, we label the relationships according to the deﬁnitions above.
F E FFICIENCY A NALYSIS
We compare the time and memory complexity of SpaLoc against NLM on the Has-Sister and
Grandparent tasks.
In Fig.
5, we plot the curve of memory consumption and inference time of each sample as a function of the number of objects in the evaluation domains.
The experimental results show that our method can reduce the space complexity from the original O ( n 3 ) complexity of
NLM to the O ( n 2 ) complexity required to solve the has-sister and grandparent task.
In terms of the inference time, there is a constant level of optimization.
On small graphs, adding sparsity does not necessarily improve the running time on GPUs because of the overhead of handling sparse tensors due to the implementation.
More efﬁcient implementation of sparse operations is interesting future work.
[Figure]
G I MPLEMENTATION OF S PA L OC IN P Y T ORCH
The following python code is a minimal implementation for a single layer SpaLoc with PyTorch and † torch scatter .
The syntax is highlighted and is best viewed in color.
from itertools import permutations import SparseTensor import torch from torch import nn import math from torch_sparse import scatter   def expand(input:
SparseTensor): """Expands a new dimension in input by duplicating each entry for n times""" (cid:44) → coo = (input.coo.unsqueeze(-1) * input.n)
+ torch.arange(input.n).unsqueeze(0) (cid:44) → val = input.val.unsqueeze(1).expand(input.length, input.n, input.channel) (cid:44) → val =
val.contiguous().view(-1, input.channel) return SparseTensor(val, input.n, input.arity + 1, input.channel, coo=coo.flatten())
(cid:44) →   def reduce(input: SparseTensor): """Reduces max at the last dimension, except for elements whose coordinates contain duplicate objects.
(cid:44) → """ val =
input.val.clone() val[reduce_mask(input)]
= -math.inf
flatten_index =
input.coo // input.n coo, inverse_index = torch.unique(flatten_index, return_inverse=
True, sorted=
True) (cid:44) → val = scatter(val, inverse_index, dim=0, reduce='max') return SparseTensor(val, input.n, input.dim - 1, input.channel, coo=coo) (cid:44) →   def neural_logic(input, hidden_dim): """An MLP layer applied on permutations of the input."
"" return MLP(permute(input), hidden_dim, activation=nn.
Sigmoid()) (cid:44) →   def spaloc_breath3(input0, input1, input2, input3, hidden_dim): """A SpaLoc layer with breath 3.
Args: predicates.
(cid:44) → input1: predicates.
(cid:44) → predicates.
(cid:44) →
input3: SparseTensor of shape [N, N, N, hidden_dim], tenary predicates.
(cid:44) → hidden_dim: int, hidden dimension.
Returns: 4 SparseTensors, output nullary, unary, binary tenary predicates respectively.
(cid:44) → """ agg0 = concat([input0, reduce(input1)]) agg1 = concat([input1, expand(input0), reduce(input2)])
agg2 = concat([input2, expand(input1), reduce(input3)]) agg3 =
concat([input3, expand(input2)]) outputs =
[neural_logic(x, hidden_dim) for x in [agg0, agg1, agg2, agg3]] (cid:44) → return outputs   def reduce_mask(input): mask = torch.zeros(input.length, dtype=torch.bool) coo = input.coo for i in range(input.arity): for j in range(i+1, input.dim): mask = torch.logical_or(mask, coo[:, i]
== coo
[:, j]) return mask
def permute(input): if input.dim <= 1: return input index = tuple(range(0, input.dim)) coo = input.coo() weight = torch.tensor([[input.n ** i for i in reversed(range(input.dim))]]) (cid:44) → ts = [] for p in permutations(index): new_coo = (weight * coo[:, p]).sum(-1) ts.append(SparseTensor(input.val, input.n, input.dim, input.channel, coo=new_coo))
(cid:44) → return input.cat(ts)  def concat(inputs): nnzs
= [t.length for t in inputs] n = inputs[0].n dim = inputs[0].dim channel = sum([t.channel for t in inputs]) coo = torch.cat([t.coo for t in inputs], dim=-1) coo, inverse_index = torch.unique( coo, return_inverse=
True, sorted=
True) val = torch.zeros((len(coo), channel))
current_channel, current_idx = 0, for idx, t in enumerate(inputs): next_channel, next_idx =
current_channel + t.channel, current_idx + t.length (cid:44) →
val[inverse_index[current_idx:next_idx], current_channel:next_channel]
= t.val current_channel, current_idx =
next_channel, next_idx return SparseTensor(val, n, dim, channel, coo=coo)