O N THE E XPRESSIVENESS AND L EARNING OF R ELATIONAL N EURAL N ETWORKS ON H YPERGRAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a framework for analyzing the expressiveness and learning of relational models applied to hypergraph reasoning tasks.
We start with a general framework that uniﬁes several relational neural network architectures: graph neural networks, neural logical machines, and transformers.
Our ﬁrst contribution is a ﬁne-grained analysis of the expressiveness of these neural networks, that is, the set of functions that they can realize and the set of problems that they can solve.
Our result is a hierarchy of problems they can solve, deﬁned in terms of various hyperparameters such as depth and width.
Next, we analyze the learning properties of these neural networks, especially focusing on how they can be trained on a small graphs and generalize to larger graphs.
Our theoretical results are further supported by the empirical results illustrating the optimization and generalization of these models based on gradient-descent training.
I NTRODUCTION
Reasoning over graph-structured data is an important task in various applications, including molecule analysis, social network modeling, and knowledge graph reasoning (Gilmer et al., 2017; Schlichtkrull et al., 2018; Liu et al., 2017).
While we have seen great success of various relational neural networks, such as Graph Neural Networks (GNNs; Scarselli et al., 2008), Neural Logical Machines (NLM;
Dong et al., 2019), and Transformers (Vaswani et al., 2017) in a variety of applications (Battaglia et al., 2018; Merkwirth & Lengauer, 2005; Veliˇckovi´c et al., 2020), we do not yet have a full understanding of how different design parameters, such as the depth of the neural network, affects the expressiveness of these models, or how effectively these models generalize from data.
In this paper we develop a general framework, generalized relational neural networks (RelNNs), that uniﬁes a large class of relational neural networks including GNNs, NLMs, and Transformers.
Unlike previous work that analyzes the expressiveness of graph neural networks (Xu et al., 2019; 2020;
Barcel´o et al., 2020), our framework applies to hypergraphs, i.e., graphs that have representations that directly relate groups of more than two nodes.
Next, we quantify the expressiveness of generalized relational neural networks in terms of their structural parameters; in particular, the maximum arity of hyperedges and the depth of the network.
We formally proved the “if and only if” conditions for the expressive power w.r.t.
the arity.
That is, k -ary hyper-graph relational neural networks are sufﬁcient and necessary for realizing FOCk , a fragment of ﬁrst-order logic which involves at most k variables.
This is a helpful result because now we can determine whether a RelNN can solve a problem by thinking whether a logic formula can represent the solution to this problem.
Next, we formally described the relationship between expressiveness and non-constant-depth RelNNs.
We proposed a conjecture about the ”depth hierarchy,” and we connect the potential proof of this conjecture to distributed computing literature.
We also develope an upper bound for RelNNs.
This is a very interesting result: applying a k -ary hyper-graph RelNN for more than O ( n k − 1 ) iterations has no effect on the expressiveness.
Finally, we study the learning capabilities of RelNNs.
We prove, under certain realistic assumptions, it is possible to train a RelNN on a ﬁnite set of graphs, and it will generalize to arbitrarily large graphs.
This is an outcome due to the weight-sharing nature of RelNNs.
Next, we proved a ﬁner-grained sample-complexity bound for RelNNs.
This result theoretically suggests that using max aggregation function is provably better than sum aggregation function when realizing a ﬁrst-order logic formula.
We hope our work can serve as a foundation for designing relational neural networks: to solve a speciﬁc problem, what arity do you need?
What depth do you need?
What aggregation function should you use?
Will my model have structural generalization (i.e., to larger graphs)?
Our theoretical results on learning are further supported by experiments, for empirical demonstration of the theorems and for exploring the tightness of the sample-complexity bounds and effectiveness of different aggregation functions.
Speciﬁcally, we show that RelNNs with sufﬁcient expressiveness power can be trained using gradient descent to realize the desired function.
Meanwhile, when the training data distribution is uniformly distributed, RelNNs do demonstrate strong generalization to larger graphs than those seen during training.
Finally, we also empirically show the generalization of RelNNs with different aggregation functions.
G RAPH R EASONING AND R ELATIONAL N EURAL N ETWORKS
In this section, we deﬁne a very general class of relational networks and show that it can represent several important classes of relational neural networks, including graph neural networks, neural logic machines, and transformers.
2. G RAPH R EASONING P ROBLEM
A graph reasoning problem is a prediction task on a hypergraph G .
It can predict a discrete or real-valued property of the whole graph, of individual nodes, or of tuples of nodes.
Much of the work on graph-structured neural-network models has focused on cases with only binary edges, but many important problems are best modeled using hypergraphs , in which a hyperedge can connect multiple nodes.
A further generalization is to labeled hypergraphs, in which each hyperedge can have one of a discrete set of labels.
We will make use of a still more general structure, in which hyperedges can have real and/or vector-valued labels.
A hypergraph representation G is a tuple ( V, X ) , where V is a set of entities (nodes), and X is a set of hypergraph representation functions .
Speciﬁcally, X = {
X k 1 , X k 2 , X k 3 , · · · } , where
X k : ( v 1 , v 2 , · · · , v k ) → S is a function mapping every tuple of k nodes to a value.
We call k the arity of the hyperedge.
The range S can be any set of discrete labels that describes relation type, or a scalar number (e.g., the length of an edge), or a vector.
In general, we will use the arityrepresentation function X 0 ( ∅ ) → S to represent any global properties of the graph as a whole.
A graph reasoning function f is a mapping from a hypergraph representation G
=
( V, X ) to another hyperedge representation function Y on V .
There are several important types:
• graph classiﬁcation function: computes a global label, Y =
{ Y 0 } ; Y 0 ( ∅ ) → S (cid:48)
= { 0 , 1 } for the whole graph.
• graph regression function: computes a global real output, Y =
{ Y 0 } , where Y 0 ( ∅ ) → R .
• node classiﬁcation/regression function: computes Y =
{ Y 1 } , where Y 1 ( v 0 )
→ { 0 , 1 } is a binary label for each individual node.
• edge classiﬁcation function: extends previous prediction notions to higher-arity hyperedges.
For example, asking whether a graph is fully connected is a graph classiﬁcation problem; computing its diameter is a graph regression problem; ﬁnding the set of disconnected subgraphs of size k is a k -ary hyperedge classiﬁcation problem.
Example (undirected graph connectivity).
The input is G =
( V, X ) , where X = { X 2 } ) , and Xis mapping from each pair of nodes v 1 , v 2 ∈ V to { 0, 1 } , indicating whether ( v 1 , v 2 ) are connected.
The input is an undirected graph; thus, X 2 ( v 1 , v 2 ) =
X 2 ( v 2 , v 1 ) , ∀ v 1 , v 2 .
The output is Y =
{ Y 0 } , where Y 0 ( ∅ ) → { 0 , 1 } is a binary classiﬁcation label.
The desired label is 1 when the graph is connected, and 0 otherwise.
2. R ELATIONAL N EURAL N ETWORKS
We now introduce introduce generalized relational neural networks (RelNNs), a general framework that uniﬁes a set of important relational neural network models that can be trained to solve graph reasoning problems.
In particular, we will describe relational neural networks following the notation and computation graph of neural logic machines (NLMs; Dong et al., 2019).
We choose NLM as a base model because its form very naturally generalizes to hyperedges with arities greater than 2.
As we will see later, other relational neural networks such as graph neural networks (GNNs; Scarselli et al., 2008) and transformers (Vaswani et al., 2017) are equivalent to NLMs and each other in terms of their expressiveness.
Showing this equivalence allows us to focus the rest of the paper on analyzing a single model type, with the understanding that the conclusions generalize to this broader class of relational neural networks.
𝑇 !
RRL #,!
RRL #,#
RRL #,$
RRL #,% 𝑇 !,# 𝑇 !,$ 𝑇 !
,% 𝑇 #,!
𝑇 #,# 𝑇 #,$ 𝑇 #,% …… ……
RRL &,!
RRL &,#
RRL &,$
RRL &,% 𝑇 ’,!
𝑇 &,# 𝑇 &,$ 𝑇 &,% …… …… …… …… …… …… ……
𝑇 !"
#,%"# 𝑇 !"
#,% 𝑇 !"
Reduce
Expand
Permute
NN !
𝜎 𝑇 !
Concat
Nonlinearity 𝜎
RRL 𝒊,𝒋 (
[Figure]
A RelNN is a multi-layer neural network that operates on hypergraph representations, in which the hypergraph representation functions are represented as tensors.
The input is a hypergraph representation ( V, X ) .
There are then several computational layers, each of which produces a hypergraph representation with nodes V and a new set of representation functions.
Speciﬁcally, a
B -ary RelNN produces hypergraph representation functions with arities from 0 up to a maximum hyperedge arity of B .
We let T i,j denote the tensor representation for the output at layer i and arity j .
Each entry in the tensor is a mapping from a set of node indices ( v 1 , v 2 , · · · , v j ) to a vector in a latent space R W .
Thus, T i,j is a tensor of j +1 dimensions, with the ﬁrst j dimensions corresponding to j -tuple of nodes, and the last feature dimension.
For convenience, we write h 0 , · for the input hypergraph representation and h D, · for the output of the RelNN.
Fig.
1a shows the overall architecture of RelNNs.
It has D × B computation blocks, namely relational reasoning layers (RRLs).
Each block RRL i,j , illustrated in Fig.
1b, takes the output from neighboring arities in the previous layer, T i − 1 ,j − 1 , T i − 1 ,j and T i − 1 ,j +1 , and produces T i,j .
Below we show the computation of each primitive operation in an RRL.
The expand operation takes tensor T i − 1 ,j − 1 (arity j − 1 ) and produces a new tensor T Ei − 1 ,j − 1 of arity j .
The reduce operation takes tensor T i − 1 ,j +1 (arity j + 1 ) and produces a new tensor T Ri − 1 ,j +1 of arity j + 1 .
Mathematically,
[Equation]
Here, Agg is called the aggregation function of a RelNN.
For example, a sum aggregation function takes the summation along the dimension j + 1 of the tensor, and a max aggregation function takes the max along that dimension.
The concat (concatenate) operation (cid:76) is applied at the “vector representation” dimension.
The permute operation generates a new tensor of the same arity, but it fuses the representations of hyperedges that share the same set of entities but in different order, such as ( v 1 , v 2 ) and ( v 2 , v 1 ) .
Mathematically, for tensor X of arity j , if Y = permute ( X ) then
[Equation]
where σ ∈ S j iterates over all permuations of { 1 , 2 , · · · j } .
NN j is a multi-layer perceptron (MLP) applied to each entry in the tensor produced after permutation, with nonlinearity σ (e.g., ReLU).
It is important to note that we intentionally name the MLPs NN j instead of NN i,j .
In generalized relational neural networks, for a given arity j , all MLPs across all layers i are shared.
It is straightforward to see that this “weight-shared” model can realize a “non-weight-shared” RelNN that uses different weights for MLPs at different layers when the number of layers is a constant.
With a sufﬁciently large length of the representation vector, we can simulate the computation of applying different transformations by constructing block matrix weights.
(A more formal proof is in Appendix B.1)
The advantage of this weight sharing is that the network can be easily extended to a “recurrent” model.
For example, we can apply the RelNN for a number of layers that is a function of n , where n is the size (the number of nodes) of the input graph.
Thus, below, we will use the term layers and iterations interchangeably.
Handling high-arity features and using deeper models usually increase the computational cost.
In appendix B.6, we show that the time and space complexity of RelNN [ D, B ] is O ( n B ) .
Exanple (GNN).
The graph neural network (GNN) of Scarselli et al.
(2008) can be described as a speciﬁc type of RelNN.
First, GNNs work on graphs with maximum arity 2.
In a single iteration in a
[Table]
GNN, we use two message-passing subroutines: (edge update) the representation associated with each edge is updated by representations of its both ends, and (node update): the representation of each node is updated by all edge presentations connected to it.
This message passing scheme can easily ﬁt in the RelNN: edge update can be realized by the expand and permute operations, while node update can be realized by the reduce operation.
In general, hypergraph GNNs (Morris et al., 2019) are also equivalent to RelNNs.
Speciﬁcally, a
GNN applied to B -ary hyperedges is equivalent to an NLM applied to B + 1 -ary hyperedges.
We provide proof details in Appendix B.2.
There are other approaches of realizing GNN on hypergraphs such as hypergraph convolution(Feng et al., 2019; Yadati et al., 2019; Bai et al., 2021), attention(Ding et al., 2020) and message passing(Huang & Yang, 2021).
These approaches can be viewed as instances of RelNNs, but are less expressive than RelNNs with equal max arity.
We provide proof details in Appendix B.3.
Example (Transformer).
The key difference between Transformers (Vaswani et al., 2017) and
GNNs is the “self-attention” opeartion, which can be viewed as a special aggregation function.
In self-attention, each “sender” computes the key and value , and each “receiver” computes the query .
Xu et al.
(2019) proved such aggregation can be simulated with a sum aggregation function.
We will use calligraphic letters M 1 , M 2 , · · · to name RelNNs.
Because even when hyperparameters such as the maximum arity and the number of iterations are ﬁxed, a RelNN is still a model family M : the weights for MLPs will be trained on some data.
Furthermore, each model M ∈ M is a RelNN with a speciﬁc set of MLP weights.
E XPRESSIVENESS OF R ELATIONAL N EURAL N ETWORKS
A RelNN is characterized by hyperparameters D (depth), and B maximum arity.
In this section, we will focus on D and B , quantifying how they affect the expressiveness of a model; i.e., the set of graph reasoning functions a model family can realize.
The W parameter affects the precise details of what functions can be realized, as it does in a regular neural network, but does not affect the structural constraints that are our focus in this paper.
When considering D , we will focus on the distinction between any constant depth vs .
adaptive depth, in which D depends on the number n of nodes in the input graph.
We will use the notation RelNN[ D , B ] to represent a RelNN family with depth D and max arity B .
A summary of some concrete results is in table 1 and discussed in detail in Appendix A.
Many frameworks for characterizing the expressiveness of neural networks make a distinction between models that have the capacity to use unbounded precision in intermediate representations.
In all results we state below, we will indicate whether they apply to the ﬁxed or unbounded precision case.
Deﬁnition 3.1 (Expressiveness) .
We say a model family M 1 is at least expressive as M 2 , written as M 1 (cid:60) M 2 , if for all M 2 ∈ M 2 , there exists M 1 ∈ M 1 such that M 1 can realize M 2 .
A model family M 1 is more expressive than M 2 , written as M 1 (cid:31) M 2 , if M 1 (cid:60) M 2 and ∃ M 1 ∈ M 1 , ∀ M 2 ∈ M 2 , M 2 can not realize M 1 .
3. A RITY H IERARCHY
We ﬁrst aim to quantify how the maximum arity B of the network’s representation affects its expressiveness and ﬁnd that, in short, the higher the maximum arity, the more expressive the relational neural network is.
A majority of the results in this section are based on the Weisfeiler-Leman graph isomophism test (WL-test)
(Leman & Weisfeiler, 1968; Xu et al., 2019).
Theorem 3.1 (Arity Hierarchy) .
For any maximum arity B , there exists a depth D ∗ such that: ∀ D ≥ D ∗ , RelNN[ D , B + 1 ] is more expressive than RelNN[ D , B ].
This theorem applies to both ﬁxed-precision and unbounded-precision networks.
Proof sketch:
Our proof slightly extends the proof of Morris et al.
First, the set of graphs distinguishable by RelNN[ D , B ] is bounded by graphs distinguishable by a D -round order- B WLtest.
If models in RelNN[ D , B ] cannot generate different outputs for two distinct hypergraphs G 1 and
G 2 , but there exists M ∈ RelNN [ D, B + 1] that can generate different outputs for G 1 and G 2 , then we can construct a graph classiﬁcation function f that RelNN[ D , B + 1 ] can realize but RelNN[ D ,
B ] cannot.
The full proof is described in Appendix B.4.
It is also important to quantify the minimum arity for realizing certain graph reasoning functions.
Theorem 3.2 (FOL realization bounds) .
Let FOC B denote a fragment of ﬁrst order logic with at most B variables, extended with counting quantiﬁers of the form ∃ ≥ n φ , which state that there are at least n nodes satisfying formula φ (Cai et al., 1992).
• (Upper Bound)
Any function f in FOC B can be realized by RelNN[ D , B ] for some D .
• (Lower Bound)
There exists a function f ∈ FOC B such that for all D , f cannot be realized by RelNN[ D , B −
Proof: The upper bound part of the claim has been proved by Barcel ´ o et al.
(2020) for B = 2 .
The results generalize easily to arbitrary B because the counting quantiﬁers can be realized by sum aggregation.
The lower bound part can be proved by applying Section 5 of Cai et al.
(1992), in which they show that FOC B is equivalent to a ( B − 1)
-dimensional WL test in distinguishing non-isomorphic graphs.
Given that RelNN[ D , B − 1 ] is equivalent to the ( B − 2) -dimensional
WL test of graph isomorphism, there must be an FOL B formula that distinguishes two non-isomorphic graphs that RelNN[ D , B − 1 ] cannot.
Hence, FOL B cannot be realized by RelNN[ · , B − 1 ].
3. D EPTH H IERARCHY
We now study the dependence of the expressiveness of RelNNs on depth D .
RelNNs are generally deﬁned to have a ﬁxed depth, but allowing them to have a depth that is dependent on the number of nodes n = | V | in the graph can substantially increase their expressive power.
In the following, we deﬁne a depth hierarchy by analogy to the time hierarchy in computational complexity theory (Hartmanis & Stearns, 1965), and we extend our notation to let RelNN [ O ( f ( n )) , B ] denote the class of adaptive-depth RelNNs in which the growth-rate of depth D is bounded by O ( f ( n )) .
Conjecture 3.3 (Depth hierarchy) .
For any maximum arity B , for any two functions f and g , if g ( n ) =
o ( f ( n ) / log n ) , that is, f grows logarithmically more quickly than g , then ﬁxed-precision
RelNN [ O ( f ( n )) , B ] is more expressive than ﬁxed-precision RelNN [ O ( g ( n )) , B ] .
There is a closely related result for the congested clique model in distributed computing, where
Korhonen & Suomela (2018) proved that
CLIQUE ( g ( n )) (cid:40) CLIQUE( f ( n )) if g ( n ) =
o ( f ( n )) .
This result does not have the log n gap because the congested clique model allows log n bits to transmit between nodes at each iteration, while ﬁxed-precision RelNN allows only a constant number of bits.
The reason why the result on congested clique can not be applied to ﬁxed-precision RelNNs is that congested clique assumes unbounded precision representation for each individual node.
However, Conjecture 3.3 is not true for RelNNs with unbounded precision, because there is an upper bound depth O ( n B − 1 ) for a model’s expressiveness power.
That is, an unbounded-precision RelNN can not achieve stronger expressiveness by increasing its depth beyond O ( n B − 1 ) .
Theorem 3.4 (Upper depth bound for unbounded-precision RelNN) .
For any maximum arity B ,
RelNN[ O ( n B − 1 ) , B ] (cid:60) RelNN
[ O ( f ( n )) , B ] for any function f ( n ) .
Proof sketch.
Theorem 3.4 can be proved by combining the fact that the B -dimensional WL test is equivalent to differentiating graphs using FOC B +1 (Cai et al., 1992), and the O ( n B ) upper bound on the computation depth for B -dimensional WL-test (Kiefer & Schweitzer, 2016).
This upper bound indicates that O ( n B ) depth is sufﬁcient for B -dimensional WL-test to reach its maximum discriminative power, and thus any two graphs distinguishable by the WL-test can be distinguished by some FOL B +1 formula with quantiﬁer depth O ( n B ) .
The full proof is described in Appendix B.5.
It is important to point out that, to realize a speciﬁc graph reasoning function, RelNNs with different maximum arity B may require different depth D . F¨urer (2001) provides a general construction for problems that higher-dimensional RelNNs can solve in asymptotically smaller depth than lowerdimensional RelNNs.
In the following we give a concrete example for computing S-T Connectivity- k , which asks whether there is a path of from nodes S and T in a graph, with length ≤ k .
Theorem 3.5 (S-T Connectivity- k with Different Max Arity) .
For any function f ( k ) , if f ( k )
=
o ( k ) ,
RelNN[ O ( f ( k )) , 2 ] cannot realize S-T Connectivity- k .
That is, S-T Connectivity-
k requires depth at least O ( k ) for a relational neural network with an maximum arity of B = 2 .
However, S-T
Connectivity- k can be realized by RelNN
[ O (log k ) , 3 ].
Proof sketch.
For any integer k , we can construct a graph with two chains of length k , so that if we mark two of the four ends as S or T , any RelNN[ k − 1 , 2 ] cannot tell whether S and T are on the same chain.
The full proof is described in Appendix A.7.
There are many important graph reasoning tasks that do not have known depth lower bounds, including all-pair connectivity and shortest distance (Karchmer & Wigderson, 1990; Pai & Pemmaraju, 2019).
L EARNING AND G ENERALIZATION IN R ELATIONAL N EURAL N ETWORKS
Given our understanding of what functions can be realized by relational neural networks, we move on to the problems of learning them: Can we effectively learn a relational neural network to solve a desired task given a sufﬁcient number of input-output examples?
What is the sample complexity?
How do they generalize?
These questions are very difﬁcult to address in full for RelNNs, but there still some interesting things to be said.
In this section we state and prove two theorems that illuminate their learning and generalization properties.
In a ﬁxed-precision setting, we can show that applying enumerative training with examples up to some ﬁxed size can ensure that the trained neural network will generalize to all graphs larger than those appearing in the training set.
In an unbounded-precision setting, we can show that, if we use gradient descent to train models that use certain aggregation functions from randomly sampled data, then with a number of training examples that scales with log( N ) , the learned RelNN will generalize to all hypergraphs of size smaller than N .
4. G ENERALIZATION OF F IXED -P RECISION R ELATIONAL N EURAL N ETWORKS
Machine learning studies of generalization focus on cases where input and output space are ﬁxeddimensional, and on how well the learned hypothesis can predict outputs for inputs that were not observed in the training set.
Although learning speciﬁc graph reasoning functions in RelNNs are susceptible to this same type of analysis, we focus on an important type of generalization ability that is not present in the classical models: the ability to train on example graphs with up to N nodes, and then generalize to make correct predictions on graphs with larger numbers of nodes.
A critical determinant of the generalization ability for RelNNs is the aggregation function they use.
Speciﬁcally, Xu et al.
(2019) have shown that using sum as the aggregation function provides maximum expressiveness for graph neural networks.
However, sum aggregation cannot be implemented in ﬁxed-precision models with an arbitrary number of nodes, because as the graph size n increases, the range of the sum aggregation also increases.
Deﬁnition 4.1 (Fixed-precision aggregation function) .
An aggregation function is ﬁxed precision if it maps from any ﬁnite set of inputs with values drawn from ﬁnite domains to a ﬁxed ﬁnite set of possible output values; that is, the cardinality of the range of the function cannot grow with the number of elements in the input set.
Two useful ﬁxed-precision aggregation functions are max , which computes the dimension-wise maximum over the set of input values, and ﬁxed-precision mean , which approximates the dimensionwise mean to a ﬁxed decimal place.
Deﬁnition 4.2 (Fixed-precision RelNN) .
A RelNN is ﬁxed precision if its aggregation function is ﬁxed precision and all of the component MLPs, NN j , are also ﬁxed precision, in the same sense as for aggregation functions above.
This guarantees that the cardinality of the range of the hypergraph representation functions cannot grow with the number of nodes in the hypergraph.
In order to focus on structural generalization in this section, we consider an enumerative training paradigm.
When the input hypergraph representation domain S is a ﬁnite set, we can enumerate the set G ≤ N of all possible input hypergraph representations of size bounded by N .
We ﬁrst enumerate all graph sizes n ≤ N ; for each n , we enumerate all possible values assigned to the hyperedges in the input.
Given training size N , we enumerate all inputs in G ≤ N , associate with each one the corresponding ground-truth output representation, and train the model with these input-output pairs.
This has much stronger data requirements than the standard sampling-based training mechanisms in machine learning.
In the simplest binary graph classiﬁcation case, enumerative training allows us to identify the optimal model in the hypothesis space.
In practice, this can be approximated well when the input domain S is small and the input data distribution is approximately uniformly distributed.
The enumerative learning setting is studied by the language identiﬁcation in the limit community (Gold, 1967), in which it is called complete presentation .
This is an interesting learning setting because even if the domain for each individual hyperedge representation is ﬁnite, as the graph size can go arbitrarily large, the number of possible inputs is enumerable but unbounded.
Theorem 4.1 (Fixed-precision generalization under complete presentation) .
For any hypergraph reasoning function f , if it can be realized by a ﬁxed-precision relational neural network model M , then there exists an integer N , such that if we train the model with complete presentation on all input hypergraph representations with size smaller than N , G ≤ N , then for all M ∈ M ,
[Equation]
That is, as long as M ﬁts all training examples, it will generalize to all possible hypergraphs in G ∞ .
Proof.
The key observation is that for any ﬁxed vector representation length W , there are only ﬁnite number of distinctive models in a ﬁxed-precision RelNN family, independent of the graph size n .
Let W b be the number of bits in each intermediate representation of a ﬁxed-precision RelNN.
There are at most (2 W b )
Wb different mappings from inputs to outputs.
Hence, if N is sufﬁciently large to enumerate all input hypergraphs, we can always identify the correct model in the hypothesis space.
In future, we hope to extend this result to encompass sampling-based rather than enumerative training.
In addition, models approximating the ﬁnite-precision assumption with sigmoid activation and max aggregation have empirically demonstrated the strong generalization properties this theorem predicts.
4. G ENERALIZATION OF U NBOUNDED -P RECISION R ELATIONAL N EURAL N ETWORKS
Intuitively, relational neural networks with unbounded precision may not generalize to larger graphs because an aggregation function such as sum may accumulate through the layers.
However, interestingly, this error can remain reasonably well-bounded for max or mean aggregation function.
In this section, we will derive a bound for sample complexity in a more standard PAC-learning setting, in which there are no precision bounds on the inputs and computation of the model.
It is difﬁcult to characterize the VC dimension of an entire RelNN, but following Xu et al.
(2021), we characterize the properties of a sequential training paradigm that considers the components independently.
Theorem 4.2 (PAC sample complexity of unbounded-precision RelNNs) .
Let M be a family of unbounded-precision relational neural networks with hidden dimension W , depth D , and maximum arity B .
Let f be a graph function that can be realized by M , and M ∈ M be a sequentially-trained model.
If the following two conditions are true:
• (a) There exists an instance M (cid:48)
∈ M such that M (cid:48) realizes f and MLPs in M are ( (cid:15), δ ) - approximations of those in M (cid:48) * ; and • (b) The MLPs NN j in M are Lipschitz-continuous i.e. (cid:107) NN j ( x 1 ) − NN j ( x 2 ) (cid:107) ≤ λ (cid:107)
x 1 − x 2 (cid:107) for some constant λ and for all j .
Then on graphs of size n , M is a (cid:0) O (cid:0) (cid:15)λ D W D/ 2 (cid:1) , O ( δDn B ) (cid:1) -approximation of f if M uses max aggregation, and a (cid:0) O (cid:0) (cid:15)λ D (cid:1) , O ( δDn B (cid:1) -approximation of f if M uses mean aggregation.
Theorem 4.2 indicates that when using max or mean aggregation, the generalization error does not scale with the graph size n (which is the case for sum ), while the failure probability δ scales with
O ( Dn B ) .
In the PAC-learning framework, the sample complexity for achieving error (cid:15) with failure probability δ is bounded by O (cid:0) 1 (cid:15) (cid:0) log ( V C ( H ))
+ log (cid:0) 1 δ (cid:1)(cid:1)(cid:1) , where V C ( H ) is the VC-dimension
[Table]
of the hypothesis space.
Hence, scaling to graphs with size n requires additional O ( B log n ) samples when considering W, D, λ as constants.
The proof of Theorem 4.2 can be found in Appendix B.7.
The sample complexity does not depend on the graph size, because under the sequential learning setting the MLPs are trained with direct supervision.
Though this setting can not be implemented in practice, in our experiments we observe good generalization of models trained end-to-end.
Theorem 4.2 is interesting because it shows that using max or mean aggregation has provably better sample complexity than using sum aggregation.
Combined with the fact that using max aggregation,
RelNN[ · , B ] can realize FOC B with only ∃ and ∀ quantiﬁers (Dong et al., 2019), RelNN with max aggregation is a strong model class.
E XPERIMENTS
We now study how our theoretical results on model expressiveness and learning apply to relational neural networks trained with gradient descent on practically meaningful problems.
We begin by describing two synthetic benchmarks: graph substructure detection and relational reasoning.
In the graph substructure detection dataset, there are several tasks of predicting whether there input graph containd a sub-graph with speciﬁc structure.
The tasks are: 3-link (length-3 path), 4-link , triangle , and 4-clique .
These are important graph properties with many potential applications.
The relational reasoning dataset is composed of two family-relationship prediction tasks and two connectivity-prediction tasks.
They are all binary edge classiﬁcation tasks.
In the family-relationship prediction task, the input contains the mother and father relationships, and the task is to predict the grandparent and uncle relationships between all pairs of entities.
In the connectivity-prediction tasks, the input is the edges in an undirected graph and the task is to predict, for all pairs of nodes, whether they are connected with a path of length ≤ 4 ( connectivity-4 ) and whether they are connected with a path of arbitrary length ( connectivity ).
The data generation for all datasets is included in Appendix C.
5. R ESULTS
Our main results on all datasets are shown in Table 2 and Table 3.
We empirically compare relational neural networks with different maximum arity B , different model architecture (GNN and NLM), and different aggregation functions (max and sum).
All models use sigmoidal activation for all MLPs.
For each task on both datasets we train on a set of small graphs ( n = 10 ) and test the trained model on both small graphs and large graphs ( n = 10 and n = 30 ).
We summarize the ﬁndings below.
Expressiveness.
We have seen a theoretical equal expressiveness between GNNs and NLMs applied to hypergraphs.
That is, a GNN applied to B -ary hyperedges is equivalent to a ( B + 1) -ary NLM.
Table 2 and 3 further suggest their similar performance on tasks when trained with gradient descent.
Formally, triangle detection requires RelNNs with at least B = 3 to solve.
Thus, we see that all
RelNNs with arity B = 2 fail on this task, but models with B = 3 perform well.
Formally, 4-clique is realizable by RelNNs with maximum arity B = 4 , but we failed to reliably train models to reach perfect accuracy on this problem.
It is not yet clear what the cause of this behavior is.
[Table]
Structural generalization.
We discussed the structural generalization properties of RelNNs in
Section 4.1, in a learning setting based on ﬁxed-precision networks and enumerative training.
This setting can be approximated by training RelNNs with max aggregation and sigmoidal activation on sufﬁcient data.
In both Table 2 and Table 3 we see that, when models can realize the desired function and are trained to 100% accuracy on the training split, they also generalize well to larger graphs.
In appendix C
[Table]
we provide a more detailed study on the generalization performance
[Table]
Sample complexity (case study).
In Section 4.2, we discussed the sampling complexity differences induced by different aggregation functions in the sequential learning setting.
The primary conclusion is that models with max aggregation are expected to have better sample complexity than models with sum aggregation.
We test this hypothesis in the end-to-end learning setting.
For the triangle detection problem, we train 3D-NLM with sum and max aggregation with fewer training samples.
We observe that sum aggregation requires about 300 training samples to reach nearly perfect accuracy, while max aggregation requires only 30 training samples.
More details are included in appendix C.
R ELATED W ORK
Solving problems on graphs of arbitrary size is studied in many ﬁelds.
RelNNs can be viewed as circuit families with constrained architecture.
In distributed computation, the congested clique model can be viewed as 2-arity RelNNs, where nodes have identities as extra information.
Common graph problems including sub-structure detection(Li et al., 2017; Rossman, 2010) and connectivity(Karchmer &
Wigderson, 1990) are studied for lower bounds in terms of depth, width and communication.
This has been connected to GNNs for deriving expressiveness bounds (Loukas, 2020).
Studies have been conducted on the expressiveness of GNNs and their variants.
Xu et al.
(2019) provide an illuminating characterization of GNN expressiveness in terms of the WL graph isomorphism test.
Barcel´o et al.
(2020) reviewed GNNs from the logical perspective and rigorously reﬁned their logical expressiveness with respect to fragments of ﬁrst-order logic.
Dong et al.
(2019) proposed
Neural Logical Machines (NLMs) to reason about higher-order relations, and showed that increasing order inreases expressiveness.
It is also possible to gain expressiveness using unbounded computation time, as shown by the work of Dehghani et al.
(2019) on dynamic halting in transformers.
It is interesting that GNNs may generalize to larger graphs.
Xu et al.
(2020; 2021) have studied the notion of algorithmic alignment to quantify such structural generalization.
Dong et al.
(2019) provided empirical results showing that NLMs generalize to much larger graphs on certain tasks.
C ONCLUSION
We have provided a framework for unifying several powerful classes of relational neural networks and characterized aspects of their representational and generalization power, both theoretically and empirically.
In particular, we have shown the substantial increase of expressive power due to higherarity relations and increasing depth, and have characterized very powerful structural generalization from training on small graphs to performance on larger ones.
Although many questions remain open about the overall generalization capacity of these models in continuous and noisy domains, we believe this work has shed some light on their utility and potential for application in a variety of problems.
Reproducibility statement.
The proof details for theorems in this paper have been included in the appendix.
A G RAPH P ROBLEMS
We listed a number of examples for graph classiﬁcation and regression tasks, and we provide the deﬁnitions and the current best known RelNNs for learning these tasks from data.
For some of the problems, we will also show why they can not be solved by a simpler problems, or indicate them as open problems.
A. N ODE C OLOR M AJORITY
Each node is assigned a color c ∈ C where C is a ﬁnite set of all colors.
The model needs to predict which color the most nodes have.
Using a single layer with sum aggregation, the model can count the number of nodes of color c for each c ∈ C on its global representation.
A. C OUNT R ED N ODES
Each node is assigned a color of red or blue.
The model needs to count the number of red nodes.
Similarly, using a single layer with sum aggregation, the model can count the number of red nodes on its global representation.
A. 3-L INK D ETECTION
Given an unweighted, undirected graph, the model needs to detect whether there is a triple of nodes ( a, b, c ) such that a (cid:54) =
c and ( a, b ) and ( b, c ) are edges.
This is equivalent to check whether there exists a node with degree at least 2 .
We can use a Reduction operation with sum aggregation to compute the degree for each node, and then use a Reduction operation with max aggregation to check whether the maximum degree of nodes is greater than or equal to 2 .
Note that this can not be done with 1 layer, because the edge information is necessary for the problem, and they require at least 2 layers to be passed to the global representation.
A. 4-L INK D ETECTION
Given an unweighted undirected graph, the model needs to detect whether there is a 4-tuple of nodes ( a, b, c, d ) such that a (cid:54) =
c, b (cid:54) =
d and ( a, b ) , ( b, c ) , ( c, d ) are edges (note that a triangle is also a 4-link).
This problem is equivalent to check whether there is an edge between two nodes with degrees ≥ 2 .
We can ﬁrst reduce the edge information to compute the degree for each node, and then expand it back to 2-dimensional representations, so we can check for each edge if the degrees of its ends are ≥ 2 .
Then the results are reduced to the global representation with existential quantiﬁer (realized by max aggregation) in 2 layers.
A. T RIANGLE D ETECTION
Given a unweighted undirected graph, the model is asked to determine whether there is a triangle in the graph i.e. a tuple ( a, b, c ) so that ( a, b ) , ( b, c ) , ( c, a ) are all edges.
This problem can be solved by RelNN [ 4 , 3 ]: we ﬁrst expand the edge to 3-dimensional representations, and determine for each 3-tuple if they form a triangle.
The results of 3-tuples require 3 layers to be passed to the global representation.
We can prove that Triangle Detection indeed requires breadth at least 3.
Let k -regular graphs be graphs where each node has degree k .
Consider two k -regular graphs both with n nodes, so that exactly one of them contains a triangle ‡ .
However, RelNNs of breadth 2 has been proven not to be stronger than WL test on distinguish graphs, and thus can not distinguish these two graphs (WL test can not distinguish any two k -regular graphs with equal size).
A. 4-C LIQUE D ETECTION AND C OUNTING
Given an undirected graph, check existence of, or count the number of tuples ( a, b, c, d ) so that there are edges between every pair of nodes in the tuple.
This problem can be easily solved by a RelNN with breadth 4 that ﬁrst expand the edge information to the 4-dimensional representations, and for each tuple determine whether its is a 4-clique.
Then the information of all 4-tuples are reduced 4 times to the global representation (sum aggregation can be used for counting those).
Though we did not ﬁnd explicit counter-example construction on detecting 4-cliques with RelNNs of breadth 3, we suggest that this problem can not be solved with RelNNs with 3 or lower breadth.
A. C ONNECTIVITY
The connectivity problems are deﬁned on unweighted undirected graphs.
S-T connectivity problems provides two nodes S and T (labeled with speciﬁc colors), and the model needs to predict if they are connected by some edges.
All pair connectivity problem require the model to answer for every pair of nodes.
Connectivity- k problems have an additional requirement that the distance between the pair of nodes can not exceed k .
S-T connectivity- k can be solved by a RelNN of breadth 2 with k iterations.
Assume S is colored with color c , at every iteration, every node with color c will spread the color to its neighbors.
Then, after k iterations, it is sufﬁcient to check whether T has the color c .
With RelNNs of breadth 3, we can use O (log k ) matrix multiplications to solve connectivity- k between every pair of nodes.
Since the matrix multiplication can naturally be realized by RelNNs of breadth 3 with two layers.
All-pair connectivity problems can all be solved with O (log k ) layers.
Theorem A.1 (S-T connectivity-
k with RelNN[ o ( k ) , 2) .
S-T connectivity- k can not be solved by a
RelNN of maximum arity within o ( k ) iterations.
Proof.
We construct two graphs each has 2 k nodes u 1 , · · · , u k , v 1 , · · · , v k .
In both graph, there are edges ( u i , u i +1 ) and ( v i , v i +1 ) for 1 ≤ i ≤ k − 1 i.e. there are two links of length k .
We then set
S = u 1 , T = u n and S = u 1 , T = v n the the two graphs.
We will analysis GNNs as RelNNs are proved to be equivalent to them by scaling the depth by a constant factor.
Now consider the node reﬁnement process where each node x is reﬁned by the multiset of labels of x ’s neighbots and the multiiset of labels of x ’s non-neighbors.
Let C ( i ) j
( x ) be the label of x in graph j after i iterations, at the beginning, WLOG, we have
[Equation] [Equation]
and all other nodes are labeled as 0 .
Then we can prove by induction: after i ≤ k 2 − 1 iterations, for 1 ≤ t ≤ i +
we have
[Equation]
and for i + 2 ≤ t ≤ k − i −
we have
[Equation]
This is true because before k 2 iterations are run, the multiset of all node labels are identical for the two graphs (say S ( i ) ).
Hence each node x is actually reﬁned by its neighbors and S ( i ) where S ( i ) is the same for all nodes.
Hence, before running k 2 iterations when the message between S and T ﬁnally meets in the ﬁrst graph, GNN can not distinguish the two graphs, and thus can not solve the connectivity with distance k − 1 .
A. M AX D EGREE
The max degree problem gives a graph and ask the model to output the maximum degree of its nodes.
Like we mentioned in 3-link detection, one layer for computing the degree for each node, and another layer for taking the max operation over nodes should be sufﬁcient.
A. M AX F LOW
The Max Flow problem gives a directional graph with capacities on edges, and indicate two nodes S and T .
The models is then asked to compute the amount of max-ﬂow from S to T .
Notice that the Breadth First Search (BFS) component in Dinic’s algorithm(Dinic, 1970) can be implemented on RelNNs as they does not require node identities (all new-visited nodes can augment to their non-visited neighbors in parallel).
Since the BFS runs for O ( n ) iteration, and the Dinic’s algorithm runs BFS O ( n 2 ) times, the max-ﬂow can be solved by RelNNs with in O ( n 3 ) iterations.
A. D ISTANCE
Given a graph with weighted edges, compute the length of the shortest between speciﬁed node pair (S-T Distance) or all node pairs (All-pair Distance).
Similar to Connectivity problems, but Distance problems now additionally record the minimum distance from S (for S-T) or between every node pairs (for All-pair), which can be updated using min operator (using Min-plus matrix multiplication for All-pair case).
B P ROOFS AND A NALYSIS B. C ONSTANT -L AYER R EL NN S AND R ECURRENT R EL NN S
Theorem B.1.
A neural network with representation width W that has D different layers
NN 1 , · · · , NN D can be realized by a neural network that applies a single layer NN (cid:48) for D iterations with width ( D + 1)( W + 1) .
Proof.
The representation for NN (cid:48) can be partitioned into D + 1 segments each of length W + 1 .
Each segment consist of a “ﬂag” element and a W -element representation, which are all 0 initially, except for the ﬁrst segment, where the ﬂag is set to 1 , and the representation is the input.
NN (cid:48) has the weights for all NN 1 , · · · , NN D , where weights NN i are used to compute the representation in segment i + 1 from the representation in segment i .
Additionally, at each iteration, segment i + 1 can only be computed if the ﬂag in segment i is 1 , in which case the ﬂag of segment
i +is set to 1 .
Clearly, after D iterations, the output of NN k should be the representation in segment
D + 1 .
Due to Theorem B.1, we consider the neural networks that recurrently apply the same layer because a) they are as expressive as those using layers of different weights, b) it is easier to analyze a single neural network layer than D layers, and c) they naturally generalize to neural networks that runs for adaptive number of iterations (e.g. GNNs that run O (log n ) iterations where n is the size of the input graph).
B. E XPRESSIVENESS E QUIVALENCE OF R ELATIONAL N EURAL N ETWORKS
We ﬁrst describe a framework for quantifying if two RelNN models are equally expressive on regression tasks.
The framework view the expressiveness from the perspective of computation.
Speciﬁcally, we will prove the expressiveness equivalence between models by showing that their computation can be aligned.
In complexity, we usually show a problem is at least as hard as the other one by showing a reduction from the other problem to the problem.
Similarly, on the expressiveness of RelNNs, we can construct reduction from model family A to model family B to show that B can realize all computation that A does, or even more.
Formally, we have the following deﬁnition.
Deﬁnition B.1 (Expressiveness reduction) .
For two model families A and B , we say A can be reduced to B if and only if there is a function r : A → B such that for each model instance A ∈ A , r ( A ) ∈ B and A have the same outputs on all inputs.
In this case, we say B is at least as expressive as A .
Deﬁnition B.2 (Expressiveness equivalence) .
For two model families A and B , if A and B can be reduced to each other, then A and B are equally expressive.
Note that this deﬁnition of expressiveness equivalence generalizes to both classiﬁcation and regression tasks.
Graph Neural Networks.
GNNs (Scarselli et al., 2008) is deﬁned based on two message passing operations.
• Edge update: the feature of each edge is updated by features of its ends.
• Note update: the feature of each node is updated by features of all edges adjacent to it
This message passing scheme can easily ﬁt in RelNN architecture by storing edge features as tuples of two nodes.
In this case, updating edges by its ends can be realized by the expand and the permute operation.
The message aggregation at each node can be realized by a reduce operation.
Hence,
GNNs can be realized by a RelNN of maximum arity B = 2 .
High-dimensional GNNs.
Computing only node-wise and edge-wise features does not handle higher-order relations, such as triangles in the graph.
In order to obtain more expressive power,
GNNs have be extend to hypergraphs of hier arity (Morris et al., 2019).
Speciﬁcally, GNNs on B -ary hypergraph maintains features for all B -tuple of nodes, and the neighborhood is extended to B -tuples accordingly: the feature of tuple ( v 1 , v 2 , · · · , v B ) is updated by the | V | element multiset (contain | V | elements for each u ∈ V ) of B -tuples of features
[Equation]
where H i − 1 [ v ] is the feature of tuple v from the previous iteration.
We now introduce the formal deﬁnition of the high-dimensional message passing.
We denote v as a B -tuple of nodes ( v 1 , v 2 , · · · , v B ) , and generalize the neighborhood to a higher dimension by deﬁning the neighborhood of v as all node tuples that differ from v at one position.
[Equation]
Then message passing scheme naturally generalizes to high-dimensional features using the highdimensional neighborhood.
Received i
[Equation]
=
[Equation]
Recall that the normal GNNs can be realize by RelNN[ · , 2 ].
Similarly, since features of B -tuples can be stored in RelNNs with arity B and the high-dimensional message passing involves at most
B + 1 nodes at the same time, ( v 1 , · · · , v B and u ), we can extend the previous result to that GNNs on B -arity hypergraphs can be realized by RelNNs with maximum arity B + 1 .
Equivalence between hypergraph GNNs and RelNNs.
We will prove the equivalence between hypergraph GNNs and RelNNs by making reductions in both directions.
Lemma B.2.
A GNN on B -ary hypergraphs and depth D can be realized by a RelNN with maximum arity B + 1 and depth 2 D .
Proof.
We prove lemma B.2 by showing that one layer of GNNs on B -ary hypergraphs can be realized by two RelNN with maximum arity B + 1 .
Firstly, a GNN layer maintain features of B -tuples, which are stored in correspondingly in an RelNN layer at dimension B .
Then we will realize the message passing scheme using the RelNN features of dimension B and B + 1 in two steps.
Recall the message passing scheme generalized to high dimensions (to distinguish, we use H for
GNN features and T for RelNN features.)
Received i ( v ) =
(cid:88) u (cid:0) NN 1 (cid:0)
[Table]
[Equation]
B.5)
At the ﬁrst step, the Expand operation ﬁrst raise the dimension to B + 1 by expanding a non-related variable u to the end, and the Permute operation can then swap u with each of the elements (or no swap).
Particularly, T i,B [ v 1 , v 2 , · · · , v B ] will be expand to
T i +1 ,B +
[ u, v 2 , v 3 , · · · , v B , v 1 ] , T i +1 , B +1 [ v 1 , u, v 3 , · · · , v B , v 2 ] , · · · ,
T i +1 ,B +
[ v 1 , v 2 , · · · , v B − 1 , u, v B ] ,and T i +1 ,B +1 [ v 1 , v 2 , · · · , v B − 1 , v B , u ]
Hence, T i +1 ,B +1 [ v 1 , v 2 , v 3 , · · · , v B , u ] receives the features from
T i,B [ v 1 , v 2 , · · · , v B ] , T i,B [ u, v 2 , v 3 , · · · , v B ] , T i,B [ v 1 , u, v 3 , · · · , v B ] , · · · , T i,B [ v 1 , v 2 , · · · , v B − 1 , u ]
These features matches the input of NN 1 in equation B.5, and in this layer NN 1 can be applied to compute things inside the summation.
Then at the second step, the last element is reduced to get what tuple v should receive, so v can be updated.
Since each GNN layer can be realized by such two RelNN layers, each GNN on B -ary hypergraphs with depth D can be realized by a RelNN of maximum arity ( B + 1) and depth 2 D .
To complete the proof we need to ﬁnd a reduction from RelNNs of maximum arity B + 1 to GNNs on B -ary hypergraphs.
The key observation here is that the features of ( B + 1) -tuples in RelNNs can only be expanded from sub-tuples, and the expansion and reduction involving ( B + 1) -tuples can be simulated by the message passing process.
Lemma B.3.
The features of ( B + 1) -tuples feature T
i,B +1 [ v 1 , v 2 , · · · , v B +1 ] can be computed from the following tuples
[Equation]
Proof.
Lemma B.3 is true because ( B + 1) -dimensional representations can either be computed from themselves at the previous iteration, or expanded from B -dimensional representations.
Since representations at all previous iterations j < i can be contained in T i,B , it is sufﬁcient to compute
T i,B +1 [ v 1 , v 2 , · · · , v B +1 ] from all its B -ary sub-tuples.
Then let’s construct the GNN for given RelNN to show the existence of the reduction.
Lemma B.4.
A RelNN of maximum arity B + 1 and depth D can be realized by a GNN on B -ary hypergraphs with no more than D iterations.
Proof.
We can realize the Expand and Reduce operation with only the B -dimensional features using the broadcast message passing scheme.
Note that Expand and Reduce between B -dimensional features and ( B + 1) -dimensional features in the RelNN is a special case where claim B.3 is applied.
Let’s start with Expand and Reduce operations between features of dimension B or lower.
For the b -dimensional feature in the RelNN, we keep n b n B − b § copies of it and store them the representation of every B -tuple who has a sub-tuple ¶ that is a permutation of the b -tuple.
That is, for each B -tuple in the GNN on B -ary hypergraphs, for its every sub-tuple of length b , we store b !
representations corresponding to every permutation of the b -tuple in the RelNN.
Keeping representation for all sub-tuple permutations make it possible to realize the Permute operation.
Also, it is easy to notice that Expand operation is realized already, as all features with dimension lower than B are naturally expanded to B dimension by ﬁlling in all possible combinations of the rest elements.
Finally, the
Reduce operation can be realized using a broadcast casting message passing on certain position of the tuple.
Now let’s move to the special case – the Expand and Reduce operation between features of dimensions
B and B + 1 .
Claim B.3 suggests how the ( B + 1) -dimensional features are stored in B -dimensional representations in GNNs, and we now show how the Reduce can be realized by message passing.
[Equation]
We ﬁrst bring in claim B.3 to the GNN message passing, where we have Received i [ v ] to be
[Equation]
Note that the last term T i − 1 ,B [ v 1 , v 2 , · · · , v B ] is contained in H i − 1 ( v ) in equation B.5, and other terms are contained in H i − 1 ( v (cid:48) ) for v (cid:48) ∈ neighbors( v , u ) .
Hence, equation B.5 is sufﬁcient to simulate the Reduce operation.
Theorem B.5.
GNNs on B -ary hypergraphs are equally expressive as RelNNs with maximum arity
B + 1 .
Proof.
This is a direct conclusion by combining Lemma B.2 and Lemma B.4.
B. E XPRESSIVENESS OF HYPERGRAPH CONVOLUTION AND ATTENTION
Hypergraph convolution(Feng et al., 2019; Yadati et al., 2019; Bai et al., 2021), attention(Ding et al., 2020) and message passing(Huang & Yang, 2021) focus on updating node features through hyperedges instead of hyperedges.
These approaches can be viewed as instances of RelNNs, and they have smaller time complexity because they do not model all high-arity tuples.
However, they are less expressive than RelNNs with equal max arity.
These approaches can be formulated to two steps at each iteration.
At the ﬁrst step, each hyperedge is updated by the features of nodes it connects.
[Equation]
At the second step, each node is updated by the features of hyperedges connecting it.
[Equation]
where f i,v is the feature of node v at iteration i , and h i,v is the aggregated message passing through hyperedge e at iteration i + 1 .
It is not hard to see that B.6 can be realized by B iterations of RelNN layers with Expand operations where B is the max arity of hyperedges.
This can be done by expanding each node feature to every high arity features that contain the node, and aggregate them at the tuple corresponding to each hyperedge.
Then, B.7 can also be realized by B iterations of RelNN layers with Reduce operations, as the tuple feature will ﬁnally be reduced to a single node contained in the tuple.
This approach has lower complexity compared to the GNNs we study applied on hyperedges, because it only requires communication between nodes and hyperedges connecting to them, which takes
O ( | V | · | E | ) time at each iteration.
Compared to them, RelNNs takes O ( | V | B ) time because RelNNs keep features of every tuple with max arity B , and allow communication from tuples to tuples instead of between tuples and single nodes.
An example is provided below that this approach can not solve while RelNNs can.
Consider a graph with 6 nodes and 6 edges forming two triangles (1 , 2 , 3) and (4 , 5 , 6) .
Because of the symmetry, the representation of each node should be identical throughout hypergraph message passing rounds.
Hence, it is impossible for these models to conclude that (1 , 2 , 3) is a triangle but (4 , 2 , 3) is not, based only on the node representations, because they are identical.
In contrast, RelNNs with max arity 3 can solve them (as standard triangle detection problem in Table 1).
B. P ROOF OF T HEOREM 3.1: A RITY H IERARCHY .
Morris et al.
(2019) have connected high-dimensional GNNs with high-dimensional WL tests.
Speciﬁcally, they showed that the GNNs on B -ary hypergraphs are equally expressive as B -dimensional WL test on graph isomorphism test problem.
In Theorem B.5 we proved that GNNs on B -ary hypergraphs are equivalent to RelNN of maximum arity B + 1 in terms of expressiveness.
Hence, RelNN of maximum arity B + 1 can distinguish if two non-isomorphic graphs if and only if B -dimensional
WL test can distinguish them.
However, Cai et al.
(1992) provided an construction that can generate a pair of non-isomorphic graphs for every B , which can not be distinguished by ( B − 1) -dimensional
WL test but can be distinguished by B -dimensional WL test.
Let G 1 B and G 2 B be such a pair of graph.
Since RelNN of maximum arity B + 1 is equally expressive as GNNs on B -ary hypergraphs, there must be such a RelNN that classify G 1 B and G 2 B into different label.
However, such RelNN can not be realized by any RelNN of maximum arity B because they are proven to have identical outputs on
G 1 B and G 2 B .
In the other direction, RelNNs of maximum arity B + 1 can directly realize RelNNs of maximum arity B , which completes the proof.
B. P ROOF OF T HEOREM 3.4: U PPER D EPTH B OUND FOR U NBOUNDED -P RECISION R EL NN.
The idea for proving an upper bound on depth is to connect RelNNs to WL-test, and use the O ( n B ) upper bound on number of iterations for B -dimensional test (Kiefer & Schweitzer, 2016), and FOC formula is the key connection.
For any ﬁxed n , B -dimensional WL test divide all graphs of size n , G = n , into a set of equivalence classes {C 1 , C 2 , · · · , C m } , where two graphs belong to the same class if they can not be distinguished by the WL test.
We have shown that RelNNs of maximum arity ( B + 1) must have the same input for all graphs in the same equivalence class.
Thus, any RelNN of maximum arity B + 1 can be view as a labeling over C 1 , · · · , C m .
Stated by Cai et al.
, B -dimensional WL test are as powerful as FOC B +1 in differentiating graphs graphs.
Combined with the O ( n B ) upper bound of WL test iterations, for each C i , there must be an FOC B +1 formula of quantiﬁer depth O ( n B ) that exactly recognize C i over G = n .
Finally, with unbounded precision, for any f ( n ) , RelNN of maximum arity B + 1 and depth f ( n ) can compute all FOC B +1 formulas with quantiﬁer depth f ( n ) .
Note that there are ﬁnite number of such formula because the supscript of counting quantiﬁers is bounded by n .
For any graph in some class C i , the class can be determined by evaluating these FOC formulas, and then the label is determined.
Therefore, any RelNN of maximum arity B + 1 can be realized by a
RelNN of maximum arity B + 1 and depth O ( n B ) .
B. T HE T IME ANS S PACE C OMPLEXITY OF R EL NN S
Handling high-arity features and using deeper models usually increase the computational cost in terms of time and space.
As an instance that use the architecture of RelNN, NLMs with depth D and max arity B takes O ( Dn B ) time when applying to graphs with size n .
This is because both
Expand and Reduce operation have linear time complexity with respect to the input size (which is
O ( n B ) at each iteration).
If we need to record the computational history (which is typically the case when training the network using back propagation), the space complexity is the same as the time complexity.
GNNs applied to ( B − 1) -ary hyperedges and depth D are equally expressive as RelNNs with depth
O ( D ) and max arity B .
Though up to ( B − 1) -ary features are kept in their architecture, the broadcast message passing scheme scale up the complexity by a factor of O ( n ) , so they also have
time and space complexity
O ( Dn B ) .
Here the length of feature tensors W is treated as a constant.
B. P ROOF OF T HEOREM 4.2: PAC B OUNDS FOR R EL NN S
We ﬁrst formally deﬁne the ( (cid:15), δ ) approximation on functions mapping real number factors.
Deﬁnition B.3.
We say M 1 is an ( (cid:15), δ ) -approximation of M 2 : R W
→ R W if and only if with probability at least (1 − δ ) , || M 1 ( x ) − M 2 ( x ) || <
(cid:15) .
The proof builds on bounding the error between the representations computed by M and M (cid:48) .
Assuming the Lipschitz continuity with constant λ , we will show four facts that are sufﬁcient to bound the error between M and M (cid:48) , and thus claim the approximation of M on f .
• (a) Feed-forward networks increase the error by at most a factor of λ .
• (b) The min-max aggregation increase the error by at most a factor √ W .
• (c)
The mean aggregation keeps the error bounded.
• (d) The probability to avoid any failures is lower bounded by 1 − O ( Dn B δ ) .
Fact (a) is guaranteed under the deﬁnition of Lipschitz continuity.
For fact (b), we denote t be the error bound on the representations before aggregation, and t also bounds the error at each position.
In the worst case, min-max aggregation will get the error t at all positions, which implies a new error bound √ Wt on the aggregated representation on the 2-norm.
Fact (c) is guaranteed by the convexity of 2-norm.
Fact (d) can be derived by a simple union bound on the failure probability on all internal representations (they are O ( Dn B ) representations to compute when applying the layer for D iterations).
Note that at each iteration we will run all feed-forward network in parallel for all representations, and aggregate some n -tuple of representations.
These operations increase the error bound by a factor of at most λ √ W for min-max aggregation and λ for mean aggregation.
Sample complexity for RelNNs with gradient decent.
Arora et al.
(2019) have derived a more ﬁne-grained analysis on the of two-layer neural networks optimized by gradient decent, which is later extended by Xu et al.
under the sequential learning setting i.e. we use a correct network instance M (cid:48) to generate the ground-truth outputs to supervise all feed-forward networks at each layer.
Theorem B.6 (Xu et al.
(2020), sample complexity for over-parameterized MLP modules) .
Let M be an over-parameterized and random initialized two-layer MLP trained with gradient decent for a sufﬁcient number of iterations.
Suppose g : R W → R W with components g ( x ) ( i ) =
(cid:80)
j α ( i ) j ( β ( i ) j (cid:62) x ) p ( i ) j , where β ( i ) j ∈ R W , α ∈ R and p ( i ) j
= 1 or p ( i )
j = 2 l ( l ∈ N + ).
The sample complexity C M ( g, (cid:15), δ ) is
C M (
[Equation]
We combine Theorem B.6 with our theorem 4.2 to derive the sample complexity in order to get ( (cid:15), δ ) -approximation.
We can compute (cid:15) NN
= O ( (cid:15) λ D W D/ 2 ) and δ NN = O ( δDn B ) for feed-forward network in the RelNN, and bring them as to Theorem B.6 to get the following equation
[Equation]
where g : R W → R W has components g ( x ) ( i )
= (cid:80)
j α ( i ) j ( β ( i ) j (cid:62) x ) p ( i ) j , and B, D, W are the dimension, number of iterations, width for specifying the model family M .
The sample complexity does not have a requirement of the sample graph size, because under the sequential learning setting the feed-forward networks are trained under supervision on each node and tuple at each layer.
Though this setting can not be implemented in practice, in our experiments we observe good generalization of models trained on small graphs.
C E XPERIMENTS C. E XPERIMENT S ETUP
For all problems, we have 800 training samples, 100 validation samples, and 300 test samples for each different n we are testing the models on.
We then provide the details on how we synthesize the data.
For most of the problems, we generate the graph by randomly selecting from all potential edges i.e. the Erd˝os–R´enyi model.
We sample the number of edges around n, 2 n, n log n and n 2 / 2 .
For all problems, with 50% probability the graph will ﬁrst be divided into 2 , 3 , 4 or 5 parts with equal number of components, where we use the ﬁrst generated component to ﬁll the edges for rest of the components.
Some random edges are added afterwards.
This make the data contain more isomorphic sub-graphs, which we found challenging empirically.
Substructure Detection.
To generate a graph that does not contain a certain substructure, we randomly add edges when reaching a maximal graph not containing the substructure or reaching the edge limit.
For generating a graph that does contain the certain substructure, we ﬁrst generate one that does not contain, and then randomly replace present edges with missing edges until we detect the substructure in the graph.
This aim to change the label from “No” to “Yes” while minimizing the change to the overall graph properties, and we found that data generated using edge replacing is much more difﬁcult for neural networks compared to random generated graphs from scratch.
Family Tree.
We generate the family trees using the algorithm modiﬁed from Dong et al.
We add people to the family one by one.
When a person is added, with probability p we will try to ﬁnd a single woman and a single man, get them married and let the new children be their child, and otherwise the new person is introduced as a non-related person.
Every new person is marked as single and set the gender with a coin ﬂip.
We adjust p based on the ratio of single population: p = 0 .
7 when more than 40% of the population are single, and p = 0 .
3 when less than 20% of the population are single, and p = 0 .
5 otherwise.
Connectivity.
For connectivity problems, we use the similar generation method as the substructure detection.
We sample the query pairs so that the labels are balanced.
C. M ODEL I MPLEMENTATION D ETAILS
For all models, we use a hidden dimension 128 except for 3-dimensional HD-GNN and 4-dimensional
NLM where we use hidden dimension 64 .
All model have 4 layers that each has its own parameters, except for connectivity where we use the recurrent models that apply the second layer k times, where k is sampled from integers in [2 log n, 3 log n ] .
The depths are proven to be sufﬁcient for solving these problems (unless the model itself can not solve).
All models are trained for 100 epochs using adam optimizer with learning rate 3 × 10 − 4 decaying at epoch 50 and 80.
We have varied the depth, the hidden dimension, and the activation function of different models.
We select sufﬁcient hidden dimension and depth for every model and problem (i.e., we stop when increasing depth or hidden dimension doesn’t increase the accuracy).
We tried linear, ReLU, and
Sigmoid activation functions, and ReLU performed the best overall combinations of models and tasks.
C. A DDITIONAL A BLATION S TUDIES
Case study on structure generalize.
We run a case study on the problem connectivity-4 about how the generalization performance changes when the test graph size gradually becomes larger.
Figurehow how these models generalize to gradually larger graphs with size increasing from10to80.
From the curves we can see that only models with sufﬁcient expressiveness can get 100% accuracy on the same size graphs, and among them the models using max aggregation generalize to larger graphs with no performance drop.
2-ary GNN and 3-ary NLM that use max aggregation have sufﬁcient expressiveness and better generalization property.
They achieve 100% accuracy on the original graph size and generalize perfectly to larger graphs.
Case study on sample complexity.
In order to better visualize how using max and sum aggregation functions affect the sample complexity.
We run an ablation study on the triangle detection problem where we try different numbers of training samples from 10 to 800 .
We test 3-ary NLM with max and sum aggregation which both reach 100% accuracy on the same graph size (we do not test generalization here since max aggregation is shown to be more generalizable than sum aggregation).
Figure 3 indicates that the NLM model with max aggregation require much fewer training samples ( ∼ 30 ) to reach the perfect accuracy compare to sum aggregation ( ∼ 300 .)