D ON ’ T STACK LAYERS IN GRAPH NEURAL NETWORKS , WIRE THEM RANDOMLY
diego.valsesia@polito.it Giulia Fracastoro ∗
Politecnico di Torino Italy giulia.fracastoro@polito.it enrico.magli@polito.it
BSTRACT
Several results suggest an inherent difﬁculty of graph neural networks in extracting better performance by increasing the number of layers.
Recent works attribute this to a phenomenon peculiar to the extraction of node features in graph-based tasks, i.e., the need to consider multiple neighborhood sizes at the same time and adaptively tune them.
In this paper, we investigate the recently proposed randomly wired architectures in the context of graph neural networks.
Instead of building deeper networks by stacking many layers, we prove that employing a randomlywired architecture can be a more effective way to increase the capacity of the network and obtain richer representations.
We show that such architectures behave like an ensemble of paths, which are able to merge contributions from receptive ﬁelds of varied size.
Moreover, these receptive ﬁelds can also be modulated to be wider or narrower through the trainable weights over the paths.
I NTRODUCTION
Several works, starting with the early GCN (Kipf & Welling, 2017), noticed an inability to build deep graph neural networks (GNNs).
When trying to build anything but very shallow networks, GNNs often result in worse performance than that of methods that disregard the graph domain.
This calls for exploring whether advances on CNN architectures can be translated to the GNN space while understanding the potentially different needs of graph representation learning.
Li et al.
(2019) suggest that GCNs suffer from oversmoothing as several layers are stacked, resulting in the extraction of mostly low-frequency features.
This is related to the lack of self-loop information in this speciﬁc graph convolution.
It is suggested that ResNet-like architectures mitigate the problem as the skip connections supply high frequency contributions.
Xu et al.
(2018) point out that the size of the receptive ﬁeld of a node, i.e., which nodes contribute to the features of the node under consideration, plays a crucial role, but it can vary widely depending on the graph and too large receptive ﬁelds may actually harm performance.
They conclude that for graph-based problems it would be optimal to learn how to adaptively merge contributions from receptive ﬁelds of multiple size.
For this reason they propose an architecture where each layer has a skip connection to the output so that contributions at multiple depths (hence sizes of receptive ﬁelds) can be merged.
Nonetheless, the problem of ﬁnding methods for effectively increasing the capacity of graph neural networks is still standing, since stacking many layers has been proven to provide limited improvements (Li et al., 2019; Oono & Suzuki, 2019; Alon & Yahav, 2020; NT & Maehara, 2019).
In this paper, we argue that the recently proposed randomly wired architectures (Xie et al., 2019) are ideal for GNNs.
In a randomly wired architecture, “layers” are arranged according to a random directed acyclic graph and data are propagated through the paths towards the output.
Such architecture is ideal for GNNs because it realizes the intuition of Xu et al.
(2018) of being able of merging receptive ﬁelds of varied size.
Indeed, the randomly wired network can be seen as a generalization of their approach where layer outputs can not only jump to the network output but to other layers as well, continuously merging receptive ﬁelds.
Hence, randomly wired architectures provide a way of effectively scaling up GNNs, mitigating the depth problem and creating richer representations.
Fig.shows a graphical representation of this concept by highlighting the six layers directly contributing to the output, having different receptive ﬁelds induced by the distribution of paths from the input.
R ANDOMLY WIRED GNN S
In recent work, Xie et al.
(2019) explore whether it is possible to avoid handcrafted design of neural network architectures and, at the same time, avoid expensive neural architecture search methods (Elsken et al., 2019), by designing random architecture generators.
They show that “layers” performing convolution, normalization and non-linear activation can be connected in a random architecture graph.
Strong performance is observed on the traditional image classiﬁcation task by outperforming state-of-the-art architectures.
The authors conjecture that random architectures generalize ResNets and similar constructions, but the underlying principles of their excellent performance are unclear, as well as whether the performance translates to tasks other than image recognition or to operations other than convolution on grids. ∑ RELU GCONV
BN
[Figure]
ia )
[Figure]
w ib [Figure] 𝜎 (w ic ) h (i) h (a) h (b) h (c) A randomly wired architecture consists of a directed acyclic graph (DAG) connecting a source architecture node, which is fed with the input data, to a sink architecture node.
One should not confuse the architecture DAG with the graph representing the GNN domain: to avoid any source of confusion we will use the terms architecture nodes (edges) and domain nodes (edges), respectively.
A domain node is a node of the graph that is fed as input to the GNN.
An architecture node is effectively a GNN layer performing the following operations (Fig. 2): i) aggregation of the inputs from other architecture nodes via a weighted sum as in (Xie et al., 2019):
[Equation]
being σ a sigmoid function, h ( i ) the output feature of the i -th node, A i the set of direct predecessors of the architecture node i , and w ij a scalar trainable weight; ii) a non-linear activation; iii) a graphconvolution operation (without activation); iv) batch normalization.
The architecture DAG is generated using a random graph generator.
In this paper, we focus on the
Erd˝os-Renyi model.
If multiple input architecture nodes are randomly generated, they are all wired to a single global input.
Multiple output architecture nodes are averaged to obtain a global output.
2. R ANDOMLY WIRED ARCHITECTURES BEHAVE LIKE PATH ENSEMBLES
It has already been shown that ResNets behave like ensembles of relatively shallow networks, where one can see the ResNet architecture as a collection of paths of varied lengths (Veit et al., 2016).
More speciﬁcally, in a ResNet with n layers, where all layers have a skip connection except the ﬁrst one and the last one, there are exactly 2 L − 2 paths, whose lengths follow a Binomial distribution (i.e., the number of paths of length l from layer k to the last layer is (cid:0) L − k − 1 l − 2 (cid:1) ), and the average path length is
L 2 + 1 (Veit et al., 2016).
A randomly wired neural network can also be considered as an ensemble of networks with varied depth.
However, in this case, the distribution of the path length is different from the one obtained with the ResNet, as shown in the following lemma.
Lemma 2.1.
Let us consider a randomly wired network with L architecture nodes, where the architecture DAG is generated according to a Erd˝os-Renyi graph generator with probability p .
The average number of paths of length l from node k to the sink, where k < L , is E [ N ( k ) l ] = (cid:0) L − k − 1 l − 2 (cid:1) p l −and the average total number of paths from node k to the sink is E [ N ( k ) ]
= p (1 + p ) L − k − 1 .
We can observe that if p = 1 , the randomly wired network converges to the ResNet architecture.
This allows to think of randomly wired architectures as generalizations of ResNets as they enable increased ﬂexibility in the number and distribution of paths instead of enforcing the use of all 2 L − 2 .
By means of the parameter p and the weights ω ij of the architecture edges, we can modulate the contributions of various path lengths and the size of the output receptive ﬁeld, which is obtained as a combination of the receptive ﬁelds of shallower networks induced by each of the paths of the ensemble.
In addition, the trainable weights ω ij enable adaptive receptive ﬁelds, tunable by training.
2. S EQUENTIAL PATH
Differently from ResNets, in a randomly wired neural network with L architecture nodes the longest path may be shorter than L , leading to a smaller receptive ﬁeld.
In order to overcome this issue, we propose to modify the generation process of the random architecture by imposing that it should also include the sequential path, i.e., the path traversing all architecture nodes.
This design of the architecture skews the initial path length distribution towards longer paths, which has the effect of promoting their usage.
Nevertheless, the trainable architecture edge weights will ultimately deﬁne the importance of such contribution.
2. M ONTE C ARLO D ROP P ATH REGULARIZATION
The randomly wired architecture offers new degrees of freedom to introduce regularization techniques.
In particular, one could delete a few architecture edges during training with probability p drop as a way to avoid co-adaptation of architecture nodes.
This is reminiscent of DropOut (Srivastava et al., 2014) and DropConnect (Wan et al., 2013), although it is carried out at a higher level of abstraction, i.e., connections between “layers” instead of neurons.
We propose to use a MonteCarlo approach where paths are also dropped in testing.
Inference is performed multiple times for different realizations of dropped architecture edges and results are averaged.
This allows to sample from the full predictive distribution induced by DropPath, as in MonteCarlo DropOut (Gal & Ghahramani, 2015).
E XPERIMENTAL RESULTS
Experimental evaluation of GNNs is a topic that has recently received attention.
Vignac et al.
(2020) showed that commonly used citation network datasets like CORA, CITESEER, PUBMED are too simple and skew results towards simpler architectures or even promote ignoring the underlying graph.
TU datasets are also recognized to be too small (Errica et al., 2019) and the high variability across splits does not allow for sound comparisons.
To evaluate the gains offered by randomly wired architectures, we adopt recently proposed benchmarking frameworks such as (Dwivedi et al., 2020) and Open Graph Benchmarks (Weihua Hu, 2020).
We focus on the ZINC (graph regression) from (Dwivedi et al., 2020), and ogbg-molpcba (graph classiﬁcation) from (Weihua Hu, 2020) datasets.
First, we use the ZINC dataset to analyse the performance differences between the baseline ResNet architecture, i.e., a feedforward architecture with skip connections after every layer, and the randomly wired architecture.
For this experiment, we test four of the most commonly used graph convolution deﬁnitions: GCN (Kipf & Welling, 2017), GIN (Xu et al., 2019) 1 , Gated GCN (Bresson & Laurent, 2017), GraphSage (Hamilton et al., 2017).
Notice that we do not attempt to optimize a speciﬁc method, nor we are interested in comparing one graph convolution to another.
A fair comparison is ensured by running both methods with the same number of trainable parameters and with the same hyperparameters, keeping exactly the same ones used in (Dwivedi et al., 2020).
The learning rate of both methods is adaptively decayed between 10 − 3 and 10 − 5 and stopping criterion is validation loss not improving for 5 epochs after reaching the minimum learning rate. Results
[Table] [Table] [Table] runs with different weight initialization and different random architecture graphs, drawn with p = 0 .
We use sequential path embedding but no DropPath
[Table]
The results shown in Table 1 show that randomly wired GNNs have compelling performance in many regards.
The superscript reports the standard deviation among runs and the subscript reports the level of signiﬁcance by measuring how many baseline standard deviations the average value of the random architecture deviates from the average value of the baseline.
Results are in bold if they are at least 1 σ signiﬁcant.
First of all, randomly wired GNNs typically provide lower error than their ResNet counterparts for the same number of parameters.
Moreover, they are more effective at increasing capacity than stacking layers: while they are essentially equivalent to ResNets for very short networks, they enable larger gains when additional layers are introduced.
This is highlighted by
Table 2, which shows the relative improvement in mean absolute error averaged over all the graph convolution deﬁnitions, with respect to the short 4-layer network, where random wiring and ResNets are almost equivalent.
We can notice that deeper ResNets always provide lower gains with respect to their shallow counterpart than the randomly wired GNNs.
Finally, we compare the proposed method against other state-of-the-art techniques, including methods that address the oversmoothing problem to build deeper GNNs (DeeperGCN) (Li et al., 2020) or argue that going wide instead of deep is more effective to increase the capacity (SIGN)
(Rossi et al., 2020).
This experiment is done on the ogbg-molpcba dataset from Open Graph Benchmarks (Weihua Hu, 2020) and results are taken from the public leaderboard.
We use a randomly wired version of
GIN and compare results with two different setups: a vanilla RAN-GIN with the same number of parameters as GIN, and a larger RAN-GIN with the virtual node trick (Gilmer et al., 2017) and FLAG augmentations (Kong et al., 2020).
Both versions additionally use DropPath with p drop = 0 .
The results are reported in Table 3.
We can see that RAN-GIN (12 layers) signiﬁcantly outperforms GIN and a number of other techniques for a comparable number of parameters.
Furthermore, RAN-GIN with virtual node and FLAG augmentations reaches state-of-the-art performance on this benchmark, outperforming DeeperGCN and being equal to the recently proposed GINE (Brossard et al., 2020) 2 .
C ONCLUSION
Randomly wired architectures are a promising way to increase the capacity of graph neural networks, going beyond the traditional depth concept.
Premilinary experiments presented in this paper show convincing improved to a variety of graph convolutions and warrant further tests.