D ON ’ T STACK LAYERS IN GRAPH NEURAL NETWORKS , WIRE THEM RANDOMLY
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph neural networks have become a staple in problems addressing learning and analysis of data deﬁned over graphs.
However, several results suggest an inherent difﬁculty in extracting better performance by increasing the number of layers.
Besides the classic vanishing gradient issues, recent works attribute this to a phenomenon peculiar to the extraction of node features in graph-based tasks, i.e., the need to consider multiple neighborhood sizes at the same time and adaptively tune them.
In this paper, we investigate the recently proposed randomly wired architectures in the context of graph neural networks.
Instead of building deeper networks by stacking many layers, we prove that employing a randomly-wired architecture can be a more effective way to increase the capacity of the network and obtain richer representations.
We show that such architectures behave like an ensemble of paths, which are able to merge contributions from receptive ﬁelds of varied size.
Moreover, these receptive ﬁelds can also be modulated to be wider or narrower through the trainable weights over the paths.
We also provide extensive experimental evidence of the superior performance of randomly wired architectures over three tasks and ﬁve graph convolution deﬁnitions, using a recent benchmarking framework that addresses the reliability of previous testing methodologies.
I NTRODUCTION
Data deﬁned over the nodes of graphs are ubiquitous.
Social network proﬁles (Hamilton et al., 2017), molecular interactions (Duvenaud et al., 2015), citation networks (Sen et al., 2008), 3D point clouds (Simonovsky & Komodakis, 2017) are just examples of a wide variety of data types where describing the domain as a graph allows to encode constraints and patterns among the data points.
Exploiting the graph structure is crucial in order to extract powerful representations of the data.
However, this is not a trivial task and only recently graph neural networks (GNNs) have started showing promising approaches to the problem.
GNNs (Wu et al., 2020) extend the deep learning toolbox to deal with the irregularity of the graph domain.
Much of the work has been focused on deﬁning a graph convolution operation (Bronstein et al., 2017), i.e., a layer that is well-deﬁned over the graph domain but also retains some of the key properties of convolution such as weight reuse and locality.
A wide variety of such graph convolution operators has been deﬁned over the years, mostly based on neighborhood aggregation schemes where the features of a node are transformed by processing the features of its neighbors.
Such schemes have been shown to be as powerful as the Weisfeiler-Lehman graph isomorphism test (Weisfeiler & Lehman, 1968; Xu et al., 2019), enabling them to simultaneuosly learn data features and graph topology.
However, contrary to classic literature on CNNs, few works (Li et al., 2019a; Dehmamy et al., 2019;
Xu et al., 2018; Dwivedi et al., 2020) addressed GNNs architectures and their role in extracting powerful representations.
Several works, starting with the early GCN (Kipf & Welling, 2017), noticed an inability to build deep GNNs, often resulting in worse performance than that of methods that disregard the graph domain, when trying to build anything but very shallow networks.
This calls for exploring whether advances on CNN architectures can be translated to the GNN space, while understanding the potentially different needs of graph representation learning.
Li et al.
(2019b) suggest that GCNs suffer from oversmoothing as several layers are stacked, resulting in the extraction of mostly low-frequency features.
This is related to the lack of self-loop information in this speciﬁc graph convolution.
It is suggested that ResNet-like architectures mitigate the problem as the skip connections supply high frequency contributions.
Xu et al.
(2018) point out that the size of the receptive ﬁeld of a node, i.e., which nodes contribute to the features of the node under consideration, plays a crucial role, but it can vary widely depending on the graph and too large receptive ﬁelds may actually harm performance.
They conclude that for graph-based problems it would be optimal to learn how to adaptively merge contributions from receptive ﬁelds of multiple size.
For this reason they propose an architecture where each layer has a skip connection to the output so that contributions at multiple depths (hence sizes of receptive ﬁelds) can be merged.
Nonetheless, the problem of ﬁnding methods for effectively increasing the capacity of graph neural networks is still standing, since stacking many layers has been proven to provide limited improvements (Li et al., 2019b; Oono & Suzuki, 2019; Alon & Yahav, 2020; NT & Maehara, 2019).
In this paper, we argue that the recently proposed randomly wired architectures (Xie et al., 2019) are ideal for GNNs.
In a randomly wired architecture, “layers” are arranged according to a random directed acyclic graph and data are propagated through the paths towards the output.
Such architecture is ideal for GNNs because it realizes the intuition of Xu et al.
(2018) of being able of merging receptive ﬁelds of varied size.
Indeed, the randomly wired network can be seen as an extreme generalization of their jumping network approach where layer outputs can not only jump to the network output but to other layers as well, continuously merging receptive ﬁelds.
Hence, randomly wired architectures provide a way of effectively scaling up GNNs, mitigating the depth problem and creating richer representations.
Fig.
1 shows a graphical representation of this concept by highlighting the six layers directly contributing to the output, having different receptive ﬁelds induced by the distribution of paths from the input.
Our novel contributions can be summarized as follows: i) we are the ﬁrst to analyze randomly wired architectures and show that they are generalizations of ResNets when looked at as ensembles of paths (Veit et al., 2016); ii) we show that path ensembling allows to merge receptive ﬁelds of varied size and that it can do so adaptively , i.e., trainable weights on the architecture edges can tune the desired size of the receptive ﬁelds to be merged to achieve an optimal conﬁguration for the problem; iii) we introduce improvements to the basic design of randomly wired architectures by optionally embedding a path that sequentially goes through all layers in order to promote larger receptive ﬁelds when needed, and by presenting MonteCarlo DropPath, which decorrelates path contributions by randomly dropping architecture edges;
iv) we provide extensive experimental evidence, using a recently introduced benchmarking framework (Dwivedi et al., 2020) to ensure signiﬁcance and reproducibility, that randomly wired architectures consistently outperform ResNets, often by large margins, for ﬁve of the most popular graph convolution deﬁnitions on three different tasks.
B ACKGROUND 2. G RAPH N EURAL N ETWORKS
A major shortcoming of CNNs is that they are unable to process data deﬁned on irregular domains.
In particular, one case that is drawing attention is when the data structure can be described by a graph and the data are deﬁned as vectors on the graph nodes.
This setting can be found in many applications, including 3D point clouds (Wang et al., 2019; Valsesia et al., 2019), computational biology (Alipanahi et al., 2015; Duvenaud et al., 2015), and social networks (Kipf & Welling, 2017).
However, extending
CNNs from data with a regular structure, such as images and video, to graph-structured data is not straightforward if one wants to preserve useful properties such as locality and weight reuse.
GNNs redeﬁne the convolution operation so that the new layer deﬁnition can be used on domains described by graphs.
The most widely adopted graph convolutions in the literature rely on message passing, where a weighted aggregation of the feature vectors in a neighborhood is computed.
The GCN (Kipf & Welling, 2017) is arguably the simplest deﬁnition, applying the same linear transformation to all the node features, followed by neighborhood aggregation and non-linear activation:
[Equation]
Variants of this deﬁnition have been developed, e.g., GraphSage
(Hamilton et al., 2017) concatenates the feature vector of node i to the feature vectors of its neighbors, so that self-information can also be exploited; GIN (Xu et al., 2019) uses a multilayer perceptron instead of a linear transform, replaces average with sum to ensure injectivity and proposes a different way of computing the output by using all the feature vectors produced by the intermediate layers.
These deﬁnitions are all isotropic because they treat every edge in the same way.
It has been observed that better representation capacity can be achieved using anistropic deﬁnitions, where every edge can have a different transformation, at the cost of increased computational complexity.
The Gated GCN (Bresson & Laurent, 2017) and GAT (Veli ˇ ckovi ´
c et al., 2017)
deﬁnitions fall in this category.
2. R ANDOMLY WIRED ARCHITECTURES
In recent work, Xie et al.
(2019) explore whether it is possible to avoid handcrafted design of neural network architectures and, at the same time, avoid expensive neural architecture search methods (Elsken et al., 2019), by designing random architecture generators.
They show that “layers” performing convolution, normalization and non-linear activation can be connected in a random architecture graph.
Strong performance is observed on the traditional image classiﬁcation task by outperforming state-of-the-art architectures.
The authors conjecture that random architectures generalize ResNets and similar constructions, but the underlying principles of their excellent performance are unclear, as well as whether the performance translates to tasks other than image recognition or to operations other than convolution on grids.
R ANDOMLY WIRED GNN S
In this section, we ﬁrst introduce randomly wired architectures and the notation we are going to use.
We then analyze their behavior when viewed as ensembles of paths.
∑
RELU
GCONV
BN
[Figure]
(w ia )
[Figure] [Figure]
[Figure]
(w ic ) h
(i) h (a) h (
[Figure]
) h (c)
A randomly wired architecture consists of a directed acyclic graph (DAG) connecting a source architecture node, which is fed with the input data, to a sink architecture node.
One should not confuse the architecture DAG with the graph representing the GNN domain: to avoid any source of confusion we will use the terms architecture nodes (edges) and domain nodes (edges), respectively.
A domain node is a node of the graph that is fed as input to the GNN.
An architecture node is effectively a GNN layer performing the following operations (Fig. 2): i) aggregation of the inputs from other architecture nodes via a weighted sum as in (Xie et al., 2019):
[Equation]
being σ a sigmoid function, A i the set of direct predecessors of the architecture node i , and w ij a scalar trainable weight; ii) a non-linear activation; iii) a graph-convolution operation (without output activation); iv) batch normalization.
The architecture DAG is generated using a random graph generator.
In this paper, we will focus on the Erd˝os-Renyi model where the adjacency matrix of the DAG is a strictly upper triangular matrix with entries being realizations of a Bernoulli random variable with probability p .
If multiple input architecture nodes are randomly generated, they are all wired to a single global input.
Multiple output architecture nodes are averaged to obtain a global output.
Other random generators may be used, e.g., small-world and scale-free random networks have been studied in (Xie et al., 2019).
However, a different generator will display a different behavior concerning the properties we study in Sec. 3.1.
3. R ANDOMLY WIRED ARCHITECTURES BEHAVE LIKE PATH ENSEMBLES
It has already been shown that ResNets behave like ensembles of relatively shallow networks, where one can see the ResNet architecture as a collection of paths of varied lengths (Veit et al., 2016).
More speciﬁcally, in a ResNet with n layers, where all layers have a skip connection except the ﬁrst one and the last one, there are exactly 2 L − 2 paths, whose lengths follow a Binomial distribution (i.e., the number of paths of length l from layer k to the last layer is (cid:0) L − k − 1 l − 2 (cid:1) ), and the average path length is
L 2 + 1 (Veit et al., 2016).
In this section, we show that a randomly wired neural network can also be considered as an ensemble of networks with varied depth.
However, in this case, the distribution of the path length is different from the one obtained with the ResNet, as shown in the following lemma (proof in the supplementary material).
Lemma 3.1.
Let us consider a randomly wired network with L architecture nodes, where the architecture DAG is generated according to a Erd˝os-Renyi graph generator with probability p .
The average number of paths of length l from node k to the sink, where k < L , is E [ N ( k ) l ] = (cid:0) L − k − 1 l − 2 (cid:1) p l −and the average total number of paths from node k to the sink is E [ N ( k ) ]
= p (1 + p ) L − k − 1 .
We can observe that if p = 1 , the randomly wired network converges to the ResNet architecture.
This allows to think of randomly wired architectures as generalizations of ResNets as they enable increased ﬂexibility in the number and distribution of paths instead of enforcing the use of all 2 L − 2 .
3. R ECEPTIVE FIELD ANALYSIS
In the case of GNNs, we deﬁne the receptive ﬁeld of a domain node as the neighborhood that affects the output features of that node.
As discussed in Sec. 1, the work in (Xu et al., 2018) highlights that one of the possible causes of the depth problem in GNNs is that the size of the receptive ﬁeld is not adaptive and may rapidly become excessively large.
Inspired by this observation, in this section we analyze the receptive ﬁeld of a randomly wired neural network.
We show that the receptive ﬁeld of the output is a combination of the receptive ﬁelds of shallower networks, induced by each of the paths.
This allows to effectively merge the contributions from receptive ﬁelds of varied size.
Moreover, we show that the trainable parameters along the path edges modulate the contributions of various path lengths and enable adaptive receptive ﬁelds, that can be tuned by the training procedure.
We ﬁrst introduce a deﬁnition of the receptive ﬁeld of a feedforward graph neural network 1 .
Deﬁnition 3.1.
Given a feedforward graph neural network with L layers, the receptive ﬁeld of radius
L of a domain node is its L -hop neighborhood.
In a randomly wired architecture, each path induces a corresponding receptive ﬁeld whose radius depends on the length of the path.
Then, the receptive ﬁeld at the output of the network is obtained by combining the receptive ﬁelds of all the paths.
In order to analyze the contribution of paths of different lengths to the receptive ﬁeld of the network, we introduce the concept of distribution of the receptive ﬁeld radius of the paths.
Notice that if we consider a feedforward network with L layers, the distribution of the receptive ﬁeld radius is a delta centered in L .
The following lemma allows to analyze the distribution of the receptive ﬁeld radius in a randomly wired architecture.
[Figure] [Figure]
Lemma 3.2.
The derivative ∂y∂x 0 of the output y of a randomly wired architecture with respect to the input x 0 is
[Equation]
where y p is the output of path p , ¯ y p is the output of path p when we consider all the aggregation weights equal to 1, λ p =
∂y p ∂x 0 / ∂ ¯ y p ∂x 0 , P is the set of all paths from source to sink, L is the number of architecture nodes, P l is the set of paths from source to sink of length l and E p is the set of edges of the path p .
Proof.
Direct computation.
From Lemma 3.2, we can observe that the contribution of each path to the gradient is weighted by its corresponding architecture edge weights.
Thus, we can deﬁne the following distribution ρ of the receptive ﬁeld radius:
[Equation]
where we have assumed that the gradient ∂ ¯ y p ∂x 0 depends only on the path length, as done in (Veit et al., 2016).
This is a reasonable assumption if all the architecture nodes perform the same operation.
The distribution of the receptive ﬁeld radius is therefore inﬂuenced by the architecture edge weights.
Figure 3 shows an example of how such weights can modify the radius distribution.
If we consider ω ij = 1 for all i and j , we obtain that the radius distribution is equal to the path length distribution.
In order to provide some insight into the role of parameter p in the distribution of the receptive ﬁeld radius, we focus on this special case and analyze the distribution of the path lengths in a randomly wired architecture by introducing the following Lemma (proof in the supplementary material).
Lemma 3.3.
Let us consider a randomly wired network with L architecture nodes, where the architecture DAG is generated according to a Erd˝os-Renyi graph generator with probability p .
The average length of the paths from node k to the sink is E [ l ( k ) ]
≈ p 1+ p
( L − k − 1)
+ 2 .
Therefore, if p = 1 and ω ij = 1 for all i and j the radius distribution is a Binomial distribution centered in L 2 + 1 (as in ResNets), instead when p < 1 the mean of the distribution is lower.
The path length distribution for different p values is shown in Fig.
This shows that, differently from feedforward networks, the receptive ﬁeld of ResNets and randomly wired architectures is a combination of receptive ﬁelds of varied sizes, where most of the contribution is given by shallow paths, i.e. smaller receptive ﬁelds.
The parameter p of the randomly wired neural network inﬂuences the distribution of the receptive ﬁeld radius: a lower p value skews the distribution towards shallower paths, instead a higher p value skews the distribution towards longer paths.
After having considered the special case where ω ij = 1 for all i and j , we now focus on the general case.
Since the edge architecture weights are trainable parameters, they can be adapted to optimize the distribution of the receptive ﬁeld radius.
This is one of the strongest advantages provided by randomly wired architectures with respect to ResNets.
This is particularly relevant in the context of
GNNs, where we may have a non-uniform growth of the receptive ﬁeld caused by the irregularity of the graph structure (Xu et al., 2018).
Notice that the randomly wired architecture can be seen as a generalization of the jumping knowledge networks proposed in (Xu et al., 2018), where all the architecture nodes, not only the last one, merge contributions from previous nodes.
We also remark that, even if we modify the ResNet architecture by adding trainable weights to each branch of the residual module, we cannot retrieve the behaviour of the randomly wired architecture.
In fact, the latter has intrinsically more granularity than a ResNet: the expected number of architecture edge weights of a randomly wired network is pL ( L +1) , instead a weighted ResNet has only 2( L − 2) weights.
Ideally, we would like to weight each path independently (i.e., directly optimizing the value of λ p in Eq.
However, this is unfeasible because the number of parameters would become excessively high and the randomly wired architecture provides an effective tradeoff.
Given an architecture node, weighting in a different way each input edge is important because to each edge corresponds a different length distribution of the paths going through such edge, as shown by the following Lemma (proof in supplementary material).
Lemma 3.4.
Let us consider a randomly wired network with n architecture nodes, where the architecture DAG is generated according to a Erd˝os-Renyi graph generator with probability p .
Given an edge { i, j } between the architecture nodes i and j where i < j , the average length of the paths from the source to the sink going through that edge is E [ l ij ] ≈ p 1+ p
( L − ( j − i ) − 3)
+ 4 .
3. S EQUENTIAL PATH
In the previous sections we have shown that a randomly wired architecture behaves like an ensemble of paths merging contribution from receptive ﬁelds of varied size, where most of the contribution is provided by shallow paths.
As discussed previously, this provides numerous advantages with respect to feedforward networks and ResNets.
However, some graph-based tasks may actually beneﬁt from a larger receptive ﬁeld (Li et al., 2019b), so it is interesting to provide randomly wired architectures with mechanisms to directly promote longer paths.
Differently from ResNets, in a randomly wired neural network with L architecture nodes the longest path may be shorter than L , leading to a smaller receptive ﬁeld.
In order to overcome this issue, we propose to modify the generation process of the random architecture by imposing that it should also include the sequential path, i.e., the path traversing all architecture nodes.
This design of the architecture skews the initial path length distribution towards longer paths, which has the effect of promoting their usage.
Nevertheless, the trainable architecture edge weights will ultimately deﬁne the importance of such contribution.
Fig.
3 shows an example of how including the sequential path changes the distribution of the receptive ﬁeld radius.
3. M ONTE C ARLO D ROP P ATH REGULARIZATION
The randomly wired architecture offers new degrees of freedom to introduce regularization techniques.
In particular, one could delete a few architecture edges during training with probability p drop as a way to avoid co-adaptation of architecture nodes.
This is reminiscent of DropOut (Srivastava et al., 2014) and DropConnect (Wan et al., 2013), although it is carried out at a higher level of abstraction, i.e., connections between “layers” instead of neurons.
It is also reminiscent of techniques used in Neural
Architecture Search (Zoph et al., 2018) and the approach used in ImageNet experiments in (Xie et al., 2019), although implementation details are unclear for the latter.
We propose to use a MonteCarlo approach where paths are also dropped in testing.
Inference is performed multiple times for different realizations of dropped architecture edges and results are averaged.
This allows to sample from the full predictive distribution induced by DropPath, as in
MonteCarlo DropOut (Gal & Ghahramani, 2015).
It is worth noting that MonteCarlo DropPath decorrelates the contributions of paths in Eq.
(3.2) even if they share architecture edges (proof in supplementary material), thus allowing ﬁner control over the modulation of the receptive ﬁeld radius.
E XPERIMENTAL RESULTS
Experimental evaluation of GNNs is a topic that has recently received great attention.
The emerging consensus is that benchmarking methods routinely used in past literature are inadequate and lack reproducibility.
In particular, Vignac et al.
(2020) showed that commonly used citation network datasets like CORA, CITESEER, PUBMED are too simple and skew results towards simpler architectures or even promote ignoring the underlying graph.
TU datasets are also recognized to be too small (Errica et al., 2019) and the high variability across splits does not allow for sound comparisons across methods.
In order to evaluate the gains offered by randomly wired architectures across a wide variety of graph convolutions and tasks, we adopt a recently proposed GNN benchmarking framework (Dwivedi et al., 2020), that has introduced new datasets and allows for reproducible experiments.
[Table] [Table]
We focus on testing ﬁve of the most commonly used graph convolution deﬁnitions: GCN (Kipf & Welling, 2017), GIN (Xu et al., 2019) 2 , Gated GCN (Bresson & Laurent, 2017), GraphSage (Hamilton et al., 2017), GAT
(Veli ˇ ckovi ´ c et al., 2017).
We select three representative tasks introduced by (Dwivedi et al., 2020): graph regression on the ZINC dataset, node classiﬁcation on the CLUSTER dataset, and graph classiﬁcation with superpixels on CIFAR10.
To ensure reproducibility, we use exactly the same setting as (Dwivedi et al., 2020).
We are interested in the performance differences between the baseline ResNet architecture, i.e., a feedforward architecture with skip connections after every layer, and the randomly wired architecture.
It was already shown in (Dwivedi et al., 2020) that
ResNet GNNs signiﬁcantly outperform architectures without residual connections.
We remark that other works proposed methods to build deeper GNN (Rong et al., 2019; Zhao & Akoglu, 2019;
Gong et al., 2020), but such techniques can be regarded as complementary to our work.
We do not attempt to optimize a speciﬁc method, nor we are interested in comparing one graph convolution to another.
A fair comparison is ensured by running both methods with the same number of trainable parameters and with the same hyperparameters.
In particular, the learning rate of both methods is adaptively decayed between 10 − 3 and 10 − 5 by halving according to the value of the validation loss, with a patience of 5 epochs.
Stopping criterion is validation loss not improving for 5 epochs after reaching the minimum learning rate.
We average the results of all experiments over 4 runs with different weight initializations and different random architecture graphs, drawn with p = 0 .
We also evaluate results for multiple values of the total number of layers (architecture nodes)
L , in order to show that randomly wired GNNs allow a more effective increase in capacity.
The random architectures use sequential paths (Sec. 3.2) in the ZINC experiment, sequential paths and DropPath in the CLUSTER experiment, and only DropPath in CIFAR10 3 .
The reason for these choices is that the regression task in ZINC and the node classiﬁcation task in CLUSTER are particularly sensitive to the size of the receptive ﬁeld, as observed by analyzing the experimental receptive radius (supplementary material).
On the other hand, CIFAR10 is bottlenecked by overﬁtting, and it greatly beneﬁts from the regularizing effect of DropPath, as also observed on CLUSTER.
The number of DropPath iterations in testing was ﬁxed to 16.
4. R ANDOM GNN BENCHMARKING
The results presented in this section show that randomly wired GNNs have compelling performance in many regards.
First of all, they typically provide higher accuracy or lower error than their ResNet counterparts for the same number of parameters.
Moreover, they are more effective at increasing capacity than stacking layers: while they are essentially equivalent to ResNets for very short networks, they enable larger gains when additional layers are introduced.
Table 1 shows the results obtained on the ZINC dataset.
The metric is mean absolute error (MAE), so lower is better.
The superscript reports the standard deviation among runs and the subscript reports the level of signiﬁcance by measuring how many baseline standard deviations the average value of the
[Table]
[Table] [Table] . param. [Table]
[Table]
random architecture deviates from the average value of the baseline.
Results are in bold if they are at least 1 σ signiﬁcant.
The results show that the randomly wired GNNs typically outperform the ResNet baseline by signiﬁcant margins.
Table 2 reports the node classiﬁcation accuracy on the CLUSTER dataset.
It can be seen that the random architectures achieve very signiﬁcant improvements on this dataset, especially for RAN-GCN, RAN-GIN and RAN-GAT.
Table 3 reports the classiﬁcation accuracy on CIFAR10 when the images are converted to graphs using superpixels.
Also in this case, the randomly wired architecture greatly outperforms the baseline, in some cases achieving gains higher than 5 σ .
Finally, Table 4 shows the relative performance gain (relative improvement in accuracy or mean absolute error), averaged over all the graph convolution deﬁnitions, with respect to a short 4-layer network, where random wiring and ResNets are almost equivalent (results in supplementary material).
We can notice that deeper ResNets always provide lower gains with respect to their shallow counterpart than the randomly wired GNNs.
Moreover, we observe monotonically increasing gains for random GNNs while deeper ResNets are either unable to signiﬁcantly extract more performance beyond L = 16 or even provide worse results than the L = 4 network.
This supports our claim that the randomly wired architecture is a superior way to increase GNN capacity.
Finally, we compare the proposed method against two other frameworks for GNNs, namely PPNP
Klicpera et al.
and SIGN Rossi et al.
(2020), which propose different approaches for solving the oversmoothing problem. Due to the signiﬁcant differences in the approaches, providing a fair comparison is challenging. We decided to equalize the number of parameters across the methods, since notions as number of layers or features cannot be translated in all the frameworks (180k,360k,720k parameters correspond to the L = 8 , 16 , 32 settings in the previous tables). Table ??
shows the obtained results.
We can observe that both PPNP on node classiﬁcation and SIGN on all tasks outperform the standard GCN architecture without skip connections, but they cannot outperform
GCN with residual connections and the randomly wired GCN.
[Table]
[Table]
[Table]
[Table]
4. A BLATION STUDY 4.2. E DGE PROBABILITY
We ﬁrst investigate the impact of the probability p of drawing an edge in the random architecture.
Table 6 shows the results for a basic random architecture without DropPath nor embedded sequential path.
It appears that an optimal value of p exists that maximizes performance.
This could be explained by a tradeoff between size of receptive ﬁeld and the ability to modulate it.
4.2. D ROP P ATH
The impact of DropPath on CIFAR10 is shown in Table 7.
We found the improvement due to
DropPath to be increasingly signiﬁcant for a higher number of architecture nodes, as expected due to the increased number of edges.
The value of the drop probability p drop = 0 .
01 was not extensively cross-validated.
However, Table 8 shows that higher drop rates lowered performance.
4.2. E MBEDDED SEQUENTIAL PATH
The impact of embedding a sequential path as explained in Sec. 3.2 is shown in Table 9.
It can be observed that its effect of promoting receptive ﬁelds with larger radius is useful on this task.
We remark that, while we do not report results due to space constraints, this is not always the case and some tasks (e.g., CIFAR10) do not beneﬁt from promoting larger receptive ﬁelds.
C ONCLUSIONS
We showed how randomly wired architectures can boost the performance of GNNs by merging receptive ﬁelds of multiple size.
Consistent and statistically signiﬁcant improvements over a wide range of tasks and graph convolutions suggest considering them as the go-to choice for new models.