Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis
Abstract
Advanced deep neural networks (DNNs), designed by either human or AutoML algorithms, are growing increasingly complex.
Diverse operations are connected by complicated connectivity patterns, e.g., various types of skip connections.
Those topological compositions are empirically effective and observed to smooth the loss landscape and facilitate the gradient flow in general.
However, it remains elusive to derive any principled understanding of their effects on the DNN capacity or trainability, and to understand why or in which aspect one specific connectivity pattern is better than another.
In this work, we theoretically characterize the impact of connectivity patterns on the convergence of DNNs under gradient descent training in fine granularity.
By analyzing a wide network’s Neural Network Gaussian Process (NNGP), we are able to depict how the spectrum of an
NNGP kernel propagates through a particular connectivity pattern, and how that affects the bound of convergence rates.
As one practical implication of our results, we show that by a simple filtration on “unpromising" connectivity patterns, we can trim down the number of models to evaluate, and significantly accelerate the large-scale neural architecture search without any overhead.
Code is available at: https://github.com/VITA-Group/architecture_convergence .
Introduction
Recent years have witnessed substantial progress in designing better deep neural network architectures.
The common objective is to build networks that are easy to optimize, of superior trade-offs between efficiency and accuracy, and are generalizable to diverse tasks and datasets.
When developing deep networks, how operations (linear transformations and non-linear activations) are connected and stacked together is vital, which is studied in network’s convergence [18, 71, 75], complexity [47, 50, 21], generalization [14, 8, 61], loss landscapes [36, 20, 51] etc.
Although it has been widely observed that the performance of deep networks keeps being improved by advanced design options our understanding remains limited on how a network’s properties are influenced by its architectures.
For example, in computer vision, design trends have shifted from vanilla chain-like stacked layers [33, 32, 53] to manually elaborated connectivity patterns (ResNet [25], DenseNet [28], etc.).
While people observed smoothed loss landscapes [36], mitigated gradient vanishing problem [4], and better generalization [29], these findings only explain the effectiveness of adding skip connections in general, but barely lead to further “finer-grained" insight on more sophisticated composition of skip connections beyond ResNet.
Recently, the AutoML community tries to relieve human efforts and propose to automatically discover novel networks from Equal gigantic architecture spaces [72, 46].
Despite the strong performance [57, 27, 62, 58], the searched architectures are often composed of highly complex (or even randomly wired) connections, leaving it challenging to analyze theoretically.
Understanding the principles of deep architecture connectivity is of significant importance.
Scientifically [Figure] this helps answer why composing complicated skip connection patterns has been such an effective “trick" in improving deep networks’ empirical performance.
Practically, this has direct guidance in designing more efficient and expressive architectures.
To close the gap between our theoretical understandings and practical architectures, in this work we target two concrete questions:
Q1:
Can we understand the precise roles of different connectivity patterns in deep networks?
Q2:
Can we summarize principles on how connectivity should be designed in deep networks?
[Figure]
One standard way to study how operations and connections affect the network is to analyze the model’s convergence under gradient descent [19, 2, 18].
Here, we systematically study the relationship between the connectivity pattern of a neural network and the bound of its convergence rate.
A deep architecture can be viewed as a directed acyclic computational graph (DAG), where feature maps are represented as nodes and operations in different layers are directed edges linking features.
Under this formulation, by analyzing the spectrum of the Neural Network Gaussian Process (NNGP) kernel, we show that the bound of the convergence rate of the DAG is jointly determined by the number of unique paths in a DAG and the number of parameterized operations on each path.
Note that, although several prior arts [44, 1, 10, 11] explored deep learning theories to predict the promise of architectures, their indicators were developed for the general deep network functions , not for fine-grained characterization for specific deep architecture topology patterns .
Therefore, their correlations for selecting better architecture topologies are only empirically observed , but not theoretically justified .
In contrast, our fine-grained conclusion is theory-ground and also empirically verified.
Based on this conclusion, we present two intuitive and practical principles for designing deep architecture DAGs: the “effective depth” ¯ d and the “effective width” ¯ m .
Experiments on diverse architecture benchmarks and datasets demonstrate that ¯ d and ¯ m can jointly distinguish promising connectivity patterns from unpromising ones.
As a practical implication, our work also suggests a cheap “plug-and-play” method to accelerate the neural architecture search by filtering out potentially unpromising architectures at almost zero cost, before any gradient-based architecture evaluations.
Our contributions are summarized below:
• We first theoretically analyze the convergence of gradient descent of diverse neural network architectures, and find the connectivity patterns largely impact their bound of convergence rate. • From the theoretical analysis, we abstract two practical principles on designing the network’s connectivity pattern: “effective depth” ¯ d and “effective width” ¯ m .
• Both our convergence analysis and principles on effective depth/width are verified by experiments on diverse architectures and datasets.
Our method can further significantly accelerate the neural architecture search without introducing any extra cost.
Related works 2.1 Global convergence of deep networks
Many works analyzed the convergence of networks trained with gradient descent [31, 68, 22, 9].
Convergence rates were originally explored for wide two-layer neural networks [60, 54, 55, 38, 19, 43, 3].
More recent works extended the analysis to deeper neural networks [2, 18, 42, 74, 71, 75, 30] and showed that over-parameterized networks can converge to the global minimum with random initialization if the width (number of neurons) is polynomially large as the number of training samples.
In comparison, we expand such analysis to bridge the gap between theoretical understandings of general neural networks and practical selections of better “fine-grained" architectures.
2.2 Residual structures in deep networks
Architectures for computer vision have evolved from plain chain-like stacked layers [33, 32, 53] to elaborated connectivity patterns (ResNet [25], DenseNet [28], etc.).
With the development of
AutoML algorithms, novel networks of complicated operations/connections were discovered.
Despite their strong performance [57, 27, 62, 58], these architectures are often composed of highly complex connections and are hard to analyze.
[52] defined the depth and width for complex connectivity patterns, and studied the impacts of network topologies in different cases.
To better understand these residual patterns, people tried to unify the architectures with different formulations.
One seminal way is to represent network structures into graphs, then randomly sample different architectures from the graph distribution and empirically correlate their generalization with graph-related metrics [62, 67].
Other possible ways include geometric topology analysis [6], network science [5], etc.
For example, [5] proposed NN-Mass by analyzing the network’s layer-wise dynamic isometry, and then empirically linked to the network’s convergence.
However, none of those works directly connect the convergence rate analysis to different residual structures.
2.3 Theory-guided design of neural architectures
Particularly related to our work is an emerging research direction, that tries to connect recent deep learning theories to guide the design of novel network architectures.
The main idea is to find theoretical indicators that have strong correlations with network’s training or testing performance. [44] pioneered a training-free NAS method, which empirically leveraged sample-wise activation patterns to rank architectures.
[45] leveraged the network’s NNGP features to approximate its predictions.
Different training-free indicators were evaluated in [1], and the “synflow” measure [59] was leveraged as the main ranking metric.
[10] incorporated two theory-inspired metrics with supernet pruning as the search method.
However, these works mainly adapted theoretical properties of the general deep neural network function : their correlations with the concrete network architecture topology are only empirically observed , but not theoretically justified .
Topology matters: convergence analysis with connectivity patterns
In this section, we study the convergence of gradient descent for deep networks, whose connectivity patterns can be formulated as small but representative direct acyclic graphs (DAGs).
Our goal is to compare the convergence rate bounds of different DAGs, and further establish links to their connectivity patterns, leading to abstracting design principles.
3.1 Problem setup and architectures notations
𝑿 (#) 𝑾 (#,’)
(Input) 𝑿 (’)
𝑿 (() 𝑿 ())
(Output) 𝑾 (#,() 𝑾 (#,)) 𝑾 ((,)) 𝑾 (’,() 𝑾 (’,))
We consider a network’s
DAG as illustrated in Figure 2.
X ( h ) ( h ∈ [0 , H ]) is the feature map (node) and W is the operation (edge).
X (0) is the input node, X ( H ) is the output node ( H = 3 in this case), and X (1) , · · · , X ( H − 1) are intermediate nodes.
The DAG is allowed to be fully con- nected: any two nodes could be connected by an operation.
Feature maps from multiple edges coming to one vertex will be directly summed up.
This DAG has many practical instances: for example, it is used in NAS-Bench-201 [17], a popular NAS benchmark.
The forward process of the network in Figure 2 can be formulated as:
[Equation]
a ⊤ X (3) .
Feature X ( s ) ∈
R m × 1 , where m is the absolute width of an edge (i.e. number of neurons), and s ∈ { 1 , 2 , 3 } .
a is the final layer and u is the network’s output.
We consider three candidate operations for each edge: a linear transformation followed by a non-linear activation, or a skip- connection (identity mapping), or a broken edge (zero mapping): W [Equation] 
[Equation]
zero
[Equation]
skip-connection
[Equation] transformation [Equation]   zero skip-connection
[Equation] linear transformation . (2) s, t ∈ { 1 , 2 , 3 } , N is the Gaussian distribution, and σ is the activation function. c σ = (cid:0) E x ∼ N (0 , 1) (cid:2) σ ( x ) 2 (cid:3)(cid:1) − 1 is a scaling factor to normalize the input in the initialization phase.
Consider the Neural Network Gaussian Process (NNGP) in the infinite-width limit [34], we define our
NNGP variance as K ( s ) ij = ⟨ X ( s ) i , X ( s ) j ⟩ and NNGP mean as b ( s ) i
= E [ X ( s ) i ] , i, j ∈ 1 , · · · , N for
N training samples in total.
Both K ij and b i are taken the expectation over the weight distributions.
3.2 Preliminary: bounding the network’s linear convergence rate via NNGP spectrum
Before we analyze different connectivity patterns (Section 3.3), we first give the linear convergence of our DAG networks (Theorem 3.1) and also show the guarantee of the full-rankness of λ min ( K ( H ) )
(Lemma 3.1).
For a sufficiently wide neural network, its bound of convergence rate to the global minimum can be governed by the NNGP kernel.
The linear convergence rate for a deep neural network of a DAG-like connectivity pattern is shown as follows:
Theorem 3.1 (Linear Convergence of DAG) .
Consider a DAG of H nodes and P H end-to-end paths.
At k -th gradient descent step on N training samples, with MSE loss L ( k ) = 12 (cid:13)(cid:13)
y − u ( k ) (cid:13)(cid:13) 2 2 , suppose the learning rate η
= O (cid:18) λ min ( K ( H ) ) ( NP H ) 2 2 O ( H ) (cid:19) and the number of neurons per layer m
= Ω (cid:18) max
[Equation]
K
[Equation]
[Equation]
, NP
[Equation]
[Equation]
log
[Equation] ( K [Equation] (cid:27)(cid:19) , we have
[Equation]
where P H is number of end-to-end paths from X (0) to X ( H ) .
Remark 3.2 .
In our work, we will use a fixed small learning rate η across different network architectures, to focus our analysis on the impact of λ min ( K ( H ) ) .
This is motivated by the widely adopted setting in popular architecture benchmarks [48, 17] that we will follow in our experiments.
The above theorem shows that the convergence rate is bounded by the smallest eigenvalue of K ( H ) , and also indicates that networks of larger least eigenvalues will more likely to converge faster.
This requires that the K ( H ) has full rankness, which is demonstrated by the following lemma:
Lemma 3.1 (Full Rankness of K ( H ) ) .
Suppose σ ( · ) is analytic and not a polynomial function.
If no parallel data points, then λ min (cid:0) K ( H ) (cid:1) > 0 .
3.3 How does NNGP propagate through DAG?
Now we are ready to link the DAG’s connectivity pattern to the bound of its convergence rate.
Although different DAGs are all of the linear convergence under gradient descent, they are very likely to have different bounds of convergence rates.
Finding the exact mapping from λ min ( K (0) ) to λ min ( K ( H ) ) will lead us to a fine-grained comparison between different connectivity patterns.
First, for fully-connected operations, we can obtain the propagation of the NNGP variance and mean between two consecutive layers:
𝑿 (#) 𝑾 (#,’)
(Input) 𝑿 (’)
𝑿 (() 𝑿 ())
(Output) 𝑾 ((,)) 𝑾 (’,() 𝑿 (#) (Input) 𝑿 (%) 𝑿 (&) 𝑿 (’)
(Output) 𝑾 (#,’)
𝑾 (&,’) 𝑾 (%,’)
𝑰 𝑰 𝑿 (#) 𝑾 (#,’)
(Input) 𝑿 (’)
𝑿 (() 𝑿 ())
(Output) 𝑾 (#,() 𝑾 ((,))
𝑰 𝑰 𝑰
Lemma 3.2 (Propagation of K ) . Define the propagation as K ( l ) = f ( K ( l − 1) ) and b ( l ) = g ( b ( l − 1) ) .
When the edge operation is a linear transformation, we have:
[Equation]
where z , z 1 and z 2 are independent standard Gaussian random variables. Besides, (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution. Remark 3.3 . Since b ( l ) i is a constant, it will not affect our analysis and we will omit it below.
Remark 3.4 .
With centered normalized inputs, K (0) ii = 1 . Our last lemma to prepare states that we can bound the smallest eigenvalue of the NNGP kernel of N × N by its 2 × 2 principal submatrix case. Lemma 3.3.
For a positive definite symmetric matrix K ∈ R N × N , the smallest eigenvalue is bounded by the smallest eigenvalue of its 2 × 2 principal sub-matrix.
[Equation]
Now we can analyze the smallest eigenvalue
λ min ( K ( H ) ) for different DAGs.
Note that here we adopt ReLU activation for three reasons: 1) Analyzing ReLU with the initialization at the edgeof-chaos [23] exhibits promising results; 2) Previous works also imply that the convergence rate of ReLU-based networks depends on λ 0 [19]; 3) ReLU is the most commonly used activation in practice.
Specifically, we set c σ = 2 for ReLU [18, 23].
In Figure 3, we show three representative DAGs ( H = 3 ).
For a fair comparison, each DAG has three linear transformation layers, i.e., they have the same number of parameters and mainly differ in how nodes are connected.
After these three case studies, we will show a more general rule for the propagation of K (0) .
DAG#1 (sequential connections).
In this case, W (0 , 1) , W (1 , 2) , W (2 , 3) = linear transformation, and W (0 , 2) , W (0 , 3) , W (1 , 2) =
zero (broken edge).
There is only one unique path connecting from
X (0) to X (3) , with three parameterized operations on it.
[Equation]
By Lemma 3.3, we calculate the upper bound of the least eigenvalue of K (3) , denoted as λ dag1 : λ min
[Equation]
where we denote the function composition f 3 ( K (0) ij )
=
f ( f ( f ( K (0) ij ))) .
DAG#2 (parallel connections).
In this case, W (0 , 3) , W (1 , 3) , W (2 , 3) = linear transformation,
W (0 , 1) , W (1 , 2) = skip-connection, and W (0 , 2) =
zero (broken edge).
There are three unique paths connecting from X (0) to X (3) , with one parameterized operation on each path.
[Equation]
where C is a constant matrix with all entries being the same.
Therefore:
[Equation]
DAG#3 (mixture of sequential and parallel connections).
In this case, W [Equation] [Equation] , W (2 , 3) = linear transformation, W (0 , 3) , W (1 , 2) , W (1 , 3) = skip connection.
There are four unique paths connecting from X (0) to X (3) , with 0/1/2 parameterized operations on each path.
[Equation]
Then we have:
[Equation]
where ˜ f ( K [Equation] 0) ij
[Equation] = [Equation]
Conclusion: by comparing Eq.
7, 9, and 11, we can show the rank of three upper bounds of least eigenvalues as: λ dag1 < λ dag2 < λ dag3 .
Therefore, the bound of the convergence rate of DAG#3 is better than DAG#2, and DAG#1 is the worst.
See our Appendix C in supplement for detailed analysis.
General rules of propagation from K to K
( H )
• Diagonal elements of K ( H ) is determined by the number of unique paths from X (0) to X ( H ) .
• Off-diagonal elements of K ( H ) is determined by the number of incoming edges of X ( H ) and the number of parameterized operations on each path.
Thus, we could simplify our rules as:
[Equation]
(12) where P is number of end-to-end paths from X (0) to X ( H ) , d p is the number of linear transformation operations on the p -th path, and f d p indicates a d p -power composition of f .
This summary is based on the propagation of the NNGP variance.
With this rule, we can quickly link the relationship between the smallest eigenvalue of K ( H ) and K (0) .
Practical principle of connectivity: effective depth and effective width
Eq.
12 precisely characterizes the impact of a network’s topology on its convergence.
Inspired by above results, we observe two factors that control the λ min ( K ( H ) ) :
(1) P , the “width” of a DAG; (2) d p ( p ∈ [1 , P ]) , the “depth” of a DAG.
However, directly comparing convergences via Eq.is non-trivial (see our Appendix C), because: (1) the complicated form of the NNGP propagation “ f ” (Eq. 3.2); (2) “ P ” and “ d p ”s are not truly free variables, as they are still coupled together in the discrete topology space of a fully connected DAG.
Motivated by these two challenges, we propose a practical version of our principle.
We simplify the analysis of Eq.
12 by reducing “ P ” and “ d p ”s to only two intuitive variables highly related to the network’s connectivity patterns: the effective depth and effective width.
Definition 4.1 (Effective Depth and Width) .
Consider the directed acyclic graph (DAG) of a network.
Suppose there are P unique paths connecting the starting and the ending vertex.
Denote the number of parameterized operations on the p -th path as d p , p ∈ [1 , P ] .
We define:
Effective
[Equation]
where 1 ( d p > 0) = 1 if d p > 0 otherwise is 0 .
Remark 4.2 .
In our experiments below, we consider fully-connected or convolutional layers as parameterized operations, ignoring their detailed configurations (kernel sizes, dilations, groups, etc.).
We consider skip-connections and pooling layers as non-parameterized operations.
The effective depth considers the averaging effects of multiple parallel paths, and the effective width considers the amount of unique information flowing from the starting vertex to the end, normalized by the overall depth.
We will demonstrate later in Section 5.2 that performances of networks from diverse architecture spaces show positive correlations with these two aspects.
While ¯ d and ¯ m may be loose to predict the best architecture, in Section 5.3 we will show that they are very stable in distinguishing bad ones.
Therefore, using these two principles, we can quickly determine if an architecture is potentially unpromising at almost zero cost, by only analyzing its connectivity without any forward or backward calculations.
Experiments 5.1 Empirical convergence confirms our analysis
[Figure]
We first experimentally verify our convergence analysis in
Section 3.3.
In all cases we use ReLU nonlinearities with
Kaiming normal initialization [24].
We build the same three computational graphs of fully-connected layers in
Figure 3.
Three networks have hidden layers of a constant width of 1024.
We train the network using SGD with a mini-batch of size 128.
The learning rate is fixed at 1 × 10 − 5 .
No augmentation, weight decay, learning rate decay, or momentum is adopted.
Based on our analysis, on both MNIST and CIFAR-10, the convergence rate of DAG#1 (Figure 3 left) is worse than DAG#
(Figure 3 middle), and is further worse than
DAG#3 (Figure 3 right).
The experimental observation is consistent with this analysis: the training accuracy of DAG#3 increases the fastest, and DAG#2 is faster than DAG#1.
Although on CIFAR-10 DAG#1 slightly outperforms DAG#2 in the end, DAG#2 still benefits from faster convergence in most early epochs.
This result confirms that by our convergence analysis, we can effectively compare the convergence of different connectivity patterns.
5.2 Extreme ¯ d or ¯ m leads to bad performance on architecture benchmarks
In this experiment, our goal is to verify the two principles we proposed in Section 4.
Specifically, we will leverage a large number of network architectures of random connectivities, calculate their ¯ d and ¯ m , and compare their performance.
We consider popular architecture benchmarks that are widely studied in the community of neural architecture search (NAS).
Licenses are publicly available.
• The NAS-Bench-
[17] provides 15,625 architectures that are stacked by repeated DAGs of four nodes (exactly the same DAG we considered in Section 3 and Figure 2).
It contains architecture’s performance on three datasets (CIFAR-10, CIFAR-100, ImageNet-16-120 [15]) evaluated under a unified protocol (i
e. same learning rate, batch size, etc., for all architectures).
The operation space contains zero , skip-connection , conv 1 × 1 , conv 3 × 3 convolution , and average pooling 3 × 3 .
• Large-scale architecture spaces: NASNet [73], Amoeba [49], PNAS [39], ENAS [46], DARTS [40].
These spaces have more operation types and more complicated rules on allowed DAG connectivity [Figure] [Figure] [Figure]
[Figure] ﬀ [Figure] i d t
[Figure] ¯ [Figure] patterns, see Figure 1 bottom as an example.
We refer to their papers for details.
A benchmark is further proposed in [48]: about 5000 architectures are randomly selected from each space above, and in total we have 24827 architectures pretrained on CIFAR-10.
We will use this prepared data 2 .
We show our results in Figure 5.
From all four plots 3 (across different architecture spaces and datasets), it can be observed that, architectures of extreme depths or widths suffer from bad performance.
We also calculate the multi-correlation 4 ( R ) between the accuracy and the joint of both ¯ d and ¯ m , and show as legends in Figure 5.
All four plots show positive correlations.
It is worth noting that here each dot represents a subset of architectures that share the same ¯ d and ¯ m , with possible different fine-grained topology or layer types, indicating the generalizability of our proposed principles. The variance of performance in each subset is represented by the radius of the dot.
Remark 5.1 .
Our notion of “extreme ¯ d or ¯ m ” targets a complete space of DAGs: given the total number of nodes, any two nodes in the graph can be connected, and there are no topological restrictions or disabled edges (i.e., one cannot introduce any prior to the distribution of graph topologies).
5.3 A “Plug-and-Play” method for accelerating
NAS bypassing extreme ¯ d
and ¯ m
Inspired by the bad performance from extreme ¯ d or ¯ m discussed in Section 5.2, we are motivated to further contribute a “plug-and-play” tool to accelerate practical NAS applications.
Method.
In NAS, the bottleneck of search efficiency is the evaluation cost during the search.
Our core idea is to use ¯ d and ¯ m to pre-select architectures that are very likely to be unpromising before we spend extra cost to evaluate them.
Training-based NAS is slow: one has to perform gradient descent on each sampled architecture and use the loss or accuracy to rank different networks.
We instead choose to improve two recent training-free NAS methods: NAS-WOT [44] and TE-NAS [10].
This is because our goal here is to accelerate NAS, and these two are state-of-the-art NAS works of extremely low search time cost.
We plan to show that our method can even further accelerate these two training-free NAS works.
The rationale is that, our ¯ d and ¯ m are even much cheaper: we only do a simple calculation on a network’s DAG structure, but NAS-WOT and TE-NAS still have to perform forward or backward on a mini-batch of real data.
Although our ¯ d and ¯ m are only inspired from the optimization perspective, not the complexity or generalization, our method mainly filters out bad architectures at a coarse level, but does not promote elites in a fine-grained way.
NAS-WOT proposed to rank architectures by their local linear maps.
Given a mini-batch of data, each point can be identified by its activation pattern through the network, and a network that can assign more unique activation patterns will be considered promising.
NAS-WOT uses random sampling to search the architectures and rank them based on this training-free score.
Our version : We will skip potentially unpromising architecture before leaving them for
NAS-WOT to calculate their scores.
TE-NAS proposed to rank architectures by combining both the trainability (condition number of
NTK) and expressivity (number of linear regions).
TE-NAS progressively pruned a super network to a single-path network, by removing unpromising edges of low scores. [Table] [Table]
Our version : We will skip potentially unpromising architecture before leaving them for
TE-NAS to prune.
We provide a pseudocode algorithm in Appendix A to demonstrate the usage of our method.
How to choose ¯ d and ¯ m ?
Although ¯ d and ¯ m are hyperparameters, it is worth noting that they can be determined in a highly principled way.
In practice, given a search space, we only calculate the center ( ¯ d ∗ , ¯ m ∗ ) =
( ¯ d max +¯ d min, ¯ m max + ¯ m min) and the radius ( r ¯ d , r ¯ m ) =
( ¯ d max − ¯ d min, ¯ m max − ¯ m min) .
We by default only keep architectures within half of the radius from the center for evaluation: | ¯ d − ¯ d ∗ | ≤ 12 r ¯ d and | ¯ m − ¯ m ∗ | ≤ 12 r ¯ m .
This principle leads to ( ¯ d ∗ , ¯ m ∗ ) =
(1 . 5 , 10 . 5) with 12 ( r ¯ d , r ¯ m ) =
8) on DARTS, and ( ¯ d ∗ , ¯ m ∗ ) =
(1 . 6 , 2 . 2) with 12 ( r ¯ d , r ¯ m ) =
(0 . 7 , 0 . 9) on NAS-Bench-201.
Implementation settings.
We train searched architectures for 250 epochs using SGD, with a learning rate as 0.5, a cosine scheduler, momentum as 0.9, weight decay as 3 × 10 − 5 , and a batch size as 768.
This setting follows previous works [1, 44, 69, 41, 66, 26, 12, 10].
DARTS space on ImageNet.
As shown in Table 1, for both two training-free NAS methods, by adopting our pre-filtration strategy, we can further reduce the search time cost and achieve better search results.
For NAS-WOT, we can save 16.7% search time cost and reduce 0.3% top-1 error.
For
TE-NAS, we can significantly save 41.2% search time cost, and still improve the top-1 error by 0.4%.
FLOPs of our search architectures are 0.68G (TE-NAS + DAG) and 0.56G (WOT + DAG).
[Table]
NAS-Bench-201 Space.
As shown in Table 2, for both NAS-WOT and TE-NAS, we reduce over 30% search time cost with strong accuracy.
We also include a training-based method FP-NAS [65], where we even achieve 46% seach cost reduction with better performance.
Moreover, we show that our method is robust to the choices of ¯ d and ¯ m .
In the ablation study in Table 3, by changing different ranges of ¯ d and ¯ m , our method remains strong over the WOT baseline.
Conclusion
In this work, we show that it is possible to conduct fine-grained convergence analysis on networks of complex connectivity patterns.
By analyzing how an NNGP kernel propagates through the networks, we can fairly compare different networks’ bounds of convergence rates.
This theoretical analysis and comparison are empirically verified on MNIST and CIFAR-10.
To make our convergence analysis more practical and general, we propose two intuitive principles on how to design a network’s connectivity patterns: the effective depth and the effective width.
Experiments on diverse architecture benchmarks and datasets demonstrate that networks with an extreme depth or width show bad performance, indicating that both the depth and width are important.
Finally, we apply our principles to the large-scale neural architecture search application, and our method can largely accelerate the search cost of two training-free efficient NAS works with faster and better search performance.
Our work bridge the gap between the Deep Learning theory and the application part, making the theoretical analysis more practical in architecture designs Acknowledgement
B. Hanin and Z. Wang are supported by NSF Scale-MoDL (award numbers: 2133806, 2133861)
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] In Remark 5.1, our claim “Extreme ¯ d or ¯ m leads to bad performance” requires architectures from a complete space of DAGs, without introducing any prior on the DAG topology.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work does not have negative societal impacts (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] We include our proofs in supplement.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include our code and Readme of instructions in supplement.
Data is publicly available online.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We run experiments for three random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Table 1. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Licenses are publicly available (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Pseudocode of our “Plug-and-Play” method
Here we provide a pseudocode of using our “Plug-and-Play” method in the simplified WOT algorithm [44].
Black lines are from original WOT.
Red lines are our method, which is extremely easy to use and cheap to calculate.
[Table]
B Proofs for Section 3.
Lemma 3.2 (Propagation of K ) .
Define the propagation as K ( l ) =
f ( K ( l − 1) ) and b ( l ) =
g ( b ( l − 1) ) .
When the edge operation is a linear transformation, we have,
[Equation]
where z , z 1 and z 2 are independent standard Gaussian random variables.
Besides, (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Proof of Lemma 3.2.
According to the property of Gaussian process for neural networks [34], for each layer l , we have
[Equation]
where u = (cid:113) K ( l − 1) ii z 1 and v = (cid:113) K ( l − 1) jj (cid:18) C ( l −
[Equation] ij z
[Equation] + (cid:113) 1 − ( C
[Equation]
) 2 z(cid:19) , with C ( l ) ij =
K ( l ) ij / (cid:113) K ( l ) ii K ( l ) jj .
Besides, (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution. Note that the diagonal entry K ( l ) ii of NNGP is a self-correlation term thus governed by one random variance.
On the other hand, K ( l ) ij is the correlation between X ( l ) i and X ( l ) j , therefore it contains two independent random variables.
Finally, we calculate the bias term:
[Equation]
Lemma 3.3.
For a positive definite symmetric matrix K ∈ R N × N , the smallest eigenvalue is bounded by the smallest eigenvalue of its 2 × 2 principal sub-matrix.
[Equation]
[Equation]
Proof of Lemma 3.3.
We first have the Cauchy Interlace Theorem : Suppose A ∈ R n × n is symmetric.
Let A ( n − 1) ∈
R ( n − 1) ×
( n − 1) be a principal submatrix of A (obtained by removing both i -th row
and i -th column for any i ∈ [1 , n ] ).
Suppose A has eigenvalues λ ( n )≥ λ ( n )≥ · · · ≥ λ ( n ) n and
A ( n − 1) has eigenvalues λ ( n − 1)≥ λ ( n − 1)≥ · · · ≥ λ ( n − 1) n − 1 . Then: λ ( n )≥ λ ( n − 1)≥ λ ( n )≥ λ ( n
[Equation]
1)≥ · · · ≥ λ ( n − 1) n
[Equation]
1 ≥ λ ( n ) n .
By repeating this theorem for n − 2 times, we can have:
[Equation]
This means we can upper bound λ ( n ) n by the minimal smallest eigenvalue of any 2 × 2 principal submatrix of A .
C Detailed Analysis on three DAGs in Figure 3 and Section 3.3.
We take the condition of ReLU that σ ( x ) = max { 0 , x } and c σ = 2 into Lemma 3.2 and obtain: [Equation] ii [Equation]
arcsin
[Equation]
We provide more properties regarding the propagation of normalized non-diagonal elements in NNGP by studying the function
[Equation]
It is known that f is differentiable and satisfies: [Equation] Therefor we have h ( C ( l − 1) ii ) > C ( l − 1) ii , for C ( l − 1) ii ∈ (0 , 1) . Finally, we show the Taylor expansion for h ( C ( l − 1) ii ) when C ( l − 1) ii is close to 1:
[Equation] This can be obtained by expanding each term in h ( C ( l − 1) ij ) . Before our analysis
[Equation]
we re-state some assumptions and facts:
• h ( · ) is a monotonically increasing function in [0 , 1) , and lim C ( l − 1) ij
[Equation] • We assume no parallel data points in the training data, thus C (0) ij ∈ [0 , 1) and K 0 ij ∈ [0 , 1) .
This further indicates h ( C (0) ij ) , h ( K (0) ij ) ∈
[1 /π, 1) .
• f ( K ( l − 1) ij ) >
K ( l − 1) ij .
We first show that λ dag1 < λ dag2 , namely, the upper bounds of the least eigenvalues of DAG#1 is smaller than that of DAG#2 in Figure 3:
[Equation]
Before we compare λ dag2 and λ dag3 , we first try to simplify λ dag3 .
Specifically, we have:
[Equation]
and
[Equation]
Now we compare λ dag2 and λ dag3 :
[Equation]
[Equation] As K (0) ij → 1 − , we have C (0) ij → 1 − and h ( C ( l − 1) ij ) → 1 − .
Thus lim K (0) ij → 1 − λ dag3 − λ dag2 = 0 .
Moreover, take the derivative of λ dag3 − λ dag2 w.r.t.
K (0) ij , we have:
[Equation]
The inequality considers the fact that h ′ ∈ [ 12 , 1) .
Therefore, λ dag3 − λ dag2 is a monotonically decreasing function w.r.t.
K (0) ij on [0 , 1) .
Thus, we have λ dag3 − λ dag2 > 0 on [0 , 1) .
In conclusion, we have λ dag3 > λ dag2 > λ dag1 .
Therefore, the convergence rates are ranked as:
DAG#3 > DAG#2 > DAG#1.
D Coefficient of Multiple Correlation R
Our correlation coefficient R between the accuracy and the joint of both ¯ d and ¯ m is calculated as blow:
[Equation]
E Proof of Theorem 3.
Theorem 3.1 (Linear Convergence of DAG) .
At k -th iteration of gradient descent on N training data samples, we have a network parameterized by θ ( k ) with its output as u ( k ) .
The network’s connectivity pattern can be formulated as a DAG of H nodes and P end-to-end paths.
With MSE loss L ( θ ( k ))
= 12 ∥ y − u ( k ) ∥ 22 , suppose the learning rate η
= O (cid:18) λ min ( K ( H ) ) ( NP )2 O ( H ) (cid:19) and the number of neurons (width) per layer m
= Ω (cid:18) max (cid:26) ( NP H ) 4 λ 4min
( K ( H ) ) , NP H H δ , ( NP H ) 2 log ( HNδ ) [Equation] ( H ) λ 2min ( K ( H ) )
(cid:27)(cid:19) , we have
[Equation]
Proof Sketch of Theorem 3.1.
The main purpose of this theorem is to extend the convergence of
MLP and ResNet studied in [18] to general DAG networks.
One can first decompose the induction hypothesis:
Condition E.1.1.
At the k -th iteration, we have
[Equation]
into:
[Equation]
where G ( H ) ( k ) = (cid:80)
{+H − 1 s =
+}
G ( s,H ) ( k ) as the NTK of incoming edges into the last node, composed by the NTK of each edge.
This induction implies that the convergence rate of gradient descent dynamics of DAG networks with an MSE loss is linear.
To prove the induction in Theorem 3.1, three core steps are:
Step 1.
Show at initialization, λ min (cid:0) G ( H ) (0) (cid:1) ≥ 3 λ min ( K ( H ) )when m is sufficiently large.
Step 2.
Show that during gradient descent, (cid:13)(cid:13) G ( H ) ( k ) − G
( H ) (0) (cid:13)(cid:13) 2 ≤ 14 λ min K ( H ) ) when m is sufficiently large.
Step
Show the last two terms, both − ( y −
u
( k ))
⊤ I ( k ) and ∥ u ( k + − u ( k ) ∥ 22 are proportional
to η 2 ∥ y − u ( k ) ∥ , and if we
set η sufficiently small , they
are smaller than ηλ min (cid:0)
G ( H ) ( k ) (cid:1) ∥
y − u ( k ) ∥ 22 , leading the MSE loss linearly decrease.
In our proof, we mainly focus on deriving the condition on m (the required number neurons per layer) and η (learning rate) in our DAG setting, by analyzing λ min (cid:0) G ( H ) (0) (cid:1) .
Our proof relies on lemmas below, which are based on first analyzing the gradient on each path and then considering the DAG as a union bound over all P H number of end-to-end paths.
We want to point out another benefit of bounding both λ min (cid:0) G ( H ) ( k ) (cid:1) and the linear convergence rate via λ min ( K ( H ) ) .
Since we try to find the DAG-specific propagation from λ min ( K (0) ) to λ min ( K ( H ) ) , and all DAGs share the same λ min ( K (0) ) , we can more fairly compare the bound of convergence rate of different DAGs. Definition E.2.
The geometric series function: [Equation]
Lemma E.1 (Lemma on Initialization Norms) .
If σ ( · ) is a L − Lipschitz activation and m = Ω (cid:16) NP 2 H Hg C ( H ) 2 δ (cid:17) , where C ≜ c σ L (cid:16) 2 | σ (0) | (cid:113) 2 π + 2 L (cid:17) and P H is the total number of end-toend paths, then with probability at least 1 − δ over random initialization, for every h ∈
[ H ] and i ∈ [ N ] , we have 0 ≤ (cid:13)(cid:13)(cid:13) X
( h ) i (0) (cid:13)(cid:13)(cid:13) 2 ≤ c x, 0 , where c x, 0 = 2 P H max(1 , c w, 0 L ) and c w, 0 is a universal constant.
We follow the proof sketch described in Section E. We first analyze the spectral property of G ( H ) (0) at the initialization phase.
The following lemma lower bounds its least eigenvalue.
Lemma E.2 (Least Eigenvalue at the Initialization) .
If m = Ω (cid:16) ( NP H ) 2 log( HN/δ )2 O ( H ) λ min ( K (0) )(cid:17) , we have [Equation]
Next, we characterize how the perturbation on the weight matrices affects the input of each layer. Lemma E.3. Suppose σ ( · ) is L -Lipschitz and for h ∈ [ H ] , (cid:13)(cid:13) W ( s,t ) (0) (cid:13)(cid:13) 2 ≤ c w, 0 √ m , (cid:13)(cid:13) X ( t ) {+(0) (cid:13)(cid:13) 
+}
≤ c x, 0 and (cid:13)(cid:13) W [Equation] s [Equation]
t ) [Equation] k ) [Equation]
W
[Equation]
s,
[Equation]
[Equation]
[Equation]
≤ √ mR for some constant c w, 0 , c x, 0 >and R ≤ c w, 0 .
Then we have
[Equation]
Next, we characterize how the perturbation on the weight matrices affect G ( H ) .
Lemma E.4.
Suppose σ ( · ) is differentiable, L − Lipschitz and β − smooth.
Suppose for s, t ∈
[ H ] ,
[Equation]
2 ≤ c x, 0 , if
[Equation]
W
[Equation]
W
[Equation]
[Equation]
F
[Equation]
∥
[Equation]
k
[Equation]
−
[Equation]
0) ∥ 2 ≤ √ mR where R ≤ cλ 0 ( P H N ) − 1 and R ≤ c for some small constant c , we have
[Equation] Lemma E.5.
If Condition E.1.1 holds for 1 , . .
, k , we have for any k ′ ∈
[ k + 1] and any s, t ∈ [ H ] (cid:13) (cid:13)
[Equation]
where R = P
[Equation]
< c for some small constant c and
Q
[Equation]
The next lemma bounds the I 2 term.
Lemma E.6.
If Condition E.1.1 holds for 1 , . .
k and η ≤ cλ 0 ( NP H ) − 2 for some small constant c , we have
[Equation]
Next we bound the quadratic term.
Lemma E.7.
If Condition E.1.1 holds for 1 , . .
k and η ≤ cλ 0 ( NP H ) − 2 for some small constant c , we have ∥ u ( k + 1) − u ( k [Equation] ∥ 22 ≤ [Equation]
ηλ 0 ∥
y − u ( k ) ∥ 22 . E.2 Proofs of Lemmas Proof of Lemma E.1. We will bound (cid:13)(cid:13)(cid:13) X ( t ) i (0) (cid:13)(cid:13)(cid:13) 2 by induction on layers.
For the first layer, when W (0 , 1) is a parameterized linear operator, we can calculate:
[Equation]
where C 2 ≜ σ (0) 4 +4 | σ (0) | 3 L (cid:112) 2 {+/π +6 σ (0) 
+}
L 2 +8 | σ (0) | L 3 (cid:112) 2 /π +32 L 4 .
Considering W (0 , 1) could be a zero or an identity mapping, We have with probability at least 1 − δn ,
[Equation]
By definition we have for 2 ≤ t ≤ H ,
[Equation]
Thus we have:
[Equation]
Applying Chebyshev’s inequality and plugging in our assumption on m
[Equation]
we have with probability 1 − δ NH over W ( s,t ) ,
[Equation]
≤
[Equation]
Since W ( s,t ) could be linear, identity, or zero mapping, we have:
[Equation]
Thus 0 ≤
[Equation] [Equation]
which implies
[Equation]
Choosing c x, 0 =
P H c w, 0 L and using union bounds over [ n ] , we prove the lemma.
Proof of Lemma E.2.
The purpose of this proof is to study how large m is needed to ensure the randomly generated Gram matrices ( X ⊤ X ) is close to the population Gram matrices ( K ) in networks of general DAG connectivity patterns, via extending the general framework in Section E in [18]. For complete definitions, we refer readers to Section E.1 in [18]. Theorem E.3.
With probability
[Equation]
over the [Equation] suppose m = Ω [Equation]
) (cid:19) , for any 1 ≤ t ≤
[Equation]
−
[Equation]
≤
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] m
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
E (cid:114)
[Equation]
( HN
[Equation]
δ ) m (19) and any t ∈ [ H − 1] , ∀ 1 ≤ i ≤ N, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
m (20)
The error constant E satisfies there exists an absolute constant C > 0 such that
[Equation]
where P ( t ) 0 and P ( t ) 1 are sets of indices of nodes with skip-connection and parameterized layers into X ( s ) , respectively. We have M = O (1) , B =
O (1) , C = O (1) , Λ = O (1) , W ( t ) = O (1) , and they are defined by: • W ( s,t ) (cid:0) K ( s ) (cid:1) =
E W (cid:2) W ( s,t ) K ( s ) ( W ( s,t ) ) ⊤ (cid:3) .
• M
= 1 + 100 max
i,j,s,t |W ( s,t ) ( K ( s ) ij ) | , • B = 1 + 100 max i,t | b ( t ) i | , • C = | ρ (0) | + sup x ∈ R | ρ ′ ( x ) | , • Λ is a constant that only depends on ρ ,
• W ( t )
= 1 + max s ∥W ( s,t ) ∥
L ∞ → L ∞ .
Proof of Theorem E.3.
The proof is by induction.
Suppose that Equation (19) and (20) hold for 1 ≤ s ≤ t with probability at least 1 − tH δ , now we want to show the equations holds for t + 1 with probability at least 1 − δ/H conditioned on previous layers satisfying Equation (19) and (20).
Let s =
t
+ 1 .
recall
[Equation]
Now we get the following formula for the expectation:
[Equation]
( s )
[Equation]
[Equation]
h
[Equation]
h
[Equation]
U
[Equation]
V )
[Equation]
E ( s )
[Equation]
[Equation]
h ′
[Equation]
with
[Equation]
To bound the differences that determine how the error propagates through layers:
[Equation]
we analyze the error:
[Equation]
K
[Equation]
where we define
[Equation]
By definition
[Equation]
we have
[Equation]
We can also estimate other terms
[Equation]
≤ C (cid:114) W ( s ) max ij ∥
ˆ K ( s ) ij ∥ ∞ max i (cid:13)(cid:13)(cid:13)
ˆ b ( h ′ )
ij − b ( h ′ ) ij (cid:13)(cid:13)(cid:13) ∞ ≤ C (cid:113) W ( s ) M max i (cid:13)(cid:13)(cid:13)
ˆ b ( h ′ )
ij − b ( h ′ ) ij (cid:13)(cid:13)(cid:13) ∞ , (cid:13)(cid:13)(cid:13) E ( U , V ) ∼ ˆ A ρ ( U ) ⊤ ( b ( h ′ )
j ) − E ( U , V ) ∼ A ρ ( U ) ⊤ ( b ( h ′ )
j ) (cid:13)(cid:13)(cid:13) ∞ ≤ BC ∥
[Equation]
ˆ A ∥ ∞ ≤ BC W ( s ) max ij ∥
ˆ K ( h ′ )
ij − K ( h ′ )
ij ∥ ∞ , and (cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
[Equation]
[Equation]
) (cid:13)(cid:13)(cid:13) ∞ ≤
[Equation]
∞
[Equation]
Putting these estimates together, we have
[Equation]
and
[Equation]
These two bounds imply the theorem.
Remark E.4 .
Combing Theorem E.3 and Lemma E.1 in [18] and standard matrix perturbation bound directly have Lemma E.2.
Proof of Lemma E.3.
We prove this lemma by induction.
Our induction hypothesis is
[Equation]
[Equation]
, where g ( t
[Equation]
√
[Equation]
For h = 1 , we have
[Equation]
which implies g (1) = √ c σ LR .
For 2 ≤ t ≤ H , following the proof of Lemma C.3.
in [18] [Equation] we have [Equation]
≤ [Equation] Lastly, simple calculations show g ( t ) ≤ 2 {+(cid:81) ts =1 P s √ c σ c w, 
 L + (cid:80) ts =1 P s √ c σ Lc x, 0 R .+}
Proof of Lemma E.4.
Following the proof of Lemma C.4.
in [18], we can obtain
[Equation]
Therefore we can bound the perturbation
[Equation]
where c x ≜
[Equation]
ts =1 P s √ c σ c
[Equation] LR + [Equation] P s √ c σ Lc x, 0 .
Plugging in the bound on R , we have the desired result.
Proof of Lemma E.5. We will prove this corollary by induction. The induction hypothesis is [Equation]
First it is easy to see it holds for k ′′ = 0 .
Now suppose it holds for k ′′ = 0 , . . .
, k ′ , we consider k ′′ =
k ′ + 1 .
Applying union bound over P H number of end-to-end paths, we have [Equation] where [Equation]
are the derivative matrices induced by the activation function.
Similarly
[Equation]
we have
[Equation]
Thus
[Equation]
Similarly,
[Equation]
Proof of Lemma E.6.
We apply union bound over P H number of end-to-end paths.
[Equation]
We have bounded the RHS in the proof for Lemma E.5, thus
[Equation]
( s
[Equation]
k )
[Equation]
Q ( k ) .
Denote θ ( k, k ′ )
= θ ( k )
− k ′ L ′ ( θ ( k ))
, we have (cid:13)(cid:13)(cid:13) u ′ ( s,t ) i
[Equation]
k
[Equation]
P
[Equation]
− X ( s ) i ( k, k ′ )
a ( k
[Equation]
k ′
[Equation]
⊤
[Equation]
Through standard calculations, we have
[Equation]
Thus we have
[Equation]
Thus:
[Equation]
where we used the bound of η and
that ∥ y − u (0) ∥ 2 = O ( √ N ) ,.
Proof of Lemma E.7.
[Equation]
[Equation]
i
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
( H ) i ( k
[Equation]
1) − X ( H ) i ( k
[Equation]
{+(cid:13)(cid:13)(cid:13) 
+}
[Figure]
[Equation]
[Figure]
[Equation]
F Proof of Lemma 3.
Lemma 3.1 (Full Rankness of K ( H ) ) .
Suppose σ ( · ) is analytic and not a polynomial function.
If no parallel data points, then λ min (cid:0) K ( H ) (cid:1) > 0 .
Proof of Lemma 3.1.
First, we know that K (0) is positive definite by our assumption on the data point.
The next step is to prove propagation of either skip-connection or linear operation will keep the positive definite property.
For a linear operation, the NNGP variance matrix can be expressed as,
[Equation]
Define the feature map induced by NNGP kernel as ϕ X i ( W ) =
σ ( W ⊤ X i ) X i .
Then we will show that ϕ X 1 ( W ) , . . .
ϕ X N ( W ) are linearly independent.
Assume that there are a i ∈ R such that, N
[Equation]
Differentiating the above equation ( N − 2) times with respect to W , we then have,
[Equation]
The linearly independents of X ⊗ ( N − 1) i can be proven by induction.
For N = 2 , X 1 X ⊤ 1 and X 2 X ⊤ 2 are linearly independent under the non-parallel assumption.
Suppose { vec( X ⊗ ( N − 1)) , . .
, vec( X ⊗ ( N − 1) N −) } are linearly independent.
Then we prove the linearly independent for case of X ⊗ N i by contradiction.
Suppose there exists α 1 , . . .
, α N ∈ R not identically 0, such that N (cid:88)
[Equation]
which implies for j = 1 , .
, d
N
[Equation]
Because we have supposed the linearly independence of X ⊗ ( N − 1) i , we know there must exist j ∈ [ d ] that X ij ̸ = 0 for all i ∈
[ N ] .
However, this is contradict with the assumption that α 1 , . . .
, α N ̸
= 0 .
Thus we can conclude that { X ⊗ ( N − 1) i } Ni =1 are linearly independent.
Finally, we have
[Equation]
Then we consider the NNGP for node t : [Equation] b
[Equation]
where S is the set of nodes in the DAG except t .
When W ( s,t ) is the skip-connection operator, then we have K ( s,t ) =
K ( s ) .
On the other hand, when W ( s,t ) corresponds to linear transformation, we have K ( s,t ) ij = E W ∼N (0 , I )
[ σ ( W ⊤ z i ) σ ( W ⊤ z j )] , with z ∈ Z and Z = D 1 / 2 U ⊤ , where UDU ⊤ =
K ( m ) .
Thus for each K ( s,t ) ij we can apply the result above and conclude each matrix is positive definite.
In addition, in each matrix b ( s 1 ,t ) b ( s 2 ,t ) , all entries are the same.
Because
[Equation]
Thus we know that K ( t ) is positive definite.
Repeat the argument above, we can conclude that K ( H ) is full rankness.
G Convergence vs. DAG Topology
In Figure 6, we sample 408 networks from our DAG space ( H = 3 ).
We empirically measure how many epochs each network requires to reach 80% training accuracy as its convergence rate, and plot these epochs with networks’ topologies.
We observe that wide and shallow networks show faster convergence. This observation is aligned with Eq.
12, which states that networks with larger widths ( P H , the number of end-to-end paths) and smaller depths ( d p , the number of “Linear + ReLU” layers on each path) will have large λ 0 . However, as convergence only partially explains the final test accuracy (with missing factors like generalization and model complexity), Figure 6 does not conflict with Figure 5.
[Figure]