Generalization Properties of NAS under Activation and Skip Connection Search
Abstract
Neural Architecture Search (NAS) has fostered the automatic discovery of stateof-the-art neural architectures.
Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS.
In this work, we study the generalization properties of NAS under a unifying framework enabling (deep) layer skip connection search and activation function search.
To this end, we derive the lower (and upper) bounds of the minimum eigenvalue of the Neural
Tangent Kernel (NTK) under the (in)finite-width regime using a certain search space including mixed activation functions, fully connected, and residual neural networks.
We use the minimum eigenvalue to establish generalization error bounds of NAS in the stochastic gradient descent training.
Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performing architectures, even in the case without training, leading to a trainfree algorithm based on our theory.
Accordingly, our numerical validation shed light on the design of computationally efficient methods for NAS.
Our analysis is non-trivial due to the coupling of various architectures and activation functions under the unifying framework and has its own interest in providing the lower bound of the minimum eigenvalue of NTK in deep learning theory.
Introduction
Neural Architecture Search (NAS)
[Zoph and Le, 2017] is a powerful technique that enables the automatic design of neural architectures.
NAS defines a set of operations (referred to as the search space ), that include various activation functions and layer types, or potential connections among layers [Elsken et al., 2019, Ren et al., 2021].
Optimization over the search space returns the optimal architecture as a subset of the possible combinations of operations.
NAS 1 obtains state-of-the-art results in image recognition [Liu et al., 2019a, Ding et al., 2020, Zhang et al., 2019, Chen et al., 2019] or can be used to further improve architectures defined by a human expert [Tan and Le, 2019].
The spectacular results obtained by NAS have led to a significant interest in the community to further improve the NAS algorithms, the search space etc.
However, to date little focus has been provided in the following question: Can NAS 1 achieve generalization guarantees similar to a typical neural network?
Neural tangent kernel (NTK)-based analysis
Jacot et al., 2018] is a powerful method for analyzing the optimization and the generalization of deep networks [Allen-Zhu et al., 2019, Cao and Gu, 2019,
Chen et al., 2020a, Arora et al., 2019a].
The minimum eigenvalue of NTK has been used in previous work to demonstrate the global convergence of gradient descent, such as two-layer networks [Du et al., 2019b], and deep networks with polynomially wide layers [Allen-Zhu et al., 2019].
Besides, the minimum eigenvalue of NTK is also used to prove generalization bounds [Arora et al., 2019a] and memorization [Montanari and Zhong, 2020].
However, previous work mainly focuses on a limited set of architectures, e.g., fully-connected (FC) neural networks [Allen-Zhu et al., 2018, Bartlett et al., 2017] or residual neural networks [He et al., 2016, Huang et al., 2020], in which a single activation function is used throughout the network.
These off-the-shelf theoretical results cannot be directly applied to analyze the rich search space (of NAS) that is covering various/mixed architectures and parameters.
That makes the non-trivial analysis on NAS worth of study on its own right.
The recent work of Oymak et al.
[2021] is the first work to provide generalization guarantees on a related problem, i.e., activation functions search.
The study provides generalization results on two-layer networks relying on the minimum eigenvalue with a strictly larger than zero assumption, i.e., λ min ( K ) > 0 for the NTK matrix K .
In this work, we introduce the first theoretical guarantees for multilayer NAS where the search space includes activation functions and skip connections.
We study the upper/lower bound of the minimum eigenvalue of NTK (in the (in)finite regime) under mixed activation functions and architectures which evade the minimum eigenvalue assumption of Oymak et al.
Then, we provide optimization and generalization guarantees of deep neural networks (DNNs) equipped with NAS.
Our results indicate that the minimum eigenvalue estimation can act as a powerful metric for NAS.
This method, called Eigen-NAS, is train-free, but still effective with experimental validation when compared to recent promising algorithms [Xu et al., 2021, Chen et al., 2021, Mellor et al., 2021].
Formally, our main contribution and findings are summarized below:
i)
We build a general theoretical framework based on NTK for NAS with search on popular activation functions in each layer, fully-connected, and skip connections.
We derive the NTK formula of these architectures in the (in)finite-width regime under the unifying framework.
ii)
We derive the upper and lower bounds of the minimum eigenvalue of the NTK under the (in)finitewidth regime for the considered architectures
We introduce a new technique to ensure the probability of concentration inequality remains positive
Our analysis
highlights
how the upper and lower bounds differs under activation function search and skip connection search and can guide
NAS
iii)
We establish a connection between the minimum eigenvalue and generalization of the searched DNN trained by stochastic gradient descent (SGD).
Our theoretical results show that the generalization performance largely depends on the minimum eigenvalue of NTK for NAS, which provides theoretical guarantees for the searched architecture.
iv)
Our theoretical results are supported by thorough experimental validations with the following findings: 1) our upper and lower bounds on the minimum eigenvalue largely depend on the activation function in the first layer rather than the activation functions in deeper layers.
The applied NAS algorithm always picks up ReLU (Rectified Linear Unit) and LeakyReLU in the optimal architecture, which coincides with our theory that predicts ReLU and LeakyReLU achieve the largest minimum eigenvalues.
The skip connections are required in each layer under our not very large DNNs.
Furthermore, our experimental evidence on Eigen-NAS indicates that the minimum eigenvalue is a promising metric to guide NAS (without training) as suggested by our theory.
Technical challenges.
The technical challenges of this paper mainly focus on how to analyze activation functions with different properties and skip connections under a unifying framework.
This work is non-trivial; previous works mainly focus on the ReLU activation function [Nguyen et al., 2021, Cao and Gu, 2019, Allen-Zhu et al., 2019] in optimization and generalization of a single fullyconnected neural network.
Their proofs heavily depend on the properties of ReLU , e.g., homogeneity and ReLU( x ) = x ReLU ′
( x ) which are invalid when other commonly-used activation functions, e.g., Tanh, Sigmoid, and Swish, are used.
This problem becomes harder when mixed activation functions and residual connections are considered.
To tackle these technical challenges, we develop the following techniques: a) to handle the non-homogeneous property of Tanh, Sigmoid, and Swish, we develop a new integral estimation approach for the minimal eigenvalue estimation.
b) To establish the connection between the minimum eigenvalues of NTK and generalization errors, we use the Lipschitz continuity to avoid the special property of ReLU.
More importantly, we introduce a new technique [Yaskov, 2014] to replace Gershgorin circle theorem for minimum eigenvalue estimation, which avoids concentration inequalities with negative probability in some certain cases [Nguyen et al., 2021].
Related work
Network architecture search (NAS): The idea of NAS stems from Zoph and Le [2017], while the idea of cell search, i.e., searching core building blocks and composing them together, emerged in
Zoph et al.
The earlier literature used discrete optimization techniques for obtaining the architecture.
DARTS
[Liu et al., 2019b] considers NAS as a continuous bi-level optimization task.
Recent variants of DARTS
[Xu et al., 2019, Wu et al., 2019] and several train-free methods [Mellor et al., 2021, Chen et al., 2021, Xu et al., 2021] have demonstrated success in reducing the search time or improving the search algorithm.
However, the aforementioned works have not provided generalization guarantees for the optimal architecture.
Optimization and generalization of DNNs via NTK:
In the NTK framework [Jacot et al., 2018, Du et al., 2019a, Chen et al., 2020b], the training dynamics of (in)finite-width networks can be exactly characterized by kernel tools.
Leveraging NTK facilitates studies on the global convergence of
GD Allen-Zhu et al.
[2019], Du et al.
[2019a], Nguyen [2021] in DNNs via the minimum eigenvalue of NTK.
In fact, it also controls the generalization performance of DNNs [Du et al., 2019b, Cao and
Gu, 2019, Allen-Zhu et al., 2018], which is further studied in Bietti and Bach [2021].
Problem Settings
In this section we introduce the problem setting of our NAS framework based on the search space and algorithm (search strategy) for our paper.
Let X ⊆
R
d be a compact metric space and Y ⊆ R .
We assume that the training set D tr = { ( x i , y i ) }
Ni =1 is drawn from a probability measure D on X × Y , with its marginal data distribution denoted by D X .
The goal of a supervised learning task is to find a hypothesis (i.e., a neural network used in this work)
f : X → Y such that f ( x ; W ) parameterized by W is a good approximation of the label y ∈ Y corresponding to a new sample x ∈ X .
In this paper, we consider the classification task, evaluated by minimizing the expected risk
[Equation]
where ℓ [ yf ( x ; W )] is the classification loss ℓ ( · ) as a surrogate of the expected 0-1 loss ℓ 0 − 1 D ( W ) :
=
E ( x ,y ) ∼D
[1 { yf ( x ; W ) < 0 } ] .
In this paper, we employ the cross-entropy loss, which is defined as ℓ ( z ) =
log[1 + exp( − z )] .
Notation: For an integer L , we use the shorthand [ L ] = { 1 , 2 , . . .
, L } .
The multivariate standard
Gaussian distribution is N ( 0 , I d ) with the zero-mean vector 0 and the identity-variance matrix I d .
We denote the direct sum by ⊕ .
We follow the standard Bachmann–Landau notation in complexity theory e.g., O , o , Ω , and Θ for order notation.
3.1 Neural Networks and Search Space
In this work, we consider a particular parametrization of f as a deep neural network (DNN) with depth L ( L ≥ 3 ) 2 which includes the fully-connected (FC) neural networks setting and the residual neural networks setting, and various activation functions in each layer.
This enables a quite general
NAS setting.
Formally, we define a single-output DNN with the output f l ( x ) in each layer f l ( x )=

 x l = 0 , σ 1 ( W 1 x ) l = 1 , σ l ( ⟨ W l , f l − 1 ( x ) ⟩ )+ α l − 1 f l − 1 ( x ) 2 ≤ l ≤ L − 1 , ⟨ W L , f L − 1 ( x ) ⟩ l = L , (1) where the weights of the neural networks are W 1 ∈ R m × d , W l ∈ R m × m , l = 2 , . . .
, L − 1 and
W L ∈ R m .
The binary parameter α l is for layer search, and the activation function is σ l ( · ) .
The neural network output is f ( x ; W ) =
f L ( x ) .
Architecture search: A binary vector α =
[ α 1 , α 2 , · · · , α L − 2 ] ⊤ represents the skip connections, where the α l ∈ { 0 , 1 } in Eq.
(1) indicates whether there is a skip connection in the l -th layer.
Notice that we unify FC and residual neural networks under the same framework.
[Table]
ensure f Sigmoid
[Table]
facilitates our theoretical analysis.
The parameter
[Table]
The
[Table]
the Tanh
[Table]
Activation function search: We select five representative activation functions defined by F σ = { ReLU , LeaklyReLU , Sigmoid , Tanh , Swish } used in Eq.
(1), that can be bounded, unbounded, smooth, non-smooth, monotonic, or non-monotonic, as reported in Table 1.
We define σ = [ σ 1 , σ 2 , · · · , σ L − 1 ] ⊤ with σ l ∈ F σ for any l ∈
[ L − 1] as the indicator to show which activation function is selected in each layer.
Our NAS framework allows for a different activation function in each layer, which enlarges the search space.
In our setting, we conduct the architecture search and the skip connection search independently, and accordingly, our search space is defined as the direct sum of them
[Equation] [Equation]
where W :
= ( α , σ , W 1 , . .
, W L ) ∈ W represents the collection of weight matrices and indicator for skips and selected activation functions for all layers.
3.2 Algorithm (Search Strategy)
The search strategy is the core part in NAS to pick up the optimal architecture from the search space.
Here we build a general Algorithm 1 combining the search strategy for NAS (the first part) and the subsequent neural network training by SGD (the second part).
We firstly utilize a typical NAS algorithm, e.g., random search WS [Li and Talwalkar, 2020] or
DARTS 3 , to search skip connections and activation functions independently, which results in the optimal architecture { ( σ ∗ i )
L − 1 i =1 , ( α ∗ i )
L − 2 i =1 } with the max probability, see sec.
5.1 for details.
In particular, Algorithm 1 also allows for the guidance of NAS in a train-free strategy via some specific metrics, e.g., the minimum eigenvalue of NTK (and its variant), see our Eigen-NAS method in sec.
Then, we conduct neural network training on the selected architecture by SGD.
For ease of theoretical analysis, we employ the constant step-size SGD with one epoch and randomly choose the weight parameters during all the iterations, which is commonly used in deep learning theory [Cao and Gu, 2019, Zou et al., 2019].
Main result
In this section, we state the main theoretical results.
We present the assumptions used in our proof in sec. 4.1.
Then in sec.
4.2 we provide the recursive form of NTK for DNNs defined by Eq.
(1) with mixed activation functions and skip connections.
The upper and lower bounds of the minimum eigenvalue of NTK in the infinite and finite-width setting is given in sec.
4.3 and 4.4, respectively.
Finally, in sec. 4.5, we connect the minimum eigenvalue of NTK and the generalization error bound of DNNs under these search schemes.
The proofs of our theoretical results presented in this section are deferred to Appendix B, C, and D, respectively.
4.1 Assumptions
We make the following assumptions on data and activation functions.
Our assumptions are frequently employed in the literature as we highlight below.
Assumption 1.
We assume that the data satisfy ∥ x ∥ 2 = 1 .
This
optimal
if Flag GuideNAS = EigenNAS then
Guide NAS from
Eigen else if Flag GuideNAS =
DARTS then
Search neural network architectures from using the DARTS algorithm.
end if
Output the probability.
// do neural network training via SGD
Gaussian initialization: W (1) l
N (0 , m
, l ∈ [ L ]
Construct the neural network for i = 1 to N do
[Equation] [Equation]
end for
Assumption 2.
The activation function σ : R → R satisfies σ ∈ L 2 ( R , e − x 2 / 2 / √ 2 π ) , where
L 2 ( R , g ) denotes the square integrable function.
Assumption 3.
We further assume that x is isotropic.
i.e. E [ xx ⊤ ] = I d /d , where the coefficient 1 /d is to satisfy Assumption 1 at the same time.
Remark: The first assumption on normalized data is commonly used in practice and theory on overparameterized neural networks [Du et al., 2019b,a, Allen-Zhu et al., 2019, Oymak and Soltanolkotabi, 2020, Malach et al., 2020].
The second assumption is general as the studied activation functions in
Table 1 satisfy it.
The third assumption is standard in statistics and machine learning [Vershynin, 2018,
Hastie et al., 2022, Klimovsky, 2012, Yaskov, 2014].
It covers Gaussian data, and data uniformly spread on the sphere, commonly used in deep learning theory [Mei et al., 2021, Ghosh et al., 2022].
4.2 Recursive NTK for DNNs defined by Eq. (1)
Recall that NTK [Jacot et al., 2018] under the infinite-width setting ( m → ∞ ) is:
[Equation]
where the NTK matrix for residual networks is derived by the following regular chain rule.
Lemma 1.
For any l ∈ [3 , L ] and s ∈ [2 , L ] , denote
[Equation]
Then the NTK for residual networks defined in Eq.
(1) can be written as
[Equation]
Remark: ( i ) Our NTK formula of ResNet differs from the one of Tirer et al.
[2022], Huang et al.
[2020], Belfer et al.
] in two critical ways: 1) each skip-layer in our model skips one fullyconnected layer and one activation function, as opposed to the two-layer skip of previous works, 2) our formulation does not require every layer to have a parallel skip connection, which increases the flexibility of the network.
Those differences also result in a different NTK matrix.
( ii ) Our NTK formulation covers different activation functions, and we adopt the same initialization (coefficient) on them to ensure fair/equal search in our NAS framework .
Lemma 1 covers both FC and residual neural networks, which facilitates the analysis of minimum eigenvalue of NTK under the unifying framework.
If α l = 0 for l ∈ [ L − 1] , our NTK formulation for residual neural networks degenerates to that of a fully-connected neural network, and A l and G l become equal.
4.3 Minimum Eigenvalue of NTK for infinite-width
We are now ready to state the main result on the infinite-width neural network.
We provide the upper and lower bounds of minimum eigenvalue of NTK for infinite-width neural network mixed with five different activation functions.
The main differences between different activation functions are illustrated in Table 1.
Theorem 1.
For a DNN defined by Eq.
(1) and a not very large L , let K ( L ) be the limiting NTK recursively defined in Lemma 1.
Then, under Assumptions 1 and 3, when N ≥ Ω( d 4 ) , with probability at least 1 − e − d , we have
[Equation]
where µ 1 ( σ 1 ) is the 1 -st Hermite coefficient of the first layer activation function, and β 1 , β 2 , β 3 are three constants on various activation functions defined in Table 1.
Remark: A not very large depth, e.g., L ≤ 10 , is often sufficient for the search phase in practical implementations [Liu et al., 2018, Dong et al., 2021].
In addition, existing NAS algorithms such as
DARTS tend to have architectures with wide and shallow cell structure as suggested by Shu et al.
Theorem 1 shows the upper and lower bounds of the minimum eigenvalue of NTK under the mix of activation functions and skip connections.
The following conclusions can be drawn from our result:
The bounds of the minimum eigenvalue depend significantly on the depth of the network L , the skip connections via α p , that makes the minimum eigenvalue increasing fast as L and the number of skip connections increase.
Besides, the minimum eigenvalue is also effected by activation functions via β 1 , β 2 , β 3 .
Nevertheless, the lower bound is independent of β 1 and β 2 . 2.
Different activation functions lead to different tendency (increase or decrease) on λ min ( K ( L ) ) . As
the depth increases, the lower bound λ
min
( K (
L )
under ReLU remains unchanged increases under LeakyReLU, and decreases when Sigmoid, Tanh or Swish applied, which brings in new findings when compared
to
the ReLU-network analysis of Nguyen et al.
For
the upper
bound for λ min ( K ( L ) ) , we can see our results are positively correlated with the depth
L One can see that µ 1 ( σ 1 ) is only related to the activation function of the first layer, which implies that the activation function in the first layer is very important as λ min ( K ( L ) ) largely depends on it.
4.4 Minimum Eigenvalue of NTK for finite-width
To study the finite-width, we firstly introduce the Jacobian of the network.
Let
F =
[ f ( x 1 ) , . .
f ( x N )] T .
Then, the Jacobian J of F with respect to W is J = (cid:104) ∂ F ∂ vec ( W 1 ) , . . .
, ∂ F ∂ vec ( W L ) (cid:105) , where J have dimension R ((( L − 2)
× m
+ d +1)
× m × N .
The empirical Neural
Tangent Kernel (NTK) matrix can be defined as ¯ K ( L ) =
JJ ⊤ =
(cid:80) Ll =1 (cid:104)
[Equation]
[Equation]
(cid:105)
[Equation]
Accordingly, we generalize Theorem 1 from the infinite-width to finite-width setting below.
Theorem 2.
For an L -layer network defined by Eq.
(1) , let K ( L ) =
JJ ⊤ be the NTK matrix, and the weights of the network be initialized as [ W l ] i,j ∼ N (0 , 1 /m ) , for all l ∈ [ L ] .
Under Assumptionsand 3, when N ≥ Ω( d 4 ) , then λ min ( JJ ⊤ ) can be bounded by
[Equation]
where the first inequality (lower bound) holds with probability
at least 1 − e − d − (cid:80) L −l =1 exp( − Ω( m ))
− exp( − Ω(1)) and the second inequality (upper bound) holds with probability at least 1 − (cid:80) L − 1 l =1 exp( − Ω( m ))
− exp( − Ω(1)) .
The definitions of β 2 , and β 3 are the same as those in Theorem 1.
Remark:
Theorem 2 achieves a similar result as Theorem 1 if the width m is large.
4.5 Connection to Generalization Error Bound
Based on the aforementioned upper and lower bounds of the minimum eigenvalue of NTK under different settings, here we establish its relationship with the generalization error of DNNs.
We provide a bound on the expected 0-1 error obtained by Algorithm 1.
Theorem 3.
Given a DNN defined by Eq.
(1) with y =
( y 1 , . . .
, y N ) ⊤ determined by Algorithmwith the step size of SGD γ
= κC 1 · (cid:112)
y ⊤
K ( L ) ) − 1 y / ( m √ N ) for some small enough absolute constant κ .
Under Assumptions 1, 2 and 3, for any δ ∈ (cid:0) 0 , e − 1 (cid:3) and a not very large L , if the width m ≥ ˆ m , where ˆ m depends on λ min ( K ( L ) ) , δ, N , and L , then with probability at least 1 − δ over the randomness of W (1) , we obtain the following high probability bound:
[Equation]
where C 1 = √ L/ (3Lip max + 1) L − 1 and C 2 = √ L (3Lip max + 1) L − 1 are two constants depending only on L and Lip max is the maximum value of the Lipschitz constants of the all activation functions.
Remark:
Theorem 3 gives an algorithm-dependent generalization error bound of DNNs defined by Eq.
(1) trained with SGD with different activation functions and skip connections.
If m is large enough, the learning rate is infinitesimal, which means the generalization error bound mainly depends on the NTK matrix, similarly to Cao and Gu [2019], Du et al.
[2019a].
Admittedly, our result is in an exponential increasing order of the depth.
However, in practice, the depth L during the search phrase is smaller than 20 , or even 10 [Liu et al., 2018, Dong et al., 2021].
Under this setting our results are better than the previous results, which are discussed in details in Appendix E.
According to Theorem 3, the generalization performance of DNNs is controlled by the minimum eigenvalue of the NTK matrix, which is in turn affected by different activation functions and skip connections, as discussed in Theorem 1.
Apart from the NTK matrix itself, the condition m ≥ ˆ m is also effected by different activation functions, which implies that the required minimum width is different in these cases.
4.6 Proof sketch
Our work extends the proofs of Nguyen and Mondelli [2020], Cao and Gu [2019] beyond ReLU, which is critical for enabling search across activations.
The extension to other activation functions and skip connections is non-trivial due to non-linearity, inhomogeneity and nonmonotonicity.
To derive the upper and lower bounds on the minimum eigenvalue, we start from Lemma 1 on the
NTK formula under the mixed activation functions and skip connections, and we transform the minimum eigenvalue estimation to the computation (estimation) of the bound G , ˙
G ( λ min ( G ) ).
The infinite-width and finite-width are included in Appendix B and C respectively.
For the upper bound, we estimate the diagonal elements of G and use the property that the minimum eigenvalue is less than the mean of the diagonal elements of a matrix to proof.
For the lower bound, we use Hermite expansion and [Yaskov, 2014, Corollary 3.1].
Combining these results concludes the proof.
To derive the generalization error bounds, we need a series of lemmas (see Appendix D).
If the input weights are close, the output of each neuron with any activation function does not change too much (see Lemma 7).
If the initilizations are close, the neural network output f ( x ; W ) is almost linear in W (see Lemma 8), and the loss function ℓ [ y i f ( x i ; W )] is almost a convex function of W for any
i ∈
[ N ] (see Lemma 9).
Accordingly, the gradient and loss of the neural network can be upper bounded by Lemmas 10 and 11, respectively, which concludes the proof when combined with some relevant results [Cao and Gu, 2019, Allen-Zhu et al., 2019].
Further discussion on the differences is deferred to Appendix E.
          /D\HU 5 H/  7 DQK 6 L J P R L G /HD N\ 5 H/  6 Z L V K $ F W L Y D W L RQ I XQ F W L RQ              
      
                      
               (a) activation functions σ       9 10 Layer 0. P r o b a b ili t y [Figure]  Numerical Validation
To validate our theoretical results, we conduct a series of experiments on NAS. Firstly, we simulate the NTK matrices under different depths in Appendix F.4 to verify the relationship between the minimum eigenvalue of NTK and the network depth L in Theorem 1.
In sec.
5.1 we use the DARTS algorithm
[Liu et al., 2019b] to conduct experiments on activation function search and skip connection search under the search space of Eq. Finally, we use the minimum eigenvalue of NTK to guide the training of NAS on the benchmark NAS-Bench-
[Dong and Yang, 2020], with a comparison of recent NAS algorithms.
Additional experiments on NAS-Bench-101 [Ying et al., 2019] and transfer learning are deferred to Appendix F.5 and F.6.
5.1 DARTS experiment
In this section we employ a typical NAS algorithm, DARTS Liu et al.
[2019b], to assess our theoretical results on activation functions and skip connections.
We select Fashion-MNIST
[Xiao et al., 2017] as a standard benchmark.
Details about Fashion-MNIST are shared in Appendix F.1.
Search space and search strategy:
Our search space is defined by Eq.
(2) on skip connections, activation functions, and weight parameters.
We follow the search strategy of Liu et al.
[2019b] in a two-level scheme, one level is for weight parameter search W and the other level is for architecture search { α , σ } , which results in the final optimal architecture { α ∗ , σ ∗ , W ∗ } .
Different from Liu et al.
[2019b], the activation function search and the skip connection search in our setting is independent.
To obtain σ ∗ , we use the softmax function to normalize the weights and choose the specific activation function with the highest probability in each layer.
To obtain α ∗ , we initialize each entry α l = 1 /( l ∈
[ L − 2] ), constrain it to [0 , 1] during training, and retain the skip connection when α ∗ l > 1 / 2 .
NAS Results: We conduct the experiment via DARTS on a feedforward neural network with L =and m = 1024 , with 5 runs.
After training, the probability of these activation functions and skip connections in each layer is reported in Figure 1(a) and 1(b), respectively.
We have the following findings: Firstly, after the search process, LeakyReLU and ReLU are selected as the activations with the highest probability in each layer.
This coincides with our theoretical results in Theorem 1.
One minor difference is that the probability of LeakyReLU is slightly inferior to ReLU in practice.
The reason behind this could be the sparsity of ReLU [de Dios and Bruna, 2020].
Secondly, in the first layer, we observe the largest difference on the probability of various activation functions.
As the network becomes deeper, such difference pales in importance until there is no difference among these activation functions in the last layer.
This phenomenon matches our theory well.
To be specific, in Theorem 1, our result on the minimum eigenvalue largely depend on the first layer and its Hermite coefficient.
Besides, this result also provides a justification on omitting the high-order terms while retaining the first layer activation terms.
Thirdly, for the skip search result, we find that the skip connections are required in each layer when L ≤ 10 , as suggested by our theoretical results in Theorem 1.
It also verifies the results of Zhou et al.
We expect that the skip connections might not be required in each layer for deep neural networks, since their capacity can already be enough [He et al., 2016]; but we defer the related study to a future work.
[Table]
[Table]
Interestingly, the search strategy favors the activation functions and the skip connections with larger minimum eigenvalue of NTK, which enjoy better generalization performance.
This result also motivates us to study the following question: can the minimum eigenvalue of NTK guide the search process in NAS?
We provide an affirmative answer in the next section with experimental validations.
5.2 NAS-Bench-201 Experiment
In this experiment, we use the minimum eigenvalue to guide NAS on NAS-Bench-
[Dong and
Yang, 2020].
Each experiment is repeated 5 times, while it can run on a single GPU in a few hours.
Benchmark and baselines: NAS-Bench-201 [Dong and Yang, 2020] is a commonly used benchmark for NAS algorithm evaluation, which includes three datasets: a) CIFAR-
 [Krizhevsky et al., 2014], b) CIFAR-100 [Krizhevsky et al., 2014] and c)
ImageNet-16 [Chrabaszcz et al., 2017] for image classification.
Details on the datasets exist in Appendix F.1.
Apart from that, we evaluate the proposed approach with some baselines including ResNet, DARTS, RL based algorithm and some train-free algorithms.
Algorithm procedure:
Our algorithm, called Eigen-NAS, also belongs in the train-free category.
Eigen-NAS follows KNAS, which leverages the minimum eigenvalue of NTK to guide NAS.
However, due to the O ( N 3 ) time complexity of computing these eigenvalues, KNAS instead computes ∥ K ∥ F .
However, from the expression λ min ( K ) ≤ 1 d (cid:80) Ni =1 K ii ≤ ∥ K ∥ F we utilize the first inequality in Eigen-NAS to obtain a tighter (and more computationally efficient) bound to λ min .
The computation cost of our method is O ( N ) , which is less than computing the Frobenius norm ( O ( N 2 ) ).
Sequentially, the top- k best candidates architectures are chosen in KNAS and our
Eigen-NAS, and then the best architecture is chosen by the validation error.
Please refer to the results in Table 2.
Due to the page limit, the algorithm is located in Appendix F.
Results: The experimental results in Table 2 verify that Eigen-NAS guided by the proposed metric above achieves the best performance on both the CIFAR-100 and ImageNet-16 datasets, and competitive performance on CIFAR-10, outperforming KNAS in all three cases when k = 20 for both methods.
Even when we consider a smaller k = 5 , Eigen-NAS can outperform KNAS, which we attribute to the more precise minimum eigenvalue estimation.
Conclusion
In this work, we explore the relationship between the minimum eigenvalue of NTK and neural architecture search.
We derive upper and lower bounds on the minimum eigenvalues of NTK for (in)finite residual networks under different mixtures of activation functions, and establish a connection between the minimum eigenvalues and the generalization properties of the special search space: activation function and skip connection search of NAS.
Our theoretical results on various activation functions and mixed activation cases can also be a tool for deep learning theory researchers to prove generic results rather than studying a single architecture, e.g., ReLU networks.
In addition, we use the minimum eigenvalue as a guide for the training of NAS in a train-free method, which greatly exceeds the efficiency of the classic NAS algorithm.
When compared with existing train-free methods, our algorithm, called Eigen-NAS, achieves a higher accuracy.
We posit that this will be useful for studying computationally efficient methods on NAS.
A core limitation is whether our proof framework can cover more general structures in NAS, such as the most commonly used convolutional neural networks (CNNs).
Even though this seems possible, this is non-trivial due to the tensors that emerge.
To be specific, it requires the element-recursive form of NTK matrices in Arora et al.
[2019b] to be transformed into a global-recursive form (similar to Lemma 1), then analyze its minimum eigenvalue.
Besides, the contraction operation of tensors, the locality and boundary effects of convolutional layer in CNNs make the analysis difficult.
Therefore, we believe this is a topic on its own right.
Another limitation of our work is that it does not analyze the various algorithms proposed for searching through the search space.
We believe that a deeper understanding of such algorithms, such as DARTS can provide further insights into how to design improved search spaces.
Acknowledgements
We are also thankful to the reviewers for providing constructive feedback.
Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-19-1-0404.
This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043).
This work was supported by SNF project – Deep Optimisation of the Swiss National
Science Foundation (SNSF) under grant number 200021 205011.
This work was supported by Zeiss.
This project has received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement n° 725594 - time-data).
Corresponding authors: Fanghui Liu and Zhenyu Zhu.
Appendix introduction
The Appendix is organized as follows:
• In Appendix A, we state the introductory notations and definitions.
• We prove Theorem 1 in Appendix B.
We also provide the result when the residual network only has the same activation function.
• In Appendix C, we extend the results of infinitely width to finite-width and provide the proof for them.
• In Appendix D, we prove Theorem 3.
• In Appendix E, we discussion some key points of the proof and the motivation of the analysis.
• In Appendix F, we detail our experimental settings, our Eigen-NAS algorithm as used in sec.
We conduct additional numerical validations.
• Finally, in Appendix G, we discuss the societal impact of this work.
A Background A.1 Symbols and Notation
In the paper, vectors are indicated with bold small letters, matrices with bold capital letters.
To facilitate the understanding of our work, we include the some core symbols and notation in Table
[Table]
[Table]
defined in Table
[Table]
A.1.1 Feature map
Here we define the core notation about feature maps that are required in the proof.
Firstly
[Table]
we define ω -neighborhood to describe the difference between two matrices
[Table]
For any W ∈ W , we define its ω -neighborhood as follows: Definition 1 ( ω -neighborhood) .
[Equation]
Then we define ( D l ) k,k
= σ l ′ (( W l f l − 1 ) k ) as the back-propagation matrix of the activation function.
We use the notation (cid:102) W ∈ B ( W , ω ) to describe the relationship of the two matrices have ω -neighborhood relationship.
In addition, we define the feature map of network and its perturbing matrix as follows:
Definition 2.
[Equation]
i
[Equation]
for i ∈ [ N ] and l
[Equation]
[Equation]
L
[Equation]
[Equation]
Let us define diagonal matrices (cid:101)
D i,l ∈ R m × m and D i,l ∈ R m × m by letting ( (cid:101)
D i,l ) k,k
= σ l ′
( (cid:101) g i,l ) k ) and ( D i,l ) k,k = σ l ′
(( g i,l ) k ) , ∀ k ∈ [ m ] . Accordingly, we let ˆ g i,l = (cid:101) g
i,l − g i,l , ˆ f i,l = (cid:101) f i,l − f i,l and diagonal matrix ˆ D i,l = (cid:101) D i,l − D i,l .
A.1.2 Other notations
For the Hadamard product of the matrices X 1 , X 2 , · · · , X r that share the same dimensions, we use the following abbreviation:
[Equation]
B The bound of the minimum eigenvalues of NTK for infinite-width
We present the details of our results on sec.
4.3 in this section.
Firstly, we provide the proof of Theorem 1 in Appendix B.2.
Then in Appendix B.3 we provide the result when several activation functions exist alone.
B.1 Proof of Lemma
Our proof mainly follows the results of Huang et al.
[2020], but due to the different network structures, the proof process and results are slightly different.
Moreover, we provide a matrix version results, which Huang et al.
[2020] does not contain.
For self-completeness, we include the proof here.
Proof.
By Huang et al.
[2020, Proposition 3], written as matrix form,we have:
[Equation]
Note that, it is slightly different from the original result because the network structure is slightly different.
Let G (1) =
A (1) , G (2) =
A (2) and G ( l ) = 2 E w ∼N ( 0 , I N ) [ σ l − 1 ( √ A ( l − 1) w ) σ l − 1 ( √ A ( l − 1) w ) ⊤ ] , then we have:
[Equation]
According to Huang et al.
[2020, Proposition 4], written as matrix form,we have:
[Equation]
where the ˙ G ( s ) satisfy that ˙
G ( s ) =
2 E w ∼N ( 0 , I N ) [ σ
[Equation]
s
[Equation]
( √ A ( s − 1) w ) σ ′ s − 1 ( √ A ( s − 1) w ) ⊤ ] .
Combining the above results, we finish the proof.
B.2 Proof of Theorem
In this part, we present the proof of Theorem 1.
Differently from Oymak and Soltanolkotabi [2020], our result allows for activation functions search in each layer.
Before we prove Theorem 1, we provide some propositions that are helpful to our proof.
To facilitate the writing of the proof, let α 0 := 0 .
Proposition 1.
When σ 1 is Tanh, the remaining layers are with LeakyReLU and for l ∈ [ L − 2] , α l = 1 , the quantity G ( l ) ii has the largest upper bound:
[Equation]
We set G max
= 2(2 + η 2 )
L − 2 as the upper bound of G ( L ) ii .
Proof.
To prove our result, we need bound G ( l ) under different activation functions.
We summarize them as below.
When σ l − 1 is ReLU:
[Equation]
When σ l − 1 is LeakyReLU:
[Equation]
When σ l − 1 is Sigmoid:
[Equation]
σ l
[Table]
ReLU LeakyReLU
Sigmoid
Tanh
Swish
[Table]
bound
[Equation]
[Equation]
Lower bound
[Equation]
l
When σ l − 1 is Tanh:
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table]
[Equation]
[Table]
[Equation]
= 2 .
When σ l − 1 is Swish:
[Equation]
[Equation]
Combining Eqs.
(4), (5), (6), (7) and (8) with Lemma 1 we draw the conclusion and finish the proof.
Proposition 2.
The relationship between A ( l ) ii and A ( l − 1) ii for different activation functions can be summarized as Table 4 according to the difference of σ l − 1 .
Proof.
To prove our result, we need to bound the ratio A ( l ) ii /A
( l − 1) ii for different activation functions.
We illustrate how this is achieved in different cases below:
For l ≥ 2 :
When σ l − 1 is ReLU by Eq.
(4) we have:
[Equation]
When σ l − 1 is LeakyReLU by Eq.
(5) we have:
[Equation]
When σ l − 1 is Swish, G ( l ) ii can be upper by Eq.
(8) and lower bounded by:
[Equation]
which implies:
[Equation]
[Equation]
When σ l − 1 is Sigmoid
[Equation]
G ( l ) ii can be upper by:
[Equation]
, holds for x ≥
[Equation]
Then G ( l ) ii can be lower bounded by:
[Equation]
where we use the fact that the penultimate line is a concave function with respect to A ( l − 1) ii . When
A ( l − 1) ii = 0 , the function value is 0.
That means G ( l ) ii /A
( l − 1) ii obtains the minimum value at G ( l ) ii =
G max .
Combined with Eq.
(3), we get the last inequality.
Then, we have:
[Equation]
σ l −
ReLU LeakyReLU
Sigmoid
Tanh
Swish
[Table]
1 + η1 / 1 .
[Table]
1 + ηf S ( G max ) f T ( G
[Equation]
When σ l − 1 is Tanh, G ( l ) ii can be upper bounded by:
[Equation]
[Equation]
[Equation]
The lower bound is:
[Equation]
Similar to the Sigmoid, the penultimate line is an concave function with respect to A ( l − 1) ii .
When
A ( l − 1) ii = 0 , the function value is 0.
That means G ( l ) ii /A
( l − 1) ii obtains the minimum value at G ( l ) ii =
G max .
Combined with Eq. (3), we get the last inequality.
Then, we have:
[Equation]
According to Eqs.
(9), (10), (12), (15) and (18), we can summarized the results about bound of
A ( l ) ii /A ( l − 1) ii in Table 4.
Proposition 3.
The bound of ˙ G ( l ) ii with respect to different activation function σ l − 1 can be summarized in Table 5.
Proof.
To prove our result, we need to bound ˙
G ( l ) ii with respect to different activation function σ l −as follows.
When σ l − 1 is ReLU:
[Equation]
= 1 .
[Equation]
When σ l − 1 is LeakyReLU:
[Equation]
When σ l − 1 is Sigmoid
[Equation]
according to the monotonicity of the f S
[Equation]
we have:
[Equation]
When σ l − 1 is Tanh, according to the monotonicity of the f T
[Equation]
we have:
[Equation]
When σ l − 1 is Swish
[Equation]
The quantity ˙ G ( l ) ii can be upper bounded by:
[Equation]
where the last inequality holds by 1 . 099 < sup x f ′ Swish ( x ) < 1 . 1 and − 0 .
1 < inf x f ′ Swish
( x ) < − 0 .
Then the quantity ˙ G ( l ) ii can be lower bounded by:
[Equation]
Combining Eqs.
(19), (20), (21), (22), (23) to (24), we can summarize the results about bound of ˙ G ( l ) ii in Table 5.
Proposition 4.
Let X ∈ R N ×
d be an matrix that every row x i sampled from data distribution D X .
when N ≥ Ω( d 4 ) , with probability at least 1 − e − d , we have
[Equation]
Proof.
Firstly, we can compute that, for any α ∈ (0 , 2] ,
[Equation]
Then from Assumption 3, we have √ d x i is an isotropic random vector in R d .
According to Yaskov [2014, Corollary 3.1], if we choose α = 1 , then with probability at least 1 − e − d we have:
[Equation]
That is, with probability at least 1 − e − d we have:
[Equation]
Now we are ready to prove Theorem 1.
Proof of Theorem 1.
Now we are ready to present the estimation on λ min ( K ( L ) ) as below.
From Lemma 1, we have the NTK formulation for residual neural networks:
[Equation]
G ( L ) + α L −
[Equation]
1 N × N ) .
It is clear that all the matrices G ( l ) , ˙ G ( l ) are positive semi-definite (PSD), then ( ˙ G ( l +1)
+ α l 1 N × N ) are also PSD.
For two PSD matrices P , Q ∈ R N × N , it holds λ min ( P ◦ Q ) ≥ λ min ( P ) min i ∈ [ N ] Q ii [Schur, 1911] .
Accordingly, we have:
[Equation]
Then we bound λ min ( G (2) ) :
[Equation]
[Nguyen et al., 2021, Lemma D.3]
[Equation]
As a reminder, the symbol ⃝ denotes the Hadamard product, which is defined in Appendix A.1.2.
We know that when N > d , XX ⊤ and X ⊤ X have the same non-zero eigenvalues.
Then according to Proposition 4, with probability at least 1 − e − d we have:
[Equation]
That means, with probability at least 1 − e −
d
we have the lower bound of λ min ( K ( L ) ) :
[Equation]
According to Table 1 and Table
[Equation]
we have:
[Equation]
According to the previous analysis
[Equation]
we know that K ( L ) has d non-zero eigenvalues.
Their sum is equal to the trace of K ( L ) .
The upper bound of λ min ( K ( L ) ) is directly given by:
[Equation]
Final result - upper bound
For G ( l ) ii we have the following bound:
[Equation]
By Eqs.
(26), (28) and (29), we have:
[Equation]
Final result - lower bound
By Eq.
(25) and Eq.
(27), with probability at least 1 − e − d , we have:
[Equation]
B.3 Special cases
To provide further insights into our proofs of mixed activation functions as provided in the previous sections
[Equation]
we now consider the special case of a single activation function in each layer
[Equation]
Corollary 4.
Under Assumption 1, 3, for a deep fully-connected ResNet with the same activation functions in every layer and for a not very large L
[Equation]
let K ( L ) be the limiting NTK recursively defined in Lemma 1.
Then, with probability at least 1 − e − d , we have:
For ReLU:
[Equation]
For LeakyReLU:
[Equation]
For Sigmoid:
[Equation]
For Tanh:
[Equation]
For Swish:
[Equation]
The µ 1 ( σ 1 ) is 1 -st Hermite coefficient of the activation function.
Proof.
By, Table 1, Eqs.
(25) and (30).
we can have this result.
It should be noted that for Sigmoid network (all of activation functions are Sigmoid) and Tanh (all of activation functions are Tanh) network , the upper bound of G max will change.
By Eq.
(6) and Eq.
(7) we have for Sigmoid G max = 1 2 , For Tanh G max = 2 .
That means f S ( G max ) in the Theorem 1 is replaced by f S ( 12 ) in Eq.
(31) and f T ( G max ) in the Theorem 1 is replaced by f T (2) in Eq.
C The bound of the minimum eigenvalues of NTK for finite-width
We present the details of our results on sec.
4.4 in this section.
Firstly, we introduce the specific expression form for NTK of finite-width network in Appendix C.1.
Then, we introduce some lemmas in Appendix C.2 to facilitate the proof of theorems, after that we provide the results of multiple activation functions are mixed in one network in Appendix C.3 directly, finally we discuss the results.
C.1 Neural Tangent Kernel for finite-width
[Equation]
∂ F ∂ vec( W l ) (cid:21) ⊤ .
Let F k =
[ f k ( x 1 ) , . . .
, f k ( x N )] T , by chain rule and some standard calculation, we have,
[Equation]
where B k ∈ R N × m is a matrix of which the i -th row is given by
[Equation]
C.2 Relevant Lemmas
Lemma 2.
Fix any k ∈ [0 , L − 1] and x ∼ P X , then for ReLU, LeakyReLU, Sigmoid, Tanh and
Swish we have
[Equation]
with probability at least 1 − (cid:80) kl =1 exp( − Ω( m )) over ( W l ) kl =1 and x .
Moreover,
[Equation]
with probability at least 1 −
(cid:80)
k
−
l =
1 exp
− Ω m
) over
( W l ) kl =1 .
Proof.
We prove this by induction.
The result holds for k = 0 due to Assumption 1 and Assumption 3.
Assume that the lemma holds for some k − 1 , i.e.
[Equation]
, with probability at least 1 − (cid:80) k − 1 l =1 exp( − Ω( m )) over ( W l ) kl =1 and x .
Let us condition on this event of ( W l ) k − 1 l =1 and study probability bounds over W k : Let W k =
[ w 1 , · · · , w m ] ⊤ where w j ∼ N (0 , I m /m ) and f [ j ] k represents the j -th element of f k .
Note that:
[Equation]
Then we have:
[Equation]
[Equation]
According to Eqs.
(4), (5), (8), (11), (13), (14), (16) and (17)
[Equation]
we know that when σ k − 1 are in ReLU,
LeakyReLU, Sigmoid, Tanh and Swish we have:
[Equation]
When σ k − 1 is ReLU, LeakyReLU or Swish, Eq.
(35) can be written as:
[Equation] According to the relationship between the vectors 1 -norm and 2 -norm
[Equation]
we have:
[Equation]
Then:
[Equation]
If we substitute into Eq.
(34), we have upper bound and lower bound for E W k ∥ f k ( x ) ∥ 22 :
[Equation]
[Equation]
That means, when σ k − 1 is ReLU, LeakyReLU or Swish we have:
[Equation]
When σ k − 1 is Sigmoid or Tanh, according to symmetry we have:
[Equation]
Then:
[Equation]
By Eq.
(36) and Eq.
(37) when σ k − 1 is ReLU, LeakyReLU, Sigmoid, Tanh or Swish we have:
[Equation]
Thus, by applying Bernstein’s inequality to the sum of i.i.d.
random variables in Eq.
(33), we have:
[Equation]
with probability at least 1 − exp( − Ω( m )) .
i.e.:
[Equation]
with probability at least 1 − (cid:80) kl =1 exp( − Ω( m )) .
The proof for E x ∥ f k ( x ) ∥ 2 2 can be done by following similar passages and using that
(cid:13)(cid:13)(cid:13) E x
[ f [ j ] k ( x ) 2 ] (cid:13)(cid:13)(cid:13) ψ 1 ≤ E x (cid:13)(cid:13)(cid:13) f
[ j ] k ( x ) 2 (cid:13)(cid:13)(cid:13) ψ 1 .
Lemma 3.
Fix any layer k ∈
[ L − 1] , and x ∼ P X .
Then, we have that ∥
D k ∥ 2F = Θ( m ) with probability at least 1 − (cid:80) kl =1 exp( − Ω( m )) over ( W l ) kl =1 and x .
Proof.
By Lemma 2, we have f k − 1 ( x ) ̸ = 0 with probability at least 1 − (cid:80) kl
=1 exp( − Ω( m )) over ( W l ) kl =1 and x .
Let us condition on this event and derive probability bounds over W k .
Let W k
=
[ w 1 , · · · , w n k ] .
Then, ∥ D k ∥ 2F = (cid:80) mj
=1 σ ′ k 2 ( ⟨ f k − 1 ( x ) , w j ⟩ ) .
Thus:
[Equation]
[Equation]
By Eqs.
(19), (20), (21), (22), (23) and (24), we know that when σ k are in ReLU, LeakyReLU,
Sigmoid, Tanh and Swish we have:
[Equation]
By Hoeffding’s inequality on bounded random variables, we have:
[Equation]
Picking t := 0 .
01 m concludes the proof.
Lemma 4. For any k ∈ [ L − 1] , k ≤ p ≤ L − 1 and x ∼ P X , we have that: Θ
[Equation]
m
[Equation]
with probability at least 1 − (cid:80) pl = k +1 exp( − Ω( m )) over ( W l )
pl = k +1 and x .
Proof.
We prove this by induction on p .
Lemma 3 implies that the statement holds for p = k .
Suppose it holds for some p − 1 .
Let S p
=
D k (cid:81) pl = k +
( W l D l + α l − 1 I m × m ) .
Then, S p =
S p − 1 ( W p D p + α p − 1 I m × m )
=
S p − 1 W p D
p
+ α p − 1 S p − 1 .
Let W p =
[ w 1 , . . .
, w n p ] .
Then: ∥
[Equation]
Then we have:
[Equation]
From the previous result Eqs.
[Equation]
[Equation]
[Equation]
(23) and (24) we have:
[Equation]
That is:
[Equation]
Moreover:
[Equation]
By Bernstein’s inequality [Vershynin
[Equation] [Equation]
we have:
[Equation]
with probability at least 1 − exp( − Ω( m )) .
Finally, taking the intersection of all the events finishes the proof.
Lemma 5.
For any layer k ∈ [ L − 2] and x ∼ P X , we have:
[Equation]
with probability at least 1 − (cid:80)
[Equation] 1 l = k +1 exp( − Ω( m ))
− exp( − Ω(1)) .
Proof.
Let B = D k (cid:81)
L − 1 l = k +
( W l D l + α l − 1 I m × m ) .
By Lemma 4, we have:
[Equation]
with probability at least 1 − (cid:80) L
[Equation]
[Equation]
= k +1 exp( − Ω( m )) .
Then, by Hanson-Wright inequality [Vershynin, 2018], we have:
[Equation]
with probability at least 1 − exp( − Ω( ∥
[Equation]
∥ 2F / ∥ B ∥ 22 )) ≥ 1 − exp( − Ω(1)) over W L .
According to Eq.
(38) and Eq.
(39) we can get the result. C.3 Results for mixed activation functions under the finite-width setting (Proof of Theorem 2)
Proof.
We firstly present the lower bound of the minimal eigenvalue of JJ ⊤ and then derive its upper bound.
Lower bound
For PSD matrices P , Q ∈ R N × N , it holds λ min ( P ◦ Q ) ≥ λ min ( P ) min i ∈ [ N ] Q ii .
Then, by Theorem 1 and Theorem 5.1 of Nguyen et al.
[Equation]
(cid:19)
[Equation]
with probability at least 1 − e − d − (cid:80) L − 1 l =1 exp( − Ω( m )) − exp( − Ω(1)) .
where the last inequality hold by Lemma 5 and Proposition 4.
Upper bound
For ReLU, LeakyReLU
and Swish we have:
[Equation]
By Lemma 2 and Lemma 5 we have:
[Equation]
L −
[Equation]
(cid:88) k
= Θ (cid:18)
L − 1 (cid:89) i
= k
+ ( β 2 ( σ i )
+ α
i − 1 ) (cid:19) , with probability at least 1 − (cid:80) L − 1 l =1 exp( − Ω( m ))
− exp( − Ω(1)) .
D Generalization error via the minimum eigenvalue of NTK
In this section, firstly, we provide some useful lemmas in Appendix D.1,then present the proof of Theorem 3 in Appendix D.2.
D.1 Relevant Lemmas
Lemma 6.
( Vershynin [2018, Theorem 4.4.5]) Let A be an N × n matrix whose entries are independent standard normal random variables.
Then for every t ≥ 0 , with probability at least 1 − 2 exp( − t 2 / 2) , one has: s ( A ) max
[Equation]
We need the following lemma to show that the output of each neuron with any activation function does not change too much if the input weights are close.
Lemma 7.
Let W ∈ R m × m be the random Gaussian matrix with W i,j ∼ N (0 , 1 /m ) , Lip max be the maximum value of the Lipschitz constants of the all activation functions, with ω
=
O ((3Lip max + 1) − ( L − 1) ) , assuming (cid:102) W ∈ B ( W , ω ) , for any l ∈ [ L ] , it holds that (cid:13)(cid:13)(cid:13)
ˆ f i,l (cid:13)(cid:13)(cid:13)
= O (1) with probability at least 1 − 2 l exp( − m/ 2) − l exp( − Ω( m )) .
Proof.
We provide the estimation on ˆ f i, 1 and ˆ f i,l ( l = 2 , 3 , · · · , L ) in Definition 2, respectively.
Firstly, ˆ f i, 1 admits: (cid:13)(cid:13)(cid:13)
[Equation]
= O (1)
[Equation]
For ˆ f i,l with l = 2 , 3 , . . .
, L
[Equation]
we have:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
By Lemma 6, choosing t = √ m , with probability at least 1 − 2 exp( − m/ 2) , we have:
[Equation]
Then, ∥ ˆ f i,l ∥ 2 in Eq.
(40) can be further upper bounded with probability at least 1 − 2 l exp( − m/ 2) − l exp( − Ω( m )) :
[Equation]
where the second inequality holds by the recursion which conclude the proof.
We also need the following lemma, demonstrating that the neural network function is almost linear in terms of its weights if the initializations are close to each other.
Lemma 8.
Let W , W ′ ∈ B ( W (0) , ω ) with ω
=
O ((3Lip max + 1) − ( L − 1) ) , for any i ∈ [ N ] , with probability at least 1 −
( L − 1) exp( − m/ 2) − L exp( − Ω( m )) − 2 /m , we have:
[Equation]
⟩| = O (1) .
Proof.
We have the following expression:
[Equation]
Here we require the derivative of the activation function σ ′ is bound, i.e., ∥ D ∥ 2 ≤ Lip max .
The considered activation functions in this paper satisfy this condition.
By Lemma 6, Lemma 7 and Lemma 2 with probability at least 1 − 2( L − 1) exp( − m/ 2) −
L exp( − Ω( m )) , we have (cid:13)(cid:13) f
′ i,L − 1 − f i,L − 1 (cid:13)(cid:13) 2 ≤ O (1) , ∥ f
i,l − 1 ∥ 2 = Θ(1) and ∥ W r ∥ 2 ≤ 3 ∀ r ∈
[ L − 1] .
Moreover, m ∥ W L ∥ 2 2 is a random Variables obey chi-square distribution with m degrees of freedom.
That means E ( m ∥ W L ∥ 22 ) =
m
and V ( m ∥ W L ∥ 22 ) = 2 m .
By Chebyshev’s Inequality we have
P ( | m ∥ W L ∥ 22 − m | ≥ m ) ≤ 2 m/m 2 .
i.e.:
[Equation]
with probability at least 1 − 2 /m .
Accordingly, Equation (41) can be further upper bounded by:
[Equation]
We define L i ( W ) =
ℓ [ y i f
( x i ; W )] , then the following lemma shows that, L i ( W ) is almost a convex function of W for any i ∈
[ N ] if the initilizations are close.
Lemma 9.
Let W , W ′ ∈ B ( W (0) , ω ) with ω
=
O ((3Lip max + 1) − ( L − 1) ) , for any i ∈ [ N ] , it holds that:
[Equation]
with probability at least 1 − 2( L − 1) exp( − m/ 2) − L exp( − Ω( m )) − 2 /m .
Proof.
By the convexity of ℓ ( z ) , we have:
[Equation]
Using the chain rule leads to:
[Equation]
Combining the above two equations, by triangle inequality, we have:
[Equation]
where ε := | ℓ ′
[ y i f ( x i ; W )] · y i · [ f ( x i ; W ′ )
− f ( x i ; W ) − ⟨∇ f
( x i ; W ) , W ′ − W ⟩ ] | .
Then by upper-bounding ε with Lemma 8 and the fact that | ℓ ′
[ y i f ( x i ; W )] · y i | ≤ 1 , we have:
[Equation]
We need the following lemma to show that, the gradient of the neural network function can be upper bounded under near initialization.
Lemma 10.
Let W ∈ B ( W (0) , ω ) with ω
=
O ((3Lip max + 1) − ( L − 1) ) , for any i ∈ [ N ] , with probability at least 1 − 2( L − l ) exp( − m/ 2) − l exp( − Ω( m )) − 2 /m , it holds that:
[Equation]
Proof.
According to the triangle inequality and definition of operator norm, we have:
[Equation]
L
[Equation]
L
[Equation]
L
[Equation]
By Lemma 2 and Lemma 6, with probability at least 1 − 2( L − l − 1) exp( − m/ 2) − l exp( − Ω( m ))
− 2 /m
we have ∥ f i,l − 1 ∥ 2 = Θ(1) , (cid:13)(cid:13)(cid:13) W (0) i (cid:13)(cid:13)(cid:13) 2 ≤ 3 for i =
l + 1 , · · · , L − 1 , (cid:13)(cid:13)(cid:13) W (0) L (cid:13)(cid:13)(cid:13) 2 ≤ √ 2 and ∥ D ∥ 2 ≤ Lip max due to σ ′ is bounded, then we have:
[Equation]
which implies: ∥∇
[Equation]
where we use the fact that | ℓ ′
[ y i f ( x i ; W )] · y i | ≤ 1 .
We need the following lemma to show that, the cumulative loss can be upper bounded under small changes on the parameters (i.e., weights).
Lemma 11.
For any ϵ, δ, R > 0 , there exists:
[Equation]
such that if m ≥ m ∗ ( ϵ, δ, R, L ) , then with probability at least 1 − δ over the randomness of
W (1) , for any W ∗
∈ B ( W (1) , Rm − 1 / 2 ) , Algorithm 1 with γ = ε/ [ m (3Lip max + 1) 2 L − 2 ] , N =
LR 2 (3Lip max + 1) 2 L − 2 / (2 ε 2 ) , the cumulative loss can be upper bounded by:
[Equation]
Remark: Discussion on the required width m refer to Appendix E.
Proof.
Set ω
= 1 / (3Lip max + 1) L − 1 such that the conditions on ω given in Lemmas 7, 8, 9 andhold.
It is easy to see that as long as m ≥ R 2 (3Lip max + 1) 2 L − 2 , we have W ∗ ∈ B ( W (1) , ω ) .
We now show that under our parameter choice, W (1) , . . .
, W ( N ) are inside B ( W (1) , ω ) as well.
This result follows by simple induction.
Clearly we have W (1) ∈ B ( W (1) , ω ) .
Suppose that
W (1) , . . .
, W ( i )
∈ B ( W (1) , ω ) .
Then by Lemma 10, for l ∈ [ L ] , we have ∥∇ W l L
i ( W ( i ) )
∥ 2 ≤ Θ(3Lip max + 1) L − l − 1 .
Therefore:
[Equation]
Plugging in our parameter choice γ = ε/ [ m (3Lip max +1)
L − 2 ] , N = LR 2 (3Lip max +1) 2 L − 2 / (2 ε 2 ) for some small enough absolute constant ν provides:
[Equation]
where the last inequality holds as long as m ≥ (3Lip max + 1) 4 L − 4 L 2 R 4 / (4 ε 2 ) .
Therefore by induction we see that W (1) , . . .
, W ( N ) ∈ B ( W (1) , ω ) .
As a result, the conditions of Lemmas. 7, 8,and 10 are satisfied for W ∗ and W (1) , . .
, W ( N ) .
In the following, we utilize the results of Lemmas 7, 8, 9 and 10 to prove the bound of cumulative loss.
First of all, by Lemma 9, we have:
[Equation]
Note that for the matrix inner product we have the equality 2 ⟨
[Equation]
Applying this equality to the right hand side above provides:
[Equation]
By Lemma 10, for l ∈ [ L ] we have ∥ W ( i ) l
[Equation]
L
[Equation]
Therefore:
[Equation]
W
[Equation]
3Lip
[Equation]
Telescoping over i = 1 , . .
, N , we obtain:
[Equation]
where in the first inequality we simply remove the term −∥ W ( N +1)
l − W ∗ l ∥ 2F / (2 γ ) to obtain an upper bound, the second inequality follows by the assumption that W ∗ ∈ B ( W (1) , Rm − 1 / 2 ) .
Plugging in the parameter choice γ = ε/ [ m (3Lip max + 1) 2 L − 2 ] , N = LR 2 (3Lip max + 1) 2 L − 2 / (2 ε 2 ) , then:
[Equation]
which finishes the proof.
D.2 Proof of Theorem
Proof.
By Lemmas 8, Lemma 11 and Theorem 3.3, Lemma 4.4, Corollary 3.10 in [Cao and Gu, 2019], let C 1 ( L ) = √ L/ (3Lip max + 1) L − 1 and C 2 ( L ) = √ L (3Lip max + 1) L − 1 , bring in our γ and
N with a not very large L , we have:
[Equation]
−
[Equation]
y
N (cid:33)
[Equation]
O (cid:32)(cid:114) log(1 /δ )
N (cid:33) .
According to the courant minimax principle [Golub and Van Loan, 1996]:
[Equation]
, that means y ⊤ (( K ( L ) ) − 1 )
y ≤ y ⊤ y λ min ( K ( L ) ) , then we have the final bound:
E [ ℓ 0 − 1 D ( ˆ W )] ≤ ˜ O
(cid:32)
C(cid:115)
y ⊤ y λ min ( K ( L ) )
N (cid:33) + O (cid:32)(cid:114) log(1 /δ )
N (cid:33) .
E Discussion on the key points and the motivation of the NTK analysis
In this section, we discuss the motivation and few key points in the proof of this paper and we also explain how the proof differs from previous results.
The motivation for studying the minimum eigenvalue of NTK :
To make this clearer, we provide an illustrative example on the significance of the minimum eigenvalue.
Let us consider the square loss Φ( θ ) = 12 (cid:80) ni
=1 ∥ f ( x i ) − y i ∥ 2 .
A simple calculation shows that Φ( θ ) ≤ [ ∇ Φ( θ )] 2 2 λ min ( K ) .
Thus if the minimum eigenvalue of NTK is strictly greater than 0, then minimizing the gradient on the LHS will drive the loss to zero.
The larger the minimum eigenvalue, the smaller the loss.
Therefore, in this work, we are using the minimum eigenvalue to derive the generalization bound of
NAS.
Key points in the proof :
• Minimum eigenvalue : Our proof framework is motivated by Nguyen et al.
[2021] on minimal eigenvalue of NTK of ReLU neural networks.
However, our proofs differ from them in two aspects.
Firstly, as we discussed in sec. 1, extension to mixed activation functions is non-trivial due to the special properties of ReLU .
More importantly, we remark that the lower bound of the minimal eigenvalue of NTK in [Nguyen et al., 2021, Theorem 3.2] holds with probability at least 1 − Ne − Ω( d ) − N 2 e − Ω (cid:18)
dN − 2 r − 0 .
5 (cid:19) , where r ≥ 2 is some constant.
It can be found that, this concentration probability decreases as the number of training data increases.
Thus, it could be negative for a large N .
This is due to the use of Gershgorin circle theorem leading to a loose probability estimation.
Instead, in this paper, we do not use this theorem, and we develop a tighter estimation based on Yaskov [2014] under the assumption of isotropic data distribution.
Accordingly, we achieve the reasonable 1 − e − d probability, c.f. Theorem 1.
• Generalization : Our proof framework is based on Cao and Gu [2019] for generalization guarantees of deep ReLU neural networks requiring m = Ω( L 56 ) .
Their results cannot be directly extended to other activation functions as the nice homogeneity and the derivative property of ReLU are used in their proof.
To make our result feasible to various activation functions, we employ Lipschitz continuous properties of all activation functions, and achieve the generalization guarantees with m = Ω(4 4 L ) ,
c.f. Theorem 3 and Lemma 11.
Admittedly, our result is in an exponential increasing order of the depth.
However, in practice, the depth of neural networks in NAS is usually smaller than 20 , or even 10 [Liu et al., 2018, Dong et al., 2021], which leads to 4 4 L ≪ L 56 in this case when compared to their result.
This result makes our theory reasonable and fair for NAS.
F Auxiliary numerical validations F.1 Dataset details and algorithm
We describe here the datasets that we have used for the numerical validation of our theory.
Those are the following five datasets:
Fashion-MNIST [Xiao et al., 2017] includes grayscale images of clothing.
The training set consists of 60 , 000 examples and the test set of 10 , 000 examples.
The resolution of each image is 28 × 28 , with each image belonging to one of the 10 classes.
MNIST [Lecun et al., 1998] includes handwritten digits images.
MNIST has a training set of 60 , 000 examples and a test set of 10 , 000 examples.
The resolution of each image is 28 × 28 .
CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2014] depicts images of natural scenes.
CIFAR-100 has a training set of 50 , 000 examples and a test set of 10 , 000 examples.
The resolution of each RGB image is 32 × 32 . 4.
ImageNet-16 [Chrabaszcz et al., 2017] is the down-sampled version of ImageNet [Deng et al., 2009] with image size 16 × 16 on 120 classes.
Our Eigen-NAS algorithm used in sec.
5.2 is summarized as below.
Algorithm
Require: Search space S , training data D tr = { ( x i , y i )
Ni =1 } , validation data D val = { ( x j , y j ) N v j =1 } .
Initialize max iteration =
M
Initialize candidate set C =
[] for search iteration in 1 , 2 , . .
, max iteration do
Randomly sample architecture s from search space S .
Compute Eigen := minimum eigenvalue of NTK.
C .
append ( s, Eigen ) update C to kept top-K best architectures end for s ⋆
= best s ( C , D tr , D val )
# Choose the best architecture based on validation error after trainingepochs.
Output s ⋆
F.2 Compared algorithms
We provide a thorough comparison with the following baselines:
Classical network: ResNet [He et al., 2016], which is the default baseline used widely in imagerelated tasks.
Reinforcement learning based algorithm: NAS-RL [Zoph and Le, 2017] with the validation accuracy as a reward, which is an classical and representative NAS Algorithm.
Differentiable algorithm:
DARTS
[Liu et al., 2019b] 4 , which is the earliest and basic gradient-based NAS algorithm.
Train-free algorithms using metrics to guide NAS: A new type of NAS algorithm
, they use some special metrics to pick models directly from candidate models
Common Train-free algorithms are:
NASWOT [Mellor et al., 2021] using the output of ReLU; TE-NAS [Chen et al., 2021] leveraging the spectrum of NTK and linear partition of the input space; KNAS [Xu et al., 2021] employing the Frobenius norm of NTK.
Our Eigen-NAS algorithm also belongs to this type.
F.3 Training/test accuracy of DNNs by NAS
Here we evaluate the classification results with 5 runs of the obtained architecture by DARTS under varying widths m ∈ { 64 , 128 , 256 , 512 , 1024 } and depths L ∈ { 5 , 10 } on Fashion-MNIST.
Figureshows that nearly 90% accuracy is achieved on the test set under different depth and width settings.
The result is competitive on FC/residual networks within 10 layers and without training tricks, e.g., data augmentation, batch norm and drop out.
We find that when compared to the depth, the network width also contributes on test accuracy.
As suggested by Eq.
(2), the amount of parameters in the neural network is approximately proportional to the depth, but squared to the width.
F.4 Simulation of minimum eigenvalues of NTK
We calculate the minimum eigenvalue of each NTK matrix under different architectures with activation functions, skip connections and depths, according to Lemma 1.
We consider four special cases on skip connections: a) no skip connections with α = 0 , i.e., fully connected neural network in Figure 3(a);
Width (m)
A cc u r a c y ( % ) train acc L= test acc L= train acc
L= test
acc L= b) skip connections between all consecutive layers with α = 1 , in Figure 3(b); c) the alpha of the first half (of the network) is 1 and the alpha of the second half is 0 shown in Figure 4(a)
; d) The alpha of the first half is 0 and the alpha of the second half is 1 .
The results are shown in Figure 4(b).
Figure 3(a) indicates that as the network depth increases, the minimum eigenvalue of NTK will become larger when LeakyReLU, ReLU, Swish and Tanh employed, but Sigmoid leads to a decreasing minimum eigenvalue, which is consistent with the upper bound shown in Theorem 1.
The LeakyReLU,
ReLU and Swish generate the fastest increasing rate of depth, while Tanh and Sigmoid are slow, which coincides with the derived lower bound in Theorem 1 and previous work Bietti and Bach [2021].
Figure 3(b) shows that, under the skip connection, the tendency of the minimum eigenvalue of NTK is similar to that of FC neural networks when various activation functions are employed.
However, the specific values and the growth rate are significantly larger than FC neural networks.
This result is consistent with the conclusion we state in Theorem 1 about skip layers leading to the increase of minimum eigenvalue of NTK with respect to the depth.
Moreover, Figure 4 show similar growth speed.
Then, we plot the comparison figure of NTK under above two settings and two settings in main paper for the same activation function in Figure 5.
In addition to reconfirming the order between different activation functions, we can also see that the effect of adding an activation layer in the second half of the 10 -layer neural network is better than the first half of the neural network.
This verifies the experimental results in Figure 1(b).
F.5 Additional experiments on NAS-Bench-101 and ranking correlations
In this section, we conduct more experiments on two new benchmarks NAS-Bench-101 [Ying et al., 2019] and Network Design Spaces (NDS)
[Radosavovic et al., 2019] using the same setting as sec.
Table 6 provides a comparison of the accuracy of Eigen-NAS, KNAS and NASWOT on four new search spaces.
For all of four search spaces, our method achieves the best results with 1% − 2% accuracy improvement.
Moreover, we conduct more detailed experiments using the CIFAR-10 dataset on NAS-Bench101.
Table 7 provides the running time and Kendall rank correlation coefficient between minimum eigenvalues and accuracy for the above three train-free NAS algorithm.
We can see that our EigenNAS method can get the best rank correlation coefficient with the fastest speed among three methods.
[Figure]
[Figure]
The scatter plot of the relationship between the minimum eigenvalue and the accuracy is shown in Figure 6.
F.6 Transfer learning experiment
Here we evaluate the proposed NAS framework on transfer learning.
The algorithm from sec.
5.1 is employed for this experiment, e.g., the same search space and search strategy.
The experiment setting is the following: we train the model on FashionMNIST for 20 epochs, then we use the pretrained weights and fine-tune them for 5 epochs on MNIST, with repeated three times.
Table 8 show that, after the fine-tuning for just 2 epochs, the method obtains up to 95% accuracy and after fine-tuning for 5 epochs it obtains up to 97% accuracy.
This verifies our intuition that the proposed NAS framework can obtain architectures that generalize well beyond the dataset they were optimized on.
[Table] [Figure]
Method
Eigen
[Table]
KNAS
[Table]
NASWOT
Running
[Table]
[Table]
− 0 .0 .− 0 .
F.7 DARTS experiment on CNN
Our theory relies on fully-connected matrices and we have indeed verified experimentally the validity of our theoretical findings.
To scrutinize our method even further, we attempt to extend our results to the popular convolutional neural networks.
We believe this will provide some further insights on future extensions of our theory.
In particular, we use DARTS (similarly with the experiment in sec. 5.1) with convolutional layers.
The standard dataset of CIFAR-10 is selected; the details of the dataset are shared in Appendix F.1.
The search space and search strategy follow sec.
5.1 with one differentiating point: we use convolutional layers instead of fully connected layers in Eq. We select DARTS on a Convolutional Neural Network with L = 10 and m = 1024 , while we repeat the experiment for 5 times.
After training, the probability of these activation functions and skip connections in each layer are reported in Figure 7(a) and 7(b), respectively.
Compared with the Figure 1, the activation function search exhibits similar characteristics with the results of the fully connected network.
Namely: (1) ReLU and LeakyReLU have the highest probability to be selected, (2) the
difference of probability between different activation functions in the first layer is the largest
[Table]
Accuracy
M i n i m u m
E i g e n v a l u e
          /D\HU 5 H/  7 DQK 6 L J P R L G /HD N\ 5 H/  6 Z L V K $ F W L Y D W L RQ I XQ F W L RQ
[Figure]

[Figure]
    
[Figure]
      
[Figure]
But for skip layer search, CNN exhibits the opposite results with fully connected network, that is, almost all of the skip connections have a probability of being selected less than 50% .
Based on the above results, our theory can still explain some of the phenomena observed in CNNs, e.g., activation functions search.
Nevertheless, our theory on skip connections search on CNNs mismatches with experimental demonstration in practice to some extent, which motivates us to conduct a refined analysis on CNNs for NAS.
F.8 β -DARTS experiment on MLP
In this section, we use an improved DARTS-based algorithm, β -DARTS
[Ye et al., 2022], for doing the activation function search.
Our experiments are performed on a 5-layers MLP and the experimental results are presented in Figure 8.
Compared with the results of DARTS in Figure 1, the experimental results of β -DARTS indicate that the probability difference between different activation functions is smaller, which may verify that DART is more easily to overfit .
This is also the advantage mentioned in the β -DARTS paper.
     /D\HU 5 H/  7 DQK 6 L J P R L G /HD N\  H/  6 Z L V K $ F W L Y D W L RQ I XQ F W L RQ            
                  
G Societal impact
This is a theoretical work that derived generalization bounds for the architectures obtained by NAS.
As such, we do not expect our work to have negative societal bias, as we do not focus on obtaining state-of-the-art results in a particular task.
On the contrary, our work can have various benefits for the community:
• We provide the first generalization bounds for the class of NAS architectures, which is expected to have a positive impact on the understanding and the application of such architectures.
• As we illustrate in sec.
5, we can use the minimum eigenvalue as a promising metric to guide NAS.
This can lead to further investigation on techniques for efficient evaluation of NAS by avoiding solving the intensive bi-level optimization of NAS explicitly.
Nevertheless, we encourage researchers to further investigate the impact of different architectures and their inductive biases on the society.