Generalization Properties of NAS under Activation and Skip Connection Search
Abstract
Neural Architecture Search (NAS) has fostered the automatic discovery of neuralarchitectures, which achieve state-of-the-art accuracy in image recognition.
Despite the progress achieved with NAS, so far there is little attention to theoreticalguarantees on NAS.
In this work, we study the generalization properties of NASunder a unifying framework enabling (deep) layer skip connection search andactivation function search.
To this end, we derive the lower (and upper) boundsof the minimum eigenvalue of Neural Tangent Kernel under the (in)finite widthregime from a search space including mixed activation functions, fully connected,and residual neural networks.
Our analysis is non-trivial due to the coupling ofvarious architectures and activation functions under the unifying framework.
Then,we leverage the eigenvalue bounds to establish generalization error bounds of NASin the stochastic gradient descent training.
Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performingarchitectures, even in the case without training, leading to a training-free algorithmbased on our theory.
Accordingly, our numerical validation shed light on the designof computationally efficient methods for NAS.
Introduction
Neural Architecture Search (NAS)
[Zoph and Le, 2017] is a powerful technique that enables theautomatic design of neural architectures.
NAS defines a set of operations, that includes variousactivation functions and layer types, or potential connections among layers [Elsken et al., 2019, Renet al., 2020], which are then referred to as the search space .
Optimization over the search spacereturns the optimal architecture as a subset of the possible combinations of operations.
NAS 1 obtainsstate-of-the-art results in image recognition [Liu et al., 2019a, Ding et al., 2020, Zhang et al., 2019,
Chen et al., 2019] or can be used to further improve architectures defined by a human expert [Tanand Le, 2019].
The spectacular results obtained by NAS have led to a significant interest in thecommunity to further improve the NAS algorithms, the search space etc.
However, to date little focushas been provided in the following question: Can NAS 1 achieve generalization guarantees similar toa typical neural network?
There has been a large number of exciting results aimed at demystifying both the optimization andgeneralization aspects in deep learning theory [Allen-Zhu et al., 2019, Cao and Gu, 2019, Chenet al., 2020a, Arora et al., 2019a].
However, previous work mainly focuses on specific architectures,e.g., fully-connected (FC) neural networks [Allen-Zhu et al., 2018, Bartlett et al., 2017] or residualneural networks [He et al., 2016, Huang et al., 2020], in which a single activation function is usedthroughout the network.
These off-the-shelf theoretical results cannot be directly applied to analyze the rich search space (of NAS) that is covering various/mixed architectures and parameters.
Thatmakes the non-trivial analysis on NAS worth of study on its own right.
The recent work of Oymak et al.
[2021] is the first work to provide generalization guarantees ona related problem, i.e., activation functions search.
The study provides generalization results ontwo-layer networks relying on the minimum eigenvalue with a strictly larger than zero assumption,i.e., λ min ( K ) > 0 for the the neural tangent kernel (NTK)
[Jacot et al., 2018] of the network, where
K is the NTK matrix.
In this work, we introduce the first theoretical guarantees for multilayer NAS where the search spaceincludes activation functions and skip connections.
We study the upper/lower bound of the minimumeigenvalue of NTK (in the (in)finite regime) under mixed activation functions and architectures toevade the minimum eigenvalue assumption of Oymak et al.
Then, we provide optimizationand generalization guarantees of deep neural networks (DNNs) equipped with NAS.
Our resultsindicate that the minimum eigenvalue estimation can act as a powerful metric for NAS.
This method,called Eigen-NAS, is train-free, but still effective with experimental validation when compared torecent promising algorithms
[Xu et al., 2021, Chen et al., 2021, Mellor et al., 2021].
Formally, ourmain contribution and findings are summarized below i)
We build a general theoretical framework based on NTK for NAS with search on popular activationfunctions in each layer, fully-connected, and skip connections.
We derive the NTK formula of thesearchitectures in the (in)finite width regime under the unifying framework.ii)
We derive the upper and lower bound of minimum eigenvalue of NTK under the (in)finite widthregime for the considered architectures.
We introduce a new technique to ensure the probabilityof concentration inequality remains positive, addressing the negative probability issue in previouswork.
Our analysis highlights how the error bounds differs under activation function search and skipconnection search and can guide NAS. iii)
By virtue of the NTK matrix and its minimum eigenvalue, we establish a connection between themand generalization of the searched DNN trained by stochastic gradient descent (SGD).
Our theoreticalresults show that the generalization performance largely depends on the minimum eigenvalue of NTKfor NAS, which provides theoretical guarantees for the searched architecture.iv)
Our theoretical results are supported by thorough experimental validations with the followingfindings: 1) our error bounds on the minimum eigenvalue largely depend on activation functionsin the first layer rather than those in deeper layers.
The applied NAS algorithm always picks up ReLU (Rectified Linear Unit) and LeakyReLU in the optimal architecture, which coincides with ourtheory that predicts ReLU and LeakyReLU achieve the largest minimum eigenvalues.
The skipconnections are required in each layer under our not very large DNNs.
Furthermore, our experimentalevidence on Eigen-NAS indicates that the minimum eigenvalue is a promising metric to guide NAS(without training) as suggested by our theory.
Technical challenges.
The technical challenges of this paper mainly focus on how to analyzeactivation functions with different properties and skip connections under a unifying framework.
Thiswork is non-trivial, firstly previous works mainly focus on the ReLU activation function [Nguyenet al., 2021, Cao and Gu, 2019, Allen-Zhu et al., 2019] in optimization and generalization of asingle fully-connected neural network.
Their proofs heavily depend on properties of ReLU , e.g.,homogeneity and ReLU( x ) = x ReLU ′ ( x ) which are invalid when other commonly-used activationfunctions, e.g., Tanh, Sigmoid, and Swish, are used.
This problem becomes quite intractable underthe consideration of mixing of activation functions and residual connections.
To tackle these technicalchallenges, we develop the following techniques: a) to handle the non-homogeneous property of
Tanh, Sigmoid, and Swish, we develop a new integral estimation approach for minimal eigenvalueestimation.
b) To establish the connection between minimum eigenvalues of NTK and generalizationerrors, we use the Lipschitz continuity to avoid the special property of ReLU.
More importantly,we introduce a new technique [Yaskov, 2014] to replace Gershgorin circle theorem for minimumeigenvalue estimation, which avoids concentration inequalities with negative probability in somecertain cases [Nguyen et al., 2021].
Related work
Network architecture search (NAS): The idea of NAS stems from Zoph and Le [2017], while theidea of cell search, i.e., searching core building blocks and composing them together, emerged in
Zoph et al.
The earlier literature used discrete optimization techniques for obtaining thearchitecture.
DARTS Liu et al.
[2019b] considers NAS as a continuous bi-level optimization task.
Recent variants of DARTS
[Xu et al., 2019, Wu et al., 2019] and several train-free methods [Melloret al., 2021, Chen et al., 2021, Xu et al., 2021] have demonstrated success in reducing the searchtime or improving the search algorithm.
However, the aforementioned works have not providedgeneralization guarantees for the optimal architecture.
Optimization and generalization of DNNs via NTK:
In the NTK framework [Jacot et al., 2018, Duet al., 2019, Chen et al., 2020b], the training dynamics of (in)finite-width networks can be exactlycharacterized by kernel tools.
Leveraging NTK facilitates studies on the global convergence of
GD Allen-Zhu et al.
[2019], Du et al.
[2019], Nguyen [2021] in DNNs via the minimum eigenvalueof NTK.
In fact, it also controls the generalization performance of DNNs [Du et al., 2018, Cao and
Gu, 2019, Allen-Zhu et al., 2018], which is further studied in Bietti and Bach [2021].
Problem Settings
In this section we introduce the problem setting of our NAS framework based on the search spaceand algorithm (search strategy) for our paper.
Let X ⊆
R
d be a compact metric space and Y ⊆ R .
We assume that the training set D tr ={ ( x i , y i ) }
Ni =1 is drawn from a probability measure D on X × Y , with its marginal data distributiondenoted by D X .
The goal of a supervised learning task is to find a hypothesis (i.e., a neural networkused in this work)
f : X → Y such that f ( x ; W ) parameterized by W is a good approximation ofthe label y ∈ Y corresponding to a new sample x ∈ X .
In this paper, we consider the classificationtask, evaluated by minimizing the expected risk
[Equation]
where ℓ [ yf ( x ; W )] is the classification loss ℓ ( · ) as a surrogate of the expected 0-1 loss ℓ 0 − 1 D ( W ) :
= 110 E ( x ,y )
∼D [1 { yf ( x ; W ) < 0 } ] .
In this paper, we employ the cross-entropy loss, which is defined asℓ ( z ) =
log[
+ exp( − z )] .
Notation: For an integer L , we use the shorthand [ L ] = { 1 , 2 , . . .
, L } .
The multivariate standard
Gaussian distribution is N ( 0 , I d ) with the zero-mean vector 0 and the identity-variance matrix I d .
We denote the direct sum by ⊕ .
We follow the standard Bachmann–Landau notation in complexitytheory e.g., O , o , Ω , and Θ for order notation.
3.1 Neural Networks and Search Space
In this work, we consider a particular parametrization of f as a deep neural network (DNN) withdepth L ( L ≥ 3 ) 2 which includes the fully-connected (FC) neural networks setting and the residualneural networks setting, and various activation functions in each layer.
This allows for a quite general
NAS setting.
Formally, we define a single-output DNN with the output f l ( x ) in each layerf l ( x )=  
  x l = 0 , σ 1 ( W 1 x ) l = 1 , σ l ( ⟨ W l , f l − 1 ( x ) ⟩ )+ α l − 1 f l − 1 ( x ) 2 ≤ l ≤ L − 1 , ⟨ W L , f L − 1 ( x ) ⟩ l = L , (1) where the weights of the neural networks are W 1 ∈ R m × d , W l ∈ R m × m , l = 2 , . . .
, L − 1 and
W L ∈ R m .
The binary parameter α l is for layer search, and the activation function is σ l ( · ) .
Theneural network output is f ( x ; W ) =
f L ( x ) .
Architecture search: A binary vector α =
[ α 1 , α 2 , · · · , α L − 2 ] ⊤ represents the skip connections,where the α l ∈ { 0 , 1 } in Eq.
(1) indicates whether there is a skip connection in the l -th layer.
Noticethat we unify FC and residual neural networks under the same framework.
Activation function search: We select five representative activation functions used in Eq.
(1),i.e.
, F σ = { ReLU , LeaklyReLU , Sigmoid , Tanh , Swish } , that can be bounded, unbounded,smooth, non-smooth, monotonic, or non-monotonic, as reported in Table 1.
We define σ =
[Table]
the integral
[Table]
Sigmoid to ensure f Sigmoid [Table] = [Table]
facilitates our theoretical analysis.
The parameter is
[Table]
The definition of f T is similar to f S by using the Tanh function.
[ σ 1 , σ 2 , · · · , σ L − 1 ] ⊤ with σ l ∈ F σ for any l ∈
[ L − 1] as the indicator to show which activation function is selected in each layer.
Importantly, our NAS framework allows for a differentactivation function in each layer, which enlarges the search space.
In our setting, we conduct the architecture search and the skip connection search independently, andaccordingly, our search space is defined as the direct sum of them
[Equation]
where W :
= ( α , σ , W 1 , . .
, W L ) ∈ W represents the collection of weight matrices and indicatorfor skips and selected activation functions for all layers.
3.2 Algorithm (Search Strategy)
The search strategy is the core part in NAS to pick up the optimal architecture from the search space.
Here we build a general Algorithm 1 combining the search strategy for NAS (the first part) and thesubsequent neural network training by SGD (the second part).
We firstly utilize a typical NAS algorithm, e.g., random search WS [Li and Talwalkar, 2020] or
DARTS 3 , to search skip connections and activation functions independently, which results in theoptimal architecture { ( σ ∗ i )
L − 1 i =1 , ( α ∗ i )
L − 2 i =1 } with the max probability, see sec.
5.1 for details.
Inparticular, Algorithm 1 also allows for the guidance of NAS in a train-free strategy via some specificmetrics, e.g., the minimum eigenvalue of NTK (and its variant), see our Eigen-NAS method in sec.
Then, we conduct neural network training on the selected architecture by SGD.
For ease of theoreticalanalysis, we employ the constant step-size SGD with one epoch and randomly choose the weightparameters during all the iterations, which is commonly used in deep learning theory [Cao and Gu,2019, Zou et al., 2019].
Main result
In this section, we state the main theoretical results.
We present the assumptions used in our proofin sec. 4.1.
Then in sec.
4.2 we provide the recursive form of NTK for DNNs defined by Eq.
(1) withmixed activation functions and skip connections.
The upper and lower bound of the corresponding
NTK in the infinite and finite width setting is given in sec.
4.3 and 4.4, respectively.
Finally, in sec.
4.5,we connect the minimum eigenvalue of NTK and the generalization error bound of DNNs underthese search schemes.
The proofs of our theoretical results presented in this section are deferred to
Appendix B, C, and D, respectively.
4.1 Assumptions
We make the following assumptions on data and activation functions.
Our assumptions are frequentlyemployed in the literature as we highlight below.
Assumption 1.
We assume that the data satisfy ∥ x ∥ 2 = 1 .
Assumption 2.
The activation function σ : R → R satisfies σ ∈ L 2 ( R , e − x 2 / 2 / √ 2 π ) , where
L 2 ( R , g ) denotes the square integrable function.
Input: search
Flag method // conduct NAS algorithms if Flag GuideNAS = EigenNAS then Guide NAS from S by our Eigen-NAS algorithm.
else if Flag GuideNAS =
DARTS then
Search neural network architectures from S using the DARTS algorithm.
end if
Output the optimal architecture { ( σ ∗ i )
L − 1 i =1 , ( α ∗ i )
L − 2 i =1 } ∈ S with max probability.
// do neural network training via SGD
Gaussian initialization: W (1) l ∼ N (0 , 1 /m ) ,
l ∈
[ L ]
Construct the neural network f ( x ; W (1) l ) based on { ( σ ∗
i
L
1 i α ∗ i
L −
=1 } for i = 1 to N do
[Equation]
end for
Assumption 3.
We further assume that x is isotropic.
i.e. E xx ⊤ =
I d /d , where the coefficient 1 /dis to satisfy Assumption 1 at the same time.
Remark: The first assumption on normalized data is commonly used in practice and theory onover-parameterized neural networks [Du et al., 2018, 2019, Allen-Zhu et al., 2019, Oymak and
Soltanolkotabi, 2020, Malach et al., 2020].
The second assumption is general as the studied activationfunctions in Table 1 satisfy it.
The third assumption is standard in statistics [Vershynin, 2018, Section3.2], and used in [Hastie et al., 2022, Klimovsky, 2012, Yaskov, 2014].
It covers Gaussian data,and data uniformly spread on the sphere, commonly used in deep learning theory [Mei et al., 2021,
Ghosh et al., 2021].
4.2 Recursive NTK for DNNs defined by Eq. (1)
Recall that NTK [Jacot et al., 2018] under the infinite width setting ( m → ∞ ) is the following:
[Equation]
where the NTK matrix for residual networks is derived by the following regular chain rule.
Lemma 1.
For any l ∈ [3 , L ] and s ∈ [2 , L ] , denote
[Equation]
Then the NTK for residual networks defined in Eq.
(1) can be written as
[Equation]
Remark: ( i ) Our NTK formula of ResNet differs from the one of Tirer et al.
[2021], Huang et al.[2020], Belfer et al.
[2021] in two critical ways: 1) each skip-layer in our model skips one fullyconnected layer and one activation function, as opposed to the two-layer skip of previous works, 2) our formulation does not require every layer to have a parallel skip connection, which increases the
flexibility of the network.
Those differences also result in a different NTK matrix.
( ii ) Our NTK formulation covers different activation functions, and we adopt the same initialization
(coefficient) on them to ensure fair/equal search in our NAS framework .
Lemma 1 is general to cover FC and residual neural networks, which facilitates the analysis ofminimum eigenvalue of NTK under the unifying framework.
If α l = 0 for l ∈ [ L − 1] , our NTK formulation for residual neural networks degenerates to that of fully-connected neural network, and
A l and G l become equal.
4.3 Minimum Eigenvalue of NTK for infinite width
We are now ready to state the main result of infinite width neural network.
We provide the errorbounds of minimum eigenvalue of NTK for infinite width neural network mixed with five differentactivation functions.
The main differences between different activation functions are illustratedin Table 1.
Theorem 1.
For a DNN defined by Eq.
(1) and a not very large L , let K ( L ) be the limiting NTKrecursively defined in Lemma 1.
Then, under Assumptions 1 and 3, with probability at least 1 − e − d ,we have
[Equation]
[Equation]
where µ 1 ( σ 1 ) is the 1 -st Hermite coefficient of the first layer activation function, and β 1 , β 2 , β 3 arethree constants on various activation functions defined in Table 1.
Remark:
Here we use a not very large L ≤ 10 , because in NAS, the users can directly specify thedepth L (in a constant order) and in practice it is often less than 10 [Liu et al., 2018, Dong et al.,2021].
In fact, existing NAS algorithms such as DARTS tend to have architectures with wide andshallow cell structure as suggested by Shu et al.
Theorem 1 shows the upper and lower boundof the minimum eigenvalue of NTK under the mix of activation functions and skip connections.
Thefollowing conclusions can be drawn from our result:
The error bounds of minimum eigenvalue significantly depend on the depth of the network L , theskip connections via α p , which makes the minimum eigenvalue increasing fast as L and the number
of skip connections increase.
Besides, the minimum eigenvalue is also effected by activation
functions via β 1 , β 2 , β 3 .
Nevertheless, the lower bound is independent of β 1 and β 2 .
Different activation functions lead to different tendency (increase or decrease) on λ min ( K ( L ) ) .
Asthe depth increases, the lower bound λ min ( K ( L ) ) under ReLU remains unchanged, increases under
LeakyReLU, and decreases when Sigmoid, Tanh or Swish applied, which brings in new findings
when compared to Nguyen et al.
[2021] sorely on ReLU.
For the upper bound for λ min ( K ( L ) ) , we
can see our results are positively correlated with the depth L .
One can see that µ 1 ( σ 1 ) is only related to the activation function of the first layer, which implies
that the activation function in the first layer is very important as λ min ( K ( L ) ) largely depends on it.
4.4 Minimum Eigenvalue of NTK for finite width
To study the finite width, we firstly introduce the Jacobian of the network.
Let
F =
[ f ( x 1 ) , . .
f ( x N )] T .
Then, the Jacobian J of F with respect to W is J = 220 (cid:104) ∂ F ∂ vec ( W 1 ) , . . .
, ∂ F ∂ vec ( W L ) (cid:105) , where J have dimension R ((( L − 2) × m
+ d +1)
× m × N .
The empirical Neural
Tangent Kernel (NTK) matrix can be defined as ¯ K ( L ) =
JJ ⊤ =
(cid:80) Ll =1 (cid:104)
[Equation]
(cid:104) ∂ F
[Equation]
⊤ .
Accordingly, we generalize Theorem 1 from the infinite-width to finite-width setting below.
Theorem 2.
For an L -layer network defined by Eq.
(1) , let K ( L ) =
JJ ⊤ be the NTK matrix, and theweights of the network be initialized as [ W l ] i,j ∼ N (0 , 1 /m ) , for all l ∈ [ L ] .
Under Assumptions 1and 3, then λ min ( JJ ⊤ ) can be bounded by
[Equation]
where the first inequality (lower bound) holds with probability at least 1 − e − d − 227 (cid:80) L −l =1 exp( − Ω( m ))
− exp( − Ω(1)) and the second inequality (upper bound) holds with probability at least 1 − (cid:80) L − 1 l =1 exp( − Ω( m ))
− exp( − Ω(1)) .
The definitions of β 2 , and β 3 are the same asthose in Theorem 1.
Remark:
Theorem 2 achieves a similar result as Theorem 1 if the width m is large.
The onlydifference in the findings is that probability that the Theorem holds increases as m increases.
4.5 Connection to Generalization Error Bound
Based on the above error bound of the minimum eigenvalue of NTK under different settings, here weestablish its relationship with the generalization error of DNNs.
We provide a bound on the expected0-1 error obtained by Algorithm 1.
Theorem 3.
Given a DNN defined by Eq.
(1) with y =
( y 1 , . . .
, y N ) ⊤ determined by Algorithm 1with
the step size of SGD γ = κC 1 · (cid:112) y ⊤ ( K ( L ) ) − 1 y / ( m √ N ) for some small enough absoluteconstant κ .
Under Assumptions 1, 2 and 3, for any δ ∈ (cid:0) 0 , e − 1 (cid:3) and a not very large L , if the widthm ≥ ˆ m , where ˆ m depends on λ min ( K ( L ) ) , δ, N , and L , then with probability at least 1 − δ over therandomness of W (1) , we obtain the following high probability bound:
[Equation]
where C 1 and C 2 are two constants only depending on L .
Remark:
Theorem 3 gives an algorithm-dependent generalization error bound of DNNs definedby Eq.
(1) trained with SGD with different activation functions and skip connections.
If m is largeenough, the learning rate is infinitesimal, which means the generalization error bound mainly dependson the NTK matrix, similarly to Cao and Gu [2019], Du et al.
According to Theorem 3, the generalization performance of DNNs is controlled by the minimumeigenvalue of the NTK matrix, which is in turn affected by different activation functions and skipconnections, as discussed in Theorem 1.
Apart from the NTK matrix itself, the condition m ≥ ˆ mis also effected by different activation functions, which implies that the required minimum width isdifferent in these cases.
4.6 Proof sketch
Our work extends the proofs of Nguyen and Mondelli [2020], Cao and Gu [2019] beyond ReLU,which is critical for enabling search across activations.
The extension to other activation functionsand skip connections is non-trivial due to non-linearity, inhomogeneity and nonmonotonicity.
To derive the error bound on the minimum eigenvalue, we start from Lemma 1 on the NTK formulaunder the mixed activation functions and skip connections, and we transform the minimum eigenvalueestimation to the computation (estimation) of the bound G , ˙ G ( λ min ( G ) ).
The infinite width andfinite width are included in Appendix B and C respectively.
For the upper bound, we estimatethe diagonal elements of G and use the minimum eigenvalue less than the mean of the diagonalelements for proof.
For the lower bound, we use Hermite expansion and [Yaskov, 2014, Corollary3.1].
Combining these results concludes the proof.
To derive the generalization error bounds, we need a series of lemmas (see Appendix D).
If the inputweights are close, the output of each neuron with any activation function does not change too much(see Lemma 7); If the initilizations are close, the neural network output f ( x ; W ) is almost linearin W (see Lemma 8), and the loss function ℓ [ y i f ( x i ; W )] is almost a convex function of W forany i ∈
[ N ] (see Lemma 9).
Accordingly, the gradient and loss of the neural network can be upperbounded by Lemma 10 and 11, respectively, which concludes the proof by combining with somerelevant results [Cao and Gu, 2019, Allen-Zhu et al., 2019].
Further discussion on the differences isdeferred to Appendix E.
Numerical Validation
To validate our theoretical results, we conduct a series of experiments on NAS.
Firstly, we simulatethe NTK matrices under different depths in Appendix F.4 to verify the relationship between the
[-Finally, we use the minimum eigenvalue of NTK to guidethe training of NAS on the benchmark NAS-Bench-
-]
          /D\HU 5 H/  7 DQK 6 L J P R L G /HD N\ 5 H/  6 Z L V K $ F W L Y D W L RQ I XQ F W L RQ              
      
                      
               (a) activation functions σ       9 10 Layer 0.
P r o b a b ili t y
[Figure] [Table]
minimum eigenvalue of NTK and the network depth L in Theorem 1. In sec. [Table] 1 we use the DARTSalgorithm [Liu et al., 2019b] to conduct experiments on activation function search and skip connectionsearch under the search space of Eq.
Finally [Table]
{+we use the minimum eigenvalue of NTK to guidethe training of NAS on the benchmark NAS-Bench-
+}
[Dong and Yang, 2020]
[Table]
with a comparison ofrecent NAS algorithms.
Additional experiments on NAS-Bench-101 [Ying et al., 2019] and transferlearning are deferred to Appendix F.5 and F.6. 5.1 DARTS experiment
In this section we employ a typical NAS algorithm, DARTS Liu et al.
[2019b], to assess our theoreticalresults on activation functions and skip connections.
We select Fashion-MNIST
[Xiao et al., 2017]
asa standard benchmark. Details about Fashion-MNIST are shared in Appendix F.1.
Search space and search strategy:
Our search space is defined by Eq. (2) on skip connections,activation functions, and weight parameters.
We follow the search strategy with Liu et al.
[2019b]in a two-level scheme, one is for weight parameter search W and the other is for architecturesearch { α , σ } , which results in the final optimal architecture { α ∗ , σ ∗ , W ∗ } .
Different from Liu et al.[2019b], the activation function search and the skip connection search in our setting is independent.
Toobtain σ ∗ , we use the softmax function to normalize their weights and choose the specific activationfunction with the highest probability in each layer.
To obtain α ∗ , we initialize each entry α l = 1 / 2( l ∈
[ L − 2] ), constrain it to [0 , 1] during training, and retain the skip connection when α ∗ l > 1 / 2 .
NAS Results: We conduct the NAS experiment via DARTS on a feedforward neural network with
L = 10 and m = 1024 , with 5 runs.
After training, the probability of these activation functions andskip connections in each layer is reported in Figure 1(a) and 1(b), respectively.
We have the followingfindings: Firstly, after the search process, LeakyReLU and ReLU are selected as the activationswith the highest probability in each layer.
This coincides with our theoretical results in Theorem 1.
One minor difference is that the probability of LeakyReLU is slightly inferior to ReLU in practice.
The reason behind this could be the sparsity of ReLU [de Dios and Bruna, 2020].
Secondly, inthe first layer, we observe the largest difference on the probability of various activation functions.
As the network becomes deeper, such difference pales in importance until there is no differenceamong these activation functions in the last layer.
This phenomenon matches our theory well.
To bespecific, in Theorem 1, our result on the minimum eigenvalue largely depends on the first layer andits Hermite coefficient.
Besides, this result also provides a justification on omitting the high-orderterms while retaining the first layer activation terms.
Thirdly, for the skip search result, we find thatthe skip connections are required in each layer when L ≤ 10 , as suggested by our theoretical resultsin Theorem 1.
It also verifies results of Zhou et al.
We expect that the skip connections mightnot be required in each layer for deep neural networks, since their capacity can already be enough [Heet al., 2016]; but we defer the related study to a future work.
Interestingly, the search strategy favors the activation functions and the skip connections with largerminimum eigenvalue of NTK, which enjoy better generalization performance.
This result alsomotivates us to study the following question: can the minimum eigenvalue of NTK guide the searchprocess in NAS?
We provide an affirmative answer in the next section with experimental validations.
5.2 NAS-Bench-201 Experiment
In this experiment, we use the minimum eigenvalue to guide NAS on NAS-Bench-
[Dong and
Yang, 2020].
Each experiment is repeated 5 times, while it can run on a single GPU in a few hours.
Benchmark and baselines: NAS-Bench-201 [Dong and Yang, 2020] is a commonly used benchmarkfor NAS algorithm evaluation, which includes three datasets: a) CIFAR-10 [Krizhevsky et al., 2014],b)
CIFAR-
[Krizhevsky et al., 2014] and c)
ImageNet-16 [Chrabaszcz et al., 2017] for imageclassification.
Details on the datasets exist in Appendix F.1.
Apart from that, we evaluate the proposedapproach with some baselines including ResNet, RL based algorithm and so on.
Algorithm procedure:
Our algorithm, called Eigen-NAS, also belongs in the train-free category.
Eigen-NAS follows KNAS, which leverages the minimum eigenvalue of NTK to guide NAS.
However, due to the O ( N 3 ) time complexity of computing these eigenvalues, KNAS insteadcomputes ∥ K ∥ F .
However, from the expression λ min ( K ) ≤ 1 d (cid:80) Ni =1 K ii ≤ ∥ K ∥ F we utilize thefirst inequality in Eigen-NAS to obtain a tighter (and more computationally efficient) bound toλ min .
The computation cost of our method is O ( N ) , which is less than computing the Frobeniusnorm ( O ( N 2 ) ).
Sequentially, the top- k best candidates architectures are chosen in KNAS and our
Eigen-NAS, and then the best architecture is chosen by the validation error, refer to the resultsin Table 2.
Due to the page limit, the algorithm is located in Appendix F.
Results: The experimental results in Table 2 verify that Eigen-NAS guided by the proposed metricabove achieves the best performance on both the CIFAR-100 and ImageNet-16 datasets, and competitive performance on CIFAR-10, outperforming KNAS in all three cases when k = 20 for bothmethods.
Even when we consider a smaller k = 5 , Eigen-NAS can outperform KNAS, which weattribute to the more precise minimum eigenvalue estimation.
Conclusion
In this work, we explore the relationship between the minimum eigenvalue of NTK and neuralarchitecture search.
We derive upper and lower bounds on the minimum eigenvalues of NTK for(in)finite residual networks under different mixtures of activation functions, and establish a connectionbetween the minimum eigenvalues and the generalization properties of the special search space:activation function and skip connection search of NAS.
Our theoretical results on various activationfunctions and mixed activation cases can also be a tool for deep learning theory researchers to provegeneric results rather than studying a single architecture, e.g., ReLU networks.
In addition, we usethe minimum eigenvalue as a guide for the training of NAS in a train-free method, which greatlyexceeds the efficiency of the classic NAS algorithm.
When compared with existing train-free methods,our algorithm, called Eigen-NAS, achieves a higher accuracy.
We posit that this will be useful forstudying computationally efficient methods on NAS.