Understanding Neural Architecture Search: Convergence and Generalization
Abstract
Neural Architecture Search (NAS) has fostered the automatic discovery of neuralarchitectures, which achieve state-of-the-art accuracy in image recognition.
Despite the progress achieved with NAS, so far there is little attention to theoreticalguarantees on NAS.
In this work, we study the optimization and generalizationproperties of NAS under a unifying framework enabling (deep) layer search and activation function search.
To this end, we derive the lower (and upper) bounds of theminimum eigenvalue of Neural Tangent Kernel under mixed activation functions,fully connected, and residual neural networks in the (in)finite width regime.
Ouranalysis is non-trivial due to the coupling of various architectures and activationfunctions under the unifying framework.
Then, we leverage the eigenvalue boundsto establish generalization error bounds in the stochastic gradient descent training.
Importantly, we theoretically and experimentally show how the derived results canguide NAS in terms of selecting the top-performing architectures, even in the casewithout training.
Our theoretical demonstration and numerical validation shed lighton the design of computationally efficient methods for NAS.
Introduction
Neural Architecture Search (NAS)
[Zoph and Le, 2017] is a powerful technique that enables theautomatic design of neural architectures.
NAS defines a set of operations, that includes variousactivation functions and layer types or potential connections among layers [Elsken et al., 2019, Renet al., 2020], which are then referred to as the search space .
Optimization over the search spacereturns the optimal architecture as a subset of the possible combinations of operations.
NAS 1 obtainsstate-of-the-art results in image recognition [Liu et al., 2019a, Ding et al., 2020, Zhang et al., 2019,
Chen et al., 2019] or can be used to further improve architectures defined by a human expert [Tanand Le, 2019].
The spectacular results obtained by NAS have led to a significant interest in thecommunity to further improve the NAS algorithms, the search space etc.
However, to date little focushas been provided in the following question: Can NAS 1 achieve generalization guarantees similar toa typical neural network?
There has been a large number of exciting results aimed at demystifying both the optimization andgeneralization aspects in deep learning theory [Allen-Zhu et al., 2019, Cao and Gu, 2019, Chenet al., 2020a, Arora et al., 2019a].
However, previous work mainly focuses on specific architectures,e.g., fully-connected (FC) neural networks [Allen-Zhu et al., 2018, Bartlett et al., 2017] or residualneural networks [He et al., 2016, Huang et al., 2020], in which a single activation function is usedthroughout the network.
These off-the-shelf theoretical results cannot be directly applied to analyzethe rich search space (of NAS) that is covering various/mixed architectures and parameters.
Thatmakes the non-trivial analysis on NAS worth of study on its own right.
The recent work of Oymak et al.
[2021] is the first work to provide generalization guarantees ona related problem, i.e., activation functions search.
The study provides generalization results ontwo-layer networks relying on the minimum eigenvalue with a strictly larger than zero assumption,i.e., λ min ( K ) > 0 for the the neural tangent kernel (NTK)
[Jacot et al., 2018] of the network, where
K is the NTK matrix.
In this work, we introduce the first theoretical guarantees on NAS with a network of arbitrary depth.
We study the upper/lower bound of the minimum eigenvalue of NTK (in the (in)finite regime) undermixed activation functions and architectures to evade the minimum eigenvalue assumption of Oymaket al.
Then, we provide optimization and generalization guarantees of deep neural networks(DNNs) equipped with NAS.
Our results indicate that the minimum eigenvalue estimation can actas a powerful metric for NAS.
This method, called Eigen-NAS, is train-free, but still effective withexperimental validation when compared to recent promising algorithms [Xu et al., 2021, Chen et al.
,2021, Mellor et al., 2021].
Formally, our main contribution and findings are summarized below:
A unifying analysis framework based on NTK for NAS: We build a general theoretical frameworkfor NAS with search on popular activation functions in each layer, fully-connected, and residualconnections.
We derive the NTK formula of these architectures in the (in)finite width regime underthe unifying framework.
Minimum eigenvalue of NTK in various search architectures: We derive the upper and lowerbound of minimum eigenvalue of NTK in different architectures under the (in)finite width regime.
We introduce a new technique to ensure the probability of concentration inequality remains positive,addressing the negative probability issue in previous papers.
Our analysis highlights how the errorbounds differs under activation function search and skip connection search and can guide NAS.
Relationship between NTK and generalization on the searched architecture: With the NTKmatrix and its minimum eigenvalue, we establish a connection between them and generalization ofthe searched DNN trained by stochastic gradient descent (SGD).
Our theoretical results show that thegeneralization performance largely depends on the minimum eigenvalue of NTK for NAS, whichprovides theoretical guarantees for the searched architecture.
Findings and guidance of NAS: Our theoretical results are supported by thorough experimentalvalidations with the following findings: 1) our error bounds on the minimum eigenvalue largelydepend on activation functions in the first layer rather than those in deeper layers.
The applied NASalgorithm always picks up ReLU (Rectified Linear Unit) and LeakyReLU in the optimal architecture,which coincides with our theory that predicts ReLU and LeakyReLU achieve the largest minimumeigenvalues.
The skip connections are required in each layer under our not very large DNNs.
Furthermore, our experimental evidence on Eigen-NAS indicates that the minimum eigenvalue is apromising metric to guide NAS (without training) as suggested by our theory.
Technical challenges.
The technical challenges of this paper mainly focus on how to analyzeactivation functions with different properties and skip connections under a unifying framework.
Thiswork is non-trivial, firstly previous works mainly focus on the ReLU activation function [Nguyenet al., 2021, Cao and Gu, 2019, Allen-Zhu et al., 2019] in optimization and generalization of asingle fully-connected neural network.
Their proofs heavily depend on properties of ReLU , e.g.,homogeneity and ReLU( x ) = x ReLU ′ ( x ) which are invalid when other commonly-used activationfunctions, e.g., Tanh, Sigmoid, and Swish, are used.
This problem becomes quite intractable underthe consideration of mixing of activation functions and residual connections.
To tackle these technicalchallenges, we develop the following techniques: a) to handle the non-homogeneous property of
Tanh, Sigmoid, and Swish, we develop a new integral estimation approach for minimal eigenvalueestimation.
b) To establish the connection between minimum eigenvalues of NTK and generalizationerrors, we use the Lipschitz continuity to avoid the special property of ReLU.
More importantly,we introduce a new technique [Yaskov, 2014] to replace Gershgorin circle theorem for minimumeigenvalue estimation, which avoids concentration inequalities with negative probability in somecertain cases [Nguyen et al., 2021].
Related work
Network architecture search (NAS): The idea of NAS stems from Zoph and Le [2017], while theidea of cell search, i.e., searching core building blocks and composing them together, emerged in
Zoph et al.
The earlier literature used discrete optimization techniques for obtaining thearchitecture.
DARTS Liu et al.
[2019b] considers NAS as a continuous bi-level optimization task.
Recent variants of DARTS
[Xu et al., 2019, Wu et al., 2019] and several train-free methods [Melloret al., 2021, Chen et al., 2021, Xu et al., 2021] have demonstrated success in reducing the searchtime or improving the search algorithm.
However, the aforementioned works have not providedgeneralization guarantees for the optimal architecture.
Optimization and generalization of DNNs via NTK:
In the NTK framework [Jacot et al., 2018, Duet al., 2019, Chen et al., 2020b], the training dynamics of (in)finite-width networks can be exactlycharacterized by kernel tools.
Leveraging NTK facilitates studies on the global convergence of
GD Allen-Zhu et al.
[2019], Du et al.
[2019], Nguyen [2021] in DNNs via the minimum eigenvalueof NTK.
In fact, it also controls the generalization performance of DNNs [Du et al., 2018, Cao and
Gu, 2019, Allen-Zhu et al., 2018], which is further studied in Bietti and Bach [2021].
Problem Settings
In this section we introduce the problem setting of our NAS framework based on the search spaceand algorithm (search strategy) for our paper.
Let X ⊆
R
d be a compact metric space and Y ⊆ R .
We assume that the training set D tr ={ ( x i , y i ) }
Ni =1 is drawn from a probability measure D on X × Y , with its marginal data distributiondenoted by D X .
The goal of a supervised learning task is to find a hypothesis (i.e., a neural networkused in this work)
f : X → Y such that f ( x ; W ) parameterized by W is a good approximation ofthe label y ∈ Y corresponding to a new sample x ∈ X .
In this paper, we consider the classificationtask, evaluated by minimizing the expected risk
[Equation]
where ℓ [ yf ( x ; W )] is the classification loss ℓ ( · ) as a surrogate of the expected 0-1 loss ℓ
[Equation]
( W ) :
= 110 E ( x ,y )
∼D [1 { yf ( x ; W ) < 0 } ] .
In this paper, we employ the cross-entropy loss, , which is defined asℓ ( z ) =
{+log[
 + exp(+} − z )] .
Notation: For an integer L , we use the shorthand [ L ] = { 1 , 2 , . . .
, L } .
The multivariate standard
Gaussian distribution is N ( 0 , I d ) with the zero-mean vector 0 and the identity-variance matrix I d .
We denote the direct sum by ⊕ .
We follow the standard Bachmann–Landau notation in complexitytheory e.g., O , o , Ω , and Θ for order notation.
3.1 Neural Networks and Search Space
In this work, we consider a particular parametrization of f as a deep neural network (DNN) withdepth L ( L ≥ 3 ) 2 which includes the fully-connected (FC) neural networks setting and the residualneural networks setting, and various activation functions in each layer.
This allows for a quite general
NAS setting.
Formally, we define a single-output DNN with the output f l ( x ) in each layer
[Equation]
where the weights of the neural networks are W 1 ∈ R m × d , W l ∈ R m × m , l = 2 , . .
, L − 1 and
W L ∈ R m .
The binary parameter α l is for layer search, and the activation function is σ l ( · ) .
Theneural network output is f ( x ; W ) =
f L ( x ) .
Architecture search: A binary vector α =
[ α 1 , α 2 , · · · , α L − 2 ] ⊤ represents the skip connections,where the α l ∈ { 0 , 1 } in Eq.
(1) indicates whether there is a skip connection in the l -th layer.
Noticethat we unify FC and residual neural networks under the same framework.
Activation function search: We select five representative activation functions used in Eq.
(1),i.e.
, F σ = { ReLU , LeaklyReLU , Sigmoid , Tanh , Swish } , that can be bounded, unbounded,smooth, non-smooth, monotonic, or non-monotonic, as reported in Table 1.
We define σ =[ σ 1 , σ 2 , · · · , σ L − 1 ] ⊤ with σ l ∈ F σ for any l ∈
[ L − 1] as the indicator to show which activation function is selected in each layer.
Importantly, our NAS framework allows for a differentactivation function in each layer, which enlarges the search space.
[Table]
consider the integral
[Table]
Sigmoid to ensure f Sigmoid (0) =
facilitates our theoretical analysis.
The parameter is
[Table]
The definition of f T is similar to f S by using the Tanh function.
Algorithm 1: SGD for training DNNs by NAS
Input: search space S , data D tr = { ( x i , y i ) Ni =1 } , step size γ and Flag method ∈ { EigenNAS , DARTS , · · · } . // conduct NAS algorithms if Flag GuideNAS =
EigenNAS then
Guide NAS from S by our Eigen-NAS algorithm.
else if Flag GuideNAS =
DARTS then
Search neural network architectures from S using the DARTS algorithm.
end if
Output the optimal architecture { ( σ ∗ i )
L − 1 i =1 , ( α ∗ i )
L − 2 i =1 } ∈ S with max probability.
// do neural network training via SGD
Gaussian initialization: W (1) l ∼ N (0 , 1 /m ) ,
l ∈
[ L ]
Construct the neural network f ( x ; W (1) l ) based on { ( σ ∗ i )
L − 1 i =1 , ( α i )
L − i =1 } for i = 1 to N do
W
[Equation]
[Equation]
x
[Equation]
; W i
y i end for
Output Randomly choose ˆ W uniformly from (cid:8) W (1) , . .
, W ( N ) (cid:9) .
In our setting, we conduct the architecture search and the skip connection search independently, andaccordingly, our search space is defined as the direct sum of them
[Equation]
where W :
= ( α , σ , W 1 , . .
, W L ) ∈ W represents the collection of weight matrices and indicatorfor skips and selected activation functions for all layers.
3.2 Algorithm (Search Strategy)
The search strategy is the core part in NAS to pick up the optimal architecture from the search space.
Here we build a general Algorithm 1 combining the search strategy for NAS (the first part) and thesubsequent neural network training by SGD (the second part).
We firstly utilize a typical NAS algorithm, e.g., random search WS [Li and Talwalkar, 2020] or
DARTS 3 , to search skip connections and activation functions independently, which results in theoptimal architecture { ( σ ∗ i )
L − 1 i =1 , ( α ∗ i )
L − 2 i =1 } with the max probability, see sec.
5.1 for details.
Inparticular, Algorithm 1 also allows for the guidance of NAS in a train-free strategy via some specificmetrics, e.g., the minimum eigenvalue of NTK (and its variant), see our Eigen-NAS method in sec.
Then, we conduct neural network training on the selected architecture by SGD.
For ease of theoreticalanalysis, we employ the constant step-size SGD with one epoch and randomly choose the weightparameters during all the iterations, which is commonly used in deep learning theory [Cao and Gu,2019, Zou et al., 2019].
Main result
In this section, we state the main theoretical results.
We present the assumptions used in our proofin sec. 4.1.
Then in sec.
4.2 we provide the recursive form of NTK for DNNs defined by Eq.
(1) withmixed activation functions and skip connections.
The upper and lower bound of the corresponding
NTK in the infinite and finite width setting is given in sec.
4.3 and 4.4, respectively.
Finally, in sec.
4.5,we connect the minimum eigenvalue of NTK and the generalization error bound of DNNs underthese search schemes.
The proofs of our theoretical results presented in this section are deferred to
Appendix B, C, and D, respectively.
4.1 Assumptions
We make the following assumptions on data and activation functions.
Our assumptions are frequentlyemployed in the literature as we highlight below.
Assumption 1.
We assume that the data satisfy ∥ x ∥ 2 = 1 .
Assumption 2.
The activation function σ : R → R satisfies σ ∈ L 2 ( R , e − x 2 / 2 / √ 2 π ) , where
L 2 ( R , g ) denotes the square integrable function.
Assumption 3.
We further assume that x is isotropic.
i.e. E xx ⊤ =
I d /d , where the coefficient 1 /dis to satisfy Assumption 1 at the same time.
Remark: The first assumption on normalized data is commonly used in practice and theory onover-parameterized neural networks [Du et al., 2018, 2019, Allen-Zhu et al., 2019, Oymak and
Soltanolkotabi, 2020, Malach et al., 2020].
The second assumption is general as the studied activationfunctions in Table 1 satisfy it.
The third assumption is standard in statistics [Vershynin, 2018, Section3.2], and used in [Hastie et al., 2022, Klimovsky, 2012, Yaskov, 2014].
It covers Gaussian data,and data uniformly spread on the sphere, commonly used in deep learning theory [Mei et al., 2021,
Ghosh et al., 2021].
4.2 Recursive
NTK for DNNs defined by Eq.
Recall that NTK [Jacot et al., 2018] under the infinite width setting ( m → ∞ ) is the following:
[Equation]
where the NTK matrix for residual networks is derived by the following regular chain rule.
Lemma 1.
For any l ∈ [3 , L ] and s ∈ [2 , L ] , denote
[Equation]
Then the NTK for residual networks defined in Eq.
(1) can be written as
[Equation]
Remark: ( i ) Our NTK formula of ResNet differs from the one of Tirer et al.
[2021], Huang et al.[2020], Belfer et al.
[2021] in two critical ways: 1) each skip-layer in our model skips one fullyconnected layer and one activation function, as opposed to the two-layer skip of previous works, 2)our formulation does not require every layer to have a parallel skip connection, which increases the flexibility of the network.
Those
differences also result in a different NTK matrix.
( ii ) Our NTK formulation covers different activation functions, and we adopt the same initialization
(coefficient) on them to ensure fair/equal search in our NAS framework
Lemma 1 is general to cover FC and residual neural networks, which facilitates the analysis ofminimum eigenvalue of NTK under the unifying framework.
If α l = 0 for l ∈ [ L − 1] , our NTKformulation for residual neural networks degenerates to that of fully-connected neural network, and
A l and G l become equal.
4.3 Minimum Eigenvalue of NTK for infinite width
We are now ready to state the main result of infinite width neural network.
We provide the errorbounds of minimum eigenvalue of NTK for infinite width neural network mixed with five differentactivation functions.
The main differences between different activation functions are illustratedin Table 1.
Theorem 1.
For a DNN defined by Eq.
(1) and a not very large L , let K ( L ) be the limiting NTKrecursively defined in Lemma 1.
Then, under Assumptions 1 and 3, with probability at least 1 − e − d ,we have
[Equation]
[Equation]
where µ 1 ( σ 1 ) is the 1 -st Hermite coefficient of the first layer activation function, and β 1 , β 2 , β 3 arethree constants on various activation functions defined in Table 1.
Remark:
Here we use a not very large L ≤ 10 , because in NAS, the users can directly specify thedepth L (in a constant order) and in practice it is often less than 10 [Liu et al., 2018, Dong et al.,2021].
In fact, existing NAS algorithms such as DARTS tend to have architectures with wide andshallow cell structure as suggested by Shu et al.
Theorem 1 shows the upper and lower boundof the minimum eigenvalue of NTK under the mix of activation functions and skip connections.
Thefollowing conclusions can be drawn from our result:
The error bounds of minimum eigenvalue significantly depend on the depth of the network L
, the skip connections via α p , which makes the
minimum eigenvalue increasing fast as L and the
numberof skip connections increase.
Besides, the
minimum eigenvalue
is also effected by activation
functions via
β β β
Nevertheless, the lower bound is independent of β and β 2 .
Different activation functions lead to different tendency (increase or decrease) on λ min ( K ( L ) ) .
Asthe depth increases, the lower bound λ min ( K ( L ) ) under ReLU remains unchanged, increases under LeakyReLU
and decreases when Sigmoid, Tanh or Swish applied, which brings in new findings when compared to Nguyen et al.
sorely on ReLU
For the upper bound for λ min ( K ( L ) ) , we can see our results are positively correlated with the depth L .
One can see that µ 1 ( σ 1 ) is only related to the activation function of the first layer, which implies that the activation function in the first layer is very important as λ min ( K ( L ) ) largely depends on it.
4.4 Minimum Eigenvalue of NTK for finite width
To study the finite width, we firstly introduce the Jacobian of the network.
Let
F =
[ f ( x 1 ) , . .
f ( x N )] T .
Then, the Jacobian J of F with respect to W is J = 220 (cid:104)
[Equation]
) (cid:105) , where J have dimension R ((( L − 2) × m
+ d +1)
× m × N .
The empirical Neural
Tangent Kernel (NTK) matrix can be defined as ¯ K
[Equation]
Accordingly, we generalize Theorem 1 from the infinite-width to finite-width setting below.
Theorem 2.
For an L -layer network defined by Eq.
(1) , let K ( L ) =
JJ ⊤ be the NTK matrix, and theweights of the network be initialized as [ W l ] i,j ∼ N (0 , 1 /m ) , for all l ∈ [ L ] .
Under Assumptions 1and 3, then λ min ( JJ ⊤ ) can be bounded by
[Equation]
where the first inequality (lower bound) holds with probability at least 1 − e − d − 227 (cid:80) L −l =1 exp( − Ω( m )) − exp( − Ω(1)) and the second inequality (upper bound) holds with probability at least 1 − (cid:80) L − 1 l =1 exp( − Ω( m ))
− exp( − Ω(1)) .
The definitions of β 2 , and β 3 are the same asthose in Theorem 1.
Remark:
Theorem 2 achieves a similar result as Theorem 1 if the width m is large.
The onlydifference in the findings is that probability that the Theorem holds increases as m increases.
4.5 Connection to Generalization Error Bound
Based on the above error bound of the minimum eigenvalue of NTK under different settings, here weestablish its relationship with the generalization error of DNNs.
We provide a bound on the expected0-1 error obtained by Algorithm 1.
Theorem 3.
Given a DNN defined by Eq.
(1) with y =
( y 1 , . . .
, y N ) ⊤ determined by Algorithm 1with
the step size of SGD γ = κC 1 · (cid:112) y ⊤ ( K ( L ) ) − 1 y / ( m √ N ) for some small enough absoluteconstant κ .
Under Assumptions 1, 2 and 3, for any δ ∈ (cid:0) 0 , e − 1 (cid:3) and a not very large L , if the widthm ≥ ˆ m , where ˆ m depends on λ min ( K ( L ) ) , δ, N , and L , then with probability at least 1 − δ over therandomness of W (1) , we obtain the following high probability bound:
[Equation]
where C 1 and C 2 are two constants only depending on L .
Remark:
Theorem 3 gives an algorithm-dependent generalization error bound of DNNs definedby Eq.
(1) trained with SGD with different activation functions and skip connections.
If m is largeenough, the learning rate is infinitesimal, which means the generalization error bound mainly dependson the NTK matrix, similarly to Cao and Gu [2019], Du et al.
According to Theorem 3, the generalization performance of DNNs is controlled by the minimumeigenvalue of the NTK matrix, which is in turn affected by different activation functions and skipconnections, as discussed in Theorem 1.
Apart from the NTK matrix itself, the condition m ≥ ˆ mis also effected by different activation functions, which implies that the required minimum width isdifferent in these cases.
4.6 Proof sketch
Our work extends the proofs of Nguyen and Mondelli [2020], Cao and Gu [2019] beyond ReLU,which is critical for enabling search across activations.
The extension to other activation functionsand skip connections is non-trivial due to non-linearity, inhomogeneity and nonmonotonicity.
To derive the error bound on the minimum eigenvalue, we start from Lemma 1 on the NTK formulaunder the mixed activation functions and skip connections, and we transform the minimum eigenvalueestimation to the computation (estimation) of the bound G , ˙ G ( λ min ( G ) ).
The infinite width andfinite width are included in Appendix B and C respectively.
For the upper bound, we estimatethe diagonal elements of G and use the minimum eigenvalue less than the mean of the diagonalelements for proof.
For the lower bound, we use Hermite expansion and [Yaskov, 2014, Corollary3.1].
Combining these results concludes the proof.
To derive the generalization error bounds, we need a series of lemmas (see Appendix D).
If the inputweights are close, the output of each neuron with any activation function does not change too much(see Lemma 7); If the initilizations are close, the neural network output f ( x ; W ) is almost linearin W (see Lemma 8), and the loss function ℓ [ y i f ( x i ; W )] is almost a convex function of W forany i ∈
[ N ] (see Lemma 9).
Accordingly, the gradient and loss of the neural network can be upperbounded by Lemma 10 and 11, respectively, which concludes the proof by combining with somerelevant results [Cao and Gu, 2019, Allen-Zhu et al., 2019].
Further discussion on the differences isdeferred to Appendix E.
Numerical Validation
To validate our theoretical results, we conduct a series of experiments on NAS.
Firstly, we simulatethe NTK matrices under different depths in Appendix F.4 to verify the relationship between theminimum eigenvalue of NTK and the network depth L in Theorem 1.
In sec.
5.1 we use the DARTSalgorithm [Liu et al., 2019b] to conduct experiments on activation function search and skip connectionsearch under the search space of Eq.
Finally, we use the minimum eigenvalue of NTK to guidethe training of NAS on the benchmark NAS-Bench-
[Dong and Yang, 2020], with a comparison ofrecent NAS algorithms.
Additional experiments on NAS-Bench-101 [Ying et al., 2019] and transferlearning are deferred to Appendix F.5 and F.6.
          /D\HU 5 H/  7 DQK 6 L J P R L G /HD N\ 5 H/  6 Z L V K $ F W L Y D W L RQ I XQ F W L RQ              
      
                      
               (a) activation functions σ       9 10 Layer 0.
P r o b a b ili t y [Figure] [Table]
al [Table]
[Table]
Le [Table] [Table]
5.1 DARTS experiment
In this section we employ a typical NAS algorithm, DARTS Liu et al.
[2019b], to assess our theoreticalresults on activation functions and skip connections. We select Fashion-MNIST
[Xiao et al., 2017] asa standard benchmark. Details about Fashion-MNIST are shared in Appendix F.1. Search space and search strategy:
Our search space is defined by Eq. (2) on skip connections,activation functions, and weight parameters.
We follow the search strategy with Liu et al.
[2019b]in a two-level scheme, one is for weight parameter search W and the other is for architecturesearch { α , σ } , which results in the final optimal architecture { α ∗ , σ ∗ , W ∗ } .
Different from Liu et al.[2019b], the activation function search and the skip connection search in our setting is independent.
Toobtain σ ∗ , we use the softmax function to normalize their weights and choose the specific activationfunction with the highest probability in each layer.
To obtain α ∗ , we initialize each entry α l = 1 / 2( l ∈
[ L − 2] ), constrain it to [0 , 1] during training, and retain the skip connection when α ∗ l > 1 / 2 .
NAS Results: We conduct the NAS experiment via DARTS on a feedforward neural network with
L = 10 and m = 1024 , with 5 runs.
After training, the probability of these activation functions andskip connections in each layer is reported in Figure 1(a) and 1(b), respectively.
We have the followingfindings: Firstly, after the search process, LeakyReLU and ReLU are selected as the activationswith the highest probability in each layer.
This coincides with our theoretical results in Theorem 1.
One minor difference is that the probability of LeakyReLU is slightly inferior to ReLU in practice.
The reason behind this could be the sparsity of ReLU [de Dios and Bruna, 2020].
Secondly, inthe first layer, we observe the largest difference on the probability of various activation functions.
As the network becomes deeper, such difference pales in importance until there is no differenceamong these activation functions in the last layer.
This phenomenon matches our theory well.
To bespecific, in Theorem 1, our result on the minimum eigenvalue largely depends on the first layer andits Hermite coefficient.
Besides, this result also provides a justification on omitting the high-orderterms while retaining the first layer activation terms.
Thirdly, for the skip search result, we find thatthe skip connections are required in each layer when L ≤ 10 , as suggested by our theoretical resultsin Theorem 1.
It also verifies results of Zhou et al.
We expect that the skip connections mightnot be required in each layer for deep neural networks, since their capacity can already be enough [Heet al., 2016]; but we defer the related study to a future work.
Interestingly, the search strategy favors the activation functions and the skip connections with largerminimum eigenvalue of NTK, which enjoy better generalization performance.
This result alsomotivates us to study the following question: can the minimum eigenvalue of NTK guide the searchprocess in NAS?
We provide an affirmative answer in the next section with experimental validations.
5.2 NAS-Bench-201 Experiment
In this experiment, we use the minimum eigenvalue to guide NAS on NAS-Bench-
[Dong and
Yang, 2020].
Each experiment is repeated 5 times, while it can run on a single GPU in a few hours.
Benchmark and baselines: NAS-Bench-201 [Dong and Yang, 2020] is a commonly used benchmarkfor NAS algorithm evaluation, which includes three datasets: a) CIFAR-10 [Krizhevsky et al., 2014],b)
CIFAR-
[Krizhevsky et al., 2014] and c)
ImageNet-16 [Chrabaszcz et al., 2017] for imageclassification.
Details on the datasets exist in Appendix F.1.
Apart from that, we evaluate the proposedapproach with some baselines including ResNet, RL based algorithm and so on.
Algorithm procedure:
Our algorithm, called Eigen-NAS, also belongs in the train-free category.
Eigen-NAS follows KNAS, which leverages the minimum eigenvalue of NTK to guide NAS.
However, due to the O ( N 3 ) time complexity of computing these eigenvalues, KNAS insteadcomputes ∥ K ∥ F .
However, from the expression λ min ( K ) ≤ 1 d (cid:80) Ni =1 K ii ≤ ∥ K ∥ F we utilize thefirst inequality in Eigen-NAS to obtain a tighter (and more computationally efficient) bound toλ min .
The computation cost of our method is O ( N ) , which is less than computing the Frobeniusnorm ( O ( N 2 ) ).
Sequentially, the top- k best candidates architectures are chosen in KNAS and our
Eigen-NAS, and then the best architecture is chosen by the validation error, refer to the resultsin Table 2.
Due to the page limit, the algorithm is located in Appendix F.
Results: The experimental results in Table 2 verify that Eigen-NAS guided by the proposed metricabove achieves the best performance on both the CIFAR-100 and ImageNet-16 datasets, and competitive performance on CIFAR-10, outperforming KNAS in all three cases when k = 20 for bothmethods.
Even when we consider a smaller k = 5 , Eigen-NAS can outperform KNAS, which weattribute to the more precise minimum eigenvalue estimation.
Conclusion and discussion
In this work, we explore the relationship between the minimum eigenvalue of NTK and neuralarchitecture search.
We derive upper and lower bounds on the minimum eigenvalues of NTK for(in)finite residual networks under different mixtures of activation functions, and establish a connectionbetween the minimum eigenvalues and the generalization properties of NAS.
Our theoretical resultson various activation functions and mixed activation cases can also be a tool for deep learning theoryresearchers to prove generic results rather than studying a single architecture, e.g., ReLU networks.
In addition, we use the minimum eigenvalue as a guide for the training of NAS in a train-free method,which greatly exceeds the efficiency of the classic NAS algorithm.
When compared with existingtrain-free methods, our algorithm, called Eigen-NAS, achieves a higher accuracy.
We posit that thiswill be useful for studying computationally efficient methods on NAS.
Limitations :
A core limitation is whether our proof framework can cover more general structuresin NAS, such as the most commonly used convolutional neural networks (CNNs).
Even thoughthis seems possible, this is non-trivial due to the tensors that emerge.
To be specific, it requires theelement-recursive form of NTK matrices in Arora et al.
[2019b] to be transformed into a globalrecursive form (similar to Lemma 1), then analyze its minimum eigenvalue.
Besides, the contractionoperation of tensors, the locality and boundary effects of convolutional layer in CNNs make theanalysis difficult.
Therefore, we believe this is a topic on its own right.