NASI: L ABEL - AND D ATA - AGNOSTIC N EURAL A RCHITECTURE S EARCH AT I NITIALIZATION
Recent years have witnessed a surging interest in Neural Architecture Search (NAS).
Various algorithms have been proposed to improve the search efﬁciency and the search effectiveness of NAS, i e., to reduce the search cost and improve the generalization performance of the selected architectures, respectively.
However, the search efﬁciency of these algorithms is severely limited by the need for model training during the search process.
To overcome this limitation we propose a novel NAS algorithm called NAS at Initialization (NASI) that exploits the capability of a Neural
Tangent Kernel in being able to characterize the performance of candidate architectures at initialization, hence allowing model training to be completely avoided to boost the search efﬁciency.
Besides the improved search efﬁciency, NASI also achieves competitive search effectiveness on various datasets like CIFAR-10/ and ImageNet.
Further, NASI is shown to be label- and data-agnostic under mild conditions which guarantees the transferability of the architectures selected by our
NASI over different datasets
I NTRODUCTION
The past decade has witnessed the wide success of deep neural networks (DNNs) in computer vision and natural language processing.
These DNNs, e.g., VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), and MobileNet (Howard et al., 2017), are typically handcrafted by human experts with considerable trials and errors.
The human efforts devoting to the design of these DNNs are, however, not affordable nor scalable due to an increasing demand of customizing DNNs for different tasks.
To reduce such human efforts, Neural Architecture Search (NAS) (Zoph & Le, 2017) has recently been introduced to automate the design of DNNs.
As summarized in (Elsken et al., 2019),
NAS conventionally consists of a search space, a search algorithm, and a performance evaluation.
Speciﬁcally, the search algorithm aims to select the best-performing neural architecture from the search space based on its evaluated performance via the performance evaluation.
In the literature, various search algorithms (Luo et al., 2018; Zoph et al., 2018; Real et al., 2019) have been proposed to search for architectures with comparable or even better performance than the handcrafted ones.
However, these NAS algorithms are inefﬁcient due to the requirement of model training for numerous candidate architectures during the search process.
To improve the search inefﬁciency, one-shot NAS algorithms (Pham et al., 2018; Dong & Yang, 2019; Liu et al., 2019; Xie et al., 2019) have trained a single one-shot architecture and then evaluated the performance of candidate architectures with model parameters inherited from this ﬁne-tuned one-shot architecture.
So, these algorithms can considerably reduce the cost of model training in NAS, but still require the training of the one-shot architecture.
This naturally begs the question whether NAS is realizable at initialization such that model training can be completely avoided during the search process?
To the best of our knowledge, only a few empirical efforts to date have been devoted to developing NAS algorithms without model training (Mellor et al., 2021;
Park et al., 2020; Abdelfattah et al., 2021; Chen et al., 2021).
This paper presents a novel NAS algorithm called NAS at Initialization (NASI) that can completely avoid model training to boost search efﬁciency.
To achieve this, NASI exploits the capability of a Neural Tangent Kernel (NTK)
(Jacot et al., 2018; Lee et al., 2019a) in being able to formally characterize the performance of inﬁnite-wide DNNs at initialization, hence allowing the performance of candidate architectures to be estimated and realizing NAS at initialization.
Speciﬁcally, given the estimated performance of candidate architectures by NTK, NAS can be reformulated into an optimization problem without model training (Sec. 3.1).
However, NTK is prohibitively costly to evaluate.
Fortunately, we can approximate it 1 using a similar form to gradient ﬂow (Wang et al., 2020)
(Sec. 3.2).
This results in a reformulated NAS problem that can be solved efﬁciently by a gradient-based algorithm via additional relaxation using Gumbel-Softmax (Jang et al., 2017)
(Sec.
Interestingly, NASI is even shown to be label- and data-agnostic under mild conditions, which thus implies the transferability of the architectures selected by NASI over different datasets (Sec. 4).
We will ﬁrstly empirically demonstrate the improved search efﬁciency and the competitive search effectiveness achieved by NASI in NAS-Bench-1Shot
(Zela et al., 2020b) (Sec. 5.1).
Compared with other NAS algorithms, NASI incurs the smallest search cost while preserving the competitive performance of its selected architectures.
Meanwhile, the architectures selected by NASI from the
DARTS (Liu et al., 2019) search space over CIFAR-10 consistently enjoy the competitive or even outperformed performance when evaluated on different benchmark datasets, e.g., CIFAR-10/ and ImageNet (Sec. 5.2), indicating the guaranteed transferability of architectures selected by our
NASI.
In Sec. 5.3, NASI is further demonstrated to be able to select well-performing architectures on CIFAR-10 even with randomly generated labels or data, which strongly supports the label- and data-agnostic search and therefore the guaranteed transferability achieved by our NASI.
R ELATED W ORKS AND B ACKGROUND 2. N EURAL A RCHITECTURE S EARCH
A growing body of NAS algorithms have been proposed in the literature (Zoph & Le, 2017; Liu et al., 2018; Luo et al., 2018; Zoph et al., 2018; Real et al., 2019) to automate the design of neural architectures.
However, scaling existing NAS algorithms to large datasets is notoriously hard.
Recently, attention has thus been shifted to improving the search efﬁciency of NAS without sacriﬁcing the generalization performance of its selected architectures.
In particular, a one-shot architecture is introduced by Pham et al.
(2018) to share model parameters among candidate architectures, thereby reducing the cost of model training substantially.
Recent works (Chen et al., 2019; Dong & Yang, 2019; Liu et al., 2019; Xie et al., 2019; Chen & Hsieh, 2020; Chu et al., 2020) along this line have further formulated NAS as a continuous and differentiable optimization problem to yield efﬁcient gradient-based solutions.
These one-shot NAS algorithms have achieved considerable improvement in search efﬁciency.
However, the model training of the one-shot architecture is still needed.
A number of algorithms are recently proposed to estimate the performance of candidate architectures without model training.
For example, Mellor et al.
(2021) have explored the correlation between the divergence of linear maps induced by data points at initialization and the performance of candidate architectures heuristically.
Meanwhile, Park et al.
(2020) have approximated the performance of candidate architectures by the performance of their corresponding Neural Network Gaussian Process (NNGP) with only initialized model parameters, which is yet computationally costly.
Abdelfattah et al.
(2021) have investigated several training-free proxies to rank candidate architectures in the search space, while Chen et al.
(2021) intuitively adopt theoretical aspects in deep networks (e.g., NTK (Jacot et al., 2018) and linear regions of deep networks (Raghu et al., 2017)) to select architectures with a good trade-off between its trainability and expressivity.
Our NASI signiﬁcantly advances this line of work in (a) providing theoretically grounded performance estimation by NTK (compared with (Mellor et al., 2021; Abdelfattah et al., 2021; Chen et al., 2021)), (b) guaranteeing the transferability of its selected architectures with its provable label- and data-agnostic search under mild conditions (compared with (Mellor et al., 2021;
Park et al., 2020; Abdelfattah et al., 2021; Chen et al., 2021))) and (c) achieving SOTA performance in a large search space over various benchmark datasets (compared with (Mellor et al., 2021;
Park et al., 2020; Abdelfattah et al., 2021)).
Based on our results, Shu et al.
(2022) recently have provided further improvement on existing training-free NAS algorithms.
2. N EURAL T ANGENT K ERNEL (NTK)
Let a dataset ( X , Y ) denote a pair comprising a set X of m n 0 -dimensional vectors of input features and a vector Y ∈ R mn × 1 concatenating the m n -dimensional vectors of output values, respectively.
Let a DNN be parameterized by θ t ∈ R p at time t and output a vector f ( X ; θ t )
∈ R mn × 1 (abbreviated to f t ) of the predicted values of Y .
Jacot et al.
(2018) have revealed that the training dynamics of DNNs with gradient descent can be characterized by an NTK.
Formally, deﬁne the NTK Θ t ( X , X ) ∈ R mn × mn (abbreviated to Θ t ) as
[Equation]
Given a loss function L t at time t and a learning rate η , the training dynamics of the DNN can then be characterized as
[Equation]
Interestingly, as proven in (Jacot et al., 2018), the NTK stays asymptotically constant during the course of training as the width of DNNs goes to inﬁnity.
NTK at initialization (i.e., Θ 0 ) can thus characterize the training dynamics and also the performance of inﬁnite-width DNNs.
Lee et al. (2019a) have further revealed that, for DNNs with over-parameterization, the aforementioned training dynamics can be governed by their ﬁrst-order Taylor expansion (or linearization) at initialization.
In particular, deﬁne
[Equation]
for all x ∈ X .
Then, f ( x ; θ t ) and f lin ( x ; θ t ) share similar training dynamics over time, as described formally in Appendix A.2.
Besides, following the deﬁnition of NTK in (1), this linearization f lin achieves a constant NTK over time.
Given the mean squared error (MSE) loss deﬁned as L t (cid:44) m − 1 (cid:107)Y − f ( X ; θ t ) (cid:107) 22 and the constant
NTK Θ t =
Θ 0 , the loss dynamics in (2) above can be analyzed in a closed form while applying gradient descent with learning rate η (Arora et al., 2019):
[Equation]
where Θ 0 = (cid:80) mni =1 λ i ( Θ 0 )
u i u (cid:62) i , and λ i ( Θ 0 )
and u i denote the i -th largest eigenvalue and the corresponding eigenvector of Θ 0 , respectively.
N EURAL A RCHITECTURE S EARCH AT I NITIALIZATION 3. R EFORMULATING NAS VIA NTK
Given a loss function L and the model parameters θ ( A ) of architecture A , we denote the training and validation loss as L train and L val , respectively.
NAS is conventionally formulated as a bi-level optimization problem (Liu et al., 2019):
[Equation]
[Equation]
Notably, model training is required to evaluate the validation performance of each candidate architecture in (5).
The search efﬁciency of NAS algorithms (Real et al., 2019; Zoph et al., 2018) based on (5) is thus severely limited by the cost of model training for each candidate architecture.
Though recent works (Pham et al., 2018) have considerably reduced this training cost by introducing a one-shot architecture for model parameter sharing, such a one-shot architecture still requires model training and hence incurs training cost.
To completely avoid this training cost, we exploit the capability of a NTK in characterizing the performance of DNNs at initialization.
Speciﬁcally, Sec. 2.2 has revealed that the training dynamics of an over-parameterized DNN can be governed by its linearization at initialization.
With the MSE loss, the training dynamics of such linearization are further determined by its constant NTK.
Therefore, the training dynamics and hence the performance of a DNN can be characterized by the constant NTK of its linearization.
However, this constant NTK is computationally costly to evaluate.
To this end, we instead characterize the training dynamics (i.e., MSE) of DNNs in Proposition 1 using the trace norm of NTK at initialization, which fortunately can be efﬁciently approximated.
For simplicity, we use this MSE loss in our analysis.
Other widely adopted loss functions (e.g., cross entropy with softmax) can also be applied, as supported in our experiments.
Note that throughout this paper, the parameterization and initialization of DNNs follow that of Jacot et al.
For a L -layer DNN, we denote the output dimension of its hidden layers and the last layer as n 1 = · · · = n L − 1 = k and n L = n , respectively.
Proposition 1.
Suppose that (cid:107) x (cid:107) 2 ≤ 1 for all x ∈ X and Y ∈ [0 , 1] mn for a given dataset ( X , Y ) of size |X| =
m , a given L -layer neural architecture A outputs f t ∈ [0 , 1] mn as predicted labels of Y with the corresponding MSE loss L t , λ min ( Θ 0 ) > 0 for the given NTK Θ 0 w.r.t.
f t at initialization, and gradient descent (or gradient ﬂow) is applied with learning rate η < λ − 1max ( Θ 0 ) .
Then, for any t ≥ 0 , there exists a constant c 0 > 0 such that as k → ∞ ,
[Equation]
with probability arbitrarily close to 1 where q is set to 2 t if t < 0 . 5 , and 1 otherwise, λ ( Θ 0 ) (cid:44) ( mn ) − 1 (cid:80) mni =1 λ i ( Θ 0 ) , and (cid:15) (cid:44) 2 c 0 (cid:112) n/ ( mk ) (cid:16) 1 + c 0 (cid:112) 1 /k (cid:17) .
Its proof is in Appendix A.3.
Proposition 1 implies that NAS can be realizable at initialization.
Speciﬁcally, given a ﬁxed and sufﬁciently large training budget t , in order to select the best-performing architecture, we can simply minimize the upper bound of L t in (6) over all the candidate architectures in the search space.
Since both theoretical (Mohri et al., 2018) and empirical (Hardt et al., 2016) justiﬁcations in the literature have shown that training and validation loss are generally highly related, we simply use L t to approximate L val .
Hence, (5) can be reformulated as
[Equation]
Note that the constraint in (7) is derived from the condition η < λ − 1max ( Θ 0 ( A )) in Proposition 1, and η and (cid:15) are typically constants 2 during the search process.
Following the deﬁnition of trace norm, (7) can be further reduced into
[Equation]
Notably, Θ 0 ( A ) only relies on the initialization of A .
So, no model training is required in optimizing (8), which achieves our objective of realizing NAS at initialization.
Furthermore, (8) suggests an interesting interpretation of NAS: NAS intends to select architectures with a good trade-off between their model complexity and the optimization behavior in their model training.
Particularly, architectures containing more model parameters will usually achieve a larger (cid:107) Θ
( A ) (cid:107)
tr according to the deﬁnition in (1), which hence provides an alternative to measuring the complexity of architectures.
So, maximizing (cid:107) Θ
( A ) (cid:107)
tr leads to architectures with large complexity and therefore strong representation power.
On the other hand, the complexity of the selected architectures is limited by the constraint in (8) to ensure a well-behaved optimization with a large learning rate η in their model training.
By combining these two effects, the optimization of (8) naturally trades off between the complexity of the selected architectures and the optimization behavior in their model training for the best performance.
Appendix C.1 will validate such trade-off.
Interestingly, Chen et al.
(2021) have revealed a similar insight of NAS to us.
3. A PPROXIMATING THE T RACE N ORM OF NTK
The optimization of our reformulated NAS in (8) requires the evaluation of (cid:107) Θ 0 ( A ) (cid:107) tr for each architecture A in the search space, which can be obtained by
[Equation]
where (cid:107) · (cid:107) F denotes the Frobenius norm.
However, the Frobenius norm of the Jacobian matrix in (9) is costly to evaluate.
So, we propose to approximate this term.
Speciﬁcally, given a γ -Lipschitz continuous loss function 3 L x (i.e., (cid:107)∇ f L x (cid:107) 2 ≤ γ for all x ∈ X ),
[Equation]
The Frobenius norm (cid:13)(cid:13) ∇ θ
( A ) f
( x , θ
( A ))
(cid:13)(cid:13) F can thus be approximated efﬁciently using its lower bound in (10) through automatic differentiation (Baydin et al., 2017).
Meanwhile, the evaluation of (cid:107) Θ
( A ) (cid:107) tr in (9) requires iterating over the entire dataset of size m , which thus incurs O ( m ) time.
Fortunately, this incurred time can be reduced by parallelization over mini-batches.
Let the set X j denote the input feature vectors of the j -th randomly sampled mini-batch of size |X
j | = b .
By combining (9) and (10),
[Equation]
[Equation]
where the last inequality follows from Jensen’s inequality.
Note that (11) provides an approximation of (cid:107) Θ 0 ( A ) (cid:107) tr incurring O ( m/b ) time because the gradients within a mini-batch can be evaluated in parallel. In practice
[Equation] we further approximate the summation over m/b mini-batches in (11) by using only one single uniformly randomly sampled mini-batch X j .
Formally, under the deﬁnition of (cid:107) [-(cid:101) Θ 
-][Equation]
( A
[Equation]
A
[Equation]
x
[Equation]
22 , our ﬁnal approximation of (cid:107) Θ 0 ( A ) (cid:107) tr becomes
[Equation]
This ﬁnal approximation incurs only O (1) time and can also characterize the performance of neural architectures effectively, as demonstrated in our experiments.
Interestingly, a similar form to (12), called gradient ﬂow (Wang et al., 2020), has also been applied in network pruning at initialization.
3. O PTIMIZATION AND S EARCH A LGORITHM
The approximation of (cid:107) Θ 0 ( A ) (cid:107) tr in Sec.
3.2 engenders an efﬁcient optimization of our reformulated
NAS in (8):
Firstly, we apply a penalty method to transform (8) into an unconstrained optimization problem.
Given a penalty coefﬁcient µ and an exterior penalty function F ( x ) (cid:44) max(0 , x ) with a pre-deﬁned constant ν (cid:44) γnη − 1 , and a randomly sampled mini-batch X j , by replacing (cid:107) Θ 0 ( A ) (cid:107) tr with the approximation in (12), our reformulated NAS problem (8) can be transformed into
[Equation]
Interestingly, (13) implies that the complexity of the ﬁnal selected architectures is limited by not only the constraint ν (discussed in Sec. 3.1) but also the penalty coefﬁcient µ : For a ﬁxed constant ν , a larger µ imposes a stricter limitation on the complexity of architectures (i.e., (cid:107) (cid:101) Θ
( A ) (cid:107) tr < ν ) in the optimization of (13).
The optimization of (13) in the discrete search space, however, is intractable.
So, we apply certain optimization tricks to simplify it:
Following that of Liu et al.
; Xie et al.
(2019), we represent the search space as a one-shot architecture such that candidate architectures in the search space can be represented as the sub-graphs of this one-shot architecture.
Next, instead of optimizing (13), we introduce a distribution p
α
( A ) (parameterized by α ) over the candidate architectures in this search space like that in (Zoph & Le, 2017; Pham et al., 2018; Xie et al., 2019) to optimize the expected performance of architectures sampled from p α ( A ) :
[Equation]
Then, we apply Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) to relax the optimization of (14) to be continuous and differentiable using the reparameterization trick.
Speciﬁcally, for a given α , to sample an architecture A , we simply need to sample g from p ( g ) =
Gumbel ( 0 , 1 ) and then determine A using α and g (more details in Appendix B.3).
Consequently, (14) can be transformed into
[Equation]
After that, we approximate (15) based on its ﬁrst-order Taylor expansion at initialization such that it can be optimized efﬁciently through a gradient-based algorithm.
In particular, given the ﬁrst-order approximation within the ξ -neighborhood of initialization α 0 (i
[Equation]
e., (cid:107) ∆ (cid:107)
[Equation]
≤ ξ ):
[Equation]
16) the maximum of (16) is achieved when
[Equation]
1: Input: dataset D (cid:44) ( X , Y ) , batch size b , steps T , penalty coefﬁcient µ , constraint constant ν , initialized model parameters θ 0 for one-shot architecture and distribution p α 0 ( A ) with initialization α 0 = 0 , set ξ =2: for step t = 1 , . .
, T do 3:
Sample data D t ∼ D of size b 4:
Sample g t ∼ p ( g ) =
Gumbel ( 0 , 1 ) and determine sampled architecture A t based on α 0 , g t 5:
Evaluate gradient G t = ∇ α 0 R ( A t ) with data D t 6: end for 7: Estimate ∆ ∗ with (18) and get α ∗ =
α 0 + ∆ ∗ 8: Select architecture A ∗ = arg max A p α ∗ ( A )
[Figure]
vs. true data. Each pair ( x, y ) in the plots denotes the approximation for one candidate architecture in the search space with true vs. random labels (or data), respectively.
The trends of these approximations are further illustrated by the lines in orange.
In addition, Pearson correlation coefﬁcient ρ of the approximations with random vs. true labels (or data) is given in the corner.
The closed-form solution in (17) follows from the deﬁnition of dual norm and requires only a one-step optimization, i.e., without the iterative update of ∆ .
Similar one-step optimizations have also been adopted in (Goodfellow et al., 2015; Wang et al., 2020).
Unfortunately, the expectation in (17) makes the evaluation of ∆ ∗ intractable.
Monte Carlo sampling is thus applied to estimate ∆ ∗ efﬁciently: Given T sequentially sampled g (i.e., g 1 , . .
. , g T ) and let
G i (cid:44) ∇ α 0 R ( A ( α 0 , g i )) , ∆ ∗ can be approximated as
[Equation]
Note that inspired by AMSGrad (Reddi et al., 2018), the expectation (cid:13)(cid:13)
E g ∼
[Equation]
[Equation] ( A [Equation] g ))]
(cid:13)(cid:13)in (17) has been approximated using max( (cid:107) G 1 (cid:107) 2 , . . .
, (cid:107) G t (cid:107) 2 ) in (18) based on a sample of g at time t .
Interestingly, this approximation is non-decreasing in t and therefore achieves a similar effect of learning rate decay, which may lead to a better-behaved optimization of ∆ ∗ .
With the optimal ∆ ∗ and α ∗
=
α 0 + ∆ ∗ , the ﬁnal architecture can then be selected as A ∗ (cid:44)
[Equation] ) , which ﬁnally completes our NAS at Initialization (NASI) algorithm detailed in Algorithm 1.
Interestingly, this simple and efﬁcient solution in (18) can already allow us to select architectures with competitive performances, as shown in our experiments (Sec. 5).
L ABEL - AND D ATA - AGNOSTIC S EARCH OF NASI
Besides the improved search efﬁciency by completely avoiding model training during search, NASI can even guarantee the transferability of its selected architectures with its provable label- and dataagnostic search under mild conditions in Sec.
4.1 and Sec. 4.2, respectively.
That is, under this provable label- and data-agnostic search, the ﬁnal selected architectures on a proxy dataset using NASI are also likely to be selected and hence guaranteed to perform well on the target datasets.
So, the transferability of the architectures selected via such label- and data-agnostic search can be naturally guaranteed, which will also be validated in Sec.
5 empirically.
4. L ABEL -A GNOSTIC S EARCH
Our reformulated NAS problem (8) has explicitly revealed that it can be optimized without the need of the labels from a dataset.
Though our approximation of (cid:107) Θ
( A ) (cid:107) tr in (12) seemingly depends on the labels of a dataset, (12) can, however, be derived using random labels.
This is because the Lipschitz continuity assumption on the loss function required by (10), which is necessary for the derivation of (12), remains satisﬁed when random labels are used.
So, the approximation in (12) (and hence our optimization objective (13) that is based on this approximation) is label-agnostic, which hence justiﬁes the label-agnostic nature of NASI.
Interestingly, NAS algorithm achieving a similar label-agnostic search has also been developed in (Liu et al., 2020), which further implies the reasonableness of such label-agnostic search.
The label-agnostic approximation of (cid:107) Θ 0 ( A ) (cid:107) tr is demonstrated in Fig.
1a using the three search spaces of NAS-Bench-1Shot1 with randomly selected labels.
According to Fig.
1a, the large Pearson correlation coefﬁcient (i.e., ρ ≈ 1 ) implies a strong correlation between the approximations with random vs. true labels, which consequently validates the label-agnostic approximation of (cid:107) Θ 0 ( A ) (cid:107) tr .
Overall, these empirical observations have veriﬁed that the approximation of (cid:107) Θ 0 ( A ) (cid:107) tr and hence
NASI based on the optimization over this approximation are label-agnostic, which will be further validated empirically in Sec.
4. D ATA -A GNOSTIC S EARCH
Besides being label-agnostic, NASI is also guaranteed to be data-agnostic.
To justify this, we prove in Proposition 2 (following from the notations in Sec. 3.1) below that (cid:107) Θ
( A ) (cid:107) tr is data-agnostic under mild conditions.
Proposition 2.
Suppose that x ∈ R n 0 and (cid:107) x (cid:107) 2 ≤ 1 for all x ∈ X given a dataset ( X , Y ) of size |X| =
m , a given L -layer neural architecture A is randomly initialized, and the γ -Lipschitz continuous nonlinearity σ satisﬁes | σ ( x ) | ≤ | x | .
Then, for any two data distributions P ( x ) and
Q ( x ) , denote Z (cid:44) (cid:82) (cid:107) P ( x ) − Q ( x ) (cid:107) d x , as n 1 , . .
. , n L − 1 → ∞ sequentially,
[Equation]
with probability arbitrarily close to 1 .
D ( γ ) is set to L if γ = 1 , and (1 − γ 2 L ) / (1 − γ 2 ) otherwise.
Its proof is in Appendix A.4.
Proposition 2 reveals that for any neural architecture A , (cid:107) Θ
( A ) (cid:107) tr is data-agnostic if either one of the following conditions is satisﬁed: (a) Different datasets achieve a small Z or (b) the input dimension n 0 is large.
Interestingly, these two conditions required by the data-agnostic (cid:107) Θ
( A ) (cid:107) tr can be well-satisﬁed in practice.
Firstly, we always have Z < 2 according to the property of probability distributions.
Moreover, many real-world datasets are indeed of high dimensions such as ∼ 10 3 for CIFAR-10 (Krizhevsky et al., 2009) and ∼ 10 5 for COCO (Lin et al., 2014).
Since (cid:107) Θ
( A ) (cid:107) tr under such mild conditions is data-agnostic, NASI using (cid:107) Θ 0 ( A ) (cid:107) tr as the optimization objective in (8) is also data-agnostic.
While (cid:107) Θ
( A ) (cid:107) tr is costly to evaluate, we demonstrate in Fig.
1b that the approximated (cid:107) Θ 0 ( A ) (cid:107) tr in (12) is also data-agnostic by using random data generated with the standard Gaussian distribution.
Similar to the results using true vs. random labels in Fig.
1a, the approximated (cid:107) Θ
( A ) (cid:107) tr using random vs. true data are also highly correlated by achieving a large Pearson correlation coefﬁcient (i.e., ρ > 0 . 9 ).
Interestingly, the correlation coefﬁcient here is slightly smaller than the one of label-agnostic approximations in Fig.
1a, which implies that our approximated (cid:107) Θ 0 ( A ) (cid:107) tr is more agnostic to the labels than data.
Based on these results, the approximated (cid:107) Θ 0 ( A ) (cid:107) tr is guaranteed to be data-agnostic.
So, NASI based on the optimization over such a data-agnostic approximation is also data-agnostic, which will be further validated empirically in Sec.
E XPERIMENTS 5. S EARCH IN NAS-B ENCH -1S HOT
We ﬁrstly validate the search efﬁciency and effectiveness of our NASI in the three search spaces of
NAS-Bench-1Shot1 (Zela et al., 2020b) on CIFAR-10.
As these three search spaces are relatively
[Figure]
[Table]
small, a lower penalty coefﬁcient µ and a larger constraint ν
(i [Table] e., µ =1 and ν =1000 ) are adopted to encourage the selection of high-complexity architectures in the optimization of (13).
Here, ν is determined adaptively as shown in Appendix B.1.
Figure 2 shows the results comparing the efﬁciency and effectiveness between NASI with a oneepoch search budget and other NAS algorithms with a maximum search budget of 20 epochs to allow sufﬁcient model training during their search process.
Figure 2 reveals that among all these three search spaces, NASI consistently selects architectures of better generalization performance than other
NAS algorithms with a search budget of only one epoch.
Impressively, the architectures selected by our one-epoch NASI can also achieve performances that are comparable to the best-performing NAS algorithms with 19 × more search budget.
Above all, NASI guarantees its beneﬁts of improving the search efﬁciency of NAS algorithms considerably without sacriﬁcing the generalization performance of its selected architectures.
5. S EARCH IN THE DARTS S EARCH S PACE
We then compare NASI with other NAS algorithms in a more complex search space than NAS-Bench1Shot1, i.e., the DARTS (Liu et al., 2019) search space (detailed in Appendix B.2).
Here, NASI
[Table]
selects the architecture with a search budget of T =100 , batch size of b =64 and µ =2 .
Besides, two different methods are applied to determine the constraint ν during the search process: the adaptive determination with an initial value of 500 and the ﬁxed determination with a value of 100 .
The ﬁnal selected architectures with adaptive and ﬁxed ν are, respectively, called NASI-ADA and NASI-FIX (visualized in Appendix C.4), which are then evaluated on CIFAR-10/100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) following Appendix B.4.
Table 1 summarizes the generalization performance of the ﬁnal architectures selected by various
NAS algorithms on CIFAR-10/100.
Compared with popular training-based NAS algorithms, NASI achieves a substantial improvement in search efﬁciency and maintains a competitive generalization performance.
Even when compared with other training-free NAS algorithm (i.e., TE-NAS), NASI is also able to select competitive or even outperformed architectures with a smaller search cost.
Besides,
NASI-FIX achieves the smallest test error on CIFAR-100, which also demonstrates the transferability of the architectures selected by NASI over different datasets.
Consistent results on ImageNet can be found in Appendix C.5.
5. L ABEL - AND D ATA -A GNOSTIC S EARCH
To further validate the label- and data-agnostic search achieved by our NASI as discussed in Sec. 4, we compare the generalization performance of the ﬁnal architectures selected by NASI using random labels and data on CIFAR-10.
The random labels are randomly selected from all possible categories while the random data is i.i.d.
sampled from the standard Gaussian distribution.
Both NAS-Bench1Shot1 and the DARTS search space are applied in this performance comparison where the same search and training settings in Sec.
5.1 and Sec.
5.2 are adopted.
Table 2 summarizes the performance comparison.
Interestingly, among all these four search spaces, comparable generalization performances are obtained on CIFAR-10 for both the architectures selected with random labels (or data) and the ones selected with true labels and data.
These results hence conﬁrm the label- and data-agnostic search achieved by NASI, which therefore also further validates the transferability of the architectures selected by NASI over different datasets.
C ONCLUSION
This paper describes a novel NAS algorithm called NASI that exploits the capability of NTK for estimating the performance of candidate architectures at initialization.
Consequently, NASI can completely avoid model training during the search process to achieve higher search efﬁciency than existing NAS algorithms.
NASI can also achieve competitive generalization performance across different search spaces and benchmark datasets.
Interestingly, NASI is guaranteed to be label- and data-agnostic under mild conditions, which therefore implies the transferability of the ﬁnal selected architectures by NASI over different datasets.
With all these advantages, NASI can thus be adopted to select well-performing architectures for unsupervised tasks and larger-scale datasets efﬁciently, which to date remains challenging to other training-based NAS algorithms.
Furthermore, NASI can also be integrated into other training-based one-shot NAS algorithms to improve their search efﬁciency while preserving the search effectiveness of these training-based algorithms.
R EPRODUCIBILITY S TATEMENT
To guarantee the reproducibility of the theoretical analysis in this paper we have provided complete proof of our propositions and also the justiﬁcation of certain assumptions in
Appendix A. Moreover, we have conducted adequate ablation studies to further investigate the impacts of these assumptions as well as the approximations used in our method on the ﬁnal search results in Appendix C.6.
Meanwhile, to guarantee the reproducibility of the empirical results in this paper, we have provided our codes in the supplementary materials and detailed experimental settings in Appendix B A CKNOWLEDGMENTS
This research is supported by A*STAR under its RIE 2020 Advanced Manufacturing and Engineering (AME) Programmatic Funds (Award A 20 H 6 b 0151 ).
A PPENDIX A T HEOREMS AND P ROOFS
We ﬁrstly introduce the theorems proposed by Jacot et al.
; Lee et al.
(2019a), which reveal the capability of NTK in characterizing the training dynamics of inﬁnite-width DNNs.
Note that the these theorems follow the parameterization and initialization of DNNs in (Jacot et al., 2018).
Our analysis based on these theorems thus also needs to follow such parameterization and initialization of
DNNs.
A. N EURAL T ANGENT K ERNEL AT I NITIALIZATION
Jacot et al.
(2018) have validated that the outputs of inﬁnite-width DNNs at initialization tends to
Gaussian process (Theorem 1) and have further revealed the deterministic limit of NTK at initialization (Theorem 2).
We denote I n L as a n L × n L matrix with all elements being 1.
Theorem 1 (Jacot et al.
For a network of depth L at initialization, with a Lipschitz nonlinearity σ , and in the limit as n 1 , · · · , n L − 1 → ∞ sequentially, the output functions f θ ,i for i = 1 , · · · , n L , tend (in law) to iid centered Gaussian processes of covariance Σ ( L ) , where Σ ( L ) is deﬁned recursively by
[Equation]
such that the expectation is with respect to a centered Gaussian process f with covariance Σ
( L − 1) .
Theorem 2 (Jacot et al.
For a network of depth L at initialization, with a Lipschitz nonlinearity σ , and in the limit as n 1 , · · · , n L − 1 → ∞ sequentially, the NTK Θ ( L ) converges in probability to a deterministic limiting kernel:
[Equation]
Kernel Θ ( L ) ∞ : R n 0 × n 0 → R is deﬁned recursively by
[Equation]
where
[Equation]
such that the expectation is with respect to a centered Gaussian process f with covariance Σ
( L − 1) and ˙ σ denotes the derivative of σ .
A. T RAINING D YNAMICS OF I NFINITE -W IDTH N EURAL N ETWORKS
Given λ min ( Θ ) as the minimal eigenvalue of NTK Θ and deﬁne η critical (cid:44) 2( λ min ( Θ )+ λ max ( Θ )) − 1 ,
Lee et al. (2019a) have characterized the training dynamics of inﬁnite-wide neural networks as below.
Theorem 3 (Lee et al.
(2019a)) .
Let n 1 = · · · = n L − 1 = k and assume λ min ( Θ ) > 0 .
Applying gradient descent with learning rate η < η critical (or gradient ﬂow), for every x ∈ R n 0 with (cid:107) x (cid:107) ≤ 1 , with probability arbitrarily close to 1 over random initialization,
[Equation]
Remark.
For the case of L = 2 , Du et al.
(2019) have revealed that if any two input vectors of a dataset are not parallel, then λ min ( Θ ) > 0 holds, which fortunately can be well-satisﬁed for most real-world datasets.
Though the training dynamics of DNNs only tend to be governed by their linearization at initialization when the inﬁnite width is satisﬁed as revealed in Theorem 3, empirical results in (Lee et al., 2019a) suggest that such linearization can also govern the training dynamics of practical over-parameterized DNNs.
A. P ROOF OF P ROPOSITION
With aforementioned theorems, especially Theorem 3, our Proposition 1 can be proved as below with an introduced lemma (Lemma 1).
Lemma 1.
Let loss function L ( f ( X ; θ t ) , Y ) , abbreviated to L ( f t ) , be γ -Lipschitz continuous within the domain V .
Under the condition in Theorem 3, there exists a constant c 0 > 0 such that as k → ∞ ,
[Equation]
with probability arbitrarily close to 1.
Proof.
Since L ( f t ) is γ -Lipschitz continuous, for any v , u ∈ V , we have
[Equation]
following the deﬁnition of Lipschitz continuity.
Besides, under the condition in Theorem 3, Theoremreveals that there exists a constant c 0 such that as k → ∞ ,
[Equation]
with probability arbitrarily close to 1.
Combining (19) and (20), we hence can ﬁnish the proof by
[Equation]
Remark.
The γ -Lipschitz continuity based on (cid:107) · (cid:107) 2 is commonly satisﬁed for widely adopted loss functions.
For example, given 1-dimensional MSE L = m − 1 (cid:80) mi ( x i − y i ) 2 , let x i , y i ∈ [0 , 1] denote the prediction and label respectively, the Lipschitz of MSE with respect to x (cid:44) ( x 1 , · · · , x m ) (cid:62) is 2 .
Meanwhile, given the n -class Cross Entropy with Softmax L = − m − 1 (cid:80) mi (cid:80) nj
y i,j log( p i,j ) as the loss function, let p i,j ∈ (0 , 1) and y i,j ∈ { 0 , 1 } denote the prediction and label correspondingly, with (cid:80) j y i,j = 1 and p i,j (cid:44) exp( x i,j ) / (cid:80) j exp( x i,j ) for input x
i,j ∈ R , the Lipschitz of Cross Entropy with Softmax with respect to x (cid:44) ( x 1 , 1 , · · · , x i,j , · · · , x m,n ) (cid:62) is then 1 .
Proof of Proposition 1.
Note that the linearization f lin ( x ; θ t ) achieves a constant NTK Θ t =
Θbecause its gradient with respect to θ t (i.e., ∇ θ 0 f ( x ; θ 0 ) ) stays constant over time.
Given MSE as the loss function, according to the loss decomposition (4) in Sec. 2.2, the training dynamics of f lin ( x ; θ t ) can then be analyzed in a closed form:
[Equation]
where Θ 0 = (cid:80) mn i = λ i ( Θ 0 )
u i u (cid:62) i , and λ i ( Θ 0 )
and u i denote the i -th largest eigenvalue and the corresponding eigenvector of Θ 0 , respectively.
With η < λ max ( Θ 0 ) − 1 and λ min ( Θ 0 ) > 0 , ηλ i ( Θ 0 ) is then under the constraint that
[Equation]
Hence, for the case of t ≥ 0 . 5 , with 0 < 1 − ηλ i ( Θ 0 ) <
1 and λ ( Θ 0 ) (cid:44) ( mn ) − 1 (cid:80) mni
=
λ
i
( Θ 0 ) ,
[Equation]
i
[Equation]
Further, given 0 ≤ t < 0 . 5 , the scalar function y = x 2 t is concave for any x ∈ R ≥ 0 .
Following from the Jensen’s inequality on this concave function,
[Equation]
With bounded labels Y ∈ [0 , 1] mn and the unit norm of eigenvectors (cid:107) [Equation]
i
[Equation] [Equation]
By introducing (24), (25) and (26) into (22), the training loss L ( f lin t ) can then be bounded by
[Equation]
where q is set to be 2 t if 0 ≤ t < 0 .
5 , and 1 otherwise.
Following from Theorem 3, while applying gradient descent (or gradient ﬂow) with learning rate η < λ − 1max < η critical (cid:44) 2( λ min + λ max ) − 1 , for all x ∈ X with (cid:107) x (cid:107) ≤ 1 , there exists a constant csuch that as k → ∞ ,
[Equation]
(28) with probability arbitrarily close to 1.
Hence, f lin t
[Equation] k [Equation] mn with f t ∈ [0 , 1] mn .
Within the extended domain f t ∈ (cid:104) − c 0 (cid:112) 1 /k, 1 + c 0 (cid:112) 1 /k
[Equation]
mn for the MSE loss function, [Equation]
[Equation]
Hence, the MSE within this extended domain is 2 (cid:112) n/m (cid:16) 1 + c 0 (cid:112) 1 /k (cid:17) -Lipschitz continuous.
Combining with Lemma 1,
[Equation]
which concludes the proof by denoting L t (cid:44) L ( f t ) .
Remark.
(cid:107) x (cid:107) 2 ≤ 1 can be well-satisﬁed for the normalized dataset, which is conventionally adopted as data pre-processing in practice.
A. P ROOF OF P ROPOSITION
According to Theorem 2, Θ 0 (cid:44) Θ ( L ) is determined by both Σ ( L ) ( x , x ) and ˙ Σ ( L ) ( x , x ) of all x ∈ X .
We hence introduce Lemma 2 below regarding Σ ( L ) ( x , x ) and ˙Σ ( L ) ( x , x ) to ease the proof of Proposition 2.
Particularly, we set β = 0 for the β in Theorem 1 and Theorem 2 throughout our analysis.
Note that this condition is usually satisﬁed by training DNNs without bias.
Lemma 2.
For a network of depth L at initialization, with the γ -Lipschitz continuous nonlinearity σ satisfying | σ ( x ) | ≤ | x | for all x ∈ R , given the input features x ∈ X ,
[Equation]
where Σ ( L ) ( x , x ) and ˙Σ ( L ) ( x , x ) are deﬁned in Theorem 1 and Theorem 2, respectively, and β = 0 .
Proof.
Following from the deﬁnition in Theorem 1,
[Equation]
in which (a) follows from | σ ( x ) | ≤ | x | and (b) results from the variance of g ∼ N (0 ,
Σ ( L − 1) ) .
By following (31) from layer L to 1, we get
[Equation]
where (a) is deﬁned in Theorem 1.
Similarly, following from the deﬁnition in Theorem 2,
[Equation]
in which (a) results from the γ -Lipschitz continuity of nonlinearity and (b) follows from the expectation of a constant.
Proof of Proposition 2.
Notably, Theorem 2 reveals that in the limit as n 1 , · · · , n L − 1 → ∞ sequentially, Θ ( L ) → Θ ( L ) ∞ ⊗ I n L with probability arbitrarily close to 1 over random initializations.
We therefore only need to focus on this deterministic limiting kernel to simplify our analysis, i.e., let Θ
= Θ ( L ) ∞ ⊗ I n L .
Particularly, given m input vectors x ∼ P ( x ) with covariance matrix Σ P and a
L -layer neural architecture of n -dimensional output, For γ (cid:54)
= 1 , we have
[Equation]
in which (a) derives from the property of Kronecker product, (b) follows from the notation n L = n and (c) results from the property of expectation and trace norm.
In addition, (d) follows from the deﬁnition of Θ ( L ) ∞ ( x , x ) in Theorem 2 and (e) is derived by introducing Lemma 2 into (d).
Finally, (f) follows from the expectation and variance of P ( x ) .
By following (c-f) from layer L to 1, we can get
[Equation]
Notably, (35) is derived under the condition that γ (cid:54) = 1 .
For the case of γ = 1 , similarly, we can get
[Equation]
Note that with (cid:107) x (cid:107) 2 ≤ 1 , we have
[Equation]
Given any two different distributions P ( x ) and Q ( x ) , deﬁne Z = (cid:82) (cid:107) P ( x ) − Q ( x ) (cid:107) d x , 4 we can construct a special probability density function (cid:101) P ( x ) =
Z − 1 (cid:107) P ( x ) − Q ( x ) (cid:107) to further employee the bounds in (35) and (36).
Speciﬁcally, for γ (cid:54) = 1 , by combining (35) and (37), we can get
[Equation]
mn
[Equation]
[Equation]
in which (a) follows from (c) in (34), (b) results from Cauchy Schwarz inequality and (c) is obtained by introducing inequality (35) and distribution (cid:101) P ( x ) .
Finally, (d) is based on (37).
Similarly, in the case of γ = 1 , we can get
[Equation]
[Equation]
which concludes the proof.
Remark.
Note that ReLU is widely adopted as the nonlinearity in neural networks, which satisﬁes the Lipschitz continuity with γ = 1 and the inequality | σ ( x ) | ≤ | x | .
Notably, many other ReLU-type nonlinearities (e.g., Leaky ReLU (Maas et al., 2013) and PReLU (He et al., 2015)) can also satisfy these two conditions.
A PPENDIX B E XPERIMENTAL S ETTINGS B. D ETERMINATION OF C ONSTRAINT ν AND P ENALTY C OEFFICIENT µ
As demonstrated in Sec. 3.1, constraint ν (derived from (cid:107) Θ 0 ( A ) (cid:107) tr < mnη − 1 in (8)) introduces a trade-off between the complexity of ﬁnal selected architectures and the optimization behavior in their model training.
This constraint ν hence is of great importance to our NASI algorithm.
Though ν has already been pre-deﬁned as ν (cid:44) γnη − 1 in Sec.
3.3, we still tend to take it as a hyper-parameter to be determined for the selection of best-performing architectures in practice.
Speciﬁcally, in this paper, two methods are adopted to determine ν during the search progress: The ﬁxed and adaptive method shown as below.
Notably, the ﬁnal architectures selected by NASI via the ﬁxed and adaptive method are called NASI-FIX and NASI-ADA, respectively.
Note that our experiments in the main text suggest that both methods can select architectures with competitive generalization performance over different tasks.
The ﬁxed determination.
We initialize and ﬁx ν with ν 0 during the whole search process.
Hence, ν 0 is required to provide a good trade-off between the complexity of architectures and their optimization behavior in the search space.
Intuitively, the expectation of architecture complexity in the search space can help to select architectures with medium complexity and hence implicitly achieve a good trade-off between the complexity of architectures and their optimization behavior.
Speciﬁcally, we randomly sample N = 50 architectures in the search space (i.e., A 1 , · · · , A N ), and then determine ν 0 before the search process by
[Equation]
Note that we can further enlarge ν 0 in practice to encourage the selection of architectures with larger complexity.
The adaptive determination.
We initialize ν with a relatively large ν
and then adaptively update it with the expected (cid:107) (cid:101) Θ
( A ) (cid:107) tr of sampled architectures during the search process.
Speciﬁcally, with sampled architectures A 1 , · · · , A t in the history, ν at time t (i.e., ν t ) during the search process is given by
[Equation]
We apply a relatively large ν 0 to ensure a loose constraint on the complexity of architectures in the ﬁrst several steps of the search process.
Note that the adaptive determination provides a more accurate approximation of the expected complexity of architectures in the search space than the ﬁxed determination method if more architectures are sampled to update ν t , i.e., t > N .
Note that (13) in Sec.
3.3 further reveals the limitation on the complexity of ﬁnal selected architectures by the penalty coefﬁcient µ .
Particularly, µ =0 indicates no limitation on the complexity of architectures.
Following from the introduced trade-off between the complexity of ﬁnal selected architectures and the optimization behavior in their model training by the constraint ν , for a search space with relatively larger-complexity architectures, a larger penalty coefﬁcient µ (i.e., µ = 2 ) is preferred to search for architectures with relatively smaller complexity to ensure a well-behaved optimization with a larger learning rate η .
On the contrary, for a search space with relatively smaller-complexity architectures, a lower penalty coefﬁcient µ (i.e., µ =1 ) is adopted to ensure the complexity and hence the representation power of the ﬁnal selected architectures.
Appendix C.6 provides further ablation study on the constraint ν and the penalty coefﬁcient µ .
B. T HE DARTS S EARCH S PACE
Following from the DARTS (Liu et al., 2019) search space, candidate architecture in our search space comprise a stack of L cells, and each cell can be represented as a directed acyclic graph (DAG) of
N nodes denoted by { x 0 , x 1 , . . .
x N − 1 } .
To select the best-performing architectures, we instead need to select their corresponding cells, including the normal and reduction cell.
Speciﬁcally, xand x 1 denote the input nodes, which are the output of two preceding cells, and the output x N of a cell is the concatenation of all intermediate nodes from x 2 to x N − 1 .
Following that of SNAS (Xie et al., 2019), by introducing the distribution of architecture in the search space (i.e., p α ( A ) in (14)), each intermediate nodes x i (2 ≤ i ≤ N − 1) denotes the output of a single sampled operation o i ∼ p
i ( o ) with (cid:80) o ∈O p
i ( o ) = 1 given a single sampled input x j ∼ p i ( x ) with j ∈ 0 , · · · , i − 1 and (cid:80) i − 1 j =
p
i ( x j ) = 1 , where O is a predeﬁned operation set.
After the search process, only operations achieving the top-2 largest p i ( o ) and inputs achieving the top-2 largest p i ( x ) for node x i are retained.
Each intermediate node x j hence connects to two preceding nodes with the corresponding selected operations.
Following from DARTS, the candidate operation set O includes following operations: 3 × 3 max pooling, 3 × 3 avg pooling, identity, 3 × 3 separable conv, 5 × 5 separable conv, 3 × 3 dilated separable conv, 5 × 5 dilated separable conv.
Note that our search space has been modiﬁed slightly based on the standard DARTS (Liu et al., 2019) search space: (a) Operation zero is removed from the candidate operation set in our search space since it can never been selected in the standard DARTS search space, and (b) the inputs of each intermediate node are selected independently from the selection of operations, while DARTS attempts to select the inputs of intermediate nodes by selecting their coupling operations with the largest weights.
Notably, following from DARTS, we need to search for two different cells:
A normal cell and a reduction cell.
Besides, a max-pooling operation in between normal and reduction cell is applied to down-sampling intermediate features during the search process inspired by NAS-Bench-1Shot
(Zela et al., 2020b).
B. S AMPLING A RCHITECTURE WITH G UMBEL -S OFTMAX
Notably, aforementioned p i ( o ) and p i ( x ) for the node x i follow the categorical distributions.
To relax the optimization of (14) with such categorical distributions to be continuous and differentiable,
Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) is applied.
Speciﬁcally, supposing p i ( o ) and p i ( x ) are parameterized by α oi,j and α xi,k respectively, with Straight-Through (ST) Gumbel
Estimator (Jang et al., 2017), we can sample the single operation and input for the node x i (2 ≤ i ≤
N − 1) during the search process by
[Equation]
where g oi,j and g xi,k are sampled from Gumbel(0, 1) and τ denotes the softmax temperature, which is conventionally set to be 1 in our experiments.
Note that in (42), j ∈ [0 , · · · , |O| − 1] and k ∈ [0 , · · · , i − 1] , which correspond to the |O| operations in the operation set O and the i candidate inputs for the node x i , respectively.
Notably, the gradient through ST can be approximated by its continuous counterpart as suggested in (Jang et al., 2017), thus allowing the continuous and differentiable optimization of (14).
By sampling the discrete operation and input with (42) for each nodes in the cells, the ﬁnal sampled architecture can hence be determined.
B. E VALUATION ON CIFAR-10/100 AND I MAGE N ET
Evaluation on CIFAR-10/100.
Following DARTS (Liu et al., 2019), the ﬁnal selected architectures consist of 20 searched cells: 18 of them are identical normal cell and 2 of them are identical reduction cell.
An auxiliary tower with weight 0.4 is located at 13-th cell of the ﬁnal selected architectures and the number of initial channels is set to be 36.
The ﬁnal selected architecture are then trained via stochastic gradient descent (SGD) of 600 epochs with a learning rate cosine scheduled from 0.
to 0, momentum 0.9, weight decay 3 × 10 − 4 and batch size 96 on a single Nvidia 2080Ti GPU.
Cutout (Devries & Taylor, 2017), and ScheduledDropPath linearly increased from 0 to 0.2 are also employed for regularization purpose.
Evaluation on ImageNet.
We evaluate the transferability of the selected architectures from CIFAR10 to ImageNet.
The architecture comprises of 14 cells (12 normal cells and 2 reduction cells).
To evaluate in the mobile setting (under 600M multiply-add operations), the number of initial channels is set to 46.
We adopt conventional training enhancements (Liu et al., 2019; Chen et al., 2019;
Chen & Hsieh, 2020) include an auxiliary tower loss of weight 0.4 and label smoothing.
Following
P-DARTS
(Chen et al., 2019)
and SDARTS-ADV (Chen & Hsieh, 2020), we train the model from scratch for 250 epochs with a batch size of 1024 on 8 Nvidia 2080Ti GPUs.
The learning rate is warmed up to 0.5 for the ﬁrst 5 epoch and then decreased to zero linearly.
We adopt the SGD optimizer with 0.9 momentum and a weight decay of 3 × 10 − 5 for the model training on ImageNet.
A PPENDIX C M ORE R ESULTS C. T RADE - OFF B ETWEEN M ODEL C OMPLEXITY AND O PTIMIZATION B EHAVIOUR
In this section, we empirically validate the existence of trade-off between model complexity of selected architectures and the optimization behavior in their model training for our reformulated NAS detailed in Sec. 3.1.
The trade-off in NAS-Bench-1Shot1.
Speciﬁcally, Figure 3 illustrates the relation between test error and approximated (cid:107) Θ
( A ) (cid:107) tr of candidate architectures in the three search spaces of NASBench-1Shot1.
Note that with the increasing of the approximated (cid:107) Θ 0 ( A ) (cid:107) tr , the test error decreases rapidly to a minimum and then increase gradually, which implies that architectures only with certain (cid:107) Θ
( A ) (cid:107) tr
(or with desirable complexity instead of the largest complexity) can achieve the best generalization performance.
These results hence validate the existence of trade-off between the model complexity and the generalization performance of selected architectures.
Note that such trade-off usually results from different optimization behavior in the model training of those selected architectures as demonstrated in the following experiments.
The trade-off in the DARTS search space.
We then illustrate the optimization behavior of the ﬁnal selected architecture from the DARTS search space with different constraint and penalty coefﬁcient in
Figure 4 to further conﬁrm the existence of such trade-off in our reformulated NAS (8).
Speciﬁcally,
[Figure]
[Figure]
right we apply NASI with the ﬁxed determination of ν to select architectures in the the DARTS search space over CIFAR-10, where ν 0 in the ﬁxed determination method introduced in Appendix B.1 is modiﬁed manually for the comparison.
Besides, a search budget of T = 100 and batch size of b =are adopted.
These ﬁnal selected architectures are then trained on CIFAR-10 following Appendix 5.2.
Note that, in Figure 4(a), with the increasing of ν , the ﬁnal selected architectures by NASI contain more parameters and hence achieve larger complexity.
Meanwhile, the ﬁnal selected architecture with ν =10 enjoys a faster convergence rate in the ﬁrst 50 epochs, but a poorer generalization performance than the one with ν = 200 .
Interestingly, the ﬁnal selected architecture with ν =100 realizes the fastest convergence and the best generalization performance by achieving a proper complexity of ﬁnal selected architectures.
These results validate the existence and also the importance of the trade-off between the complexity of ﬁnal selected architectures and the optimization behavior in their model training.
However, the trade-off introduced by the penalty coefﬁcient µ shown in Figure 4(b) is hard to be observed, which indicates that the constraint ν is of greater importance than the penalty coefﬁcient µ in terms of the trade-off between the complexity of architectures and their optimization behavior.
Interestingly, Figure 4(b) can still reveal the slower convergence rate and poorer generalization performance caused by the selection of architecture with relatively larger complexity, i.e., µ =1 .
C. C OMPARISON TO O THER T RAINING -F REE M ETRICS
We compare our methods with other training-free NAS methods using both the Spearman correlation and the Kendall’s tau between the training-free metrics and the test accuracy on CIFAR-10 in the three search spaces of NAS-Bench-1Shot1.
We adopt one uniformly randomly sampled mini-batch data to evaluate these two correlations and apply the same implementations of these training-free
[Table] [Table] NAS methods in (Abdelfattah et al., 2021) for the comparison.
Table 3 summarizes the comparison, where the results of our metric are reported under the constraint in (8).
Interestingly, our metric generally achieves a higher positive correlation than other training-free metrics [Table] which conﬁrms the reasonableness and also the effectiveness of our training-free metric
[Table]
C. S EARCH IN NAS-B ENCH -
To further justify the improved search efﬁciency and competitive search effectiveness of our NASI algorithm, we also compare it with other state-of-the-art NAS algorithms in NAS-Bench-201 (Dong &
Yang, 2020) on CIFAR-10/100 (C10/100) and ImageNet-16-200 (IN-16-200) 5 .
Table 4 summarizes the comparison.
Note that the baselines in Table 4 are obtained from TE-NAS (Chen et al., 2021) paper.
Notably, compared with training-based NAS algorithms, our NASI algorithm can achieve signiﬁcantly improved search efﬁciency while maintaining a competitive or even outperforming test performance.
Furthermore, our NASI algorithm is shown to be able to enjoy both improved search efﬁciency and effectiveness when compared with most other training-free baselines.
Although
TE-NAS, as the best-performing training-free NAS algorithm on both CIFAR-10/100, achieves a relatively improved test accuracy than our NASI ( T ), our NASI with a search budget of T = 30 s is 50 × more efﬁcient than TE-NAS and is able to achieve compelling test performance on all the three datasets.
Moreover, by providing a larger search budget, our NASI algorithm (i.e., NASI (4 T )) can in fact achieve comparable (on CIFAR-10/100) or even better (on ImageNet-16-120) search results with 12 × lower search cost compared with TE-NAS.
C. A RCHITECTURES S ELECTED BY NASI
The ﬁnal selected cells in the DARTS search space (i.e., NASI-FIX and NASI-ADA used in Sec. 5.2) are illustrated in Figure 5, where different operations are denoted with different colors for clariﬁcation. [Figure] [Table]
Interestingly, according to the deﬁnitions and ﬁndings in (Shu et al., 2020), these ﬁnal selected cells by
NASI are relatively deeper and shallower than the ones selected by other NAS algorithms and hence may achieve worse generalization performance.
Nonetheless, in our experiments, the architectures constructed with these cells (i.e., NASI-FIX and NASI-ADA) achieve competitive or even better generalization performance than the ones selected by other NAS algorithms as shown in Table 1 and
Table 5.
A possible explanation is that our NASI algorithm provides a good trade-off between the complexity of architectures and the optimization in their model training, while other NAS algorithms implicitly prefer architectures with smaller complexity and faster convergence rate as revealed in (Shu et al., 2020).
Note that the ﬁnal selected cells in NASI-FIX and NASI-ADA are of great similarity to each other, which hence implies that the adaptive determination and the ﬁxed determination of constraint ν share similar effects on the selection of ﬁnal architectures.
C. E VALUATION ON I MAGE N ET
We also evaluate the performance of the ﬁnal architectures selected by NASI on ImageNet and summarize the results in Table 5.
Notably, NASI-FIX and NASI-ADA outperform the expertdesigned architecture ShufﬂeNet 2 × (v2), NAS-based architecture MnasNet-92 and DARTS by a large margin, and are even competitive with best-performing one-shot NAS algorithm DARTS-.
Notably, while achieving better generalization performance than TE-NAS (ImageNet), NASI (C10) is even shown to be more efﬁcient by directly transferring the architectures selected on CIFARto ImageNet based on its provable transferability in Sec. 4.
Meanwhile, by directly searching on
ImageNet, NASI-ADA (ImageNet) is able to achieve further improved performance over NASI-ADA (C10) while the performance of NASI-FIX (ImageNet) and NASI-FIX (C10) are quite similar.
Above all, these results on ImageNet further conﬁrm the good transferability of the architectures selected by NASI to larger-scale datasets.
[Table]
[Table]
C. A BLATION S TUDY AND A NALYSIS
The impacts of architecture width.
As our theoretically grounded performance estimation of neural architectures relies on an inﬁnite width assumption (i.e., n → ∞ in Proposition 1), we investigate the impacts of varying architecture width on the generalization performance of ﬁnal selected architectures by our NASI.
We adopt the same search settings in Sec.
but with a varying architecture width (i.e., a varying number of initial channels) on CIFAR-10 for the three search spaces of NAS-Bench-1Shot1.
Table 6 shows results of the effectiveness of our NASI in NAS-Bench-1Shot with varying architecture widths N : NASI consistently selects well-performing architectures and a larger width ( N = 32 ) enjoys better search results.
Hence, the inﬁnite width assumption for NTK does not cause any empirical issues for our NASI.
The effectiveness of NTK trace norm approximations.
Since the trace norm of NTK is costly to evaluate, we have provided our approximation to it in Sec. 3.2.
In this section, we empirically validate the effectiveness of our NTK trace norm approximation.
Speciﬁcally, we evaluate the Pearson correlation between our approximations (including the approximations using the sum of sample gradient norm in the ﬁrst inequality of (11) and the approximations using mini-batch gradient norm in (12)) and the exact NTK trace norm under varying batch size in the three search spaces of NASBench-1Shot1.
The results are summarized in Table 7.
The results conﬁrm that our approximation is reasonably good to estimate the exact NTK trace norm of different architectures by achieving a high Pearson correlation between our approximations and the exact NTK trace norm.
Interestingly, a larger batch size of mini-batch gradient norm generally achieves a better approximation, and the sum of sample gradient norm achieves the best approximation, which can be explained by the possible approximation errors we introduced when deriving our (11) and (12).
The impacts of batch size.
Since we only adopt a mini-batch to approximate the NTK trace norm shown in Sec. 3.2, we further examine the impacts of varying batch size on the search results in the three search spaces of NAS-Bench-1Shot1 in this section.
Table 8 summarizes the search results.
Interestingly, the results show that our approximation under varying batch sizes achieves comparable search results, further conﬁrming the effectiveness of our approximations in select well-performing architectures.
The impacts of constraint ν and penalty coefﬁcient µ .
Based on the analysis in Sec.
3.3 and the results in Appendix C.1, the choice of constraint ν and penalty coefﬁcient µ is thus non-trivial to select best-performing architectures since they trade off the complexity of ﬁnal selected architectures and the optimization in their model training.
In this section, we demonstrate their impacts on the generalization performance of the ﬁnally selected architectures and also the effectiveness of our ﬁxed determination on the constraint ν in detail.
Notably, we adopt the same settings (including the search and training settings) as those in Appendix C.1 on the DARTS search space, where ν 0 in the ﬁxed determination method introduced in Appendix B.1 is modiﬁed manually for the comparison. [Table] [Figure] [Table]
[Figure] [Table] [Figure] Figure 6 summarizes their impacts into two parts: the joint impacts in Figure 6(a) and the individual impacts in Figure 6(b).
Notably, in both plots, with the increasing of ν or µ , the test error of the ﬁnal selected architecture by NASI gradually decreases to a minimal and then increase steadily, hence further validating the existence of the trade-off introduced by ν and µ .
Moreover, the ﬁnal selected architectures with ν =100 signiﬁcantly outperform other selected architectures.
This result thus supports the effectiveness of our ﬁxed determination method for ν in NASI.
Interestingly, Figure 6(b) reveals a less obvious decreasing trend of test error for µ compared with ν , which hence further implies the greater importance of constraint ν than penalty coefﬁcient µ in terms of the generalization performance of the ﬁnal selected architectures.
Therefore, in our experiments, we conventionally set penalty coefﬁcient µ =1 for small-complexity search spaces and µ =2 for large-complexity search spaces as introduced in Appendix B.1.