G RAD S IGN : M ODEL P ERFORMANCE I NFERENCE WITH T HEORETICAL I NSIGHTS
A BSTRACT
A key challenge in neural architecture search (NAS) is quickly inferring the predictive performance of a broad spectrum of neural networks to discover statistically accurate and computationally efﬁcient ones.
We refer to this task as model performance inference (MPI).
The current practice for efﬁcient MPI is gradient-based methods that leverage the gradients of a network at initialization to infer its performance.
However, existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical foundations to consolidate their designs.
We propose GradSign, an accurate, simple, and ﬂexible metric for model performance inference with theoretical insights.
A key idea behind GradSign is a quantity Ψ to analyze the sample-wise optimization landscape of different networks.
Theoretically, we show that Ψ is an upper bound for both the training and true population losses of a neural network under reasonable assumptions.
However, it is computationally prohibitive to directly calculate Ψ for modern neural networks.
To address this challenge, we design GradSign, an accurate and simple approximation of Ψ using the gradients of a network evaluated at a random initialization state.
Evaluation on seven NAS benchmarks across three training datasets shows that
GradSign generalizes well to real-world neural networks and consistently outperforms state-of-the-art gradient-based methods for MPI evaluated by Spearman’s ρ and Kendall’s Tau.
Additionally, we have integrated GradSign into four existing NAS algorithms and show that the GradSign-assisted NAS algorithms outperform their vanilla counterparts by improving the accuracies of best-discovered networks by up to 0.3%, 1.1%, and 1.0% on three real-world tasks.
I NTRODUCTION
As deep learning methods evolve, neural architectures have gotten progressively larger and more sophisticated (He et al., 2015; Ioffe & Szegedy, 2015; Krizhevsky et al., 2017; Devlin et al., 2019;
Rumelhart et al., 1986;
Srivastava et al., 2014; Kingma & Ba, 2017), making it increasingly challenging to manually design model architectures that can achieve state-of-the-art predictive performance.
To alleviate this challenge, recent work has proposed several approaches to automatically discovering statistically accurate and computationally efﬁcient neural architectures.
The most common approach is neural architecture search (NAS), which explores a comprehensive search space of potential network architectures that use a set of predeﬁned network modules as basic building blocks. Recent work shows that NAS is able to discover architectures that outperform human-designed counterparts (Liu et al., 2018a; Zoph & Le, 2016; Pham et al., 2018).
A key challenge in NAS is quickly assessing the predictive performance of a diverse set of candidate architectures to discover performant ones.
We refer to this task as model performance inference (MPI).
A straightforward approach to MPI is directly training each candidate architecture on a dataset until convergence and recording the achieved training loss and validation accuracy (Frankle & Carbin, 2018;
Chen et al., 2020a; Liu et al., 2018a; Zoph & Le, 2016).
Though accurate, this approach is computationally prohibitive and cannot scale to large networks or datasets.
The current practice to efﬁcient MPI is gradient-based methods that leverage the gradient information of a network at initialization to infer its predictive performance (Lee et al., 2018;
Wang et al., 2020;
Tanaka et al., 2020).
Compared to directly measuring the accuracy of candidate networks on a training dataset, gradient-based methods are computationally more efﬁcient since they only require evaluating a mini-batch of gradients at initialization.
However, existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical insights to consolidate their designs.
In this paper, we propose GradSign, a simple yet accurate metric for MPI with theoretical foundations.
GradSign is inspired by analyzing the sample-wise optimization landscape of a network.
GradSign takes as inputs a mini-batch of sample-wise gradients evaluated at a random initialization point and outputs a statistical evidence of a network that highly correlates to its well-trained predictive performance measured by accuracy on the entire dataset.
Prior theoretical results (Allen-Zhu et al., 2019) show that the optimization landscape of a randomly initialized network is nearly convex and semi-smooth for a sufﬁciently large neighborhood.
To realize its potential for MPI, we generalize these results to sample-wise optimization landscapes and propose a quantity Ψ to measure the density of sample-wise local optima in the convex areas around a random initialization point.
Additionally, we prove that both the training loss and generalization error of a network are proportionally upper bounded by Ψ 2 under reasonable assumptions.
Based on our theoretical results, we design GradSign, an accurate and simple approximation of Ψ . Empirically, we show that GradSign can also generalize to realistic setups that may violate our assumptions.
In addition, GradSign is efﬁcient to compute and easy to implement as it uses only the sample-wise gradient information of a network at a random initialization point.
Extensive evaluation of GradSign on seven NAS benchmarks (i.e., NAS-Bench-101, NAS-Bench-201, and ﬁve design spaces of NDS) across three datasets (i.e., CIFAR-10, CIFAR-100, and ImageNet16120) shows that GradSign consistently outperforms existing gradient-based methods in all circumstances.
Furthermore, we have integrated GradSign into existing NAS algorithms and show that
the
GradSign-assisted variants of these NAS algorithms lead to more accurate architectures.
Contributions.
This paper makes the following contributions:
• We provide a new perspective to view the overall optimization landscape of a network as a combination of sample-wise optimization landscapes.
Based on this insight, we introduce a new quantity Ψ that provides an upper bound on both the training loss and generalization error of a network under reasonable assumptions.
• To infer Ψ , we propose GradSign, an accurate and simple estimation of Ψ .
GradSign enables fast and efﬁcient MPI using only the sample-wise gradients of a network at initialization.
• We empirically show that GradSign generalizes to modern network architectures and consistently outperforms existing gradient-based MPI methods.
Additionally, GradSign can be directly integrated into a variety of NAS algorithms to discover more accurate architectures.
R ELATED W ORK 2. M ODEL P ERFORMANCE I NFERENCE
Table 1 summarizes existing approaches to inferring the statistical performance of neural architectures.
Sample-based methods assess the performance of a neural architecture by training it on a dataset.
Though accurate, sample-based methods require a surrogate training procedure to evaluate each architecture.
EconNAS (Zhou et al., 2020) mitigates the cost of training candidate architectures by reducing the number of training epochs, input dataset sizes, resolution of input images, and model sizes.
Theory-based methods leverage recent advances in deep learning theory, such as Neural Tangent
Kernel (Jacot et al., 2018) and Linear Region Analysis (Serra et al., 2018), to assess the predictive performance of a network (Chen et al., 2020a; Mellor et al., 2021; Park et al., 2020).
In particular, NNGP (Park et al., 2020) infers a network’s performance by ﬁtting its kernel regression parameters on a training dataset and evaluating its accuracy on a validation set, which alleviates the burden of training.
As another example, Chen et al. (2020a) utilizes the kernel condition number proposed in Xiao et al.
(2020), which can be theoretically proved to correlate to training convergence rate and generalization performance.
However, this theoretical evidence is only guaranteed for extremely wide networks with a specialized initialization mode.
While the linear region analysis used in Mellor et al.
(2021), Lin et al.
(2021) and Chen et al.
(2020a) is easy to implement, such technique is only applicable to networks with ReLU activations (Agarap, 2018). [Table]
Learning-based methods train a separate network (e
[Table]
g., graph neural networks) to predict a network’s accuracy (Liu et al., 2018a;
Luo et al., 2020; Dai et al., 2019; Wen et al., 2020; Chen et al., 2020b; Siems et al., 2020).
Though these learned models can achieve high accuracies on a speciﬁc task, this approach requires constructing a training dataset with sampled architectures for each downstream task.
As a result, existing learning-based methods are generally task-speciﬁc and computationally prohibitive.
Gradient-based methods infer the statistical performance of a network by leveraging its gradient information at initialization, which can be easily obtained using an automated differentiation tool of today’s ML frameworks, such as PyTorch (Paszke et al., 2017) and TensorFlow (Abadi et al., 2016).
The weight-wise salience score computed by several pruning at initialization (Lee et al., 2018; Wang et al., 2020; Tanaka et al., 2020) methods can easily be adapted to MPI settings by summing scores up.
Though lack of theoretical foundations, such migrations have been empirically proven to be effective as baselines in recent works (Abdelfattah et al., 2021a; Mellor et al., 2021; Lin et al., 2021).
An alternative stream of work (Turner et al., 2019; 2021; Theis et al., 2018) uses approximated secondorder gradients, known as empirical Fisher Information Matrix (FIM), at a random initialization point to infer the performance of a network.
Empirical FIM (Martens, 2014) is a valid approximation of a model’s predictive performance only if the model’s parameters are a Maximum Likelihood Estimation (MLE).
However, this assumption is invalid at a random initialization point, making FIM-based algorithms inapplicable.
A key difference between GradSign and existing gradient-based methods is that GradSign is based on a ﬁne-grained analysis of sample-wise optimization landscapes rather than heuristic insights.
In addition, GradSign also provides the ﬁrst attempt for MPI by leveraging the optimization landscape properties contained in sample-wise gradient information, while prior gradient-based methods only focus on gradients evaluated in a full batch fashion.
2. N EURAL A RCHITECTURE S EARCH
Recent work (He et al., 2021;
Cai et al., 2019; 2018; Tan & Le, 2019; Howard et al., 2019) has proposed several algorithms to explore a NAS search space and discover highly accurate networks.
RS (Bergstra & Bengio, 2012) is one of the baseline algorithms that generates and evaluates architectures randomly in the search space.
REINFORCE (Williams, 1992) moves a step forward by reframing NAS as a reinforcement learning task where accuracy is the reward and architecture generation is the policy action.
Given limited computational resources, BOHB (Falkner et al., 2018) uses Bayesian Optimization (BO) to propose candidates while uses HyperBand(HB)
(Li et al., 2017) for searching resource allocation.
REA (Real et al., 2019) uses a simple yet effective evolutionary searching strategy that achieves state-of-the-art performance.
GradSign is complementary to and can be combined with existing NAS algorithms.
We integrate GradSign into the NAS algorithms mentioned above and show that GradSign can consistently assist these NAS algorithms to discover more accurate architectures on various real-world tasks.
2. O PTIMIZATION L ANDSCAPE A NALYSIS
Inspired by the fact that over-parameterized networks always ﬁnd a remarkable ﬁt for a training dataset (Zhang et al., 2016), optimization landscape analysis has been one of the main focuses in deep learning theory (Brutzkus & Globerson, 2017; Du et al., 2018;
Ge et al., 2017; Li & Yuan, 2017; Soltanolkotabi, 2017; Allen-Zhu et al., 2019).
Even though existing theoretical results for θ * θ *l ( f θ ( x 1 ), y 1 ) θ *l ( f θ ( x 2 ), y 2 ) | θ * 1 − θ * 2 | θ l
[Figure]
Optimization
[Figure]
worse J ( θ ∗ ) .
θ l θ * θ *l ( f θ ( x 1 ), y 1 ) θ *l ( f θ ( x 2 ), y 2 ) | θ * 1 − θ * 2 |
[Figure]
optimization landscape analysis rely on strict assumptions on the landscape’s smoothness, convexity, and initialization point, we can leverage theoretical insights to guide the design of GradSign.
In addition, SGD-based optimizers trained from randomly initialized points hardly encounter nonsmoothness or non-convexity in practice for a variety of architectures (Goodfellow et al., 2014).
Furthermore, Allen-Zhu et al.
(2019) provides theoretical evidence that for a sufﬁciently large neighborhood of a randomly initialized point, the optimization landscape is nearly convex and semismooth.
Different from existing optimization landscape analyses depending on objectives evaluated across a mini-batch of training samples, we propose a new perspective that decomposes a mini-batch objective into the aggregation of sample-wise optimization landscapes.
To the best of our knowledge, our work is the ﬁrst attempt to MPI by leveraging sample-wise optimization landscapes.
T HEORETICAL F OUNDATIONS 3. I NSIGHTS
Conventional optimization landscape analyses focus on objectives across a mini-batch of training samples and miss potential evidence hidden in the optimization landscapes of individual samples.
By decomposing a mini-batch objective into the summation of sample-wise objectives across individual samples in a mini-batch, we can distinguish better local optima as illustrated in Figure 1.
Both Figure 1a and Figure 1b reach a local optimum at θ ∗ for the mini-batch objective
J = 12 ( l ( f θ ∗ ( x 1 ) , y 1 )
+ l ( f θ ∗ ( x 2 ) , y 2 )) .
However, the optimization landscape in Figure 1b contains a better local optimum θ ∗ (i.e., a lower J ).
This can be distinguished by analyzing the relative distance between local optima across training samples (i.e., | θ ∗ 1 − θ ∗ 2 | in Figure 1).
For a mini-batch with more than two samples, we use a sample-wise local optima density measurement Ψ deﬁned in Section 3.2 to represent the overall closeness of sample-wise local optima.
Intuitively, as the distances between the local optima across samples reduce (shown as the red areas in Figure 1), there is a higher probability that the gradients of different samples evaluated at a random initialization point have the same sign (shown as the green areas in Figure 1).
Driven by this insight, we propose
GradSign to infer the sample-wise local optima density Ψ statistically.
The design of GradSign is based on our theoretical results that a network with denser sample-wise local optima has lower training and generalization losses under reasonable assumptions.
We introduce the notations and assumptions in Section 3.2, provide a formal derivation of our theoretical results in Section 3.3, and present GradSign in Section 4.
3. P RELIMINARIES
We use S =
{ ( x i , y i ) } i ∈ [ n ] to denote training samples, where each x i ∈ R
d is a feature vector, and y i is the corresponding label.
We use l ( ˆ y i , y i ) to represent a loss function where ˆ y i is the prediction of our model.
We use f θ ( · ) :
R d → R o to denote the model parameterized by θ and use θ 0 ∈
R m to denote random initialized parameters where m is the number of model parameters.
Bold font constant denotes a constant vector such as 0 =
[0 , 0 , · · · , 0] whose dimension depends on the corresponding situation.
D denotes the underline data distribution, which is the same for training and testing.
Our theoretical results rely on an assumption that there exists a neighborhood Γ θ 0 for a random initialization point θ 0 in which sample-wise optimization landscapes are almost convex and semismooth.
Note that our assumption is weaker than that of (Allen-Zhu et al., 2019) since their analysis is focusing on the overall optimization landscape while we only consider the sample-wise optimization landscape which is a simpler case.
We use { θ ∗ i } i ∈ [ n ] ∈ Γ θ 0 to denote a local optima in the convex areas attached to the i -th sample near the initialization point θ 0 .
Under this assumption, the overall optimization landscape is also convex and semi-smooth within the neighborhood Γ θas additive operations preserve both.
We use θ ∗ to denote a local optimum within Γ θ 0 for the mini-batch optimization landscape.
Note that θ ∗ always lie in the convex hull of { θ ∗ i } i ∈ [ n ] .
Second, we assume that only gradient-based optimizers 1 are used during training.
Thus the optimizer eventually converges to θ ∗ .
Third, our theoretical analysis assumes that every Hessian in the set {∇ 2 l ( f θ ( x i ) , y i ) |∀ i ∈ [ n ] , θ ∈ Γ θ 0 } is almost diagonal as in the Neural Tangent Kernel (NTK) regime (Jacot et al., 2018).
Section 5 shows that our method generalizes well to real-world networks that may violate this assumption.
Sample-wise local optima density.
We use sample-wise local optima density to represent the relative closeness of { θ ∗ i } i ∈ [ n ] .
Given a dataset S
= { ( x i , y i ) } i ∈
[ n ] , an objective function l ( ˆ y i , y i ) , and a model class f θ ( · ) , we use Ψ S ,l ( f θ 0 ( · )) to measure the average distance between the local optima across samples { θ ∗ i } i ∈ [ n ] near a random initialization point θ 0 : Ψ
[Equation]
[Equation]
H ∈ R is a smoothness upper bound: ∀ k ∈ [ m ] , i ∈ [ n ] , [ ∇ 2 l ( f θ ( x i ) , y i )] k,k ≤ H .
This upper bound always exists due to the smoothness assumption.
Intuitively, Ψ S ,l ( f θ 0 ( · )) can be interpreted as the mean Manhattan distance with respect to each pair of { θ ∗ i } i ∈ [ n ] normalized by the inverse of the square root of the smoothness upper bound.
The denser { θ ∗ i } i ∈ [ n ] are, the smaller Ψ S ,l ( f θ 0 ( · )) is.
In an ideal case, Ψ S ,l ( f θ 0 ( · ))
=
when all local optima are located at the same point.
3. M AIN R ESULTS
We show the local optimum property of sample-wise optimization landscapes in the following lemma.
Lemma 1 There exists no saddle point in a sample-wise optimization landscape and every local optimum is a global optimum.
Using Lemma 1, we can draw a relation between the training error J = 1 n (cid:80) i l ( f θ ( x i ) , y i ) and Ψ S ,l ( f θ 0 ( · )) using the following theorem.
Theorem 2 The training error of a network on a dataset J = 1 n (cid:80) i
l
( f θ ( x i ) , y i ) is upper bounded by n 3 2 Ψ 2 S ,l ( f θ 0 ( · )) , and the bound is tight when Ψ S ,l ( f θ 0 ( · ))
= 0 .
Finally, we show that Ψ S ,l ( f θ 0 ( · )) also provides an upper bound for the generalization performance of a network measured by population loss.
Theorem 3 Given that Var ( x u ,y u )
∼D [ (cid:107) θ ∗ − θ ∗ u (cid:107) 21 ] is bounded by σ 2 where θ ∗ u is a local optimum attached to the convex area near θ 0 for l ( f θ ( x u ) , y u ) .
With probability 1 − δ , the true population loss is upper bounded by n 3 2 Ψ 2 S ,l ( f θ 0 ( · ))
+ σ √ nδ .
A formal proof of all theoretical results is available in Appendix A.2.
Main takeaways: A key takeaway of our theoretical results is that Ψ S ,l ( f θ 0 ( · )) closely relates to an upper bound of the training and generalization performance of a network.
Albeit theoretically sound, Ψ S l ( f θ 0 ( · )) is intractable to be directly measured.
Instead, we derive a simple yet accurate metric to reﬂect Ψ S ,l ( f θ 0 ( · )) , which we present in the next section.
G RAD S IGN
Inspired by the theoretical results derived above, we introduce GradSign, a simple yet accurate metric for model performance inference.
The key idea behind GradSign is a quantity to statistically reﬂect the relative value of Ψ S ,l ( f θ 0 ( · )) .
Speciﬁcally,
[Equation]
where C is a constant and k is the vector index.
Detailed derivation is given in Appendix A.2.
Using the above relation, we can infer Ψ S ,l ( f θ 0 ( · )) by directly measuring the signs of the gradients for a mini-batch of training samples at a randomly initialized point instead of going through an end-to-end training process.
To enable more efﬁcient calculation of Ψ S ,l ( f θ 0 ( · )) , we make a further simpliﬁcation and use the following sample observation:
[Equation]
to infer the true probability (cid:80)
[Equation]
(cid:80)
[Equation]
whose relation is given by:
[Equation]
The proof of this simpliﬁcation is included in Appendix A.2.
Given the above relationship, we formally state our algorithm pipeline in Algorithm 1.
Note that a higher GradSign score indicates better model performance as we have an inverse correlation in Equation (2). Algorithm 1: GradSign
Result:
GradSign score τ f for a function class f θ
Given S
= { ( x i , y i ) } i ∈
[ n ] , randomly select initialization point θ 0 ;
Initialize g [ n, m ] ; for i = 1 , 2 , · · · , n do for k = 1 , 2 , · · · , m do g
[ i, k ]
= sign ([ ∇ θ l ( f θ ( x i ) , y i )
| θ 0 ] k ) end end τ f = (cid:80) k
| (cid:80) i
g
[ i, k ] | ; return τ f
E XPERIMENTS
In this section, we empirically verify the effectiveness of our metric against existing gradient-based methods on three neural architecture search (NAS) benchmarks, including NAS-Bench-101 (Ying et al., 2019), NAS-Bench-201 (Dong & Yang, 2020) and NDS (Radosavovic et al., 2019) 2 .
Theorybased, sample-based, and learning-based methods are excluded in our evaluation, as they either require further training processes or have strong assumptions not suitable for generic architectures.
Baselines.
We compare GradSign against existing gradient-based methods, including snip (Lee et al., 2018), grasp (Wang et al., 2020), ﬁsher (Turner et al., 2019), and Synﬂow.
In addition, we also include grad_norm as a heuristic method and a one-shot MPI metric NASWOT(Mellor et al., 2021).
Since all gradient-based methods share a similar calculation pipeline (i.e., evaluating the gradients of a mini-batch at a random initialization point), we set the initialization mode and batch size to be the same across all methods to guarantee fairness.
Experimental setup details are included in Appendix A.3.
To align with the experimental setup of prior work (Abdelfattah et al., 2021b;
Mellor et al., 2021), we use two criteria to evaluate the correlations between different metrics and test accuracies across approximately 20k networks:
Spearman’s ρ (Daniel et al., 1990) characterizes the monotonic relationships between two variables.
The correlation score is restricted in range [-1, 1], where ρ = 1 denotes a perfect positive monotonic relationship and ρ = − 1 denotes a perfect negative monotonic relationship.
Following prior work, we use Spearman’s ρ to evaluate gradient-based methods on NAS-Bench-101 and NAS-Bench-201.
Kendall’s Tau .
Similar to Spearman’s ρ as a correlation measurement, Kendall’s Tau is also restricted between [-1, 1].
While Spearman’s ρ is more sensitive to error and discrepancies, Kendall’s Tau is more robust with a smaller gross error sensitivity.
We use Kendall’s Tau to quantify the correlation between one-shot metric scores and model testing accuracies over the NDS search space.
5. NAS-B ENCH -
NAS-Bench-101 is the ﬁrst dataset targeting large-scale neural architecture space, containing 423k unique convolutional architectures trained on the CIFAR-10 dataset.
The benchmark provides the test accuracy of each architecture in the search space, which we use to calculate the corresponding
Spearman’s ρ .
We use a randomly sampled subset with approximately 4500 architectures of the original search space and a batch size of 64 in this experiment.
Table 2 summarizes the results.
GradSign signiﬁcantly outperforms existing gradient-based methods and heuristic approaches and improves the Spearman’s ρ score by 25% compared to the best existing method ( 0 . 363 → 0 . 449 ).
[Table]
NAS-Bench-201 is an extended version of NAS-Bench-101 with a different search space, containing 15,625 cell-based candidate architectures evaluated across three datasets: CIFAR-10, CIFAR100 (Krizhevsky, 2009) and ImageNet 16-120 (Russakovsky et al., 2015).
The benchmark provides the test accuracies on the three datasets for all candidate architectures in the search space.
We evaluate Spearman’s ρ scores for GradSign and existing gradient-based methods.
The experiments were conducted overall 15,265 architectures in NAS-Bench-201.
The batch size is set to 64.
The results on the three datasets are summarized in Table 3.
GradSign consistently achieves the best performance across all three datasets and improves the Spearman’s ρ scores by ≈ 4% over the best existing approaches.
This improvement is signiﬁcant as the more Spearman’s ρ approaches 1 , the more difﬁcult it can be further improved.
[Table]
We further select 1000 architecture candidates randomly in the NAS-Bench-201 search space and visualize their testing accuracies against GradSign scores in Figure 2.
Figure 2 shows a highly positive correlation between the GradSign score and actual test accuracy of 1000 architectures.
A higher GradSign score indicates higher conﬁdence for the statistical performance of architecture.
Note that the GradSign scores show a clustering pattern, which may correspond to different architecture classes in the NAS-Bench-201 search space.
5. NAS D ESIGN S PACE (NDS)
NDS is a uniﬁed searching framework that includes ﬁve different design spaces: NAS-Net (Zoph et al., 2018), AmoebaNet (Real et al., 2019), PNAS (Liu et al., 2018a), ENAS (Pham et al., 2018),
DARTS (Liu et al., 2018b).
Each space contains approximately one thousand networks fully trained on the CIFAR-10 dataset.
We include the performance of our method along with grad_norm , Synﬂow, and NASWOT on all ﬁve design spaces evaluated by Kendall’s Tau and show the results in Table 4.
GradSign signiﬁcantly and consistently outperforms all other MPI methods in all ﬁve design spaces.
[Table]
5. A RCHITECTURE S ELECTION
We evaluate whether GradSign can be directly used to select highly accurate architectures in a NAS search space.
To pick a top architecture, we randomly sample N candidates in a NAS search space, choose the one with the highest GradSign score, and measure its validation/test accuracies (mean ± std).
We compare GradSign with Synﬂow, NASWOT, Random , and Optimal , where
Random uniformly samples architectures in the search space, while Optimal always chooses the best architecture across N candidates.
The results 3 are summarized in Table 5.
N in parenthesis indicates the number of architectures sampled in each run.
All methods can generally ﬁnd more accurate architectures with a high N .
In addition to outperforming Synﬂow and NASWOT, GradSign ( N = 100 ) can also ﬁnd better networks even compared to NASWOT( N = 1000 ).
The results show that GradSign can directly identify accurate architectures besides highly correlating to networks’ test accuracies.
[Table]
5. G RAD S IGN -A SSISTED N EURAL A RCHITECTURE S EARCH
Besides evaluating GradSign on the Spearman’s ρ and Kendall’s Tau scores as prior work [Table] we also integrate GradSign into various neural architecture search algorithms and evaluate how GradSign can assist neural architecture search on real-world tasks.
Speciﬁcally [Table] we integrate GradSign into four NAS algorithms: REA, REINFORCE, BOHB and RS.
We design a corresponding method for each NAS algorithm that uses the GradSign scores of candidate architectures to guide the search.
Speciﬁcally, we integrate GradSign into each NAS algorithm by replacing the random selection of architectures with GradSign-assisted selection.
We name these GradSign-assisted variants G-REA,
G-REINFROCE, G-HB, and G-RS, and describe their algorithm details in Appendix A.
To evaluate how GradSign can improve the search procedure of NAS algorithms, we run each algorithm with and without GradSign’s assistance for 500 runs on NAS-Bench-201 and report the validation and test accuracies of the best-discovered architecture in each run.
Following prior work (Mellor et al., 2021; Dong & Yang, 2020), all searches are conducted on the CIFAR-10 dataset with a time budget of 12000s while the performance is evaluated on CIFAR-10, CIFAR-100 and
ImageNet16-120.
The baselines also include A-REA (Mellor et al., 2021), a variant of REA that uses the NASWOT scores at the initial population selection phase.
Table 6 shows the results.
The GradSign-assisted NAS algorithms outperform their counterparts by improving test accuracy by up to 0.3%, 1.1%, and 1.0% on the three datasets.
[Table]
C ONCLUSION
In this paper
[Table]
we propose a model performance inference metric GradSign and provide theoretical foundations to support our metric.
Instead of focusing on full batch optimization landscape analysis, we move a step further to sample-wise optimization landscape properties [Table] which give us additional information to uncover the quality of the local optima encountered on the optimization trajectory.
We propose Ψ S ,l ( f θ 0 ( · )) to quantitatively characterize the potential of a model f θ ( · ) at a random initialization point θ 0 based on our theory results.
Finally, we design the GradSign metric to statistically infer the value of Ψ S ,l ( f θ 0 ( · )) to give out our ﬁnal score for model performance inference.
Empirically, we have demonstrated that our method consistently achieves the best correlation with true model performance among all other gradient-based metrics.
In addition, we also veriﬁed the practical value of our method in assisting existed NAS algorithms to achieve better results.
Given that our metric is generic and promising, we believe that our work not only assists in accelerating
MPI-related applications but sheds some light on optimization landscape analysis as well.
Meanwhile, the effectiveness of our method may further reduce the energy cost introduced by modern NAS algorithms.
In addition, one of the future work of GradSign can be adding normalization across different architecture classes to tackle the clustering problem in Figure
[Table] A. P ROOF
Lemma 1 Proof:
For a single training sample ( x i , y i ) , we minimize its objective function l ( f θ ( x i ) , y i ) with gradient descent: [Equation]
[Equation]
At any local optimal point θ ∗ i , we have ∇ θ l ( f θ ( x i ) , y i )
| θ ∗ i = 0 , but f (cid:48) θ ( x i ) (cid:54) = 0 for conventional neural architectures with at least one dense layer 4 .
Therefore, we show ∂l ( f θ ( x i ) ,y i )
∂f θ ( x i ) | θ ∗ i must be equal to 0 .
For the commonly used objective functions, such as Mean Squared Error Loss and Cross
Entropy Loss, this derivative is equal to C ( f θ ( x i ) − y i ) , where C is a non-zero constant.
Hence we have f θ ∗ i ( x i ) =
y i and l ( f θ ∗ i ( x i ) , y i )
=
at local optima θ ∗ i , which makes θ ∗ i also a global optima as it is impossible to obtain a lower loss value for this single sample.
In addition, at local optima θ ∗ i , we have:
[Equation]
The above equation implies that ∇ 2 θ l ( f θ ( x i ) , y i ) is a positive semi-deﬁnite matrix, since C > 0 .
This concludes the proof of non existence of saddle points in a sample-wise optimization landscape.
This result aligns with our convexity assumptions in Section 3.2.
Theorem 2 Proof: Recall that θ ∗ denotes a local optima of J which could be reached by a gradientﬂow based optimizer start from θ 0 .
Since θ 0 is randomly sampled and [ ∇ 2 l ( f θ ( x i ) , y i )] k,k ≤ H , we have: [Equation]
[Equation]
where Eq 10 → Eq 11 uses the basic property of the smoothness upper bound H and the fact that each local optimum θ ∗ i satisﬁes l ( f θ ∗ i ( x i ) , y i )
= 0 .
Eq 11 → Eq 12 uses Jensen Inequality for square root operators.
Eq 12 → Eq 13 is derived from the fact that for each dimension in ( θ ∗ − θ ∗ i ) we have:
[Equation]
Otherwise, θ ∗ dose not lie in the convex hull of { θ ∗ i } i ∈ [ n ] which contradicts with our assumption stated in Section 3.2.
The bound is tight when θ ∗ i
= θ ∗ j , ∀ i, j ∈ [ n ] .
Eq 10 → Eq 11: As we have ∇ 2 l ( f θ ( x i ) , y i ) (cid:22) H I and l ( f θ ∗ i ( x i ) , y i ) = 0 , ∇ θ l ( f θ ( x i ) , y i ) | θ ∗ i = 0 , we could derive the following inequality:
[Equation]
17) we thus have 1 n
[Equation]
i
[Equation]
Theorem 3 Proof: Let E ( x u ,y u ) ∼D
[ l ( f ∗ θ ( x u ) , y u )] denotes the true population error.
With probability 1 − δ , we have:
[Equation]
where n and σ are constants, S is a training dataset, and D denotes its underlying data distribution.
This implies that Ψ S ,l ( f θ 0 ( · )) is an accurate indicator for the true population loss.
Eq 18 → Equses Chebyshev’s inequality, while Eq 19 → Eq 20 uses the same inequality derived in Claim 2 .
Algorithm Proof:
Given that sample-wise local optima { [ θ ∗ i ] , i ∈ [ n ] } are contained in the convex area around θ 0 .
We derive the following property:
[Equation]
Since θ 0 is a randomly chosen initialization point, without loss of generality, we assume θ 0 is sampled from a hypercube [ − a, a ] .
Thus we have:
[Equation]
Where P ( sign ([ ∇ θ l ( f θ ( x i ) , y i ) | θ 0 ] k ) (cid:54) =
sign ([ ∇ θ l ( f θ ( x j ) , y j ) | θ 0 ] k )) denotes the probability for [ ∇ θ l ( f θ ( x i ) , y i )
| θ 0 ]
k and [ ∇ θ l ( f θ ( x j ) , y j ) | θ 0 ]
k having different signs.
Notice that we have completely dropped the dependency for θ ∗ i at this point and can simply infer from sign ([ ∇ θ l ( f θ ( x i ) , y i )
| θ 0 ] k ) .
To complete our proof:
[Equation]
Simpliﬁcation
Proof: : for each i, j and a given k , we could estimate
P ( sign ([ ∇ θ l ( f θ ( x i ) , y i ) | θ 0 ] k ) = sign ([ ∇ θ l ( f θ ( x j ) , y j ) | θ 0 ] k )) using:
[Equation]
(26) which is valid (not equal to zero) only when sign ([ ∇ θ l ( f θ ( x i ) , y i ) | θ 0 ] k )
=
= sign ([ ∇ θ l ( f θ ( x j ) , y j ) | θ 0 ] k ) .
Suppose we have p positive and n-p negative sign ([ ∇ θ l ( f θ ( x i ) , y i ) | θ 0 ] k ) for n samples, since we only care about samples share the same sign, the original probability estimation is ( n − p ) 2 + p 2 n= 1 2 + ( n − 2 p ) 2 2 n 2 .
Thus we only need to measure the quantity | n − 2 p | which simply equals to | (cid:80) i sign ([ ∇ θ l ( f θ ( x i ) , y i ) | θ 0 ] k ) | .
A. E XPERIMENTS S ETUP
The code we used during experimentation was mainly based on existed code base Abdelfattah et al.
(2021b)
; Mellor et al.
(2021)Abdelfattah et al.
(2021b) which is under Apache-2.0 License.
The hardwares we used were Amazon EC2 C5 instances with no GPU involved and p3 instance with one V100 Tensor Core GPU.
As our methods is gradient-based which is training free, we don’t need to split our dataset.
For
Spearman’s correlation measurement on NAS-Bench-201, we set batch size to 64, which is used by most baselines.
For the Kendall’s Tau experiment, other accuracy comparison experience and the GradSign assisted algorithms, we used a batch size of 128, also to match the batch size in other baselines (NASWOT).
We use Pytorch default parameter initialization for all architectures.
Random seed in correlation experiments is set to 42 which is also randomly chosen.
For accuracy experiments, our results are summarized over 500 runs whose random seed are chosen randomly for each run.
For the correlation evaluation of each individual architecture, we only use one θ 0 for minimizing computational cost.
Our approach can be easily generalized to an average of multiple θ 0 s and can trade-off between efﬁciency and accuracy.
A. A DDITIONAL RESULTS
Sample-based: Fig 3 compares EconNAS, a sample-based method, with existing gradient-based methods on MPI.
Results of EconNAS are referenced from Abdelfattah et al.
(2021b).
To achieve a similar MPI performance as GradSign, EconNAS needs 500 minibatches of samples for each candidate’s proxy training, while all gradient-based methods (including GradSign) require only one minibatch.
By increasing the number of minibatches, EconNAS can achieve higher Spearman’s ρ scores, which eventually converge to 0.85.
At that point overﬁtting takes place and the score cannot be further improved.
[Figure]
EconNAS requires more than 500 minibatches to have a better performance
Learning-based:
Table 7 compares GradSign and existing learning-based methods (MLP, LSTM, and GATES) on the Kendall’s Tau correlation score.
The MLP, LSTM, and GATES results are referenced from Ning et al.
For MLP and LSTM (Wang et al., 2019), the predictor uses Multi Layer Perceptron (MLP) and Long Short Term Memory (LSTM) as the base predictor, while
GATES (Ning et al., 2020) uses Graph Neural Network (GNN) as the base predictor.
All results are obtained on NAS-Bench-201 and the GradSign’s score is averaged over all three datasets (CIFAR-10, CIFAR-100 and ImageNet16-120) as (Ning et al., 2020) does not provide which dataset is used for calculating Kendall’s Tau.
To achieve a similar score as GradSign, MLP, LSTM and GATES-1 require an average of 1959, 978 and 1959 minibatches per sample respectively to prepare the dataset for training the predictors.
Although GATES-2 achieves a better correlation score than GradSign, it still needs 195 minibatches per sample to prepare the dataset for training the GATES-2 predictor.
In addition to the cost of preparing a training dataset, each predictor also has to be trained on the dataset as well, which involves 200 more epochs while the cost of evaluating GradSign is one mini-batch.
With less minibatches evaluated for learning-based methods, their training set sizes shrink signiﬁcantly (e [Table] g.,mini-batches equal to 78 training samples and 7813 testing samples).
This may result in overﬁtting to the training set.
[Table]
We also includes the evaluation of GradSign for both MPI correlation performance and GradSignassisted NAS algorithms on the latest version of NAS-Bench-201 (NATS-Bench) across three datasets (CIFAR-10, CIFAR-100 and ImageNet16-120) in Table 8, Table 9.
Results show GradSign is robust against hyper-parameter tuning as long as the trained networks can converge to near optimal.
Also notice that GradSign-assisted NAS algorithms could not only achieve a better accuracy but lower variance as well compared to their non-assisted counterparts.
[Table]
To make a fair comparison across all the methods, the search is performed on CIFAR-100 dataset while the
All the methods have a search time budget of 12000s.
Note that the benchmark results might not match with the original paper as we have run all the experiments from start in a environment different from Dong & Yang (2020).
[Table]
To demonstrate the potential for GradSign in a more complicated computer vision task, we compare the performance of GradSign with ZenNAS following their setups and search space.
Due to the limitation of computational resources 5 , we only run 10000 evolution iterations using solely Zen score or GradSign score to select the architecture candidate and 20 epochs to train the selected architecture.
Following ZenNAS’s setup, EfﬁcientNet-B3 is used as teacher network when training selected architectures.
Though the top-1 validation accuracy of GradSign in the ﬁrst 20 epochs is slightly better than Zen, we should note that this process can highly depend on the random seed for evolution search phase.
As we mentioned before, ZenNAS uses linear region analysis which makes it less ﬂexible for arbitrary activation functions.
On the other hand, since ZenNAS calculates an architecture complexity related score which is both dataset independent and initialization independent, it can be too general and results in low Spearman’s ρ as shown in Table 3.
[Figure]
Result: Find the best performing architecture given the time constraint for 12000s Population
History = ; population_size; sample
; pool ; for population_size do model
; model.acc, model.time_cost eval model);
Population.append(
History.append(model); endwhile not exceeding time budget do
Sample = []; for 1 , 2 , · · · , sample_size do
Sample.append(random.choice(Population))
end parent = max_acc(Sample);
GradSign_pool =
[]; for 1 , 2 , · · · , pool_size do /*
GradSign assisted part */ model = mutate_arch(parent); model.score = GradSign(model);
GradSign_pool.append(model) ; end child = max_score(GradSign_pool); Population.append(child);
History.append(child);
Population.popleft
; endreturn max_acc(History)
Algorithm 3: RS
Result: Find the best performing architecture given the time constraint for 12000s
History =
[]; pool_size; while not exceeding time budget do
GradSign_pool =
[]; for 1 , 2 , · · · , pool_size do /*
GradSign assisted part */ model =
random_arch(); model.score = GradSign(model);
GradSign_pool.append(model)
; end arch =
max_score(GradSign_pool) arch.acc = eval(arch) History.append(arch); endreturn max_acc(History);
Result
Input do