ZARTS: On Zero-order Optimization for Neural Architecture Search
Abstract
Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency.
It introduces trainable architecture parameters to represent the importance of candidate operations and proposes first/secondorder approximation to estimate their gradients, making it possible to solve NAS by gradient descent algorithm.
However, our in-depth empirical results show that the approximation often distorts the loss landscape, leading to the biased objective to optimize and, in turn, inaccurate gradient estimation for architecture parameters.
This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation.
Specifically, three representative zero-order optimization methods are introduced:
RS, MGS, and GLD, among which MGS performs best by balancing the accuracy and speed.
Moreover, we explore the connections between RS/MGS and the gradient descent algorithm and show that our ZARTS can be seen as a robust gradient-free counterpart to DARTS.
Extensive experiments on multiple datasets and search spaces show the remarkable performance of our method.
In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue.
Also, we search on the search space of DARTS to compare with peer methods, and our discovered architecture achieves 97.54% accuracy on CIFAR-10 and 75.7% topaccuracy on ImageNet.
Finally, we combine our ZARTS with three orthogonal variants of DARTS for faster search speed and better performance.
Introduction
Despite their success, neural networks are still designed mainly by humans [27, 14, 15].
It remains open to search for efficient architectures automatically.
Neural architecture search (NAS) has attracted wide attention, which can be modeled as bi-level optimization for network architectures and operation weights.
One-shot NAS [1] is a popular search framework that regards neural architectures as directed acyclic graphs (DAG) and constructs a supernet with all possible connections and operations in the search space.
DARTS [20] further introduces trainable architecture parameters to represent the importance of candidate operations, which are alternately trained by SGD optimizer along with network weights.
It proposes a first-order approximation to estimate the gradients of architecture parameters, which is biased and may lead to the severe instability issue shown by [3].
Other works [38, 6] point out that architecture parameters will converge to a sharp local minimum resulting in the instability issue, so they introduce extra regularization items making architecture parameters converge to a flat local minimum.
In this paper, we empirically show that the first-order approximation of optimal network weights sharpens the loss landscape and results in the instability issue of DARTS.
It also shifts the global minimum, misleading the training of architecture parameters.
To this end, we discard such approximation and turn to zero-order optimization algorithms, which can run without the requirement that the search loss is differentiable w.r.t.
architecture parameters.
Specifically, we introduce a novel NAS scheme named ZARTS, which outperforms DARTS by a large margin and can discover efficient architectures stably on multiple public benchmarks.
This paper sheds light on the frontier of NAS by: 1) Establishing zero-order based robust paradigm to solve bi-level optimization for NAS.
Differentiable architecture search has been a well-developed area [20, 34, 31] which solves the bi-level optimization of NAS by gradient descent algorithms.
However, this paradigm suffers from the instability issue during search since biased approximation for optimal network weights distorts the loss landscape, as shown in Fig.
1 (a) and (b).
To this end, we propose a flexible zero-order optimization NAS framework to solve the bi-level optimization problem, which is compatible with multiple potential gradient-free algorithms in the literature.
Uncovering the connection between zero-order architecture search and DARTS.
This work introduces three representative zero-order optimization algorithms without enforcing the unverified differentiability assumption for search loss w.r.t.
architecture parameters.
We reveal the connections between the zero-order algorithms and gradient descent algorithm, showing that two implementations of ZARTS can be seen as gradient-free counterparts to DARTS, being more stable and robust.
Strong empirical performance and robustness.
Experiments on four datasets and five search spaces show that, unlike DARTS which suffers the severe instability issue [
ZARTS
can stably discover effective
architectures
on various
benchmarks
In particular, the searched
architecture
achieves 75. top- accuracy on ImageNet outperforming DARTS and most of its variants
Also,
our ZARTS
can
be
further improved by combining with orthogonal DARTS variants , achieving on CIFAR
after searching in 0.
GPU
day.
Related Work
One-shot Neural Architecture Search.
[1] construct a supernet so that all candidate architectures can be seen as its sub-graph.
DARTS [20] introduces architecture parameters to represent the importance of operations in the supernet and update them by gradient descent algorithm.
Some works [34, 31, 10] reduce the memory requirement of DARTS in the search process.
Other works [38, 6] point out the instability issue of DARTS, i.e., skip-connection gradually dominates the normal cells, leading to performance collapse during the search stage.
Bi-level Optimization for NAS.
NAS can be modeled as a bi-level optimization for architecture parameters and network weights.
DARTS [20] proposes the first/second-order approximations to estimate gradients of architecture parameters so that they can be trained by gradient descent.
However, we show that such an approximation will distort the loss landscape and mislead the training of architecture parameters.
Amended-DARTS [3] derives an analytic formula of the gradients w.r.t. architecture parameters that require the Hessian inverse of network weights, which is even unfeasible to compute.
This work discards the approximation in DARTS and attempts to solve the bi-level optimization by gradient-free algorithms.
Zero-order Optimization.
Unlike gradient-based optimization methods that require the objective differentiable w.r.t.
the parameters, zero-order optimization can train parameters when the gradient is unavailable or difficult to obtain, which has been widely used in adversarial robustness for neural networks [5, 16], meta learning [28], and transfer learning [29].
[22] aim at AutoML and utilize zero-order optimization to discover optimal configurations for ML pipelines.
In this work (to our best knowledge), we make the first attempt to apply zero-order optimization to NAS and experiment with multiple algorithms, from vanilla random search [11] to more advanced and effective direct search [12], showing its great superiority against gradient-based methods.
Bi-level Optimization in DARTS
Following one-shot NAS [1], DARTS constructs a supernet stacked by normal and reduced cells.
Cells in the supernet are denoted by directed acyclic graphs (DAG) with N nodes { x i } Ni =1 , which [Figure] represents latent feature maps.
Each edge e i,j contains multiple operations { o i,j , o ∈ O} , whose importance is represented by architecture parameters
α o i,j .
Therefore, NAS can be modeled as a bi-level optimization problem by alternately updating the operation weights ω (parameters within candidate operations on each edge) and the architecture parameters α :
[Equation]
3.1 Fundamental Limitations in DARTS
By enforcing an unverified (and in fact difficult to verify) assumption that the search loss
L val ( ω ∗ ( α ) , α ) is differentiable w.r.t.
α , DARTS [20] proposes a second-order approximation for the optimal weights
ω ∗ ( α ) by applying one-step gradient descent:
[Equation]
(2) where ξ is the learning rate to update network weights.
Thus the gradient of the loss w.r.t.
α , ∇ α L val ( ω ∗ ( α ) , α ) , can be computed by the chain rule: ∇ α L val ( ω ∗ ( α ) , α )
≈ ∇ α L val ( ω ′ , α ) − ξ ∇ 2 α ,
ω L train ( ω , α ) ∇ ω ′ L val ( ω ′ , α ) .
Nevertheless, the second-order partial derivative is hard to compute, so the authors adopt the difference method, which is proved in the appendix.
To further reduce the computational cost, first-order approximation is introduced by assuming ω ∗ ( α ) being independent of α , as shown in Eq. 3, which is much faster and widely used in many variants of
DARTS [7, 31, 38].
[Equation]
The gradient is then simplified as: ∇ α L val ( ω ∗ ( α ) , α )
≈ ∇ α L val ( ω , α ) , which exacerbates the estimation bias.
Reexamining the definition of ω ∗ ( α ) in Eq.
1, one would note that it is intractable to derive a mathematical expression for ω ∗ ( α ) , making L val ( ω ∗ ( α ) , α ) even non-differentiable w.r.t.
α .
Yet DARTS has to compromise with such approximations as Eq.
and Eq.
3 so that differentiability is established, and SGD can be applied.
However, such sketchy estimation of optimal operation weights can distort the loss landscape w.r.t.
architecture parameters and thus mislead the search procedure, shown in Fig. 1 and analyzed in the next section.
3.2 Distorted Landscape and Biased Optimization
Fig.
1 illustrates the loss landscape with perturbations on architecture parameters α , showing how different approximations of ω ∗ affect the search process.
We train a supernet for 50 epochs and randomly select two orthonormal vectors as the directions to perturb α .
The same group of perturbation directions is used to draw landscapes in Fig.
1(a)
and (b) for a fair comparison.
Fig.
1(a) shows the loss landscape with the first-order approximation in DARTS, ω ∗ 1 st ( α ) = ω , while Fig. 1(b) shows the loss landscape with more accurate ω ∗ ( α ) , which is obtained by fine-tuning the network weights ω for 10 iterations for each α .
Landscapes (contours) are plotted by evaluating L at grid points ranging from -1 to 1 at an interval of 0.02 in both directions.
Global minima are marked with stars on the landscapes, from which we have two observations: 1)
The approximation ω ∗ 1 st ( α ) =
ω
Hyper-parameters: Operation weights ω , architecture parameters α , sampling number N , iteration number M , update estimation function ϕ ( · ) .
repeatSample candidates: { u i } Ni =1 , and get α ± i = α ± u
i .
Estimate optimal operation weights ω ∗ ( α ± i ) by descending ∇ ω L train ( ω , α ± i ) for M iterations;
Get update direction: u ∗ =
ϕ (cid:0) { u i , ω ∗ ( α ± i ) }
Ni =1 (cid:1)
Update architecture parameters: α ← α + u ∗ ; until Converged ⋆
The sampling strategies and update estimation functions ϕ ( · ) for three different zero-order optimization algorithms are detailed in Table 1.
shifts the global minimum and sharpens the landscape 2 , which is the representative characteristic of instability issue as pointed out by [38]. 2)
Accurate estimation for ω ∗ leads to a flatter landscape, indicating that the instability issue can be alleviated.
Moreover, we display the landscape with second-order approximation ω ∗ 2 nd in the appendix, which is also sharp but slightly flatter than Fig.(a).
Consequently, we discard the first/second-order approximation in DARTS and instead use more accurate ω ∗ coordinated with zero-order optimization.
Fig.
1 (c) shows the optimization paths of DARTS and three methods of ZARTS, illustrating how the approximation in DARTS affects the search process.
Starting from the same randomly generated position, we update architecture parameters α for ten iterations by DARTS and ZARTS and draw the tracks of the optimization path.
ZARTS can gradually converge to the global minimum, while
DARTS converges to an incorrect point.
Zero-order Optimization for NAS
This paper goes beyond the 1st/2nd-order approximation in DARTS and proposes to train architecture parameters α by zero-order optimization, allowing for more accurate estimation for ω ∗ ( α ) .
Alg.outlines the generic form of our ZARTS framework.
We adopt three representative techniques: a vanilla zero-order optimizer, random search (RS) [21], and two advanced algorithms: Maximumlikelihood Guided Parameter Search (MGS) [32] and GradientLess Descent (GLD) [12], presented in Sec.
4.1, 4.2, 4.3 as preliminaries.
Further, we theoretically establish the connection between
ZARTS and DARTS, showing that ZARTS with RS and MGS optimizer can be seen as an expansion of DARTS.
In the following, we denote L ( α ) ≜
L val ( ω ∗ ( α ) , α ) as the objective w.r.t.
architecture parameters α ∈ R d
(Eq. 1), and L ( α + u ) ≜
L val ( ω ∗ ( α + u ) , α + u ) , where u is the update for α .
4.1 Optimizer I: ZARTS-RS
We adopt Multi-point estimator , shown in Table 1, as a baseline of our ZARTS:
[Equation]
where u ∼ q is sampled from a spherically symmetric distribution q , µ > 0 is a smoothing parameter, and φ ( d ) is a dimension-dependent factor related to q .
Specifically, φ ( d ) = 1 when q is a standard normal distribution N ( 0 , I ) , and φ ( d ) =
d when q is a uniform distribution on a unit sphere S d − 1 .
4.2 Optimizer II: ZARTS-MGS
Maximum-likelihood guided parameter search (MGS) is an advanced zero-order optimization algorithm for machine translation [32].
We attempt to apply it to the NAS task.
We first define a distribution for the update of architecture parameters, u , as follows:
[Equation]
[Table] where (cid:101) p ( u | α ) = exp ( − [ L ( α + u ) − L ( α )] /τ ) is an unnormalized exponential distribution [Table] and
Z ( α ) =
(cid:82) (cid:101) p ( u | α ) d u is its normalization coefficient.
τ is a temperature parameter controlling the variance of the distribution
[Table]
Intuitively, u with higher probability contributes more to the objective.
Hence, the optimal update of architecture parameters can be estimated by u ∗ =
E u ∼ p ( u | α )
[ u ] .
However, since the probability p ( u | α ) is an implicit function relying on L ( α + u ) , making it impractical to obtain the expectation, we refer to [32] and apply importance sampling to sample from a proposal distribution q ( u | α ) with known probability function:
[Equation]
where { u i } Ni =1 are sampled from the proposal distribution q ( u | α ) .
Similarly, the normalization coefficient Z ( α ) can be computed as follows:
[Equation]
For convenience [Equation] we define a ratio representing the weight on each sample as (cid:101) c
( u | α ) = (cid:101) p ( u | α ) q ( u | α ) .
The optimal update for architecture parameters in Eq.
6 can be computed by:
[Equation]
Finally, the architecture parameters are updated by α ← α + ˆ u ∗ .
Importance sampling diagnostics are conducted to verify the effectiveness of ZARTS-MGS (in the appendix).
4.3 Optimizer III: ZARTS-GLD
Unlike the above two algorithms that estimate gradient or the update for α , [12] propose the so-called
GradientLess Descent (GLD) algorithm, which falls into the category of truly gradient-free (or direct search) methods.
The authors provide solid theoretical proof of the efficacy and efficiency of the GLD algorithm and suggestions on the choice of search radius boundaries.
Specifically, they prove that the distance between the optimal minimum and the solution given by GLD is bounded and positively correlated with the condition number of the objective function, where the condition number Q is defined as:
[Equation]
The loss landscape in Fig. 1(b) is pretty flat, implying a low condition number, thus the high efficiency of ZARTS-GLD.
Specifically, at each iteration, with a predefined search radius boundary [ r, R ] , we independently sample candidate updates { u i } for architecture parameters on spheres with various radii { 2 − k R } log( R/r ) k
=  and perform function evaluation at these points.
By comparing L ( α ) and {L ( α + u i ) } , α steps to the point with minimum value, or stay at the current point if none of them makes an improvement.
The architecture parameters are then updated by α ← α + u ∗ .
[Equation]
4.4 Connection between DARTS and ZARTS
The similarity between gradient-estimation-based zero-order optimization and SGD builds an essential connection when the objective function is differentiable.
For ZARTS-RS, we adopt a multi-point estimator to approximate the gradient with a limited bound [23, 2].
Details are put in the supplementary material.
Therefore, ZARTS-RS degenerates to second-order DARTS if L ( α ) is indeed differentiable w.r.t α and the iteration number M is set to 1.
Next, we theoretically show that MGS [32] will degenerate to the gradient descent algorithm if the first-order Taylor approximation is applied.
Then we analyze the relationship between ZARTS-MGS and DARTS.
Proposition 1.
Assuming that L ( α ) in Eq.
1 is differentiable w.r.t.
α , MGS algorithm [32] degenerates to SGD (used in vanilla DARTS) by the first-order Taylor approximation for L ( α ) , i.e., u ∗ ∝ −∇ α L ( α ) .
Proof.
Denote g ≜ ∇ α L ( α ) as the gradient of L .
The Taylor series of L at α up to the first order gives L ( α + u ) − L ( α ) ≈ u ⊤ g .
Applying it to p ( u | α ) in Eq.
5 yields:
[Equation]
Since ∥ u ∥ is constrained within ε to make sure the rationality of first-order Taylor approximation, the optimal update u ∗ then becomes:
[Equation]
Note that u ⊤ g = −∥ u ∥∥ g ∥ cos η , where η is the angle between u and g .
According to the symmetry of integral, Z ( g ) is determined once ∥ g ∥ is given.
Therefore, we can formulate Z ( g ) as Z ( ∥ g ∥ ) .
According to the chain rule:
[Equation]
Since Z ( ∥ g ∥ ) , ∇ ∥ g ∥ Z ( ∥ g ∥ ) ,
∥ g ∥ are all scalars, we have
[Equation]
That is, the optimal update u ∗ in MGS algorithm shares a common direction with the negative gradient −∇ α L ( α ) , as used by gradient descent.
Based on Proposition 1, ZARTS-MGS can be seen as an expansion of DARTS, and it degrades to first-order and second-order DARTS when ω ∗ is estimated by ω ∗ 1 st and ω ∗ 2 nd , respectively.
In general,
ZARTS-RS/-MGS can degenerate to DARTS, given the differentiability assumption.
However, unlike DARTS, which has to estimate ω ∗ ( α ) by ω ∗ 1 st or ω ∗ 2 nd to satisfy the differentiablity property of L val and update α by gradient descent algorithm, ZARTS, without such assumptions, can compute ω ∗ ( α ) by training network weights ω for arbitrary numbers of iterations, leading to more robust and effective training for architecture parameters, as shown in the next section.
4.5 Variants and Speedup of ZARTS
ZARTS can be flexibly and seamlessly combined with variants of DARTS for boosting as it can be seen as an expansion of DARTS as analyzed in section 4.4.
Here, we derive three variants of
ZARTS by combining it with prior works: P-ZARTS based on P-DARTS [7], GZAS based on
GDAS [10], and MergeZARTS based on MergeNAS [31].
Implementation details are illustrated in the appendix.
These variants can speed up ZARTS twice and reduce the search cost to 0.5 GPU-day.
Moreover, GZAS and MergeZARTS also reduce the GPU memory cost during the search process.
The performance of these variants is reported in Table 5.
[Table]
Experiments
We first verify the stability of ZARTS (with three zero-order optimization methods RS, MGS, GLD) on the four popular search spaces of R-DARTS [38] on three datasets including CIFAR-10 [17],
CIFAR-100 [17] [Table] and SVHN [24].
We then follow Amended-DARTS [3] and empirically evaluate the convergence ability of our method by searching for 200 epochs.
Performance trends of the discovered architectures are drawn in Fig. 2(a).
Next, we compare with the peer methods on the search space of
DARTS [20] to show the efficacy of our method.
Finally
[Table] we derive three variants of ZARTS with faster search speed by combining ZARTS with the variants of DARTS [Table] showing the flexibility and potential of our ZARTS framework.
All the experiments are conducted on NVIDIA 2080Ti.
The details of search spaces and experiment settings are given in the appendix.
5.1 Stability Evaluation
[Table]
The instability issue of DARTS has recently drawn increasing attention.
Amended-DARTS [3] shows that skipconnection gradually dominates the discovered architectures after searching by DARTS for 200 epochs.
RDARTS [38] proposes four search spaces, S1-S4, which amplify the instability of DARTS, as such dominance occurs after only 50 epochs of searching.
These studies expose the instability of gradient-based methods.
To verify the stability of our method, we search on S1-S4 proposed by R-DARTS and conduct convergence analysis following AmendedDARTS.
Performance on S1-S4.
We first search on the four spaces on CIFAR10, CIFAR-100, and SVHN.
Our evaluation settings are the same as R-DARTS [38].
Four parallel tests are conducted on each benchmark, among which the best is reported in Table 2.
We observe that ZARTS achieves outstanding performance with great robustness on 12 benchmarks.
All three zero-order algorithms outperform DARTS notably.
Even the vanilla zero-order algorithm ZARTS-RS achieves similar robust performance as R-DARTS, which verifies our analysis in Fig.
1, i.e., the coarse estimation ω ∗ 1 st in DARTS distorts the landscape and causes instability.
Note that though
ZARTS-GLD performs best in Table 2, it falls into the category of direct search methods, requiring
[Figure]
more sampling for candidate updates u and thus more search cost (2.2 GPU-days on the search space of DARTS) than ZARTS-MGS (1.0 GPU-days).
ZARTS-MGS is chosen by default if not otherwise specified, considering the trade-off between accuracy and speed.
Convergence Analysis.
The convergence ability of NAS methods describes whether a search method can stably discover effective architectures along the search process, i.e., whether the discovered architectures’ ultimate performance (top-1 accuracy) can converge to a high value.
AmendedDARTS [3] empirically shows that DARTS has a poor convergence ability: accuracy of the supernet increases but the ultimate performance of the searched network drops.
Following Amended-DARTS, we run ZARTS and DARTS for 200 epochs and show the trend of performance and number of parameters in Fig.
Specifically, we derive one network every 25 epochs during the search process and train each network for 600 epochs to evaluate its ultimate performance.
We observe that the networks discovered by ZARTS perform stably well (around 97.40% accuracy), while the performance of networks found by DARTS gradually drops.
Moreover, the parameter number of networks searched by DARTS decreases significantly after 50 epochs, indicating that parameterless operations dominate the topology and the instability issue [39] occurs.
On the contrary, ZARTS consistently discovers effective networks with about 4.0M parameters, showing the remarkable stability of our method.
More details and visualization of architectures are shown in the appendix.
5.2 Performance on the Search Space of DARTS
[Table]
Results on CIFAR.
We conduct four parallel runs by searching with different random seeds and separately training the searched architectures for 600 epochs
[Table]
The best and average accuracy of four parallel tests are reported in Table 3.
In particular,
ZARTS achieves 97.46% average accuracy and 97.54% best accuracy on CIFAR-10, outperforming DARTS and its variants.
Also, compared with AmendedDARTS that approximates optimal operation weights ω ∗ ( α ) by
Hessian matrix, our method can stably discover effective architectures in fewer GPU days.
Moreover, Table 3 shows our method achieves 83.71% and 84.54% for mean and best accuracy (i.e., 1- error%), on CIFAR-100, outperforming the compared methods by more than 1%.
Results on ImageNet.
For the transferability test, we follow the settings of DARTS to transfer the network discovered on CIFAR-10 to ImageNet.
Models are constructed by stacking 14 cells with 48 initial channels.
We train 250 epochs with a batch size of 1024 by SGD with a momentum of 0.9 and a base learning rate of 0.5.
We utilize the same data pre-processing strategies and auxiliary classifiers as DARTS.
Table 4 shows the performance of our searched networks.
ZARTS (5.6M) [-has5.
 M-]has 5.6M parameters and achieves 75.7% top-1 accuracy on the validation set of ImageNet, and ZARTS (5.0M) has 5.0M parameters and achieves 75.5% accuracy.
Their structure details are given in the appendix, which has fewer skip connection operations than DARTS.
5.3 Variants and Speedup of ZARTS
[Table]
The results of three variants of
ZARTS, including the best and mean performance among four independent searches, are illustrated in Table 5.
We search for 50 epochs by each variant and train the discovered architectures for 600 epochs.
The search and evaluation settings are introduced in the appendix.
Specifically, GZAS outperforms GDAS by 0.3% with a similar search cost, only 0. GPU-day.
P-ZARTS outperforms PDARTS by 0.15% and even surpasses ZARTS due to the handcrafted criteria in P-DARTS: setting the number of skip connection operations as 2.
MergeZARTS achieves state-of-theart performance with low search cost (0.5 GPU-day).
We analyze that the weight merge technique among convolutions in MergeNAS reduces the redundant network weights ω , which alleviates the training difficulty for supernet and thus makes ω ∗ ( α ) more accurate after M iterations of gradient descent.  Conclusion and Future Work
DARTS has been a dominant paradigm in NAS, while its instability issue has received increasing attention [3, 38, 6]. In this work, we have empirically shown that the instability issue results from the first-order approximation for optimal network weights and the optimization gap in DARTS, which is also raised in the recent study [3]. To step out of such a bottleneck, this work proposes a robust search framework named ZARTS, allowing for higher-order approximation for ω ∗ ( α ) and supporting multiple combinations of zero-order optimization algorithms. Specifically, we adopt three representative methods for experiments and reveal the connection between ZARTS and DARTS.
Extensive experiments on various benchmarks show the effectiveness and robustness of ZARTS. To our best knowledge, this is the first work that applies zero-order optimization to one-shot NAS, providing a promising paradigm to solve the bi-level optimization problem for NAS.
Limitation & future work: i)
We adopt three existing zero-order solvers in Table 1, which also suggests new solvers may also be readily reused to improve ZARTS.
In contrast, this feature is not allowed in DARTS as there is little option for the gradient-descent solver.
ii) Our experiments focus on classification tasks, while ZARTS can also search network architectures for more complex tasks.
We derive three ZARTS variants from DARTS variants to speed up the search process.
GZAS and MergeZARTS can also reduce the GPU memory requirements, making it possible to search on more complex search spaces or tasks, e.g., object detection and segmentation. iii) ZARTS supports searching for the targets whose gradients are intractable or hard to obtain, e.g., FLOPs and latency, since ZARTS discards the differentiability assumption and leverages gradient-free algorithms into NAS.
Acknowledgement
This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] see Section ?? .
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Section 4.4.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codes are included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 2.2 and 2.3 in supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The standard deviation of repeated runs is reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] All the experiments are conducted on a single NVIDIA 2080Ti.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]