DARTS without a Validation Set: Optimizing the Marginal Likelihood
Abstract
The success of neural architecture search (NAS) has historically been limited by excessive compute requirements.
While modern weight-sharing NAS methods such as DARTS are able to ﬁnish the search in single-digit GPU days, extracting the ﬁnal best architecture from the shared weights is notoriously unreliable.
Training-Speed-Estimate (TSE), a recently developed generalization estimator with a Bayesian marginal likelihood interpretation, has previously been used in place of the validation loss for gradient-based optimization in DARTS.
This prevents the
DARTS skip connection collapse, which signiﬁcantly improves performance on
NASBench-201 and the original DARTS search space.
We extend those results by applying various DARTS diagnostics and show several unusual behaviors arising from not using a validation set.
Furthermore, our experiments yield concrete examples of the depth gap and topology selection in DARTS having a strongly negative impact on the search performance despite generally receiving limited attention in the literature compared to the operations selection.
Introduction
Neural architecture search (NAS) algorithms have been able to automatically ﬁnd network architectures that outperform the best human designs in test set performance on several benchmarks, making it an important subﬁeld of AutoML [10, 23].
However, earliest specialized NAS methods using evolutionary algorithms [22] or reinforcement learning [32] required thousands of GPU days to achieve good performance because each architecture was trained separately.
Weight-sharing algorithms such as ENAS [21] or DARTS [18], in which all the architectures share the same set of weights in a large supernetwork, have reduced the computational cost to single-digit GPU days while often delivering better performance.
However, identifying the single best architecture based on the shared weights is known to be unstable [2, 18, 30].
DARTS in particular suffers from overﬁtting to too many skip connections [28] and Dong et al.
[7] have shown that DARTS might even select architectures with all skip connections and no parameter-based operations.
Most weight-sharing NAS extracts the best architecture out of the supernetwork either by evaluating the validation accuracy of single architectures with weights inherited from the supernetwork [17, 13], or by explicitly optimizing the validation loss end-to-end in a differentiable fashion as in DARTS [18].
However, the performance of architectures using the shared weights tends to be signiﬁcantly different from their performances when trained separately [2, 30], which makes it difﬁcult to identify the standalone best architecture.
Ru et al.
[24] have shown that Training-Speed-Estimate (TSE), a recent generalization estimator based on Bayesian model selection [20] that does not require a validation set, can be used to provide high ﬁdelity yet computationally cheap early-stopping estimates of architecture test set performance in non-weight-sharing NAS.
They also showed promising results on TSE being applicable to various weight-sharing NAS algorithms.
Our work extends those results by providing additional benchmarks of TSE applied to DARTS (TSE-DARTS) and its variants on the NASBench-series.
The main novelty of our work lies in showing that using TSE both improves performance and fundamentally changes the behavior of DARTS, thus expanding the insights from previous work on explaining DARTS [28, 25].
Background
We ﬁrst introduce the basics behind DARTS and TSE, and then follow up by showing how to efﬁciently compute the gradients of TSE to use for the differentiable optimization in DARTS.
2.1 DARTS
DARTS [18] searches for a single architecture cell that is stacked repeatedly to form the ﬁnal network.
Such search space design is ubiquitous in modern NAS work [21, 22, 17].
A cell in the DARTS search space is simply a directed acyclic graph (DAG) with 7 nodes.
The DAG is fully connected, and every edge between a pair of nodes ( i, j ) represents one operation o from the 8 operations in the search space (such as skip connections, zero op, convolution or max pool).
The output x j of intermediate nodes in the DAG is equal to a sum of the operations corresponding to all the edges ( · , j ) with input x i :
[Equation]
During the search, DARTS relaxes the constraint that each edge ( i, j ) must represent only a single operation, and instead uses a softmax with learnable weights α to make the sum differentiable.
Therefore, the node output is a mixture of all the available operations at once deﬁned as
[Equation]
DARTS searches for normal and reduction cells.
Most of the network is composed of normal cells, whereas there are two reduction cells in total at 1 / 3 and 2 / 3 of the network depth.
The difference is that reduction cells have all operations with stride two.
The whole architecture encoding can thus be expressed as α =
( α normal , α reduce ) .
DARTS formulates the optimization of the weights w and architecture α as a bi-level optimization problem
[Equation]
L train and L val are losses on the train and validation sets, respectively.
This formulation is equivalent to the training loops in other problems such as meta-learning [11] or gradient-based hyperparameter optimization [1].
The optimal weights w ∗ are approximated by only a single step of SGD, which means that the architecture gradients actually descend L val ( w − η ∇ w L train ( w, α )) .
At the end of training, the supernetwork is discretized into a single architecture by taking argmax over each edge to select the operation that has the highest edge within the softmax.
It is therefore implicitly assumed that the most useful operations will simultaneously have the highest weights.
In practice, selecting the architecture this way is known to result in too many skip connections [28, 5], which frequently results in poor performance of the ﬁnal architecture.
2.2 TSE
TSE is deﬁned as the sum over training losses during a model’s optimization.
Let D =
( x 1 , y 1 ) , ( x 2 , y 2 ) , .., ( x n , y n ) } be the training dataset, f θ t ( x i ( t ) ) represent a model’s output for the training sample
x i ( t ) in the t -th iteration with θ t being the model weights at time t , L be a loss function and T be the total training iterations.
We deﬁne TSE as:
[Equation]
Lyle et al.
[20] have shown that TSE provably corresponds to an evidence lower bound in Bayesian linear regression, making it an estimate of the marginal likelihood.
For non-linear models such as neural networks, TSE is a theoretically-inspired metric that was already shown to be useful as a predictor of generalization in NAS by Ru et al.
Similar optimization-based generalization estimators have also been noted to correlate well with generalization by Jiang et al.
It is possible to integrate TSE into DARTS by replacing descent of the validation loss gradient ∇ L val ( w ∗ ( α ) , α ) by descending the TSE gradient.
Note that optimizing TSE amounts to minimizing the sum over training losses during training.
Therefore, when using TSE-DARTS, no validation set is required, and we only use the training set to ﬁnd both the weights and the architecture.
We compute TSE over T = 100 SGD iterations and iteratively update the architecture as in normal
DARTS.
Algorithm 1 shows the TSE-DARTS training loop, which simply uses different gradients for updating the architecture compared to the original DARTS training loop [18].
We discuss how to compute the TSE gradient in Section 2.3.
Algorithm 1: TSE-DARTS
Create mixed operations o ( i,j ) parameterized by α ( i,j ) for each edge ( i, j )
Set T=100 steps of unrolling for computing TSE while not converged do
Approximate the optimal weights w ∗ with T steps of SGD by computing w T = w 0 − η (cid:80)
Tt =0 ∇ w L train ( f ( x t , w t , α ) , y t ) ; 2.
Update architecture α by descending TSE gradient via Eq.
Update the original weights with T steps of SGD w T = w 0 − η (cid:80) Tt =0 ∇ w L train ( f ( x t , w t , α ) , y t ) using the new architecture encoding;
end
2.3 Computing the TSE gradient
While optimizing TSE instead of the validation loss is conceptually simple, obtaining the TSE gradients is very computationally demanding in practice.
Naively computing TSE gradient requires differentiating through the whole training history to compute gradients with respect to the architecture α , which leads to O ( T ) memory requirements due to the workings of reverse-mode auto-differentiation [1].
This would make it impossible to use large-scale networks.
Precisely, the exact gradient for the ﬁnal training loss at time T after training with SGD is equal to (proof is included in Appendix A):
[Equation]
5) where we abbreviate L train ( f ( x T , w T , α ) , y T ) as L Ttrain .
The TSE gradient would be equivalent to summing such gradients over all the time steps 0 , 1 , .., T .
Instead, we follow Ru et al.
[24], who proposed to use a ﬁrst-order approximation for each training loss gradient that only includes the direct gradient ∂L Ttrain ∂α from Eq.
(12) at each time step.
The whole TSE gradient can then be approximated as
[Equation]
This form of the TSE gradient has several advantages.
Most importantly, the ∇ α L ttrain can be computed for free concurrently with the weights gradients.
As a result, computing the approximate
TSE gradient has the same time and memory costs as normal training via backprop.
An example implementation is shown in Appendix D.
Experiments
We ﬁrst show the performance of TSE-DARTS compared to baseline DARTS on NASBench-201 [7] and NASBench-1shot1 [27].
While those search spaces are small, they tabulate the ground truth test set performance of every architecture in the search space. The tabular nature of those benchmarks
[Table]
makes it especially viable to use those spaces for diagnosing the behavior of TSE-DARTS, which we do in Sections 3.3 and 3.4 as our main empirical contribution.
For the DARTS search space, we retrain our best architectures using the 600 epochs training protocol as originally used by DARTS [18].
Moreover, we use the DARTS search protocol, where we ﬁrst run the search four times, retrain all the resulting architectures once, and then retrain the best out of the four for three more seeds to compute its mean ground truth test set accuracy.
We also evaluate TSE versions of DrNAS [4] and
PDARTS [5].
Our experiments with TSE algorithms always reuse the default parameters from their corresponding public implementations.
3.1 NASBench-201 and NASBench-1shot
Searching on NASBench-201 [7] is a notorious example of DARTS overﬁtting to skip connections as the ﬁnal selected architecture often contains only skip connections with no parameter-based operations.
This makes DARTS have a consistently poor performance.
Table 1 shows that using TSE-DARTS is able to prevent the skip connection collapse and achieve 92 .
66% test set accuracy, which is competitive with other weight-sharing algorithms such as GDAS [8] or SPOS [17] in NASBench-201.
On the other hand, DARTS has a strong performance on all three search spaces from NASBench-1shot1 [27], from which we show results on search space 3, denoted as NB101-3.
Note that none of the search spaces have skip connections in the operations set, hence DARTS cannot overﬁt to them.
In comparison, TSE-DARTS fares poorly on this benchmark.
Section 3.3 shows that this is due to picking cells that are too deep rather than due to the operations choice, which is unusual given the consistent overﬁtting to skip connections in normal DARTS.
Appendix B includes results on additional NASBench-201 datasets and NASBench-1shot1 search spaces, which are mostly analogous to those already shown.
3.2 DARTS search space and the role of depth
Table 2 shows the results of TSE-DARTS on the original DARTS search space.
The best architecture found by TSE-DARTS achieves 2 .
73% test set accuracy, which is a marginal improvement over the 2 .
76% of second-order DARTS.
However, it is a signiﬁcant improvement compared to the 3 . {+
+}
% of ﬁrst-order DARTS, which is particularly notable since both TSE-DARTS and ﬁrst-order DARTS have the same compute and memory requirements.
Moreover, TSE-DARTS ﬁnds very different architectures compared to normal DARTS.
Even in the DARTS search space, there is no overﬁtting to skip connections for TSE-DARTS.
Instead, the discovered architectures tend to be very heavy on separable convolutions, often with no parameter-free operations at all.
The best architecture found by
TSE-DARTS is visualized in Appendix C.
We hypothesize that skip connections are never chosen because they are unnecessary in the DARTS search setup, which uses a shallow search supernetwork.
In contrast, note that skip connections are known to be more beneﬁcial for deeper architectures [14, 16].
The search uses only 8 layers in the supernetwork even though the evaluation with retraining from scratch is done using 20 layers networks.
This inequity between the search and evaluation networks is often referred to as the depth gap.
To test whether TSE-DARTS would pick skip connections with a deeper search supernetwork, we also ran the search using 20 layers (denoted as TSE-DARTS-20) instead of the original 8 layers.
[Table]
TSE outperforms the
Figure 1c compares the number of skip connections for TSE-DARTS and TSE-DARTS-20.
The increased depth clearly favors picking a moderate amount of skip connections without overﬁtting to them as in original DARTS, whereas the shallow TSE-DARTS has no skip connections whatsoever.
The best architecture of TSE-DARTS-20 has standalone test accuracy of 2 .
68% , further improving on the 2 .
73% of TSE-DARTS.
We also benchmarked ﬁrst-order DARTS-20, which reached 2 . 89% ﬁnal test set accuracy, improving on the 3 .
% of original ﬁrst-order DARTS.
To better assess the effects of depth gap, we also trained the standalone architectures from scratch using 8 layers for the evaluation.
This again completely eliminates the depth gap except that now both the search and evaluation networks are shallow.
Table 2 shows the results with 8 layers evaluation using the same architectures discovered by each search algorithm as the corresponding 20 layers evaluation.
In this setup, the TSE variants consistently improve on their baseline versions.
This is true particularly for TSE-DrNAS, which has a weak performance of 2 . 90% in the originallayers evaluation setup, but top tier results in 8 layers evaluation.
Because some of the TSE variants signiﬁcantly improve in the 8 layers evaluation but are close or worse than the respective baselines using 20 layers evaluation, it suggests that the architectural optimum with 8 layers networks can be quite different from the 20 layers optimum.
Hence, even if the search algorithm itself is improved, it might still lead to worse results in the DARTS 20 layers evaluation if the search only uses 8 layers, since the performance gains might not transfer across different depths due to the depth gap.
Related work including PDARTS [5] and DrNAS [4] tries to tackle the depth gap by progressively deepening the search network while removing some operations from the operations set during multiple phases of the search.
We also tried to do this and tested TSE-PDARTS, but we found the resulting architectures to be very poor with only 3 .
19% accuracy.
Furthermore, the ﬁnal architectures never had any skip connections because the skip connections already got removed from the search space while the network was still shallow.
This suggests that even though the progressive deepening might be a key factor in PDARTS’s impressive 2 .
50% performance, the improvement might not necessarily be coming from reducing the depth gap since the architectures found by TSE-DARTSand TSE-PDARTS are very different.
It thus appears that searching without the depth gap can be more effective compared to progressive deepening.
DARTS keeps the search supernetwork small primarily because of excessive memory requirements due to the continuous relaxation.
However, the full-depth search is computationally feasible when using techniques such as gradient accumulation, which allows to trade-off extra compute time for memory, or partial channel connections as in PC-DARTS [26].
The TSE-DARTS20 search only takes around 1 day to complete, which is the same as second-order DARTS, and around 23GB of VRAM when using the original 64 batch size.
Note that optimizing TSE has the added advantage of making DARTS less memory-intensive due to not using any second-order gradients, hence making it easier to use higher depth supernetworks. At the same time, searching with full depth
[Figure]
supernetworks might even be computationally cheaper overall if it were to improve on the search instability in DARTS, reducing the number of seeds that need to be ran to obtain good results.
3.3 Bias towards shallow architectures
In Section 3.2, we investigated the effect of the number of layers in the supernetwork considered as a ﬁxed hyperparameter.
Now, we investigate the depth of the architectural cells themselves which is determined by the search.
Shu et al.
[25] have shown that DARTS tends to be biased towards shallow cells.
This is explained by shallower architectures having smoother loss landscapes, which makes them easier to train.
DARTS favors them since the gradient-based search is naturally greedy towards architectures which train fast one step ahead.
Following Shu et al.
[25], we measure depth as the length of the longest path from input to output within the cell DAG.
Figure 1 shows that TSE-DARTS tends to select signiﬁcantly deeper cells than normal DARTS.
This is inconsistent with the intuition that DARTS is biased towards shallow architectures because they train faster, since optimizing TSE means that the bias towards architectures which achieve low training loss faster should be even more prominent.
We propose an explanation for this paradox in Section 3.4.
Additionally, we found the depth of TSE-DARTS-20 architectures to be inbetween DARTS and TSE-DARTS, whereas DARTS-20 again found the shallowest cells, same as normal DARTS.
It appears that TSE-DARTS with both 8 and 20 layers maintains a roughly constant total depth of the network since searching with 8 cells ﬁnds deep individual cells whereas searching with 20 cells ﬁnds shallower cells.
Next, we consider the selection of skip connections, which have been shown to make the loss landscapes of deep neural networks smoother as deep networks tend to otherwise have very nonsmooth loss regions [16].
Therefore, TSE-DARTS-20 picking more skip connections can also be interpreted in the context of smoothing the loss landscapes, which only becomes necessary when the search supernetwork has 20 layers.
Skip connections might have limited utility with only 8 layers because the network is still shallow, and hence are not selected by TSE-DARTS.
Our results would thus suggest that the bias towards shallow architectures observed by Shu et al.
[25] is more strongly related to the higher loss landscape smoothness of such architectures rather than skip connections making the architecture train faster, which is often believed to be the reason for DARTS’s bias towards skip connections [31, 5].
Otherwise, TSE-DARTS would have even stronger overﬁtting to skip connections than normal DARTS, which is the opposite of what happens empirically.
We further show that the poor results of TSE-DARTS on NASBench-1shot1 are caused by ﬁnding architectures with excessive depth.
Both DARTS and TSE-DARTS ﬁnd architectures which are predominantly if not all 3x3 convolutions, but the performance of DARTS is high because it ﬁnds the shallowest cells possible.
Note that NASBench-1shot1 search spaces contain no skip connections, which makes it impossible for DARTS to overﬁt to them.
Figure 2a shows the performance of the architectures selected by DARTS and TSE-DARTS on NB101-3 alongside the cell depth, where lower depth clearly correlates with better performance.
[Figure]
Overall, DARTS consistently ﬁnds shallow architectures even without any skip connections in the search space while TSE-DARTS ﬁnds deep cells.
Whether this leads to good or bad performance then depends on the search space.
The results here also highlight that selecting the cell topology can be a serious issue in differentiable NAS as the poor performance of TSE-DARTS on NASBench-1shot1 is almost entirely caused by the suboptimal cell topology.
However, most work on improving DARTS focuses on the skip connection collapse rather than improving the topology selection.
3.4 Eigenvalues of the architecture Hessian
Zela et al.
[28] argue that the skip connection overﬁtting in DARTS is related to the growing dominant eigenvalues of the architecture Hessian.
We additionally conﬁrm the trend of rising eigenvalues for DARTS on the NASBench search spaces [6, 27], which have been released after the original work by
Zela et al.
On the other hand, we also show that the behavior of TSE-DARTS eigenvalues is much more varied.
We ﬁrst reproduce experiments on the reduced S2 search space proposed by Zela et al.
[28], which contains only skip connections and 3 × 3 separable convolutions for two operations total.
Figureshows that TSE-DARTS picks zero skip connections even on this space whereas DARTS severely overﬁts to skip connections.
DARTS-20 overﬁts to skip connections even more while TSE-DARTSchooses a small but nonzero number of skip connections, which attests to skip connections being more prominent when the search supernetwork is deep.
The architecture eigenvalues increase for DARTS but stay at low levels for TSE-DARTS.
Figure 2 shows the same experiment on NB101-3, where we see that the eigenvalues for TSE-DARTS actually go down but so does its performance, whereas
DARTS has rising eigenvalues and rising performance.
On NASBench-201, the eigenvalues increase for both DARTS and TSE-DARTS.
However, the performance keeps improving for TSE-DARTS but it gets worse over time for DARTS.
We visualize the NASBench-201 results in Appendix E.
Overall, it appears that the eigenvalue phenomenon observed by Zela et al.
[28] also has signiﬁcant differences for DARTS and TSE-DARTS.
While DARTS always has rising eigenvalues and simultaneously overﬁts to skip connections, neither appears to be the case for TSE-DARTS.
A potential explanation might be that TSE-DARTS is more biased towards architectures with smooth loss landscapes as proposed by Shu et al.
[25], which by deﬁnition results in smaller architecture Hessian eigenvalues by the virtue of the search algorithm ﬁnding architectures without excessively sharp loss regions.
This would be in disagreement with the observation in Section 3.3 showing that TSE-DARTS discovers deeper architectures, which should instead have less smooth loss landscapes.
Nonetheless, we see that in practice, DARTS has higher eigenvalues despite ﬁnding shallower architectures.
A plausible explanation might be that the architecture depth is confounded by the architecture selection in DARTS being suboptimal as we had explicitly shown in Section 3.3.
The TSE-DARTS bias towards deep architectures might then be caused by the architecture selection procedure rather than [Figure] the optimization itself.
An explicit treatment of cell topology could therefore allow to better reconcile our results with those of Shu et al.
[25] and Zela et al.
That said, note that analyzing the supernetwork eigenvalues already has the advantage of being independent of architecture selection, and TSE-DARTS having smaller architecture eigenvalues can also be understood as supporting evidence for TSE-DARTS to be ﬁnding shallow architectures.
Conclusion
In this work, we analyzed TSE-DARTS, a variant of DARTS where the architecture is updated to minimize TSE rather than validation loss.
Our results show that TSE-DARTS tends to have a stronger performance than DARTS, and we demonstrated that TSE-DARTS also has several interesting qualitative behaviors.
In particular, it is free of the skip connection collapse that is known to be a major issue in DARTS.
We applied several DARTS diagnostics previously proposed in the literature and shown that TSE-DARTS has fundamentally different biases than the original DARTS.
TSEDARTS appears to be ﬁnding signiﬁcantly deeper architectures than DARTS [25], and its architecture Hessian eigenvalues appear to have different trajectories depending on the search space, whereas the DARTS eigenvalues rise consistently as observed by Zela et al.
Our experiments also yielded a concrete example of the depth gap that cannot be ﬁxed with progressive deepening [5, 4], where TSE-DARTS with an increased number of layers in the search supernetwork starts selecting skip connections despite not picking any otherwise.
We have also shown that the DARTS architecture selection via argmax on the architecture coefﬁcients can lead to problems with the cell topology rather than just the selected operations as is commonly assumed.
Ultimately, while TSE-DARTS is able to outperform the baseline DARTS, it appears to have its own distinct set of problems, and it cannot be trivially applied to other state-of-the-art DARTS variants to further advance their performance.
However, the follow-up work on DARTS might be effectively overﬁtted to ﬁxing DARTS’s problems, which are no longer present in TSE-DARTS.
It is possible that if the same amount of attention was devoted to ﬁxing the problems of TSE-DARTS, it could achieve much better performance.
Such extensions are an interesting direction for future work.
Acknowledgments and Disclosure of Funding
Binxin Ru was supported by the Clarendon Scholarship.
Clare Lyle was supported by the Open
Philantropy Foundation AI Fellows Program.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See the Conclusions section.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work has no direct societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix A, which proves our only theoretical result (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A, which proves our only theoretical result.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Most hyperparameters are defaults from the corresponding benchmarks.
We speciﬁed the parameters speciﬁc to our method.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All tabular results include error bars and so do the ﬁgures where clarity allows.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We speciﬁed the search and evaluation costs of our methods as well as the baselines.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all the NASBench and other relevant baseline code implementations of methods that we adapt.
We used no other relevant data or models apart from well-known public datasets such as CIFAR10.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Derivation of the exact TSE gradient
The derivation here is analogous to those in other work involving differentiation through optimization [19, 12].
Assume we optimize the weights by SGD, then the weights w T after T time steps from initial w 0 can be written as
[Equation]
With this in mind, we will now compute the ﬁnal training loss gradient after T iterations ∇ α L train ( f ( x T , w T , α ) , y T ) as the ﬁrst step to computing the TSE gradients, where we abbreviate L train ( f ( x T , w T , α ) , y T ) as L Ttrain :
[Equation]
The ∂L Ttrain ∂α is sometimes referred to as the direct gradient, and the ∂L Ttrain ∂w ∂w T ∂α is also known as the indirect gradient or hypergradient.
Now given the expression for w T given in Eq.
(7), we calculate
[Equation]
The ∂w T ∂α at the end of Eq.
(9) gives a recurrent relation that can be further expanded to give ∂w T
[Equation]
If we unroll the whole history until w 0 , for which it holds ∂w 0 ∂α =
[Equation]
we get a string of summands that can be summarized as
[Equation]
In total [Equation] we have that Eq.
(8) is equivalent to
[Equation]
[Equation]
This is the exact unrolled differentiation hypergradient using the last training loss.
The TSE gradient computed over T steps of training is simply equal to a sum of the individual training loss gradients: [Equation]
(cid:88)
[Equation]
([ (cid:89)
[Equation]
I
[Equation]
train ∂w∂w ]
[Equation]
train ∂w∂α )))
B Additional results on NASBench-201 and NASBench-1shot
We also show the results of TSE-DARTS on all the three datasets in NASBench-201 [6] and all the three search spaces in NASBench-1shot1 [27].
For NASBench-201, we additionaly show that training for 150 rather than 50 epochs still retains strong performance and the robustness to skip connection collapse holds even over longer training.
CIFAR
CIFAR
ImageNet16REA [22] 94.02 (0.31) 72.23 (0.95) 45.77 (0.80)
Random Search [3] 93.90 (0.26) 71.86 (0.89) 45.28 (0.97)
SPOS [17] 91.05 (0.66) 68.26 (0.96) 40.69 (0.36)
GDAS [8] 93.23 (0.58) 68.17 (2.50) 39.40 (0.00)
ENAS [21] 93.76 (0.00) 70.67 (0.62) 41.44 (0.00) DrNAS [4] 93.76 (0.00) 68.82 (2.06) 41.44 (0.00) DARTS (ﬁrst-order) [18] 59.84 (7.84) 61.26 (4.43) 37.88 (2.91) DARTS (second-order) [18] 65.38 (7.84) 60.49 (4.95) 36.79 (7.59)
TSE-DARTS 92.66 (0.00) 68.13 (2.18) 36.60 (0.00) TSE-DARTS (150 epochs) 89.89 (2.52)
[Table] [Table]
C Best discovered architectures by TSE-DARTS
Figure 4 shows the ﬁnal best architectures found by TSE-DARTS and TSE-DARTS-20.
Notably,
TSE-DARTS ﬁnds signiﬁcantly deeper architectures for both the normal and reduction cells than
TSE-DARTS-20.
[Figure]
D TSE-DARTS implementation
We showcase an example PyTorch implementation of Algorithm 1 (TSE-DARTS) to demonstrate that TSE is very easy to implement on top of standard training loops.
The most important part is on lines 12-17, which show that it is trivial to accumulate the approximate TSE gradient by not zeroing out the architecture gradients after each weight update.
All implementations of TSE-DARTS variants proceed analogously to this example.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
[Figure]