DARTS for Inverse Problems: a Study on Stability
Abstract
Differentiable architecture search (DARTS) is a widely researched tool for neural architecture search, due to its promising results for image classiﬁcation.
The main beneﬁt of DARTS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efﬁcient architecture search.
In this work, we investigate
DARTS in a systematic case study of inverse problems, which allows us to analyze these potential beneﬁts in a controlled manner.
Although we demonstrate that the success of DARTS can be extended from classiﬁcation to reconstruction, our experiments yield a fundamental difﬁculty in the evaluation of DARTS-based methods: The results show a large variance in all test cases and the weight-sharing performance of the architecture found during training does not always reﬂect its ﬁnal performance.
We conclude the necessity to 1) report the results of any DARTSbased methods from several runs along with its underlying performance statistics and 2) show the correlation between training and ﬁnal architecture performance.
Introduction
Recent progress in computer vision and related ﬁelds has illustrated the importance of suitable neural architecture designs and training schemes [4, 5, 1].
Neural architecture search (NAS) is the task of optimizing the architecture of a neural network automatically without resorting to human selection, scaling to larger search spaces with the promise of novel, improved architectures.
To avoid the computational costs of training candidate architectures for black-box optimization models [16, 17, 6, 13, 11, 10], differentiable architecture search (DARTS) [9] proposes a continuous relaxation of the search problem, i e. all candidate architectures within a given search space are jointly optimized using shared network parameters while the network also learns to weigh these operations.
The ﬁnal architecture can then be deduced by selecting the highest weighted operations.
This is appealing as practically good architectures are proposed within a single optimization run.
However, previous works such as [14] also indicate that the proposed results are often sub-optimal, especially when the search space is not well chosen (e.g. contains sub-optimal operations).
Speciﬁcally, since network weights are randomly initialized, promising operations can have poor initial weights such that the architecture optimization tends to entirely discard them.
As a result, the practical relevance of DARTS-proposed architectures depends heavily on network initialization.
Here, we apply DARTS to inverse problems, which have hardly been addressed in NAS, and allow us the choice of parameters and search space like a practitioner would do.
Our main focus is the analysis of DARTS w.r.t.
the stability of results.
We investigate one-dimensional inverse problems which allow to conduct several evaluation runs to analyze the robustness of DARTS.
We show that DARTS can automatically ﬁnd well performing architectures in our setup, if the search space is well preconditioned.
Yet, importantly, we ﬁnd that the estimated network performance using jointly optimized, shared weights is often not well correlated
[Figure]
N
[Figure]
the reconstruction ability of the
DARTS
We introduce DARTS [9] in a more detailed way in the following, as it is the technical basis of our analysis.
While the originally proposed method optimizes so-called cells , which are stacked in order to deﬁne the overall neural network architecture, and deﬁnes each cell in the form of a directed acyclic graph (DAG), we conduct our study of the behavior of DARTS on special, sequential and easy-to-interpret meta-architectures to be described below.
Our sequential architecture merely consists of N nodes x ( i ) , where x (0) represents the input data and the result x ( j +1) of any layer is computed by applying some operation o ( j ) to the predecessor node x ( j ) , i.e.,
[Equation]
where θ ( j ) are the (learnable) parameters of operation o ( j ) .
To determine which operation o ( j ) is most suitable to be applied to the feature x ( j ) , one deﬁnes a set of candidate operations o t ∈ O , t ∈ { 1 , . .
, | O | =: T } and searches over the continuous relaxation of Eq.
(1) for
[Equation]
where α = ( α ( j ) o t ) are architecture parameters that determine the selection of exactly one candidate operation in the limit of β becoming binary.
Instead of looking for binary parameters directly, the optimization is relaxed to the soft-max of continuous parameters α .
DARTS formulates this search as a bi-level optimization problem in which both, the network parameters θ = { θ ( j ) } Tj =1 and the architecture parameters α , are jointly optimized on the training and validation set, respectively, via
[Equation]
where L val and L train denote suitable loss functions for the validation and training data.
The optimization is done by approximating (4) by one (or zero) iterations of gradient descent.
After optimization, the discrete architecture is obtained by choosing ˆ o ( j ) =
argmax
o t α ( j ) o t for each node.
Subsequently, the ﬁnal network given by the architecture eq.
(1), but using ˆ o ( j ) instead of o ( j ) , is retrained from scratch.
Thus, the fundamental assumption of DARTS is that the performance of the ﬁnal network architecture on the validation set (the architecture validation ) is highly correlated with the performance of the relaxed DARTS approach obtained in (3) (the one-shot validation ).
Experimental Setup
We investigate the meta-architecture shown in ﬁg. 1, which should be speciﬁcally well-suited for examples of signal recovery from known data formation processes such as blurring and subsampling with noise.
We choose one operation out of several sequentially before adding the output to a residual branch.
Image recovery networks such as DnCNN [15] are contained in this meta-architecture.
We search for the optimal architecture that can be deﬁned as a sequence of operations selected from the set O l which contains four operation types.
Two of the four operation types in O l are [Table] by design potentially beneﬁcial operations: The ﬁrst is a plain 2-layer convolutional network.
The second is motivated from rolled-out architectures (see e.g. [3, 12, 7]) and tries to embed modelbased knowledge about the recovery problems into the networks architecture.
As all considered problems can be phrased as linear inverse problems in which a quantity x ought to be recovered from data y
=
Ax + noise for a linear operator A , gradient descent on a quadratic data ﬁdelity term yields updates x k +
= x k − τA T (
Ax k − y ) , which we augment by an additive 2-layer convolutional network to obtain a learnable gradient descent layer .
To complement these beneﬁcial layers we include two harmful operation types in O l , a layer that merely adds white Gaussian noise and a roll layer, which rolls the inputs in all dimensions, both of which the optimization should discard.
We provide additional details for these operations in the supp.
material.
In total we set
O l = { learnable gradient descent , 2-layer-CNN , roll , noise } , and search for 10 successive layers.
This allows an evaluation of the effectiveness of DARTS in two different cases: Training on all operations versus training only on beneﬁcial operations.
A good differentiable architecture search algorithm should reliably ﬁnd the optimal operations, even when presented with sub-optimal choices.
For a fast test, we generate one-dimensional data sampling cosine waves of varying magnitude, amplitude and offset, and search for models to recover these samples from distorted measurements.
We consider two distortion processes with varying difﬁculty: First, Gaussian noise and blurring and second, in addition to these, a subsampling by a factor of 4.
For more details, we refer to the appendix.
We evaluate the performance of these models on a validation set of previously unseen examples by computing their average peak signal-to-noise ratio (PSNR).
Evaluating DARTS
Plainly, DARTS does work for inverse problems as shown by results in table 1 in the Max.
column, which shows the validation PSNR of the best architecture found over 100 trials: DARTS is able to propose good architectures in two different applications with a noticeably higher PSNR than either a random selection of operations or the choice of just one of the good operations.
When comparing
DARTS with all operations versus DARTS with only beneﬁcial operations, then the best architectures in both settings are close.
In particular, the best architecture DARTS found when searching over all operations did not contain noise or roll , which implies DARTS found one of the top 0 .
1% out of the 4 10 many architectures in 100 trials.
Evaluating Robustness: While DARTS is able to propose good architectures according to table 1, the assertion that it works well in this case study is predicated on the evaluation criterion of maximal
PSNR over all trials.
If we turn towards the second column in table 1 and evaluate mean PSNR scores the picture changes considerably.
While DARTS with only good operations is stable, we see that the default of DARTS with all operations suffers considerably, dropping 7 PSNR values for the blur example and 3 PSNR for downsampling.
Especially of note here is that DARTS performs on average worse than the maximum over 100 random architectures.
This prompts a conundrum.
On the one hand, best benchmark results are achieved by running DARTS for as many trials as possible and returning the best found architecture, but on the other hand this setup treats the DARTS algorithm, originally a replacement of 0-th order NAS methods, itself as a component of an overarching NAS algorithm that sometimes proposes good architectures.
While
[Figure] Validation [Figure]
V a li d a t i o n P S N
R ( a r c h . )
OperationsGood Operations, c=0.24 All Operations, c=0.
Validation PSNR (search)
V a li d a t i o n P S N
R ( a r c h . )
this strategy of repeatedly using DARTS to propose trials for NAS is clearly a feasible architecture search tactic, it does fall short of the original goal, which is to ﬁnd a good architecture within a single optimization of the weight-shared DARTS problem.
Correlation of Architecture and DARTS Performance : Figure 2 takes a closer look at the trials considered in table 1, and plots for all trials separately the direct validation performance of the one-shot architecture (x-axis) versus the "true" architecture performance after retraining (y-axis).
We also plot a regression line over all trials and report the correlation of all trials in the legend, showing the linear ﬁt has limited expressiveness.
As discussed above, the correlation of these quantities is a fundamental assumption of DARTS.
Yet these plots show that DARTS’ behavior is highly problemdependent: The downsampling dataset (right) shows that, although the mean value of DARTS can be non-optimal, search performance and architecture performance are weakly correlated, even if the best architecture only has average search performance.
The closely related blur dataset (left) shows a different behavior with 3 observable "failure" cases: Although the trials are nominally well correlated, we ﬁnd a cluster of trials for which both search and architecture validation PSNR are low, a cluster where search performance is good, but architecture performance is low, and a cluster where architecture performance is high (close to DARTS with only good operations) but seemingly uncorrelated to search performance.
In summary, the three problems we observe are that either
DARTS fails directly, or that DARTS works but does not predict a useful architecture, or that DARTS does predict a useful architecture, but is unrelated to its search performance.
Discussion on DARTS: From our result, we ﬁnd two schools of thought for DARTS:
For maximal performance, DARTS can be used as as a component in a larger search that proposes trial architectures.
For average performance, and immediate performance with a single DARTS run, we should maximize its correlation with architecture performance.
Positioning future DARTS papers in this respect, and understanding between the relation between maximal performance and maximal correlation, are important task in future research.
Discussion on optimal network architectures: From the applied inverse problems perspective, it is interesting to see that the optimal architectures found for both problems, deblurring and super resolution, are hybrid versions that mix learnable grad- and net-operations.
The optimal mixed architectures have a remarkable advantage of more than 1dB over the best plain (only net or only grad) architecture, possibly suggesting that the best way to approach inverse problems with deep learning are neither plain (convolutional) networks nor pure unrolling schemes.
Conclusions
In this paper we analyzed DARTS in a systematic study on one-dimensional inverse reconstruction problems, and made the following ﬁndings: While it is possible to ﬁnd very well performing architectures using DARTS, multiple runs of the same setting yield a high variance.
Unfortunately, judging the success of any DARTS-based model right after the one-shot training is difﬁcult, since a strong correlation to the actual architecture performance is missing.
Therefore, we advertise to look at a full statistical evaluation of DARTS performances over multiple trials, in all applications where this is feasible, and to show a reasonable correlation between the search and ﬁnal architecture performances for any method that reports improved results based on a more faithful minimization of the one-shot DARTS objective.
A Operations
While the precise type of algorithm is typically dictated by (smoothness) properties of the regularization, a partially parameterized network-based approach has a lot of freedom to choose from template layers based on the inverse problem y
=
Ax + noise
[Equation]
where D is a data formation term arising from the distribution of noise present, i.e. D ( v, f ) = 12 || v − f || 2 for Gaussian noise.
This optimization objective yields templates such as a gradient descent layer (operation set O u ):
[Equation]
for input u k and output u k +1 of a new layer.
For suitable chosen τ , the application of this layer is guaranteed to reduce the objective (5).
The gradient layer can be turned into a learnable operation by introducing a learnable mapping n ( u, θ ) after the gradient step,
[Equation]
as a learnable gradient descent layer in our operation set O l .
We also consider a fully-learned neural network layer in our operation set O l , that learns an appropriate mapping n ( u, θ ) without knowledge of the operator A :
[Equation]
The unlearnable counterpart to the neural network layer in O u is a skip layer .
For both layers, the learnable mapping n ( u, θ ) is parametrized by a small convolutional network, consisting of a convolution layer, followed by batch normalization, ReLU and a second convolution layer.
These four layers are by design beneﬁcial operations.
In contrast to these beneﬁcial layers we also include two negative operations to each operation set; a gradient layer with white Gaussian noise noise layer and a roll layer , which rolls the inputs.
B Hyperparameters
In table 2 we show the hyperparameters used for our experiments in the main paper.
C Data Generation
We generate synthethic one-dimensional cosine data from N = 50 equally spaced points ω i on the interval [ − π 2 , π 2 ] with the model
[Equation]
for a random frequency f drawn uniformly from the interval [0 , 2 π ] and offsets O x and O y drawn from a normal Gaussian distribution.
Such random drawn waves comprises our ground truth training data.
We then generate measured data via the linear operation A and addition of noise,
[Equation]
These pairs ( y, x ) represent our training data.
We sample new examples on-the-ﬂy during both training and validation, so that no confounding effects of dataset size exist.
All validation and training loss evaluations are each based on 2432 randomly drawn samples.
For all experiments we chose σ n = 0 .
For the blur experiments, the linear operator A is a Gaussian blur with kernel size 7 and σ b = 0 .
For the downsampling experiments, this Gaussian blur is follows by a subsampling by a factor 4.
D Computational Setup
All experiments in the main body were run on a single Nvidia GTX 2080ti graphics card of which two were utilized.
[Table] [Table]
E Hyperparameter Optimization
As we have seen in section 4 the DARTS search entails a discrepancy between being a feasible architecture search tactic and indeed ﬁnding a good architecture within a single optimization of the
DARTS problem.
This prompts us to also take a closer look at hyperparameter tuning.
Our one-dimensional case study allows us to optimize DARTS training hyperparameters with more granularity than it would be possible for image classiﬁcation tasks, and we stress that we consider this mainly as an analysis tool - given that the neural architecture search itself is a hyperparameter optimization on which we stack another.
To improve hyperparameters we apply BOHB [2], a Bayesian optimization method with hyperbanding [8] and run BOHB for 128 hyperband iterations, which is an affordable budget in this one-dimensional data setting.
This hyperparameter tuning was conducted on a single Nvidia GTX
Ti graphics card.
The BOHB optimized hyperparameters are listed in table 4; the search range for the BOHB search is given in the second column.
It is important to note, that BOHB is not an exhaustive search and thus there are no guarantees for success within our budget or even in general.
The usage of BOHB as such covers the problem of hyperparameter optimization partially, but in general there is no simple ﬁx of DARTS via hyperparameter search.
When applying BOHB to target one-shot validation performance, the resulting plot over all trials in ﬁg. 3 shows a rather surprising outcome: In the case of downsampling, the correlation between one-shot validation and architecture validation increases notably and the average performance of
DARTS improves in the case of blur, but the best architecture PSNR over 100 trials decreases in both cases.
Overall, the apparent stabilization via optimization of the search loss removes not only negative, but also positive outliers.
Conversely, we can use BOHB to search for hyperparameters that maximize the ﬁnal architecture performance.
This is twice as expensive (on top of the already expensive hyperparameter) search, due to the need for retraining the architecture, but feasible in our synthetic case.
Figure 4 again shows blur and downsampling with hyperparameters optimized for the ﬁnal architecture performance for blur data.
These hyperparameters successfully increase the max. architecture performance, but the overall distribution is similar to the three failure cases discussed for ﬁg. 2 for blur.
Overall, hyperparameters optimized with BOHB on the one-shot validation have to be considered with caution: When regarding the search of architectures over all operations for the case of blur in table 5, the dedicated BOHB-Blur hyperparameters are not as stable as the BOHB-DS hyperparameters.
When changing the domain to downsampling, the exact opposite holds: BOHB-Blur hyperparameters improve over BOHB-DS hyperparameters in every statistic.
Note that this could be due to both, the missing correlation between one-shot and architecture validation as well as the missing guarantee of any Bayesian search to ﬁnd the optimal hyperparameters.
In addition, table 5 even demonstrates that using the manual hyperparameters H1 leads to a better average performance compared with
BOHB-DS and BOHB-Blur
in most cases.
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
In summary
[Figure]
even hyperparameter tuning for different data formations and objectives does not improve the stability of the DARTS search itself.
Also in this experiment we still ﬁnd that DARTS either fails directly
[Figure]
or that it works but does not predict a useful architecture
[Figure]
or that is does predict a useful architecture
[Figure]
but it is unrelated to its search performance
[Figure] [Figure]
V a li d a t i o n P S N
R ( a r c h . )
[Figure]