Saliency-Aware Neural Architecture Search
Abstract
Recently a wide variety of NAS methods have been proposed and achieved considerable success in automatically identifying highly-performing architectures of neural networks for the sake of reducing the reliance on human experts.
Existing
NAS methods ignore the fact that different input data elements (e.g., image pixels) have different importance (or saliency) in determining the prediction outcome.
They treat all data elements as being equally important and therefore lead to suboptimal performance.
To address this problem, we propose an end-to-end framework which dynamically detects saliency of input data, reweights data using saliency maps, and searches architectures on saliency-reweighted data.
Our framework is based on four-level optimization, which performs four learning stages in a unified way.
At the first stage, a model is trained with its architecture tentatively fixed.
At the second stage, saliency maps are generated using the trained model.
At the third stage, the model is retrained on saliency-reweighted data.
At the fourth stage, the model is evaluated on a validation set and the architecture is updated by minimizing the validation loss.
Experiments on several datasets demonstrate the effectiveness of our framework.
Introduction
Neural architecture search (NAS) [78, 41, 48], which aims to automatically identify highly performant neural architectures, has received much attention recently.
Existing NAS methods treat all elements in an input data example (such as pixels in an image, tokens in a sentence, etc.) as being equally important, without considering the different saliency of individual elements, which leads to lessoptimal performance.
In machine learning applications, an input data example typically consists of many data elements.
For instance, an image example consists of a grid of pixel elements and a sentence example consists of a sequence of token elements.
When used to make a prediction, different data elements have different importance (or saliency).
For example, when predicting which object category an image belongs to, pixels in foreground object regions are more important than those in background regions.
Such saliency information is not leveraged by existing NAS methods.
In this paper, we aim to bridge this gap, by proposing a saliency-aware NAS method which automatically identifies the saliency of data elements and leverages that to search for better architectures.
Our framework is formulated as a four-level optimization problem.
At the first level, the model tentatively fixes its architecture and trains its first set of network weights.
At the second level, the trained model generates saliency maps using an adversarial attack based method [18].
At the third level, input data is reweighted using saliency maps and the second set of model weights are trained using reweighted data.
At the fourth level, the two sets of trained model weights are evaluated on a human-provided validation set and the architecture is optimized by minimizing the validation losses.
The major contributions of this paper are as follows:
• We propose a framework which performs end-to-end detection of saliency and leverage saliencyreweighted data to improve neural architecture search.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
• We formulate our method as a multi-level optimization problem which performs model training on unweighted data, saliency map generation, model retraining on saliency-reweighted data, and architecture update in a unified manner.
Related works
Neural architecture search (NAS).
Early NAS approaches [78, 46, 79] are mostly based on reinforcement learning (RL) which use a policy network to generate architectures and evaluate these architectures on validation sets.
The validation loss is used as a reward to optimize the policy network and train it to produce high-quality architectures.
Differentiable search methods [5, 41, 67] parameterize architectures as differentiable functions and perform search using efficient gradientbased methods.
In these methods, the search space of architectures is composed of a large set of building blocks where the output of each block is multiplied with a smooth variable indicating how important this block is.
Under such a formulation, search becomes solving a mathematical optimization problem defined on the importance variables where the objective is to find an optimal set of variables that yield the lowest validation loss.
This optimization problem can be solved efficiently using gradient-based methods.
Differentiable NAS research is initiated by DARTS [41] and further improved by subsequent works such as P-DARTS [11], PC-DARTS [68], etc.
P-DARTS [11] grows the depth of architectures progressively in the search process.
PC-DARTS [68] samples sub-architectures from a super network to reduce redundancy during search.
Our proposed framework is orthogonal to existing NAS methods and can be used in combination with any differentiable
NAS method to further improve these methods.
Besides RL-based approaches and differentiable
NAS, another paradigm of NAS methods [40, 48] are based on the evolutionary algorithm where architectures are formulated as individuals in a population.
High-quality architectures produce offspring to replace low-quality architectures, where the quality is measured using fitness scores.
Saliency detection.
Many methods [72, 77, 57, 45, 52, 51, 47] have been proposed for saliency detection, based on perturbing inputs [72, 77], propagating gradients [3, 56, 57], attention [37, 71, 45], model approximation [50, 2], etc.
Several works [52, 51, 47] show that leveraging saliency of input data can enhance model’s predictive power.
Rieger et al.
[51] leverage domain-specific rules or knowledge to provide “groundtruth” saliency.
Such rules/knowledge are difficult to obtain in many applications.
Pillai and Pirsiavash [47] encourage a prediction model to produce saliency maps that are consistent with those generated by GradCAM [54].
GradCAM is an unsupervised approach; without any supervision from humans, its generated saliency maps may not be reliable.
For example, it is shown in [58] that GradCAM cannot highlight adversarial image patches that cause wrong predictions.
In [52], saliency maps are either labeled by humans or auto-generated based on gradient magnitude with no human supervision, which suffers the same problems as [54, 51].
Different from existing works, our method generates saliency maps with weak supervision such as human-provided class labels.
Such weak supervision is much easier to obtain than human annotations of saliency maps and can yield more reliable saliency maps than using no supervision at all.
Bi-level optimization.
Many ML methods [19, 4, 20, 41, 55, 74] have been formulated as bi-level optimization (BLO) problems.
In these methods, network weights are learned by solving an inner optimization problem defined on a training set while meta parameters are learned by solving an outer optimization problem defined on a validation set.
BLO-based methods have been applied for neural architecture search [41], hyperparameter tuning [19], learning rate adaptation [4], data selection [55, 49, 65], meta learning [20], label correction [74], etc., where meta parameters are neural architectures, hyperparameters, importance weights of data examples, meta network weights, etc.
Many optimization algorithms [14, 21, 23, 32, 42, 69] have been developed for solving BLO problems where convergence analysis is provided.
Methods
In this section, we propose a four-level optimization based framework to perform saliency-aware neural architecture search.
For the ease of presentation, we assume the task is image classification.
In the experiments, we show that our method can be applied for other tasks as well.
[Figure]
3.1 A four-level optimization framework
In our framework (Figure 1), a model has a learnable architecture A and two sets of learnable network weights W 1 and W 2 .
W 2 is a tensor that has the same dimensions as W 1 .
The weight values in W 2 and
W 1 are different.
It consists of four stages performed end-to-end.
At the first stage, the model trains its network weights W 1 with the architecture A tentatively fixed.
At the second stage, the trained Wgenerates saliency maps for input images: a saliency score is calculated for each pixel.
At the third stage, images are reweighted using saliency maps and saliency-reweighted images are used to train model weights W 2 .
At the fourth stage, the trained W 2 is evaluated on a human-labeled validation set and the architecture A is updated by minimizing the validation loss.
Stage I.
At the first stage, the model trains its first set of network weights W 1 by minimizing a loss
L on training dataset D ( tr ) , with the architecture A tentatively fixed:
[Equation]
To define the training loss, we need to use both the architecture parameters A and network weights
W 1 .
However, A cannot be updated by minimizing the training loss.
Otherwise, a trivial solution of
A will be yielded: A can perfectly overfit the training data but will make incorrect predictions on unseen data examples.
W ∗ 1 ( A ) denotes that the optimal weights W ∗ 1 depends on A .
This is because
L ( W 1 , A , D ( tr ) ) is a function of A , and W ∗ 1 depends on L ( W 1 , A , D ( tr ) ) .
Stage II.
At the second stage, the trained W ∗ 1 ( A ) is used to generate saliency maps.
Specifically, given an input image x , we first use W ∗ 1 ( A ) and A to predict the class label (denoted by f ( x ; W ∗ 1 ( A ) , A ) ) of x .
Then an adversarial attack based approach [22, 18] is leveraged to calculate saliency scores of pixels.
Adversarial attack adds small random perturbations δ to pixels in x so that the prediction outcome on the perturbed image x
+ δ is no longer f
( x ; W ∗ 1 ( A ) , A ) .
Pixels perturbed more are more correlated with the prediction outcome f ( x ; W ∗ 1 ( A ) , A ) and are considered to be more salient.
This process amounts to solving the following optimization problem:
[Equation]
where δ i is the perturbation added to image x i and ε is a small norm-bound.
N is the number of images.
f ( x i + δ i ; W ∗ 1 ( A ) , A ) and f ( x i ; W ∗ 1 ( A ) , A ) are predictions made by W ∗ 1 ( A ) and A on x i
+ δ i and x i .
Assume the number of classes is K .
f ( x i + δ i ; W ∗ 1 ( A ) , A ) and f ( x i ; W ∗ 1 ( A ) , A ) are K -dimensional vectors containing prediction probabilities on individual classes.
ℓ ( · , · ) is the cross-entropy loss with ℓ ( a , b ) =
− ∑ Kk = 1 b i log a i .
In this optimization problem, we aim to find perturbations for each image so that the predicted outcome on the perturbed image is largely different from that on the original image.
The learned optimal perturbations are considered as saliency scores of pixels: larger perturbations indicate that the corresponding pixels are more correlated with the prediction outcome and therefore are more salient.
δ ∗ i depends on W ∗ 1 ( A ) and
A since δ ∗ i depends on the loss in Eq.(2), and the loss is a function of W ∗ 1 ( A ) and A .
Stage III.
At the third stage, given the saliency scores { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 , the second set of model weights W 2 are trained.
We use the saliency scores to reweight the pixels: x ⊙ δ , where ⊙ denotes element-wise multiplication (we compare with other reweighting mechanisms in Table 4).
Pixels that are more salient are given more weights.
Then W 2 is trained on these weighted pixels:
[Equation]
[Equation]
where f ( δ ∗ i ( W ∗ 1 ( A ) , A ) ⊙ x i ; W 2 , A ) is the prediction made by W 2 and A on the weighted image δ ∗
i ( W ∗ 1 ( A ) , A ) ⊙ x i , and t i is the class label.
W ∗ depends on { δ ∗ i ( W ∗ 1 ( A ) , A } Ni = 1 and A since W ∗ depends on the loss in Eq.(3), and the loss is a function of { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 and A .
Stage IV.
At the fourth stage, W ∗ 2 ( { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 , A ) and W ∗ 1 ( A ) are evaluated on a humanlabeled validation set D ( val ) .
The architecture A is updated by minimizing the validation losses:
[Equation]
where γ is a tradeoff parameter.
Four-level optimization framework.
We integrate the four stages into a unified four-level optimization framework and obtain the following formulation:
[Equation]
In this framework, there are four optimization problems, each corresponding to a learning stage.
From bottom to up, the optimization problems correspond to learning stage I to IV respectively.
The first three optimization problems are nested on the constraint of the fourth optimization problem.
These four stages are conducted end-to-end in this unified framework.
The solution W ∗ 1 ( A ) obtained at the first stage is used to generate explanations at the second stage.
The saliency maps { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni =obtained at the second stage are used to train W 2 at the third stage.
The solutions obtained at the first and third stage are used to calculate validation losses at the fourth stage.
The architecture A updated at the fourth stage changes the training loss at the first stage and consequently changes the solution
W ∗ 1 ( A ) , which subsequently changes { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 and W ∗ 2 ( { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 , A ) .
Optimization algorithm.
To solve the problem in Eq.(5), we used a standard algorithm developed in [41], which is broadly used in many previous works and demonstrated to be effective in the literature.
The convergence analysis of this algorithm has been given in many works [21, 23, 32, 42, 69].
The optimization algorithm is not the focus or contribution of our work.
In this algorithm, we calculate the gradient of L ( W 1 , A , D ( tr ) ) w.r.t W 1 and approximate W ∗ 1 ( A ) using a one-step gradient descent update of W 1 .
We plug the approximation W ′ 1 of W ∗ 1 ( A ) into ℓ ( f ( x i + δ i ; W ∗ 1 ( A ) , A ) , f ( x i ; W ∗ 1 ( A ) , A )) and obtain an approximated objective denoted by O δ i .
Then we approximate δ ∗
i ( W ∗ 1 ( A ) , A ) using a one-step gradient ascent update of δ i based on the gradient of O δ i .
Next, we plug the approximation δ ′ i of δ ∗ i ( W ∗ 1 ( A ) , A ) into ℓ ( f ( δ ∗ i ( W ∗ 1 ( A ) , A ) ⊙ x i ; W 2 , A ) , t i ) and get another approximated objective denoted by O W 2 .
Then we approximate W ∗ 2 ( { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 , A ) using a one-step gradient descent update of W 2 based on the gradient of O W 2 .
Finally, we plug the approximation W ′ 1 of W ∗ 1 ( A ) and the approximation W ′ 2 of W ∗ 2 ( { δ ∗ i ( W ∗ 1 ( A ) , A ) } Ni = 1 , A ) into the validation loss and get the third approximate objective O A .
A is updated by descending the gradient of O A .
These steps iterate until convergence.
Experiments
In this section, we present experimental results.
Please refer to the supplements for detailed hyperparameter settings and additional results (e.g., on inference costs, ablation studies).
4.1 Experiments on image classification
Following [41] the architecture A is parameterized in a differentiable way.
A contains a set of importance weights, each multiplied to the output of a candidate architecture block.
Architecture search amounts to learning these weights using gradient methods.
After learning, blocks with top weights compose an architecture.
Each experiment consists of two phrases: 1) architecture search where an optimal cell is identified, and 2) architecture evaluation where multiple copies of the optimal cell are stacked into a larger network, which is retrained from scratch.
[Table] Datasets We used three datasets: CIFAR-10 [35], CIFAR-100 [36], and ImageNet [15].
Both
CIFAR-10 and CIFAR-100 contain 60K images from 10 classes.
For each of them, we split it into a train, validation, and test set with 25K, 25K, and 10K images respectively.
ImageNet contains 1.2M training images and 50K test images from 1000 objective classes.
Following [68], we randomly sample 10% of the 1.2M images to form a new training set and another 2.5% to form a validation set, then perform a search on them.
Experimental settings We experimented with the search spaces in DARTS [41], P-DARTS [11], and PC-DARTS [68].
The tradeoff parameter γ is set to 2.
The norm bound ε of perturbations is set to 0.03.
During architecture search, for CIFAR-10 and CIFAR-100, the architecture of the target is a stack of 8 cells.
Each cell consists of 7 nodes.
Initial channel number is 16.
The search algorithm was based on SGD, with a batch size of 64, an initial learning rate of 0.025 with cosine scheduling, an epoch number of 50, a weight decay of 3e-4, and a momentum of 0.9.
We update the architecture A every 5 mini-batches (iterations), update model weights W 2 and perturbations δ every 3 mini-batches, and update W 1 on every mini-batch.
The rest of hyperparameters mostly follow those in DARTS, P-DARTS [Table] and PC-DARTS.
We compare with the following baselines: 1) explanation constraints (EC) [52]; 2) contextual decomposition explanation penalization (CDEP)
[51]; 3) global max pooling + GradCAM (GMPGC) [47].
The mean and standard deviation of 10 random runs are reported.
Results and analysis on CIFAR-100 and CIFAR-10 Table 1 shows results on CIFAR-100 and CIFAR-10.
Applying our framework to DARTS, P-DARTS, and PC-DARTS, test errors are greatly reduced, which shows that by end-to-end detecting and leveraging saliency of pixels, the quality of searched architectures can be improved.
Another observation from Table 1 is: the performance gain of our method does not come at the cost of substantially increasing model size (number of parameters) or search cost [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] Table 1 shows that our methods outperform EC, CDEP, and GMPGC.
These methods use GradCAM, gradient magnitude, and pretrained semantic segmentation models to calculate saliency scores, which are not very reliable.
In contrast, the calculation of saliency in our method is weakly supervised by the validation loss of the second model calculated on human-provided class labels, which have higher fidelity.
As analyzed earlier, saliency with higher fidelity can result in higher-quality architectures.
Results on ImageNet In Table 2, we compare different methods on ImageNet.
In experiments based on PC-DARTS, architectures are searched on a subset of ImageNet.
In other experiments, architectures are searched on CIFARand CIFAR-100.
Ours-darts2nd-cifar denotes that our method is applied to DARTS-2nd and performs search on CIFAR10.
Similar meanings hold for other notations in such a format.
The observations made from these results are consistent with those made from Table 1.
The architectures searched using our methods are consistently better than those searched by corresponding baselines.
These results again show that by end-to-end detecting and leveraging saliency can improve architecture search.
Sanity check of saliency maps We evaluate saliency maps generated by the adversarial saliency method using model parameter cascading randomization tests [1].
The model architecture is searched by SANAS on ImageNet.
Figure 2(left) shows that saliency maps change considerably as more layers are randomized, on multiple ImageNet examples.
Figure 2(right) shows the Spearman rank correlation (with absolute values) between original saliency maps and randomized saliency maps, on ImageNet.
The rank correlation consistently decreases as more layers are randomized.
These results demonstrate that saliency maps generated by the adversarial saliency method are sensitive to model parameters and pass the sanity check.
[Table] Human evaluation of saliency We randomly sampled 100 images from the test set of ImageNet and generated saliency maps for them using different methods.
Then we asked three undergraduates to judge whether the saliency maps are sensible.
The ratings are from 1-5 (higher is better).
Table 3 summarizes the mean and standard deviation of ratings.
Our methods achieve significantly higher ratings (significance test results are in the supplements) [Table] which demonstrates that our methods can generate more accurate saliency maps than the baselines.
Inter-annotator Kappa score is 0.71, which shows strong agreements among annotators.
Visualization of saliency In Figure 3, we visualize the saliency maps generated for some randomlysampled ImageNet images.
These saliency maps are very sensible.
Warmer color (representing higher saliency) regions correspond to objects.
Colder color regions correspond to background.
These results show that our method is effective in generating correct saliency maps.
In contrast, the saliency maps generated by baselines are less sensible.
For example, in the schipperke, hay, container ship, ice lolly images, higher saliency regions of baselines are in the background instead of on objects.
Ablation studies In terms of how to use saliency scores to reweight pixels, we compare the elementwise product between pixels and saliency scores in Eq.(3) with 1) element-wise addition between pixels and saliency scores; 2) element-wise product between pixels and the absolute values of saliency
[Table]
scores; 3) concatenate saliency map with input image and feed the concatenation into the second model W 2 ; 4) no reweighting.
Table 4 shows results where our method is applied to DARTS2nd and P-DARTS.
From this table, we make the following observations.
First , product works better than addition.
The reason is: magnitude of saliency scores (perturbations) is very small; adding them to pixels does not render a significant change of pixel values, and consequently cannot distinguish important pixels from unimportant ones.
In contrast, the relative difference between saliency scores is significantly large; multiplying them to pixels can better distinguish important and unimportant pixels.
Second , reweighting pixels using signed saliency score and absolute saliency scores does not have a significant difference.
This shows that signs of saliency scores do not matter too much.
Third , product works better than concatenation.
The possible reason is: compared with concatenation, product can better differentiate important and unimportant pixels using the saliency scores.
Fourth , reweighting pixels works better than no reweighting.
This demonstrates that multiplying saliency scores to inputs is indeed helpful in identifying important pixels, which helps to improve classification performance.
In the next ablation study, we compare the adversarial attack (AA) based saliency detection method with another two saliency detection methods, including integrated gradients (IG) [60] and SmoothGrad (SG) [57] by plugging them into our framework.
These studies are performed on Darts2nd and Pdarts.
Table 5 shows the results, where we make two observations.
First, our framework with IG and SG still outperforms vanilla Darts2nd and Pdarts.
This demonstrates that our framework is a general one that generalizes beyond a single saliency detection method.
Second, IG and SG perform worse than AA.
A possible reason is: IG and SG restrict the definition of saliency to be gradient-based.
In contrast, AA treats saliency scores as optimization variables and automatically learns them by solving an optimization problem, which is more flexible.
To better understand the effectiveness of the proposed four stages performed end-to-end, we compare with the following two ablation settings, which are performed on Darts2nd and Pdarts.
• Perform the four stages separately (denoted as Separate ) instead of end-to-end.
• Perform stages I, II, III by optimizing the weighted sum of their objective functions with weights 1, 0.5, 1, in a multi-task learning (MTL) manner (denoted as MTL ). [Table]
Table 7 shows the results on Separate and MTL.
We make two observations.
First, our end-to-end method works better than Separate which conducts the four stages separately.
Conducting the four stages end-toend can enable them to mutually influence each other to achieve the overall best performance.
In contrast, when conducted separately, earlier stages cannot be influenced by later stages (e.g., stage I cannot be influenced by stage IV), which leads to worse performance.
Second, our method performs better than MTL.
The tasks in stages I-III have an inherent order: before detecting saliency maps using a model, we first need to train this model; before training the second model on saliency-reweighted data, we need to detect the saliency maps first.
MTL performs these three tasks simultaneously by minimizing a single objective, which breaks their inherent order and therefore leads to worse performance.
In contrast, our method preserves this order using multi-level optimization.
[Table]
Figure 4(right) shows how the test error of SANAS-darts2nd on CIFAR100 varies with γ .
When γ = 0, the validation loss of W 1 is not used for architecture search and the performance is inferior (compared with γ = 2).
A γ in the middle ground which properly balances the two validation losses yields the optimal performance.
Using the validation loss of W 1 only is equivalent to vanilla DARTS-2nd (results are in the supplements).
4.2 Experiments on text classification
In this section, we apply the proposed framework for text classification.
The Gumbel softmax trick [31] is leveraged to deal with non differentiability of texts.
Dataset We conduct experiments on six datasets in the GLUE benchmark [64]: SST-2, MRPC,
QQP, MNLI, QNLI and RTE.
SST-2 is a sentiment classification dataset where the input text is movie review and the output label is whether the review is positive or negative.
In MRPC and QQP, the input is a pair of sentences and the output is whether they are semantically equivalent.
MNLI, QNLI, and RTE are textual entailment recognition datasets.
Baselines We compare with the following baselines: 1) BERT [16], 2) BERT-PKD [59], 3) DistilBERT [53], 4) TinyBERT [33], 5)
BiLSTM SOFT [63], 6) AdaBERT
[7], 7) EC-AdaBERT [52], 8)
CDEP-AdaBERT [51], and 9) GMPGC-AdaBERT [47].
[Table]
Hyperparameter settings Candidate operations are commonly used operations in convolutional networks, including 1D convolution, dilated convolution, pooling, identify, and zero.
In dilated convolution, the kernel size includes 3, 5, and 7.
Each convolution operation consists of an Relu-Conv-BatchNorm sequence.
For pooling, we used average pooling and max pooling, where the kernel size is set to 3.
The “SAME" padding is utilized for convolution and pooling.
We optimize weight parameters using SGD.
The initial learning rate is set to 2 e − 2.
It is annealed using a cosine scheduler.
The momentum is set to 0.9.
We use Adam [34] to optimize the architecture variables.
The learning rate is set to 3 e − 4 and weight decay is set to 1 e − 3.
Main results Table 6 shows the results.
We make the following observations.
First, our method works better than AdaBERT, which is an NAS method without saliency detection.
This further demonstrates the effectiveness of saliency detection in improving NAS.
Second, our method works better than EC, CDEP, and GMPGC.
This further shows that performing saliency detection and NAS jointly is better than conducting them separately as the three baselines do.
Third, compared with BERT 12 and BERT 12 -T, our method achieves similar performance while using much fewer parameters and being much faster during inference.
Qualitative results Table 8 shows salient words detected by different methods on a randomly sampled sentence (whose sentiment is labeled as being positive).
As can be seen, our method can successfully recognize the words “entertaining" and “please" which are mostly relevant to a positive sentiment.
In contrast, the two baselines fail to do that.
Conclusions and discussions
E rr o r ( % ) Gamma CIFAR-
In this paper, we propose to leverage the saliency information of input data to improve NAS.
Our work makes the following contributions.
First, our method can detect saliency and perform NAS end-to-end, based on a fourlevel optimization framework.
The framework performs four stages in a unified way: train a preliminary model, generate saliency maps using the preliminary model, retrain the model on saliency-reweighted data, and update architecture by minimizing validation losses.
Second, our framework is end-to-end differentiable, allowing using efficient gradient-based algorithms as solvers.
Third, our method provides a mechanism to evaluate generated saliency maps by checking whether they are helpful for improving classification performance.
We demonstrate the effectiveness of our method on several datasets.
EC an entertaining mix of period drama and flat-out farce that should please history fans.
GMPGC an entertaining mix of period drama and flat-out farce that should please history fans.
Ours an entertaining mix of period drama and flat-out farce that should please history fans. EC a very witty take
on change, risk and romance and the film uses humor to make its points about acceptance and growth
GMPGC
a very witty take on change, risk and romance and the
film
uses humor to make its points about acceptance and growth.
Ours a very witty take on change, risk and romance, and the film uses humor to make its points about acceptance and growth.
One major limitation of this work is that it cannot be easily applied to nondifferentiable NAS methods that are based on reinforcement learning (RL) and evolutionary algorithm (EA).
The reason is that our method uses a gradient-based optimization algorithm to solve the multi-level optimization problem.
For non-differentiable NAS methods, their non-differentiable objective functions do not have gradients, which therefore are not compatible with the gradient-based algorithm used by our method.
Please see Appendix A.1 for discussion on how to extend our method to non-differentiable NAS methods.
Another limitation of our method is its higher time cost than baselines, due to the extra computation needed for detecting saliency maps.
Considering the benefits and limitations of our method, we recommend using our method in applications that strongly need high-performance architectures capable of generating sensible saliency maps but do not have strong efficiency requirements on architecture search time.
For applications which have high restrictions on search cost but allow sacrificing some performance and ignoring saliency maps, other NAS methods might be better choices.
Please see Appendix A.1 for a more detailed discussion.
One potential negative societal impact of our work is: in mission-critical applications such as disease diagnosis and autonomous driving, if saliency maps generated by our method are not correct, they may mislead human decision-makers.
For future works, we plan to investigate these ideas: 1) formulate saliency-based network pruning [76] as a saliency-aware NAS problem and automatically search for the optimal pruning decisions based on detected saliency; 2) extend the notion of saliency from input data to blocks in neural networks, develop multi-level optimization based frameworks to detect the saliency of blocks, and perform pruning on blocks based on detected saliency.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] The contributions are summarized in Lines 37-42, which are consistent with the main claims made in the abstract and introduction.
(b) Did you describe the limitations of your work?
[Yes] The limitations are described in Lines 352-354.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] The potential negative societal impacts are discussed in Lines 354-356.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The data is publicly available.
The code is proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The major training details are described in Lines 170-229.
Additional training details are described in the supplements.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Error bars (standard deviations) are reported in Table 1, 2, 4, and 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] They are included in the supplements.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] They are cited in Line 199.
(b) Did you mention the license of the assets?
[N/A] These datasets are public.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] These datasets are public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] These datasets are well-known public datasets, which do not contain personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] The instructions are included in the supplements.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
Our study does not need IRB approvals.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
The study is a voluntary study, which does not involve compensation.
A Appendix A.1 More discussion of limitations
Apply SANAS to non-differentiable NAS methods.
To apply SANAS to non-differentiable NAS methods, we have to change the current gradient-based optimization algorithm to some other nongradient-based optimization algorithms (such as REINFORCE [66] for reinforcement learning), which might incur higher computational costs.
To apply SANAS to reinforcement learning (RL) based NAS methods, we perform the following procedures.
First, we use an RL controller [78] to generate a set of candidate architectures.
Second, given a candidate architecture, we train its weight parameters on a training dataset, similar to stage I in SANAS.
Third, given the trained model, we perform adversarial attacks to detect the saliency maps of the training data, similar to stage II in
SANAS.
Fourth, we use saliency maps to reweight training data and retrain the model on reweighted data, similar to stage III in SANAS.
Fifth, we evaluate the retrained model on a validation set and use validation accuracy as a reward for this architecture.
We repeat steps 2-5 for every candidate architecture, calculate the mean reward on all candidate architectures, and update the RL controller by maximizing the mean reward using policy gradient [78].
These procedures repeat until convergence.
Similar procedures can be conducted to perform saliency-aware architecture search in evolutionary algorithm based NAS methods.
When to use our method and when not.
It is recommended to use SANAS in applications that strongly need high-performance neural architectures capable of generating sensible saliency maps but do not have strong requirements on the time spent on architecture search.
For example, imagingbased disease diagnosis is a good application scenario of SANAS, for two reasons.
First, disease diagnosis needs to be highly accurate and needs high-fidelity saliency maps for interpreting predicted diagnosis outcomes.
Second, to use an automatically searched neural architecture in hospitals, FDA approval is needed, which usually takes several months.
To successfully pass the FDA approval, it is acceptable to take some extra time to search for a high-quality neural architecture.
For applications which have high restrictions on search cost but allow sacrificing some performance and ignoring saliency maps, other NAS methods that have higher search efficiency but lower performance and weaker saliency-generation capability than our method might be better choices.
Examples of such applications are online learning applications which need to update architectures in real time.
A.2 More examples of visual saliency maps
Figure 5 shows more examples of visual saliency maps.
The saliency maps detected by our method are more sensible than those detected by baselines.
A.3 More examples of salient word detection
Table 9 shows more examples of salient words detected by different methods.
In each example, the top2 words detected by our method are more salient than baselines.
The prediction task corresponding to
Table 8 and 9 is sentiment classification.
A word is more salient if it has a stronger correlation with a sentiment (either positive or negative).
For example, the word “entertaining” in Table 8 implies a positive sentiment, and therefore is considered to be salient.
In contrast, the word “mix” is a neutral word that is irrelevant to sentiments, and therefore is not considered to be salient.
EC with youthful high spirits, tautou remains captivating throughout michele’s religious and romantic quests, and she is backed by a likable cast.
GMPGC with youthful high spirits, tautou remains captivating throughout michele’s religious and romantic quests, and she is backed by a likable cast.
Ours with youthful high spirits, tautou remains captivating throughout michele’s religious and romantic quests, and she is backed by a likable cast.
EC the title’s lameness should clue you in on how bad the movie is.
GMPGC the title’s lameness should clue you in on how bad the movie is.
Ours the title’s lameness should clue you in on how bad the movie is.
A.4 Improve computational efficiency of SANAS
We improved the computational efficiency of our method from both the algorithm side and implementation side.
On the algorithm side, we speed up computation by approximating the optimal solution at each stage using a one-step gradient descent update [41] and reducing the frequencies of these updates.
Specifically, we update the architecture A every 5 mini-batches.
In contrast, baselines (including Darts, Parts, Pcdarts, Prdarts) update A on every mini-batch.
We update model weights Wand perturbations δ every 3 mini-batches, and update W 1 on every mini-batch.
We empirically found that reducing the update frequencies of certain parameters can significantly speed up convergence without sacrificing performance.
Besides, when calculating hypergradients of A , we recursively approximate matrix-vector multiplications using finite-difference calculations [41], which reduces the computation cost from being quadratic in matrix dimensions down to linear.
On the implementation side, we speed up computation by leveraging techniques and tricks including 1) automatic mixed precision [44], 2) using multiple (4, specifically) workers and pinned memory in
PyTorch DataLoader, 3) using cudNN autotuner, 4) kernel fusion, etc.
A.5 Hyperparameter tuning strategies
Most hyperparameters in our method follow their default values used in baseline methods.
The only hyperparameter needing to be tuned is the tradeoff parameter γ .
To tune γ on CIFAR-100, we randomly sample 2.5K data from the 25K training set and sample 2.5K data from the 25K validation set.
Then we use the 5K sampled data as a hyperparameter tuning set.
γ is tuned in 0.1, 0.5, 1, 2, 3.
For each configuration of γ , we use the remaining 22.5K training data and 22.5K validation data to perform architecture search and use their combination to perform architecture evaluation (retraining a larger stacked network from scratch).
Then we measure the performance of the stacked network on the 5K sampled data.
γ value yielding the best performance on the 5K sampled data is selected.
For γ in CIFAR-10 and ImageNet experiments, we simply used the value tuned on CIFAR-100 and did not conduct further tuning.