P ERFORMANCE P REDICTION VIA U NSUPERVISED D OMAIN A DAPTATION FOR A RCHITECTURE S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
Performance predictors can directly predict the performance value of given neural architectures without training, thus broadly being studied to alleviate the prohibitive cost of Neural Architecture Search (NAS).
However, existing performance predictors still require training a large number of architectures from scratch to get their performance labels as the training dataset, which is still computationally expensive.
To solve this issue, we develop an performance predictor by applying the unsupervised domain adaptation technique called USPP, which can avoid costly dataset construction by using existing fully-trained architectures.
Speciﬁcally, a progressive domain-invariant feature extraction method is proposed to assist in extracting domain-invariant features due to the great transferability challenge caused by the rich domain-speciﬁc features.
Furthermore, a learnable representation (denoted as operation embedding) is designed to replace the ﬁxed encoding of the operations to transfer more knowledge about operations to the target search space.
In experiments, we train the predictor by the labeled architectures in NAS-Bench101 and predict the architectures in the DARTS search space.
Compared with other state-of-the-art NAS methods, the proposed USPP only costs 0 .
02 GPU days but ﬁnds the architecture with 97 .
86% on CIFAR-10 and 76 .
50% top-1 accuracy on ImageNet.
I NTRODUCTION
Neural Architecture Search (NAS) (Elsken et al., 2019) aims to automatically design highperformance neural architectures and has been a popular research ﬁeld of machine learning.
In recent years, the architectures searched by NAS have outperformed manually-designed architectures in many ﬁelds (Howard et al., 2019; Real et al., 2019).
However, NAS generally requires massive computation resources to estimate the performance of architectures obtained during the search process (Real et al., 2019; Zoph et al., 2018).
In practice, this is unaffordable for most researchers interested.
As a result, how to speed up the estimation of neural architectures has become a hot topic among the NAS community.
Performance predictor (Wen et al., 2020) is a popular accelerated method for NAS.
It can directly predict the performance of neural architectures without training, thus greatly accelerating the NAS process.
A large number of related works are carried out because of its superiority in reducing the costs of NAS.
For example, E2EPP (Sun et al., 2019) adopted a random forest (Breiman, 2001) as the regression model to effectively ﬁnd promising architectures.
ReNAS (Xu et al., 2021) used a simple LeNet-5 network (LeCun et al., 1998) as the regression model, and creatively employed a ranking-based loss function to train the predictor, thus improving the prediction ability of the performance predictor.
Although existing performance predictors gain huge success in improving the efﬁciency of NAS, sufﬁcient architectures need to be sampled from the target search space and be fully trained to obtain their performance value as the label (Wen et al., 2020).
The performance predictor is trained by these labeled architectures, and then is used to predict the performance of architectures.
In order to ensure the prediction accuracy of the performance predictor, it is usually necessary to train at least hundreds of architectures as the dataset, which is a huge cost.
In recent years, many benchmark datasets such as NAS-Bench-101 (Ying et al., 2019), NAS-Bench201 (Dong & Yang, 2020), NAS-Bench-NLP (Klyuchnikov et al., 2020) are released for promoting the research on NAS.
There are a large number of architecture pairs (i.e., the architecture and its performance) in these datasets.
As a result, we are motivated to utilize the rich architecture knowledge in these datasets to predict the architectures in the target search space (i.e., the search space in which the architecture needs to be predicted.).
In this way, we can avoid training a large number of architectures in the target search space, thereby alleviating the expensive cost of building the dataset for performance predictors.
However, the search space designed in the benchmark datasets is very different from the real-world search spaces.
The performance predictor trained on existing labeled architectures cannot be applied to the target search space.
In this paper, we proposed an U n S upervised domain adaptation-based P erformance P redictor ( USPP ) with the usage of the domain adaptation technique.
Different from the traditional performance predictors that need the training data and the predicted data in the same search space, USPP can leverage the labeled architectures in existing benchmark datasets (e.g., NAS-Bench-
(Ying et al., 2019)) to build a powerful performance predictor for the target search space (e.g., the DARTS search space (Liu et al., 2018b)).
As a result, USPP can avoid expensive data collection for the target search space.
Speciﬁcally, the contributions can be summarized as follows:
• A progressive domain-invariant feature extraction method is proposed to reduce the transfer difﬁculty caused by the huge difference between source and target search spaces.
The progressive method explicitly models the domain-speciﬁc features and gradually separates them from the domain-invariant features, thus assisting in the alignment of the source and target search spaces.
• A learnable representation for the operations in architectures,
i
e., operation embedding, is designed to transfer more knowledge about operations to the target search space.
Compared to the widely used ﬁxed encoding method, the operation embedding can effectively capture the inner meaning and structural role of each operation in the source search space and applied them in the target search space to reduce transfer difﬁculty.
• USPP only costs 0 .
02 GPU days to search for the architectures in the DARTS search space because there is no need to annotate the architectures in the target search space.
Furthermore, the searched architecture by USPP achieves 97 .
86% classiﬁcation accuracy on CIFAR-10 and 76 .
50% classiﬁcation accuracy on ImageNet and outperforms all the stateof-the-art methods compared.
R ELATED W ORK 2. NAS AND P ERFORMANCE P REDICTORS
NAS can automatically design high-performance neural architecture and consists of search space, search strategy, and performance estimation strategy (Elsken et al., 2019).
Speciﬁcally, the search space deﬁnes the collections of the candidate architectures.
The search strategy corresponds to the employed optimization algorithms for the search, which can be mainly classiﬁed into evolutionary algorithms (B¨ack et al., 1997), reinforcement learning (Kaelbling et al., 1996) and gradient descent (Liu et al., 2018b).
The performance estimation strategy deﬁnes how to evaluate the architectures to obtain their performance.
During the search, the NAS algorithms use the search strategy to search architecture in the predeﬁned search space and obtain the performance value of the searched architectures by the performance estimation strategy.
No matter what search strategy is used, a lot of neural architectures need to be estimated.
Because of the heavy cost of the traditional GPU-based estimation method, many accelerated methods are proposed such as early stopping policy (Sun et al., 2018), proxy dataset (Sapra & Pimentel, 2020), weight-sharing method (Bender et al., 2018).
However, the ﬁrst two methods may lead to poor generalization and low-ﬁdelity approximation of performance value, and the weight-sharing method may be unreliable in predicting the relative ranking among architectures (Li et al.
; Yang et al.), which disobeys the goal of ﬁnding the architecture with the highest ranking.
Performance predictor is free from the aforementioned shortcomings and has received great attention in recent years.
However, existing predictors have the constraint that the source search space and the target search space must be the same.
The proposed USPP breaks the limitation and creatively uses the existing labeled architectures in the source search space to predict the architectures in the target search space, thus removing the reliance on potentially costly labels in the target search space.
2. D OMAIN A DAPTATION
In many machine learning methods, there is a major assumption that the training data and testing data are from the same distribution (Wilson & Cook, 2020).
When it is not held, the network trained on the source domain will face a performance decline when testing on the target domain (Patel et al., 2015).
Transfer learning (Weiss et al., 2016) can solve the problem by learning knowledge from the source domain and applying it to the target domain.
Domain Adaptation (DA) (Wilson & Cook, 2020) is a subcategory of transfer learning, and its goal is to train a network that performs well on a different but related target domain by the labeled data in the source domain.
The scenario in this paper is a problem of Unsupervised Domain Adaptation (UDA), where the labeled data are only available in the source domain.
The mainstream method to address the UDA is aligning source and target domains by learning the domain-invariant feature representation.
Speciﬁcally, the features are said to be domain-invariant if features extracted from data in both source and target domains follow the same distribution.
If a network performs well in the source domain using domain-invariant feature representation, the network can be generalized well to the target domain.
Generally, the domain-invariant methods can be classiﬁed into two categories.
The ﬁrst category explicitly reduces the domain discrepancy to obtain the domain-invariant by some distribution discrepancy metrics such as Maximum Mean
Discrepancy (MMD) (Gretton et al., 2006; 2012), correlation alignment (Sun et al., 2016), and contrastive domain discrepancy (Kang et al., 2019).
Motivated by the Generative Adversarial Network (GAN) (Goodfellow et al., 2014), another category learns domain-invariant representation by adversarial training (Ganin et al., 2016), and this is also what our method belongs to.
The methods generally train a domain discriminator to distinguish the source domain from the target domain, and train a feature extractor to fool the discriminator to extract domain-invariant representations.
Adversarial domain adaptation achieves great success in many ﬁelds, and a lot of works are proposed.
For example, symnets (Zhang et al., 2019) designed symmetric classiﬁers to play the role of domain discriminator.
GVB (Cui et al., 2020b) constructed bridge layers on both the generator and discriminator to reduce the overall transfer difﬁculty.
However, most existing adversarial methods are performed on the classiﬁcation task of Computer Vision (CV) or Neural Language Processing (NLP), and cannot be directly exploited in performance predictor which is a regression problem and deals with a completely different data type (i.e., graph data).
In this paper, we apply the adversarial methods in performance predictors and propose a progressive domain-invariant feature extraction method to reduce the transferring difﬁculty.
As far as we know, this is the ﬁrst work to alleviate the high costs of performance predictors by the adversarial domain adaptation techniques.
A PPROACH 3. F ORMULATION
In the scenarios of our paper, the architecture dataset in the source domain is denoted as D S
= (cid:8)(cid:0) x
S i , y S i (cid:1)(cid:9) N
s i =1 with N S labeled architectures, where x S i and y S
i denote the i th architecture and its performance value, respectively.
The dataset in the target domain is denoted as D T = (cid:8)
x Ti (cid:9) N T i = with N T unlabeled architectures.
USPP aims to train a performance predictor on D S and D T , and achieve promising results in the performance prediction of the architectures in target domain.
The overall framework of the proposed USPP is shown in Fig 1.
Speciﬁcally, to build the predictor, the neural architectures ﬁrst should be represented into a form that can be fed into the predictor.
Generally speaking, most neural architectures can be regarded as the Directed Acyclic Graph (DAG).
The node in the DAG corresponds to one speciﬁc operation (e.g., convolution 3 × 3), and the edge stands for the connection between operations.
Consequently, each architecture can be represented by the adjacency matrix A and the operation list L op , where A stands for the topology connections between nodes and L op represents the operations of nodes.
Because the most commonly used encoding method for L op (i.e., one-hot method) ignores the intrinsic relationship between different operations, we adopt a learnable operation embedding E to map L op to a continuous space R to avoid the drawback, and the operation embedding will be elaborated in Sec. 3.4.
Then, we use a Graph
Convolution Network (GCN) as the ﬁrst component of USPP to map every architecture into the feature space Z .
In speciﬁc, GCN is a powerful technique to learn useful features from graph data which is a kind of non-Euclidean data, and has achieved success in processing neural architecture
[Figure]
[Equation] [Equation]
[Figure]
data.
After learning the latent representation of the architectures by G , regressor R , as the second component of USPP, is exploited to predict the performance of the architectures.
To ensure the prediction accuracy in target domain, the prediction results in source domain needs to be guaranteed ﬁrst.
It can be achieved by minimizing the regression loss L reg on G and R when training.
The regression loss can be calculated as:
[Equation]
where L is the L 2 loss fuction in USPP.
3. A DVERSARIAL T RAINING
As mentioned above, we are motivated to use the existing labeled architectures in the benchmark datasets to build a performance predictor for the target search space.
However, the gap between the architectures in different search spaces is so large that the performance predictor trained with the labeled architectures in the source domain cannot be directly applied to predict the architectures in the target domain.
Accordingly, we leverage the adversarial training (Wilson & Cook, 2020) to learn domain-invariant representations, thus reducing the discrepancy between source and target domains.
In this way, the performance predictor can generalize well to the target domain.
To perform adversarial training, a discriminator D is designed as the third component of USPP, and it is a fully connected layer.
Similar to the regressor R , D is attached after the feature extractor G .
Speciﬁcally, the discriminator is essentially a domain classiﬁer to differentiate between the features from the source domain and the target domain, and only works during the training process of USPP.
During the training, the discriminator D is optimized to correctly classify the domains, while the feature extractor G is optimized to fool the discriminator so that the discriminator cannot distinguish the source domain from the target domain.
Through the adversarial training, the feature extractor is encouraged to extract the common features shared by domains (i.e., domain-invariant features).
To adversarially train the discriminator and the feature extractor, the domain classiﬁcation loss L cls is designed, which can be formulated as:
[Equation]
To make D distinguish between the source domain and target domain, we train D to minimize the domain classiﬁcation loss L cls .
At the same time, we train G to maximize L cls , thus making the feature distributions from source and target domains as similar as possible to fool D .
3. P ROGRESSIVE D OMAIN - INVARIANT F EATURE E XTRACTION
Although the adversarial training can align the source and target domains to a certain extent, it is difﬁcult to reduce the divergence between source and target domains to zero in practice due to the existence of rich domain-speciﬁc features in respective domains.
Speciﬁcally, domain-speciﬁc features refer to the features unique to each domain and greatly hinder the alignment of the data distributions from source and target domains.
To further reduce the negative inﬂuence of domainspeciﬁc features, we propose a simple but effective progressive domain-invariant feature extraction method.
The method reduces the inﬂuence of domain-speciﬁc features mainly by explicitly modeling them.
In fact, there are some similar works to mitigate domain-speciﬁc features in this way.
For example,
DSN (Bousmalis et al., 2016) separately modeled the domain-speciﬁc features of each domain and the domain-invariant feature shared by source and target domains and reconstructed the input data by the extracted domain-speciﬁc and domain-invariant features to add generalizability.
However, the modeled domain-invariant features also participated in the reconstruction, leading to them with a lot of domain-speciﬁc properties.
HDA (Cui et al., 2020a) proposed a heuristic framework, and leveraged the modeled domain-speciﬁc features as heuristics to gradually gain domain-invariant features.
However, the multiple sub-networks in the heuristic network made it difﬁcult to optimize.
The proposed progressive feature extraction method can effectively model the domain-speciﬁc features and gradually separate them from the features for the architecture data processed in this paper.
To alleviate the domain-speciﬁc features more easily, we hypothesize that the domain-speciﬁc features are easier to be captured compared with the domain-invariant features according to HDA.
This is because the architectures in the same domain generally share common domain-speciﬁc characteristics.
For example, the cell in the NAS-Bench-101 only has one input while that in the DARTS search space has two inputs.
Furthermore, the operations in NAS-Bench-101 are completely different from those in the DARTS search space.
Compared with the domain-invariant features, these domain-speciﬁc features are easier to be extracted.
The proposed method is embodied in the design of the regressor R .
The regressor is used to map the extracted feature from G to the predicted label, and consists of fundament layer F , bridge layer B , and prediction layer P .
Speciﬁcally, fundament layer F is used to model the current feature representation
r i , and bridge layer B is utilized to explicitly model the discrepancy from the current feature to the ideal domain-invariant feature, i.e., the domain-speciﬁc part b i in the current feature.
By subtracting the results of bridge layer b i from the results of r i , the domain-invariant features d i can be obtained.
This is because b i is easier to be obtained than d i and can give guide to the constructing of d i .
Finally, the obtained d i is fed to the prediction layer P to get the ﬁnal prediction label ˆ y .
The overall process can be formulated as:
[Equation]
where f i denotes the extracted feature from G .
During training, we seek to reduce the inﬂuence of the modeled domain-speciﬁc feature
b
i to help the extraction of domain-invariant representation.
To achieve this goal, we design an extra loss L bri as following:
[Equation]
where L is the size of b i .
During the training of USPP, we aim to gradually minimize L bri , thus mitigating domain-speciﬁc representation.
Therefore, at the beginning of training, the designed bridge layer assists in the progressive extraction of the domain-invariant feature.
In the later stage of training, b i is close to zero, which will not have too much impact on training.
In conclusion, USPP has three objectives in the training phase: 1.
Minimizing L reg on G and R to get high prediction accuracy in the source domain.
Minimizing L cls on D and maximizing L cls on G to perform adversarial training.
Minimizing L bri on R and G to assist in the extraction of the domain-invariant features.
Let L dom =
−L cls , the overall objectives during training is as following:
[Equation]
where γ and µ are the hyper-parameters.
3. O PERATION E MBEDDING
As is mentioned above, neural architectures can be represented by the adjacency matrix A and the operation list L op , where A represents the connection relationship between different nodes, and
L op represents the operation type of every node.
Generally, the one-hot method is used to encode the operation list L op to a ﬁxed matrix M len ( L op )
× K (Liu et al., 2021).
Then, the ﬁxed matrix
M len ( L op )
× K and A are fed into GCN to obtain the feature representation of a given architecture.
However, there is a large limitation that the process ignores the computational relationship between different operations.
This is because every operation is thought to be independent in the one-hot method, but it is obviously not in line with reality.
For example, the role of 1 × 1 convolution may be more similar to that of 3 × 3 convolution compared with that of the max pooling operation.
According to this motivation, the trainable operation embedding is proposed instead of the one-hot method.
Speciﬁcally, we use operation embedding E to map each operation in L op to a continuous space
R len ( L op )
× K .
K is the dimension of the mapped vector and can be set during the training.
Then, the K -dimensional vector obtained by E along with the adjacency matrix is mapped to the feature representation by GCN.
During the training process of USPP, the operation embedding is regarded as a part of GCN, and the weight of the operation embedding is optimized with the weights of GCN.
Through this, the operation embedding E can seek the inner relationship between different operations during the training.
In speciﬁc, the operation embedding is similar to the word embedding in
NLP, except that the mapped object words are replaced by the operation in the architecture.
Word embedding can encode the semantic and syntactic information of every word, where semantic information is related to the meaning of words, and syntactic information corresponds to the structural roles of words (Li & Yang, 2018).
Similarly, the inner meaning and structural information of each operation can be also learned from the source domain and applied in the target domain through the operation embedding.
As a result, operation embedding helps USPP transfer more information about the operation from the source domain to the target domain, thus improving the prediction performance in the target domain.
3. T HEORETICAL ANALYSIS
In this section, we analyze the expected target-domain error for the proposed method based on the theories of domain adaptation (Ben-David et al., 2010).
Let S and T be the source and target distributions for a family of fundament layer F in the hypothesis space H .
ε T ( F ) and ε S ( F ) be the expected error of a hypothesis F ∈ H on target and source domains.
To bound the target error, we use the H ∆ H -distance to measure the distance between distributions S and T , which is deﬁned as the following:
[Equation]
We can get the target error of hypothesis F by the theory in (Ben-David et al., 2010):
[Equation]
(7) where λ =
ε T ( F ∗ ) + ε S ( F ∗ ) is the combined error of ideal hypothesis F ∗ = arg min F ( ε S ( F ) + ε T ( F )) on both domains.
By the deﬁnition of H ∆ H -distance, d H ∆ H ( D S , D T ) = 2 sup | ε S ( h, h (cid:48) ) − ε T
( h, h (cid:48) )
| .
Hence,
[Equation]
Because the ideal discrepancy B ∗ from the current feature to the ideal domain-invariant feature may be not fully represented by the practice discrepancy B , B ≤ B ∗ .
Furthermore, B ∗ =
F − F ∗ .
Hence, there exists a positive correlation between B and F :
[Equation]
As mentioned in Subsec.
3.3, F = B + P .
Furthermore, the goal of F and P is both to better obtain the performance of the given architecture.
As a result, the ideal hypothesis F ∗ =
P ∗ .
We could obtain the relationship between F and P :
[Equation] [Equation]
Equation (10) is held on source and target distributions.
Hence, ε S ( P, P ∗ ) =
(1 − k )
ε S ( F, F ∗ ) and ε T ( P, P ∗ ) =
(1 − k )
ε T ( F, F ∗ ) .
The P and F can both correctly predict the performance
[Table]
of architectures, the differences between them on the source domain are small.
Hence, ε S ( P ) =
ε S ( F ) .
Combined with Equation (10), we can obtain Equation (11):
[Equation]
It can be seen from Equation (11) that the upper bound of the target-domain error for P with the usage of B is lower than that for F without the usage of B .
Therefore, the proposed method can help to reduce the upper bound of the target-domain error, which also proves the effectiveness of our proposed method from a theoretical point of view.
E XPERIMENTS
In this part, we choose the NAS-Bench-101 as source domain and the DARTS search space as target domains to train USPP.
We ﬁrst report the results of neural architecture search in
DARTS to verify the effectiveness of USPP.
Then, we perform extensive ablation studies on various components.
Please note that we perform additional experiments (NAS-Bench-101 → NASBench-201, NAS-Bench-201 → NAS-Bench-101, NAS-Bench-201 → DARTS, NDS ResNet → NDS
ResNeXt Radosavovic et al.
(2019), and NDS ResNeXt → NDS ResNet) in Appendix B.
All experiments in the paper are performed on NVIDIA GTX 2080Ti
GPU.
The experimental settings is presented in Appendix A.
4. N EURAL A RCHITECTURE S EARCH ON DARTS
Training process.
The regression loss L reg , the domain classiﬁcation loss L cls , and the bridge loss
L bri are shown in Fig.
4.1(a).
As expected, L reg , L cls , and L bri reduce repaidly.
Kendall’s Tau (KTau) (Sen, 1968), which can describe the correlation between the predicted value and the groundtruth values, is used to measure the prediction performance of USPP.
The trend of the KTau value of USPP is shown in Fig. 4.1(b).
It can be seen that the KTau of USPP goes up very quickly and converges.
CIFAR-10.
The experimental results on CIFAR-10 are shown in Table 1.
It can be seen that the proposed USPP only costs 0 .
02 GPU days, which is the least among all methods compared.
Please
[Table]
note that we do not take the costs for building the NAS-Bench-101 into consideration because NASBench-101 is a public dataset and not constructed for our work.
Besides, TENAS with the least cost among other methods, takes 2.5 times as long as USPP.
This is because USPP takes advantage of the fully-trained architectures in NAS-Bench-101, thus eliminating the cost of building new architectures in the DARTS search space.
Furthermore, the searched architecture by USPP gets the highest test accuracy 97 .
86% compared with all the other state-of-the-art methods.
Speciﬁcally, the accuracy of the architecture searched is still 0 .
32% higher than that of the best performance architecture searched by other methods.
This shows that our method can effectively transfer knowledge from the source search space to the target search space.
[Figure]
ImageNet.
The comparison results on ImageNet are shown in Table 2.
It can be seen from the results that the search cost of USPP is the lowest, and the PC-DARTS method with the least cost of the other methods is ﬁve times of the USPP.
Besides, the searched architecture by USPP trained without any tricks can obtain the highest top-1 and top-5 accuracy, and the training result with positive technique largely surpasses existing methods in top-1 and top-5 accuracy.
As a result,
USPP can transfer useful information from NAS-Bench-101 and learn the inner correspondence between architectures and performance in DARTS.
[Table]
× 0.2679 √ 0.
[Table] [Table]
4. A BLATION S TUDY
In this part, we conduct ablation experiments on the progressive domain-invariant feature extraction method, operation embedding, and domain adaptation methods.
We randomly sample 100 architectures from NAS-Bench-301 Siems et al.
(2020) to evaluate the proposed USPP.
The progressive domain-invariant feature extraction.
To validate the effectiveness of the proposed progressive method, we have performed ablation experiments on the core components of the progressive domain-invariant feature extraction method, i.e., the bridge layer.
It can be seen from
Table 3 that the performance predictor with the bridge layer gains a higher Kendall’s Tau value and 0 .
1434 more than the result without the bridge.
The result indicates that the progressive domaininvariant feature extraction plays a positive role in improving the performance of USPP.
[Figure]
Operation embedding.
The results of the ablation study on the operation embedding are reported in Table 4.
It can be seen from the result with operation embedding is more 0 .
1009 than that without operation embedding, which illustrates that operation embedding improves the performance of the predictor.
Furthermore, the length of operation embedding is set as 3 in the experiment because of the small number of the operations, and the 3D visual result is shown in Fig.
Surprisingly, convolution 1 × 1 and output are the most similar pair in all pairs of operations.
Furthermore, just as we suspected, the distance between convolution operations is smaller than the distance between convolution and pooling operations.
Different domain adaptation methods.
We adapt the distribution discrepancy metric-based method MMD (Long et al., 2015) and the adversarial training-based methods DANN (Ganin & Lempitsky, 2015), GVB (Cui et al., 2020b), and HDA (Cui et al., 2020a).
The hyperparameters of these methods are optimized, and the best results of them are reported in Table 7.
It is obvious that our method outperforms all the peer competitors.
C ONCLUSION
In this paper, we propose an performance predictor with the unsupervised domain adaptation technique to reducing the high cost of annotating architectures.
Speciﬁcally, we design a progressive domain-invariant feature extraction to reduce the difﬁculty of alignment by explicitly modeling the domain-speciﬁc and separating them of domain-invariant features.
Moreover, we use operation embedding to map the operation list of every architecture to continuous space, and learn the deeper meaning and structural information of each operation during training.
The USPP achieves the test accuracy of 97 .
86 % and 76 .
5 % on CIFAR-10 and ImageNet, respectively.
Unfortunately, this paper does not consider sequence-based search spaces and we will propose a more general domain adaptation framework in the future.
A E XPERIMENTAL S ETTING
In this section we report the expermental settings of the experiments in Section 4.
Data processing.
In order to keep the DARTS search space and NAS-Bench-101 search space uniform, we convert the architecture of the DARTS search space with operations on edges to one with operations on nodes following the convention (Liu et al., 2021).
Then, since the types of operations in the two search spaces are very different, we group all operation types into ﬁve categories: input, output, convolution 1 × 1 , convolution 3 × 3 , convolution 5 × 5 , and pooling.
Setting of USPP.
The Gradient Reversal Layer (GRL) is utilized to simplify the adversarial training.
Speciﬁcally, GRL can ﬂip the gradients from D to G so that we can train G and R to minimize the sum of L reg , L dom and L bri .
We train the USPP with the batch size of 1 , 000 for 100 epoch.
We employ the SGD with the weight decay of 0 .
and the learning rate is initialed with 0 . 1 .
We set γ = 1 and λ = 1 in the loss which is veriﬁed to perform well in the experiments.
Search setting.
Followed the convention of performance predictor (Wen et al., 2020), we randomly sample 100 , 000 architectures from the DARTS search space, and predict them by the trained USPP.
Then, we select the architecture with the highest predicted performance value as the searched architecture.
Training setting of the searched architecture.
For the training setting on CIFAR-10, we follow the convention of (Lu et al., 2021b).
As for the training setting on ImageNet, we train the searched architecture under two types of training settings to compare with other methods fairly.
In the ﬁrst setting, we train architecture for 350 epoch with the batch size of 512 .
Besides, we use the SGD optimizer with a momentum of 0 .
9 , an initial learning rate of 0 . 2 , and the weight decay of 4 × 10 − 5 .
In the second setting, the cosine learning rate strategy is used during the training on the basis of the ﬁrst setting.
In addition, we use some enhancement techniques to improve the classiﬁcation accuracy on ImageNet.
Speciﬁcally, label smoothing (Szegedy et al., 2016) and autoaugment (Cubuk et al., 2019) which are effective regularization methods are utilized in the training.
Furthermore, the
Squeeze-and-Excitation (SE) module (Hu et al., 2018) is attached after each cell.
[Table] [Table]
B S UPPLEMENTARY EXPERIMENTS
In this part, we perform additional experiments on cell-based search space and block-based search space, and compare our method with other zero-cost methods.
Similar to USPP, the zero-cost methods do not need to train any architectures during the workﬂow.
As a result, we compare USPP with these methods to perform relatively fair comparisons.
The detail of the cell-based search space and the block-based search space used are shown in Fig.
6 and Fig.
7, respectively.
Speciﬁcally, NASBench-101, NAS-Bench-
[Table]
and DARTS belong to the cell-based search space, while NDS ResNet and NDS ResNeXt belong to the block-based search space.
B. C ELL - BASED SEARCH SPACE
[Table]
[Table]
Transferring from NAS-Bench-101 to NAS-Bench-201.
The comparisons of the KTau value on
NAS-Bench-201 between USPP and other zero-cost methods are shown in Table.
Please note synﬂow, grad norm, snip, grasp, ﬁsher, plain are all proposed to evaluate the
Abdelfattah et al., 2021), and the experimental results in Appendix are all obtained from the
[Table]
NASWOT (Mellor et al., 2021) 0.
USPP (ours) 0.
[Table]
paper Ning et al.
It can be seen that the USPP gain the highest KTau value among all peer competitors.
This illustrates that the proposed method can effectively transfer knowledge from
NAS-Bench-101 to NAS-Bench-201.
Transferring from NAS-Bench-201 to NAS-Bench-101.
Table 8 shows the comparison results of USPP on NAS-Bench-101.
It can be seen that USPP achives the highest KTau value among all peer competitors, which verify the effectiveness of USPP to transfer from NAS-Bench-201 to
NAS-Bench-101.
Transferring from NAS-Bench-201 to DARTS.
USPP is evaluated in NAS-Bench-301.
The comparison results of transferring from NAS-Bench-201 to DARTS search space are shown in Table.
It can be observed that the USPP learned in NAS-Bench-201 has a poor performance in the DARTS search space.
We infer that this is because NAS-Bench-201 is too small compared to DARTS (15.6k vs 10 18 ), and the domain discrepancy between this NAS-Bench-201 and DARTS is too large.
The information learned from NAS-Bench-201 is not enough to accurately predict the architecture in
DARTS.
B. B LOCK - BASED SEARCH SPACE
[Table]
value of USPP on NDS
ResNeXt (NDS ResNet → NDS ResNeXt).g
[Table]
Transferring from NDS ResNet to NDS ResNeXt.
We apply the 25k public annotated architectures in the NDS ResNeXt search space and the unlabeled architectures in the NDS ResNet search space to train USPP.
Please note that we do not use the proposed operation embedding method, and replace the GCN (i.e., feature extractor) with the MLP.
The comparison results of the KTau value are shown in Table.
The results show that the proposed USPP surpasses all competitors, and is 0.135 higher than the second-place Ktau value.
This veriﬁes the effectiveness of our approach in
NDS ResNet which belongs to the block-based search space.
Transferring from NDS ResNeXt to NDS ResNet.
Similar to the last experiment (i.e., NDS
ResNeXt → NDS ResNet), We train USPP on 25k labeled architectures in the NDS ResNet search space and test USPP on the public annotated architectures in the ResNeXt search space.
The comparison results are shown in Table 12.
USPP achieves the second-highest KTau value and is only 0.074 lower than the ﬁrst place.
This illustrates that the proposed method can effectively learn the correlation between architecture and performance and transfer it to the NDS ResNeXt search space.
C D ETAILS OF THE TRAINING PROCESS
Quantity of the domain-adaptation features.
We have performed experiments on NAS-Bench201 → DARTS and NAS-Bench-101 → NAS-Bench-201, and computed the mean KL-Divergence and the mean cosine similarity.
Speciﬁcally, the KL-Divergence is 0.0086, and the mean cosine similarity is 0.5018.
These results show that the domain invariant features we learn are similar in different cases (i.e., NAS-Bench-201 → DARTS, NAS-Bench-101 → NAS-Bench-201), and further prove the quality of domain invariant features.
The regression error and the domain classiﬁcation accuracy.
The results are shown in Table.
The regression error is low, and illustrates that the regressor learns the correspondence between the architecture and performance in the source domain.
Furthermore, the domain classiﬁcation accuracy can verify that the discriminator cannot distinguish between domains.