Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor
Abstract
Neural Architecture Search (NAS) can automatically design promising neural architectures without artiﬁcial experience.
Though it achieves great success, prohibitively high search cost is required to ﬁnd a high-performance architecture, which blocks its practical implementation.
Neural predictor can directly evaluate the performance of neural networks based on their architectures and thereby save much budget.
However, existing neural predictors require substantial annotated architectures trained from scratch, which still consume many computational resources.
To solve this issue, we propose a Cross-Domain Predictor (CDP), which is trained based on the existing NAS benchmark datasets ( e.g. , NAS-Bench-101), but can be used to ﬁnd high-performance architectures in large-scale search spaces.
Particularly, we propose a progressive subspace adaptation strategy to address the domain discrepancy between the source architecture space and the target space.
Considering the large difference between two architecture spaces, an assistant space is developed to smooth the transfer process.
Compared with existing NAS methods, the proposed CDP is much more efﬁcient.
For example, CDP only requires the search cost of 0.1 GPU Days to ﬁnd architectures with 76.9% top-1 accuracy on
ImageNet and 97.51% on CIFAR-10.
The source code will be available 3 .
Introduction
Neural Architecture Search (NAS) [16] can automatically design promising architectures of deep neural networks for real-world applications, such as image classiﬁcation, language modeling and medical image segmentation [50, 46, 62].
In recent years, there have been many architectures searched by NAS methods, which can surpass the performance of those manually designed [43, 23].
Generally, performing NAS methods often requires thousands of GPU Days.
Unfortunately, such a large amount of computing resources is not necessarily available to most researchers and enterprises interested.
As a result, NAS researchers are paying increasing attention to improve the efﬁciency of
NAS through developing effective acceleration methods, to complete the search within an acceptable cost [34].
Generally speaking, most of the existing search acceleration methods are developed based on some heuristic strategies, e.g. , reducing the size of search space [65, 15], searching upon a smaller proxy dataset [43, 33], and sharing the parameters of different architectures (weight sharing)
However, the shortcomings of the above methods are obvious.
Reducing the size of search space also limits the diversity of architectures [52], and the architectures optimized on the proxy dataset are not guaranteed to perform best on the target dataset [4].
Weight sharing ﬁrst trains a supergraph that is manually designed in advance and then searches subgraphs as the potential architectures in the supergraph.
The weights in the same components between different architectures are shared from the trained supergraph without further training, thus the efﬁciency of performing of NAS is greatly improved.
However, the weight sharing also suffers from performance instability and dramatic performance collapse [8, 6], which severely limits its application.
Neural predictor is a promising acceleration method, which directly estimates the performance of neural architectures without any training of the neural network during NAS [55].
For example,
Sun et al.
[49] proposed an end-to-end performance predictor that employed random forest [21] which reduced the consumed time of NAS signiﬁcantly.
Tang et al.
[54] used a graph convolution network (GCN) as the neural predictor and fully exploited the unlabeled architectures to improve the prediction accuracy.
Though these predictor-based methods tend to be more efﬁcient than other methods, sufﬁcient architectures which play as the annotated examples to train the predictor, are still required to sample from the target search space.
These sampled architectures are usually required to train fully to obtain their actual performance [55, 49].
A large search space requires a mass of samples, which becomes a bottleneck for further improving predictor-based methods.
Consequently, neural predictor still requires massive computing costs largely due to annotating the sampled architectures.
Fortunately, we note that many benchmark datasets have been released for NAS research, such as NAS-Bench-101 [60], NAS-Bench-201 [13], NAS-Bench-nlp [29], NAS-Bench-ASR [38], etc.
These datasets contain a great number of architectures with known real performance, and constructing them has consumed massive computational resources.
If these data can come in handy, there is no need to annotate other architectures for predictors and the search cost is greatly reduced.
However, the search spaces of these datasets are small and have a huge difference from the large search spaces adopted by popular NAS algorithms such as DARTS [33] and ProxylessNAS [4].
Although the architectures are all made up of operations, the settings are quite different, such as the types and the maximum number of operations.
As a result, these well-trained benchmark architectures cannot be directly used to build predictors.
In this paper, we develop a Cross-Domain Predictor (dubbed as CDP) by exploiting the existing NAS benchmarks, and then the predictor is implemented to ﬁnd high-performance neural architectures in large search spaces.
The contributions of CDP are summarized as follows: 1) We divide the architecture space for subspace adaptation, and propose a progressive subspace partition to accurately exploit the local information.
2) We design an assistant space, to achieve smoother transfer from the source space to the target one because of the signiﬁcant differences between searching domains.
3) We experimentally show that CDP can search promising architectures with only 0.1 GPU days on ImageNet and CIFAR-10, achieving 76.9% and 97.51% classiﬁcation accuracy, respectively.
Related Works 2.1 NAS and Neural Predictor
[Figure]
In general, there are three important parts in NAS [16], namely architecture space 1 , search strategy, and performance evaluation.
During the search process, each NAS method ﬁrst speciﬁes the architecture space that includes all the potential architectures to be searched.
Secondly, a particular search strategy is utilized to perform the search.
In practice, reinforcement learning [26], evolutionary computation [1], and gradient-based methods are the mainstream search strate- gies widely used.
Finally, the performance evaluation assesses the architectures searched to guide the search strategy.
In recent years, how to accelerate the performance evaluation is one of the most valuable research directions in NAS [34].
As a promising acceleration method, neural predictors have received great attention from researchers since 2017 [11].
The red and black parts in Fig.
1 illustrate a typical neural predictor-assisted NAS for a glance.
Recently, with the design of new predictors [49, 54, 55], the prediction performance of neural predictors has been continuously improved.
However, existing predictors request the source and the target architecture spaces to be the same.
If the predictor makes predictions for a speciﬁc NAS method, it must collect sufﬁcient labeled architectures from the search space of the corresponding NAS method in advance.
The CDP algorithm we proposed in this paper gets rid of the constraint that the source and the target architecture spaces must be the same.
Particularly, CDP directly utilizes the existing benchmark datasets as the training architectures in the source space, yet to predict the architectures in other target spaces.
In this way, the neural predictor will not need to sample labeled data from the target space of the particular NAS method, thus greatly reducing the cost of annotating the training architectures.
2.2 Transfer Learning and Domain Adaptation
Transfer learning is to apply knowledge learned in previous domains to other domains, thus reducing the need and effort to recollect labeled data [40].
Speciﬁcally, domain adaptation is a subcategory of transfer learning, where the source and target tasks are the same and labeled data is available in the source domain, which is the scenario of CDP.
According to [64], transfer learning can be classiﬁed into two different categories based on the consistency between the source and the target domains.
If the feature spaces are different in both domains, the scenario is termed heterogeneous transfer learning.
Otherwise, it is homogeneous transfer learning.
The proposed CDP algorithm in this paper follows heterogeneous transfer learning because the architecture spaces are different.
But observing that much more current literature addresses the scenario of homogeneous transfer learning [10], we proposed a uniform encoding strategy that uniﬁes the architecture spaces, so that it can be treated as homogeneous transfer learning.
The feature-based transfer is one of the most prevalent methods.
It aims at reducing the distribution difference between the source domain and the target domain with the help of some instances.
A common metric for measuring the difference is Maximum Mean Discrepancy (MMD) [18], which can quantify the difference by calculating the average distance of the instances sampled from both domains [64].
Inspired by MMD, many variants and improved approaches are proposed such as
MK-MMD [19], weighted MMD [59], and Local Maximum Mean Discrepancy (LMMD) [63].
Among them, LMMD can take advantage of local afﬁnity for more delicate adaptation and can achieve extraordinary results.
Speciﬁcally, LMMD ﬁrst classiﬁes the domain into several subspaces and then aligns the subspaces of the same category in pairs.
Secondly, LMMD uses the pseudo-labels output by the neural network to classify the target data throughout the training process.
However, it is well known that the neural network cannot achieve satisfactory results in the initial training stage, that is, the pseudo-label obtained through it may be seriously inconsistent with the real label in the early training process.
In this paper, we propose a progressive approach to alleviate the disadvantages of misclassiﬁcation.
Approach
In this section, we will document how the proposed cross-domain predictor bridges the gap between different architecture spaces.
The overall process of the proposed CDP algorithm is shown in Fig.
3.1 Formulation
Supposing the architectures in the search space X are X
= { x 1 , x 2 , ..., x n } , and their respective performances in space Y are Y
=
{ y 1 , y 2 , ..., y n } , the predictor P : X → Y , which is trained to predict the performance of architecture, can be formulized by Equation (1):
[Equation]
LMMD
FC layers
[Figure]
GCN
[Figure]
ො𝑦 s y s
[Figure]
Architecture
[Figure]
Progressive Subdomain
[Figure]
where W denotes the trainable weight of predictor P , and L is the loss function. After the training, the predictor with trained weight W will evaluate other architectures, say x i where i is not in { 1 , 2 , ..., n } , via Equation (2): [Equation] Mathematically, neural architectures can be presented as directed graphs, and a common encoding scheme of architecture x is to use feature vectors of operations O 0 and adjacency matrix M , which are further collectively employed to represent the type of each operation and the connection between operations.
GCN [28] is a prevalent technique tackling non-Euclidean data such as graphs and has achieved great success in representing neural architectures [55].
Therefore, it is used as the ﬁrst component in P to extract features of neural architecture G : X → Z , where Z is the feature space.
In this paper, we follow the convention and choose the GCN version implemented in [55] as the neural predictor, and the feature vectors are updated by Equation (3):
[Equation]
where M T is the transposed matrix of M and ReLU is an activation function.
W + l and W − l are two trainable weights of the l -th layer in GCN.
Speciﬁcally, the last two terms of Equation (3) allow the graph information to ﬂow forward and backward, respectively, which makes it more suitable for dealing with directed graphs like neural architectures.
In addition, the second component of P is a hypothesis h : Z → Y , consisting of several fully connected layers, and is tailed to the GCN for obtaining the predicted performance values.
As for loss function L , we also follow the convention suggested in the reference [55] using mean square error.
3.2 Progressive Subspace Adaptation
Although neural predictor can largely save search cost for NAS, the limitation of itself is also serious: it cannot be well trained due to the lack of available labeled architectures of the corresponding NAS algorithm.
Observing that many architecture datasets have been released, these data can be potentially used to train the neural predictors.
However, the distribution of these massive labeled architectures is signiﬁcantly different from that of the architectures from the popular search space that is often large.
In principle, domain adaptation is useful for reducing the distribution gap between these architecture spaces.
Let (cid:101) D S and (cid:101)
D T be the source and target distributions over Z .
We will speciﬁcally introduce how to bridge the gap between (cid:101) D S and (cid:101) D T in the proposed method as below.
Research has shown that deep neural networks have the ability to learn the transferable features well [61, 39].
In addition, many methods [35, 63] add metrics of discrepancy to the objective function to jointly optimize the neural network.
Based on Equation (1), the term of adaptation regularizer d ( · , · ) is added, and the objective function becomes:
[Equation]
where x sn and y sn denote architectures with labels in source search space X S , and θ > 0 is a penalty parameter.
Maximum Mean Discrepancy (MMD)
[18] is a classic metric and has been widely used in domain adaptation.
Let H k denote the reproducing kernel Hilbert space (RKHS) endowed with a characteristic kernel k , and (cid:101) U S , (cid:101) U T are samples of sizes n s , n t drawn from (cid:101) D S , (cid:101) D T respectively.
The regularizer term d ( · , · ) can be implemented by MMD, and the unbiased estimator of MMD is deﬁned by latent representation z s , z t :
[Equation]
where φ denotes feature map function mapping original space to H k and it is associated with k , k
( z s , z t ) =
(cid:104) φ ( z s ) , φ ( z t ) (cid:105) where (cid:104)· , ·(cid:105) means the inner product operation.
Considering that the differences between search spaces are quite large, the gap cannot be handled well if only MMD is used.
In addition, MMD is a global domain adaptation, which often ignores the local relationship.
In order to take full use of the local information between architectures, we present a progressive subspace adaptation strategy.
First, the source space and the target space are divided into several subspaces whose number will increase during the training process.
Secondly,
MMD loss is used between corresponding subspaces for adaptation.
Speciﬁcally, these two spaces will be divided according to the performance Y =
{ y 1 , y 2 , ..., y n } of architectures.
Between these two spaces, the architectures with good performance may share some characteristics, such as long shortcut connections and preferred operations.
We directly use the true labels y of architectures in the sources space and predict the pseudo-labels ˆ y of architectures in the target space to divide the subspaces.
The performance value y
i is a scalar, which needs to be classiﬁed ﬁrst to become a classiﬁcation label y
i that is a vector.
Secondly, the weight w ci which presents the x i belonging relationship to the c -th subspace can be calculated via
Equation (6):
[Equation]
where y ci denotes the c -th element of label vector y i .
With the weight w ci and Equation (5), the unbiased estimator of the discrepancy between subspaces becomes Equation (7):
[Equation]
This discrepancy is termed Local Maximum Mean Discrepancy (LMMD) [63] that takes the correlation from the same category of subspaces into consideration.
With the help of the kernel k , Equation (7) can be further modiﬁed to obtain the following formula for easy calculation of LMMD:
[Equation]
However, there is a severe issue that the pseudo-label ˆ y may be incorrect, especially when the neural network is not fully trained.
This will further affect the accuracy of ˆ d k ( (cid:101) D S , (cid:101) D T ) in Equation (8).
In order to alleviate the misclassiﬁcation due to inaccurate pseudo-labels, we propose a progressive subspace partition strategy.
All the architectures are partitioned into one space in the early stage of neural network training so that there will be no classiﬁcation errors caused by incorrect pseudo labels.
As the training progresses, the neural network can predict more accurately.
At this time, we slowly increase the number of subspaces.
Finally, the number of subspaces will end at the speciﬁed maximum K .
Progressive Subspace Partition: Supposing the neural network is speciﬁed to be trained for a total of E epochs, and the maximum number of subspaces is K , when the training progresses arrive at the e -th epoch, the number of subspace categories C e can be scheduled by Equation (9):
[Equation]
and ∀ i < j, Sche ( i ; E, K ) <
Sche ( j ; E, K ) .
Based on experience and experiments, in this paper we let C e
= K − (cid:98) cos ( π 2 E e ) ∗ K (cid:99) , where (cid:98)·(cid:99) denotes the round-down operation.
C e calculated by this schedule meets the characteristics discussed above, i.e. , when e is relatively small, C e is equal to 1 , while e is close to E , C e will be K .
Furthermore, we use the cosine function to increase the number of epochs with a small value of C e .
This strategy ensures that the neural network has been sufﬁciently trained before the C e becomes larger.
Therefore, the occurrence of classiﬁcation errors will be greatly reduced.
After determining the number of subspace categories C e during the e -th training epoch, we can use fractile to generate a new label y for the original label y which is a scalar.
To be more speciﬁc, the fractiles { f 0 , f 1 , f 2 , ..., f C e } of scalar labels need to be found at ﬁrst, where f C e is the largest one in the labels and f 0 is the lower bound.
Then, we can ﬁnd the corresponding subspace category c for y i if f c − 1 < y i ≤ f c .
After that, we set the c -th element of y i to one and other elements as zeros.
Finally, the classiﬁcation vector y i is used by Equation (6) for the calculation.
Here, we will further analyze the role of the proposed progressive subspace partition strategy.
When the number of subspaces is one, the numerator y ci in Equation (6) is always one, and the denominator (cid:80) y j ∈ Y y cj is equal to the number of source (or target) data.
Hence, w sci is equal to 1 /n s and w tci is equal to 1 /n t .
As a result, the unbiased estimator of LMMD (Equation (7)) becomes that of MMD (Equation (5)) which is a global adaptation.
While the number of subspaces increases to the speciﬁed maximum number K , the measurement of the discrepancy we used is actually LMMD which focuses on the relationship between the same categories of subspaces to help the subspace adaptation.
3.3 Assistant Architecture Space
[Figure]
Considering that the source architecture space and target architecture space are quite different, it is difﬁcult to directly transfer the source architecture space to the target space.
To address this issue, we design an assistant architecture space to make the transfer smoother.
Generally, the source space, where all architectures can be trained from scratch to obtain the performance of each architecture, is often relatively small.
However, the target architecture spaces are often large in practice.
An example is given based on two widely used architecture spaces, namely NAS-Bench-101 [60] and DARTS [33].
In terms of the number of possible architectures,
NAS-Bench-101 is 423K and DARTS is about 10 18 .
In terms of the maximum number of operations in a cell, NAS-Bench-101 is 5 while DARTS is 8.
Thus, the size of the assistant space should be between the size of the source space and the target space.
Besides, other features should be similar to these two architecture spaces.
In this paper, we specify the assistant space more similar to the target space.
Speciﬁcally, an assistant space for NAS-Bench-101, NAS-Bench-201 [13], and DARTS are designed following the above principles, dubbed as ‘tiny DARTS’ space.
Fig.
3 is an illustration of this assistant space, where we use dark black to represent the large architecture space and gray to represent the smaller one.
The participation of the assistant space makes the adaptation be a continuous process and alleviates the inadaptability caused by large differences.
The only difference between tiny DARTS and DARTS is that the number of intermediate nodes has changed from 4 to 3.
This directly reduces the number of operations from 8 to 6 which is equal to the situation in NAS-Bench-201.
Except for this change, any other settings, including the types of operations, etc. , have not been changed.
Therefore, the features in DARTS have also been preserved.
Speciﬁcally, there are two input nodes in tiny DARTS, and all intermediate nodes are connected to the output node.
In the early training epochs of CDP, the assistant space ( e.g. , tiny DARTS) is used as the target domain, i.e. , the input at the bottom in Fig.
In this stage, CDP focuses on bridging the smaller gap between the source space and the assistant space.
After several training epochs, the gap between the source space and the target space can be also reduced as the assistant space and target space are very similar.
Therefore, it is much easier to reduce the gap between the source and target spaces at this time.
As a result, the space transfer process becomes smoother with the assistant space, which improves the prediction performance.
After the determination of the target domain, an encoding strategy is required for different search spaces.
We propose a uniform strategy with four main steps, where The detailed descriptions are shown in Appendix A.
3.4 Theoretical Analysis
In this section, we give a theoretical analysis of the expected error of target architecture search space.
We ﬁrst introduce Lemma 1 which is a basic theorem in domain adaptation, and then further deduce the target error bound of CDP.
Lemma 1.
[3] Let (cid:15) T ( h ) and (cid:15) S ( h ) be the expected error on target and source domain and H be a hypothesis space, for h ∈ H :
[Equation]
(10) where λ = (cid:15) T ( h ∗ )
+ (cid:15) S ( h ∗ ) is the combined error of ideal hypothesis h ∗ =
arg min h ∈H ( (cid:15) T ( h )
+ (cid:15) S ( h )) on both domains and d H ( (cid:101) D S , (cid:101) D T ) is the upper bound of the A -distance [27].
Let (cid:101) U S,train , (cid:101) U S,valid be the training and validation datasets randomly sampled from the distribution
D S respectively, and (cid:101) U S,train ∩ (cid:101) U S,valid = ∅ .
After the predictor is trained on the training dataset (cid:101) U S,train , the validation dataset (cid:101) U S,valid is used to validate it.
The validation hypothesis H (cid:48) is a subset of H , and is only determined by (cid:101) U S,train .
The following theorem provides an upper bound of expected target-domain error in terms of the empirical error on (cid:101) U S,valid and MMD d k ( (cid:101) D S , (cid:101) D T ) .
Theorem 2.
Let d (cid:48) be the VC-dimension of H (cid:48) , m be the size of (cid:101) U S,valid , m (cid:48) be the size of unlabeled samples (cid:101) U S and (cid:101) U T .
With probability of 1 − δ , for h ∈ H (cid:48) :
[Equation]
[Equation]
Proof sketch: The proof is mainly divided into two steps.
We ﬁrst use the empirical error on source validation dataset ˆ (cid:15) S,valid ( h ) to represent the upper bound of expected error on source domain (cid:15) S ( h ) : [Equation] S ( h ) ≤ ˆ (cid:15) S,valid ( h )
+ [Equation]
The detailed proof can be found in Equations (19)-(24) in Appendix B. Secondly, the upper bound of the distance d H ( (cid:101) D S , (cid:101) D T ) is constrained by MMD d k ( (cid:101) D S , (cid:101) D T ) , i.e. :
[Equation]
and Equations (25)-(26) in Appendix B show the proof in detail.
Finally, by applying these inequalities to Lemma 1, the theorem can be obtained.
Theorem 2 shows that the expected error on the target domain is bound by two terms, the empirical validation error in source domain ˆ (cid:15) S,valid ( h ) and MMD d k ( (cid:101) D S , (cid:101) D T ) which is minimized as one term of the objective function.
This means that when observing the predictor performs well on the source validation dataset, the expected error on the target domain will also be small. We report the results with the lowest cost if there are multiple architectures searched in the literature of the compared methods. The symbol ‘—’ denotes that no results are available in original papers. [Table]
Experiments
In this section, we will empirically investigate the effectiveness of the proposed method, including the experiments conducted on ImageNet [12] and CIFAR-10 [30], and the ablation studies.
Our experiments are conducted with PyTorch [41] and MindSpore [25].
4.1 Experimental Settings
Architecture spaces for CDP:
NAS-Bench-101 and NAS-Bench-201 are chosen as the source spaces and DARTS is the target space.
The training labels in NAS-Bench-101 and NAS-Bench-201 are normalized separately.
This can alleviate the problems caused by different performance scales.
Search strategy: Following the convention in neural predictor [55], a large set of architectures in the target search space are randomly sampled and their performance is predicted.
In addition, according to the previous work on DARTS search space [5], we limit the max number of skip connections to 2 .
After that, we choose the architecture with the best-predicted performance as the searched architecture.
4.2 Search Results on ImageNet and CIFAR-
[Table]
The experimental results of ImageNet are shown in Table 1, where the competitors are all the state-of-the-art NAS methods (shown in the second row) and neural predictorassisted NAS methods (shown in the third row).
As can be seen from Table 1, the proposed CDP method only consumed 0.1 GPU
Days to complete the search, and it is one of the least costly search methods.
Speciﬁcally, compared with the chosen neural predictorassisted NAS methods, the search cost of
CDP is 2000 × less than NAO, 40 × less than SemiNAS, and 25 × less than WeakNAS.
The main reason for the extremely low cost of CDP lies in that CDP can make full use of a large amount of existing data from the benchmark datasets.
In addition, although CDP consumes a similar budget to that of PC-DARTS which is the state-of-the-art NAS method using weight sharing, CDP achieves the best top-1 accuracy.
The searched architectures are displayed in Appendix C.
[Table] [Table] [Table] The comparison results of CIFAR-10 are shown in Table 2.
In this experiment, in addition to the methods compared in Table 1, we also compare with three other state-of-the-art methods.
They are
ENAS [42], P-DARTS [5], PVLL-NAS [31], and a predictor-assisted NAS method BRP-NAS [14].
The parameters of all comparison methods are similar.
In terms of the search cost, CDP requires only 0.1 GPU Days, which is one of the lowest cost methods in the table.
In addition, the architecture searched by CDP also outperforms all the NAS methods compared.
4.3 Ablation Study
To the best of our knowledge, CDP is the ﬁrst work of neural predictors designed from the crossdomain view, by directly exploring the architectures of existing NAS benchmark datasets.
Therefore, it is not practical to conduct any experiments on NAS-Bench-101 and NAS-Bench-201 by tradition.
To deal with it, we create a new dataset for neural predictor termed shallow DARTS by randomly sampling 100 shallow architectures from DARTS search space.
In order to save training cost, CIFAR10 is chosen as the dataset, and these architectures have only 8 layers which are shallower than the architectures reported in Table 2.
Moreover, we train these shallow architectures for 50 epochs to further save cost.
After CDP has been trained on NAS-Bench-101 and NAS-Bench-201, it is used to predict the performance of 100 architectures in the shallow DARTS dataset.
Following the conventions, we also pay more attention to the relative ranking between the architectures.
Therefore, we use Kendall’s Tau (KTau) [45] which is widely used to measure the quality of neural predictors [49, 54] in this ablation study.
KTau concerns the relative ranking between the architectures, and its values vary in [ − 1 , 1] .
The larger the value, the more the predicted ranking matches the real ranking.
Table 3 displays the results with and without the proposed progressive subspace partition strategy.
Speciﬁcally, we specify that the numbers of subspaces are all 3 in the end.
‘None’ denotes no progressive strategy is used.
In addition, progressive subspace partition with the linear increasing rate in Equation (9) is also tested to show the effectiveness of the cosine function.
Generally, the performance of the predictor is much better when working with the proposed progressive subspace partition strategy.
Furthermore, the cosine function also works better than the linear increasing rate.
In summary, the ablation study proves the effectiveness of the proposed progressive subspace partition.
The results of the ablation study on the assistant space are reported in Table 4.
The KTau value of the ﬁrst line is obtained with the help of the progressive subspace partition.
On this basis, two assistant spaces are added separately.
The ﬁrst one is similar to DARTS and the number of internal nodes is set as 2, which makes its size be smaller than both the source and target architecture spaces.
The second is the tiny DARTS whose size is between the source and target architecture spaces.
As can be seen from Table 4, the small assistant space fails to improve the performance of the predictor.
But with the help of tiny DARTS (the medium space), the prediction performance can be further improved on the basis of the progressive subspace partition.
Finally, we conduct an ablation study on domain adaptation approaches.
In addition to the MMD and LMMD introduced above, domain-adversarial neural network (DANN) [17] and CORrelation ALignment (CORAL) [48] are also compared.
We ﬁne-tune the hyper-parameters of these approaches (if feasible), and report the best KTau values they achieved in Table 5.
It is obvious that our approach is far better than peer competitors.
To sum up, the ablation studies prove that the proposed progressive subspace partition and assistant space can signiﬁcantly improve the prediction performance of CDP.
Other ablation studies can be found in Appendix E.
Conclusion
This paper aims to develop a cross-domain predictor, named CDP, which does not require training neural architectures in the particular search space.
This goal has been achieved by the two proposed components.
Speciﬁcally, the progressive subspace partition has been proposed to largely alleviate the issue of inaccurate pseudo-label.
In addition, the assistant space has been developed to address the inadaptability caused by large differences between source and target domain.
The training architectures of CDP are collected from the existing NAS benchmark datasets, so it does not have to spend a lot of computation cost on training neural architectures as the existing neural predictors do.
CDP can ﬁnd high-performance architectures in the large search space (such as DARTS) [-with0.
-]with 0.1 GPU Days and achieves the classiﬁcation accuracy of 76.9% and 97.51% on ImageNet and
CIFAR-10, respectively
Acknowledgments
This work was sponsored in part by Zhejiang Lab (NO.2022PG0AB02), in part by CAAI-Huawei
MindSpore Open Fund and in part by the Science and Technology Innovation Talent Project of Sichuan Province under Grant 2021JDRC0001. We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]We followed all the guidelines.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofs are in Appendix B. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provided runnable code in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The training details can be found in Section 4.1, 4.3 and Appendix D, E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We reported the mean value of ﬁve repeated experiments in Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We reported them in Tables 1 and 2. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cited NAS-Bench-101 and NAS-Bench-201.
(b) Did you mention the license of the assets?
[Yes] We mention the license in the README ﬁle in the code of the provided supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The new assets are in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
In the appendix, we ﬁrst provide the details of the uniform encoding strategy.
We then give a detailed proof of the Theorem 2.
In Section C, we illustrate the searched architectures with ﬁgures.
Supplementary experimental details and more ablation studies are provided in Section D and E respectively.
In practice, the encoding strategies of different architecture spaces are required to be uniﬁed for implementing the cross-domain predictor. Taking NAS-Bench-101, NAS-Bench-201, and DARTS as examples, although these three search spaces all use cells to make up architectures, the representation methods of architectures in different search spaces vary greatly. In addition, the original encoding strategies for architectures in different search spaces are also quite different, so the progressive subspace adaptation cannot be carried out directly.
To solve this issue, we propose a uniform encoding strategy that consists of four main steps.
These steps unify the encoding from four aspects, including the number of input, operation location, mapping of operations, and cell types.
After these four main steps, all the cells in NAS-Bench-201 and DARTS can be converted to the formulation of NAS-Bench-101.
And then, we will introduce them in detail next.
[Figure]
The number of inputs: We ﬁrstly separate the cell having two inputs in DARTS into two cells with a single input of each.
An example is given based on Fig.
4 (c), and we will introduce how to separate it into two cells in detail.
Supposing that the ﬁrst cell contains ‘In_1’, then the operations only connected to node ‘In_2’ need to be deleted in the ﬁrst cell.
A similar process will be carried out in the second cell.
This step uniﬁes the number of ‘input’ nodes in different cells and ensures that there is only one input in each cell.
At the same time, this step can reduce the number of operations in cells of DARTS and make it more similar to the smaller cells in
NAS-Bench-101 and NAS-Bench-201.
Operation location: Another obvious difference is the operation location.
According to
Fig.
4, the operations are represented by vertices in NAS-Bench-101, while are represented by edges in the remaining two search spaces.
In addition, the skip connection and the zeroize operation will not appear in the form of NASBench-101 because they are both implicitly represented by the edges, i.e. , the connections.
This step is to convert the cells to the formulation of NAS-Bench-101 because it can be more intuitively converted into adjacency matrix M and operation feature O 0 which are the inputs of CDP.
Because the conversion methods of NAS-Bench-201 and DARTS are almost the same, we only introduce how to convert the cell in NAS-Bench-201 to the form of NAS-Bench-101.
The vertices in cells of
NAS-Bench-201 are densely connected since the vertices in the back may receive the connection from all the vertices in the front.
Then we specify an order for these six edges.
After that, the template adjacency matrix M t , which should be the same in all cells, can be obtained by regarding the edges as vertices.
Finally, the adjacency matrix is pruned according to skip connection and zeroize operation.
Mapping of operations: One intractable issue is how to encode the feature of operations caused by the various types of operations.
We hope to map the operation types in the target architecture space and the source architecture space as much as possible and as accurately as possible, and try to manually divide similar operations into one category based on their characteristics.
Firstly, there are two types of pooling operations, namely max pooling and average pooling.
We make their feature encoding the same because of their similar functions.
Secondly, there are separable convolutions and dilated separable convolutions with kernel sizes of 3 and 5 in DARTS.
Fortunately, 3 × 3 convolutions are utilized in both NAS-Bench-101 and NAS-Bench-201.
According to the above principle, the convolutions with the same kernel size are classiﬁed into one category.
However, we cannot ﬁnd any suitable operation corresponding to the 5 × 5 convolutions in the source architecture space.
One solution is to map the 5 × 5 convolution to the 1 × 1 convolution which is also isolated.
But their roles in the neural network are too far apart, so we do not choose this solution.
Finally, we deﬁne these operations as unknown type, and the ablation study provided in the manuscript proves the beneﬁts of doing so.
Cell types:
There are two types of cells in DARTS, namely normal cell and reduction cell.
In the end, one architecture in DARTS will be represented by totally 4 cells multiple by 2 cells which are separated from the ﬁrst step.
The weighted sum of these four cell scores is used as the performance of the architecture.
We use the weighted sum score of these four cells as the performance value of the architecture.
Considering the contribution of different types of cells to the entire architecture, four neural predictors are trained with different datasets split from the whole training datasets of NASBench-101 and NAS-Bench-201.
The size of the training dataset is proportional to the proportion of the cell type predicted by the neural predictor in the architecture.
After the above four main steps, all the cells in NAS-Bench-201 and DARTS can be converted to the form of NAS-Bench-101.
The last step is to pad zeros to ensure the sizes of all the adjacency matrices are the same.
Following the conventions in [57], the positions of the zero-padding are in the second to last row and second to last column.
In addition, we use one-hot encoding to encode the operations, namely input , output , convolution 1 × 1 , convolution 3 × 3 , pooling , and zeroize .
And the unknown operation is encoded with zeros.
B Proofs B
Proof of Lemma 1.
Lemma 1.
Let (cid:15) T ( h ) and (cid:15) S ( h ) be the expected error on target and source domain and H be a hypothesis space, for h ∈ H :
[Equation]
where λ = (cid:15) T ( h ∗ )
+ (cid:15) S ( h ∗ ) is the combined error of ideal hypothesis h ∗ =
arg min h ∈H ( (cid:15) T ( h )
+ (cid:15) S ( h )) on both domains.
Proof.
We ﬁrst introduce the A -distance which can measure the difference between probability distributions.
The formulation is shown as follows:
[Equation]
We follow the conventions in domain adaptation to assume a binary classiﬁcation scenario, where
Y = { 0 , 1 } .
For a binary hypothesis space H , we use d H to indicate the A -distance.
Let Z h
= { z ∈ Z : h ( z ) = 1 } represents the set of latent representations in the latent space Z that are classiﬁed as category 1 by h , h ∗ = arg min h ∈H ( (cid:15) T ( h )
+ (cid:15) S ( h )) , and we have the following inequalities:
[Equation]
[Equation]
(17) where ∆ is the XOR operation.
The reason why inequality (16) holds is as follows.
The ﬁrst term after the inequality sign (cid:15) T ( h ∗ ) is the error rate of h ∗ , including the error when the hypotheses of h, h ∗ are in agreement, and the second term is the error probability when the hypotheses of h, h ∗ are in disagreement.
Thus, the expected error on the target domain is less than the sum of these two terms.
The reason why inequality (17) holds as follows.
The ﬁrst term Pr D S
[ Z h ∆ Z h ∗ ] denotes the hypotheses of h, h ∗ are in disagreement, i.e. , h ( z ) ⊕ h ∗ ( z )
= 1 , ⊕ : XOR .
At this time, only one of the hypotheses is wrong, so it is less than the sum of the errors of h and h ∗ .
Let λ =
(cid:15) T ( h ∗ )
+ (cid:15) S ( h ∗ ) , and we can have:
[Equation]
B.2 Proof of Theorem
Theorem 2.
Let d (cid:48) be the VC-dimension of H (cid:48) , m be the size of (cid:101) U S,valid , m (cid:48) be the size of unlabeled samples (cid:101) U S and (cid:101) U T .
With probability of 1 − δ , for h ∈ H (cid:48) :
[Equation]
Proof.
Firstly, we found an upper bound represented by empirical error on source validation dataset ˆ (cid:15) S,valid ( h ) for the expected error on source domain (cid:15) S ( h ) in Equation (14).
Following the proof in [32], let κ i =
(cid:15) S ( h ) − L ( h ( z i ) , y i ) for h ∈ H and z i ∈ (cid:101) U S,valid , and m is the size of (cid:101) U S,valid .
We have:
[Equation]
As 0 ≤ (cid:15) S ( h ) ≤ 1 and 0 ≤ L ( h ( z i ) , y i ) ≤ 1 , we can derive the boundary − 1 ≤ κ i ≤ 1 and E [ κ i ( h ) 2 ] ≤ 1 , | κ i | ≤ 1 .
By using Bernstein inequality, we have:
[Equation]
Considering union bound over all h ∈ H , Equation (21) can be derived from Equation (20) with
VC-dimension d (cid:48) of H :
[Equation]
(cid:33)
[Equation]
Let δ = m d (cid:48) exp (cid:16) − ξ 2 m 2(1+ ξ 3 ) (cid:17) and solve the Equation (21) for ξ :
[Equation]
Because ξ ≥ 0 and √ a + b ≤ √ a + √ b
[Equation]
we can simplify the Equation (22) as:
[Equation]
Therefore
[Equation]
with a probability of at least 1 − δ
[Equation]
for h ∈ H :
[Equation]
Furthermore, by enlarging the expected domain distance to its upper bound with the empirical domain distance measured by ﬁnite samples according to [3], we have:
[Equation]
Secondly
[Equation]
we found an upper bound represented by MMD for the distance ˆ d H ( (cid:101) U S , (cid:101) U T ) in Equation (25).
Following the assumption in [35]
[Equation]
we also choose Parzen window classiﬁer [47] as h
[Equation]
and the empirical distance can be bounded by:
[Equation]
where L ( · ) is the loss function of Parzen window classiﬁer.
Finally, by applying the bound between ˆ d H ( (cid:101) U S , (cid:101) U T ) and d k ( (cid:101) D S , (cid:101) D T ) on Equation (25), we can have
Equation (18)
[Figure]
C Searched Architectures
We use four ﬁgures to display the searched architectures.
Speciﬁcally, the searched normal cell and reduction cell on ImageNet are shown in Fig.
While the searched cells on CIFAR-10 are shown in
Fig.
[Figure]
D Supplementary Experimental Details
This section is a supplement to the experimental details in the manuscript.
[Figure]
Settings of GCN: GCN is set up to fully utilize GPU memory and computing power.
Speciﬁcally, the training batch size is set as 1K and the learning rate is 2e-3.
In addition, because the neural architectures in multiple spaces are complex, we set the number of hidden layers as 5.
The other parameters remain the same as suggested in [55].
Training settings of the searched architectures: Based on the convention, we also follow the positive training methods suggested in [2] to improve the classiﬁcation accuracy on ImageNet.
Speciﬁcally, the total training epoch is set as 350.
The cosine learning rate with the decay 4e-5 is used as the optimizer.
Regularization methods such as labeling smoothing [51] and AutoAugment [9] are also adopted.
In addition, the squeeze-and-excitation module [24] is attached after each cell.
Furthermore, the training batch size on ImageNet is set as 512, and the learning rate is 0.2.
As for the training details on CIFAR-10, we follow the conventions of the previous works [33, 58].
E More Ablation Studies
This section is a supplement to the ablation study of the manuscript.
4 C 0.
[Figure]
The numbers of categories: We ﬁrst conduct an ablation study on the numbers of categories C in
LMMD, and the results are shown in Fig.
As discussed in Subsection 3.2, when C is equal to 1 , LMMD is mathematically equivalent to MMD.
Because of the generalizability of MMD, not surprisingly, the predictor has been improved.
Then we increase C to display the effect of subspace adaptation.
As can be seen from Fig.
7, LMMD works better than MMD when C is set as 2.
However, when C is increased to 3, the prediction performance does not improve but exacerbates.
Further increasing C to 4 results in continued deterioration of performance.
This phenomenon may be caused by the incorrect pseudo-labels discussed in Subsection 3.2.
The hyper-parameter K of Equation (9): We decide the maximum number of subspaces K by an ablation study.
Table 6 shows increasing K from 2 to 3, the performance is improved.
But increasing
K to 4, the performance is worse.
This is because when K increases, the probability that the predicted
[Table]
pseudo-label is wrong will also increase.
In addition, when K increases, the search cost will also increase.
So we set K to 3 .
[Table]
The impact of kernel φ : We have tried three different φ , i.e. , Rational Quadratic Kernel (RQK), Laplace kernel, and Gaussian kernel.
As can be seen from Table 7, the
Gaussian kernel is signiﬁcantly better than the other two in terms of KTau.
And this is why we use it.
Other improvements: As shown in Table 8, we divided the improvements into two parts.
The ﬁrst part is the baseline where no improvements are applied.
The second part includes three tricks.
The ﬁrst trick is to normalize the performance values in NAS-Bench-101 and NAS-Bench-201 separately.
A great improvement is obtained in terms of KTau value.
On this basis, we continue to ﬁne-tune the penalty parameter θ in Equation (4) of the manuscript.
In the end, θ = 2 1+ exp {− 10 eE } − 1 after ﬁne-tuning, and a smaller improvement can be observed.
On the basis of these two tricks, we use the unknown operation to represent 5 × 5 convolution which is explored in the uniform encoding strategy.
As can be seen from Table 8, this trick also brings positive effects.
Therefore, we used these three tricks throughout the experiments introduced in the manuscript.
[Table]