Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Neural Architecture Search (NAS) can automatically design promising neuralarchitectures without artiﬁcial experience.
Though it achieves great success, prohibitively high search cost is required to ﬁnd a high-performance architecture,which blocks its practical implementation.
Neural predictor can directly evaluatethe performance of neural networks based on their architectures and thereby savemuch budget.
However, existing neural predictors require substantial annotatedarchitectures trained from scratch, which still consume many computational resources.
To solve this issue, we propose a Cross-Domain Predictor (CDP), whichis trained based on the existing NAS benchmark datasets ( e.g. , NAS-Bench-101),but can be used to ﬁnd high-performance architectures in large-scale search spaces.
Particularly, we propose a progressive subspace adaptation strategy to address thedomain discrepancy between the source architecture space and the target space.
Considering the large difference between two architecture spaces, an assistant spaceis developed to smooth the transfer process.
Compared with existing NAS methods,the proposed CDP is much more efﬁcient.
For example, CDP only requires thesearch cost of 0.1 GPU Days to ﬁnd architectures with 76.9% top-1 accuracy on
ImageNet and 97.51% on CIFAR-10.
Introduction
Neural Architecture Search (NAS) [16] can automatically design promising architectures of deepneural networks for real-world applications, such as image classiﬁcation, language modeling andmedical image segmentation [48, 44, 60].
In recent years, there have been many architecturessearched by NAS methods, which can surpass the performance of those manually designed [41, 23].
Generally, performing NAS methods often requires thousands of GPU Days.
Unfortunately, such alarge amount of computing resources is not necessarily available to most researchers and enterprisesinterested.
As a result, NAS researchers are paying increasing attention to improve the efﬁciency of
NAS through developing effective acceleration methods, to complete the search within an acceptablecost [33].
Generally speaking, most of the existing search acceleration methods are developed based on someheuristic strategies, e.g. , reducing the size of search space [63, 15], searching upon a smaller proxydataset [41, 32], and sharing the parameters of different architectures (weight sharing)
However, the shortcomings of the above methods are obvious.
Reducing the size of search spacealso limits the diversity of architectures [50], and the architectures optimized on the proxy datasetare not guaranteed to perform best on the target dataset [4].
Weight sharing ﬁrst trains a supergraphthat is manually designed in advance and then searches subgraphs as the potential architectures inthe supergraph.
The weights in the same components between different architectures are sharedfrom the trained supergraph without further training, thus the efﬁciency of performing of NAS is greatly improved.
However, the weight sharing also suffers from performance instability and dramaticperformance collapse [8, 6], which severely limits its application.
Neural predictor is a promising acceleration method, which directly estimates the performance ofneural architectures without any training of the neural network during NAS [53].
For example,
Sun et al.
[47] proposed an end-to-end performance predictor that employed random forest [21]which reduced the consumed time of NAS signiﬁcantly.
Tang et al.
[52] used a graph convolutionnetwork (GCN) as the neural predictor and fully exploited the unlabeled architectures to improvethe prediction accuracy.
Though these predictor-based methods tend to be more efﬁcient than othermethods, sufﬁcient architectures which play as the annotated examples to train the predictor, are stillrequired to sample from the target search space.
These sampled architectures are usually required totrain fully to obtain their actual performance [53, 47].
A large search space requires a mass of samples,which becomes a bottleneck for further improving predictor-based methods.
Consequently, neuralpredictor still requires massive computing costs largely due to annotating the sampled architectures.
Fortunately, we note that many benchmark datasets have been released for NAS research, suchas NAS-Bench-101 [58], NAS-Bench-201 [13], NAS-Bench-nlp [28], NAS-Bench-ASR [37], etc.
These datasets contain a great number of architectures with known real performance, and constructingthem has consumed massive computational resources.
If these data can come in handy, there is noneed to annotate other architectures for predictors and the search cost is greatly reduced.
However, the search spaces of these datasets are small and have a huge difference from the largesearch spaces adopted by popular NAS algorithms such as DARTS [32] and ProxylessNAS [4].
Although the architectures are all made up of operations, the settings are quite different, such as thetypes and the maximum number of operations.
As a result, these well-trained benchmark architecturescannot be directly used to build predictors.
In this paper, we develop a Cross-Domain Predictor (dubbed as CDP) by exploiting the existing NASbenchmarks, and then the predictor is implemented to ﬁnd high-performance neural architectures inlarge search spaces.
The contributions of CDP are summarized as follows: 1) We divide the architecture space for subspaceadaptation, and propose a progressive subspace partition to accurately exploit the local information.2)
We design an assistant space, to achieve smoother transfer from the source space to the targetone because of the signiﬁcant differences between searching domains.
3) We experimentally showthat CDP can search promising architectures with only 0.1 GPU days on ImageNet and CIFAR-10,achieving 76.9% and 97.51% classiﬁcation accuracy, respectively.
Related Works
2.1 NAS and Neural Predictor
[Figure]
In general, there are three important parts in
NAS [16], namely architecture space 1 , searchstrategy, and performance evaluation.
Duringthe search process, each NAS method ﬁrst speciﬁes the architecture space that includes all thepotential architectures to be searched.
Secondly,a particular search strategy is utilized to perform the search.
In practice, reinforcementlearning [25], evolutionary computation [1],and gradient-based methods are the mainstreamsearch strategies widely used.
Finally, the performance evaluation assesses the architecturessearched to guide the search strategy.
In recent years, how to accelerate the performance evaluation is one of the most valuable researchdirections in NAS [33].
As a promising acceleration method, neural predictors have received greatattention from researchers since 2017 [11].
The red and black parts in Fig.
1 illustrate a typical neuralpredictor-assisted NAS for a glance.
Recently, with the design of new predictors [47, 52, 53], the prediction performance of neural predictors has been continuously improved.
However, existingpredictors request the source and the target architecture spaces to be the same.
If the predictormakes predictions for a speciﬁc NAS method, it must collect sufﬁcient labeled architectures from thesearch space of the corresponding NAS method in advance.
The CDP algorithm we proposed in thispaper gets rid of the constraint that the source and the target architecture spaces must be the same.
Particularly, CDP directly utilizes the existing benchmark datasets as the training architectures in thesource space, yet to predict the architectures in other target spaces.
In this way, the neural predictorwill not need to sample labeled data from the target space of the particular NAS method, thus greatlyreducing the cost of annotating the training architectures.
2.2 Transfer Learning and Domain Adaptation
Transfer learning is to apply knowledge learned in previous domains to other domains, thus reducingthe need and effort to recollect labeled data [39].
Speciﬁcally, domain adaptation is a subcategory oftransfer learning, where the source and target tasks are the same and labeled data is available in thesource domain, which is the scenario of CDP.
According to [62], transfer learning can be classiﬁedinto two different categories based on the consistency between the source and the target domains.
If the feature spaces are different in both domains, the scenario is termed heterogeneous transferlearning.
Otherwise, it is homogeneous transfer learning.
The proposed CDP algorithm in this paperfollows heterogeneous transfer learning because the architecture spaces are different.
But observingthat much more current literature addresses the scenario of homogeneous transfer learning [10], weproposed a uniform encoding strategy that uniﬁes the architecture spaces, so that it can be treated ashomogeneous transfer learning.
The feature-based transfer is one of the most prevalent methods.
It aims at reducing the distributiondifference between the source domain and the target domain with the help of some instances.
Acommon metric for measuring the difference is Maximum Mean Discrepancy (MMD) [18], whichcan quantify the difference by calculating the average distance of the instances sampled from bothdomains [62].
Inspired by MMD, many variants and improved approaches are proposed such as
MK-MMD [19], weighted MMD [57], and Local Maximum Mean Discrepancy (LMMD) [61].
Among them, LMMD can take advantage of local afﬁnity for more delicate adaptation and canachieve extraordinary results.
Speciﬁcally, LMMD ﬁrst classiﬁes the domain into several subspacesand then aligns the subspaces of the same category in pairs.
Secondly, LMMD uses the pseudo-labelsoutput by the neural network to classify the target data throughout the training process.
However, it iswell known that the neural network cannot achieve satisfactory results in the initial training stage,that is, the pseudo-label obtained through it may be seriously inconsistent with the real label in theearly training process.
In this paper, we propose a progressive approach to alleviate the disadvantagesof misclassiﬁcation.
Approach
In this section, we will document how the proposed cross-domain predictor bridges the gap betweendifferent architecture spaces.
The overall process of the proposed CDP algorithm is shown in Fig.
3.1 Formulation
Supposing the architectures in the search space X are X
= { x 1 , x 2 , ..., x n } , and their respectiveperformances in space Y are Y =
{ y 1 , y 2 , ..., y n } , the predictor P : X → Y , which is trained topredict the performance of architecture, can be formulized by Equation (1):
[Equation]
where W denotes the trainable weight of predictor P , and L is the loss function.
After the training, thepredictor with trained weight W will evaluate other architectures, say x i where i is not in { 1 , 2 , ..., n } ,via Equation (2):
[Equation]
Mathematically, neural architectures can be presented as directed graphs, and a common encodingscheme of architecture x is to use feature vectors of operations O 0 and adjacency matrix M , which
LMMD
FC layers
[Figure] [Figure]
are further collectively employed to represent the type of each operation and the connection betweenoperations.
GCN [27] is a prevalent technique tackling non-Euclidean data such as graphs and hasachieved great success in representing neural architectures [53].
Therefore, it is used as the ﬁrstcomponent in P to extract features of neural architecture G : X → Z , where Z is the feature space.
In this paper, we follow the convention and choose the GCN version implemented in [53] as theneural predictor, and the feature vectors are updated by Equation (3):
[Equation]
(3) where M T is the transposed matrix of M and ReLU is an activation function.
W + l and W − l aretwo trainable weights of the l -th layer in GCN.
Speciﬁcally, the last two terms of Equation (3) allowthe graph information to ﬂow forward and backward, respectively, which makes it more suitable fordealing with directed graphs like neural architectures.
In addition, the second component of P is ahypothesis h : Z → Y , consisting of several fully connected layers, and is tailed to the GCN forobtaining the predicted performance values.
As for loss function L , we also follow the conventionsuggested in the reference [53] using mean square error.
3.2 Progressive Subspace Adaptation
Although neural predictor can largely save search cost for NAS, the limitation of itself is also serious:it cannot be well trained due to the lack of available labeled architectures of the corresponding NASalgorithm.
Observing that many architecture datasets have been released, these data can be potentiallyused to train the neural predictors.
However, the distribution of these massive labeled architectures issigniﬁcantly different from that of the architectures from the popular search space that is often large.
In principle, domain adaptation is useful for reducing the distribution gap between these architecturespaces.
Let (cid:101) D S and (cid:101)
D T be the source and target distributions over Z .
We will speciﬁcally introducehow to bridge the gap between (cid:101) D S and (cid:101) D T in the proposed method as below.
Research has shown that deep neural networks have the ability to learn the transferable featureswell [59, 38].
In addition, many methods [34, 61] add metrics of discrepancy to the objective functionto jointly optimize the neural network.
Based on Equation (1), the term of adaptation regularizerd ( · , · ) is added, and the objective function becomes:
[Equation]
where x sn and y sn denote architectures with labels in source search space X S , and θ > 0 is a penaltyparameter.
Maximum Mean Discrepancy (MMD)
[18] is a classic metric and has been widely used in domainadaptation.
Let H k denote the reproducing kernel Hilbert space (RKHS) endowed with a characteristickernel k , and (cid:101) U S , (cid:101) U T are samples of sizes n s , n t drawn from (cid:101) D S , (cid:101) D T respectively.
The regularizerterm d ( · , · ) can be implemented by MMD, and the unbiased estimator of MMD is deﬁned by latentrepresentation z s , z t :
[Equation]
where φ denotes feature map function mapping original space to H k and it is associated with k ,k
( z s , z t ) =
(cid:104) φ ( z s ) , φ ( z t ) (cid:105) where (cid:104)· , ·(cid:105) means the inner product operation.
Considering that the differences between search spaces are quite large, the gap cannot be handledwell if only MMD is used.
In addition, MMD is a global domain adaptation, which often ignoresthe local relationship.
In order to take full use of the local information between architectures, wepresent a progressive subspace adaptation strategy.
First, the source space and the target space aredivided into several subspaces whose number will increase during the training process.
Secondly,
MMD loss is used between corresponding subspaces for adaptation.
Speciﬁcally, these two spaceswill be divided according to the performance Y =
{ y 1 , y 2 , ..., y n } of architectures.
Between thesetwo spaces, the architectures with good performance may share some characteristics, such as longshortcut connections and preferred operations.
We directly use the true labels y of architectures in the sources space and predict the pseudo-labelsˆ y of architectures in the target space to divide the subspaces.
The performance value y
i is a scalar,which needs to be classiﬁed ﬁrst to become a classiﬁcation label y
i that is a vector.
Secondly, theweight w ci which presents the x i belonging relationship to the c -th subspace can be calculated via
Equation (6):
[Equation]
where y ci denotes the c -th element of label vector y i .
With the weight w ci and Equation (5), theunbiased estimator of the discrepancy between subspaces becomes Equation (7):
[Equation]
This discrepancy is termed Local Maximum Mean Discrepancy (LMMD) [61] that takes the correlation from the same category of subspaces into consideration.
With the help of the kernel k , Equation (7) can be further modiﬁed to obtain the following formulafor easy calculation of LMMD:
[Equation]
j = w sci w tcj k
( z si , z tj )] .
However, there is a severe issue that the pseudo-label ˆ y may be incorrect, especially when the neuralnetwork is not fully trained.
This will further affect the accuracy of ˆ d k ( (cid:101) D S , (cid:101) D T ) in Equation (8).
In order to alleviate the misclassiﬁcation due to inaccurate pseudo-labels, we propose a progressivesubspace partition strategy.
All the architectures are partitioned into one space in the early stageof neural network training so that there will be no classiﬁcation errors caused by incorrect pseudolabels.
As the training progresses, the neural network can predict more accurately.
At this time, weslowly increase the number of subspaces.
Finally, the number of subspaces will end at the speciﬁedmaximum K .
Progressive Subspace Partition: Supposing the neural network is speciﬁed to be trained for a totalof E epochs, and the maximum number of subspaces is K , when the training progresses arrive at thee -th epoch, the number of subspace categories C e can be scheduled by Equation (9):
[Equation]
and ∀ i < j, Sche ( i ; E, K ) < Sche ( j ; E, K ) . Based on experience and experiments, in this paperwe let C e = K − (cid:98) cos ( π 2 E e ) ∗ K (cid:99) , where (cid:98)·(cid:99) denotes the round-down operation.
C e calculated bythis schedule meets the characteristics discussed above, i.e. , when e is relatively small, C e is equalto 1 , while e is close to E , C e will be K .
Furthermore, we use the cosine function to increase thenumber of epochs with a small value of C e .
This strategy ensures that the neural network has beensufﬁciently trained before the C e becomes larger.
Therefore, the occurrence of classiﬁcation errorswill be greatly reduced.
After determining the number of subspace categories C e during the e -th training epoch, we can usefractile to generate a new label y for the original label y which is a scalar.
To be more speciﬁc, thefractiles { f 0 , f 1 , f 2 , ..., f C e } of scalar labels need to be found at ﬁrst, where f C e is the largest one inthe labels and f 0 is the lower bound.
Then, we can ﬁnd the corresponding subspace category c fory i if f c − 1 < y i ≤ f c .
After that, we set the c -th element of y i to one and other elements as zeros.
Finally, the classiﬁcation vector y i is used by Equation (6) for the calculation.
Here, we will further analyze the role of the proposed progressive subspace partition strategy.
Whenthe number of subspaces is one, the numerator y ci in Equation (6) is always one, and the denominator 215 (cid:80) y j ∈ Y y cj is equal to the number of source (or target) data.
Hence, w sci is equal to 1 /n s and w tci isequal to 1 /n t .
As a result, the unbiased estimator of LMMD (Equation (7)) becomes that of MMD(Equation (5)) which is a global adaptation.
While the number of subspaces increases to the speciﬁedmaximum number K , the measurement of the discrepancy we used is actually LMMD which focuseson the relationship between the same categories of subspaces to help the subspace adaptation. [Figure]
3.3 Assistant Architecture Space
[Figure]
Considering that the source architecture spaceand target architecture space are quite different,it is difﬁcult to directly transfer the source architecture space to the target space.
To address thisissue, we design an assistant architecture spaceto make the transfer smoother.
Generally, the source space, where all architectures can be trained from scratch to obtain the performance of each architecture, is often relatively small.
However, the target architecture spacesare often large in practice.
An example is given based on two widely used architecture spaces,namely NAS-Bench-101 [58] and DARTS [32].
In terms of the number of possible architectures,
NAS-Bench-101 is 423K and DARTS is about 10 18 .
In terms of the maximum number of operationsin a cell, NAS-Bench-101 is 5 while DARTS is 8.
Thus, the size of the assistant space should bebetween the size of the source space and the target space.
Besides, other features should be similar tothese two architecture spaces.
In this paper, we specify the assistant space more similar to the targetspace.
Speciﬁcally, an assistant space for NAS-Bench-101, NAS-Bench-201 [13], and DARTS are designedfollowing the above principles, dubbed as ‘tiny DARTS’ space.
Fig.
3 is an illustration of this assistantspace, where we use dark black to represent the large architecture space and gray to represent thesmaller one.
The participation of the assistant space makes the adaptation be a continuous process andalleviates the inadaptability caused by large differences.
The only difference between tiny DARTSand DARTS is that the number of intermediate nodes has changed from 4 to 3.
This directly reducesthe number of operations from 8 to 6 which is equal to the situation in NAS-Bench-201.
Exceptfor this change, any other settings, including the types of operations, etc. , have not been changed.
Therefore, the features in DARTS have also been preserved.
Speciﬁcally, there are two input nodes intiny DARTS, and all intermediate nodes are connected to the output node.
In the early training epochs of CDP, the assistant space ( e.g. , tiny DARTS) is used as the targetdomain, i.e. , the input at the bottom in Fig.
In this stage, CDP focuses on bridging the smaller gapbetween the source space and the assistant space.
After several training epochs, the gap betweenthe source space and the target space can be also reduced as the assistant space and target space arevery similar.
Therefore, it is much easier to reduce the gap between the source and target spaces atthis time.
As a result, the space transfer process becomes smoother with the assistant space, whichimproves the prediction performance.
After the determination of the target domain, an encoding strategy is required for different searchspaces.
We propose a uniform strategy with four main steps, where The detailed descriptions areshown in Appendix A.
3.4 Theoretical Analysis
In this section, we give a theoretical analysis of the expected error of target architecture search space.
We ﬁrst introduce Lemma 1 which is a basic theorem in domain adaptation, and then further deducethe target error bound of CDP.
Lemma 1.
[3] Let (cid:15) T ( h ) and (cid:15) S ( h ) be the expected error on target and source domain and H be ahypothesis space, for h ∈ H :
[Equation]
where λ = (cid:15) T ( h ∗ )
+ (cid:15) S ( h ∗ ) is the combined error of ideal hypothesis h ∗ =
arg min h ∈H ( (cid:15) T ( h )
+(cid:15) S ( h )) on both domains and d H ( (cid:101) D S , (cid:101) D T ) is the upper bound of the A -distance
Let (cid:101) U S,train , (cid:101) U S,valid be the training and validation datasets randomly sampled from the distribution
D S respectively, and (cid:101) U S,train ∩ (cid:101) U S,valid = ∅ .
After the predictor is trained on the training dataset(cid:101) U S,train , the validation dataset (cid:101) U S,valid is used to validate it.
The validation hypothesis H (cid:48) is asubset of H , and is only determined by (cid:101) U S,train .
The following theorem provides an upper bound ofexpected target-domain error in terms of the empirical error on (cid:101) U S,valid and MMD d k ( (cid:101) D S , (cid:101) D T ) .
Theorem 2.
Let d (cid:48) be the VC-dimension of H (cid:48) , m be the size of (cid:101) U S,valid , m (cid:48) be the size of unlabeledsamples (cid:101) U S and (cid:101) U T .
With probability of 1 − δ , for h ∈ H (cid:48) : [Equation] T
[Equation]
h
[Equation]
ˆ
[Equation]
,valid
[Equation]
Proof sketch: The proof is mainly divided into two steps.
We ﬁrst use the empirical error on sourcevalidation dataset ˆ (cid:15) S,valid ( h ) to represent the upper bound of expected error on source domain (cid:15) S ( h ) :
[Equation]
The detailed proof can be found in Equations (19)-(24) in Appendix B. Secondly, the upper bound ofthe distance d H ( (cid:101) D S , (cid:101) D T ) is constrained by MMD d k ( (cid:101) D S , (cid:101) D T ) , i.e. : [Equation]
[Equation]
and Equations (25)-(26) in Appendix B show the proof in detail.
Finally, by applying these inequalitiesto Lemma 1, the theorem can be obtained.
Theorem 2 shows that the expected error on the target domain is bound by two terms, the empiricalvalidation error in source domain ˆ (cid:15) S,valid ( h ) and MMD d k ( (cid:101) D S , (cid:101) D T ) which is minimized as one termof the objective function.
This means that when observing the predictor performs well on the sourcevalidation dataset, the expected error on the target domain will also be small.
Experiments
In this section, we will empirically investigate the effectiveness of the proposed method, includingthe experiments conducted on ImageNet [12] and CIFAR-10 [29], and the ablation studies.
All ourexperiments are conducted on NVIDIA RTX 2080Ti GPUs.
4.1 Experimental Settings
Architecture spaces for CDP:
NAS-Bench-101 and NAS-Bench-201 are chosen as the source spacesand DARTS is the target space.
The training labels in NAS-Bench-101 and NAS-Bench-201 arenormalized separately.
This can alleviate the problems caused by different performance scales.
We report the results with the lowest cost if there are multiple architectures searched in the literature of the compared methods.
The symbol ‘—’ denotes that no results are available in original papers.
[Table]
Search strategy: Following the convention in neural predictor [53], a large set of architectures in thetarget search space are randomly sampled and their performance is predicted.
In addition, accordingto the previous work on DARTS search space [5], we limit the max number of skip connectionsto 2 .
After that, we choose the architecture with the best-predicted performance as the searchedarchitecture.
4.2 Search Results on ImageNet and CIFAR-10
[Table]
The experimental results of ImageNet areshown in Table 1, where the competitors areall the state-of-the-art NAS methods (shownin the second row) and neural predictorassisted NAS methods (shown in the thirdrow).
As can be seen from Table 1, the proposed CDP method only consumed 0.1 GPU
Days to complete the search, and it is one ofthe least costly search methods.
Speciﬁcally,compared with the chosen neural predictorassisted NAS methods, the search cost of
CDP is 2000 × less than NAO, 40 × lessthan SemiNAS, and 25 × less than WeakNAS.
The main reason for the extremely lowcost of CDP lies in that CDP can make fulluse of a large amount of existing data fromthe benchmark datasets.
In addition, although CDP consumes a similar budget to that of PC-DARTSwhich is the state-of-the-art NAS method using weight sharing, CDP achieves the best top-1 accuracy.
The searched architectures are displayed in Appendix C.
The comparison results of CIFAR-10 are shown in Table 2.
In this experiment, in addition to themethods compared in Table 1, we also compare with three other state-of-the-art methods.
They are
ENAS [40], P-DARTS [5], PVLL-NAS [30], and a predictor-assisted NAS method BRP-NAS [14].
The parameters of all comparison methods are similar.
In terms of the search cost, CDP requires only0.1 GPU Days, which is one of the lowest cost methods in the table.
In addition, the architecturesearched by CDP also outperforms all the NAS methods compared.
4.3 Ablation Study
To the best of our knowledge, CDP is the ﬁrst work of neural predictors designed from the crossdomain view, by directly exploring the architectures of existing NAS benchmark datasets.
Therefore,it is not practical to conduct any experiments on NAS-Bench-101 and NAS-Bench-201 by tradition.
[Table]
[Table] [Table]
To deal with it, we create a new dataset for neural predictor termed shallow DARTS by randomlysampling 100 shallow architectures from DARTS search space.
In order to save training cost, CIFAR10 is chosen as the dataset, and these architectures have only 8 layers which are shallower than thearchitectures reported in Table 2.
Moreover, we train these shallow architectures for 50 epochs tofurther save cost.
After CDP has been trained on NAS-Bench-101 and NAS-Bench-201, it is used to predict theperformance of 100 architectures in the shallow DARTS dataset.
Following the conventions, we alsopay more attention to the relative ranking between the architectures.
Therefore, we use Kendall’s Tau(KTau)
[43] which is widely used to measure the quality of neural predictors [47, 52] in this ablationstudy.
KTau concerns the relative ranking between the architectures, and its values vary in [ − 1 , 1] .
The larger the value, the more the predicted ranking matches the real ranking.
Table 3 displays the results with and without the proposed progressive subspace partition strategy.
Speciﬁcally, we specify that the numbers of subspaces are all 3 in the end.
‘None’ denotes noprogressive strategy is used.
In addition, progressive subspace partition with the linear increasingrate in Equation (9) is also tested to show the effectiveness of the cosine function.
Generally, theperformance of the predictor is much better when working with the proposed progressive subspacepartition strategy.
Furthermore, the cosine function also works better than the linear increasing rate.
Insummary, the ablation study proves the effectiveness of the proposed progressive subspace partition.
The results of the ablation study on the assistant space are reported in Table 4.
The KTau value of theﬁrst line is obtained with the help of the progressive subspace partition.
On this basis, two assistantspaces are added separately.
The ﬁrst one is similar to DARTS and the number of internal nodes is setas 2, which makes its size be smaller than both the source and target architecture spaces.
The secondis the tiny DARTS whose size is between the source and target architecture spaces.
As can be seenfrom Table 4, the small assistant space fails to improve the performance of the predictor.
But with thehelp of tiny DARTS (the medium space), the prediction performance can be further improved on thebasis of the progressive subspace partition.
Finally, we conduct an ablation study on domain adaptation approaches.
In addition to the MMD and LMMD introduced above, domain-adversarial neural network (DANN) [17] and CORrelation ALignment (CORAL) [46] are also compared.
We ﬁne-tune the hyper-parameters of these approaches (iffeasible), and report the best KTau values they achieved both on whole dataset and Top-5 architecturesin Table 5.
It is obvious that our approach is far better than peer competitors.
To sum up, the ablation studies prove that the proposed progressive subspace partition and assistantspace can signiﬁcantly improve the prediction performance of CDP.
Other ablation studies can befound in Appendix E.
Conclusion
This paper aims to develop a cross-domain predictor, named CDP, which does not require trainingneural architectures in the particular search space.
This goal has been achieved by the two proposedcomponents.
Speciﬁcally, the progressive subspace partition has been proposed to largely alleviatethe issue of inaccurate pseudo-label.
In addition, the assistant space has been developed to addressthe inadaptability caused by large differences between source and target domain.
The trainingarchitectures of CDP are collected from the existing NAS benchmark datasets, so it does not haveto spend a lot of computation cost on training neural architectures as the existing neural predictorsdo.
CDP can ﬁnd high-performance architectures in the large search space (such as DARTS)
with0.
GPU Days and achieves the classiﬁcation accuracy of 76.9% and 97.51% on ImageNet and
CIFAR-10, respectively.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We followed all the guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofs
are in Appendix B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provided
runnable code in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] The training details can be found in Section 4.1, 4.3 and Appendix D, E.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We reported the mean value of ﬁve repeated experiments
in Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
[Table]
of GPUs, internal cluster, or cloud provider)?
[Yes] We reported them in Tables[-
-]
and 2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited NAS-Bench101 and NAS-Bench-201. [Table]
(b) Did you mention the license of the assets?
[Yes] We mention the license in the
[Table]
README ﬁle in the code of the provided supplementary material.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
The new assets are in the supplemental material.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]