VAENAS: S AMPLING M ATTERS IN N EURAL A RCHITECTURE S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural Architecture Search (NAS) aims at automatically ﬁnding neural network architectures within an enormous designed search space.
The search space usually contains billions of network architectures which causes extremely expensive computing costs in searching for the best-performing architecture.
One-shot and gradient-based NAS approaches have recently shown to achieve superior results on various computer vision tasks such as image recognition.
With the weight sharing mechanism, these methods lead to efﬁcient model search.
Despite their success, however, current sampling methods are either ﬁxed or hand-crafted and thus ineffective.
In this paper, we propose a learnable sampling module based on variational auto-encoder (VAE) for neural architecture search (NAS), named as VAENAS , which can be easily embedded into existing weight sharing NAS framework, e.g., one-shot approach and gradient-based approach, and signiﬁcantly improve the performance of searching results.
VAENAS generates a series of competitive results on CIFAR-10 and ImageNet in NasNet-like search space.
Moreover, combined with one-shot approach, our method achieves a new state-of-the-art result for ImageNet classiﬁcation model under 400M FLOPs with 77.4% in ShufﬂeNetlike search space.
Finally, we conduct a thorough analysis of VAENAS on NASbench-101 dataset, which demonstrates the effectiveness of our proposed methods.
I NTRODUCTION
Deep neural networks have greatly pushed the frontier of various inﬂuential applications by designing novel neural architectures (Krizhevsky et al.
; Goodfellow et al.
; He et al.
Automatic model design of neural network architectures without human intervention, known as neural architecture search (NAS) , has drawn much attention of the community recently.
It has resulted in state-of-the-art performance in the domain of image recognition (Zoph et al.
; Real et al.
(2019)), object detection (Ghiasi et al.
; Chen et al.
and semantic segmentation (Liu et al.
Generally, the magnitude of search space for NAS tasks is enormous.
For example, NasNet (Zoph et al., 2018) presents a search space with 6 × 10 9 possible cells.
Searching on such huge designed space cost 2400 GPU days.
Weight sharing mechanism has shown to be a promising avenue for efﬁcient NAS.
Latest algorithms on efﬁcient NAS fall into two categories: one-shot approaches (Bender et al.
and gradient-based approaches (Liu et al.
(2018b)).
In one-shot approaches, prior works focus on adopting a ﬁxed sampling strategy (Guo et al.
; Bender et al.
Chu et al.
In gradient-based approaches, search is typically performed without sampling procedure (Liu et al.
(2018b);
Xie et al.
or hand-crafted sampling (Liu et al.
(2018a)).
Despite the success of these NAS methods on various benchmarks, however, these sampling approaches do not interactively learn the architecture distribution as the search process goes along, which makes the sampling procedure ineffective.
In this paper, we propose a learnable and interactive architecture sampling module based on VAE for
NAS, named as VAENAS .
There are two advantages of VAENAS: 1) it learns the good-performing architecture distribution which could be used to reduce search space.
2) it can be embedded into existing NAS framework and improve the performance of current NAS methods.
After developing the generic VAENAS approach, we study in detail the application of VAENAS module via two mainstream NAS approaches: one-shot approach (Brock et al.
; Guo et al.
and gradient-based approach (Liu et al.
(2018b)
; Cai et al.
(2018)), both of which have obtained state-of-the-art performance on neural architecture search tasks.
We show how VAENAS module can be embedded in both approaches and make performance improvement consistently.
We validate VAENAS with various search space on two benchmark datasets (CIFAR-10 and ImageNet) for image recognition.
On NASNET-like search space, we apply VAENAS on both gradientbased methods and one-shot methods.
Speciﬁcally, combined with one-shot approach, VAENAS achieves 2.26% test error on CIFAR-10 and 75.8% accuracy on ImageNet, outperforming state-ofthe-art NAS methods.
On a Shufﬂenet-like search space, VAENAS combined with one-shot achieves 77.4% top-1 accuracy with 365M FLOPs on ImageNet classiﬁcation, outperforming state-of-the-art
Efﬁcient-B0 by 1.1% with 6.5% less computational complexity.
Finally, we perform a thorough analysis of VAENAS on NAS-101 benchmark, to show the effectiveness of our proposed architecture sampling methods.
R ELATED W ORK
Neural Architecture Search.
Recently the design of efﬁcient neural networks has largely shifted from leveraging human knowledge to automatic methods, which is known as neural architecture search(NAS).
Early NAS methods adopt reinforcement learning (RL)(Zoph et al., 2018) or evolutionary strategy (Real et al., 2019) to search on thousands of individually evaluated networks, which is computational-consuming.
Recent works focus on efﬁcient search methods, which falls into two categories: one-shot approach (Guo et al.
; Chu et al.
and gradient-based approach (Liu et al.
(2018b)
; Cai et al.
, both achieve state-of-the-art results on a series of benchmark dataset (Chen et al.
; Ghiasi et al.
Liu et al.
with various search space.
We argue that sampling method matters in NAS and a learnable sampling algorithm can signiﬁcantly improve the NAS performance.
Sampling.
Some prior works have applied sampling methods to NAS framework.
For one-shot approach, Bender et al.
(2018) randomly zero out a subset of the operations during super-network training.
Guo et al.
; Stamoulis et al.
(2019) adopt uniform sampling and Chu et al.
(2019) employ fair uniform sampling to reduce training bias in one-shot models.
For gradient-based approach,
Cai et al.
(2018) propose a multinomial distribution sampling to alleviate large memory consumption issue in conventional gradient NAS methods (Liu et al.
(2018b)).
Our approach presents a new perspective of sampling methods of NAS application.
VAENAS 3. M OTIVATION
Many prior works in NAS have discussed modeling the architecture distribution Γ( α ) .
The typical Γ( α ) is to adopt either ﬁxed architecture distribution (e.g., uniform distribution Bender et al.
Guo et al.
(2019)), or hand-crafted architecture distribution (e.g., Gumbel softmax distribution in
Liu et al. (2018b); Xie et al.
Conceptually, it is expected that the architecture distribution Γ( α ) can interactively learn from the weight sharing super-network as the searching process goes along.
To this end, we propose a learnable architecture distribution, where the distribution Γ t ( α ) at time t is learned from the new incoming architecture α and prior architecture distribution at t − 1 time is Γ t − 1 ( α ) , we can formalize the distribution learning process as:
[Equation]
where Metric denotes the predeﬁned metric, e.g., accuracy or loss, and T denotes the stochastic process.
In equation 1, the distribution of architectures at time t is learned by both the prior knowledge on architecture distribution Γ( α ) and new architectures data α at t − 1 time, as in Figure 1.
As the architecture search proceeds, we randomly sample multiple architectures, rank these architectures according to the predeﬁned metric.
The T incrementally reﬁnes the distribution of architectures by continually learning the good-performing architectures from the weight sharing network.
Alternatively, the weight sharing network updates the architectures weights, where the architectures is sampled from T .
There are many methods that can be used to model this distribution, and we choose VAE due to its training stability as well as the convenience to be embedded into the existing
NAS frameworks.
3. T HE USAGE OF VAE
VAE approximates the marginal likelihood of architecture distribution log p θ ( α ) by maximizing variational lower bound:
[Equation]
Speciﬁcally, the left term is the Kullback-Leibler divergence between the approximated posterior (recognition model) q φ ( z | α ) and the prior p ( z ) .
And the right term is the expectation of reconstruction loss where p θ ( α | z ) denotes variational inference.
As a result, to sample α from the marginal distribution p θ ( α ) , VAE samples latent variable z according to the prior distribution p θ ( z ) and feed z into the inference model p θ ( α | z ) .
Commonly, the prior distribution p θ ( z ) can be set as standard
Gaussian distribution N (0 , I ) .
Let α = { o i } ni be a sequence of operation
o i representing an architecture string, where n is the total number of nodes in a network or cell.
We use an encoder E to take α as input and maps it into parameters µ α and σ α corresponding to a normal distribution N ( µ α , σ α ) .
Symmetrically, a decoder D is used to map z e to a reconstructed architecture ¯ α , where z e ∼ N ( µ e , σ e ) .
The encoder
E corresponds q φ ( z | α ) and the decoder D corresponds p θ ( α | z ) .
Speciﬁcally, the encoder E and the decoder D are implemented by two LSTM networks.
VAENAS FOR N EURAL A RCHITECTURE S EARCH
In this section, we demonstrate how VAENAS is integrated into existing NAS frameworks.
Section 4.1 gives the formal deﬁnition of weight sharing mechanism.
Section 4.2 gives detailed description of VAENAS combined with one-shot framework.
Section 4.3 shows VAENAS combined with gradient-based framework.
W EIGHT -S HARING
M ECHANISM
Weight sharing approaches construct a super-network G ( α , W ) that contains all the valid architectures α with shared weights W .
Any architecture α is a sub-graph in the super-network and inherits corresponding weights W α .
Thus we can formalize the training of weight sharing mechanism as solving a bi-level optimization problem:
[Equation]
where Γ( α ) is an architecture distribution.
There are two main frameworks which utilize weightsharing mechanism: one-shot approach and gradient-based approach.
4. O NE - SHOT F RAMEWORK
Review of One-Shot Architecture Search.
Generally, one-shot approaches (Liu et al.
(2018b)
; Cai et al.
; Xie et al.
(2018)) consist of three steps: 1) train a weight sharing super-network.
2) rank architectures based on its performance on the super-network.
3) derive the ﬁnal architecture via a search strategy.
Previous methods typically adopt hand-crafted sampling methods (Bender et al., 2018; Guo et al., 2019; Chu et al., 2019).
Combine One-Shot Approach with VAENAS.
Algorithm 2 shows the detailed pipeline of VAENAS combined with one-shot approach.
VAENAS-OS utilizes one-shot framework and optimizes sampling distribution:
[Equation] [Equation]
where U ( α ) is random uniform distribution, p θ ( α | z ) is variational distribution, and (cid:15) is exploitation parameter.
On supernet training, architectures are sampled from p θ ( α | z ) and U ( α ) with probability
(cid:15) and 1 − (cid:15) .
As the search phase continues, the (cid:15) increases, thus the super-network trains more samples from VAENAS.
The good-performing architectures, for example, with k % highest accuracy, are collected every n vae epochs and fed into the VAENAS module to train.
As the training phase proceeds, the variational distribution p θ ( α | z ) would converge to generating good-performing architectures.
On search phase, the initial seeds are sampled from p θ ( α | z ) and then search with genetic algorithm as Guo et al.
Due to the weights of good-performing architectures are trained more adequately, intuitively, the super-network would be more predictive.
Algorithm 2 shows the detailed pipeline of VAENAS combined with the one-shot approach.
Why Architecture-Dependent Sampling for One-Shot?
The conventional sampling methods for one-shot NAS, such as zero out random operations (Bender et al.
(2018)), uniform sampling (Guo et al.
and fair sampling (Chu et al.
(2019)) are ﬁxed sampling methods.
One issue of ﬁxed architecture distribution in one-shot approach is its ineffectiveness to sample good-performing architectures, both during super-network training and architecture search.
When train a super-network with architectures sampled from a pre-deﬁned architecture distribution such as uniform distribution, it is likely that the best-performing architectures are not picked; thus the search algorithm fails to seek them out due to the low performance of best architectures on the super-network.
Moreover, the search strategy usually adopts evolution/RL algorithms, where a set of architectures are selected as seeds via sampling method.
Due to the evaluation process is expensive, only a small set of architectures are evaluated and ranked.
Thereby the initialized set for search strategy can be sub-optimal.
VAENAS resolve above issues by adopting an architecturedependent distribution on super-network training and search phase.
By learning the distribution of good-performing architectures, VAENAS ensures the sampling procedure effective since the samples from the variational distribution p θ ( α | z ) are guaranteed to have good performance on the supernetwork.
4. G RADIENT - BASED F RAMEWORK
Review of Gradient-Based Architecture Search.
Gradient-based approaches (Liu et al.
(2018b);
Cai et al.
; Xie et al.
(2018)) employ an independent discrete architecture parameter to model the networks, thus we can follow the exact bi-level optimization process as we describe in equation 3 and 4.
The common practice of gradient-based approach does not contain sampling process.
Proxylessnas (Cai et al.
(2018)) introduce a multinomial distribution sampling process prior to the optimization step of architecture weights.
Combine Gradient-Based Approach With VAENAS.
Alternatively, we consider replacing the sampling procedure by VAENAS:
[Equation]
[Table]
Speciﬁcally [Table] we sample an architecture α i from p θ ( α | z )
[Table]
and update super-network weights W α i and multinomial distribution parameters
x i accordingly.
Algorithm 3 shows the detailed pipeline.
Why Independent Sampling Module Necessary?
Conventional gradient-based methods (Liu et al.
(2018b)) do not contain the sampling process: all architecture operations are updated during training at each iteration.
These methods bring overwhelming memory and computation consumption issue, such that direct model search and direct dataset search are incapable.
Proxylessnas (Cai et al.
present multinomial distribution to sample the architectures in the search phase.
However, the probability of multinomial distribution is obtained by applying softmax to architecture weights.
The principal limitation is that the model weights and architecture weights are mutually interactional.
To be concrete, if an architecture gets more training, more likely it will be sampled again in the next iteration.
This winner takes all effect causes premature convergence and reduces the diversity of architectures.
Moreover, the gradient tends to ﬂow through non-parameter operations, such as identity (the architectures derived by gradient-based method is typically small, i.e., 3.4M on CIFAR10 (Liu et al.
(2018b)).
The VAENAS sampling module, which serves as an independent sampling module, helps to 1) increase the diversity of architectures that can be exploited in the training process, as a result, alleviate the premature convergence phenomenon, 2) search for large models.
Large models naturally obtain better performance than small models.
E XPERIMENTS 5. NASNET- LIKE S EARCH S PACE .
The detailed training strategy, search space and hyper-parameters settings can be found in Appendix
A.1.
Our VAENAS is evaluated on both gradient-based and one-shot approaches, and compared with the state-of-the-art NAS methods under the same search space.
We use ’OS’ to denote one-shot approach and use ’G’ to denote gradient-based approach.
Performance on CIFAR-10.
We present the results on Table 1.
Combined with gradient-based method, our method achieves 2.4% test error with 4.4M parameters.
Combined with one-shot approach, our method gives two results.
The ﬁrst one is searched with parameter constraint of 3.5M parameters, and obtain 2.5% test error, and the second one is trained without any constraint and obtain 2.26% test error with 5.2M parameters.
Performance on ImageNet
We present the results on Table 2.
Combined with gradient-based approach, our method obtains 75.6% top-1 accuracy with 568M FLOPs.
Combined with one-shot
[Table] [Table]
approach, our method obtains 75.8% top-1 accuracy with 573M FLOPs.
Both results surpass previous state-of-the-art results.
5. I MAGE N ET C LASSIFICATION UNDER M OBILE S ETTING
Comparison with State-of-the-art methods.
Table 3 lists a number of state-of-the-art models’
FLOPs on ImageNet dataset.
We select models with the range of FLOPS from 200M to 400M.
Notably
[Table]
VAENAS-OS outperforms state-of-the-art models by a large margin.
In particular, the accuarcy of VAENAS is 1.1% higher than EfﬁcientNet-B0 (Tan & Le, 2019), recent state-of-theart method, while using 6.5% less computational budget.
To best of our knowledge, VAENAS-OS is the ﬁrst model surpass 77% under 400M FLOPs without using strong regularization and data augmentation techniques such as AutoAugment.
5. VAENAS ON NAS
B D
NAS-Bench-101 dataset (Ying et al., 2019) is a carefully constructed architecture dataset for NAS research, including over 5 million already trained architectures on CIFAR-10.
The ground-truth accuracy of architectures can be gotten from NAS-Bench-101 dataset directly.
We compare VAENAS sampling with uniform sampling (also known as random search Li & Talwalkar (2019)).
[Table]
We design two methods for training VAENAS.
The weight-sharing method follows Section 4.1, where VAENAS is trained with a super-network on NAS-Bench-101 search space.
The design of super-network is described in Appendix A.5.
The non-weight-sharing method trains VAENAS with a set of high-accuracy architectures.
6000 trained architectures are sampled and the top-1500 models form the training set.
Inﬂuence of noise on accuracy.
The training of VAENAS is based on high-accuracy architectures, while ground-truth accuracy of architectures is hard to be obtained in most cases.
We add various levels of noise on the accuracy, then rank architectures and select high-accuracy architectures to train corresponding VAENAS module.
As displayed in Table 4 and Fig. 2, VAENAS is robust to a certain extent of noise.
Futhermore, the accuracy of architectures in weight-sharing methods have a noise on super-network, while the accuracy on super-network is inconsistent with ground-truth accuracy.
The results of VAENAS trained by weight-sharing method proof that VAENAS works well with NAS frameworks.
Comparisons of sampling methods.
We sample a number architectures by VAENAS and random method and evaluate the maximum accuracy of architectures, which is shown in Table 4.
VAENAS, trained with super-network or clear ground-truth accuracy, outpeforms random sampling (+0.79% with 10 samples).
In addition, we sample 5000 architectures with random sampling and VAENAS sampling, then display the distribution of samples in Fig.
The distribution of VAENAS covers high-accuracy architectures, while the distribution of random sampling is relatively dispersed.
Combination with evolution search.
We execute evolution search on NAS-Bench-101 and replace random sampling by VAENAS during the initialization of population.
The results are shown in
Table 4.
VAENAS outperforms uniform sampling by 0.7%.
It proves that VAENAS is helpful to the search strategy, since the convergence of evolution search is accelerated.
[Figure]
C ONCLUSION
In this paper, we argue sampling matters in NAS and present a learnable sampling method based on
VAE.
The VAENAS learns the architecture distribution and is interactive with NAS search process.
As a result, by embedding into the existing efﬁcient NAS framework, we achieve a series of state-ofthe-art results on various dataset and search space.
Overall, we believe that the VAENAS approach provides a practical new perspective on sampling methods in neural architecture search.
A A PPENDIX A. I MPLEMENTATION D ETAILS OF VAENAS
VAENAS-OS ShufﬂeNet training details.
VAENAS-OS is trained for 480 epochs with batch size 1024 on 8 2080-Ti GPUs.
It is based on the Pytorch (Paszke et al.
The network parameters are optimized using an SGD optimizer with an initial learning rate of 0.5 (decayed linearly after each iteration), a momentum of 0.9 and a weight decay of 3 × 10 − 5 .
Additional enhancements including label smoothing (Szegedy et al.
and Dropout with 0.5 on last FC layer.
We follow the common practice to use inception augmentation.
No other data augmentation method is used.
VAENAS-OS/VAENAS-G NASNet training details:
ImageNet.
The model is trained for 300 epochs with batch size 512 on 8 2080-Ti GPUs.
It is based on the Pytorch (Paszke et al.
The network parameters are optimized by using an SGD optimizer with an initial learning rate of 0.5 (decayed linearly after each iteration), a momentum of 0.
and a weight decay of 3 × 10 − 5 .
Additional enhancements include label smoothing (Szegedy et al.
and auxiliary tower with 0.4.
We follow the common practice to use inception augmentation.
The initial width is 48 and depth is 14, which follows the same setting as Liu et al.
(2018b).
CIFAR-10.
The model is trained by following the common pipeline (Liu et al.
(2018b); Real et al.
(2019); Zoph et al.
; Pham et al.
; Liu et al.
(2018a)).
it is trained for 600 epochs with batch size 48 on single GPU.
Additional enhancements include label smoothing (Szegedy et al.
(2016)), auxiliary tower with 0.4, drop path with 0.2 and Cutout (DeVries & Taylor (2017)).
A. I MPORTANCE OF VAENAS
More ablation study will be available in the Appendix A.3.
In this section, we use gradient-based
NAS approach and one-shot NAS approach as two baseline framework, and compare the result with
VAENAS and without VAENAS using the same search and training strategy.
The baseline method of gradient-based approach is re-implemented based on Cai et al.
(2018) and one-shot approach is re-implemented based on Guo et al.
The results are shown in Table 5.
We can observed that the the framework with VAENAS achieve signiﬁcantly better performance on both CIFAR-10 and
ImageNet.
A. A BLATION S TUDY
Inﬂuence of Depth Gap between Search and Evaluation.
One issue encountered in prior works with both one-shot NAS approach and gradient-based approach is the challenge of search on the network with the same depth as the evaluated models.
We therefore study the effect of VAENAS from the perspective of direct model search.
Table 6 shows the results of baseline with different depth that one-shot method fails to ﬁnd the good architecture when directly search on target model.
One explanation is that the search space is too large for the conventional uniform sampling.
[Table]
A. N ETWORK A RCHITECTURE
This section shows the details of network structures which we searched for NASNet-like search space and ShufﬂeNet-like search space.
ShufﬂeNet-like search space.
We designed a search space based on ShufﬂeNetV2.
Speciﬁcally, we construct a over-parameterized network with 20 layers, each layer can choose between four operations:
• Shufﬂe Convolution 3x • Shufﬂe Convolution 5x • Shufﬂe Convolution 7x • Shufﬂe Xception 3x
ReLU non-linearity are replaced by Swish if conv 5x5, conv 7x7 or Xception are chosen.
Squeezeand-Exited module are also adopted if conv 7x7 and Xception are chosen.
We utilized VAENAS combined with one-shot framework to do the search.
NASNet search space.
We modify the search space of DARTS (Liu et al., 2018b).
A cell is a directed acyclic graph, and every edge in the graph represents a kind of computation operation.
Different from DARTS, in our search space, one node could have more than two predecessors in one cell.
The maximum number of edges in one cell is 2 + 3 + 4 + 5 = 14 , while ’None’ is an available operation in our search space.
The list of candidate operations is listed as follows:
• None • Average Pooling 3x • Max Pooling 3x • Skip Connect • Separate Convolution 3x • Separate Convolution 5x • Separate Convolution 7x • Dilated Convolution 3x
The magnitude of search space is 8 2 × 14 , while normal cell and reduction cell are searched jointly.
We present the cell searched by our methods in Fig.4 and Fig.5.
A. S UPER - NETWORK AND VAENAS ON NAS-B ENCH -101 SEARCH SPACE
Super-network.
NAS-Bench-101 is a cell-based search space, and the details of NAS-Benchsearch space is introduced in Ying et al.
While NAS-Bench-101 is not proposed for weightsharing NAS methods, we design one super-network which covers architectures in NAS-Bench-101.
The output of one cell in NAS-Bench-101 search space is the concatenation of input nodes.
The illustration Fig.6 shows one sampled cell on the super-network.
However, the output channels of
SF 3 x SF 3 x
SF 3 x
X c e p t i o n
SF 3 x
SF 5 x SF 5 x
SF 5 x
SF 3 x 3 / S E
SF 5 x 5 / S E
X c e p t i o n / S E
SF 3 x 3 / S E
SF 5 x 5 / S E
SF 7 x 7 / S E SF 7 x 7 / S E
SF 5 x 5 / S E SF 5 x 5 / S E
SF 7 x 7 / S E
X c e p t i o n / S E
SF 7 x 7 / S E 112 x 112 x 56 x 56 x56 x 56 x56 x
56 x28 x 28 x 28 x 28 x 28 x 28 x 28 x 28 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 7 x 7 x7 x 7 x7 x 7 x7 x 7 x
C o n v 1 x
C o n v 3 x 3 224 x 224 x
P oo li n g F C 7 x 7 x7 x 7 x one middle node is not ﬁxed, so weight-shared tensor is utilized to save the parameters of convolutional operations.
The illustration Fig.7 shows one example of dividing convolutional kernel for computation
Representation and Sampling.
In NAS-Bench-101 dataset, one architecture can be represented as a 7x7 upper-triangle adjacent matrix and 5 operations.
To simply the representation of architectures, the adjacent matrix is converted to a list of edges a = { a 1 , a 2 , ..., a 21 } , a i ∈ { 0 , 1 } , and operations compose a vector op = { op 1 , op 2 , ..., op 5 } , op i ∈ { conv 1 × 1 , conv 3 × 3 , maxpool } .
The integrated architecture is a vector α = concat ( a , op ) .
Two constraints of edges determine the validity of one sampled cell.
One is the number of edges in one cell is no more than 9 in a cell.
The other is that at least one path connect the input node and output node in a cell, which indicate the connectivity of the cell.
Under these two constraints, we design the following sampling strategy.
21 number in range [0 , 1] is sampled by random sampling or VAE sampling, then the indexes of top-9 numbers is selected as IDX .
We set a i = 1 when a i ∈ IDX , and set a j = 0 when j / ∈ IDX .
If the sampled cell is not connected, we perform above sampling process again.
Evolution Search on NAS-Bench-101.
The setting of evolution search follows (Guo et al.
In addition, to make sure the validity of descendants, the mutation and crossover of the edge list a should be planned carefully.
For mutation, an index i, a i = 1 and another index j, a j = 0 are selected randomly.
Another new sample a (cid:48) is generated by exchanging a i and a j .
We repeat this process until a (cid:48) satisfy connectivity.
For crossover, two edge lists a x and a y are selected randomly.
Then an index i, a xi = 1 , a yi = 0 and another index j, a xj = 0 , a yj = 1 are selected.
One new sample a (cid:48) is generated by exchanging a xi and a xj .
If a (cid:48) does not maintain connectivity, it will be dropped, and the crossover process will be repeated.