A Genetic Programming Approach To Zero-Shot Neural Architecture Ranking
Abstract
Neural networks are becoming increasingly ubiquitous in a wide range of use cases.
A primary hurdle in deploying neural networks in many scenarios is the tedious and difﬁcult neural network architectural design process, which was reliant on expert knowledge and iterative design.
Neural Architecture Search (NAS) reduces the human effort required for design, but still has considerable resource requirements and is extremely slow.
To address the inefﬁciencies of conventional NAS, ZeroShot NAS is a new paradigm, which introduces zero shot neural architecture scoring metrics (NASMs) to identify good neural network designs without training them.
While applying Zero Shot NASMs is cheap and requires no training resources, we identify that there is a lack of NASMs that generalize well across neural architecture design spaces.
In this paper, we present a program representation for NASMs and automate its search with genetic programming.
We discover effective NASMs for
Image Classiﬁcation as well as Automatic Speech Recognition.
We believe that our work indicates a new direction for NASM design and can greatly beneﬁt from recent advances in program synthesis.
Introduction
As the diversity of hardware platforms and end use-cases of neural networks increase practical deployment of neural networks is becoming increasingly difﬁcult.
Maximizing accuracy of neural network architectures is no longer the sole concern, with signiﬁcant emphasis being placed on the deployment efﬁciency and the carbon footprint of discovering efﬁcient architectures.
Further, hardware performance metrics are often non-differentiable in nature as these costs can vary arbitrarily depending on the the memory access pattern, cache hierarchy and other several factors.
Given the irregular deployment costs of neural networks in hardware aware settings, the efﬁciency of differentiable neural architecture design is signiﬁcantly lower than in accuracy aware settings.
The sample efﬁciency of non-differentiable methods also suffer due to the irregular nature of the loss landscape of hardware performance.
Zero-shot Neural Architecture Search aims to alleviate the training costs in the Neural Architecture
Search process by ’scoring’ neural networks at initialization.
Ideally, a higher score should correspond to a higher test accuracy of the neural network after training.
We refer to algorithms that can score neural network architectures without training them as Zero Shot Neural Architecture Scoring Metrics (NASMs).
There exist several zero-shot NASMs such as NASWOT, ZenNAS, AngleNAS, TENAS and SynFlow [3, 4, 5, 6, 7].
Design of these NASMs are driven by human intuition or are theoretically inspired by discovering metrics that quantify the trainability and expressivity of neural networks.
[4] empirically studies metrics such as synflow , which was introduced as a techinque to pruning neural network weights at initialization based on a saliency metric [8].
NASWOT [3] treats output of each layer of a neural network as a binary indicator (zero if value is negative, one if value is positive) and uses the hamming distance between two binary codes induced by an untrained network at two inputs as a measure of dissimilarity.
This is driven by the intuition that the more similar two inputs are, ), Sydney, Australia. the more challenging it should be for the network to learn how to separate them.
Such methods of scoring neural network architectures can work well for a small sub-set of neural architecture design spaces, but in practice often fail to transfer across NN design spaces.
These methods may also impose structural restrictions on the neural networks that limit the architecture design space.
For instance,
ZenNAS [6] truncates the neural network representation by removing residual connections.
This limits the ability to distinguish differing connectivity patterns of the neural network architecture.
In this paper, we identify a domain speciﬁc language (DSL) which can effectively represent existing
SoTA NASMs.
We also identify an appropriate choice of input-output example and formulate the discovery of programs (NASMs) with the DSL as a program synthesis task driven by evolutionary search.
This method allows us to discover NASMs that impose minimal restrictions on the neural network representation and task.
Our contributions are as follows:
• Utilizing an expression tree structure to represent NASMs in a manner that is sufﬁciently expressive to capture the task of neural architecture ranking across neural architecture design spaces and tasks.
• Demonstration of our genetic programming driven methodology (EZNAS - E volutionarily Generated Z ero Shot N eural
A rchitecture S coring Metric) to automate zero-shot NASM design for Image Classiﬁcation and Automatic Speech Recognition.
• Outline of future work to motivate research in more robust input-output example representation, DSL design and program search strategies.
Evolutionary Framework
Domain Speciﬁc Language (DSL) .
DSLs are computer languages that are suitable for a specialized domain and are more restrictive than full-features programming languages [9].
An appropriate DSL construction is crucial to discovering NASMs efﬁciently.
We explored two alternative program representations for evolutionary search of NASMs.
Our initial choice of DSL was inspired by AutoML-Zero [10] and posed minimal structural restrictions on the program.
The program representation and application is depicted in Figure 1 (Left).
A total of 22 tensors were generated at every ReLU-Conv2D-BatchNorm2D ( RCB tensors ) instance of the neural network (weights, activations and gradients).
Each NASM was applied to every RCB instance and the score was generated by averaging the score assigned by the NASM at every RCB instance of the neural network.
We had 22 static memory addresses to store the RCB tensors (referenced with integers 0-21 ), 80 dynamic memory addresses to store intermediate tensors generated by the program and 20 dynamic memory addresses to store intermediate scalars.
The ﬁtness of each individual program is calculated by measuring the Kendall Tau rank correlation between the scores generated of 80 neural network architectures, and their test accuracy.
The test accuracy is readily available to us as we use public NAS data-sets (NASBench-201 [11] and NDS [12]).
As seen in Figure 1 (Left), each NASM was sequentially represented as a set of integers, with every four integers indicating an expression with a result, operation ID and two operands.
We initialize valid random integer arrays and convert them to programs to evaluate and fetch the ﬁtness.
Valid integer arrays are the arrays that can be converted to programs and generate a ﬂoat score.
We allow Mate,
InsertOP, RemoveOP, MutateOP & MutateOPArg as variation functions described further in the appendix.
While we discover weak NASMs with this formulation, we observe that there are too many redundancies in the programs discovered.
Program length bloating as well as operations that do not contribute to the ﬁnal output made the run time intractable.
In EZNAS, each individual NASM has to be evaluated on several gigabytes of input-output examples with no approximations to generate precise ﬁtness values.
To reduce the computational complexity of search, we necessitate an expression tree structure on the NASM program to capture the executional ordering of the program.
As depicted in Figure(Right), the program output appears at a root node, and the child (terminal) nodes are the arguments of the expression tree.
These arguments are the 22 RCB tensors.
The advantage of this program representation is that there is only a single root node with dense connectivity form the root to terminal nodes, leading to lesser redundancies.
As the expression tree describes the executional ordering of the mathematical operations available to us, it is crucial to provide a varied set of mathematical operations.
We provide 34 unique operations in our program search space, including basic mathematical operations such as Addition, Product, Log as well as operations such as Cosine
Similarity, KL Divergence and Hamming Distance.
We provide the full list of mathematical operations available in the supplementary material.
Search Technique .
Our search algorithm discovers programs by modifying the expression tree representation.
At each generation, ﬁtness score is generated for every expression tree (NASM) in the population.
A selection and variation algorithm is then used to generate new offspring.
We utilize the
VarOr implementation from Distributed Evolutionary Algorithms in Python (DEAP) [13] framework for variation of individual programs.
We generate n valid offspring programs at each generation. n/ 2 offspring are generated as a result of three operations; crossover, mutation or reproduction.
To encourage diversity, we also randomly generate n/ 2 valid individuals at every generation.
Invalid programs that either fail to execute or give inf, nan outputs are replaced by new individuals.
We give an algorithmic description of our search method in the Appendix.
Results
We choose the most consistent NASM from all our evolutionary runs on the image classiﬁcation task ( EZNAS-A ) and the automatic speech recognition task ( EZNAS-ASR ) and compare with existing Image Classiﬁcation [Table] NASM
Amoeba
DARTS
ENAS PNAS NASNet NASM
ASR TIMIT
EZNAS-A 0.
EZNAS-ASR 0.
NASWOT 0.
SNIP 0.
grad_
[Table]
[Table]
[Table]
syn_flow -0. syn_flow 0. NASMs from recent works. We can see that we are able to obtain competitive Spearman rank correlation coefﬁcient on the NASBench-ASR automatic speech recognition task.
Our EZNAS-A
NASM is able to outperform competitive NASMs consistently and able to generalize across the architecture design spaces, as indicated by the test in Table 2 (Left).
Discussion
In this paper, we present EZNAS, a novel genetic programming driven approach to discover Zero Shot
Neural Architecture Scoring Metrics (NASMs).
EZNAS automates the discovery of relevant features of a neural network with minimal human intervention, and outperforms most existing human-designed
NASMs in both generalizability and Kendall Tau Ranking Correlation.
We believe that this is a ﬁrst step towards not only efﬁcient neural architecture search but also discovering interpretable programs that indicate the attributes of neural network that contribute to accuracy.
In the rest of this section, we discuss the limitations of our current approach and outline future work that may enable us to discover better NASMs.
Program Design.
To simplify the search problem, we take a mean ( aggregation_function in Figure 1) of the scores generated across all ReLU-Conv2D-BatchNorm2D instances.
This is a major limitation, as it hopes to capture layer connectivity patterns implicitly in the network attribute tensors (22 tensors).
Extending our evolutionary search to take into account the global connectivity pattern of the neural networks or learning weighted averaging techniques for the individual layer scores along with the scoring program could help us discover better NASMs.
Research in graph neural networks along with evolutionary program synthesis may allow us to obtain higher performance NASMs.
Further, we impose structural restrictions on our program by necessitating a ﬁxed structure ( ReLU-Conv2D-BatchNorm2D ).
We have to truncate instances of
ReLU-Conv2D-Conv2D-BatchNorm2D from the NDS space by ignoring the second convolution.
As more diverse architectures are introduced in the ﬁeld, we must reformulate collection of network statistics in a more robust manner.
It is important to note that NASWOT measures the similarity in the binary codes generated from the ReLU units for two different inputs.
Our implemented program design limits us to only using a single/batch of inputs and does not compare different input samples directly.
Thus, we would not be able to discover the NASWOT NASM metric with our current formulation.
Fortunately, integrating such functionality into the program design space is trivial and is an interesting line of future work to generate more complex metrics.
Network Statistics.
Due to the computational resources required to generate network statistics, we have to pre-compute them and use the generated data as an evolutionary task dataset .
We only use network statistics with input batch-size of 1 for evolution.
A single sample statistic of a neural network is insufﬁcient to describe the architecture.
Unfortunately, this would cause a linear increase in memory requirement as well as increase in run-time of the evolutionary search.
If we were to generate network statistics of all neural networks in the NASBench-201 and NDS spaces (excluding
ImageNet), it would be 7 terabytes of tensors with a batch size of just 1.
Computing network statistics at run-time can be made more efﬁcient by using lower precision numerical formats, or exploring proxy tasks on smaller datasets (down-sampled image datasets, smaller neural network design architecture etc.) to discover metrics that can inﬂuence accuracy.
Evolutionary Search.
Recent advances in the ﬁeld of Neuro-Symbolic Program Synthesis [15] to learn mappings from input-output examples (neural network statistics to neural architecture scores) can motivate improvements in our evolutionary search.
Further, exploring architecture connectivity encodings [16] for neural architecture search and discovering programs to rank neural architecture encodings may enable discovery of effective connectivity patterns to enable a deeper understanding of important features in neural architecture design beyond those contained in activations maps and weights.
Details of Evolutionary Framework
EZNAS uses evolutionary search to discover programs that can score neural network architectures in proportion to their accuracy.
There are several components that are instrumental in facilitating efﬁcient search for such a program in the prohibitively large program space.
In this section, we go over the different components of our framework in detail.
1.1 Program Representation
In our evolutionary search, each individual program has to be evaluated on tens of gigabytes of network statistics with no approximations to generate precise NASM score-accuracy KTR ﬁtness.
To make such search computationally tractable, it is crucial to not introduce redundant operations in the program.
Our initial attempt described in the Appendix 2.3 resembled that of AutoML-Zero [1] where
NASMs were represented as a sequence of instructions and a memory space to store intermediate tensors.
This led to discovery of NASMs with many redundant computations and an intractably large run-time.
To reduce the complexity of search, we necessitate a expression tree structure on the NASM program to capture the executional ordering of the program.
The program output appears at a root node, and the child (terminal) nodes are the arguments of the expression tree.
These arguments are the network statistics .
The advantage of this program representation is that there is only a single root node with dense connectivity from the root to the terminal nodes, leading to lesser redundancies.
1.2 Neural Network Statistics Generation
To search for genetic programs that can rank neural network architectures effectively, we need to generate a dataset of ’network statistics’ .
As depicted in Figure 1, for any sampled neural network from the NDS or NASBench-201 spaces, we identify every ReLU-Conv2D-BatchNorm2D instance after randomly initializing the neural network.
We register the input and output activation ( T1, T2 ) and gradient ( T1G, T2G ) to the ReLU unit, weight and weight gradient of the Conv2D layer ( T3,
T3G ) and the output and output gradient of BatchNorm2D ( T4, T4G ).
We register these values for a single randomly sampled training data item (a single image D ), a random noise input ( N ) with mean 0 and variance 1 and a training sample perturbed by random noise ( D + √ 0 .
01 N ).
In the Appendix 2.8, we also present an alternate formulation which identiﬁes every Conv2D-BatchNorm2D-ReLU instance for network statistics generation to demonstrate that the evolutionary framework is not restricted to a ReLU-Conv2D-BatchNorm2D structure.
We register 24 different tensors for each ReLU-Conv2D-BatchNorm2D instance, however since the weight tensor is the same for the three types of input ( D , N , D + √ 0 .
01 N ), we have 22 unique tensors.
For evolutionary search, we generate an evolution task dataset by randomly samplingneural networks from each available search space (NASBench-201 and NDS) and dataset (CIFAR-10,
CIFAR-100, ImageNet-16-120).
At the time of evolution, we only have access to evolution task dataset .
At the end of the search, we use a test task dataset to identify the best programs.
The test task dataset has 1000 neural networks statistics for each dataset on the NASBench-201 space and 200 neural network statistics for each design space on NDS.
We generate a smaller network statistics dataset on NDS due to RAM constraints.
Since we are generating statistics for each layer on over 4640 ( = 1000 × 3 + 200 × 5 + 80 × 8 ) neural networks, the dataset size is approximately 400 GB for a batch size of 1.
1.3 Mathematical Operations
As an expression tree describes the execution order of the mathematical operations available to us, it is crucial to provide a varied set of operations to process the neural network statistics effectively.
We provide 34 unique operations in our program search space.
We include basic mathematical operations such as Addition, Product, Log as well as some advanced operations such as Cosine Similarity, KL
Divergence, Hamming Distance.
We provide the full list of mathematical operations at the end.
1.4 Program Application
Majority of the neural network architectures available in NASBench-201 and NDS have overinstances of ReLU-Conv2D-BatchNorm2D .
This may mean that an expression tree would have as many as 2200 ( 22 × 100 ) possible arguments (terminal nodes), each of which can be used multiple times.
This would result in a computationally intractable expression tree.
To simplify the search problem, we generate a single expression tree with 22 possible inputs.
This expression tree is then applied on every ReLU-Conv2D-BatchNorm2D instance and the output is aggregated across all instances using an aggregation_function .
It is not necessary that the root node of an expression tree would give a scalar value, so we add a to_scalar operation above the root node of the expression tree.
This serves as the ’score’ of a single layer of the the sampled neural network architecture.
We demonstrate how an expression tree is applied to a neural network architecture to generate a score in
Figure 3.
In EZNAS-A , the aggregation_function and to_scalar are both mean .
In Appendix 2.8, we explore L2-Norm as a to_scalar function as well and ﬁnd that we are able to discover effective NASMs.
input tensor represents a single sample image from the dataset such as CIFAR-10, CIFARserving as input to the neural network.
N represents a randomly initialized noise tensor.
We refer to D + √ 0 .
01 N as ’Perturb’.
It represents an input to the neural network which is a data-sample perturbed by noise.
1.5 Evolutionary Algorithm
Our search algorithm discovers programs by modifying the expression tree representation.
A ﬁtness score is generated for every expression tree in the population.
A selection and variation algorithm is then used to generate new offspring.
1.5.1 Population Initialization
We initialize a population of n programs.
We do not impose any restrictions on the operations the nodes can use in the expression tree.
Due to this, the number of valid expression trees several is orders of magnitude lesser than the total number of expression trees that can be generated with our mathematical operators and network statistics.
To increase search efﬁciency, we would like to ensure we sample and evolve only valid expression trees.
To enable this, we ensure that all individuals in the population are valid programs by testing program execution on a small sub-set of network statistics data.
All programs that produce outputs with inf , nan or fail to execute are replaced by new randomly initialized valid programs.
1.5.2 Fitness Objective
As the goal of zero shot NAS is to be able to rank neural network architectures well, we utilize the
Kendall Tau rank correlation coefﬁcient as the ﬁtness objective.
The search objective is to maximize the Kendall Tau rank correlation coefﬁcient between the scores generated by a program and the test accuracy of the neural networks.
1.5.3 Variation Algorithms
In our tests, we utilize the VarOr implementation from Distributed Evolutionary Algorithms in
Python (DEAP) [2] framework for variation of individual programs.
We generate n (hyper-parameter) offspring programs at each generation.
n/2 offspring are generated as a result of three operations; crossover, mutation or reproduction.
These variations are depicted in Figure 2.
For crossover, two individual programs are randomly selected from the population and mated.
Our mating function randomly selects a crossover point from each individual and exchanges the sub-trees with the selected point as root between each individual.
The ﬁrst child is appended to the offspring population.
For mutation, we randomly select a point in the individual program, and replace the sub-tree at that point by a randomly generated expression tree.
We repeat the variation algorithm on the population till n/2 valid individuals are generated.
To encourage diversity, we also randomly generate n/2 valid individuals.
We have placed static limits on the height of all expression trees at 10.
1.6 Program Evaluation Methodology
At each generation, the ﬁtness of the entire population is invalidated and recalculated.
Calculating the ﬁtness of each program on the entire dataset (which can be approximately 1 TB) is computationally infeasible.
Further, we may want to ﬁnd generalized programs that give high ﬁtness on many different architecture design spaces and datasets.
This means we have to evaluate each individual on 7 TB of data.
Reducing the computation by evaluating the ﬁtness of the population on a single small ﬁxed sub-set of neural networks from the search space causes discovered programs to trivially over-ﬁt to the sub-set statistics in our tests.
To address over-ﬁtting of programs to small datasets of network statistics while minimizing compute resources required for evaluating on the entire dataset of network statistics, we use our evolution task dataset .
The evolution task dataset contains statistics of 80 neural networks on each search space.
We evaluate individuals by randomly choosing s search spaces, and sampling 20 neural networks from each of the chosen search spaces.
We take the minimum ﬁtness achieved by the individual program on the s spaces.
We consider s as a hyper-parameter.
In our tests, this is consistently kept at 4.
1.7 Program Testing Methodology
At the end of the evolutionary search, our primary goal is to test whether the programs discovered are able to provide high ﬁtness on previously unseen neural network architecture statistics.
We test the ﬁttest program from our ﬁnal population as well as the two ﬁttest programs encountered through-out the evolutionary search.
At test time, we take the program and ﬁnd the score-accuracy KTR over the entire test task dataset .
Examination Of Our Best Programs
We analyze two NASMs discovered by EZNAS to give a deeper understanding of the nature of programs our search ﬁnds.
The ﬁrst program we analyze is EZNAS-A .
This evolution task dataset for discovering this program was NDS DARTS.
The input to the NASM in Figure 4 is the T3GNoise (Weight Gradient with Random Noise Input).
To understand how the NASM score varies with the weight gradient size, we generate random tensors of varying sizes and average the NASM outputtimes.
We ﬁnd that the score increases as the number of channels or depth increases, we also observe that kernel sizes of 1 and 7 give higher scores than 3 and 5 with the lowest score being assigned to kernel of size 3.
It is interesting to see that the expectation value of EZNAS-A NASM in Figure 4 translates to a weighted form of parameter counting, with a non-linear monotonically increasing scaling of score with the number of input/output channels and a locally parabolic relationship between the score and the kernel size with the minimum score (among integers) at kernel size of 3.
The second program we analyze ( EZNAS-B ) is depicted in Figure 5.
This evolution task dataset for discovering this program was NDS ENAS and its score-accuracy KTR on the test task dataset is provided in Figure 13.
We analyze this NASM in a manner similar to Figure 4.
We provide random tensors in place of T1GPerturb (Difference in pre-activation gradients for a random noisy input and a data-sample input).
This is an approximation to understand how the NASM responds to change in activation gradient size.
We ﬁnd that the activation map size is exponentially more inﬂuential to the score when compared to the number of channels.
With the random initialization approximation done in our analysis, we ﬁnd that our NASM EZNAS-A is counting the number of parameters in a ’weighted fashion’.
EZNAS-B generates a score by giving higher weightage to feature map size over number of channels as a proxy for accuracy.
It is interesting to note that when compared to the methods from recent papers in zero shot NASMs, this form of weighted parameter counting works more consistently and better.
This is further supported by our ﬁnding that FLOPs and parameter count are competitive NASMs which work more consistently as opposed to synflow or grad_norm which work on the NASBench-201 space but fail on the NDS space.
2.1 NASBench-201 and NDS
For image classiﬁcation, we utilize the NASBench-201 [3] and NDS [4] NAS search spaces for our evolutionary search as well as testing.
NASBench-201 consists of 15,625 neural networks trained on the CIFAR-10, CIFAR-100 and ImageNet-16-120 datasets.
Neural Networks in Network Design
Spaces (NDS) uses the DARTS [5] skeleton.
The networks are comprised of cells sampled from each of AmoebaNet [6], DARTS [5], ENAS [7], NASNet [8] and PNAS [9].
There approximately 5000 neural network architectures
2.2 Our evolutionary algorithm
We explicitly describe our evolutionary search method in Algorithm 1.
evolution_space refers to the evolution task dataset.
the evaluate function evaluates every individual in the offspring and assigns a ﬁtness value to them.
This ﬁtness value is the minimum score-accuracy KTR obtained by testing it on each of the evolution_space .
test_space refers to the test task dataset, and is used after the last generation to identify good programs.
The genValidPopulation function generates N valid individuals.
The VarOr function is the variation algorithm we have used to mate, mutate and cross-over the population.
We repeat the VarOr function generates half the offsprings, while the other half is randomly initialized at each generation.
This is to promote diversity in the population.
The selectValid function selects all the valid individuals produced by the VarOr function by evaluating each individual on a single neural network statistic.
The selTop3 selects the three individuals with the highest ﬁtness.
The top 3 individuals are propagated to the next generation with no variation. selTournament selects the best individual among x randomly chosen individuals, k times.
We set the x=4 and k=N .
2.3 Details of Sequential Program Representation
Our initial attempts at discovering NASMs took a different approach to program representation.
This sequential program representation posed no structural limitations on the program.
We have 22 static memory addresses, which contained network statistics and are referenced with integers 0-21 .
To store intermediate tensors generated by the program, we allocate 80 dynamic memory addresses, which can be over-written in the program as well.
To store intermediate scalars generated by the program, we allocate 20 memory addresses.
As seen in Figure 8, we represent the programs as integers, where each instruction is expressed as 4 integers.
The ﬁrst integer provides the write address, the second integer provides the operation ID and the third and fourth integers provide the read addresses for the operation.
We initialize valid random integer arrays and convert them to programs to evaluate and fetch the ﬁtness (score-accuracy KTR).
We allow Mate, InsertOP, RemoveOP, MutateOP & MutateOpArg as variation functions.
The Mate function takes two individuals, and takes the ﬁrst half of each individual.
Then, these components are interpolated to generate a new individual.
The
InsertOP function inserts an operation at a random point in the program.
The RemoveOP function removes an operation at a random point in the program.
The MutateOP changes a random operation in program without changing read/write addresses.
The MutateOpArg function simply replaces one of the read arguments of any random instruction with another argument from the same address space (dynamic address argument cannot be replaced by a scalar address argument).
While we are able to discover weak NASMs with this formulation, we observe that there are too many redundancies in the programs discovered.
Program length bloating as well as operations that do not contribute to the ﬁnal output were frequently observed.
Due to these issues, the evolution time evaluation of individual ﬁtness quickly became an intractable problem.
To address this, we change our program representation to a expression tree representation in the results reported in the paper.
This representation necessitates contribution of each operation to the ﬁnal output, which means there is no redundant compute.
While the sequential program representation is valid, we believe evolution_space =
[’DARTS’, ’NB201-CIFAR10’]
test_space =
[’PNAS’, ’ENAS’] population = genValidPopulation(N) evol_dataset = loadDataset(evolution_space) evaluate(population) top_gen=[] for gen=1:T do offspring =
[] if len(top_gen) > 0 then offspring.append(top_gen) top_gen
= [] while len(offspring) < N//2 do # Applies crossover, mutation & reproduction individuals = VarOr(population) offspring.append(selectValid(individuals))
# Half the population is randomly # initialized at each generation individuals = genValidPopulation(N//2) offspring.append(individuals) population=
evaluate(offspring) top_gen.append(selTop3(population))
if gen!=T then population = selTournament(population) test(population) top_NASMs=selTop3(population) that signiﬁcant engineering efforts are required to ensure discovery of meaningful programs.
Our sequential program representation is directly inspired by the formulation used in AutoML-Zero [1].
AutoML-Zero makes signiﬁcant approximations in the learning task to evolutionarily discover MLPs.
While AutoML-Zero has a much larger program space to search for, approximations in computing individual ﬁtness are not feasible in our formulation as generating exact score-accuracy KTR is an important factor in selecting individuals with high ﬁtness.
2.4 Noise and Perturbation for Network Statistics
To generate network statistics, we use three types of input data.
The ﬁrst is simply a single random sample from the dataset (e.g. a single image or a batch of images from
CIFAR-10).
To generate a noisy input, we simply use the default torch.randn function as input = torch.randn(data_sample.shape) .
The third type of input we provide is a data-sample which has been perturbed by random noise ( input = data_sample + 0. [Table] .randn(data_sample.shape) ).
2.5 Network Initialization Seed Test
In Figure 9, we use different seeds to change the initialization and input tensors, but keep the neural architectures being sampled ﬁxed in the respective spaces.
The variance in the score accuracy KTR is much lesser than in Figure 10 where the seed also controls the neural architectures being sampled.
This shows the true variation in our EZNAS-A NASM with respect to network initialization.
2.6 Hardware used for evolution and testing
Our evolutionary algorithm runs on Intel(R)
Xeon(R)
Gold 6242 CPU with 630GB of RAM.
Our
RAM utilization for evolving programs on a single Image Classiﬁcation dataset was approximately 60GB.
RAM utilization can vastly vary (linearly) based on the number of neural network statistics that are being used for the evolutionary search.
Our testing to generate the statistics for the seed experiments as well as the ﬁnal Spearman ρ and Kendall Tau Rank Correlation Coefﬁcient is done on an NVIDIA DGX-2 server with 4 NVIDIA V-100 32GB GPUs.
2.7 Hyper-parameters for discovering [-EZNAS-Anum_generation: 15population_size: 50tournament_size: 
-]EZNAS-A num_generation: population_size: tournament_size:
MU:lambda_ : [-50crossover_prob: 0.4mutation_prob: 0.4min_tree_depth: 2max_tree_depth: 
-]crossover_prob: 0. mutation_prob: 0. min_tree_depth: max_tree_depth:
2.8 Varying network statistics and to_scalar
In Figure 11, we detail 3 tests while evolving on the NDS_DARTS CIFAR-10 search space in an identical fashion to EZNAS-A (referred to as EZNAS-(R-C-B)-(Mean) here).
EZNAS-(C-B-R)-(Mean) and EZNAS-(C-B-R)-(L2) correspond to alternate to_scalar and network statistics collection tests respectively.
We demonstrate that while EZNAS-(R-C-B)-(Mean) is superior, we are able to discover NASMs with all three formulations.