EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
Abstract
Neural Architecture Search (NAS) has signiﬁcantly improved productivity in the design and deployment of neural networks (NN).
As NAS typically evaluates multiple models by training them partially or completely, the improved productivity comes at the cost of signiﬁcant carbon footprint.
To alleviate this expensive training routine, zero-shot/cost proxies analyze an NN at initialization to generate a score, which correlates highly with its true accuracy.
Zero-cost proxies are currently designed by experts conducting multiple cycles of empirical testing on possible algorithms, datasets, and neural architecture design spaces.
This experimentation lowers productivity and is an unsustainable approach towards zero-cost proxy design as deep learning use-cases diversify in nature.
Additionally, existing zerocost proxies fail to generalize across neural architecture design spaces.
In this paper, we propose a genetic programming framework to automate the discovery of zero-cost proxies for neural architecture scoring.
Our methodology efﬁciently discovers an interpretable and generalizable zero-cost proxy that gives state of the art score-accuracy correlation on all datasets and search spaces of NASBench201 and Network Design Spaces (NDS).
We believe that this research indicates a promising direction towards automatically discovering zero-cost proxies that can work across network architecture design spaces, datasets, and tasks.
Introduction
The manual trial-and-error method of designing neural network architectures and assessing whether it meets performance and accuracy requirements is not scalable to complex neural architecture design spaces and hardware deployment scenarios.
Neural Architecture Search is intended to address this problem of iterative and expensive design by automating the search of neural network architectures.
Enabling such automation requires accurate and efﬁcient prediction of the accuracy of candidate neural network architectures.
The process of predicting the accuracy often involves partial or complete training of many neural network architectures in the search space.
Such methods are infeasible for very large architecture design spaces, as considerable sampling and training of neural networks would be required to sufﬁciently describe the entire design space.
Popular NAS techniques often involve generating and training neural network architectures and using this accuracy to train accuracy predictors to serve as feedback in the search algorithm.
This architecture generation and feedback methodology can be leveraged by algorithms such as reinforcement learning (Luo et al.
[2019]) and evolutionary algorithms (Real et al.
One-shot methods (Cai et al.
[2019, 2020], Xie et al. [2020], Yang et al.
[2020], Liu et al.
[2019]) maintain a super-network, from which sub-networks are sampled.
The weights are shared between the sub-networks which reduces NAS cost as there are lesser parameters to train.
The primary challenge of Neural Architecture Search is the evaluation of candidate architectures, which can take hours to days to estimate (Abdelfattah et al.
Further, practical deployment of neural networks is no longer limited to accuracy-oriented neural architecture design.
The need for efﬁcient deployment has given rise to signiﬁcant literature in co-design of hardware and neural network architectures (Choi et al.
[2021], Akhauri et al.
[2021], Zhang et al.
However, hardware performance metrics in such co-design tasks are often non-differentiable in nature due to factors such as memory access patterns and cache hierarchy.
The interaction between the workload and the hardware makes this optimization problem signiﬁcantly more complex, and being able to score neural network architectures without the expensive training step would allow for the development of more efﬁcient co-design algorithms.
Most zero-cost proxies utilize a single minibatch of data and a single forward/backward propagation pass to score a neural network (Abdelfattah et al.
We refer to algorithms that score neural network architectures without training as Zero-Cost Neural Architecture Scoring Metrics (ZC-NASM).
Design of existing ZC-NASMs are driven by human intuition or are theoretically inspired.
These algorithms attempt to quantify the trainability and expressivity of neural networks.
The primary hurdle with existing ZC-NASM design is that they are not generalizable to different neural architecture design spaces.
In this paper, we introduce a genetic programming driven methodology to automatically discover
ZC-NASMs that are interpretable, generalizable and deliver state of the art score-accuracy correlation.
We refer to our method as EZNAS ( E volutionarily Generated Z ero-Cost N eural A rchitecture S coring
Metric).
Our approach is:
• Interpretable: We discover ZC-NASMs as expression trees that explicitly indicate which NN features and mathematical operations are being utilized.
• Generalizable:
The ZC-NASM discovered by EZNAS delivers SoTA score-accuracy correlation on unseen neural architecture design spaces and datasets (NASBench-201, NDS & NATS-Bench).
Our framework utilizes simple mathematical operations and an expression tree structure that can be trivially extended to implement arbitrarily complex ZC-NASMs • Efﬁcient: We are able to discover our SoTA ZC-NASM on an Intel(R) Xeon(R) Gold 6242 CPU in under 24 hours.
This requires 12.5 × lesser CO2e than a NAS search.
Our ZC-NASM is generalizable to multiple design spaces, increasing the end-to-end efﬁciency of NN architecture search by over two orders of magnitude.
Related Work
Neural Architecture Search (NAS) was proposed to reduce the human effort that goes into manually designing complex neural network architectures.
Early efforts in the ﬁeld of NAS adopted brute force methods by training candidate architectures and using the obtained accuracy as a proxy to discover better architectures.
AmoebaNet (Real et al.
[2019]) utilized evolutionary algorithms (EA) and [-
-]3150 GPU days of compute to achieve 74.5% top-1 accuracy on the ImageNet dataset.
Many EA and Reinforcement Learning (RL) driven methods have since signiﬁcantly improved the efﬁciency of the search process (Yang et al.
[2020], Liu et al.
[2018], Tan et al.
[2019], Zoph et al.
These methods often require sampling and training of several candidate architectures.
One-shot (Hu et al. [2020], Xie et al.
[2020], Cai et al.
[2019]) methods of NAS do not require training of candidate architectures to completion, but train large super-networks and identify sub-networks that can give high accuracy.
Such super-networks can be generated automatically from pre-trained models (Muñoz et al.
These improvements have signiﬁcantly reduced the cost of NAS.
However, as the search space continues to get larger with new architectural innovations, we need more efﬁcient methods to predict the accuracy of neural networks in intractably large design spaces.
Zero-Cost Neural Architecture Scoring is a promising paradigm which explores zero-cost proxies for estimating the true accuracy of neural networks.
The majority of research in this ﬁeld introduces novel methods of scoring neural networks at initialization along with a theoretical or intuitive explanation of their scoring strategy.
For instance, (Abdelfattah et al.
[2021]) empirically studies metrics such as synflow , SNIP and FISHER .
Figure 2 depicts the expression tree representations of these metrics.
NASWOT (Mellor et al.
[2021]) works by treating the output of each layer as a binary indicator (zero if value is negative, one if value is positive) using the hamming distance between two binary codes induced by an untrained network from two inputs as a measure of dissimilarity. The intuition here is that the more similar two inputs are, the more challenging it is for the network to learn how to separate them. Other works such as TENAS (Chen et al. [2021]) are motivated by creating proxies for trainability and expressivity of neural networks to rank them.
Program Synthesis : Program synthesis is the task of automatically discovering programs that satisfy user constraints.
AutoML-Zero (Real et al.
[2020]) evolves entire machine learning algorithms from scratch with little restrictions on form and using only simple mathematical operations.
The algorithms are learnt symbolically, representing programs as a sequence of instructions.
We posit that the current endeavor of identifying fundamental architectural properties that correlate strongly with testing accuracy can beneﬁt from a methodology that minimizes human bias and intervention in a similar fashion.
This stems from the observation that majority of the existing ZCNASMs can be represented as simple programs that utilize the neural network statistics as inputs which can be discovered with genetic programming.
Evolutionary Framework
EZNAS uses evolutionary search to discover programs that can score neural network architectures such that it correlates highly with accuracy.
In this section, we detail the speciﬁcs of how ZC-NASM programs are constructed, evaluated and evolved with EZNAS.
3.1 Program Representation
[Table]
[Table]
Each individual ZC-NASM has to be evaluated on tens of gigabytes of tensors of NN architectures with no approximations to generate precise ZCNASM score-accuracy correlation to serve as the measure of ﬁtness.
To make such search computationally tractable, it is crucial to not introduce redundant operations in the program.
Our initial attempt described in the appendix resembled that of AutoML-Zero (Real et al.
where
ZC-NASMs were represented as a sequence of instructions and a memory space to store intermediate tensors.
This led to discovery of ZC-NASMs with many redundant computations due to program length bloating and an intractably large runtime.
To reduce the complexity of search, we necessitate a expression tree structure on the ZC-NASM program to capture the executional ordering of the program.
The program output appears at a root node, and the child (terminal) nodes are the arguments of the expression tree.
These arguments are the network statistics .
The advantage of this program representation is that there is only a single root node with dense connectivity from the root to the terminal nodes, leading to fewer redundancies.
3.2 Neural Network Statistics Generation
Each ZC-NASM requires a set of arguments as inputs.
These arguments are the intermediate tensors of the neural network.
As depicted in Figure 1, for any sampled neural network from the
NDS or NASBench-201 spaces, we identify every ReLU-Conv2D-BatchNorm2D (RCB) instance at initialization, and register the activations, weights and the corresponding gradients for three types of network inputs (a mini-batch of data ( D ), a random noisy tensor ( N ), and a mini-batch of data perturbed by random noise ( P )).
We collect T 1 { D,N,P } , T 2 { D,N,P } , T 3 , T 4 { D,N,P } (10 tensors) and T 1 G { D,N,P } , T 2 G { D,N,P } , T 3 G { D,N,P } , T 4 G { D,N,P } (12 tensors) from each RCB instance.
T3 represents the Conv2D weights and does not change for the network input type, thus giving us 22 tensors for each RCB instance.
We present an alternate formulation by identifying Conv2D-BatchNorm2D-ReLU (CBR) instance for network statistics generation to demonstrate that the evolutionary framework is not restricted to a
RCB structure in the appendix.
3.3 Mathematical Operations
The expression tree describes the execution order of the mathematical operations available to us.
It is crucial to provide a varied set of operations to process the neural network statistics effectively.
We provide 34 unique operations in our program search space.
We include basic mathematical operations (Addition, Product) as well as some advanced operations such as Cosine Similarity and Hamming
Distance.
We provide the full list of mathematical operations in the supplementary material.
3.4 Program Application
Majority of the neural network architectures available in NASBench-201 and NDS have overinstances of ReLU-Conv2D-BatchNorm2D .
This may mean that an expression tree would have as many as 2200 ( 22 × 100 ) possible arguments (terminal nodes), each of which can be used multiple times.
This would result in a computationally intractable expression tree.
To simplify the search problem, we generate a single expression tree with 22 possible inputs.
It is not necessary that the root node of an expression tree would give a scalar value, so we add a to_scalar operation above the root node of the expression tree.
As seen in Figure 1 the expression tree is then applied on every
ReLU-Conv2D-BatchNorm2D instance and the output is aggregated across all instances using an aggregation_function .
This serves as the ’score’ of the the sampled neural network architecture.
In EZNAS-A , the aggregation_function and to_scalar are both mean .
In the appendix, we explore L2-Norm as a to_scalar function as well and ﬁnd that we are able to discover effective
ZC-NASMs.
Our search algorithm discovers programs by modifying the expression tree representation of the population by variation_functions .
A ﬁtness score is generated for every expression tree in the population at each generation.
The variation_function (VarOr) is then used to generate new offspring.
We describe our search algorithm in Algorithm[-
-]
3.5.1 Population Initialization
We initialize a population of n programs.
We do not impose any restrictions on the operations the nodes can use in the expression tree.
Due to this, the number of valid expression trees is several orders of magnitude lesser than the total number of expression trees that can be generated with our mathematical operators and network statistics.
To increase search efﬁciency, we would like to ensure that we sample and evolve only valid expression trees.
To enable this, we ensure that all individuals in the population are valid programs by testing program execution on a small sub-set of network statistics data.
All programs that produce outputs with inf , nan or fail to execute are replaced by new randomly initialized valid programs.
3.5.2 Fitness Objective
As the goal of zero-cost NAS is to be able to rank neural network architectures well, we utilize the
Kendall Tau rank correlation coefﬁcient as the ﬁtness objective.
The search objective is to maximize the Kendall Tau rank correlation coefﬁcient between the scores generated by a program and the test accuracy of the neural networks in the evol_space .
3.5.3 Variation Algorithms
In our tests, we utilize the VarOr implementation from Distributed Evolutionary Algorithms in Python (DEAP)
(Fortin et al.
[2012]) framework for the variation of individual programs.
We generate n (hyper-parameter) offspring programs at each generation.
(cid:98) n/ 2 (cid:99) offspring are generated as a result of three operations; crossover, mutation or reproduction.
These variations are depicted in Figure 3.
For crossover, two individual programs are randomly selected from the population and mated.
Our mating function randomly selects a crossover point from each individual and exchanges the sub-trees with the selected point as root between each individual.
The ﬁrst child is appended to the offspring population.
For mutation, we randomly select a point in the individual program, and replace the sub-tree at that point by a randomly generated expression tree.
We repeat the variation algorithm on the population till (cid:98) n/ 2 (cid:99) valid individuals are generated.
To encourage diversity, we also randomly generate (cid:98) n/ 2 (cid:99) valid individuals.
We have placed static limits on the depth of all expression trees at 10.
3.6 Program Evaluation Methodology
At each generation, the ﬁtness of the entire population is invalidated and recalculated.
Calculating the ﬁtness of each program on the entire dataset (which can be approximately 1 TB) is computationally infeasible.
Further, we may want to ﬁnd generalized programs that give high ﬁtness on many different architecture design spaces and datasets.
This would translate to evaluating each individual on over[-
-]
TB of data on the NDS and NASBench-201 search spaces.
Reducing the computation by evaluating the ﬁtness of the population on a single small ﬁxed sub-set of neural networks from the search space causes discovered programs to trivially over-ﬁt to the sub-set statistics in our tests.
To address over-ﬁtting of programs to small datasets of network statistics while minimizing compute resources required for evaluating on the entire dataset of network statistics, we generate an evolution task dataset .
This is generated by randomly sampling 80 neural networks
[Table] [Table]
[Table] [Table]
from each available search space (NASBench-201 and NDS) and dataset (CIFAR-10, CIFAR-100,
ImageNet-16-120).
We evaluate individuals by randomly choosing s search spaces, and sampling 20 neural networks from each of the chosen search spaces.
We take the minimum ﬁtness achieved by the individual program on the s spaces.
We consider s as a hyper-parameter.
In our tests, this is consistently kept at 4.
3.7 Program Testing Methodology
At the end of the evolutionary search, our primary goal is to test whether the programs discovered are able to provide high ﬁtness on previously unseen neural network architectures.
We test the ﬁttest program from our ﬁnal population as well as the two ﬁttest programs encountered through-out the evolutionary search.
At test time, we take the program and ﬁnd the score-accuracy correlation over 4000 neural network architectures sampled from the NASBench-201 and NDS design spaces.
Results
As described in Algorithm 1, we search for effective ZC-NASMs on every design space from NDS
CIFAR-10 and the NASBench-201 datasets.
Each search takes approximately 24 hours on a Intel(R)
Xeon(R) Gold 6242 CPU with 1 terabyte of RAM.
We choose the most consistent ZC-NASM ( EZNAS-A , discovered by evolving programs on the NDS-DARTS CIFARsearch space) from our evolutionary search and compare it with existing ZC-NASMs from recent works.
We report both the
Kendall τ rank correlation coefﬁcient and the Spearman ρ rank-order correlation coefﬁcient to fairly compare EZNAS-A with a broad set of ZC-NASMs from recent literature.
We provide our Spearman ρ correlation on the NATS-Bench SSS search space and NDS ImageNet design spaces as well.
NASBench-201: We report our scoreaccuracy correlation by scoring all neural networks in the NASBench-201 design space.
As seen in Table 3, we obtain the highest scoreaccuracy Kendall Tau Ranking Correlation on all datasets.
We also benchmark our work against the pruning based saliency metrics formalized in (Abdelfattah et al.
[2021]) and ﬁnd that our program
EZNAS-A has the highest Spearman ρ among the ZC-NASMs in Table 1.
NDS: We report our score-accuracy correlation by scoring all neural networks in the NDS design space for the CIFAR-10 dataset and scoring 40 random neural networks over 5 seeds for the ImageNet dataset (Table 5).
We ﬁnd that EZNAS-A has the highest score accuracy Kendall Tau Ranking
Correlation on ea ch of the design spaces of NDS CIFAR-10 in Table 4.
NATS-Bench: We report our score-accuracy correlation by scoring 200 randomly sampled neural networks in the NATS-Bench TSS (same as NB-201) and SSS (Dong et al.
[2021]) design space averaged over 5 seeds for the CIFAR-10, CIFAR-100 and ImageNet16-120 dataset in Table 2.
Neural Architecture Search Integration: We integrate our ZC-NASMs EZNAS-A with the Aging
Evolution (AE) Search algorithm from (Real et al.
Our evolutionary search discovers a
ZC-NASM that is competitive in search efﬁciency with existing ZC-NASMs in Figure 4.
Examination Of Best Programs
[Figure]
EZNAS discovers a set of ZC-NASMs at the end of the evolutionary search.
In this section, we analyze two of the best ZC-NASMs discovered by our method to give a deeper understanding of the nature of programs.
We refer to these programs as EZNAS-A and EZNAS-B .
For each program, we generate random input tensors of varying sizes and average the ZC-NASM score 5000 times.
This is an approximate method to understand how the ZC-NASM responds to change in architectural parameters (kernel size, number of channels, activation size).
EZNAS-A :
The evolution task dataset for discovering this program was NDS-DARTS.
The input to the ZC-NASM in Figure 6 is the T 3 G N (Weight Gradient with Random Noise Input) tensor.
We ﬁnd that the score increases as the number of channels or depth increases, we also observe that kernel sizes of 1 and 7 give higher scores than 3 and 5 with the lowest score being assigned to kernel of size 3.
It is interesting to see that the expectation value of EZNAS-A in Figure 6 translates to a weighted form of parameter counting, with a non-linear monotonically increasing scaling of score with the number of input/output channels and a locally parabolic relationship between the score and the kernel size with the minimum score at kernel size of 3.
EZNAS-B : The evolution task dataset for discovering this program was NDS-ENAS.
The input to the
ZC-NASM in Figure 6 is T 1 G P
(Difference in pre-activation gradients for a noise perturbed data mini-batch).
We ﬁnd that the activation map size is exponentially more inﬂuential to the score when compared to the number of channels.
Through this analysis, we see that EZNAS-A & EZNAS-B generate a score which is correlated with the activation or weight sizes.
It is interesting to note that when compared to ZC-NASMs from recent literature, this form of weighted parameter counting works more consistently.
This is supported by our ﬁnding that FLOPs and Params are more generalizable than ZC-NASMs which work on the
NASBench-201 space but fail on the NDS space.
Discussion and Future Work
With the EZNAS formulation, we discover effective ZC-NASMs which generalize across design spaces and datasets, as well as give SoTA score-accuracy correlation.
This is a promising new direction, but there is much work to be done.
In this section, we discuss the current limitations of
EZNAS with scope for future work.
Program Design: To simplify the search problem, we take a mean ( aggregation_function ) of the scores generated across all ReLU-Conv2D-BatchNorm2D instances.
This formulation does not take layer connectivity patterns into account explicitly.
Extending our evolutionary search to take into account the global connectivity pattern of the neural networks or learning weighted averaging techniques for the individual layer (RCB instance) scores could help us discover better ZC-NASMs.
Further, we impose structural restrictions on our program by necessitating a ﬁxed structure ( ReLU-Conv2D-BatchNorm2D ).
We have to truncate instances of
ReLU-Conv2D-Conv2D-BatchNorm2D from the NDS space by ignoring the second convolution.
As more diverse architectures are introduced in the ﬁeld, we must re-formulate collection of network statistics in a more robust manner.
Note that NASWOT measures the similarity in the binary codes generated from the ReLU units for two different inputs.
Our program design limits us to only using a single mini-batch of inputs and does not compare inputs directly.
Thus, we would not be able to discover the NASWOT ZC-NASM metric with our approach.
Fortunately, integrating such functionality into the program design space is feasible and is an interesting line of future work to generate more complex metrics.
It is also important to have a robust and diverse set of mathematical operations to discover effective
ZC-NASMs.
We utilize 34 mathematical operations inspired by AutoML-Zero (Real et al.
and operations found in existing ZC-NASMs.
None of our operations or network statistics generation have any scalar hyper parameters.
For example, the Power operation is ﬁxed to do an element-wise square, and the noise generation for input tensor is ﬁxed to N (0 , 1) .
Optimizing these values dynamically and ensuring a sufﬁciently diverse set of operations may enable discovery of better ZC-NASMs.
Network Statistics: Due to the computational resources required to generate network statistics, we have to pre-compute them and use the generated data as an evolutionary task dataset .
We only use network statistics with input batch-size of 1 for evolution.
A single sample statistic of a neural network is insufﬁcient to describe the architecture, as evidenced by our increase in score-accuracy correlation with the batch size in Figure 9.
Further, from Figure 9 we observe high variance in the ﬁtness of a ZC-NASM when evaluated for multiple seeds with a batch size of 1.
Testing over a large number of seeds or increasing the batch-size would cause a linear increase in memory requirement as well as increase in run-time of the evolutionary search.
Ranking Architectures: It is interesting to observe that in the entire neural architecture design space, FLOPs and Params are competitive proxies and sometimes better than many existing ZCNASMs.
A deeper investigation reveals that ranking the top 10% of the neural networks in the design space is a signiﬁcantly more difﬁcult task.
In the top 10% of neural networks, FLOPs and Params are extremely weak proxies.
Further, (Abdelfattah et al.
[2021]) ﬁnds that among their ZC-NASMs, synflow is the only one which is able to serve as a weak correlator for performance in top 10% of the neural networks on the NASBench-201 CIFAR-100 and ImageNet16-120 design spaces.
However, synflow is not able to rank networks effective in the top 10% of CIFAR-10 NASBench-201 and the entire NDS design space.
EZNAS-A is able to rank neural networks robustly across both the NASBench-201 and NDS design spaces, but does not rank the top 10% of neural networks effectively.
We discover alternative
ZC-NASMs that can weakly rank the top 10% of neural networks, but such ZC-NASMs fail to generalize across neural architecture design search spaces.
This ability to identify good architectures but inability to distinguish between the best architectures may be an inherent limitation of ZC-NASMs.
Alternatively, improvements to our evolutionary framework may result in robust ZC-NASMs that can rank the top 10% neural networks effectively.
This is an interesting research direction and may require creation of more neural architecture design spaces like NASBench-201/NDS which exhibit lower correlation with FLOPs/Params.
Evolutionary Search:
In this paper, we introduce an intuitive program representation and appropriate variation methodologies to enable discovery of ZC-NASMs with minimal human intervention.
However, recent advances in the ﬁeld of Neuro-Symbolic Program Synthesis (Parisotto et al.
[2017]) to learn mappings from input-output examples (neural network statistics to neural architecture scores) can motivate improvements in our evolutionary search.
Further, exploring architecture connectivity encodings (White et al.
[2021]) for neural architecture search and discovering programs to rank neural architecture encodings may enable discovery of effective connectivity patterns to enable a deeper understanding of important features in neural architecture design beyond those contained in activations maps and weights.
Efﬁciency: EZNAS is able to discover a set of ZC-NASMs on a single CPU in under 24 hours, which translates to 358.6 g of CO2e (Lannelongue et al.
In comparison, a single NAS search (assuming 8 GPU days (Zhou et al.
[2020])) can take over 4.49 kg of CO2e.
Recent works integrate their zero-cost proxies with differentiable architecture search (Xiang et al.
[2021]) to deliver 40 × NAS speed-up.
As the architecture design spaces that ZC-NASMs need to be discovered on diversify, the efﬁciency of our methodology must be improved to scale to larger problems.
Seeding the initial population with viable candidates can enable faster discovery of robust ZC-NASMs.
The search process can be made more efﬁcient by using lower precision numerical formats, or exploring proxy tasks on smaller datasets (down-sampled image datasets, smaller neural network design architecture etc.).
Conclusion
In this paper, we present EZNAS, a novel genetic programming driven approach to discover Zero-Cost
Neural Architecture Scoring Metrics (ZC-NASMs) .
The key advantage of EZNAS is that it is an interpretable approach to discover generalizable methods to rank neural networks.
Generalizability of a ZC-NASM is crucial for its practical utility, as a robust ZC-NASM should be able to rank NNs in previously unseen neural architecture search spaces.
With our approach, we are able to discover a ZC-NASM ( EZNAS-A ) which evolved only on the NDS
DARTS space, but delivers state of the art score-accuracy correlation across both the NASBench201 and NDS design space.
We also demonstrate the generalizability of EZNAS-A by providing extremely strong correlation results on NATS-Bench as well as NDS ImageNet.
We demonstrate competitive search efﬁciency of EZNAS-A by integrating our metric with the Aging Evolution (AE) search algorithm from (Abdelfattah et al.
Further, we provide an in-depth analysis of the nature of programs we have discovered, along with a detailed account of existing limitations of our methodology to motivate future work in this direction.
EZNAS has the potential to reduce human bias in design of ZC-NASMs, and aid in discovery of programs that provide deeper insights into properties that make a neural network perform well.  Appendix 8.1 NASBench-201 and NDS
For image classiﬁcation, we utilize the NASBench-201 [7] and NDS [19] NAS search spaces for our evolutionary search as well as testing. NASBench-201 consists of 15,625 neural networks trained on the CIFAR-10, CIFAR-100 and ImageNet-16-120 datasets. Neural Networks in Network Design Spaces (NDS) uses the DARTS [13] skeleton.
The networks are comprised of cells sampled from each of AmoebaNet [20], DARTS [13], ENAS [18], NASNet [29] and PNAS [12].
There exists approximately 5000 neural network architectures in each NDS design space.
8.2 Sequential Program Representation
Our initial attempts at discovering ZC-NASMs took a different approach to program representation.
The sequential program representation described in Figure 8 posed no structural limitations on the program.
We have 22 static memory addresses, which contained network statistics and are referenced with integers 0-21 .
To store intermediate tensors generated by the program, we allocate 80 dynamic memory addresses, which can be over-written during the program execution as well.
To store intermediate scalars generated by the program, we allocate 20 memory addresses. As seen in Figure 8, we represent the programs as integers, where each instruction is expressed as 4 integers. The ﬁrst integer provides the write address, the second integer provides the operation ID and the third and fourth integers provide the read addresses for the operation.
We initialize valid random integer arrays and convert them to programs to evaluate and fetch the ﬁtness (score-accuracy correlation).
We allow Mate, InsertOP, RemoveOP, MutateOP & MutateOpArg as variation functions.
The Mate function takes two individuals, and takes the ﬁrst half of each individual.
Then, these components are interpolated to generate a new individual.
The InsertOP function inserts an operation at a random point in the program.
The RemoveOP function removes an operation at a random point in the program.
The MutateOP changes a random operation in program without changing read/write addresses.
The MutateOpArg function simply replaces one of the read arguments of any random instruction with another argument from the same address space (dynamic address argument cannot be replaced by a scalar address argument).
While we are able to discover weak ZC-NASMs with this formulation, we observe that there are too many redundancies in the programs discovered.
Program length bloating as well as operations that do not contribute to the ﬁnal output were frequently observed.
Due to these issues, the evolution time evaluation of individual ﬁtness quickly became an intractable problem.
To address this, we change our program representation to a expression tree representation in the results reported in the paper.
This representation necessitates contribution of each operation to the ﬁnal output, which means there is no redundant compute.
While the sequential program representation is valid, we believe that signiﬁcant engineering efforts are required to ensure discovery of meaningful programs.
Our sequential program representation is directly inspired by the formulation used in AutoML-Zero [21].
AutoML-Zero makes signiﬁcant approximations in the learning task to evolutionarily discover MLPs.
While AutoML-Zero has a much larger program space to search for, approximations in computing individual ﬁtness are not feasible in our formulation as generating exact score-accuracy correlation is an important factor in selecting individuals with high ﬁtness.
8.3 Noise and Perturbation for Network Statistics
To generate network statistics, we use three types of input data.
The ﬁrst is simply a single random sample from the dataset (e.g. a single image or a batch of images from
CIFAR-10).
To generate a noisy input, we simply use the default torch.randn function as input = torch.randn(data_sample.shape) .
The third type of input we pro- vide is a data-sample which has been perturbed by random noise ( input = data_sample + 0.01**0.5*torch.randn(data_sample.shape) ).
8.4 Network Initialization Seed Test
In Figure 9 (Left), we use different seeds to change the initialization and input tensors, but keep the neural architectures being sampled ﬁxed in the respective spaces.
The variance in the score accuracy correlation is much lesser than in Figure 9 (Right) where the seed also controls the neural architectures being sampled.
This shows the true variation in our EZNAS-A ZC-NASM with respect to network initialization. 8.5 Hardware used for evolution and testing Our evolutionary algorithm runs on Intel(R) Xeon(R)
Gold 6242 CPU with 630GB of RAM.
Our
RAM utilization for evolving programs on a single Image Classiﬁcation dataset was approximately 60GB.
RAM utilization can vastly vary (linearly) based on the number of neural network statistics that are being used for the evolutionary search.
Our testing to generate the statistics for the seed experiments as well as the ﬁnal Spearman ρ and Kendall Tau Rank Correlation Coefﬁcient is done on an NVIDIA DGX-2 server with 4 NVIDIA V-100 32GB GPUs.
8.6 Varying network statistics and to_scalar
In Figure 10, we detail 3 tests while evolving on the NDS_DARTS CIFAR-10 search space in an identical fashion to EZNAS-A (referred to as EZNAS-(R-C-B)-(Mean) here).
EZNAS-(C-B-R)-(Mean) and EZNAS-(C-B-R)-(L2) correspond to alternate to_scalar and network statistics collection tests respectively.
We demonstrate that while EZNAS-(R-C-B)-(Mean) is more effective, we are able to discover ZC-NASMs with all three formulations.
Hyper-
parameters for
discovering
EZNAS
-A num_generation: population_size: tournament_size: MU:lambda_ : crossover_prob: 0. mutation_prob: 0.
min_tree_depth: max_tree_depth:
[Table]
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] Our method increases the efﬁciency of NAS signiﬁcantly, and is aimed at reducing the negative societal impact of AI.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We plan to open-source our work as part of a larger framework in the future.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report error bars for the score-accuracy correlation of our discovered metric against variation of batc hsize as well as seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Crediting the curators was sufﬁcient as speciﬁed in their open-source license.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[Table] ID Operation Description
Output: C , Input: A, B OP Element-wise Sum C = A+B
OP
Element-wise Difference
C =
A-B
OP
Element-wise Product
C = A*B
OP
Matrix Multiplication
C =
A@B
OP
Lesser Than
C =
(A<B).bool()
OP
Greater Than
C =
(A>B).bool()
OP
Equal To
C =
(A==B).bool()
OP
Log
A[A<=0] =
C = torch.log(A)
OP
AbsLog
A[A==0] =
C = torch.log(torch.abs(A))
OP
Abs
C = torch.abs(A)
OP
Power
C = torch.pow(A, 2)
OP
Exp
C = torch.exp(A)
OP
Normalize
C =
(A - A mean )/A
std
C[C!=C] =
OP
ReLU
C = torch.functional.
F.relu(A)
OP
Sign
C = torch.sign(A)
OP
Heaviside
C = torch.heaviside(A, values=[0])
OP
Element-wise Invert
C = 1/A
OP
Frobenius Norm
C = torch.norm(A, p=’fro’)
OP
Determinant
C = torch.det(A)
OP
LogDeterminant
C = torch.logdet(A)
C[C!=C]=
OP
SymEigRatio
A = A.reshape(A.shape[0], -1)
A = A@A.T
A =
A + A.T e,v = torch.symeig(A, eigenvectors=
False)
C = e[-1]/e[0]
OP
EigRatio
A = A.reshape(A.shape[0], -1)
A = torch.einsum(’nc,mc->nm’, [A,A]) e,v = torch.eig(A)
C = (e[-1]/e[0])[0]
[Table]
C = torch.sum(A)/A.numel()
[Table]
torch.sum(abs(A))/A.numel()
[Table]
A = Heaviside(A)
B = Heaviside(B)
C = sum(A!=B)
[Table]
C = torch.nn.
KLDivLoss(’batchmean’)(A,B)
[Table]
A = A.reshape(A.shape[0], -1)
B =
B.reshape(B.shape[0], -1)
C = torch.nn.
CosineSimilarity()(A, B)
C = torch.sum(C)
[Table]
C = torch.functional.
F
[Table]
softmax(A)
[Table]
C = torch.functional.
F.sigmoid(A)
[Table]
C = torch.
[Table]
[Table]
= torch.
[Table]
=
[Table]
C = torch.
Tensor([A.numel()])