Original: N EURAL A RCHITECTURE S EARCH WITHOUT T RAINING
Revised: N EURAL A RCHITECTURE S EARCH WITHOUT T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The time and effort involved in hand-designing deep neural networks is immense.
Revised: The time and effort involved in hand-designing deep neural networks is immense.

Original: This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design.
Revised: This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design.

Original: However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process.
Revised: However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process.

Original: This could be remedied if we could infer a network’s trained accuracy from its initial state.
Revised: This could be remedied if we could infer a network’s trained accuracy from its initial state.

Original: In this work, we examine the correlation of linear maps induced by augmented versions of a single image in untrained networks and motivate how this can be used to give a measure which is highly indicative of a network’s trained performance.
Revised: In this work, we examine the correlation of linear maps induced by augmented versions of a single image in untrained networks and motivate how this can be used to give a measure which is highly indicative of a network’s trained performance.

Original: We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Benchand NAS-Bench-201.
Revised: We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Benchand NAS-Bench-201.

Original: Finally, we show that our approach can be readily combined with more expensive search methods for added value: we modify regularised evolutionary search to produce a novel algorithm that outperforms its predecessor.
Revised: Finally, we show that our approach can be readily combined with more expensive search methods for added value: we modify regularised evolutionary search to produce a novel algorithm that outperforms its predecessor.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The success of deep learning in computer vision is in no small part due to the insight and engineering efforts of human experts, allowing for the creation of powerful architectures for widespread adoption (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015;
Revised: The success of deep learning in computer vision is in no small part due to the insight and engineering efforts of human experts, allowing for the creation of powerful architectures for widespread adoption (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015;

Original: He et al., 2016; Szegedy et al., 2016;
Revised: He et al., 2016; Szegedy et al., 2016;

Original: Huang et al., 2017).
Revised: Huang et al., 2017).

Original: However, this manual design is costly, and becomes increasingly more difﬁcult as networks get larger and more complicated.
Revised: However, this manual design is costly, and becomes increasingly more difﬁcult as networks get larger and more complicated.

Original: Because of these challenges, the neural network community has seen a shift from designing architectures to designing algorithms that search for candidate architectures (Elsken et al., 2019; Wistuba et al., 2019).
Revised: Because of these challenges, the neural network community has seen a shift from designing architectures to designing algorithms that search for candidate architectures (Elsken et al., 2019; Wistuba et al., 2019).

Original: These Neural Architecture Search (NAS) algorithms are capable of automating the discovery of effective architectures (Zoph & Le, 2017; Zoph et al., 2018;
Revised: These Neural Architecture Search (NAS) algorithms are capable of automating the discovery of effective architectures (Zoph & Le, 2017; Zoph et al., 2018;

Original: Pham et al., 2018; Tan et al., 2019; Liu et al., 2019; Real et al., 2019).
Revised: Pham et al., 2018; Tan et al., 2019; Liu et al., 2019; Real et al., 2019).

Original: NAS algorithms are broadly based on the seminal work of Zoph & Le (2017).
Revised: NAS algorithms are broadly based on the seminal work of Zoph & Le (2017).

Original: A controller network generates an architecture proposal, which is then trained to provide a signal to the controller through
Revised: A controller network generates an architecture proposal, which is then trained to provide a signal to the controller through

Original: REINFORCE (Williams, 1992), which then produces a new proposal, and so on.
Revised: REINFORCE (Williams, 1992), which then produces a new proposal, and so on.

Original: Training a network for every controller update is extremely expensive; utilising 800 GPUs for 28 days in Zoph & Le (2017).
Revised: Training a network for every controller update is extremely expensive; utilising 800 GPUs for 28 days in Zoph & Le (2017).

Original: Subsequent work has sought to ameliorate this by (i) learning stackable cells instead of whole networks (Zoph et al., 2018) and (ii) incorporating weight sharing ; allowing candidate networks to share weights to allow for joint training (Pham et al., 2018).
Revised: Subsequent work has sought to ameliorate this by (i) learning stackable cells instead of whole networks (Zoph et al., 2018) and (ii) incorporating weight sharing ; allowing candidate networks to share weights to allow for joint training (Pham et al., 2018).

Original: These contributions have accelerated the speed of NAS algorithms e.g. to half a day on a single GPU in Pham et al.
Revised: These contributions have accelerated the speed of NAS algorithms e.g. to half a day on a single GPU in Pham et al.

Original: For some practitioners, NAS is still too slow; being able to perform NAS quickly (i.e. in seconds) would be immensely useful in the hardware-aware setting where a separate search is typically required for each device and task (Wu et al., 2019; Tan et al., 2019).
Revised: For some practitioners, NAS is still too slow; being able to perform NAS quickly (i.e. in seconds) would be immensely useful in the hardware-aware setting where a separate search is typically required for each device and task (Wu et al., 2019; Tan et al., 2019).

Original: Moreover, recent works have scrutinised
Revised: Moreover, recent works have scrutinised

Original: NAS with weight sharing (Li & Talwalkar, 2019; Yu et al., 2020); there is continued debate as to whether it is clearly better than simple random search.
Revised: NAS with weight sharing (Li & Talwalkar, 2019; Yu et al., 2020); there is continued debate as to whether it is clearly better than simple random search.

Original: The issues of cost and time, and the risks of weight sharing could be avoided entirely if a NAS algorithm did not require any network training .
Revised: The issues of cost and time, and the risks of weight sharing could be avoided entirely if a NAS algorithm did not require any network training .

Original: In this paper, we show that this can be achieved.
Revised: In this paper, we show that this can be achieved.

Original: We explore two recently released NAS benchmarks, NAS-Bench-101 (Ying et al., 2019), and NASBench-201 (Dong & Yang, 2020) and examine the relationship between the linear maps induced by an untrained network for a minibatch of augmented versions of a single image (Section 3).
Revised: We explore two recently released NAS benchmarks, NAS-Bench-101 (Ying et al., 2019), and NASBench-201 (Dong & Yang, 2020) and examine the relationship between the linear maps induced by an untrained network for a minibatch of augmented versions of a single image (Section 3).

Original: These maps are easily computed using the Jacobian.
Revised: These maps are easily computed using the Jacobian.

Original: The correlations between these maps (which we denote
Revised: The correlations between these maps (which we denote

Original: [Figure]
Revised: [Figure]

Original: by Σ J ) are distinctive for networks that perform well when trained on both NAS-Benches; this is immediately apparent from visualisation alone (Figure 1).
Revised: by Σ J ) are distinctive for networks that perform well when trained on both NAS-Benches; this is immediately apparent from visualisation alone (Figure 1).

Original: We devise a score based on Σ J and perform an ablation study to demonstrate its robustness to inputs and network initialisation.
Revised: We devise a score based on Σ J and perform an ablation study to demonstrate its robustness to inputs and network initialisation.

Original: We incorporate our score into a simple search algorithm that doesn’t require training (Section 4).
Revised: We incorporate our score into a simple search algorithm that doesn’t require training (Section 4).

Original: This allows us to perform architecture search quickly, for example, on CIFAR-10 (Krizhevsky, 2009) we are able to search for a network that achieve 93 .
Revised: This allows us to perform architecture search quickly, for example, on CIFAR-10 (Krizhevsky, 2009) we are able to search for a network that achieve 93 .

Original: 36% accuracy in 29 seconds within the
Revised: 36% accuracy in 29 seconds within the

Original: NAS-Bench-201 search space; several orders of magnitude faster than traditional NAS methods for a modest change in ﬁnal accuracy (e.g. REINFORCE ﬁnds a 93 .
Revised: NAS-Bench-201 search space; several orders of magnitude faster than traditional NAS methods for a modest change in ﬁnal accuracy (e.g. REINFORCE ﬁnds a 93 .

Original: 85% net in 12000 seconds).
Revised: 85% net in 12000 seconds).

Original: Finally, we show that we can combine our approach with regularised evolutionary search (REA, Pham et al., 2018) to produce a new NAS algorithm, Assisted-REA (AREA) that outperforms its precedessor, attaining 94 .
Revised: Finally, we show that we can combine our approach with regularised evolutionary search (REA, Pham et al., 2018) to produce a new NAS algorithm, Assisted-REA (AREA) that outperforms its precedessor, attaining 94 .

Original: 16% accuracy on NAS-Bench-101 in 12,000 seconds.
Revised: 16% accuracy on NAS-Bench-101 in 12,000 seconds.

Original: Code for reproducing our experiments is available in the supplementary material.
Revised: Code for reproducing our experiments is available in the supplementary material.

Original: We believe this work is an important proof-of-concept for NAS without training, and shows that the large resource costs associated with NAS can be avoided.
Revised: We believe this work is an important proof-of-concept for NAS without training, and shows that the large resource costs associated with NAS can be avoided.

Original: The beneﬁt is two-fold, as we also show that we can integrate our approach into existing NAS techniques for scenarios where obtaining as high an accuracy as possible is of the essence.
Revised: The beneﬁt is two-fold, as we also show that we can integrate our approach into existing NAS techniques for scenarios where obtaining as high an accuracy as possible is of the essence.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Designing a neural architecture by hand is a challenging and time-consuming task.
Revised: Designing a neural architecture by hand is a challenging and time-consuming task.

Original: It is extremely difﬁcult to intuit where to place connections, or which operations to use.
Revised: It is extremely difﬁcult to intuit where to place connections, or which operations to use.

Original: This has prompted an abundance of research into neural architecture search (NAS); the automation of the network design process.
Revised: This has prompted an abundance of research into neural architecture search (NAS); the automation of the network design process.

Original: In the pioneering work of Zoph & Le (2017), the authors use an RNN controller to generate descriptions are
Revised: In the pioneering work of Zoph & Le (2017), the authors use an RNN controller to generate descriptions are

Original: and used to controller using reinforcement learning to improve the quality of the candidates it generates.
Revised: and used to controller using reinforcement learning to improve the quality of the candidates it generates.

Original: This algorithm is very expensive: searching for an architecture to classify CIFAR-10 required 800 GPUs for 28 days.
Revised: This algorithm is very expensive: searching for an architecture to classify CIFAR-10 required 800 GPUs for 28 days.

Original: It is also inﬂexible; the ﬁnal network obtained is ﬁxed and cannot be scaled e.g. for use on mobile devices or for other datasets.
Revised: It is also inﬂexible; the ﬁnal network obtained is ﬁxed and cannot be scaled e.g. for use on mobile devices or for other datasets.

Original: The subsequent work of Zoph et al.
Revised: The subsequent work of Zoph et al.

Original: (2018) deals with these limitations.
Revised: (2018) deals with these limitations.

Original: Inspired by the modular nature of successful hand-designed networks (Simonyan & Zisserman, 2015; He et al., 2016; Huang et al., 2017), they propose searching over neural building blocks, instead of over whole architectures.
Revised: Inspired by the modular nature of successful hand-designed networks (Simonyan & Zisserman, 2015; He et al., 2016; Huang et al., 2017), they propose searching over neural building blocks, instead of over whole architectures.

Original: These building blocks, or cells , form part of a ﬁxed overall network structure.
Revised: These building blocks, or cells , form part of a ﬁxed overall network structure.

Original: Speciﬁcally, the authors learn a standard cell, and a reduced cell (incorporating pooling) for CIFAR-10 classiﬁcation.
Revised: Speciﬁcally, the authors learn a standard cell, and a reduced cell (incorporating pooling) for CIFAR-10 classiﬁcation.

Original: These are then used as the building blocks of a larger network for ImageNet (Russakovsky et al., 2015) classiﬁcation.
Revised: These are then used as the building blocks of a larger network for ImageNet (Russakovsky et al., 2015) classiﬁcation.

Original: While more ﬂexible—the number of cells can be adjusted according to budget— and cheaper, owing to a smaller search space, this technique still utilises 500 GPUs across 4 days.
Revised: While more ﬂexible—the number of cells can be adjusted according to budget— and cheaper, owing to a smaller search space, this technique still utilises 500 GPUs across 4 days.

Original: ENAS (Pham et al., 2018) reduces the computational cost of searching by allowing multiple candidate architectures to share weights.
Revised: ENAS (Pham et al., 2018) reduces the computational cost of searching by allowing multiple candidate architectures to share weights.

Original: This facilitates the simultaneous training of candidates, reducing the search time on CIFAR-10 to half a day on a single GPU.
Revised: This facilitates the simultaneous training of candidates, reducing the search time on CIFAR-10 to half a day on a single GPU.

Original: Weight sharing has seen widespread adoption in a host of NAS algorithms (Liu et al., 2019; Luo et al., 2018;
Revised: Weight sharing has seen widespread adoption in a host of NAS algorithms (Liu et al., 2019; Luo et al., 2018;

Original: Cai et al., 2019; Xie et al., 2019;
Revised: Cai et al., 2019; Xie et al., 2019;

Original: Brock et al., 2018).
Revised: Brock et al., 2018).

Original: However, there is evidence that it inhibits the search for optimal architectures (Yu et al., 2020).
Revised: However, there is evidence that it inhibits the search for optimal architectures (Yu et al., 2020).

Original: Moreover, random search proves to be an extremely effective NAS baseline (Yu et al., 2020; Li & Talwalkar, 2019).
Revised: Moreover, random search proves to be an extremely effective NAS baseline (Yu et al., 2020; Li & Talwalkar, 2019).

Original: This exposes another problem: the search space is still vast—there are 1 .
Revised: This exposes another problem: the search space is still vast—there are 1 .

Original: 6 × 10 29 possible architectures in Pham et al.
Revised: 6 × 10 29 possible architectures in Pham et al.

Original: (2018) for example—that it is impossible to isolate the best networks and demonstrate that NAS algorithms ﬁnd them.
Revised: (2018) for example—that it is impossible to isolate the best networks and demonstrate that NAS algorithms ﬁnd them.

Original: An orthogonal direction for identifying good architectures is the estimation of accuracy prior to training (Deng et al., 2017; Istrate et al., 2019), although these differ from this work in that they rely on training a predictive model, rather than investigating more fundamental architectural properties.
Revised: An orthogonal direction for identifying good architectures is the estimation of accuracy prior to training (Deng et al., 2017; Istrate et al., 2019), although these differ from this work in that they rely on training a predictive model, rather than investigating more fundamental architectural properties.

Original: 2. NAS B ENCHMARKS
Revised: 2. NAS B ENCHMARKS

Original: A major barrier to evaluating the effectiveness of a NAS algorithm is that the search space (the set of all possible networks) is too large for exhaustive evaluation.
Revised: A major barrier to evaluating the effectiveness of a NAS algorithm is that the search space (the set of all possible networks) is too large for exhaustive evaluation.

Original: Moreover, popular search spaces have been shown to be over-engineered, exhibiting little variety in their trained networks (Yang et al., 2020).
Revised: Moreover, popular search spaces have been shown to be over-engineered, exhibiting little variety in their trained networks (Yang et al., 2020).

Original: This has led to the creation of several benchmarks (Ying et al., 2019; Zela et al., 2020;
Revised: This has led to the creation of several benchmarks (Ying et al., 2019; Zela et al., 2020;

Original: Dong & Yang, 2020) that consist of tractable NAS search spaces, and metadata for the training of networks within that search space.
Revised: Dong & Yang, 2020) that consist of tractable NAS search spaces, and metadata for the training of networks within that search space.

Original: Concretely, this means that it is now possible to determine whether an algorithm is able to search for a good network.
Revised: Concretely, this means that it is now possible to determine whether an algorithm is able to search for a good network.

Original: In this work we utilise NAS-Bench-101 (Ying et al., 2019) and NAS-Bench-201 (Dong & Yang, 2020) to evaluate the effectiveness of our approach.
Revised: In this work we utilise NAS-Bench-101 (Ying et al., 2019) and NAS-Bench-201 (Dong & Yang, 2020) to evaluate the effectiveness of our approach.

Original: NAS-Bench-101 consists of 423,624 neural networks that have been trained exhaustively, with three different initialisations, on the CIFAR-10 dataset for 108 epochs.
Revised: NAS-Bench-101 consists of 423,624 neural networks that have been trained exhaustively, with three different initialisations, on the CIFAR-10 dataset for 108 epochs.

Original: NAS-Bench-201 consists of 15, networks trained multiple times on CIFAR-10, CIFAR-100, and ImageNet-16-120 (Chrabaszcz et al., 2017).
Revised: NAS-Bench-201 consists of 15, networks trained multiple times on CIFAR-10, CIFAR-100, and ImageNet-16-120 (Chrabaszcz et al., 2017).

Original: Both benchmarks are described in detail in Appendix B.
Revised: Both benchmarks are described in detail in Appendix B.

Original:  S CORING N ETWORKS AT I NITIALISATION
Revised:  S CORING N ETWORKS AT I NITIALISATION

Original: Our goal is to devise a means to score a network architecture at initialisation in a way that is indicative of its ﬁnal trained accuracy.
Revised: Our goal is to devise a means to score a network architecture at initialisation in a way that is indicative of its ﬁnal trained accuracy.

Original: This can either replace the expensive inner-loop training step in NAS, or better direct exploration in existing NAS algorithms.
Revised: This can either replace the expensive inner-loop training step in NAS, or better direct exploration in existing NAS algorithms.

Original: Given a neural network with rectiﬁed linear units, we can, at each unit in each layer, identify a binary indicator as to whether the unit is inactive (the value is negative and hence is multiplied by zero) or active (in which case its value is multiplied by one).
Revised: Given a neural network with rectiﬁed linear units, we can, at each unit in each layer, identify a binary indicator as to whether the unit is inactive (the value is negative and hence is multiplied by zero) or active (in which case its value is multiplied by one).

Original: Fixing these indicator variables, it is well known that the network is now locally deﬁned by a linear operator (Hanin & Rolnick, 2019); this operator is obtained by multiplying the linear maps at each layer interspersed with the binary rectiﬁcation units.
Revised: Fixing these indicator variables, it is well known that the network is now locally deﬁned by a linear operator (Hanin & Rolnick, 2019); this operator is obtained by multiplying the linear maps at each layer interspersed with the binary rectiﬁcation units.

Original: Consider a minibatch of data X
Revised: Consider a minibatch of data X

Original: = { x i } Ni =1 .
Revised: = { x i } Ni =1 .

Original: Let us denote the linear map for input x i ∈ R D by column vector w i , which maps the input through the network f ( x i ) to a ﬁnal choice of scalar representation z i ∈ R 1 .
Revised: Let us denote the linear map for input x i ∈ R D by column vector w i , which maps the input through the network f ( x i ) to a ﬁnal choice of scalar representation z i ∈ R 1 .

Original: This linear map can be easily computed using the Jacobian w i =
Revised: This linear map can be easily computed using the Jacobian w i =

Original: ∂f ( x i ) ∂ x .
Revised: ∂f ( x i ) ∂ x .

Original: How differently a network acts at each data point can be summarised by comparing the corresponding local linear operators.
Revised: How differently a network acts at each data point can be summarised by comparing the corresponding local linear operators.

Original: Correlated operators for nearby points (such as small perturbations from a training point) relate to a potential difﬁculty in handling the two points differently during learning.
Revised: Correlated operators for nearby points (such as small perturbations from a training point) relate to a potential difﬁculty in handling the two points differently during learning.

Original: The Frobenius inner product Tr[( w i − µ i ) T ( w j − µ j )] provides a natural basis for deﬁning how two linear operators corresponding to data points x i and x j covary ( µ are mean Jacobian elements, and usually close to zero).
Revised: The Frobenius inner product Tr[( w i − µ i ) T ( w j − µ j )] provides a natural basis for deﬁning how two linear operators corresponding to data points x i and x j covary ( µ are mean Jacobian elements, and usually close to zero).

Original: We can examine the correspondences for the whole minibatch by computing
Revised: We can examine the correspondences for the whole minibatch by computing

Original: [Equation]
Revised: [Equation]

Original: and observing the covariance matrix C J =
Revised: and observing the covariance matrix C J =

Original: ( J − M J )
Revised: ( J − M J )

Original: ( J − M J ) T where M J is the matrix with entries ( M J ) i,t = 1 D (cid:80)
Revised: ( J − M J ) T where M J is the matrix with entries ( M J ) i,t = 1 D (cid:80)

Original: Dd =1 J i,d ∀ t , where d, t index over the D elements of each input (i.e. channels × pixels).
Revised: Dd =1 J i,d ∀ t , where d, t index over the D elements of each input (i.e. channels × pixels).

Original: It is more salient to focus on the the correlation matrix Σ J as the appropriate scaling in input space around each point is arbitrary.
Revised: It is more salient to focus on the the correlation matrix Σ J as the appropriate scaling in input space around each point is arbitrary.

Original: The ( i, j ) th element of Σ J is given by ( Σ J ) i,j = (
Revised: The ( i, j ) th element of Σ J is given by ( Σ J ) i,j = (

Original: [Equation]
Revised: [Equation]

Original: J ) i,j √ ( C J )
Revised: J ) i,j √ ( C J )

Original: i,i (
Revised: i,i (

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: j .
Revised: j .

Original: We want an untrained neural network to be sufﬁciently ﬂexible to model a complex target function.
Revised: We want an untrained neural network to be sufﬁciently ﬂexible to model a complex target function.

Original: However, we also want a network to be invariant to small perturbations.
Revised: However, we also want a network to be invariant to small perturbations.

Original: These two requirements are antagonistic.
Revised: These two requirements are antagonistic.

Original: For an untrained neural network to be sufﬁciently ﬂexible it would need to be able to distinguish the local linear operators associated with each data point: if two are the same then the two points are coupled.
Revised: For an untrained neural network to be sufﬁciently ﬂexible it would need to be able to distinguish the local linear operators associated with each data point: if two are the same then the two points are coupled.

Original: To be invariant to small perturbations the same local linear operators would need to be weakly coupled.
Revised: To be invariant to small perturbations the same local linear operators would need to be weakly coupled.

Original: Ideally a network would have low correlated local maps associated with each data point to be able to model each local region.
Revised: Ideally a network would have low correlated local maps associated with each data point to be able to model each local region.

Original: We empirically demonstrate this by computing Σ J for a random subset of NAS-Bench101 (Ying et al., 2019) and NAS-Bench-201 (Dong & Yang, 2020) networks at initialisation for a minibatch of a single CIFAR-10 image replicated 256 times with a different cutout (DeVries & Taylor, 2017) perturbation applied to each replicant.
Revised: We empirically demonstrate this by computing Σ J for a random subset of NAS-Bench101 (Ying et al., 2019) and NAS-Bench-201 (Dong & Yang, 2020) networks at initialisation for a minibatch of a single CIFAR-10 image replicated 256 times with a different cutout (DeVries & Taylor, 2017) perturbation applied to each replicant.

Original: We use torchvision.transforms.
Revised: We use torchvision.transforms.

Original: RandomErasing(p=0.9, scale=(0.02, 0.04)) in our experiments.
Revised: RandomErasing(p=0.9, scale=(0.02, 0.04)) in our experiments.

Original: To form J we ﬂatten the Jacobian for each input (so D = 3 × 32 × 32 = 3072 ), and adjust the ﬁnal classiﬁer layer to output a scalar.
Revised: To form J we ﬂatten the Jacobian for each input (so D = 3 × 32 × 32 = 3072 ), and adjust the ﬁnal classiﬁer layer to output a scalar.

Original: The plots of the histograms of Σ J for different networks, categorised according to the validation accuracy when trained is given in Figure 1 for a sample of networks in both benchmarks.
Revised: The plots of the histograms of Σ J for different networks, categorised according to the validation accuracy when trained is given in Figure 1 for a sample of networks in both benchmarks.

Original: Further plots are given in Appendix A.
Revised: Further plots are given in Appendix A.

Original: The histograms are very distinct: high performing networks in both benchmarks have their mass tightly around zero with a small positive skew.
Revised: The histograms are very distinct: high performing networks in both benchmarks have their mass tightly around zero with a small positive skew.

Original: We can therefore use these histograms to predict the ﬁnal performance of untrained networks, in place of the expensive training step in NAS.
Revised: We can therefore use these histograms to predict the ﬁnal performance of untrained networks, in place of the expensive training step in NAS.

Original: Speciﬁcally, we score networks by counting the entries in Σ J that lie between 0 and an small upper bound β .
Revised: Speciﬁcally, we score networks by counting the entries in Σ J that lie between 0 and an small upper bound β .

Original: A Σ J where inputs are marginally positively-correlated will have a higher score.
Revised: A Σ J where inputs are marginally positively-correlated will have a higher score.

Original: Our score is given by
Revised: Our score is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]
Revised: [Figure]

Original: where 1 is the indicator function.
Revised: where 1 is the indicator function.

Original: In this work we set β = 14 .
Revised: In this work we set β = 14 .

Original: An overview is provided in Figure 2.
Revised: An overview is provided in Figure 2.

Original: To be clear, we are not claiming this score is particularly optimal; rather we use it to demonstrate that there are ways of scoring untrained networks that provide signiﬁcant value for architecture search.
Revised: To be clear, we are not claiming this score is particularly optimal; rather we use it to demonstrate that there are ways of scoring untrained networks that provide signiﬁcant value for architecture search.

Original: We sample 1000 different architectures at random from NAS-Bench-101 and NAS-Bench-201 and plot our score on the untrained network versus their validation accuracies when trained for the datasets in these benchmarks in Figure 3.
Revised: We sample 1000 different architectures at random from NAS-Bench-101 and NAS-Bench-201 and plot our score on the untrained network versus their validation accuracies when trained for the datasets in these benchmarks in Figure 3.

Original: In all cases there is a strong correlation between our score and the ﬁnal accuracy, although it is noisier for ImageNet-16-120; this dataset has smaller images compared to the other datasets so it may be that different cutout parameters would improve this.
Revised: In all cases there is a strong correlation between our score and the ﬁnal accuracy, although it is noisier for ImageNet-16-120; this dataset has smaller images compared to the other datasets so it may be that different cutout parameters would improve this.

Original: In
Revised: In

Original: Section 4 we demonstrate how our score can be used in a NAS algorithm for extremely fast search.
Revised: Section 4 we demonstrate how our score can be used in a NAS algorithm for extremely fast search.

Original: 3. A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: How important is the image used to compute the score?
Revised: How important is the image used to compute the score?

Original: Our score relies on a single image being repeatedly augmented to form a minibatch.
Revised: Our score relies on a single image being repeatedly augmented to form a minibatch.

Original: To determine how important the image is we randomly select 9 architectures in NAS-Bench-201 and compute the score separately for 10 random CIFARimages.
Revised: To determine how important the image is we randomly select 9 architectures in NAS-Bench-201 and compute the score separately for 10 random CIFARimages.

Original: The resulting box-and-whisker plot is given in Figure 4(left): different architectures vary in
Revised: The resulting box-and-whisker plot is given in Figure 4(left): different architectures vary in

Original: [Figure]
Revised: [Figure]

Original: Uninitialised network

Original: [Figure]

Original: [Figure]

Original: c

Original: [Figure]
Revised: [Figure]

Original: e

Original: CIFAR-10 images
Revised: CIFAR-10 images

Original: [Figure]
Revised: [Figure]

Original: Uninitialised network
Revised: Uninitialised network

Original: [Figure]
Revised: [Figure]

Original: Random images sensitivity, although the ranking of the scores is reasonably robust to the speciﬁc choice of image.
Revised: Random images sensitivity, although the ranking of the scores is reasonably robust to the speciﬁc choice of image.

Original: In Figure 4(right) we compute our score using normally distributed random inputs, which seems to have little impact on the general trend.
Revised: In Figure 4(right) we compute our score using normally distributed random inputs, which seems to have little impact on the general trend.

Original: This leads us to believe the score captures a property of the network architecture, rather than something data-speciﬁc.
Revised: This leads us to believe the score captures a property of the network architecture, rather than something data-speciﬁc.

Original: Does the score change for different intialisations?
Revised: Does the score change for different intialisations?

Original: Figure 7(bottom-left) in Appendix C shows the score for different initial weights.
Revised: Figure 7(bottom-left) in Appendix C shows the score for different initial weights.

Original: This has a small effect on the score, supporting the idea that we are quantifying an architectural property.
Revised: This has a small effect on the score, supporting the idea that we are quantifying an architectural property.

Original: Does the size of the minibatch matter?
Revised: Does the size of the minibatch matter?

Original: The size of Σ J scales with minibatch size.
Revised: The size of Σ J scales with minibatch size.

Original: As our score sums entries of this matrix, we need to normalise our score to compare different minibatch sizes.
Revised: As our score sums entries of this matrix, we need to normalise our score to compare different minibatch sizes.

Original: Figure 7(bottom-right) in Appendix C presents this normalised score for different minibatch sizes, which shows little variation.
Revised: Figure 7(bottom-right) in Appendix C presents this normalised score for different minibatch sizes, which shows little variation.

Original: What cell types score highly?
Revised: What cell types score highly?

Original: We were unable to detect any obvious patterns in the types of network architecture our score ranks highly.
Revised: We were unable to detect any obvious patterns in the types of network architecture our score ranks highly.

Original: High scoring networks contained a range of operations, and connectivities.
Revised: High scoring networks contained a range of operations, and connectivities.

Original: We leave a ﬁne-grained analysis of this for future work.
Revised: We leave a ﬁne-grained analysis of this for future work.

Original:  N EURAL A RCHITECTURE S EARCH WITHOUT T RAINING - NASWOT
Revised:  N EURAL A RCHITECTURE S EARCH WITHOUT T RAINING - NASWOT

Original: evolutionary search to select an architecture ( runs), Random selects one architecture (500 runs).
Revised: evolutionary search to select an architecture ( runs), Random selects one architecture (500 runs).

Original: AREAL (assisted-REAL) uses our score to select the starting population for REAL (50 runs).
Revised: AREAL (assisted-REAL) uses our score to select the starting population for REAL (50 runs).

Original: Search times for REAL and AREAL were calculated using the NAS-Bench-101 API.
Revised: Search times for REAL and AREAL were calculated using the NAS-Bench-101 API.

Original: [Table]
Revised: [Table]

Original: In Section 3 we derived a score for cheaply ranking networks at initialisation based on their expected performance (Equation 2).
Revised: In Section 3 we derived a score for cheaply ranking networks at initialisation based on their expected performance (Equation 2).

Original: Here as a proof of concept, we integrate this score into a simple search algorithm and evaluate its ability to alleviate the need for training in NAS.
Revised: Here as a proof of concept, we integrate this score into a simple search algorithm and evaluate its ability to alleviate the need for training in NAS.

Original: Code for reproducing our experiments is available in the supplementary material.
Revised: Code for reproducing our experiments is available in the supplementary material.

Original: Many NAS algorithms are based on that of Zoph & Le (2017), which is illustrated in Algorithm 1.
Revised: Many NAS algorithms are based on that of Zoph & Le (2017), which is illustrated in Algorithm 1.

Original: It consists of learning a generator network which proposes an architecture.
Revised: It consists of learning a generator network which proposes an architecture.

Original: The weights of the generator are learnt by training the networks it generates, either on a proxy task or on the dataset itself, and using their trained accuracies as signal through e.g. REINFORCE (Williams, 1992).
Revised: The weights of the generator are learnt by training the networks it generates, either on a proxy task or on the dataset itself, and using their trained accuracies as signal through e.g. REINFORCE (Williams, 1992).

Original: This is repeated until the generator is
Revised: This is repeated until the generator is

Original: Algorithm generator = GeneratorNet() for i=1:N do net = generator.generate() trained net = net.train() (cid:46) Training a net every step is expensive generator.update(trained net) chosen net = generator.generate()
Revised: Algorithm generator = GeneratorNet() for i=1:N do net = generator.generate() trained net = net.train() (cid:46) Training a net every step is expensive generator.update(trained net) chosen net = generator.generate()

Original: Algorithm generator =
Revised: Algorithm generator =

Original: RandomGenerator() best net, best score =
Revised: RandomGenerator() best net, best score =

Original: None,for i=1:N do net = generator.generate() score =
Revised: None,for i=1:N do net = generator.generate() score =

Original: net.score
Revised: net.score

Original: () if score > best score then best net, best score = net, score chosen net = best network trained; it then produces a ﬁnal network which is the output of this algorithm.
Revised: () if score > best score then best net, best score = net, score chosen net = best network trained; it then produces a ﬁnal network which is the output of this algorithm.

Original: The vast majority of the cost is incurred by having to train candidate architectures for every single controller update.
Revised: The vast majority of the cost is incurred by having to train candidate architectures for every single controller update.

Original: Note that there exist alternative schema utilising e.g. evolutionary algorithms such as (Real et al., 2019) or bilevel optimisation (Liu et al., 2019) but all involve a training element.
Revised: Note that there exist alternative schema utilising e.g. evolutionary algorithms such as (Real et al., 2019) or bilevel optimisation (Liu et al., 2019) but all involve a training element.

Original: We instead propose a simple alternative—NASWOT—illustrated in Algorithm 2.
Revised: We instead propose a simple alternative—NASWOT—illustrated in Algorithm 2.

Original: Instead of having a neural network as a generator, we randomly propose a candidate from the search space and then rather than training it, we score it in its untrained state using Equation 2.
Revised: Instead of having a neural network as a generator, we randomly propose a candidate from the search space and then rather than training it, we score it in its untrained state using Equation 2.

Original: We do this N times—i.e. we have a sample size of N architectures—and then output the highest scoring network.
Revised: We do this N times—i.e. we have a sample size of N architectures—and then output the highest scoring network.

Original: As scoring only requires computing gradients for a single minibatch, this takes very little time.
Revised: As scoring only requires computing gradients for a single minibatch, this takes very little time.

Original:  We compare NASWOT to 12000 seconds of REAL—our implementation of REA (Real et al., 2019)— and random selection on NAS-Bench-101 (Ying et al., 2019) in Table 1.
Revised: NAS-Bench-101. We compare NASWOT to 12000 seconds of REAL—our implementation of REA (Real et al., 2019)—and random selection on NAS-Bench-101 (Ying et al., 2019) in Table 1.

Original: We can see that we are able to ﬁnd a network with a ﬁnal accuracy within a percent of REAL in under a minute on a single GPU.
Revised: We can see that we are able to ﬁnd a network with a ﬁnal accuracy within a percent of REAL in under a minute on a single GPU.

Original: NAS-Bench-201 (Dong & Yang, 2020) benchmarks a wide range of NAS algorithms, both with and without weight sharing, that wecan compare NASWOT against.
Revised: NAS-Bench-201. Dong & Yang (2020) benchmarks a wide range of NAS algorithms, both with and without weight sharing, that we compare to NASWOT.

Original: The weight sharing methods are random search with parameter sharing (RSPS, Li & Talwalkar, 2019), ﬁrst-order DARTS (DARTSV1, Liu et al., 2019), second order DARTS (DARTS-V2, Liu et al., 2019), GDAS (Dong & Yang, 2019b), SETN (Dong & Yang, 2019a), and ENAS (Pham et al., 2018).
Revised: The weight sharing methods are random search with parameter sharing (RSPS, Li & Talwalkar, 2019), ﬁrst-order DARTS (DARTS-V1, Liu et al., 2019), second order DARTS (DARTS-V2, Liu et al., 2019), GDAS (Dong & Yang, 2019b), SETN (Dong & Yang, 2019a), and ENAS (Pham et al., 2018).

Original: The non-weight sharing methods are random search with training (RS), REA (Real et al., 2019), REINFORCE (Williams, 1992), and BOHB (Falkner et al., 2018).
Revised: The non-weight sharing methods are random search with training (RS), REA (Real et al., 2019), REINFORCE (Williams, 1992), and BOHB (Falkner et al., 2018).

Original: For implementation details we refer the reader to Dong & Yang (2020).
Revised: For implementation details we refer the reader to Dong & Yang (2020).

Original: It is important to note that the hyperparameter setup in NAS-Bench-201 forboth training and search is ﬁxed —it is possible that these results are not  invariant to hyperparameter choices, which may well reﬂect the low performance of e.g. DARTS.
Revised: The hyperparameters in NAS-Bench-201 for training and search are ﬁxed — these results may not be invariant to hyperparameter choices, which may explain the low performance of e.g. DARTS.

Original: We report results on the validation and test sets of CIFAR-10, CIFAR-100, and ImageNet-16in Table 2.
Revised: We report results on the validation and test sets of CIFAR-10, CIFAR-100, and ImageNet-16in Table 2.

Original: Search times are reported for CIFAR-10 on a single GeForce GTX 1080 Ti GPU.
Revised: Search times are reported for CIFAR-10 on a single GeForce GTX 1080 Ti GPU.

Original: As per the NAS-Bench-201 setup, the non-weight sharing methods are given a time budget ofseconds.
Revised: As per the NAS-Bench-201 setup, the non-weight sharing methods are given a time budget ofseconds.

Original: For NASWOT and the non-weight sharing methods, accuracies are averaged over 500 runs.
Revised: For NASWOT and the non-weight sharing methods, accuracies are averaged over 500 runs.

Original: For weight-sharing methods, accuracies are reported over 3 runs.
Revised: For weight-sharing methods, accuracies are reported over 3 runs.

Original: We report NASWOT for sample sizes of N =10 and N =100.
Revised: We report NASWOT for sample sizes of N =10 and N =100.

Original: With the exception of GDAS (Dong & Yang, 2019b), NASWOT is able to outperform all of the weight sharing methods while requiring a fraction of the search time.
Revised: With the exception of GDAS (Dong & Yang, 2019b), NASWOT is able to outperform all of the weight sharing methods while requiring a fraction of the search time.

Original: The non-weight sharing methods do outperform NASWOT, though they also incur a large search time cost.
Revised: The non-weight sharing methods do outperform NASWOT, though they also incur a large search time cost.

Original: REA for instance, also requires a large memory budget due to the maintenance of a population of parallel architectures which may suffer from scalability issues as model and dataset sizes increase.
Revised: REA for instance, also requires a large memory budget due to the maintenance of a population of parallel architectures which may suffer from scalability issues as model and dataset sizes increase.

Original: For CIFAR-10 and CIFAR-100 NASWOT is able to ﬁnd networks with performance close to the best non-weight sharing algorithms, suggesting that network architectures themselves contain almost as much information about ﬁnal performance at initialisation as after training.
Revised: For CIFAR-10 and CIFAR-100 NASWOT is able to ﬁnd networks with performance close to the best non-weight sharing algorithms, suggesting that network architectures themselves contain almost as much information about ﬁnal performance at initialisation as after training.

Original: Table 2 also shows the effect of sample size ( N ).
Revised: Table 2 also shows the effect of sample size ( N ).

Original: We show the accuracy of networks chosen by our method for each N .
Revised: We show the accuracy of networks chosen by our method for each N .

Original: We list optimal accuracy for each N , and random selection over the whole benchmark, both averaged over 500 runs.
Revised: We list optimal accuracy for each N , and random selection over the whole benchmark, both averaged over 500 runs.

Original: We observe that sample size increases NASWOT performance.
Revised: We observe that sample size increases NASWOT performance.

Original: A key practical beneﬁt of NASWOT is its execution time.
Revised: A key practical beneﬁt of NASWOT is its execution time.

Original: Where time is a constraint, our method is very appealing.
Revised: Where time is a constraint, our method is very appealing.

Original: This may be important when repeating NAS several times, for instance for several hardware devices or datasets.
Revised: This may be important when repeating NAS several times, for instance for several hardware devices or datasets.

Original: This affords us the ability in future to specialise neural architectures for a task and resource environment cheaply, demanding only a few seconds per setup.
Revised: This affords us the ability in future to specialise neural architectures for a task and resource environment cheaply, demanding only a few seconds per setup.

Original: A visualisation of the time/accuracy trade-off for different mehods is shown in Appendix D.
Revised: A visualisation of the time/accuracy trade-off for different mehods is shown in Appendix D.

Original: [Table]
Revised: [Table]

Original:  A SSISTED R EGULARISED EA - AREA
Revised:  A SSISTED R EGULARISED EA - AREA

Original: Our proposed score can also be incorporated into existing NAS algorithms.
Revised: Our proposed score can also be incorporated into existing NAS algorithms.

Original: To demonstrate this we implemented a variant of REA (Real et al., 2019), which we call Assisted-REA (AREA).
Revised: To demonstrate this we implemented a variant of REA (Real et al., 2019), which we call Assisted-REA (AREA).

Original: REA starts with a randomly-selected population (in our experiments the population size is 10).
Revised: REA starts with a randomly-selected population (in our experiments the population size is 10).

Original: AREA instead randomly-samples a larger population (in our experiments 1000) and uses our score to select the initial population for the REA algorithm.
Revised: AREA instead randomly-samples a larger population (in our experiments 1000) and uses our score to select the initial population for the REA algorithm.

Original: Pseudocode can be found in Appendix E.
Revised: Pseudocode can be found in Appendix E.

Original: When implementing
Revised: When implementing

Original: AREA using the NAS-Bench-201 API the precise epoch accuracy used was not speciﬁed in Dong &
Revised: AREA using the NAS-Bench-201 API the precise epoch accuracy used was not speciﬁed in Dong &

Original: Yang (2020) for their REA results across all datasets.
Revised: Yang (2020) for their REA results across all datasets.

Original: Therefore we opted to use ‘ori-test@12’ for this benchmark.
Revised: Therefore we opted to use ‘ori-test@12’ for this benchmark.

Original: We denote the algorithms REAL and AREAL (for Local implementations) to distinguish them from those using the settings in Dong & Yang (2020).
Revised: We denote the algorithms REAL and AREAL (for Local implementations) to distinguish them from those using the settings in Dong & Yang (2020).

Original: Tables 1 and 2 shows AREAL improves upon REAL in most cases across NAS-Bench-101 and NAS-Bench-201.
Revised: Tables 1 and 2 shows AREAL improves upon REAL in most cases across NAS-Bench-101 and NAS-Bench-201.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: NAS has previously suffered from intractable search spaces and heavy search costs.
Revised: NAS has previously suffered from intractable search spaces and heavy search costs.

Original: Recent advances in producing tractable search spaces, through NAS benchmarks, have allowed us to investigate if such search costs can be avoided.
Revised: Recent advances in producing tractable search spaces, through NAS benchmarks, have allowed us to investigate if such search costs can be avoided.

Original: In this work, we have shown that it is possible to run a search algorithm—
Revised: In this work, we have shown that it is possible to run a search algorithm—

Original: NASWOT—in a matter of seconds, relying on simple, intuitive observations made on initialised neural networks, that challenges more expensive black box methods.
Revised: NASWOT—in a matter of seconds, relying on simple, intuitive observations made on initialised neural networks, that challenges more expensive black box methods.

Original: We also demonstrate how our score can be combined into an existing NAS algorithm.
Revised: We also demonstrate how our score can be combined into an existing NAS algorithm.

Original: We ﬁnd our score-assisted algorithm—
Revised: We ﬁnd our score-assisted algorithm—

Original: AREA—performs better than the original version on a number of benchmarks.
Revised: AREA—performs better than the original version on a number of benchmarks.

Original: This work is not without its limitations; our scope is restricted to convolutional architectures for image classiﬁcation on benchmark search spaces.
Revised: This work is not without its limitations; our scope is restricted to convolutional architectures for image classiﬁcation on benchmark search spaces.

Original: However, we hope that this will be a powerful ﬁrst step towards removing training from NAS and making architecture search cheaper, and more readily available to practitioners.
Revised: However, we hope that this will be a powerful ﬁrst step towards removing training from NAS and making architecture search cheaper, and more readily available to practitioners.

Original: A H ISTOGRAMS OF C ORRELATIONS
Revised: A H ISTOGRAMS OF C ORRELATIONS

Original: [Figure]
Revised: [Figure]

Original: (b) Additional NAS-Bench-201 histograms.
Revised: (b) Additional NAS-Bench-201 histograms.

Original: B NAS B ENCHMARKS B. N AS -B ENCH - 
Revised: B NAS B ENCHMARKS B. N AS -B ENCH -

Original: In NAS-Bench-101, the search space is restricted as follows: algorithms must search for an individual cell which will be repeatedly stacked into a pre-deﬁned skeleton, shown in Figure 6c.
Revised: In NAS-Bench-101, the search space is restricted as follows: algorithms must search for an individual cell which will be repeatedly stacked into a pre-deﬁned skeleton, shown in Figure 6c.

Original: Each cell can be represented as a directed acyclic graph (DAG) with up to 9 nodes and up to 7 edges.
Revised: Each cell can be represented as a directed acyclic graph (DAG) with up to 9 nodes and up to 7 edges.

Original: Each node represents an operation, and each edge represents a state.
Revised: Each node represents an operation, and each edge represents a state.

Original: Operations can be chosen from: 3 ×convolution, 1 × 1 convolution, 3 × 3 max pool.
Revised: Operations can be chosen from: 3 ×convolution, 1 × 1 convolution, 3 × 3 max pool.

Original: An example of this is shown in Figure 6a.
Revised: An example of this is shown in Figure 6a.

Original: After de-duplication, this search space contains 423,624 possible neural networks.
Revised: After de-duplication, this search space contains 423,624 possible neural networks.

Original: These have been trained exhaustively, with three different initialisations, on the CIFAR-10 dataset for 108 epochs.
Revised: These have been trained exhaustively, with three different initialisations, on the CIFAR-10 dataset for 108 epochs.

Original: B. NAS-B ENCH - 
Revised: B. NAS-B ENCH -

Original: In NAS-Bench-201, networks also share a common skeleton (Figure 6c) that consists of stacks of its unique cell interleaved with ﬁxed residual downsampling blocks.
Revised: In NAS-Bench-201, networks also share a common skeleton (Figure 6c) that consists of stacks of its unique cell interleaved with ﬁxed residual downsampling blocks.

Original: Each cell (Figure 6b) can be represented as a densely-connected DAG of 4 ordered nodes (A, B, C, D) where node A is the input and node D is the output.
Revised: Each cell (Figure 6b) can be represented as a densely-connected DAG of 4 ordered nodes (A, B, C, D) where node A is the input and node D is the output.

Original: In this graph, there is an edge connecting each node to all subsequent nodes (A → B, A → C, A → D, B → C, B → D, C → D) for a total of 6 edges, and each edge can perform one of 5 possible operations (Zeroise, Identity, 3 × 3 convolution, 1 × 1 convolution, 3 ×average pool).
Revised: In this graph, there is an edge connecting each node to all subsequent nodes (A → B, A → C, A → D, B → C, B → D, C → D) for a total of 6 edges, and each edge can perform one of 5 possible operations (Zeroise, Identity, 3 × 3 convolution, 1 × 1 convolution, 3 ×average pool).

Original: The search space consists of every possible cell.
Revised: The search space consists of every possible cell.

Original: As there are 6 edges, on which there may be one of 5 operations, this means that there are 5 6 = 15625 possible cells.
Revised: As there are 6 edges, on which there may be one of 5 operations, this means that there are 5 6 = 15625 possible cells.

Original: This makes for a total of 15625 networks as each network uses just one of these cells repeatedly.
Revised: This makes for a total of 15625 networks as each network uses just one of these cells repeatedly.

Original: The authors have manually split CIFAR-10, CIFAR-100, and ImageNet-16-120 (Chrabaszcz et al., 2017) into train/val/test, and provide full training results across all networks for (i) training on train,
Revised: The authors have manually split CIFAR-10, CIFAR-100, and ImageNet-16-120 (Chrabaszcz et al., 2017) into train/val/test, and provide full training results across all networks for (i) training on train,

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: A BLATION F IGURES
Revised: A BLATION F IGURES

Original: [Figure]
Revised: [Figure]

Original: Uninitialised 
Revised: Uninitialised network

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: s c
Revised: s c

Original: o r e
Revised: o r e

Original: CIFAR-10 images {+
+}
Revised: CIFAR-10 images {+
+}

Original: [Figure]
Revised: [Figure]

Original: Uninitialised [Figure]      s c o r e
Revised: Uninitialised network      s c o r e

Original: Random images
Revised: Random images

Original: [Figure]
Revised: [Figure]

Original: Uninitialised
Revised: Uninitialised

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: s c
Revised: s c

Original: o r e
Revised: o r e

Original: Initialisations
Revised: Initialisations

Original: [Figure]
Revised: [Figure]

Original: Mini
Revised: Mini-batch size [Figure] CIFAR-10 images Kendall = 0.

Original: [Figure]
Revised: [Figure]

Revised: Random images

Revised: Kendall = 0.

Revised: [Figure]

Revised: Initialisations

Revised: Kendall = 0.

Revised: [Figure]

Revised: CIFAR-10 images

Revised: Kendall = 0.

Revised: [Figure]

Revised: Random images

Original: size
Revised: Kendall = 0. [Figure] Initialisations Kendall = 0.

Original: E AREA A LGORITHM
Revised: E AREA A LGORITHM

Original: Algorithm 

Revised: Algorithm

Original: Assisted Regularised EA - AREA population =
Revised: Assisted Regularised EA - AREA population =

Original: [] generator = RandomGenerator() for i=1:
Revised: [] generator = RandomGenerator() for i=1:

Original: M do net = generator.generate() scored net = net.score() population.append(scored net)
Revised: M do net = generator.generate() scored net = net.score() population.append(scored net)

Original: Keep the top N scored networks in the population history =
Revised: Keep the top N scored networks in the population history =

Original: [] for net in population do trained net = net.train() history.append(trained net) while time limit not exceeded do
Revised: [] for net in population do trained net = net.train() history.append(trained net) while time limit not exceeded do

Original: Sample sub-population, S, without replacement from population
Revised: Sample sub-population, S, without replacement from population

Original: Select network in S with highest accuracy as parent
Revised: Select network in S with highest accuracy as parent

Original: Mutate parent network to produce child
Revised: Mutate parent network to produce child

Original: Train child network
Revised: Train child network

Original: Remove oldest network from population population.append(child network) history.append(child network) chosen net =
Revised: Remove oldest network from population population.append(child network) history.append(child network) chosen net =

Original: Network in history with highest accuracy
Revised: Network in history with highest accuracy

Revised: F V ARYING INITIALISATIONS

Revised: In Figure 11 we examine the effect of different initialisation functions on the efﬁcacy of our score.

Revised: We ﬁnd that the scoring function is robust to many different intialisation strategies, though it fails for

Revised: Uniform(0,1) initialisation.

Revised: In Figure 12 we conﬁrm that this effect is similar when the score is aggregated over multiple re-initialisations of the network

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: score

Revised: G V ARYING β

Revised: The scoring mechanism introduced in Section 3 relies on a cutoff value β that decides which portion of the histogram of correlations we look at.

Revised: Given that our goal is to assign high scores to histograms with a small positive skew, we set the default value of β = 0 .

Revised: In Figure 13 we investigate different values of β , reporting the CIFAR-10 test accuracy for networks in NAS-Bench-101 and

Revised: NAS-Bench-201.

Revised: We linearly sample ﬁve values of β between 0 .

Revised: 025 and 1 and report mean accuracy over 500 runs.

Revised: We also show the standard deviation in accuracy for each β in the shaded area.

Revised: The ﬁgure shows that in both benchmarks, the scoring mechanism is reasonably robust to different choices of β , but deteriorates at very large values.

Revised: [Figure]

Revised: H S CORE BY EPOCH

Revised: We evaluated the score during training and plotted the trajectory of the score for 100 networks for

Revised: NAS-Bench-201 trained using the CIFAR-10 dataset in Figure 14.

Revised: [Figure]

Revised: I A UGMENTING WITH G AUSSIAN N OISE AND C OLOUR J ITTER

Revised: [Figure]

Revised: J F URTHER R ANDOM I MAGE E XPERIMENTS

Revised: We ran further evaluations of the score for random images for 6 seeds.

Revised: These are shown in Figure

Revised: [Figure]