Original: L EARNABLE E MBEDDING S PACE FOR E FFICIENT N EURAL A RCHITECTURE C OMPRESSION
Revised: L EARNABLE E MBEDDING S PACE FOR E FFICIENT N EURAL A RCHITECTURE C OMPRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during neural architecture search (NAS).
Revised: We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search.

Original: In this paper, we focus on the task of network architecture compression.

Original: Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function deﬁned over our proposed embedding space to select architectures for evaluation.
Revised: Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function deﬁned over our proposed embedding space to select architectures for evaluation.

Original: We demonstrate that our search algorithm can signiﬁcantly outperform various baseline methods, such as random search and N2N (Ashok et al., 2018).
Revised: We demonstrate that our search algorithm can signiﬁcantly outperform various baseline methods, such as random search and N2N (Ashok et al., 2018).

Original: The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShufﬂeNet (Zhang et al., 2018).
Revised: The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShufﬂeNet (Zhang et al., 2018).

Original: We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.
Revised: We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One signiﬁcant bottleneck of neural architecture search (NAS) – the automatic discovery of neural network architectures – is the need to repeatedly evaluate different network architectures, as each evaluation is extremely costly ( e.g. , back-propagation to learn the parameters of a single deep network can take several days on a single GPU).

Original: This means that any efﬁcient search algorithm must be judicious when selecting architectures to evaluate.

Original: Learning a good embedding space over the domain of network architectures is important because it can be used to deﬁne a distribution on the architecture space that can be used to generate a priority ordering of architectures for evaluation.

Original: To enable the careful selection of architectures for evaluation, we propose a method to incrementally learn an embedding space over the domain of network architectures.

Original: While the method developed in this work can be used for general NAS, we focus on the tasks of network architecture compression as we believe that it is the more pressing need for many realworld applications.

Original: In many application domains, it is common practice to make use of well-known deep network architecture ( e.g., VGG (Simonyan & Zisserman, 2014), GoogleNet (Szegedy et al., 2015), ResNet (He et al., 2016)) and to adapt them to a new task without optimizing the architecture for that task.
Revised: In many application domains, it is common practice to make use of well-known deep network architecture ( e.g., VGG (Simonyan & Zisserman, 2014), GoogleNet (Szegedy et al., 2015), ResNet (He et al., 2016)) and to adapt them to a new task without optimizing the architecture for that task.

Original: While this process of transfer learning is surprisingly successful, it often results in oversized networks which have many redundant or unused parameters.
Revised: While this process of transfer learning is surprisingly successful, it often results in over-sized networks which have many redundant or unused parameters.

Original: Inefﬁcient network architectures can waste computational resources and over-sized networks can prevent them from being used on embedded systems.
Revised: Inefﬁcient network architectures can waste computational resources and over-sized networks can prevent them from being used on embedded systems.

Original: There is a pressing need to develop algorithms that can take large networks with high accuracy as input and compress their size while maintaining similar performance.
Revised: There is a pressing need to develop algorithms that can take large networks with high accuracy as input and compress their size while maintaining similar performance.

Revised: In this paper, we focus on the task of compressed architecture search – the automatic discovery of compressed network architectures based on a given large network.

Revised: One signiﬁcant bottleneck of compressed architecture search is the need to repeatedly evaluate different compressed network architectures, as each evaluation is extremely costly ( e.g. , backpropagation to learn the parameters of a single deep network can take several days on a single

Revised: GPU).

Original:  In the network compression paradigm, we are given a teacher network and we aim to search for a compressed network architecture, a student network, containing as few parameters as possible while maintaining similar performance to the teacher network.
Revised: This means that any efﬁcient search algorithm must be judicious when selecting architectures to evaluate. Learning a good embedding space over the domain of compressed network architectures is important because it can be used to deﬁne a distribution on the architecture space that can be used to generate a priority ordering of architectures for evaluation. To enable the careful selection of architectures for evaluation, we propose a method to incrementally learn an embedding space over the domain of network architectures. In the network compression paradigm, we are given a teacher network and we aim to search for a compressed network architecture, a student network, containing as few parameters as possible while maintaining similar performance to the teacher network.

Original: We address the task of NAS in the context of network compression by using Bayesian Optimization (BO) with a kernel function deﬁned over our proposed embedding space to select architectures for evaluation.
Revised: We address the task of compressed architecture search by using Bayesian Optimization (BO) with a kernel function deﬁned over our proposed embedding space to select architectures for evaluation.

Original: As modern neural architectures can have multiple layers, multiple branches and multiple skip connections, deﬁning an embedding space over all architectures is non-trivial.
Revised: As modern neural architectures can have multiple layers, multiple branches and multiple skip connections, deﬁning an embedding space over all architectures is non-trivial.

Original: In this work, we propose a method for mapping a diverse range of discrete architectures to a continuous embedding space through the use of recurrent neural networks.
Revised: In this work, we propose a method for mapping a diverse range of discrete architectures to a continuous embedding space through the use of recurrent neural networks.

Original: The learned embedding space allows us to perform BO to efﬁciently search for compressed student architectures that are also expected to have high accuracy.
Revised: The learned embedding space allows us to perform BO to efﬁciently search for compressed student architectures that are also expected to have high accuracy.

Original: We demonstrate that our search algorithm can signiﬁcantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018).
Revised: We demonstrate that our search algorithm can signiﬁcantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018).

Original: For example, our search algorithm can compress VGG-19 (Simonyan & Zisserman, 2014) by 10 × on CIFAR-100 (Krizhevsky & Hinton, 2009) while maintaining accuracy on par with the teacher network.
Revised: For example, our search algorithm can compress VGG-19 (Simonyan & Zisserman, 2014) by 8 × on CIFAR-100 (Krizhevsky & Hinton, 2009) while maintaining accuracy on par with the teacher network.

Original: The automatically found compressed architectures can also achieve higher accuracy than the state-of-the-art manuallydesigned compact architecture ShufﬂeNet (Zhang et al., 2018) with a similar size.
Revised: The automatically found compressed architectures can also achieve higher accuracy than the state-of-the-art manuallydesigned compact architecture ShufﬂeNet (Zhang et al., 2018) with a similar size.

Original: We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.
Revised: We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.

Original: Contributions : (1) We present a framework ofsearching for compressed network architectures with
Revised: Contributions : (1) We propose a novel method to incrementally learn an embedding space over the domain of network architectures.

Original: BO. The key to our framework is a learnable embedding space for the neural architecture domain.
Revised: Based on the learnable embedding space, we present a framework of searching for compressed network architectures with BO.

Original: The learned embedding provides a feature space over which the kernel function of BO is deﬁned.
Revised: The learned embedding provides a feature space over which the kernel function of BO is deﬁned.

Original: (2) We propose a set of architecture operators for generating architectures for search.
Revised: (2) We propose a set of architecture operators for generating architectures for search.

Original: Operators for modifying the teacher network are: layer removal, layer shrinkage and skip connection addition.
Revised: Operators for modifying the teacher network are: layer removal, layer shrinkage and skip connection addition.

Original:  We propose a multiple kernel strategy to prevent the whole search process from being biased by thevery ﬁrst few searchsteps and encourage the search algorithm to explore more diverse architectures.
Revised: (3) We propose a multiple kernel strategy to prevent the premature convergence of the search and encourage the search algorithm to explore more diverse architectures during the search process.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Computationally Efﬁcient Architecture : There has been great progress in designing computationally efﬁcient network architectures.
Revised: Computationally Efﬁcient Architecture : There has been great progress in designing computationally efﬁcient network architectures.

Original: Representative examples include SqueezeNet (Iandola et al., 2016), MobileNet (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018), CondenseNet (Huang et al., 2018) and ShufﬂeNet (Zhang et al., 2018).
Revised: Representative examples include SqueezeNet (Iandola et al., 2016), MobileNet (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018), CondenseNet (Huang et al., 2018) and ShufﬂeNet (Zhang et al., 2018).

Original: Different from them, we aim to develop an algorithm that can automatically search for an efﬁcient network architecture with minimal human efforts involved in the architecture design.
Revised: Different from them, we aim to develop an algorithm that can automatically search for an efﬁcient network architecture with minimal human efforts involved in the architecture design.

Original: Neural Architecture Search (NAS) : NAS has recently been an active research topic (Zoph & Le, 2016; Zoph et al., 2017;
Revised: Neural Architecture Search (NAS) : NAS has recently been an active research topic (Zoph & Le, 2016; Zoph et al., 2017;

Original: Real et al., 2018; Pham et al., 2018; Liu et al., 2017a;b; 2018;
Revised: Real et al., 2018; Pham et al., 2018; Liu et al., 2017a;b; 2018;

Original: Luo et al., 2018).
Revised: Luo et al., 2018).

Original: Some existing works in NAS are focused on searching for architectures that not only can achieve high performance but also respect some resource or computation constraints (Ashok et al., 2018; Tan et al., 2018;
Revised: Some existing works in NAS are focused on searching for architectures that not only can achieve high performance but also respect some resource or computation constraints (Ashok et al., 2018; Tan et al., 2018;

Original: Zhou et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018a).
Revised: Zhou et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018a).

Original: Our work is mostly related to N2N (Ashok et al., 2018), which searches for a compressed architecture based on a given teacher network using reinforcement learning.
Revised: Our work is mostly related to N2N (Ashok et al., 2018), which searches for a compressed architecture based on a given teacher network using reinforcement learning.

Original: Our search algorithm is developed based on Bayesian Optimization (BO), which is different from N2N and many other existing works.
Revised: Our search algorithm is developed based on Bayesian Optimization (BO), which is different from N2N and many other existing works.

Original: We will compare our approach to other BO based NAS methods in the next paragraph.
Revised: We will compare our approach to other BO based NAS methods in the next paragraph.

Original: Readers can refer to Elsken et al.
Revised: Readers can refer to Elsken et al.

Original: (2018b)
Revised: (2018b)

Original: for a more complete literature review of NAS.
Revised: for a more complete literature review of NAS.

Original: Bayesian Optimization (BO) : BO is a popular method for hyper-parameter optimization in machine learning.
Revised: Bayesian Optimization (BO) : BO is a popular method for hyper-parameter optimization in machine learning.

Original: BO has been used to tune the number of layers and the size of hidden layers (Bergstra et al., 2011; Swersky et al., 2014), the width of a network (Snoek et al., 2012) or the size of the ﬁlter bank (Bergstra et al., 2013), along with other hyper-parameters, such as the learning rate, number of iterations. 
Revised: BO has been used to tune the number of layers and the size of hidden layers (Bergstra et al., 2011; Swersky et al., 2014), the width of a network (Snoek et al., 2012) or the size of the ﬁlter bank (Bergstra et al., 2013), along with other hyper-parameters, such as the learning rate, number of iterations. Mendoza et al.

Original:  Jenatton et al.
Revised: , Jenatton et al.

Original: (2017) and Zela et al.
Revised: (2017) and Zela et al.

Original: (2018) also fall into this category.
Revised: (2018) also fall into this category.

Original:  However, most existing works on BO for NAScan only tune feed-forward structures, which restricts their applications.
Revised: Our work is also related to Hern´andez-Lobato et al. (2016), which presents a Bayesian method for identifying the Pareto set of multi-objective optimization problems and applies the method to searching for a fast and accurate neural network. However, most existing works on BO for NAS only show results on tuning network architectures where the connections between network layers are ﬁxed, i.e., most of them do not optimize the connections between layers.

Original: Kandasamy et al.
Revised: Kandasamy et al.

Original: (2018) proposes a distance metric OTMANN to compare network architectures with complex skip connections and branch layers, based on which NASBOT is developed, a BO based NAS framework.
Revised: (2018) proposes a distance metric OTMANN to compare network architectures with complex skip connections and branch layers, based on which NASBOT is developed, a BO based NAS framework, which can tune the connections between network layers.

Original: Although the OTMANN distance is designed with thoughtful choices, it is deﬁned based on some empirically identiﬁed factors that can inﬂuence the performance of a network, rather than the actual performance of networks.
Revised: Although the OTMANN distance is designed with thoughtful choices, it is deﬁned based on some empirically identiﬁed factors that can inﬂuence the performance of a network, rather than the actual performance of networks.

Original: Different from OTMANN, the distance metric (or the embedding) for network architectures in our algorithm is automatically learned according to the actual performance of network architectures instead of manually designed.
Revised: Different from OTMANN, the distance metric (or the embedding) for network architectures in our algorithm is automatically learned according to the actual performance of network architectures instead of manually designed.

Revised: Our work also can be viewed as carrying out optimization in the latent space of a high dimensional and structured space, which shares a similar idea with previous literature (Lu et al., 2018;

Revised: G´omez-Bombarelli et al., 2018).

Revised: For example, Lu et al.

Revised: (2018) presents a new variational autoencoder to map kernel combinations produced by a context-free grammar into a continuous and low-dimensional latent space.

Original: Deep Kernel Learning :
Revised: Deep Kernel Learning :

Original: Our work is also related to recent works on deep kernel learning (Wilson et al., 2016a;b).
Revised: Our work is also related to recent works on deep kernel learning (Wilson et al., 2016a;b).

Original: They aim to learn more expressive kernels by representing the kernel function as a neural network to incorporate the expressive power of deep networks.
Revised: They aim to learn more expressive kernels by representing the kernel function as a neural network to incorporate the expressive power of deep networks.

Original: The follow-up work (AlShedivat et al., 2017) extends the kernel representation to recurrent networks to model sequential data.
Revised: The follow-up work (AlShedivat et al., 2017) extends the kernel representation to recurrent networks to model sequential data.

Original: Our work shares a similar motivation with them and tries to learn a kernel function for the neural architecture domain by leveraging the expressive power of deep networks.
Revised: Our work shares a similar motivation with them and tries to learn a kernel function for the neural architecture domain by leveraging the expressive power of deep networks.

Original:  A PPROACH
Revised:  A PPROACH

Original: In this work, we focus on searching for a compressed network architecture based on a given teacher network and our goal is to ﬁnd a network architecture which contains as few parameters as possible but still can obtain a similar performance to the teacher network.
Revised: In this work, we focus on searching for a compressed network architecture based on a given teacher network and our goal is to ﬁnd a network architecture which contains as few parameters as possible but still can obtain a similar performance to the teacher network.

Original: Formally, we aim to solve the following optimization problem:
Revised: Formally, we aim to solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where X denotes the domain of neural architectures and the function f ( x ) : X (cid:55)→ R evaluates how well the architecture x meets our requirement.
Revised: where X denotes the domain of neural architectures and the function f ( x ) : X (cid:55)→ R evaluates how well the architecture x meets our requirement.

Original: We adopt the reward function design in N2N (Ashok et al., 2018) for the function f , which is deﬁned based on the compression ratio of the architecture x and its validation performance after being trained on the training set.
Revised: We adopt the reward function design in N2N (Ashok et al., 2018) for the function f , which is deﬁned based on the compression ratio of the architecture x and its validation performance after being trained on the training set.

Original: More details about the exact form of f are given in Appendix 6.1 due to space constraints.
Revised: More details about the exact form of f are given in Appendix 6.1 due to space constraints.

Original: As evaluating the value of f ( x ) for a speciﬁc architecture x is extremely costly, the algorithm must judiciously select the architectures to evaluate.
Revised: As evaluating the value of f ( x ) for a speciﬁc architecture x is extremely costly, the algorithm must judiciously select the architectures to evaluate.

Original: To enable the careful selection of architectures for evaluation, we propose a method to incrementally learn an embedding space over the domain of network architecture.
Revised: To enable the careful selection of architectures for evaluation, we propose a method to incrementally learn an embedding space over the domain of network architecture.

Original: We develop the search algorithm based on BO with a kernel function deﬁned over our proposed embedding space.
Revised: We develop the search algorithm based on BO with a kernel function deﬁned over our proposed embedding space.

Original: In the following text, we will ﬁrst introduce the sketch of the
Revised: In the following text, we will ﬁrst introduce the sketch of the

Original: BO algorithm
Revised: BO algorithm

Original: and then explain how the proposed embedding space is used in the loop of BO.
Revised: and then explain how the proposed embedding space is used in the loop of BO.

Original: We adopt the Gaussian process (GP) based BO algorithms to maximize the function f ( x ) , which is one of the most popular algorithms in BO.
Revised: We adopt the Gaussian process (GP) based BO algorithms to maximize the function f ( x ) , which is one of the most popular algorithms in BO.

Original: A GP prior is placed on the function f , parameterized by a mean function µ ( · ) :
Revised: A GP prior is placed on the function f , parameterized by a mean function µ ( · ) :

Original: X (cid:55)→ R and a covariance function or kernel k ( · , · ) : X × X (cid:55)→ R .
Revised: X (cid:55)→ R and a covariance function or kernel k ( · , · ) : X × X (cid:55)→ R .

Original: To search for the solution, we start from an arbitrarily selected architecture x 1 .
Revised: To search for the solution, we start from an arbitrarily selected architecture x 1 .

Original: At step t , we evaluate the architecture x t , i.e. , obtaining the value of f ( x t ) .
Revised: At step t , we evaluate the architecture x t , i.e. , obtaining the value of f ( x t ) .

Original: Using the t evaluated architectures up to now, we compute the posterior distribution on the function f :
Revised: Using the t evaluated architectures up to now, we compute the posterior distribution on the function f :

Original: [Equation]
Revised: [Equation]

Original: where f ( x 1: t )
Revised: where f ( x 1: t )

Original: =
Revised: =

Original: [ f ( x 1 ) , . . .
Revised: [ f ( x 1 ) , . . .

Original: , f ( x t )] and µ t ( x ) and σ 2 t ( x ) can be computed analytically based on the
Revised: , f ( x t )] and µ t ( x ) and σ 2 t ( x ) can be computed analytically based on the

Original: GP prior (Williams & Rasmussen, 2006).
Revised: GP prior (Williams & Rasmussen, 2006).

Original: We then use the posterior distribution to decide the next architecture to evaluate.
Revised: We then use the posterior distribution to decide the next architecture to evaluate.

Original: In particular, we obtain x t +1 by maximizing the expected improvement acquisition function EI t ( x ) :
Revised: In particular, we obtain x t +1 by maximizing the expected improvement acquisition function EI t ( x ) :

Original: X (cid:55)→ R , i.e. , x t +1 = arg max x ∈X EI t ( x ) .
Revised: X (cid:55)→ R , i.e. , x t +1 = arg max x ∈X EI t ( x ) .

Original: The expected improvement function EI t ( x ) (Mockus & Mockus, 1991) measures the expected improvement over the current maximum value according to the posterior distribution:
Revised: The expected improvement function EI t ( x ) (Mockus & Mockus, 1991) measures the expected improvement over the current maximum value according to the posterior distribution:

Original: [Equation]
Revised: [Equation]

Original: where E t indicates the expectation is taken with respect to the posterior distribution at step t p ( f ( x ) |
Revised: where E t indicates the expectation is taken with respect to the posterior distribution at step t p ( f ( x ) |

Original: f ( x 1: t )) and f ∗ t is the maximum value among { f ( x 1 ) , . . .
Revised: f ( x 1: t )) and f ∗ t is the maximum value among { f ( x 1 ) , . . .

Original: , f ( x t ) } .
Revised: , f ( x t ) } .

Original: Once obtaining x t +1 , we repeat the above described process until we reach the maximum number of steps.
Revised: Once obtaining x t +1 , we repeat the above described process until we reach the maximum number of steps.

Original: Finally, we return the best evaluated architecture as the solution.
Revised: Finally, we return the best evaluated architecture as the solution.

Original: The main difﬁculty in realizing the above optimization procedure is the design of the kernel function k ( · , · ) for X and the maximization of the acquisition function EI t ( x ) over X , since the neural architecture domain X is discrete and highly complex.
Revised: The main difﬁculty in realizing the above optimization procedure is the design of the kernel function k ( · , · ) for X and the maximization of the acquisition function EI t ( x ) over X , since the neural architecture domain X is discrete and highly complex.

Original: To overcome these difﬁculties, we propose to learn an embedding space for the neural architecture domain and deﬁne the kernel function based on the learned embedding space.
Revised: To overcome these difﬁculties, we propose to learn an embedding space for the neural architecture domain and deﬁne the kernel function based on the learned embedding space.

Original: We also propose a search space, a subset of the neural architecture domain, over which maximizing the acquisition function is feasible and sufﬁcient.
Revised: We also propose a search space, a subset of the neural architecture domain, over which maximizing the acquisition function is feasible and sufﬁcient.

Original: 3. L EARNABLE E MBEDDING S PACE AND K ERNEL F UNCTION
Revised: 3. L EARNABLE E MBEDDING S PACE AND K ERNEL F UNCTION

Original: The kernel function, which measures the similarity between network architectures, is fundamental for selecting the architectures to evaluate during the search process.
Revised: The kernel function, which measures the similarity between network architectures, is fundamental for selecting the architectures to evaluate during the search process.

Original: As modern neural architectures can have multiple layers, multiple branches and multiple skip connections, comparing two architectures is non-trivial.
Revised: As modern neural architectures can have multiple layers, multiple branches and multiple skip connections, comparing two architectures is non-trivial.

Original: Therefore, we propose to map a diverse range of discrete architectures to a continuous embedding space through the use of recurrent neural networks and then deﬁne the kernel function based on the learned embedding space.
Revised: Therefore, we propose to map a diverse range of discrete architectures to a continuous embedding space through the use of recurrent neural networks and then deﬁne the kernel function based on the learned embedding space.

Original: We use h ( · ; θ ) to denote the architecture embedding function that generates an embedding for a network architecture according to its conﬁguration parameters.
Revised: We use h ( · ; θ ) to denote the architecture embedding function that generates an embedding for a network architecture according to its conﬁguration parameters.

Original: θ represents the weights to be learned in the architecture embedding function.
Revised: θ represents the weights to be learned in the architecture embedding function.

Original: With h ( · ; θ ) , we deﬁne the kernel function k
Revised: With h ( · ; θ ) , we deﬁne the kernel function k

Original: ( x, x (cid:48) ; θ ) based the RBF kernel:
Revised: ( x, x (cid:48) ; θ ) based the RBF kernel:

Original: [Equation]
Revised: [Equation]

Original: where σ is a hyper-parameter.
Revised: where σ is a hyper-parameter.

Original: h ( · ; θ ) represents the proposed learnable embedding space and k ( x, x (cid:48) ; θ ) is the learnable kernel function.
Revised: h ( · ; θ ) represents the proposed learnable embedding space and k ( x, x (cid:48) ; θ ) is the learnable kernel function.

Original: They are parameterized by the same weights θ .
Revised: They are parameterized by the same weights θ .

Original: In the following text, we will ﬁrst introduce the architecture embedding function h ( · ; θ ) and then describe how we learn the weights θ during the search process.
Revised: In the following text, we will ﬁrst introduce the architecture embedding function h ( · ; θ ) and then describe how we learn the weights θ during the search process.

Original: The architecture embedding function needs to be ﬂexible enough to handle a diverse range of architectures that may have multiple layers, multiple branches and multiple skip connections.
Revised: The architecture embedding function needs to be ﬂexible enough to handle a diverse range of architectures that may have multiple layers, multiple branches and multiple skip connections.

Original: Therefore we adopt a Bidirectional LSTM to represent the architecture embedding function, motivated by the layer removal policy network in N2N (Ashok et al., 2018).
Revised: Therefore we adopt a Bidirectional LSTM to represent the architecture embedding function, motivated by the layer removal policy network in N2N (Ashok et al., 2018).

Original: The input to the Bi-LSTM is the conﬁguration information of each layer in the network, including the layer type, how this layer connects to other layers, and other attributes.
Revised: The input to the Bi-LSTM is the conﬁguration information of each layer in the network, including the layer type, how this layer connects to other layers, and other attributes.

Original: After passing the conﬁguration of each layer to the Bi-LSTM, we gather all the hidden states, apply average pooling to these hidden states and then apply Lnormalization to the pooled vector to obtain the architecture embedding.
Revised: After passing the conﬁguration of each layer to the Bi-LSTM, we gather all the hidden states, apply average pooling to these hidden states and then apply Lnormalization to the pooled vector to obtain the architecture embedding.

Original: We would like to emphasize that our representation for layer conﬁguration encodes the skip connections between layers.
Revised: We would like to emphasize that our representation for layer conﬁguration encodes the skip connections between layers.

Original: Skip connections have been proven effective in both human designed network architectures, such as ResNet (He et al., 2016) and DenseNet (Huang et al., 2017), and automatically found network architectures (Zoph & Le, 2016).
Revised: Skip connections have been proven effective in both human designed network architectures, such as ResNet (He et al., 2016) and DenseNet (Huang et al., 2017), and automatically found network architectures (Zoph & Le, 2016).

Original: N2N only supports the kind of skip connections used in ResNet (He et al., 2016) and does not generalize to more complex connections between layers, where our representation is still applicable.
Revised: N2N only supports the kind of skip connections used in ResNet (He et al., 2016) and does not generalize to more complex connections between layers, where our representation is still applicable.

Original: We give the details about our representation for layer conﬁguration in Appendix 6.2.
Revised: We give the details about our representation for layer conﬁguration in Appendix 6.2.

Original: The weights of the Bi-LSTM θ , is learned during the search process.
Revised: The weights of the Bi-LSTM θ , is learned during the search process.

Original: The weights θ determines the architecture embedding function h ( · ; θ ) and the kernel k ( · , · ; θ ) .
Revised: The weights θ determines the architecture embedding function h ( · ; θ ) and the kernel k ( · , · ; θ ) .

Original: Further, θ controls the GP prior and the posterior distribution of the function value conditioned on the observed data points.
Revised: Further, θ controls the GP prior and the posterior distribution of the function value conditioned on the observed data points.

Original: The posterior distribution guides the search process and is essential to the performance of our search algorithm.
Revised: The posterior distribution guides the search process and is essential to the performance of our search algorithm.

Original: Our goal is to learn a θ such that the function f is consistent with the GP prior, which will result in a posterior distribution that accurately characterizes the statistical structure of the function f .
Revised: Our goal is to learn a θ such that the function f is consistent with the GP prior, which will result in a posterior distribution that accurately characterizes the statistical structure of the function f .

Original: Let D denote the set of evaluated architectures.
Revised: Let D denote the set of evaluated architectures.

Original: In step t , D = { x 1 , . . .
Revised: In step t , D = { x 1 , . . .

Original: , x t } .
Revised: , x t } .

Original: For any architecture x i in D , we can compute p ( f ( x i ) | f
Revised: For any architecture x i in D , we can compute p ( f ( x i ) | f

Original: ( D \ x i ); θ ) based on the GP prior, where \ refers to the set difference operation, f ( x i ) is the value obtained by evaluating the architecture x i and f ( D \ x i ) =
Revised: ( D \ x i ); θ ) based on the GP prior, where \ refers to the set difference operation, f ( x i ) is the value obtained by evaluating the architecture x i and f ( D \ x i ) =

Original: [ f ( x 1 ) , . . .
Revised: [ f ( x 1 ) , . . .

Original: , f ( x i − 1 ) , f ( x i +1 ) . . .
Revised: , f ( x i − 1 ) , f ( x i +1 ) . . .

Original: , f ( x t )] .
Revised: , f ( x t )] .

Original: p ( f ( x i ) | f
Revised: p ( f ( x i ) | f

Original: ( D \ x i ); θ ) is the posterior probability of f ( x i ) conditioned on the other evaluated architectures in D .
Revised: ( D \ x i ); θ ) is the posterior probability of f ( x i ) conditioned on the other evaluated architectures in D .

Original: The higher the value of p ( f ( x i ) | f
Revised: The higher the value of p ( f ( x i ) | f

Original: ( D \ x i ); θ ) is, the more accurately the posterior distribution characterizes the statistical structure of the function f and the more the function f is consistent with the GP prior.
Revised: ( D \ x i ); θ ) is, the more accurately the posterior distribution characterizes the statistical structure of the function f and the more the function f is consistent with the GP prior.

Original: Therefore, we learn θ by minimizing the negative log posterior probability:
Revised: Therefore, we learn θ by minimizing the negative log posterior probability:

Original: [Equation]
Revised: [Equation]

Original: p ( f ( x i ) | f
Revised: p ( f ( x i ) | f

Original: ( D \ x i ); θ ) is a Gaussian distribution and its mean and covariance matrix can be computed analytically based on k ( · , · ; θ ) .
Revised: ( D \ x i ); θ ) is a Gaussian distribution and its mean and covariance matrix can be computed analytically based on k ( · , · ; θ ) .

Original: Thus L is differentiable with respect to θ
Revised: Thus L is differentiable with respect to θ

Original: and we can learn the weights θ using backpropagation.
Revised: and we can learn the weights θ using backpropagation.

Original: 3. A CQUISITION F UNCTION AND S EARCH S PACE
Revised: 3. A CQUISITION F UNCTION AND S EARCH S PACE

Original: In each optimization step, we obtain the next architecture to evaluate by maximizing the acquisition function EI t ( · ) over the neural architecture domain X .
Revised: In each optimization step, we obtain the next architecture to evaluate by maximizing the acquisition function EI t ( · ) over the neural architecture domain X .

Original: On one hand, maximizing EI t ( · ) over all the network architectures in X is unnecessary.
Revised: On one hand, maximizing EI t ( · ) over all the network architectures in X is unnecessary.

Original: Since our goal is to search for a compressed architecture based on the given teacher network, we only need to consider those architectures that are smaller than the teacher network.
Revised: Since our goal is to search for a compressed architecture based on the given teacher network, we only need to consider those architectures that are smaller than the teacher network.

Original: On the other hand, maximizing EI t ( · ) over X is non-trivial.
Revised: On the other hand, maximizing EI t ( · ) over X is non-trivial.

Original: Gradientbased optimization algorithms cannot be directly applied to optimize EI t ( · ) as X is discrete.
Revised: Gradientbased optimization algorithms cannot be directly applied to optimize EI t ( · ) as X is discrete.

Original: Also, exhaustive exploration of the whole domain is infeasible.
Revised: Also, exhaustive exploration of the whole domain is infeasible.

Original: This calls for a search space that covers the compressed architectures of our interest and easy to explore.
Revised: This calls for a search space that covers the compressed architectures of our interest and easy to explore.

Original: Motivated by N2N (Ashok et al., 2018), we propose a search space for maximizing the acquisition function, which is constrained by the teacher network, and provide a practical method to explore the search space.
Revised: Motivated by N2N (Ashok et al., 2018), we propose a search space for maximizing the acquisition function, which is constrained by the teacher network, and provide a practical method to explore the search space.

Original: We deﬁne the search space based on the teacher network.
Revised: We deﬁne the search space based on the teacher network.

Original: The search space is constructed by all the architectures that can be obtained by manipulating the teacher network with the following three operations: (1) layer removal, (2) layer shrinkage and (3) adding skip connections.
Revised: The search space is constructed by all the architectures that can be obtained by manipulating the teacher network with the following three operations: (1) layer removal, (2) layer shrinkage and (3) adding skip connections.

Original: Layer removal and shrinkage : The two operations ensure that we only consider architectures that are smaller than the given big network.
Revised: Layer removal and shrinkage : The two operations ensure that we only consider architectures that are smaller than the given big network.

Original: Layer removal refers to removing one or more layers from the network.
Revised: Layer removal refers to removing one or more layers from the network.

Original: Layer shrinkage refers to shrinking the size of layers, in particular, the number of ﬁlters in convolutional layers, as we focus on convolutional networks in this work.
Revised: Layer shrinkage refers to shrinking the size of layers, in particular, the number of ﬁlters in convolutional layers, as we focus on convolutional networks in this work.

Original: Different from N2N, we do not consider shrinking the kernel size, padding or other conﬁgurable variables and we ﬁnd that only shrinking the number of ﬁlters already yields satisfactory performance.
Revised: Different from N2N, we do not consider shrinking the kernel size, padding or other conﬁgurable variables and we ﬁnd that only shrinking the number of ﬁlters already yields satisfactory performance.

Original: Adding skip connections : The operation of adding skip connections is employed to increase the network complexity.
Revised: Adding skip connections : The operation of adding skip connections is employed to increase the network complexity.

Original: N2N (Ashok et al., 2018), which uses reinforcement learning to search for compressed network architectures, does not support forming skip connections in their action space.
Revised: N2N (Ashok et al., 2018), which uses reinforcement learning to search for compressed network architectures, does not support forming skip connections in their action space.

Original: We believe when searching for compressed architectures, adding skip connections to the compressed network is crucial for it to achieve similar performance to the teacher network and we will show ablation study results to verify this.
Revised: We believe when searching for compressed architectures, adding skip connections to the compressed network is crucial for it to achieve similar performance to the teacher network and we will show ablation study results to verify this.

Original: The way we deﬁne the search space naturally allows us to explore it by sampling the operations to manipulate the architecture of the teacher network.
Revised: The way we deﬁne the search space naturally allows us to explore it by sampling the operations to manipulate the architecture of the teacher network.

Original: To optimize the acquisition function over the search space, we randomly sample architectures in the search space by randomly sampling the operations.
Revised: To optimize the acquisition function over the search space, we randomly sample architectures in the search space by randomly sampling the operations.

Original: We then evaluate EI t ( · ) over the sampled architectures and return the best one as the solution.
Revised: We then evaluate EI t ( · ) over the sampled architectures and return the best one as the solution.

Original: We also have tried using evolutionary algorithm to maximize EI t ( · ) but it yields similar results with random sampling.
Revised: We also have tried using evolutionary algorithm to maximize EI t ( · ) but it yields similar results with random sampling.

Original: So for the sake of simplicity, we use random sampling to maximize
Revised: So for the sake of simplicity, we use random sampling to maximize

Original: EI t ( · ) .
Revised: EI t ( · ) .

Original: We attribute the good performance of random sampling to the thoughtful design of the operations to manipulate the teacher network architecture.
Revised: We attribute the good performance of random sampling to the thoughtful design of the operations to manipulate the teacher network architecture.

Original: These operations already favor the compressed architectures of our interest.
Revised: These operations already favor the compressed architectures of our interest.

Original: 3. M ULTIPLE K ERNEL S TRATEGY
Revised: 3. M ULTIPLE K ERNEL S TRATEGY

Original: We implement the search algorithm with the proposed learnable kernel function but notice that the highest function value among evaluated architectures stops increasing after a few steps.
Revised: We implement the search algorithm with the proposed learnable kernel function but notice that the highest function value among evaluated architectures stops increasing after a few steps.

Original: We conjecture this is due to that the whole search process is biased by the  architecturesevaluated in the very ﬁrst few steps.
Revised: We conjecture this is due to that the learned kernel is overﬁtted to the training samples since we only evaluate hundreds of architectures in the whole search process.

Original: In the implementation, the weights of the kernel function are learned based on all the evaluated architectures available at the current step and then the learned kernel is used to determine the architecture to evaluate in the next step. Due to the sequential dependence, it is possible that the architectures evaluated at the very beginning bias the whole search process.
Revised: An overﬁtted kernel may bias the following sampled architectures for evaluation.

Original: To alleviate this issue, we propose a multiple kernel strategy, motivated by the bagging algorithm, which is usually employed to avoid overﬁtting.
Revised: To encourage the search algorithm to explore more diverse architectures, we propose a multiple kernel strategy, motivated by the bagging algorithm, which is usually employed to avoid overﬁtting.

Original: In bagging, instead of training one single model on the whole dataset, multiple models are trained on different subsets of the whole dataset.
Revised: In bagging, instead of training one single model on the whole dataset, multiple models are trained on different subsets of the whole dataset.

Original: Likewise, in each step of the search process, we train multiple kernel functions on uniformly sampled subsets of D , the set of all the available evaluated architectures.
Revised: Likewise, in each step of the search process, we train multiple kernel functions on uniformly sampled subsets of D , the set of all the available evaluated architectures.

Original: Technically, learning multiple kernels refers to learning multiple architecture embedding spaces, i.e. , multiple sets of weights θ .
Revised: Technically, learning multiple kernels refers to learning multiple architecture embedding spaces, i.e. , multiple sets of weights θ .

Original: After training the kernels, each kernel is used separately to compute one posterior distribution and determine one architecture to evaluate in the next step.
Revised: After training the kernels, each kernel is used separately to compute one posterior distribution and determine one architecture to evaluate in the next step.

Original: That is to say, if we train K kernels in the current step, we will obtain K architectures to evaluate in the next step.
Revised: That is to say, if we train K kernels in the current step, we will obtain K architectures to evaluate in the next step.

Original: The proposed multiple kernel strategy encourages the search process to explore more diverse architectures and can help ﬁnd better architectures than training one single kernel only. 
Revised: The proposed multiple kernel strategy encourages the search process to explore more diverse architectures and can help ﬁnd better architectures than training one single kernel only. do Evaluate the x x Kt . do

Original: When training kernels, we randomly initializetheir weights and learn the weights from the scratch on subsets of evaluated architectures.
Revised: Randomly initialize the weights of kernel k , denoted as θ Randomly sample a subset of D , denoted as D k . Learn θ k on D k using the objective function in Eq. Compute the posterior distribution conditioned on the architectures in D k with kernel k .

Revised: Maximize the acquisition function denote the solution as x end for

Original:  We do not learn the weights of the kernel based on the weights learned in the last step, i.e. , ﬁne-tuning the Bi-LSTM from the last step.
Revised: Return the best D the When training kernels, we randomly initialize their weights and learn the weights from the scratch on subsets of evaluated architectures. We do not learn the weights of the kernel based on the weights learned in the last step, i.e. , ﬁne-tuning the Bi-LSTM from the last step.

Original: The training of the Bi-LSTM is fast since we usually only evaluate hundreds of architectures during the whole search process.
Revised: The training of the Bi-LSTM is fast since we usually only evaluate hundreds of architectures during the whole search process.

Original: A formal sketch of our search algorithm in shown Algorithm 1.
Revised: A formal sketch of our search algorithm in shown Algorithm 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst extensively evaluate our algorithm with different teacher architectures and datasets.
Revised: We ﬁrst extensively evaluate our algorithm with different teacher architectures and datasets.

Original: We then compare the automatically found compressed architectures to the state-of-the-art manually-designed compact architecture, ShufﬂeNet (Zhang et al., 2018).
Revised: We then compare the automatically found compressed architectures to the state-of-the-art manually-designed compact architecture, ShufﬂeNet (Zhang et al., 2018).

Original: We also evaluate the transfer performance of Input : Number of steps T . Number of kernels K . Teacher network x teacher .
Revised: We also evaluate the transfer performance of the learned embedding space and kernel.

Original: Randomly sample K architectures x 11 , . . .

Original: , x K 1 from the search space deﬁned based on x teacher .

Original: Initialize the set of evaluated architectures D = ∅ .

Original: for t = 1 , . . .

Original: , T do

Original: Evaluate the K architectures x 1 t , . . .

Original: , x Kt .

Original: D

Original: = D ∪ { x 1 t , . . .

Original: , x Kt } .

Original: for k = 1 , . . .

Original: K do

Original: Randomly initialize the weights of kernel k , denoted as θ k .

Original: Randomly sample a subset of D , denoted

Original: [Table]

Original: k .

Original: Learn

Original: [Table]

Original: k on D k using the objective function in Eq.

Original: Compute the posterior distribution conditioned on the architectures in D k with kernel k .

Original: Maximize the acquisition function and denote the solution as x kt +1 .

Original: end for end for

Original: Return the best architecture in D as the solution.

Original: [Table]

Original: ShufﬂeNet

Original: [Table]

Original: the learned embedding space and kernel.

Original: We perform ablation study to understand how the number of kernels K and other design choices in our search algorithm inﬂuence the performance.
Revised: We perform ablation study to understand how the number of kernels K and other design choices in our search algorithm inﬂuence the performance.

Original: Due to space constraints, the ablation study is included in Appendix 6.3.
Revised: Due to space constraints, the ablation study is included in Appendix 6.3.

Original: 4. C OMPRESSION E XPERIMENTS
Revised: 4. C OMPRESSION E XPERIMENTS

Original: We use two datasets: CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).
Revised: We use two datasets: CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).

Original: CIFAR-10 contains 60K images in 10 classes, with 6K images per class.
Revised: CIFAR-10 contains 60K images in 10 classes, with 6K images per class.

Original: CIFAR-100 also contains 60K images but in 100 classes, with 600 images per class.
Revised: CIFAR-100 also contains 60K images but in 100 classes, with 600 images per class.

Original: Both CIFAR-10 and CIFAR-100 are divided into a training set with 50K images and a test set with 10K images.
Revised: Both CIFAR-10 and CIFAR-100 are divided into a training set with 50K images and a test set with 10K images.

Original: We sample 5K images from the training set as the validation set.
Revised: We sample 5K images from the training set as the validation set.

Original: We provide results on four architectures as the teacher network: VGG19 (Simonyan & Zisserman, 2014), ResNet-18, ResNet-34 (He et al., 2016) and ShufﬂeNet (Zhang et al., 2018).
Revised: We provide results on four architectures as the teacher network: VGG19 (Simonyan & Zisserman, 2014), ResNet-18, ResNet-34 (He et al., 2016) and ShufﬂeNet (Zhang et al., 2018).

Original: We consider two baselines algorithms for comparison: random search (RS) and a reinforcement learning based approach, N2N (Ashok et al., 2018).
Revised: We consider two baselines algorithms for comparison: random search (RS) and a reinforcement learning based approach, N2N (Ashok et al., 2018).

Original: Here we use RS to directly maximize the compression objective f ( x ) .
Revised: Here we use RS to directly maximize the compression objective f ( x ) .

Original: To be more speciﬁc, RS randomly samples architectures in the search space, then evaluates all of them and returns the best architecture as the optimal solution.
Revised: To be more speciﬁc, RS randomly samples architectures in the search space, then evaluates all of them and returns the best architecture as the optimal solution.

Original: In the following experiments, RS evaluates 160 architectures.
Revised: In the following experiments, RS evaluates 160 architectures.

Original: For our proposed method, we run 20 architecture search steps, where each step generates K = 8 architectures for evaluation based on the the
Revised: For our proposed method, we run 20 architecture search steps, where each step generates K = 8 architectures for evaluation based on the the

Original: K different kernels.
Revised: K different kernels.

Original: This means our proposed method evaluates 160 ( 20 × 8 ) architectures in total during the search process.
Revised: This means our proposed method evaluates 160 ( 20 × 8 ) architectures in total during the search process.

Original: Note that when evaluating an architecture during the search process, we only train it for 10 epochs to reduce computation time.
Revised: Note that when evaluating an architecture during the search process, we only train it for 10 epochs to reduce computation time.

Original: So for both RS and our proposed, we fully train the top 4 architectures among the 160 evaluated architectures and choose the best one as the solution.
Revised: So for both RS and our proposed, we fully train the top 4 architectures among the 160 evaluated architectures and choose the best one as the solution.

Original: When learning the kernel function parameters, we randomly sample from the set of the evaluated architectures with a probability of 0.5 to form the training set for one kernel.
Revised: When learning the kernel function parameters, we randomly sample from the set of the evaluated architectures with a probability of 0.5 to form the training set for one kernel.

Original: The results of N2N are identical to the original paper Ashok et al.
Revised: The results of N2N are identical to the original paper Ashok et al.

Revised: [Table]

Original: The compression results are summarized in Table 1.
Revised: The compression results are summarized in Table 1.

Original: For a compressed network x , ‘Ratio’ refers to the compression ratio of x , which is deﬁned as (cid:16) 1 − # params( x )
Revised: For a compressed network x , ‘Ratio’ refers to the compression ratio of x , which is deﬁned as (cid:16) 1 − [Table] params

Original: # params ( x teacher ) (cid:17) .
Revised: [Table] x ) [Table] params ( x teacher ) (cid:17) .

Original: ‘Times’ refers to the ratio between the size of the teacher network and the size of the compressed network, i.e. , # params ( x teacher )
Revised: ‘Times’ refers to the ratio between the size of the teacher network and the size of the compressed network, i.e. , # params ( x teacher )

Original: # params ( x ) .
Revised: # params ( x ) .

Revised: [Table]

Original: We also show the value of f ( x ) as an indication of how well each architecture x meets our requirement in terms of both the accuracy and the compression ratio.
Revised: We also show the value of f ( x ) as an indication of how well each architecture x meets our requirement in terms of both the accuracy and the compression ratio.

Revised: For ‘Random Search’ and ’Ours’, we run the experiments for three times and report the average results as well as the standard deviation.

Original: We ﬁrst apply our algorithm to compress three popular network architectures: VGG-19, ResNet18 and ResNet-34, and use them as the teacher network.
Revised: We ﬁrst apply our algorithm to compress three popular network architectures: VGG-19, ResNet18 and ResNet-34, and use them as the teacher network.

Original: We can see that on both CIFAR-10 and
Revised: We can see that on both CIFAR-10 and

Original: CIFAR-100, our proposed method consistently ﬁnds architectures that can achieve higher value of f ( x ) than all baselines.
Revised: CIFAR-100, our proposed method consistently ﬁnds architectures that can achieve higher value of f ( x ) than all baselines.

Original: For VGG-19 on CIFAR-100, the architecture found by our algorithm is 10 times smaller than the original teacher network while the accuracy only drops by 1 . 51% .
Revised: For VGG-19 on CIFAR-100, the architecture found by our algorithm is 8 times smaller than the original teacher network while the accuracy only drops by 2 .

Revised: For

Original: For ResNet-18 on CIFAR-100, the architecture found by our algorithm has a similar number of parameters as the RSresult but can improved accuracy by 3 .
Revised: ResNet-18 on CIFAR-100, the architecture found by our algorithm has a little bit more parameters than that found by RS but has higer accuracy by about 4% .

Original: For ResNet-34 on CIFAR-100, the architecture found by our proposed method has a similar accuracy as the architecture discovered by RS but only uses about 57% of the number of parameters.
Revised: For ResNet-34 on CIFAR-100, the architecture found by our proposed method has a higher accuracy as the architecture discovered by RS but only uses about 65% of the number of parameters.

Original: Also for ResNet-34 on CIFAR-100, N2N only provides the results of layer removal, denoted as ‘N2N-removal’.
Revised: Also for ResNet-34 on CIFAR-100, N2N only provides the results of layer removal, denoted as ‘N2N - removal’.

Original: ‘Ours-removal’ refers to only considering the layer removal operation in the search space for fair comparison.
Revised: ‘Ours - removal’ refers to only considering the layer removal operation in the search space for fair comparison.

Original: We can see that ‘Ours-removal’ also signiﬁcantly outperforms ‘N2N-removal’ in terms of both the accuracy and the compression ratio.
Revised: We can see that ‘Ours - removal’ also signiﬁcantly outperforms ‘N2N - removal’ in terms of both the accuracy and the compression ratio.

Original: ShufﬂeNet is an extremely computation-efﬁcient human-designed CNN architecture (Zhang et al., 2018).
Revised: ShufﬂeNet is an extremely computation-efﬁcient human-designed CNN architecture (Zhang et al., 2018).

Original: We also have tried to use ShufﬂeNet as the teacher network and see if we can further optimize this architecture.
Revised: We also have tried to use ShufﬂeNet as the teacher network and see if we can further optimize this architecture.

Original: As shown in Table 1, our search algorithm successfully compresses ‘ShufﬂeNet 1 × ( g = 2) ’ by 7 . 50 × and 4 .
Revised: As shown in Table 1, our search algorithm successfully compresses ‘ShufﬂeNet 1 × ( g = 2) ’ by 10 . 43 × and 4 .

Original: 08 × on CIFAR-10 and CIFAR-100 respectively and the compressed architectures can still achieve similar accuracy to the original teacher network.
Revised: 74 × on CIFAR-10 and CIFAR-100 respectively and the compressed architectures can still achieve similar accuracy to the original teacher network.

Original: Here ‘ 1 × ’ indicates the number of channels in the teacher ShufﬂeNet and ‘ ( g = 2) ’ indicates that the number of groups is 2.
Revised: Here ‘ 1 × ’ indicates the number of channels in the teacher ShufﬂeNet and ‘ ( g = 2) ’ indicates that the number of groups is 2.

Original: Readers can refer to Zhang et al.
Revised: Readers can refer to Zhang et al.

Original: (2018) for more details about the speciﬁc conﬁguration.
Revised: (2018) for more details about the speciﬁc conﬁguration.

Original: 4. C OMPARISON TO S HUFFLE N ET
Revised: 4. C OMPARISON TO S HUFFLE N ET

Original: We now compare the compressed architectures found by our algorithm to the state-of-the-art manually-designed compact network architecture ShufﬂeNet.
Revised: We now compare the compressed architectures found by our algorithm to the state-of-the-art manually-designed compact network architecture ShufﬂeNet.

Original: We vary the number of channels and the number of groups in ShufﬂeNet and compare the compressed architectures of our proposed method against these different conﬁgurations of ShufﬂeNet.
Revised: We vary the number of channels and the number of groups in ShufﬂeNet and compare the compressed architectures of our proposed method against these different conﬁgurations of ShufﬂeNet.

Original: We conduct experiments on CIFAR100 and the results are summarized in Table 2.
Revised: We conduct experiments on CIFAR100 and the results are summarized in Table 2.

Revised: For ’Ours’ in Table 2, we use the mean results of 3 runs of our method.

Original: In Table 2, VGG-19, ResNet-18, ResNet-34 and[Table] [Table] ShufﬂeNet refer to the compressed architectures found by our algorithm based on the corresponding teacher network and do not refer to the original architecture indicated by the name.
Revised: In Table 2, VGG-19, ResNet-18, ResNet-34 and ShufﬂeNet refer to the compressed architectures found by our algorithm based on the corresponding teacher network and do not refer to the original architecture indicated by the name.

Original: The teacher ShufﬂeNet used in the experiments is ‘ShufﬂeNet 1 × ( g = 2) ’ as mentioned above.
Revised: The teacher ShufﬂeNet used in the experiments is ‘ShufﬂeNet 1 × ( g = 2) ’ as mentioned above.

Original: 5 × ( g = 1) ’ and so on in Table 2 refer to different conﬁgurations of ShufﬂeNet and we show the accuracy of these original ShufﬂeNet in the table.
Revised: 5 × ( g = 1) ’ and so on in Tablerefer to different conﬁgurations of ShufﬂeNet and we show the accuracy of these original ShufﬂeNet in the table.

Original: The compressed architectures found based on VGG-19, ResNetand ResNet-34 have a similar number of parameters with ShufﬂeNet 1 .
Revised: The compressed architectures found based on ResNet-18 and ResNet-34 have a similar number of parameters with ShufﬂeNet 1 .

Original: 5 ×
Revised: 5 ×

Original: but they can all achieve much higher accuracy than ShufﬂeNet 1 . 5 × .
Revised: but they can all achieve much higher accuracy than ShufﬂeNet 1 . 5 × .

Original: The compressed architecture found based on ShufﬂeNet 1 × ( g = 2) can obtain higher accuracy than ShufﬂeNet 0 .
Revised: The compressed architecture found based on ShufﬂeNet 1 × ( g = 2) can obtain higher accuracy than ShufﬂeNet 0 .

Original: 5 × while using a similar number of parameters.
Revised: 5 × while using a similar number of parameters.

Original: 4. K ERNEL T RANSFER
Revised: 4. K ERNEL T RANSFER

Original: We now study the transferability of the learned embedding space or the learned kernel.
Revised: We now study the transferability of the learned embedding space or the learned kernel.

Original: We would like to know to what extent a kernel learned in one setting can be generalized to a new setting.
Revised: We would like to know to what extent a kernel learned in one setting can be generalized to a new setting.

Original: To be more speciﬁc about the kernel transfer, we ﬁrst learn one kernel or multiple kernels in the  source setting.
Revised: To be more speciﬁc about the kernel transfer, we ﬁrst learn one kernel or multiple kernels in the [Table] source setting.

Original: Then we maximize the acquisition function within the search space in the target setting and the acquisition function is computed based on the kernel learned in the source setting.
Revised: Then we maximize the acquisition function within the search space in the target setting and the acquisition function is computed based on the kernel learned in the source setting.

Original: The maximizer of the acquisition function is a compressed architecture for the target setting.
Revised: The maximizer of the acquisition function is a compressed architecture for the target setting.

Original: We evaluate this architecture in the target setting and compare it with the architecture found by applying algorithms directly to the target setting.
Revised: We evaluate this architecture in the target setting and compare it with the architecture found by applying algorithms directly to the target setting.

Original: We consider the following four settings: (a) ResNet-18 on CIFAR-10, (b) ResNet-34 on CIFAR-10, (c) VGG-19 on CIFAR-10 and (d) ResNet-18 on CIFAR-100.
Revised: We consider the following four settings: (a) ResNet-18 on CIFAR-10, (b) ResNet-34 on CIFAR-10, (c) VGG-19 on CIFAR-10 and (d) ResNet-18 on CIFAR-100.

Original: ‘ResNet-18 on CIFAR-10’ refers to searching for a compressed architecture with ResNet-18 as the teacher network for the dataset
Revised: ‘ResNet-18 on CIFAR-10’ refers to searching for a compressed architecture with ResNet-18 as the teacher network for the dataset

Original: CIFAR-10 and so on.
Revised: CIFAR-10 and so on.

Original: We ﬁrst run our search algorithm in setting (a) and transfer the learned kernel to setting (b), (c) and (d) respectively to see how much the learned kernel can transfer to a larger teacher network in the same architecture family (this means a larger search space), a different architecture family (this means a totally different search space) or a harder dataset.
Revised: We ﬁrst run our search algorithm in setting (a) and transfer the learned kernel to setting (b), (c) and (d) respectively to see how much the learned kernel can transfer to a larger teacher network in the same architecture family (this means a larger search space), a different architecture family (this means a totally different search space) or a harder dataset.

Original: We learn K kernels in the source setting (a) and we transfer all the K kernels to the target setting, which will result in K compressed architectures for the target setting.
Revised: We learn K kernels in the source setting (a) and we transfer all the K kernels to the target setting, which will result in K compressed architectures for the target setting.

Original: We report the best one among the K architectures.
Revised: We report the best one among the K architectures.

Original: We have tried K = 1 and K = 8 and the results are shown in Table 3.
Revised: We have tried K = 1 and K = 8 and the results are shown in Table 3.

Original: In all the three transfer scenarios, the learned kernel in the source setting (a) can help ﬁnd reasonably good architectures in the target setting without actually training the kernel in the target setting, whose performance is better than the architecture found by applying N2N directly to the target setting.
Revised: In all the three transfer scenarios, the learned kernel in the source setting (a) can help ﬁnd reasonably good architectures in the target setting without actually training the kernel in the target setting, whose performance is better than the architecture found by applying N2N directly to the target setting.

Original: These results proves that the learned architecture embedding space or the learned kernel is able to generalize to new settings for architecture search without any additional training.
Revised: These results proves that the learned architecture embedding space or the learned kernel is able to generalize to new settings for architecture search without any additional training.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We address the task of searching for a compressed network architecture by using BO.
Revised: We address the task of searching for a compressed network architecture by using BO.

Original: Our proposed method can ﬁnd more efﬁcient architectures than all the baselines on CIFAR-10 and CIFAR-100.
Revised: Our proposed method can ﬁnd more efﬁcient architectures than all the baselines on CIFAR-10 and CIFAR-100.

Original: One key contribution is our proposed method to learn an embedding space over the domain of network architectures.
Revised: One key contribution is our proposed method to learn an embedding space over the domain of network architectures.

Original: We also demonstrate that the learned embedding space can be transferred to new settings for architecture search without any training.
Revised: We also demonstrate that the learned embedding space can be transferred to new settings for architecture search without any training.

Revised: Possible future directions include extending our method to the general NAS problem to search for desired architectures from the scratch and combing our proposed embedding space with Hern´andez-Lobato et al.

Revised: (2016) to identify the Pareto set of the architectures that are both small and accurate.

Original:  A PPENDIX 6. D EFINITION OF F UNCTION f
Revised:  A PPENDIX 6. D EFINITION OF F UNCTION f

Original: We now discuss the form of the function f .
Revised: We now discuss the form of the function f .

Original: We aim to ﬁnd a network architecture which contains as few parameters as possible but still can obtain a similar performance to the teacher network.
Revised: We aim to ﬁnd a network architecture which contains as few parameters as possible but still can obtain a similar performance to the teacher network.

Original: Usually compressing a network leads to the decrease in the performance.
Revised: Usually compressing a network leads to the decrease in the performance.

Original: So the function f needs to provide a balance between the compression ratio and the performance.
Revised: So the function f needs to provide a balance between the compression ratio and the performance.

Original: In particular, we hope the function f favors architectures of high performance but low compression ratio more than architectures of low performance but high compression ratio.
Revised: In particular, we hope the function f favors architectures of high performance but low compression ratio more than architectures of low performance but high compression ratio.

Original: So we adopt the reward function design in N2N (Ashok et al., 2018) for the function f .
Revised: So we adopt the reward function design in N2N (Ashok et al., 2018) for the function f .

Original: Formally, f is deﬁned as:
Revised: Formally, f is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: (6) where C ( x ) is the compression ratio of the architecture x , A ( x ) is the validation performance of x and A ( x teacher ) is the validation performance of the teacher network.
Revised: where C ( x ) is the compression ratio of the architecture x , A ( x ) is the validation performance of x and A ( x teacher ) is the validation performance of the teacher network.

Original: The compression ratio C ( x ) is deﬁned as C ( x ) = 1 −
Revised: The compression ratio C ( x ) is deﬁned as C ( x ) = 1 −

Original: [Equation]
Revised: [Equation]

Original: Note that for any x , to evaluate f ( x ) we need to train the architecture x on the training data and test on the validation data.
Revised: Note that for any x , to evaluate f ( x ) we need to train the architecture x on the training data and test on the validation data.

Original: This is time-consuming so during the search process, we do not fully train x .
Revised: This is time-consuming so during the search process, we do not fully train x .

Original: Instead, we only train x for a few epochs and use the validation performance of the network obtained by early stopping as an approximation for A ( x ) .
Revised: Instead, we only train x for a few epochs and use the validation performance of the network obtained by early stopping as an approximation for A ( x ) .

Original: We also employ the Knowledge Distillation (KD) strategy (Hinton et al., 2015) for faster training as we are given a teacher network.
Revised: We also employ the Knowledge Distillation (KD) strategy (Hinton et al., 2015) for faster training as we are given a teacher network.

Original: But when we fully train the architecture x to see its true performance, we ﬁne tune it from the weights obtained by early stopping with cross entropy loss without using KD.
Revised: But when we fully train the architecture x to see its true performance, we ﬁne tune it from the weights obtained by early stopping with cross entropy loss without using KD.

Original: 6. R EPRESENTATION FOR L AYER C ONFIGURATION
Revised: 6. R EPRESENTATION FOR L AYER C ONFIGURATION

Original: We represent the conﬁguration of one layer by a vector of length ( m + 2 n + 6) , where m is the number of types of layers we consider and n is the maximum number of layers in the network.
Revised: We represent the conﬁguration of one layer by a vector of length ( m + 2 n + 6) , where m is the number of types of layers we consider and n is the maximum number of layers in the network.

Original: The ﬁrst m dimensions of the vector are a one-hot vector, indicating the type of the layer.
Revised: The ﬁrst m dimensions of the vector are a one-hot vector, indicating the type of the layer.

Original: Then the following 6 numbers indicate the value of different attributes of the layer, including the kernel size, stride, padding, group, input channels and output channels of the layer.
Revised: Then the following 6 numbers indicate the value of different attributes of the layer, including the kernel size, stride, padding, group, input channels and output channels of the layer.

Original: If one layer does not have any speciﬁc attribute, the value of that attribute is simply set to zero.
Revised: If one layer does not have any speciﬁc attribute, the value of that attribute is simply set to zero.

Original: The following 2 n dimensions encode the edge information of the network, if we view the network as a directed acyclic graph with each layer as a node in the graph.
Revised: The following 2 n dimensions encode the edge information of the network, if we view the network as a directed acyclic graph with each layer as a node in the graph.

Original: In particular, the 2 n dimensions are composed of two n -dim vectors, where one represents the edges incoming to the code and the other one represents the edges outgoing from the node.
Revised: In particular, the 2 n dimensions are composed of two n -dim vectors, where one represents the edges incoming to the code and the other one represents the edges outgoing from the node.

Original: The nodes in the directed acyclic graph can be topologically sorted, which will give each layer an index.
Revised: The nodes in the directed acyclic graph can be topologically sorted, which will give each layer an index.

Original: For an edge from node i to j , the ( j − i ) th element in the outgoing vector of node i and the incoming vector of node j will be 1 .
Revised: For an edge from node i to j , the ( j − i ) th element in the outgoing vector of node i and the incoming vector of node j will be 1 .

Original: We are sure that j is larger than i because all the nodes are topologically sorted.
Revised: We are sure that j is larger than i because all the nodes are topologically sorted.

Original: With this representation, we can describe the connection information in a complex network architecture.[Table]
Revised: With this representation, we can describe the connection information in a complex network architecture.

Original:  A BLATION S TUDY
Revised: 6. A BLATION S TUDY

Original: Impact of number of kernels K : We study the impact of the number of kernels K .
Revised: Impact of number of kernels K : We study the impact of the number of kernels K .

Original: We conduct experiments on CIFAR-100 and use ResNet-34 as the teacher network.
Revised: We conduct experiments on CIFAR-100 and use ResNet-34 as the teacher network.

Original: We vary the value of K and ﬁx the number of evaluated architectures to 160.
Revised: We vary the value of K and ﬁx the number of evaluated architectures to 160.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: We can see  that K = 4 , 8 , 16 yield much better results than K = 1 .
Revised: We can see [Table] that K = 4 , 8 , 16 yield much better results than K = 1 .

Original: Also the performance is not sensitive to K as K = 4 , 8 , 16 yield similar results.
Revised: Also the performance is not sensitive to K as K = 4 , 8 , 16 yield similar results.

Original: In our main experiments, we ﬁx K = 8 .
Revised: In our main experiments, we ﬁx K = 8 .

Original: Impact of adding skip connections : Our search space is deﬁned based on three operations: layer removal, layer shrinkage and adding skip connections.
Revised: Impact of adding skip connections : Our search space is deﬁned based on three operations: layer removal, layer shrinkage and adding skip connections.

Original: A key difference between our search space and N2N Ashok et al.
Revised: A key difference between our search space and N2N Ashok et al.

Original: (2018) is that they only support layer removal and shrinkage do not support adding skip connections.
Revised: (2018) is that they only support layer removal and shrinkage do not support adding skip connections.

Original: To validate the effectiveness of adding skip connections, we conduct experiments on CIFAR-100 and on three architectures.
Revised: To validate the effectiveness of adding skip connections, we conduct experiments on CIFAR-100 and on three architectures.

Original: In Table 5, ’Ours - removal + shrink’ refers to the search space without considering adding skip connections and ’Ours’ refers to using the full search space.
Revised: In Table 5, ’Ours - removal + shrink’ refers to the search space without considering adding skip connections and ’Ours’ refers to using the full search space.

Original: We can see that ’Ours’ consistently outperforms ’Ours - removal + shrink’ across different teacher networks, proving the effectiveness of adding skip connections.
Revised: We can see that ’Ours’ consistently outperforms ’Ours - removal + shrink’ across different teacher networks, proving the effectiveness of adding skip connections.

Original: Impact of the maximization of the acquisition function : As mentioned in Section 3.2, we have two choices to maximize the acquisition function EI t ( x ) : randomly sampling (RS) and evolutionary algorithm (EA).
Revised: Impact of the maximization of the acquisition function : As mentioned in Section 3.2, we have two choices to maximize the acquisition function EI t ( x ) : randomly sampling (RS) and evolutionary algorithm (EA).

Original: We conduct the experiments to compare RS and ES to compress ResNet-34 on
Revised: We conduct the experiments to compare RS and ES to compress ResNet-34 on

Original: CIFAR-100.
Revised: CIFAR-100.

Original: We ﬁnd that although EA is empirically better than RS in terms of maximizing EI t ( x ) ,
Revised: We ﬁnd that although EA is empirically better than RS in terms of maximizing EI t ( x ) ,

Original: EA is slightly worse than RS in terms of the ﬁnal search performance as shown in Table 6.
Revised: EA is slightly worse than RS in terms of the ﬁnal search performance as shown in Table 6.

Original: For any EI t ( x ) , the solution found by EA x EA may be better than the solution found by RS x RS , i
Revised: For any EI t ( x ) , the solution found by EA x EA may be better than the solution found by RS x RS , i.e. ,

Original: [Table] e. , EI t ( x EA ) > EI t ( x RS ) .
Revised: EI t ( x EA ) > EI t ( x RS ) .

Original: However, we observe that f ( x EA ) and f ( x RS ) are usually similar.
Revised: However, we observe that f ( x EA ) and f ( x RS ) are usually similar.

Original: We also plot the values of f ( x ) for the evaluated architectures when using RS and EA to maximize the acquisition function respectively in Figure 1.
Revised: We also plot the values of f ( x ) for the evaluated architectures when using RS and EA to maximize the acquisition function respectively in Figure 1.

Original: We can see that the function value of the evaluated architectures grows slightly more stable when using RS to maximize the acquisition function then using EA.
Revised: We can see that the function value of the evaluated architectures grows slightly more stable when using RS to maximize the acquisition function then using EA.

Original: Therefore, we choose RS in the following experiments for the sake of simplicity.
Revised: Therefore, we choose RS in the following experiments for the sake of simplicity.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2.68M 0.

Original: 7.97x 0.

Original: RS, K =73.45% 2.12M 0.

Original: 10.06x 0.

Original: EA, K =71.52% 1.24M 0.

Original: 17.23x 0.

Original: EA, K =72.40% 2.15M 0.

Original: 9.90x 0.

Original: [Figure]

Original: [Figure]

Original: 6. R ANDOM S AMPLING IN S EARCH S PACE
Revised: 6. R ANDOM S AMPLING IN S EARCH S PACE

Original: We need to randomly sample architectures in the search space when optimizing the acquision function.
Revised: We need to randomly sample architectures in the search space when optimizing the acquisition function.

Original: As mentioned in Section 3.2, we sample the architectures by sampling the operations to  manipulate the architecture of the teacher network.
Revised: As mentioned in Section 3.2, we sample the architectures by sampling the operations to [Figure] [Figure] manipulate the architecture of the teacher network.

Original: During the process, we need to make sure the layers in the network are still compatible with each other in terms of the dimension of the feature map.
Revised: During the process, we need to make sure the layers in the network are still compatible with each other in terms of the dimension of the feature map.

Original: Therefore, We impose some conditions when we sample the operations in order to maintain the consistency between between layers.
Revised: Therefore, We impose some conditions when we sample the operations in order to maintain the consistency between between layers.

Original: For layer removal, only layers whose input dimension and output dimension are the same are allowed to be removed.
Revised: For layer removal, only layers whose input dimension and output dimension are the same are allowed to be removed.

Original: For layer shrinkage, we divide layers into groups and for layers in the same group, the number of channels are always shrinked with the same ratio.
Revised: For layer shrinkage, we divide layers into groups and for layers in the same group, the number of channels are always shrunken with the same ratio.

Original: The layers are grouped according to their input and output dimension.
Revised: The layers are grouped according to their input and output dimension.

Original: For adding skip connections, only when the output dimension of one layer is the same as the input dimension of another layer, the two layers can be connected.
Revised: For adding skip connections, only when the output dimension of one layer is the same as the input dimension of another layer, the two layers can be connected.

Original: When there are multiple incoming edges for one layer in the computation graph, the outputs of source layers are added up to form the input for that layer.
Revised: When there are multiple incoming edges for one layer in the computation graph, the outputs of source layers are added up to form the input for that layer.

Original: When compressing ShufﬂeNet, we also slightly modify the original architecture before compression.
Revised: When compressing ShufﬂeNet, we also slightly modify the original architecture before compression.

Original: We insert a 1 × 1 convolutional layer before each average pooling layer.
Revised: We insert a 1 × 1 convolutional layer before each average pooling layer.

Original: This modiﬁcation increases parameters by about 10% and does not signiﬁcantly inﬂuence the performance of ShufﬂeNet.
Revised: This modiﬁcation increases parameters by about 10% and does not signiﬁcantly inﬂuence the performance of ShufﬂeNet.

Original: Note that the modiﬁcation only happens when we need to compress ShufﬂeNet and does not inﬂuence the performance of the original ShufﬂeNet shown in Table 2.
Revised: Note that the modiﬁcation only happens when we need to compress ShufﬂeNet and does not inﬂuence the performance of the original ShufﬂeNet shown in Table 2.

Original: 6. A NALYSIS OF R ANDOM S EARCH B ASELINE
Revised: 6. A NALYSIS OF R ANDOM S EARCH B ASELINE

Original: We observe that the random search (RS) baseline which maximizes f ( x ) with random sampling can achieve very good performance.
Revised: We observe that the random search (RS) baseline which maximizes f ( x ) with random sampling can achieve very good performance.

Original: To analyze RS in more detail, we show the value of f ( x ) for the 160 architectures evaluated in the search process in Figure 2.
Revised: To analyze RS in more detail, we show the value of f ( x ) for the 160 architectures evaluated in the search process in Figure 2.

Original: The speciﬁc setting we choose is
Revised: The speciﬁc setting we choose is

Original: ResNet-34 on CIFAR-100.
Revised: ResNet-34 on CIFAR-100.

Original: We can see that although RS can sometimes sample good architectures with high f ( x ) value, it is much more unstable than our method.
Revised: We can see that although RS can sometimes sample good architectures with high f ( x ) value, it is much more unstable than our method.

Original: The function value of the evaluated architectures selected by our method has a strong tendency to grow as we search more steps while
Revised: The function value of the evaluated architectures selected by our method has a strong tendency to grow as we search more steps while

Original: RS does not show such trend.
Revised: RS does not show such trend.

Original: Also, from the histogram of values of f ( x ) , we can see that RS has a much lower chance to get architectures with high function values than our method.
Revised: Also, from the histogram of values of f ( x ) , we can see that RS has a much lower chance to get architectures with high function values than our method.

Original: This is expected since our method leverages the learned architecture embedding or the kernel function to carefully select the architecture for evaluation while RS just randomly samples from the search space.
Revised: This is expected since our method leverages the learned architecture embedding or the kernel function to carefully select the architecture for evaluation while RS just randomly samples from the search space.

Original: We can conclude that our method is much more efﬁcient than RS.
Revised: We can conclude that our method is much more efﬁcient than RS.