Original: N EURAL O PERATOR S EARCH
Revised: N EURAL O PERATOR S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing neural architecture search (NAS) methods explore a limited featuretransformation-only search space, ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions.
Revised: Existing neural architecture search (NAS) methods explore a limited featuretransformation-only search space, ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions.

Original: This disables the NAS algorithms to discover more advanced network architectures.
Revised: This disables the NAS algorithms to discover more advanced network architectures.

Original: We address this limitation by additionally exploiting feature self-calibration operations, resulting in a heterogeneous search space.
Revised: We address this limitation by additionally exploiting feature self-calibration operations, resulting in a heterogeneous search space.

Original: To solve the challenges of operation heterogeneity and signiﬁcantly larger search space, we formulate a neural operator search (NOS) method.
Revised: To solve the challenges of operation heterogeneity and signiﬁcantly larger search space, we formulate a neural operator search (NOS) method.

Original: NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in a uniﬁed structure, and an attention guided search strategy for facilitating the search process over a vast space.
Revised: NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in a uniﬁed structure, and an attention guided search strategy for facilitating the search process over a vast space.

Original: Extensive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.
Revised: Extensive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent advances of Neural Architecture Search (NAS) are remarkable in challenging tasks, e.g. image classiﬁcation (Zoph & Le, 2017), object detection (Ghiasi et al., 2019), and semantic segmentation (Liu et al., 2019a; Nekrasov et al., 2019), greatly alleviating the demands for human knowledge and interventions by automating the laborious process of designing neural network architectures.
Revised: Recent advances of Neural Architecture Search (NAS) are remarkable in challenging tasks, e.g. image classiﬁcation (Zoph & Le, 2017), object detection (Ghiasi et al., 2019), and semantic segmentation (Liu et al., 2019a; Nekrasov et al., 2019), greatly alleviating the demands for human knowledge and interventions by automating the laborious process of designing neural network architectures.

Original: One common scheme for the standard proxy-based neural architecture search methods (Pham et al., 2018; Zoph et al., 2018; Liu et al., 2019b) is to factorise the search space via repeatedly stacking the same cell structure, within which a computing block generates an output tensor F k by combining the transformations of two input feature tensors F i and F j :
Revised: One common scheme for the standard proxy-based neural architecture search methods (Pham et al., 2018; Zoph et al., 2018; Liu et al., 2019b) is to factorise the search space via repeatedly stacking the same cell structure, within which a computing block generates an output tensor F k by combining the transformations of two input feature tensors F i and F j :

Original: [Equation]
Revised: [Equation]

Original: where o i Ñ k and o j Ñ k are the i -th and j -th primitive operations for feature transformation, selected from a candidate operation set O , and ‘ is the element-wise addition.
Revised: where o i Ñ k and o j Ñ k are the i -th and j -th primitive operations for feature transformation, selected from a candidate operation set O , and ‘ is the element-wise addition.

Original: Existing NAS methods use only the standard feature learning/transformation operations (convolution, pooling and identity mapping) as the building components.
Revised: Existing NAS methods use only the standard feature learning/transformation operations (convolution, pooling and identity mapping) as the building components.

Original: Besides, extensive studies (Hu et al., 2018b; Bertinetto et al., 2016; Wang et al., 2017; Jia et al., 2016;
Revised: Besides, extensive studies (Hu et al., 2018b; Bertinetto et al., 2016; Wang et al., 2017; Jia et al., 2016;

Original: Wu et al., 2019b; Zhu et al., 2019) have proven that other advanced operations for feature self-calibration , such as attention learning and dynamic convolutions , can bring great beneﬁts for representation learning.
Revised: Wu et al., 2019b; Zhu et al., 2019) have proven that other advanced operations for feature self-calibration , such as attention learning and dynamic convolutions , can bring great beneﬁts for representation learning.

Original: For example, Hu et al.
Revised: For example, Hu et al.

Original: (2018b) proposes Squeeze-and-Excitation Networks to explicitly model inter-dependencies between channels by learning channel-wise self-attention.
Revised: (2018b) proposes Squeeze-and-Excitation Networks to explicitly model inter-dependencies between channels by learning channel-wise self-attention.

Original: Jia et al.
Revised: Jia et al.

Original: (2016) presents Dynamic Filter Networks to generate context-aware ﬁlters for increasing the ﬂexibility and adaptiveness of networks.
Revised: (2016) presents Dynamic Filter Networks to generate context-aware ﬁlters for increasing the ﬂexibility and adaptiveness of networks.

Original: However, these useful feature calibration elements have never been well exploited in NAS, signiﬁcantly limiting the potentials of NAS which aims for automatically discovering more sophisticated and advanced network architectures without human engineering.
Revised: However, these useful feature calibration elements have never been well exploited in NAS, signiﬁcantly limiting the potentials of NAS which aims for automatically discovering more sophisticated and advanced network architectures without human engineering.

Original: In this work, we aim to address this limitation by extending the search space of NAS with feature self-calibration operations for scaling up the search boundary.
Revised: In this work, we aim to address this limitation by extending the search space of NAS with feature self-calibration operations for scaling up the search boundary.

Original: This makes a heterogeneous search space .
Revised: This makes a heterogeneous search space .

Original: Consequently, the way of feature tensor interaction and combination is dramatically diversiﬁed, from the conventional addition operator ‘ only to the combination of addition ‘ , multiplication d for attention modelling, and dynamic convolution (cid:126) .
Revised: Consequently, the way of feature tensor interaction and combination is dramatically diversiﬁed, from the conventional addition operator ‘ only to the combination of addition ‘ , multiplication d for attention modelling, and dynamic convolution (cid:126) .

Original: In this regard,we call the proposed method
Revised: In this regard,we call the proposed method

Original: Neural Operator Search (NOS).
Revised: Neural Operator Search (NOS).

Original: Such a search space enhancement is critical since NAS is enabled to explore stronger and previously undiscovered network architectures, which opens a door to potentially take the NAS research to the next level.
Revised: Such a search space enhancement is critical since NAS is enabled to explore stronger and previously undiscovered network architectures, which opens a door to potentially take the NAS research to the next level.

Original: In the no free lunch saying, this also comes with two new challenges: (i) It is non-trivial and more challenging to assemble such heterogeneous tensors and operations (i.e. features, attentions and dynamic weights) in a uniﬁed computing block, as compared to the conventional homogeneous feature-tensor-to-feature-tensor transformation; (ii) The search space increases exponentially which leads to a much harder NAS problem.
Revised: In the no free lunch saying, this also comes with two new challenges: (i) It is non-trivial and more challenging to assemble such heterogeneous tensors and operations (i.e. features, attentions and dynamic weights) in a uniﬁed computing block, as compared to the conventional homogeneous feature-tensor-to-feature-tensor transformation; (ii) The search space increases exponentially which leads to a much harder NAS problem.

Original: To address the ﬁrst challenge, we formulate a heterogeneous operator cell characterised by a novel heterogeneous residual block.
Revised: To address the ﬁrst challenge, we formulate a heterogeneous operator cell characterised by a novel heterogeneous residual block.

Original: This block, formulated in a residual learning spirit (He et al., 2016), is designed specially for fusing all the different types of tensors and operations synergistically.
Revised: This block, formulated in a residual learning spirit (He et al., 2016), is designed specially for fusing all the different types of tensors and operations synergistically.

Original: To solve the second challenge, we propose leveraging the attention transfer (Zagoruyko & Komodakis, 2017) idea to facilitate the search behaviour across this signiﬁcantly larger network space via following the attention guidance of a pretrained teacher model.
Revised: To solve the second challenge, we propose leveraging the attention transfer (Zagoruyko & Komodakis, 2017) idea to facilitate the search behaviour across this signiﬁcantly larger network space via following the attention guidance of a pretrained teacher model.

Original: As we will show, this guidance not only makes the search more efﬁcient but also improves the search result.
Revised: As we will show, this guidance not only makes the search more efﬁcient but also improves the search result.

Original: Our contributions in this work are: (1) We present a novel heterogeneous search space for NAS characterised by richer primitive operations including both conventional feature transformations and newly introduced feature self-calibration.
Revised: Our contributions in this work are: (1) We present a novel heterogeneous search space for NAS characterised by richer primitive operations including both conventional feature transformations and newly introduced feature self-calibration.

Original: This breaks the conventional selection limit of candidate neural networks and enables the NAS process to ﬁnd stronger architectures, many of which are impossible to be discovered in the conventional space.
Revised: This breaks the conventional selection limit of candidate neural networks and enables the NAS process to ﬁnd stronger architectures, many of which are impossible to be discovered in the conventional space.

Original: This opens new territories for supporting stronger NAS algorithms and new possibilities for most expressive architectures ever to be revealed.
Revised: This opens new territories for supporting stronger NAS algorithms and new possibilities for most expressive architectures ever to be revealed.

Original: (2) We formulate a novel Neural Operator Search (NOS) method dedicated for NAS in the proposed heterogeneous search space, with a couple of key designs – heterogeneous residual block for fusing different types of tensor operations synergistically and attention guided search for facilitating the search process over a vast search space more efﬁciently and more effectively.
Revised: (2) We formulate a novel Neural Operator Search (NOS) method dedicated for NAS in the proposed heterogeneous search space, with a couple of key designs – heterogeneous residual block for fusing different types of tensor operations synergistically and attention guided search for facilitating the search process over a vast search space more efﬁciently and more effectively.

Original: (3) With extensive comparisons to the state-of-the-art NAS methods, the experiments show that our approach is highly competitive on both CIFAR and ImageNet-mobile image classiﬁcation tests.
Revised: (3) With extensive comparisons to the state-of-the-art NAS methods, the experiments show that our approach is highly competitive on both CIFAR and ImageNet-mobile image classiﬁcation tests.

Original:  R ELATED W
Revised:  R ELATED W

Original: ORK
Revised: ORK

Original: Neural Architecture Search.
Revised: Neural Architecture Search.

Original: Since the seminal work by Zoph & Le (2017), neural architecture search has gained a surge of interest, effectively replacing laborious human designs by the computational process.
Revised: Since the seminal work by Zoph & Le (2017), neural architecture search has gained a surge of interest, effectively replacing laborious human designs by the computational process.

Original: From the strategy point of view, NAS methods can be categorised into two types: (1) proxy-based (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018;
Revised: From the strategy point of view, NAS methods can be categorised into two types: (1) proxy-based (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018;

Original: Liu et al., 2019b) and (2) proxy-less (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019a) NAS.
Revised: Liu et al., 2019b) and (2) proxy-less (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019a) NAS.

Original: Speciﬁcally, to alleviate the computational cost during search, the proxy-based NAS methods search for building cells on proxy tasks, with one or more of following compromised strategies: starting with fewer cells; using a smaller dataset (e.g. CIFAR-10); learning with fewer epochs.
Revised: Speciﬁcally, to alleviate the computational cost during search, the proxy-based NAS methods search for building cells on proxy tasks, with one or more of following compromised strategies: starting with fewer cells; using a smaller dataset (e.g. CIFAR-10); learning with fewer epochs.

Original: Then, to transfer to the large-scale target task, one can build a network by stacking searched cells without further exploration.
Revised: Then, to transfer to the large-scale target task, one can build a network by stacking searched cells without further exploration.

Original: However, suffering from lacking of directness and specialisation, the searched cells by proxy-based NAS methods are not guaranteed to be optimal on the target task.
Revised: However, suffering from lacking of directness and specialisation, the searched cells by proxy-based NAS methods are not guaranteed to be optimal on the target task.

Original: In contrast, proxy-less NAS methods directly learns architectures on a target task by starting with an over-parameterised network ( supernet ) that contains all possible paths, in which the redundant paths are pruned to derive the optimised architecture.
Revised: In contrast, proxy-less NAS methods directly learns architectures on a target task by starting with an over-parameterised network ( supernet ) that contains all possible paths, in which the redundant paths are pruned to derive the optimised architecture.

Original: Notwithstanding signiﬁcant better results than proxy-based approaches, proxy-less
Revised: Notwithstanding signiﬁcant better results than proxy-based approaches, proxy-less

Original: NAS methods require massive computational cost and GPU memory assumption, due to learning with the vast-size supernet .
Revised: NAS methods require massive computational cost and GPU memory assumption, due to learning with the vast-size supernet .

Original: From the optimisation point of view, existing NAS methods usually fall into three groups: reinforcement learning (RL) based methods, evolutionary algorithm (EA) based methods, and gradient differentiable (GD) methods.
Revised: From the optimisation point of view, existing NAS methods usually fall into three groups: reinforcement learning (RL) based methods, evolutionary algorithm (EA) based methods, and gradient differentiable (GD) methods.

Original: In particular, RL-based NAS methods (Zoph &
Revised: In particular, RL-based NAS methods (Zoph &

Original: Le, 2017; Pham et al., 2018; Tan et al., 2019) control the selection of architecture component in a sequential order with policy networks.
Revised: Le, 2017; Pham et al., 2018; Tan et al., 2019) control the selection of architecture component in a sequential order with policy networks.

Original: EA-based NAS methods (Real et al., 2019;
Revised: EA-based NAS methods (Real et al., 2019;

Original: Liu et al., 2018b) employ the validation accuracies to guide the evolution of a population of initialised architectures.
Revised: Liu et al., 2018b) employ the validation accuracies to guide the evolution of a population of initialised architectures.

Original: RL- an EA-based NAS methods usually suffer from low efﬁciency and high computational resource demand, due to the fundamental searching challenge in a discrete space.
Revised: RL- an EA-based NAS methods usually suffer from low efﬁciency and high computational resource demand, due to the fundamental searching challenge in a discrete space.

Original: In contrast, GD-based NAS methods (Liu et al., 2019b; Xie et al., 2019;
Revised: In contrast, GD-based NAS methods (Liu et al., 2019b; Xie et al., 2019;

Original: Luo et al., 2018) conduct searching over a continuous space by relaxation or mapping, substantially reducing the search cost to a few GPU days.
Revised: Luo et al., 2018) conduct searching over a continuous space by relaxation or mapping, substantially reducing the search cost to a few GPU days.

Original: Whilst varying in the algorithmic aspects, all these works commonly explore the feature-transformationonly search spaces without more diverse and advanced operations as we investigate here.
Revised: Whilst varying in the algorithmic aspects, all these works commonly explore the feature-transformationonly search spaces without more diverse and advanced operations as we investigate here.

Original: To show the NAS potential of the proposed richer search space with self-calibration learning operations, we take the efﬁcient proxy-based GD optimisation due to the resource constraint.
Revised: To show the NAS potential of the proposed richer search space with self-calibration learning operations, we take the efﬁcient proxy-based GD optimisation due to the resource constraint.

Original: Self-Calibration.
Revised: Self-Calibration.

Original: Self-calibration is a type of mechanism enabling a network to dynamically perform input-conditional self-adjustment, which has been studied extensively in both the computer vision (Hu et al., 2018b; Jia et al., 2016;
Revised: Self-calibration is a type of mechanism enabling a network to dynamically perform input-conditional self-adjustment, which has been studied extensively in both the computer vision (Hu et al., 2018b; Jia et al., 2016;

Original: Li et al., 2018; Park et al., 2018) and natural language processing (NLP) literature (Wu et al., 2019b; Vaswani et al., 2017).
Revised: Li et al., 2018; Park et al., 2018) and natural language processing (NLP) literature (Wu et al., 2019b; Vaswani et al., 2017).

Original: There are two typical paradigms of self-calibration: self-attention learning and dynamic convolutions , realised via an element-wise multiplication operator d and a dynamic convolution operator (cid:126) , respectively.
Revised: There are two typical paradigms of self-calibration: self-attention learning and dynamic convolutions , realised via an element-wise multiplication operator d and a dynamic convolution operator (cid:126) , respectively.

Original: Despite showing signiﬁcant efﬁcacy, self-calibration is only exploited independently after architecture hand-design (Hu et al., 2018b) or auto-search (Tan et al., 2019).
Revised: Despite showing signiﬁcant efﬁcacy, self-calibration is only exploited independently after architecture hand-design (Hu et al., 2018b) or auto-search (Tan et al., 2019).

Original: We move a step further by fully exploring the potential of self-calibration along with feature transformation in joint optimisation, bringing a richer search space for neural architecture search.
Revised: We move a step further by fully exploring the potential of self-calibration along with feature transformation in joint optimisation, bringing a richer search space for neural architecture search.

Original: Knowledge Distillation.
Revised: Knowledge Distillation.

Original: There are recent works that use knowledge distillation to help computer vision and NLP tasks.
Revised: There are recent works that use knowledge distillation to help computer vision and NLP tasks.

Original: Three types of knowledge are usually considered in distillation: features (Yim et al., 2017), attention (Zagoruyko & Komodakis, 2017), and predictions (Hinton et al., 2015).
Revised: Three types of knowledge are usually considered in distillation: features (Yim et al., 2017), attention (Zagoruyko & Komodakis, 2017), and predictions (Hinton et al., 2015).

Original: We leverage the attention distillation with a different objective – alleviating the intrinsic training-test discrepancy issue of the proxy-based NAS strategy, particularly with a more expressive search space.
Revised: We leverage the attention distillation with a different objective – alleviating the intrinsic training-test discrepancy issue of the proxy-based NAS strategy, particularly with a more expressive search space.

Original: This represents a novel exploitation of attention distillation (Zagoruyko & Komodakis, 2017).
Revised: This represents a novel exploitation of attention distillation (Zagoruyko & Komodakis, 2017).

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we start by formulating a heterogeneous search space for NAS (Sec 3.1), followed by a dedicated heterogeneous operator cell to enable composing the heterogeneous operations in a uniﬁed computing block with synergistic interaction and cooperation (Sec 3.2).
Revised: In this section, we start by formulating a heterogeneous search space for NAS (Section 3.1), followed by a dedicated heterogeneous operator cell to enable composing the heterogeneous operations in a uniﬁed computing block with synergistic interaction and cooperation (Section 3.2).

Original: To overcome the intrinsic architecture discovery challenges from more expressive search space, we further develop an attention guided search scheme (Sec. 3.3).
Revised: To overcome the intrinsic architecture discovery challenges from more expressive search space, we further develop an attention guided search scheme (Section 3.3).

Original: 3. H ETEROGENEOUS S EARCH S PACE
Revised: 3. H ETEROGENEOUS S EARCH S PACE

Original: image
Revised: b denotes matrix multiplication.

Original: To enrich the NAS search space so that more advanced network architectures can be discovered, we introduce a heterogeneous search space A that considers three different types of representation learning capabilities: (1)
Revised: To enrich the NAS search space so that more advanced network architectures can be discovered, we introduce a heterogeneous search space A that considers three different types of representation learning capabilities: (1)

Original: Feature transformations ; (2) Attention learning ; and (3)
Revised: Feature transformations ; (2) Attention learning ; and (3)

Original: Dynamic convolutions .
Revised: Dynamic convolutions .

Original: More concretely, we form three sets of primitive computing operations that produce features , attentions and dynamic weights , respectively.
Revised: More concretely, we form three sets of primitive computing operations that produce features , attentions and dynamic weights , respectively.

Original: This novel search space generalises the conventional counterpart which is limited to the ﬁrst type of operations (Liu et al., 2019b; Pham et al., 2018), and incorporates the self-calibration learning capabilities (i.e. the second and third types) in NAS.
Revised: This novel search space generalises the conventional counterpart which is limited to the ﬁrst type of operations (Liu et al., 2019b; Pham et al., 2018), and incorporates the self-calibration learning capabilities (i.e. the second and third types) in NAS.

Original: Importantly, while the search space changes, the generic search strategies still apply therefore being largely open for collaborating with existing NAS methods.
Revised: Importantly, while the search space changes, the generic search strategies still apply therefore being largely open for collaborating with existing NAS methods.

Original: For instance, in the proxy-based NAS strategy we may ﬁrst search for a computing cell with heterogeneous operations as the building block and then form the ﬁnal network architecture by sequentially stacking multiple such cells layer-by-layer.
Revised: For instance, in the proxy-based NAS strategy we may ﬁrst search for a computing cell with heterogeneous operations as the building block and then form the ﬁnal network architecture by sequentially stacking multiple such cells layer-by-layer.

Original: Next, let us describe the heterogeneous primitive operation set O which consists of the following three disjoint subsets: O f , O a and O d , along with their aggregation or application operators.
Revised: Next, let us describe the heterogeneous primitive operation set O which consists of the following three disjoint subsets: O f , O a and O d , along with their aggregation or application operators.

Original: Feature Transformation Operations O f .
Revised: Feature Transformation Operations O f .

Original: We adopt the feature transformation/learning operation set O f same as in Liu et al.
Revised: We adopt the feature transformation/learning operation set O f same as in Liu et al.

Original: (2019b; 2018a), including the following 7 operations: 3 ˆ 3 and 5 ˆseparable convolutions, 3 ˆ 3 and 5 ˆ 5 dilated separable convolutions, 3 ˆ 3 average pooling, 3 ˆ 3 max pooling, and identity.
Revised: (2019b; 2018a), including the following 7 operations: 3 ˆ 3 and 5 ˆseparable convolutions, 3 ˆ 3 and 5 ˆ 5 dilated separable convolutions, 3 ˆ 3 average pooling, 3 ˆ 3 max pooling, and identity.

Original: Every operation o f P
Revised: Every operation o f P

Original: O f takes as input a feature tensor and outputs another feature tensor, i.e. homogeneous feature-tensor-to-feature-tensor transformation.
Revised: O f takes as input a feature tensor and outputs another feature tensor, i.e. homogeneous feature-tensor-to-feature-tensor transformation.

Original: For multiple feature tensor aggregation, the element-wise addition operator ‘ is typically used.
Revised: For multiple feature tensor aggregation, the element-wise addition operator ‘ is typically used.

Original: Attention Learning Operations O a .
Revised: Attention Learning Operations O a .

Original: Inspired by recent exquisite designs of attention learning modules (Hu et al., 2018b; Li et al., 2018; Park et al., 2018), we form the O a by considering two types of attention learning prototypes: spatial-wise and channel-wise attentions.
Revised: Inspired by recent exquisite designs of attention learning modules (Hu et al., 2018b; Li et al., 2018; Park et al., 2018), we form the O a by considering two types of attention learning prototypes: spatial-wise and channel-wise attentions.

Original: Speciﬁcally, a spatial-wise attention operation learns a saliency map for an input feature tensor in order to calibrate the importance of different spatial positions.
Revised: Speciﬁcally, a spatial-wise attention operation learns a saliency map for an input feature tensor in order to calibrate the importance of different spatial positions.

Original: In contrast, a channel-wise attention operation produces a vector of scaling factors from the aggregated global context of an input tensor for adaptively calibrating the channel dependency.
Revised: In contrast, a channel-wise attention operation produces a vector of scaling factors from the aggregated global context of an input tensor for adaptively calibrating the channel dependency.

Original: To enforce attentive calibration on feature tensor, the elementwise multiplication operator d is a typical choice for both spatial-wise and channel-wise attentions.
Revised: To enforce attentive calibration on feature tensor, the elementwise multiplication operator d is a typical choice for both spatial-wise and channel-wise attentions.

Original: Dynamic Convolution Operations O d .
Revised: Dynamic Convolution Operations O d .

Original: Dynamic convolutions, designed for the sake of selfadaptation, generate dynamic kernel weights in accordance with the input feature tensor.
Revised: Dynamic convolutions, designed for the sake of selfadaptation, generate dynamic kernel weights in accordance with the input feature tensor.

Original: It is often in form of depth-wise separable convolution as the feature transformation operation.
Revised: It is often in form of depth-wise separable convolution as the feature transformation operation.

Original: Tailored for either NLP or dense prediction tasks, existing dynamic convolution designs (Wu et al., 2019b; Jia et al., 2016) are not suitable for image classiﬁcation (our focus) with different problem nature.
Revised: Tailored for either NLP or dense prediction tasks, existing dynamic convolution designs (Wu et al., 2019b; Jia et al., 2016) are not suitable for image classiﬁcation (our focus) with different problem nature.

Original: It hence needs to be reformulated in order to be effective for learning discriminative image representations.
Revised: It hence needs to be reformulated in order to be effective for learning discriminative image representations.

Original: We consider two design principles: (i) structurally lightweight whilst (ii) functionally strong with great modelling capability.
Revised: We consider two design principles: (i) structurally lightweight whilst (ii) functionally strong with great modelling capability.

Original: To that end, we present an exquisite dynamic convolution structure specialised for cost-effective image classiﬁcation, as shown in Fig.
Revised: To that end, we present an exquisite dynamic convolution structure specialised for cost-effective image classiﬁcation, as shown in Fig.

Original: Concretely, it consists of three compact modules composed in an exquisite cooperation: (a) a bottleneck module, to compress an input feature tensor by a ratio of r ;
Revised: Concretely, it consists of three compact modules composed in an exquisite cooperation: (a) a bottleneck module, to compress an input feature tensor by a ratio of r ;

Original: (b) a kernel transform module,
Revised: (b) a kernel transform module,

Original: to
Revised: to

Original: learn
Revised: learn

Original: latent
Revised: latent

Original: rep-
Revised: rep-

Original: resentations
Revised: resentations

Original: with a
Revised: with a

Original: kernel dimension
Revised: kernel dimension

Original: of k ˆ k ; (c)
Revised: of k ˆ k ; (c)

Original: a kernel decode module, to read out the dynamic kernel weights with the channel dimension same as the input feature tensor.
Revised: a kernel decode module, to read out the dynamic kernel weights with the channel dimension same as the input feature tensor.

Original: This design is motivated, in part, by the long-range dependency modeling (Wang et al., 2018; Cao et al., 2019) and global context aggregation (Hu et al., 2018b;a), elegantly integrating their merits via a uniﬁed formulation.
Revised: This design is motivated, in part, by the long-range dependency modeling (Wang et al., 2018; Cao et al., 2019) and global context aggregation (Hu et al., 2018b;a), elegantly integrating their merits via a uniﬁed formulation.

Original: For the output of dynamic convolutions, we consider two common kernel sizes: 3 ˆ 3 and 5 ˆ 5 .
Revised: For the output of dynamic convolutions, we consider two common kernel sizes: 3 ˆ 3 and 5 ˆ 5 .

Original: In a depth-wise manner, we apply a standard or dilated convolution operator (cid:126) to transform the input feature tensor.
Revised: In a depth-wise manner, we apply a standard or dilated convolution operator (cid:126) to transform the input feature tensor.

Original: It is noteworthy to point out that, this type of convolutional kernel is speciﬁc for each feature tensor of a particular image sample (i.e. dynamic), rather than learned from a training dataset and ﬁxed for all the input samples (i.e. static) as the conventional convolutional operations in the feature transformation set.
Revised: It is noteworthy to point out that, this type of convolutional kernel is speciﬁc for each feature tensor of a particular image sample (i.e. dynamic), rather than learned from a training dataset and ﬁxed for all the input samples (i.e. static) as the conventional convolutional operations in the feature transformation set.

Original: Detailed implementations of self-calibration operations are presented in Appendix A.1.
Revised: Detailed implementations of self-calibration operations are presented in Appendix A.1.

Original: 3. H ETEROGENEOUS O PERATOR C ELL
Revised: 3. H ETEROGENEOUS O PERATOR C ELL

Original: Due to different natures of heterogeneous computing capabilities, a uniﬁcation structure is needed for composing the primitive operations
Revised: Due to different natures of heterogeneous computing capabilities, a uniﬁcation structure is needed for composing the primitive operations

Original: O “ O f Y O
Revised: O “ O f Y O

Original: a Y O d and aggregation/application operators
Revised: a Y O d and aggregation/application operators

Original: C “ t‘ , d , (cid:126) u in such a way that their representation learning potentials can be well mined.
Revised: C “ t‘ , d , (cid:126) u in such a way that their representation learning potentials can be well mined.

Original: To that end, we formulate a heterogeneous operator cell , a directed acyclic graph (DAG) G “ p V , E q , joining conventional feature transformations and proposed self-calibration operations synergistically.
Revised: To that end, we formulate a heterogeneous operator cell , a directed acyclic graph (DAG) G “ p V , E q , joining conventional feature transformations and proposed self-calibration operations synergistically.

Original: Formally, a heterogeneous operator cell consists of N ordered feature (tensor) nodes V “ t F k | , 1 ă“ k ă“ N u .
Revised: Formally, a heterogeneous operator cell consists of N ordered feature (tensor) nodes V “ t F k | , 1 ă“ k ă“ N u .

Original: Following (Zoph et al., 2018), F 1 and F 2 are the outputs from the previous cells regarded as two input and the N -th node F N is the cell output node formed as the concatenation of all the inner nodes, i.e.
Revised: Following (Zoph et al., 2018), F 1 and F 2 are the outputs from the previous cells regarded as two input and the N -th node F N is the cell output node formed as the concatenation of all the inner nodes, i.e.

Original: F N “ concat pt F k u N ´ 1 k “ 3 q .
Revised: F N “ concat pt F k u N ´ 1 k “ 3 q .

Original: The edge e i Ñ k “p i, k qP E speciﬁes the connection between the i -th and k -th nodes (the information ﬂow i Ñ k ), associated with a speciﬁc operation
Revised: The edge e i Ñ k “p i, k qP E speciﬁes the connection between the i -th and k -th nodes (the information ﬂow i Ñ k ), associated with a speciﬁc operation

Original: o i Ñ k selected from the heterogeneous primitive operation set O .
Revised: o i Ñ k selected from the heterogeneous primitive operation set O .

Original: The key is to design a computing block for the inner nodes with heterogeneous computations.
Revised: The key is to design a computing block for the inner nodes with heterogeneous computations.

Original: Heterogeneous Residual Block.
Revised: Heterogeneous Residual Block.

Original: It is non-trivial to design a heterogeneous computing block due to being not straightforward feature-tensor-to-feature-tensor transformation as in the conventional homogeneous operation.
Revised: It is non-trivial to design a heterogeneous computing block due to being not straightforward feature-tensor-to-feature-tensor transformation as in the conventional homogeneous operation.

Original: It involves self-calibrating the input feature tensor itself in addition to the homogeneous feature transformation.
Revised: It involves self-calibrating the input feature tensor itself in addition to the homogeneous feature transformation.

Original: To facilitate adding the extra capacity, we formulate a heterogeneous residual block (see Fig. 2) characterised by a surrogate node k 1 in the computing block associated with each inner node k , for enabling richer feature tensor manipulations.
Revised: To facilitate adding the extra capacity, we formulate a heterogeneous residual block (see Fig. 2) characterised by a surrogate node k 1 in the computing block associated with each inner node k , for enabling richer feature tensor manipulations.

Original: This is in a residual learning spirit (He et al., 2016), allowing to conduct self-calibration reliably.
Revised: This is in a residual learning spirit (He et al., 2016), allowing to conduct self-calibration reliably.

Original: Moreover, we design a two-tier computing hierarchy: the ﬁrst tier for individual computation per input feature tensor to capture the speciﬁcity, and the second tier for collective computation on the set of all the input feature tensors as a whole to capture the intrinsic structural relations between feature tensors and the global input properties.
Revised: Moreover, we design a two-tier computing hierarchy: the ﬁrst tier for individual computation per input feature tensor to capture the speciﬁcity, and the second tier for collective computation on the set of all the input feature tensors as a whole to capture the intrinsic structural relations between feature tensors and the global input properties.

Original: The two tiers are connected by the surrogate node k 1 .
Revised: The two tiers are connected by the surrogate node k 1 .

Original: Formally, we take as input all the previous nodes t F i | , i ă k u , process them separately with heterogeneous operations, and combine the processed results by summation (Fig. 2 (a)):
Revised: Formally, we take as input all the previous nodes t F i | , i ă k u , process them separately with heterogeneous operations, and combine the processed results by summation (Fig. 2 (a)):

Original: [Equation]
Revised: [Equation]

Original: where F k 1 , A k 1 , and D k 1 are the three types of intermediate outputted tensors, i.e. features, attentions, and dynamic weights, respectively.
Revised: where F k 1 , A k 1 , and D k 1 are the three types of intermediate outputted tensors, i.e. features, attentions, and dynamic weights, respectively.

Original: These are subsequently aggregated into an intermediate calibrated tensor , i.e. the surrogate node F k 1 , using element-wise addition in-between on feature self-calibration and transformation as:
Revised: These are subsequently aggregated into an intermediate calibrated tensor , i.e. the surrogate node F k 1 , using element-wise addition in-between on feature self-calibration and transformation as:

Original: [Equation]
Revised: [Equation]

Original: Next, F k 1 is used as the input for the second-tier set-level collective computation (Fig.
Revised: Next, F k 1 is used as the input for the second-tier set-level collective computation (Fig.

Original: 2 (b)).
Revised: 2 (b)).

Original: Likewise, we consider the same three types of operations:
Revised: Likewise, we consider the same three types of operations:

Original: [Equation]
Revised: [Equation]

Original: and form the inner node F k via further feature self-calibration and transformation as:
Revised: and form the inner node F k via further feature self-calibration and transformation as:

Original: [Equation]
Revised: [Equation]

Original: In doing so, our heterogeneous residual block presents a two-tier combinatorial operations structure for each inner node, resulting in a more expressive search space (see Sec. 4.2).
Revised: In doing so, our heterogeneous residual block presents a two-tier combinatorial operations structure for each inner node, resulting in a more expressive search space (see Section 4.2).

Original: 3. A TTENTION G UIDED S EARCH O PTIMISATION IN A H ETEROGENEOUS S EARCH S PACE
Revised: 3. A TTENTION G UIDED S EARCH O PTIMISATION IN A H ETEROGENEOUS S EARCH S PACE

Original: To showcase the effectiveness of the proposed heterogeneous search space and operator cell, we adopt the proxy-based NAS strategy, due to the computing resource constraints and the enormous search space.
Revised: To showcase the effectiveness of the proposed heterogeneous search space and operator cell, we adopt the proxy-based NAS strategy, due to the computing resource constraints and the enormous search space.

Original: This search is done by constructing a small proxy network parametrised by Θ .
Revised: This search is done by constructing a small proxy network parametrised by Θ .

Original: Attention Guided Search.
Revised: Attention Guided Search.

Original: Compared with proxyless search strategy, proxy-based NAS is more efﬁcient but relatively less optimal due to not directly optimising the ﬁnal network architecture.
Revised: Compared with proxyless search strategy, proxy-based NAS is more efﬁcient but relatively less optimal due to not directly optimising the ﬁnal network architecture.

Original: This training-test discrepancy problem can be worsened when the search space provides more ﬂexibility and combinatorial capability, such as the proposed space.
Revised: This training-test discrepancy problem can be worsened when the search space provides more ﬂexibility and combinatorial capability, such as the proposed space.

Original: To solve this obstacle, we propose attention guided search, which optimises the proxy network in a knowledge distillation manner by injecting an external guidance from a pre-trained teacher network into the NAS process.
Revised: To solve this obstacle, we propose attention guided search, which optimises the proxy network in a knowledge distillation manner by injecting an external guidance from a pre-trained teacher network into the NAS process.

Original: and S i ( i P t 0 , 1 , 2 u ) denote thei -th stage of the teacher and proxy (student) networks.
Revised: ) denote the of the teacher and proxy (student) networks.

Original: Speciﬁcally, we leverage the attention transfer idea (Zagoruyko & Komodakis, 2017) that encourages a student (the proxy network in our case) to hierarchically imitate a teacher’s hidden attention knowledge.
Revised: Speciﬁcally, we leverage the attention transfer idea (Zagoruyko & Komodakis, 2017) that encourages a student (the proxy network in our case) to hierarchically imitate a teacher’s hidden attention knowledge.

Original: Intuitively, this may beneﬁt the search for self-calibration learning.
Revised: Intuitively, this may beneﬁt the search for self-calibration learning.

Original: Formally, let us denote a feature tensor at the j -th stage of the teacher and student network as
Revised: Formally, let us denote a feature tensor at the j -th stage of the teacher and student network as

Original: F jT and F jS , separately.
Revised: F jT and F jS , separately.

Original: Attention transfer is realised by imposing an alignment loss function across the two networks as:
Revised: Attention transfer is realised by imposing an alignment loss function across the two networks as:

Original: [Equation]
Revised: [Equation]

Original: where x jS { T is the spatial-wise accumulated feature vector.
Revised: where x jS { T is the spatial-wise accumulated feature vector.

Original: An overview of attention guided search is depicted in Fig.

Original: An overview of attention guided search is depicted in Fig.
Revised: An overview of attention guided search is depicted in Fig.

Original: Optimisation.
Revised: Optimisation.

Original: For NAS optimisation, we adopt the DARTS method (Liu et al., 2019b).
Revised: For NAS optimisation, we adopt the DARTS method (Liu et al., 2019b).

Original: In our context, we conduct the continuous relaxation over all the possible heterogeneous operations O for making a continuous search space:
Revised: In our context, we conduct the continuous relaxation over all the possible heterogeneous operations O for making a continuous search space:

Original: [Equation]
Revised: [Equation]

Original: where an architecture vector a i Ñ j o
Revised: where an architecture vector a i Ñ j o

Original: P R | O | is used for each possible connection i Ñ j .
Revised: P R | O | is used for each possible connection i Ñ j .

Original: We summarise the architecture vector of all the connections as a matrix
Revised: We summarise the architecture vector of all the connections as a matrix

Original: A “ “ [Equation] P [Equation]
Revised: A “ “ a 1 , ¨ ¨ ¨ , a | E | ‰ P R | E |ˆ| O | .

Original: | [Equation] |ˆ| O | . With this relaxation, we can jointly optimise the architecture parameters A and the network weights Θ in a fully gradient differentiable manner.
Revised: With this relaxation, we can jointly optimise the architecture parameters A and the network weights Θ in a fully gradient differentiable manner.

Original: Equipped with the proposed attention guidance search, the search objective function is ﬁnally formulated as the following bilevel optimisation process:
Revised: Equipped with the proposed attention guidance search, the search objective function is ﬁnally formulated as the following bilevel optimisation process:

Original: [Equation]
Revised: [Equation]

Original: where λ denotes the weighting hyper-parameter.
Revised: where λ denotes the weighting hyper-parameter.

Original: For the ﬁrst level Eq.
Revised: For the ﬁrst level Eq.

Original: (8), we learn the optimal parameters Θ ˚ for a given architecture A w.r.t a training objective L train and the attention alignment loss L AT .
Revised: (8), we learn the optimal parameters Θ ˚ for a given architecture A w.r.t a training objective L train and the attention alignment loss L AT .

Original: The second level Eq.
Revised: The second level Eq.

Original: (9) then explores the optimal architecture A ˚ over the heterogeneous search space
Revised: (9) then explores the optimal architecture A ˚ over the heterogeneous search space

Original: A w.r.t a validation objective L val and L AT .
Revised: A w.r.t a validation objective L val and L AT .

Original: For image classiﬁcation, L train and L val usually take the cross-entropy loss function.
Revised: For image classiﬁcation, L train and L val usually take the cross-entropy loss function.

Original: Search Outcome.
Revised: Search Outcome.

Original: Once the above alternated optimisation is done, we derive an amenable cell architecture with heterogeneous operators.
Revised: Once the above alternated optimisation is done, we derive an amenable cell architecture with heterogeneous operators.

Original: In practice, for each heterogeneous computing block we retain the top-2 strongest incoming operations with at least one feature transformation operation for the ﬁrst-tier (Fig. 2(a)), and the top-1 strongest operation for the second-tier (Fig. 2(b)).
Revised: In practice, for each heterogeneous computing block we retain the top-2 strongest incoming operations with at least one feature transformation operation for the ﬁrst-tier (Fig. 2(a)), and the top-1 strongest operation for the second-tier (Fig. 2(b)).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated the proposed NOS method on image classiﬁcation using three common datasets.
Revised: We evaluated the proposed NOS method on image classiﬁcation using three common datasets.

Original: CIFAR10/100:
Revised: CIFAR10/100:

Original: Both CIFAR10 and CIFAR100 have 50K/10K train/test RGB images of size 32 ˆ 32 ˆ 3 , categorised into 10 and 100 classes, respectively (Krizhevsky et al., 2009).
Revised: Both CIFAR10 and CIFAR100 have 50K/10K train/test RGB images of size 32 ˆ 32 ˆ 3 , categorised into 10 and 100 classes, respectively (Krizhevsky et al., 2009).

Original: ImageNet: We use the
Revised: ImageNet: We use the

Original: ILSVRC2012 version for large-scale image classiﬁcation evaluation, containing 1.28M training images, 50K validation samples, and 1K classes (Russakovsky et al., 2015).
Revised: ILSVRC2012 version for large-scale image classiﬁcation evaluation, containing 1.28M training images, 50K validation samples, and 1K classes (Russakovsky et al., 2015).

Original: We ﬁrst conduct preliminary experiments on CIFAR10/100 to select the heterogeneous primitive operations O .
Revised: We ﬁrst conduct preliminary experiments on CIFAR10/100 to select the heterogeneous primitive operations O .

Original: To test the efﬁcacy and transferability of NOS, we search the cell structures on CIFAR only, and compare the performance with existing methodson CIFAR10/100 and ImageNet.
Revised: To test the efﬁcacy and transferability of NOS, we search the cell structures on CIFAR only, and compare the performance with existing methods

Original: [Table]
Revised: [Table]

Original: 4. P RELIMINARY S TUDY OF F EATURE S ELF -C ALIBRATION O PERATIONS
Revised: 4. P RELIMINARY S TUDY OF F EATURE S ELF -C ALIBRATION O PERATIONS

Original: We conducted a controlled experiment to test the introduced self-calibration operations on CIFAR10 and CIFAR-100.
Revised: We conducted a controlled experiment to test the introduced self-calibration operations on CIFAR10 and CIFAR-100.

Original: Speciﬁcally, for the proposed dynamic convolutions , we considered both normal and dilated convolutions and two kernel sizes ( 3 ˆ 3 and 5 ˆ 5 ).
Revised: Speciﬁcally, for the proposed dynamic convolutions , we considered both normal and dilated convolutions and two kernel sizes ( 3 ˆ 3 and 5 ˆ 5 ).

Original: We adopted the channel-wise and spatial-wise attention learning .
Revised: We adopted the channel-wise and spatial-wise attention learning .

Original: For the baseline model, we used ResNet-18 (He et al., 2016) with 4 stages in the backbone.
Revised: For the baseline model, we used ResNet-18 (He et al., 2016) with 4 stages in the backbone.

Original: To build a model with self-calibration, we added each self-calibration operation at the stages 1 , 2 , 3 of ResNet-18, respectively.
Revised: To build a model with self-calibration, we added each self-calibration operation at the stages 1 , 2 , 3 of ResNet-18, respectively.

Original: For fair comparison, we trained each model in the same setting (see Appendix A.2.1).
Revised: For fair comparison, we trained each model in the same setting (see Appendix A.2.1).

Original: In Table 1, we summarised the model parameters and FLOPs in addition to the test set performance (error rates).
Revised: In Table 1, we summarised the model parameters and FLOPs in addition to the test set performance (error rates).

Original: We observed that: (1) Both attention operations and our normal dynamic convolutions outperform the baseline consistently; (2) Adding dilated dynamic convolutions causes performance drop in most cases.
Revised: We observed that: (1) Both attention operations and our normal dynamic convolutions outperform the baseline consistently; (2) Adding dilated dynamic convolutions causes performance drop in most cases.

Original: We hence exclude it from the candidate set; (3) Very marginal FLOPs and parameters increase from these self-calibration operations over the baseline, suggesting their high cost-effectiveness.
Revised: We hence exclude it from the candidate set; (3) Very marginal FLOPs and parameters increase from these self-calibration operations over the baseline, suggesting their high cost-effectiveness.

Original: C ELL S EARCH
Revised: C ELL S EARCH

Original: Search Space.
Revised: Search Space.

Original: As found out above, the heterogeneous primitive operation set O contains 11 operations in total: | O f | “ 7 feature transformation operations, | O a | “ 2 attention learning operations, | O d | “ 2 dynamic convolutions, respectively.
Revised: As found out above, the heterogeneous primitive operation set O contains 11 operations in total: | O f | “ 7 feature transformation operations, | O a | “ 2 attention learning operations, | O d | “ 2 dynamic convolutions, respectively.

Original: We constructed the proposed heterogeneous operator cell ( G “ p V , E q ) with | V | “ 7 nodes (2 input nodes, 4 inner nodes and 1 output node).
Revised: We constructed the proposed heterogeneous operator cell ( G “ p V , E q ) with | V | “ 7 nodes (2 input nodes, 4 inner nodes and 1 output node).

Original: So, allheterogeneous residual blocks contain | E | “ 18 edges in total ( 14 ﬁrst-tier connections and 4 secondtier connections).
Revised: So, allheterogeneous residual blocks contain | E | “ 18 edges in total ( 14 ﬁrst-tier connections and 4 secondtier connections).

Original: To derive the ﬁnal cell architecture, we kept 2 ﬁrst-tier connections and 1 secondtier connection for each block.
Revised: To derive the ﬁnal cell architecture, we kept 2 ﬁrst-tier connections and 1 secondtier connection for each block.

Original: As a result, there is a total number of ś 4 n “ 1 p n ` 1 q nˆ 11 3 « 10possible choices, 5 orders of magnitude larger than the conventional size of ś 4 n “ 1 p n ` 1 q nˆ 7 2 « 10as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019).
Revised: As a result, there is a total number of ś 4 n “ 1 p n ` 1 q nˆ 11 3 « 10possible choices, 5 orders of magnitude larger than the conventional size of ś 4 n “ 1 p n ` 1 q nˆ 7 2 « 10as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019).

Original: Training.
Revised: Training.

Original: Following the setup of existing methods (Real et al., 2019;
Revised: Following the setup of existing methods (Real et al., 2019;

Original: Liu et al., 2019b; 2018a;
Revised: Liu et al., 2019b; 2018a;

Original: Akimoto et al., 2019), we searched the convolutional architectures on CIFAR10.
Revised: Akimoto et al., 2019), we searched the convolutional architectures on CIFAR10.

Original: We constructed a
Revised: We constructed a

Original: [Table]
Revised: [Table]

Original: (Zhou et al
Revised: (Zhou et al

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: small proxy network with 8 heterogeneous operator cells, and two reduction cells at 1/3 and 2/3 of the total network depth for feature shape reduction.
Revised: small proxy network with 8 heterogeneous operator cells, and two reduction cells at 1/3 and 2/3 of the total network depth for feature shape reduction.

Original: We used 25K images split from the training set for validation.
Revised: We used 25K images split from the training set for validation.

Original: We randomly initialised the architecture parameters A P R 18 ˆ 11 in the normal distribution.
Revised: We randomly initialised the architecture parameters A P R 18 ˆ 11 in the normal distribution.

Original: We used a pre-trained PyramidNet-110 (bottleneck, α “ 84 ) (Han et al., 2017) as the teacher model.
Revised: We used a pre-trained PyramidNet-110 (bottleneck, α “ 84 ) (Han et al., 2017) as the teacher model.

Original: We set the weight λ “ 10 3 for attention guidance loss L AT .
Revised: We set the weight λ “ 10 3 for attention guidance loss L AT .

Original: After 25 epochs of training on the proxy network, we derived the ﬁnal heterogeneous operator cells from the architecture matrix
Revised: After 25 epochs of training on the proxy network, we derived the ﬁnal heterogeneous operator cells from the architecture matrix

Original: A .
Revised: A .

Original: See Appendix A.2.1 for more conﬁgurations for training the proxy and teacher networks.
Revised: See Appendix A.2.1 for more conﬁgurations for training the proxy and teacher networks.

Original: The search on CIFAR10 took only 8.4 hours using a single NVIDIA Tesla V100 GPU.
Revised: The search on CIFAR10 took only 8.4 hours using a single NVIDIA Tesla V100 GPU.

Original: The searched heterogeneous operator cells by NOS is shown in Fig.
Revised: The searched heterogeneous operator cells by NOS is shown in Fig.

Original: 4, in which the self-calibration operators d and (cid:126) appear in both ﬁrst-tier and second-tier.
Revised: 4, in which the self-calibration operators d and (cid:126) appear in both ﬁrst-tier and second-tier.

Original: For example, there are two attention operations in ﬁrst-tier and two dynamic convolutions in second-tier in the normal cell.
Revised: For example, there are two attention operations in ﬁrst-tier and two dynamic convolutions in second-tier in the normal cell.

Original: 4. A RCHITECTURE E VALUATION
Revised: 4. A RCHITECTURE E VALUATION

Original: CIFAR.
Revised: CIFAR.

Original: To measure the ﬁnal image classiﬁcation performance of the searched heterogeneous operator cells on CIFAR10 and CIFAR100, we created an evaluation network with 20 cells, 36 initial channels, and an auxiliary tower with loss weight 0.4.
Revised: To measure the ﬁnal image classiﬁcation performance of the searched heterogeneous operator cells on CIFAR10 and CIFAR100, we created an evaluation network with 20 cells, 36 initial channels, and an auxiliary tower with loss weight 0.4.

Original: See Appendix A.2.1 for more conﬁgurations for training the evaluation network.
Revised: See Appendix A.2.1 for more conﬁgurations for training the evaluation network.

Original: Due to high variance of results on CIFAR, we conductedindependent runs and reported both the best and average results.
Revised: Due to high variance of results on CIFAR, we conductedindependent runs and reported both the best and average results.

Original: We summarised the results of NOS and the state-of-the-art methods in Table 2.
Revised: We summarised the results of NOS and the state-of-the-art methods in Table 2.

Original: The comparisons show that: (1) NOS achieves a very competitive result (second best) on CIFAR10, whilst enjoying the smallest model parameters (only 2.6M).
Revised: The comparisons show that: (1) NOS achieves a very competitive result (second best) on CIFAR10, whilst enjoying the smallest model parameters (only 2.6M).

Original: Comparing with the best performer ProxylessNAS at the size of 5.7M (searched with a supernet )
Revised: Comparing with the best performer ProxylessNAS at the size of 5.7M (searched with a supernet )

Original: (Cai et al., 2019), it shows the signiﬁcant cost-effectiveness and compactness advantages of our method.
Revised: (Cai et al., 2019), it shows the signiﬁcant cost-effectiveness and compactness advantages of our method.

Original: (2) Despite a signiﬁcantly larger search space ( 10 14 vs 10 9 in (Liu et al., 2019b;
Revised: (2) Despite a signiﬁcantly larger search space ( 10 14 vs 10 9 in (Liu et al., 2019b;

Original: Akimoto et al., 2019; Xie et al., 2019; Dong & Yang, 2019; Akimoto et al., 2019)), NOS shows high cost-effectiveness in computing cost (only 0.35 GPU day).
Revised: Akimoto et al., 2019; Xie et al., 2019; Dong & Yang, 2019; Akimoto et al., 2019)), NOS shows high cost-effectiveness in computing cost (only 0.35 GPU day).

Original: (3) NOS achieves the best result on CIFAR100 by directly transferring the CIFAR10 searched network, signiﬁcantly outperforming
Revised: (3) NOS achieves the best result on CIFAR100 by directly transferring the CIFAR10 searched network, signiﬁcantly outperforming

Original: DARTS (Liu et al., 2019b) and GDAS (Dong & Yang, 2019).
Revised: DARTS (Liu et al., 2019b) and GDAS (Dong & Yang, 2019).

Original: This challenging cross-dataset test indicates a superior transferability of the network searched by NOS.
Revised: This challenging cross-dataset test indicates a superior transferability of the network searched by NOS.

Original: ImageNet.
Revised: ImageNet.

Original: To evaluate the transferability of architecture discovered by NOS on large scale ImageNet, we used the mobile setting same as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019), where the number of multiply-add operations is restricted to be less than 600M at the input
Revised: To evaluate the transferability of architecture discovered by NOS on large scale ImageNet, we used the mobile setting same as in (Liu et al., 2019b; Dong & Yang, 2019; Xie et al., 2019), where the number of multiply-add operations is restricted to be less than 600M at the input

Original: [Table]
Revised: [Table]

Original: RL
Revised: RL

Original: ProxylessNAS (GPU) Cai et al.
Revised: ProxylessNAS (GPU) (Cai et al., 2019) 24.

Original: GD
Revised: GD

Original: FBNet-C Wu et al. (2019a)
Revised: FBNet-C (Wu et al., 2019a) 25.

Original: GD
Revised: GD

Original: GHN Zhang et al.
Revised: GHN (Zhang et al., 2019) 27.

Original: GD
Revised: GD

Original: DARTS Liu et al. (2019b) 26.
Revised: DARTS (Liu et al., 2019b) 26.

Original: GD
Revised: GD

Original: SNAS Xie et al.
Revised: SNAS (Xie et al., 2019) 27.

Original: GD
Revised: GD

Original: GDAS Dong & Yang (2019) 26.
Revised: GDAS (Dong & Yang, 2019) 26.

Original: GD
Revised: GD

Original: BayesNAS (0.005) Zhou et al. [Table] [Table]
Revised: BayesNAS (0.005) (Zhou et al., 2019) 26.

Original: GD
Revised: GD

Original: NOS (searched on CIFAR10) 
Revised: NOS (searched on CIFAR10) 25.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: GD
Revised: GD

Original: [Table]
Revised: [Table]

Original: size of 224 ˆ 224 .
Revised: size of 224 ˆ 224 .

Original: Speciﬁcally, we constructed an evaluation network with 14 cells and 48 initial channels.
Revised: Speciﬁcally, we constructed an evaluation network with 14 cells and 48 initial channels.

Original: An auxiliary tower with loss weight 0.4 was also applied.
Revised: An auxiliary tower with loss weight 0.4 was also applied.

Original: See Appendix A.2.2 for more training details.
Revised: See Appendix A.2.2 for more training details.

Original: Table 3 shows the ImageNet results in the mobile setting.
Revised: Table 3 shows the ImageNet results in the mobile setting.

Original: Notably, the cell architectures found by NOS on CIFAR10 can achieve highly competitive performance with signiﬁcantly less computational cost (0.35 day on 1 GPU vs 4.5 days using 64 TPUv2 devices required by MnasNetA3 (Tan et al., 2019)).
Revised: Notably, the cell architectures found by NOS on CIFAR10 can achieve highly competitive performance with signiﬁcantly less computational cost (0.35 day on 1 GPU vs 4.5 days using 64 TPUv2 devices required by MnasNetA3 (Tan et al., 2019)).

Original: Unlike MnasNet-A3 (Tan et al., 2019) and ProxylessNAS (Cai et al., 2019) searching the network on ImageNet directly (resource-intensive), the network searched by NOS on
Revised: Unlike MnasNet-A3 (Tan et al., 2019) and ProxylessNAS (Cai et al., 2019) searching the network on ImageNet directly (resource-intensive), the network searched by NOS on

Original: CIFAR10 can be successfully transferred.
Revised: CIFAR10 can be successfully transferred.

Original: Also, compared to other state-of-the-art gradient based proxy-based NAS (GHN, DARTS, SNAS, GDAS and BayesNAS), NOS discovers a cell structure that performs better with higher efﬁciency (only 440M FLOPs).
Revised: Also, compared to other state-of-the-art gradient based proxy-based NAS (GHN, DARTS, SNAS, GDAS and BayesNAS), NOS discovers a cell structure that performs better with higher efﬁciency (only 440M FLOPs).

Original: 4. F URTHER A NALYSIS
Revised: 4. F URTHER A NALYSIS

Original: [Table]
Revised: [Table]

Original: We evaluated attention guided search (AGS) on CIFAR10/100 by comparing a NOS variant without attention transfer loss.
Revised: We evaluated attention guided search (AGS) on CIFAR10/100 by comparing a NOS variant without attention transfer loss.

Original: The same training setting was used (Appendix A.2.1).
Revised: The same training setting was used (Appendix A.2.1) 1 .

Original: We used a pre-trained
Revised: We used a pre-trained

Original: PyramidNet-110 as teacher.
Revised: PyramidNet-110 as teacher.

Original: Table 4 shows that learning with attention guidance can signiﬁcantly beneﬁt the NOS search process.
Revised: Table 4 shows that learning with attention guidance can signiﬁcantly beneﬁt the NOS search process.

Original: We further showed the training curves in Fig.
Revised: We further showed the training curves in Fig.

Original: 5 and observed that
Revised: 5 and observed that

Original: AGS clearly improves the train/val accuracies.
Revised: AGS clearly improves the train/val accuracies.

Original: This suggests that AGS is effective to alleviate the architecture training-test discrepancy issue involved in the proxy-based NAS.
Revised: This suggests that AGS is effective to alleviate the architecture training-test discrepancy issue involved in the proxy-based NAS.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented Neural Operator Search (NOS), featured by a heterogeneous search space for neural architecture search (NAS).
Revised: We presented Neural Operator Search (NOS), featured by a heterogeneous search space for neural architecture search (NAS).

Original: This search space expansion enables NAS to discover more expressive and previously undiscovered architectures, signiﬁcantly expanding the search horizon and enriching the possible search outcomes.
Revised: This search space expansion enables NAS to discover more expressive and previously undiscovered architectures, signiﬁcantly expanding the search horizon and enriching the possible search outcomes.

Original: We further formulated heterogeneous residual block and attention guided search to solve the intrinsic search challenges involved.
Revised: We further formulated heterogeneous residual block and attention guided search to solve the intrinsic search challenges involved.

Original: Extensive experiments on image classiﬁcation show that NOS can discover novel and high-quality cell architectures in a cost-effective process.
Revised: Extensive experiments on image classiﬁcation show that NOS can discover novel and high-quality cell architectures in a cost-effective process.

Original: We hope that this work will shed light on the future directions for the NAS community
Revised: We hope that this work will shed light on the future directions for the NAS community

Original:  A.1. D YNAMIC C ONVOLUTIONS 
Revised: A A PPENDIX A. S ELF -C ALIBRATION O PERATIONS A.1. A TTENTION L EARNING

Revised: import torch 2 import torch.nn as nn 3 import torch.nn.functional as F 46 # channel-wise attention 7 class AttentionC(nn. Module): 8def _init affine

Original: {+y = self.sigm(self.conv_2(y)) return y 22 
+}
Revised: {+y = self.sigm(self.conv_2(y)) return y 22 
+}

Revised: # spatial-wise attention 24 class AttentionS(nn.

Revised: Module): 25def __init__(self, C_in, C_out, stride, reduction=16, affine=

Original: {+True): super(AttentionS, self).__init__() 28self.conv_
+}
Revised: {+True): super(AttentionS, self).__init__() 28self.conv_
+}

Original: {+= nn.Conv2d(C_in, C_in // reduction, 1, stride=stride, padding=0, bias=False) self.bn_
+}
Revised: {+= nn.Conv2d(C_in, C_in // reduction, 1, stride=stride, padding=0, bias=False) self.bn_
+}

Revised: = nn.

Revised: BatchNorm2d(C_in // reduction, affine=affine) self.relu = nn.ReLU(inplace=

Revised: False) self.conv_2 = nn.

Revised: Conv2d(C_in // reduction, 1, 3, stride=1, padding=1, bias=False) self.sigm = nn.

Revised: Sigmoid() 34def forward(self, x): 36y = self.relu(self.bn_1(self.conv_1(x)))

Revised: y

Revised: = self.sigm(self.conv_2(y)) return y

Revised: D YNAMIC C ONVOLUTIONS

Original: 1 # dynamic convolution 2 class DynamicF(nn.
Revised: 1 # dynamic convolution 2 class DynamicF(nn.

Original: Module): 3def __init__(self, C_in, C_out, F_size, reduction=8, affine=
Revised: Module): 3def __init__(self, C_in, C_out, F_size, reduction=8, affine=

Original: True): super(DynamicF, self).__init__() self.f_size =
Revised: True): super(DynamicF, self).__init__() self.f_size =

Original: F_size self.reduction = reduction self.c_in =
Revised: F_size self.reduction = reduction self.c_in =

Original: C_in self.c_out = C_out self.conv_1 = nn.Conv2d(C_in, C_in // reduction, 1, stride=1, padding =0, [-bias=False) self.bn_
-]
Revised: C_in self.c_out = C_out self.conv_1 = nn.Conv2d(C_in, C_in // reduction, 1, stride=1, padding =0, [-bias=False) self.bn_
-]bias= False) self

Original: = nn.
Revised: [Table]

Original: BatchNorm2d(C_in // reduction, affine
Revised: [Table] [Table]

Original: : 18input_x = self (self bn_1(self conv_1(x)))
Revised: forward x): 18input_x = self.relu(self.bn_1(self.conv_1(x)))

Original: N, C, H, W = input_x size() input_x = input_x view(N, C, H * W) # [N, C, H * W] input_x = input_x unsqueeze(1)
Revised: N, C, H, W = input_x.size() input_x = input_x view(N, C, H * W) # [N, C, H * W] input_x = input_x.unsqueeze(1)

Original: # [N, 1, C, H * W]
Revised: # [N, 1, C, H * W]

Original: dynamic_mask = self conv_2(x) # [N, -1, H , W] dynamic_mask =
Revised: dynamic_mask = self.conv_2(x) # [N, -1, H , W] dynamic_mask =

Original: dynamic_mask view(N, -1, H * W)# [N, -1, H * W]
Revised: dynamic_mask.view(N, -1, H * W)

Original:  dynamic_mask = self soft(dynamic_mask)
Revised: # [N, -1, H * W] dynamic_mask = self soft(dynamic_mask)

Original: # [N, -1, H * W] dynamic_mask = dynamic_mask unsqueeze(3) # [N, -1, H * W, 1] dynamic mask =
Revised: # [N, -1, H * W] dynamic_mask = dynamic_mask.unsqueeze(3) # [N, -1, H * W, 1] dynamic_mask =

Original: dynamic_mask permute(0,
Revised: dynamic_mask.permute(0,

Original: 1) # [N, 1, H * W, -1] input_x, dynamic_mask)
Revised: 1) # [N, 1, H * W, -1] dynamic = torch matmul(input_x, dynamic_mask)

Original: # [N, 1, C, -1] dynamic = dynamic squeeze(1) # [N, C, -1] 
Revised: # [N, 1, C, -1] dynamic = dynamic squeeze 1) # [N, C, -1] dynamic =

Original: = view(N, C, self f_size, self f_size) self (dynamic) dynamic = view( return
Revised: dynamic.view(N, C, self.f_size, self.f_size)

Original: A. D ETAILS OF T RAINING C ONFIGURATIONS A.2. CIFAR
Revised: A. D ETAILS OF T RAINING C ONFIGURATIONS A.2. CIFAR

Original: ResNet-18 and PyramidNet-110.
Revised: ResNet-18 and PyramidNet-110.

Original: We trained these models for 300 epochs with batch size 32.
Revised: We trained these models for 300 epochs with batch size 32.

Original: The learning rate was initialised as 0.025, which was decayed by 10 every 30 epochs.
Revised: The learning rate was initialised as 0.025, which was decayed by 10 every 30 epochs.

Original: The standard
Revised: The standard

Original: SGD optimiser with momentum of 0.9 was employed.
Revised: SGD optimiser with momentum of 0.9 was employed.

Original: We set a weight decay value of 1 ˆ 10 ´to avoid overﬁtting.
Revised: We set a weight decay value of 1 ˆ 10 ´to avoid overﬁtting.

Original: Other additional enhancements were not involved except the standard data augmentations.
Revised: Other additional enhancements were not involved except the standard data augmentations.

Original: Cell Search.
Revised: Cell Search.

Original: For network parameters Θ of proxy network, we used SGD with an initial learning rate 0.025 and set the momentum value as 0.9.
Revised: For network parameters Θ of proxy network, we used SGD with an initial learning rate 0.025 and set the momentum value as 0.9.

Original: This learning rate was decayed to 0 with a cosine scheduler.
Revised: This learning rate was decayed to 0 with a cosine scheduler.

Original: A weight decay value of 3 ˆ 10 ´ 4 was imposed to avoid over-ﬁtting.
Revised: A weight decay value of 3 ˆ 10 ´ 4 was imposed to avoid over-ﬁtting.

Original: For learning architecture matrix A , we used the Adam optimiser with a ﬁxed learning rate value 6 ˆ 10 ´ 4 and set the weight decay to 1 ˆ 10 ´ 3 .
Revised: For learning architecture matrix A , we used the Adam optimiser with a ﬁxed learning rate value 6 ˆ 10 ´ 4 and set the weight decay to 1 ˆ 10 ´ 3 .

Original: Cell Evaluation.
Revised: Cell Evaluation.

Original: The evaluation network was trained from scratch directly for 600 epochs with batch size 128.
Revised: The evaluation network was trained from scratch directly for 600 epochs with batch size 128.

Original: Note that, the attention transfer was not involved for training.
Revised: Note that, the attention transfer was not involved for training.

Original: We set the weight decay values for CIFAR-10 and CIFAR-100 to 3 ˆ 10 ´ 4 and 5 ˆ 10 ´ 4 individually.
Revised: We set the weight decay values for CIFAR-10 and CIFAR-100 to 3 ˆ 10 ´ 4 and 5 ˆ 10 ´ 4 individually.

Original: The standard
Revised: The standard

Original: SGD optimiser with a momentum of 0.9 was applied.
Revised: SGD optimiser with a momentum of 0.9 was applied.

Original: The initial learning rate was 0.25, decayed to 0 with a cosine scheduler.
Revised: The initial learning rate was 0.25, decayed to 0 with a cosine scheduler.

Original: Following existing works (Liu et al., 2019b; Pham et al., 2018; Zoph et al., 2018; Real et al., 2019), we performed two additional enhancements: the cutout regularisation DeVries & Taylor (2017) with length 16 and the drop-path Larsson et al. (2017) of probability 0.3.
Revised: Following existing works (Liu et al., 2019b; Pham et al., 2018; Zoph et al., 2018; Real et al., 2019), we performed two additional enhancements: the cutout regularisation (DeVries & Taylor, 2017) with length 16 and the drop-path (Larsson et al., 2017) of probability 0.3.

Original: A.2. I MAGE N ET
Revised: A.2. I MAGE N ET

Original: We trained the evaluation model for ImageNet using SGD optimiser for 300 epochs with batch size 512.
Revised: We trained the evaluation model for ImageNet using SGD optimiser for 300 epochs with batch size 512.

Original: We initialised the learning rate as 0.25 and reduced it to 0 by a linear scheduler.
Revised: We initialised the learning rate as 0.25 and reduced it to 0 by a linear scheduler.

Original: Learning rate warmup Goyal et al.
Revised: Learning rate warmup (Goyal et al., 2017) was applied for the ﬁrst 5 epochs to deal with the large batch size and learning rate. A. V ISUALISATION OF S EARCH S PACE A. D ISTILLATION E FFECT IN A TTENTION G UIDED S EARCH We examined the effect of knowledge distillation in the proposed Attention Guided Search (AGS).

Revised: We conducted this analysis on CIFAR10.

Revised: In this evaluation, we compared three methods: (1) Vanilla

Revised: Search: Using the original DARTS search method; (2) Distillation Only: Using the attention transfer loss for training the proxy network only; (3) AGS:

Revised: The proposed method (full).

Revised: We tracked the model performance on both training (train) and validation (val) data sets.

Revised: Figure 7 shows that (i) knowledge distillation brings a positive performance gain over the vanilla DARTS and (ii) using attention guidance for the network search can further improve the searched architecture.

Revised: A. G ENERALITY OF A TTENTION G UIDED S EARCH

Original: (2017) was applied for the ﬁrst 5 epochs to deal with the large batch size and learning rate.
Revised: We tested the general effect of the proposed Attention Guided Search (AGS) using the original DARTS search space ( O f + zero operation) on CIFAR10. During the search process, we followed the same settings as DARTS with the ﬁrst-order optimisation. We obtained the error rates: 3.

Revised: ˘ 0.14 (DARTS) vs. 2.92 ˘ 0.05 (DARTS + AGS).

Revised: This suggests a general efﬁcacy of AGS over different search spaces.