Original: IT-NAS: I NTEGRATING L ITE -T RANSFORMER INTO NAS FOR A RCHITECTURE S ELETION
Revised: IT-NAS: I NTEGRATING L ITE -T RANSFORMER INTO NAS FOR A RCHITECTURE S ELETION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural Architecture Search (NAS) aims to search for the best network in the predefined search space.
Revised: Neural Architecture Search (NAS) aims to search for the best network in the predefined search space.

Original: However, much work focuses on the search strategy but little on the architecture selection process.
Revised: However, much work focuses on the search strategy but little on the architecture selection process.

Original: Despite the fact that the weight-sharing based NAS has promoted the search efficiency, we notice that the architecture selection is quite unstable or circuitous.
Revised: Despite the fact that the weight-sharing based NAS has promoted the search efficiency, we notice that the architecture selection is quite unstable or circuitous.

Original: For instance, the differentiable NAS may derive the suboptimal architecture due to the performance collapse caused by bilevel optimization, or the One-shot NAS requires sampling and evaluating a large number of candidate structures.
Revised: For instance, the differentiable NAS may derive the suboptimal architecture due to the performance collapse caused by bilevel optimization, or the One-shot NAS requires sampling and evaluating a large number of candidate structures.

Original: Recently, the self-attention mechanism achieves better performance in terms of the long-range modeling capabilities.
Revised: Recently, the self-attention mechanism achieves better performance in terms of the long-range modeling capabilities.

Original: Considering that different operations are widely distributed in the search space, we suggest leveraging the self-attention mechanism to extract the relationship among them and to determine which operation is superior to others.
Revised: Considering that different operations are widely distributed in the search space, we suggest leveraging the self-attention mechanism to extract the relationship among them and to determine which operation is superior to others.

Original: Therefore, we integrate
Revised: Therefore, we integrate

Original: Lite-Transformer into NAS for architecture selection.
Revised: Lite-Transformer into NAS for architecture selection.

Original: Specifically, we regard the feature map of each candidate operation as distinct patches and feed them into the Lite-Transformer module along with an additional Indicator Token (called IT).
Revised: Specifically, we regard the feature map of each candidate operation as distinct patches and feed them into the Lite-Transformer module along with an additional Indicator Token (called IT).

Original: The cross attention among various operations can be extracted by the self-attention mechanism, and the importance of each candidate operation is then shown by the softmax result between the query of indicator token (IT) and other values of operational tokens.
Revised: The cross attention among various operations can be extracted by the self-attention mechanism, and the importance of each candidate operation is then shown by the softmax result between the query of indicator token (IT) and other values of operational tokens.

Original: We experimentally demonstrate that our framework can select the truly representative architecture in different search spaces and achieves 2.39% test error on CIFAR-10 in DARTS search space, and 24.1% test error on ImageNet in the ProxylessNAS  search space, as well as the stable and better performance in NAS-Bench-201 search space and S1-S4 search spaces, outperforming state-ofthe-art NAS methods.
Revised: We experimentally demonstrate that our framework can select the truly representative architecture in different search spaces and achieves 2.39% test error on CIFAR-10 in DARTS search space, and 24.1% test error on ImageNet in the ProxylessNAS (w/o SE module) search space, as well as the stable and comparable performance in NAS-Bench-201 search space, S1-S4 search spaces and NAS-Bench-1Shot1 search space.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Architecture Search (NAS) is emerging as a new paradigm for designing network structures.
Revised: Neural Architecture Search (NAS) is emerging as a new paradigm for designing network structures.

Original: It has been demonstrated to outperform manually designed networks in many tasks, including image classification (Zoph & Le, 2016;
Revised: It has been demonstrated to outperform manually designed networks in many tasks, including image classification (Zoph & Le, 2016;

Original: Zoph et al., 2018; Guo et al., 2020), object detection (Chen et al., 2019; Ghiasi et al., 2019), semantic segmentation (Chen et al., 2018;
Revised: Zoph et al., 2018; Guo et al., 2020), object detection (Chen et al., 2019; Ghiasi et al., 2019), semantic segmentation (Chen et al., 2018;

Original: Liu et al., 2019) and so on.
Revised: Liu et al., 2019) and so on.

Original: The fundamental disadvantage of earlier NAS methods, which primarily relied on heuristic algorithms like reinforcement learning (Baker et al., 2016; Bello et al., 2017; Zoph et al., 2018) or evolutionary algorithms (Real et al., 2017; Liu et al., 2018a; Real et al., 2019), is the necessity to train each architecture from scratch for validation, thereby impeding further advancement of NAS.
Revised: The fundamental disadvantage of earlier NAS methods, which primarily relied on heuristic algorithms like reinforcement learning (Baker et al., 2016; Bello et al., 2017; Zoph et al., 2018) or evolutionary algorithms (Real et al., 2017; Liu et al., 2018a; Real et al., 2019), is the necessity to train each architecture from scratch for validation, thereby impeding further advancement of NAS.

Original: Fortunately, ENAS (Pham et al., 2018) proposes a weight-sharing mechanism, which greatly improves search efficiency.
Revised: Fortunately, ENAS (Pham et al., 2018) proposes a weight-sharing mechanism, which greatly improves search efficiency.

Original: More recently, the differentiable NAS methods (Liu et al., 2018b; Xie et al., 2018) and the One-shot NAS methods (Guo et al., 2020; Chu et al., 2021b;
Revised: More recently, the differentiable NAS methods (Liu et al., 2018b; Xie et al., 2018) and the One-shot NAS methods (Guo et al., 2020; Chu et al., 2021b;

Original: You et al., 2020) have been more popular.
Revised: You et al., 2020) have been more popular.

Original: They both firstly train a super-network, and then derive the final architecture based on different strategies.
Revised: They both firstly train a super-network, and then derive the final architecture based on different strategies.

Original: However, the differentiable approach selects the target network with the largest architecture parameters, which cannot fully reflect the true operation strength (Wang et al., 2021a;
Revised: However, the differentiable approach selects the target network with the largest architecture parameters, which cannot fully reflect the true operation strength (Wang et al., 2021a;

Original: Xie et al., 2021b).
Revised: Xie et al., 2021b).

Original: Nevertheless, DARTS-PT requires fine-tuning after discretizing each edge, resulting in additional selection time.
Revised: Nevertheless, DARTS-PT requires fine-tuning after discretizing each edge, resulting in additional selection time.

Original: The One-shot method selects the optimal one by sampling and evaluating a large number of candidate structures, yielding the time-consuming issues (Chen et al., 2021a).
Revised: The One-shot method selects the optimal one by sampling and evaluating a large number of candidate structures, yielding the time-consuming issues (Chen et al., 2021a).

Original: Whereas, BN-NAS is not suitable for search space without batch normaliza- tion layer.
Revised: Whereas, BN-NAS is not suitable for search space without batch normaliza- tion layer.

Original: We are endeavoring to explore whether there is a more appropriate way for both popular search spaces to robustly and quickly select the architecture, as has rarely been studied before.
Revised: We are endeavoring to explore whether there is a more appropriate way for both popular search spaces to robustly and quickly select the architecture, as has rarely been studied before.

Original: The attention mechanism can be used to emphasize the important components of the input while ignoring other trivial ones.
Revised: The attention mechanism can be used to emphasize the important components of the input while ignoring other trivial ones.

Original: The Transformer architecture (Vaswani et al., 2017) has reignited a boom in the field of Computer Vision (CV) (Touvron et al., 2021; Liu et al., 2021) since ViT (Dosovitskiy et al., 2021) achieves competitive performance compared to Convolutional Neural Networks (CNNs).
Revised: The Transformer architecture (Vaswani et al., 2017) has reignited a boom in the field of Computer Vision (CV) (Touvron et al., 2021; Liu et al., 2021) since ViT (Dosovitskiy et al., 2021) achieves competitive performance compared to Convolutional Neural Networks (CNNs).

Original: It simply slices the image into small patches and then model the cross-attention between long sequences to locate key information, resulting in better performance.
Revised: It simply slices the image into small patches and then model the cross-attention between long sequences to locate key information, resulting in better performance.

Original: Meanwhile, Batchformer (Hou et al., 2022) introduces a batch transformer module that is applied to the batch dimension of each mini-batch of data to implicitly explore the sample relationships.
Revised: Meanwhile, Batchformer (Hou et al., 2022) introduces a batch transformer module that is applied to the batch dimension of each mini-batch of data to implicitly explore the sample relationships.

Original: Inspired by this, we can intuitively analogize the candidate operations in the NAS search space to patches, and then leverage the self-attention mechanism to describe the interactions among different operations.
Revised: Inspired by this, we can intuitively analogize the candidate operations in the NAS search space to patches, and then leverage the self-attention mechanism to describe the interactions among different operations.

Original: As a result, the optimal architecture can be selected according to the self-attention weights.
Revised: As a result, the optimal architecture can be selected according to the self-attention weights.

Original: In summary, we suggest integrating Lite-Transformer into NAS for architecture selection.
Revised: In summary, we suggest integrating Lite-Transformer into NAS for architecture selection.

Original: Specifically, we insert the Lite-Transformer module on each edge and determine the optimal operation associated with that edge in the cell-based search space.
Revised: Specifically, we insert the Lite-Transformer module on each edge and determine the optimal operation associated with that edge in the cell-based search space.

Original: While in the chain-style search space, where the network is defined by a sequence of layers containing various choice blocks, the LiteTransformer module is inserted on each layer to select the appropriate block.
Revised: While in the chain-style search space, where the network is defined by a sequence of layers containing various choice blocks, the LiteTransformer module is inserted on each layer to select the appropriate block.

Original: We adopt a more broad perspective that treats candidate operations as patches.
Revised: We adopt a more broad perspective that treats candidate operations as patches.

Original: The patches are then linearly mapped and further packed into three matrices, namely Q, K, and V. The softmax result between Q and K is a square matrix termed the attention map, which can be regarded as the attention weight between different candidate operations.
Revised: The patches are then linearly mapped and further packed into three matrices, namely Q, K, and V. The softmax result between Q and K is a square matrix termed the attention map, which can be regarded as the attention weight between different candidate operations.

Original: Furthermore, to address the issue of asymmetric attention matrix, i.e., the mutual attention value between any two operations cannot determine which is more favorable.
Revised: Furthermore, to address the issue of asymmetric attention matrix, i.e., the mutual attention value between any two operations cannot determine which is more favorable.

Original: We introduce an additional indicator token (called IT) to calculate the cross-attention between IT and the other operational tokens, which is inspired by the truth claimed by EViT (Liang et al., 2022) that the class token can be used to determine the importance of other tokens.
Revised: We introduce an additional indicator token (called IT) to calculate the cross-attention between IT and the other operational tokens, which is inspired by the truth claimed by EViT (Liang et al., 2022) that the class token can be used to determine the importance of other tokens.

Original: In this way, the importance corresponding to each operation can be represented by the row of the indicator token (IT) in the attention matrix.
Revised: In this way, the importance corresponding to each operation can be represented by the row of the indicator token (IT) in the attention matrix.

Original: After the super-network along with the Lite-Transformer module is trained to convergence, we simply need to forward propagate once on the validation dataset in order to determine the optimal architecture by computing the self-attention weights based on the indicator token (IT).
Revised: After the super-network along with the Lite-Transformer module is trained to convergence, we simply need to forward propagate once on the validation dataset in order to determine the optimal architecture by computing the self-attention weights based on the indicator token (IT).

Original: In general, our main contributions can be summarized as follows:
Revised: In general, our main contributions can be summarized as follows:

Original: • We revisit the architecture selection process of neural architecture search (NAS) in a fresh perspective and, to our knowledge, are the first to integrate Lite-Transformer into NAS for architecture selection, utilizing the self-attention mechanism to explore the interaction among different candidate operations by regarding each one as operational token.
Revised: • We revisit the architecture selection process of neural architecture search (NAS) in a fresh perspective and, to our knowledge, are the first to integrate Lite-Transformer into NAS for architecture selection, utilizing the self-attention mechanism to explore the interaction among different candidate operations by regarding each one as operational token.

Original: • We introduce an additional indicator token (called IT) to compute the cross-attention between IT and the other operational tokens.
Revised: • We introduce an additional indicator token (called IT) to compute the cross-attention between IT and the other operational tokens.

Original: In this case, the row of IT in self-attention weights matrix can be used to establish the priority of each candidate operation.
Revised: In this case, the row of IT in self-attention weights matrix can be used to establish the priority of each candidate operation.

Original: • Experimental results show that IT-NAS achieves stable and state-of-the-art performance infour popular search spaces, including DARTS search space, ProxylessNAS search space,  NAS-Bench-201, and S1-S4.
Revised: • Experimental results show that IT-NAS achieves better performance in DARTS and ProxylessNAS search space, as well as stable and comparable performance in NAS-Benches, including S1-S4, NAS-Bench-201, and NAS-Bench-1Shot1.

Original: • More comprehensive experiments demonstrate the robustness and effectiveness of IT-NAS in selecting architectures.
Revised: • More comprehensive experiments demonstrate the robustness and effectiveness of IT-NAS in selecting architectures.

Original: We also theoretically and empirically analyze why the selfattention mechanism can effectively select optimal architectures, proving the priority of our proposed method.
Revised: We also theoretically and empirically analyze why the selfattention mechanism can effectively select optimal architectures, proving the priority of our proposed method.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Neural Architecture Search.
Revised: Neural Architecture Search.

Original: Neural Architecture Search (NAS) aims to select the optimal architecture in the pre-defined search space.
Revised: Neural Architecture Search (NAS) aims to select the optimal architecture in the pre-defined search space.

Original: Earlier NAS approaches (Baker et al., 2016; Zoph et al., 2018;
Revised: Earlier NAS approaches (Baker et al., 2016; Zoph et al., 2018;

Original: Real et al., 2017; 2019) incurred substantial search overhead due to the requirement to train each candidate architecture from scratch.
Revised: Real et al., 2017; 2019) incurred substantial search overhead due to the requirement to train each candidate architecture from scratch.

Original: ENAS (Pham et al., 2018) firstly proposed the weight-sharing mechanism such that weights can be shared among different sub-structures in the super-network, greatly reducing the validation time.
Revised: ENAS (Pham et al., 2018) firstly proposed the weight-sharing mechanism such that weights can be shared among different sub-structures in the super-network, greatly reducing the validation time.

Original: Based on the mechanism, differentiable NAS and One-shot
Revised: Based on the mechanism, differentiable NAS and One-shot

Original: NAS became the two most mainstream paradigms.
Revised: NAS became the two most mainstream paradigms.

Original: Thereafter, a series of subsequent works focused on the design of search strategies and the improvement of super-network training.
Revised: Thereafter, a series of subsequent works focused on the design of search strategies and the improvement of super-network training.

Original: The differentiable NAS (Liu et al., 2018b) introduced architecture parameters which are mainly trained alternately with the network weights.
Revised: The differentiable NAS (Liu et al., 2018b) introduced architecture parameters which are mainly trained alternately with the network weights.

Original: However, the search process is very unstable and suffers from the performance collapse issue.
Revised: However, the search process is very unstable and suffers from the performance collapse issue.

Original: To improve the robustness of the search process, RobustDARTS (Zela et al., 2020) and SDARTS (Chen & Hsieh, 2020) analyzed and fixed optimization errors from a mathematical point of view.
Revised: To improve the robustness of the search process, RobustDARTS (Zela et al., 2020) and SDARTS (Chen & Hsieh, 2020) analyzed and fixed optimization errors from a mathematical point of view.

Original: To reduce the discretization gap of architecture selection,
Revised: To reduce the discretization gap of architecture selection,

Original: SNAS (Xie et al., 2018), GDAS (Dong & Yang, 2019), DATA (Chang et al., 2019) leveraged reparameterization techniques, and FairDARTS (Chu et al., 2020) introduced auxiliary loss, all with the aim of optimizing architecture parameters to approximate discrete forms.
Revised: SNAS (Xie et al., 2018), GDAS (Dong & Yang, 2019), DATA (Chang et al., 2019) leveraged reparameterization techniques, and FairDARTS (Chu et al., 2020) introduced auxiliary loss, all with the aim of optimizing architecture parameters to approximate discrete forms.

Original: These prior methods select target network based on architecture parameters, but recent work Wang et al.
Revised: These prior methods select target network based on architecture parameters, but recent work Wang et al.

Original: (2021a) pointed out that the operation associated with the largest magnitude of architecture parameters does not necessarily result in the highest validation accuracy after discretization.
Revised: (2021a) pointed out that the operation associated with the largest magnitude of architecture parameters does not necessarily result in the highest validation accuracy after discretization.

Original: It proposed a perturbation-based architecture selection strategy, which, however, requires fine-tuning after discretizing each edge, leading to substantial computation costs.
Revised: It proposed a perturbation-based architecture selection strategy, which, however, requires fine-tuning after discretizing each edge, leading to substantial computation costs.

Original: The One-shot NAS methods (Guo et al., 2020) focused on the super-network training.
Revised: The One-shot NAS methods (Guo et al., 2020) focused on the super-network training.

Original: SPOS (Guo et al., 2020) proposed to sample and train single-path sub-network uniformly; FairNAS (Chu et al., 2021b) guaranteed all paths are sampled each time for fairness; RLNAS (Zhang et al., 2021) focused on training with random labels.
Revised: SPOS (Guo et al., 2020) proposed to sample and train single-path sub-network uniformly; FairNAS (Chu et al., 2021b) guaranteed all paths are sampled each time for fairness; RLNAS (Zhang et al., 2021) focused on training with random labels.

Original: These methods then select the optimal substructure by sampling many sub-networks based on the evolutionary algorithm to evaluate their performance, which is very time-consuming.
Revised: These methods then select the optimal substructure by sampling many sub-networks based on the evolutionary algorithm to evaluate their performance, which is very time-consuming.

Original: BN-NAS (Chen et al., 2021a) proposed to train only BN layers and then select the architecture according to the parameters of BN layer.
Revised: BN-NAS (Chen et al., 2021a) proposed to train only BN layers and then select the architecture according to the parameters of BN layer.

Original: But it is obvious that this method may not be effective for some networks lacking BN layer in the search space, which limits its application.
Revised: But it is obvious that this method may not be effective for some networks lacking BN layer in the search space, which limits its application.

Original: Vision Transformer.
Revised: Vision Transformer.

Original: Transformer (Vaswani et al., 2017) has drawn much attention to computer vision recently due to its strong capability of modeling long-range relations.
Revised: Transformer (Vaswani et al., 2017) has drawn much attention to computer vision recently due to its strong capability of modeling long-range relations.

Original: ViT (Dosovitskiy et al., 2021) firstly introduced the pure Transformer Encoder without any convolutional layers and achieved comparable performance to CNNs.
Revised: ViT (Dosovitskiy et al., 2021) firstly introduced the pure Transformer Encoder without any convolutional layers and achieved comparable performance to CNNs.

Original: From then on, many works attempted to modify the
Revised: From then on, many works attempted to modify the

Original: ViT architecture to image classification (Yuan et al., 2021;
Revised: ViT architecture to image classification (Yuan et al., 2021;

Original: Zhou et al., 2021), object detection (Carion et al., 2020; Zhu et al., 2020; Liu et al., 2021), and semantic segmentation (Wang et al., 2021b;
Revised: Zhou et al., 2021), object detection (Carion et al., 2020; Zhu et al., 2020; Liu et al., 2021), and semantic segmentation (Wang et al., 2021b;

Original: Xie et al., 2021a; Chu et al., 2021a).
Revised: Xie et al., 2021a; Chu et al., 2021a).

Original: To further improve the performance of Transformer-like network, some works combine with NAS to automatically discover better architecture.
Revised: To further improve the performance of Transformer-like network, some works combine with NAS to automatically discover better architecture.

Original: For example, AutoFormer (Chen et al., 2021c) firstly aimed to search ViT. GLiT (Chen et al., 2021b) introduced locality modules into the search space and searched ViT from both global and local levels.
Revised: For example, AutoFormer (Chen et al., 2021c) firstly aimed to search ViT. GLiT (Chen et al., 2021b) introduced locality modules into the search space and searched ViT from both global and local levels.

Original: ViT-ResNAS (Liao et al., 2021) proposed to search for multi-stage ViT architecture.
Revised: ViT-ResNAS (Liao et al., 2021) proposed to search for multi-stage ViT architecture.

Original: Besides, other approaches make ViT more efficient by reducing the number of tokens.
Revised: Besides, other approaches make ViT more efficient by reducing the number of tokens.

Original: DynamicViT (Rao et al., 2021) introduced an additional learnable neural network to reduce the tokens of a fully trained ViT, TokenLearner (Ryoo et al., 2021) aggregated the entire feature map weighted by a dynamic attention map, and EViT (Liang et al., 2022) focused on the progressive selection of informative tokens during training.
Revised: DynamicViT (Rao et al., 2021) introduced an additional learnable neural network to reduce the tokens of a fully trained ViT, TokenLearner (Ryoo et al., 2021) aggregated the entire feature map weighted by a dynamic attention map, and EViT (Liang et al., 2022) focused on the progressive selection of informative tokens during training.

Original: Unlike searching the ViT network, we leverage Transformer to boost NAS.
Revised: Unlike searching the ViT network, we leverage Transformer to boost NAS.

Original: Inspired by the fact that the number of tokens can be reduced with little impact on performance, we analogize the candidate operations to tokens, and then use self-attention mechanism to select the truly important operations.
Revised: Inspired by the fact that the number of tokens can be reduced with little impact on performance, we analogize the candidate operations to tokens, and then use self-attention mechanism to select the truly important operations.

Original:  M ETHODOLOGY 3. NAS O VERVIEW
Revised:  M ETHODOLOGY 3. NAS O VERVIEW

Original: Differentiable NAS.
Revised: Differentiable NAS.

Original: In the traditional cell-based search space, each cell is defined as a directed acyclic graph (DAG) with N nodes, and each edge ( i, j ) between every node is associated with mixed operation
Revised: In the traditional cell-based search space, each cell is defined as a directed acyclic graph (DAG) with N nodes, and each edge ( i, j ) between every node is associated with mixed operation

Original: ¯ o ( i,j ) that is parameterized as architecture parameters
Revised: ¯ o ( i,j ) that is parameterized as architecture parameters

Original: α
Revised: α

Original: ( i,j ) by using softmax relaxation.
Revised: ( i,j ) by using softmax relaxation.

Original: The differentiable architecture search can be formulated as a bi-level optimization problem:
Revised: The differentiable architecture search can be formulated as a bi-level optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Indicator Token Operational Tokens
Revised: Indicator Token Operational Tokens

Original: Lite-Transformer
Revised: Lite-Transformer

Original: Lite-Transformer
Revised: Lite-Transformer

Original: Lite-Transformer ... O

Revised: Lite-Transformer ... O

Original: O
Revised: O

Original: On
Revised: On

Original: O 1 IT O
Revised: O 1 IT O

Original: O n ...
Revised: O n ...

Original: IT O 2 ...
Revised: IT O 2 ...

Original: O n O
Revised: O n O

Original: Self-Attention
Revised: Self-Attention

Original: Add & Norm
Revised: Add & Norm

Original: Feed Forward
Revised: Feed Forward

Original: Add & Norm
Revised: Add & Norm

Original: Positional Encoding ...
Revised: Positional Encoding ...

Original: Score of Operation
Revised: Score of Operation

Original: [Figure]
Revised: [Figure]

Original: ... O1 O
Revised: ... O1 O

Original: On ... O1 O
Revised: On ... O1 O

Original: On
Revised: On

Original: The optimal cell architecture is selected according to the architecture parameters α by discretizing each edge.
Revised: The optimal cell architecture is selected according to the architecture parameters α by discretizing each edge.

Original: In this way, the differentiable NAS methods argue that the optimized architecture parameters can represent the true importance of the candidate operation, which is unfortunately not the case (Wang et al., 2021a;
Revised: In this way, the differentiable NAS methods argue that the optimized architecture parameters can represent the true importance of the candidate operation, which is unfortunately not the case (Wang et al., 2021a;

Original: Xie et al., 2021b).
Revised: Xie et al., 2021b).

Original: One-shot NAS.
Revised: One-shot NAS.

Original: In the chain-style search space, One-shot NAS methods mainly focus on supernetwork training, which is expressed as:
Revised: In the chain-style search space, One-shot NAS methods mainly focus on supernetwork training, which is expressed as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Compared to Eq.(1), the continuous architecture parameters are discarded, and only network weights are optimized.
Revised: Compared to Eq.(1), the continuous architecture parameters are discarded, and only network weights are optimized.

Original: So the architecture selection process needs to sample a large number of sub-networks by evolution algorithm and evaluate each of them by sharing the super-network weights, resulting in the time-consuming issues (Guo et al., 2020; Chen et al., 2021a).
Revised: So the architecture selection process needs to sample a large number of sub-networks by evolution algorithm and evaluate each of them by sharing the super-network weights, resulting in the time-consuming issues (Guo et al., 2020; Chen et al., 2021a).

Original: No matter in the cell-based search space or the chain-style search space, any two network layers are composed of all candidate operations, with the goal of selecting out the optimal one from them.
Revised: No matter in the cell-based search space or the chain-style search space, any two network layers are composed of all candidate operations, with the goal of selecting out the optimal one from them.

Original: We take a new perspective for architecture selection, that is, neither using architecture parameters nor sampling sub-networks, but leveraging the self-attention mechanism to evaluate which operation is superior, and this has never been studied in NAS before.
Revised: We take a new perspective for architecture selection, that is, neither using architecture parameters nor sampling sub-networks, but leveraging the self-attention mechanism to evaluate which operation is superior, and this has never been studied in NAS before.

Original: 3. S ELF -A TTENTION M ECHANISM
Revised: 3. S ELF -A TTENTION M ECHANISM

Original: To model the relationship between individual different candidate operations, we first regard them as patches and then package all candidate operations of the layer into a sequence and feed them to the
Revised: To model the relationship between individual different candidate operations, we first regard them as patches and then package all candidate operations of the layer into a sequence and feed them to the

Original: Lite-Transformer Encoder.
Revised: Lite-Transformer Encoder.

Original: Let’s denote the feature maps sequence of n candidate operations on the searchable layer L as
Revised: Let’s denote the feature maps sequence of n candidate operations on the searchable layer L as

Original: O i L ∈ R b × c × h × w ( i = 1 , 2 , ..., n ) .
Revised: O i L ∈ R b × c × h × w ( i = 1 , 2 , ..., n ) .

Original: We first flatten each feature to O i L ∈ R b × 1 × ( c ∗ h ∗ w ) so that each one can be analogized as a patch similar to ViT (Dosovitskiy et al., 2021).
Revised: We first flatten each feature to O i L ∈ R b × 1 × ( c ∗ h ∗ w ) so that each one can be analogized as a patch similar to ViT (Dosovitskiy et al., 2021).

Original: And then we concatenate n feature maps in the second dimension as O L ∈ R b × n × ( c ∗ h ∗ w ) .
Revised: And then we concatenate n feature maps in the second dimension as O L ∈ R b × n × ( c ∗ h ∗ w ) .

Original: For shorthand, let d = c ∗ h ∗ w denotes the embedding dimension.
Revised: For shorthand, let d = c ∗ h ∗ w denotes the embedding dimension.

Original: Self-Attention.
Revised: Self-Attention.

Original: A transformation layer projects each operational sequence X ∈ R n × d to three different sequential vectors (namely, query Q , key K , and value V ), where n and d are the length and dimension of the input sequences, respectively.
Revised: A transformation layer projects each operational sequence X ∈ R n × d to three different sequential vectors (namely, query Q , key K , and value V ), where n and d are the length and dimension of the input sequences, respectively.

Original: The two sequences of query and key input scheme are referred to as cross-attention mechanism.
Revised: The two sequences of query and key input scheme are referred to as cross-attention mechanism.

Original: Specifically, it explicitly aggregates the query with the corresponding key, assigns them to the value, and updates the output vector:
Revised: Specifically, it explicitly aggregates the query with the corresponding key, assigns them to the value, and updates the output vector:

Original: [Equation]
Revised: [Equation]

Original: where d q is the length of query vector.
Revised: where d q is the length of query vector.

Original: The result of Softmax( · ) is the self-attention weight square matrix, where each entry represents the cross-attention between any two query and key.
Revised: The result of Softmax( · ) is the self-attention weight square matrix, where each entry represents the cross-attention between any two query and key.

Original: However, this matrix is not symmetric as shown in Figure 1.
Revised: However, this matrix is not symmetric as shown in Figure 1.

Original: That is, the attention weight between query O i and key
Revised: That is, the attention weight between query O i and key

Original: O j is not equal to the attention weight between query O j and key
Revised: O j is not equal to the attention weight between query O j and key

Original: O i , so it is not possible to directly measure whether O i or O j is more important.
Revised: O i , so it is not possible to directly measure whether O i or O j is more important.

Original: Indicator Token.
Revised: Indicator Token.

Original: We introduce an additional indicator token (called IT), which aims to explicitly indicate the importance of each candidate operation.
Revised: We introduce an additional indicator token (called IT), which aims to explicitly indicate the importance of each candidate operation.

Original: The IT is concatenated in series in front of the operational tokens before feeding them into the Lite-Transformer Encoder.
Revised: The IT is concatenated in series in front of the operational tokens before feeding them into the Lite-Transformer Encoder.

Original: The interaction between
Revised: The interaction between

Original: IT and other operational tokens is computed by the self-attention mentioned before:
Revised: IT and other operational tokens is computed by the self-attention mentioned before:

Original: [Equation]
Revised: [Equation]

Original: where q it is the query of indicator token, K and V are operational key matrix and value matrix.
Revised: where q it is the query of indicator token, K and V are operational key matrix and value matrix.

Original: So, Z it denotes the linear combination of the value vectors V =
Revised: So, Z it denotes the linear combination of the value vectors V =

Original: [ v 1 , v 2 , ..., v n ] with the attention weights vector a =
Revised: [ v 1 , v 2 , ..., v n ] with the attention weights vector a =

Original: [ a 1 , a 2 , ..., a n ] .
Revised: [ a 1 , a 2 , ..., a n ] .

Original: Because v i is the linear transformation from i -th operation, the attention weight a
Revised: Because v i is the linear transformation from i -th operation, the attention weight a

Original: i therefore can be seen how importance of this operation over other operations.
Revised: i therefore can be seen how importance of this operation over other operations.

Original: 3. A RCHITECTURE S ELECTION
Revised: 3. A RCHITECTURE S ELECTION

Original: Supernet Training.
Revised: Supernet Training.

Original: The Lite-Transformer Encoder is simplified to repeat only once and consists of a single-head Self-Attention (SA) and a Feed-Forward-Network (FFN).
Revised: The Lite-Transformer Encoder is simplified to repeat only once and consists of a single-head Self-Attention (SA) and a Feed-Forward-Network (FFN).

Original: Residual connections alongside layer normalization (LN) (Ba et al., 2016) are employed after each layer.We integrate
Revised: Residual connections alongside layer normalization (LN) (Ba et al., 2016) are employed after each layer.

Original: Lite-Transformer T into super-network N , and then train themon the training datasets following the settings same as DARTS (Liu et al., 2018b) for differentiable NAS or SPOS (Guo et al., 2020) for One-shot NAS.
Revised: We integrate LiteTransformer T into super-network N , and then train them following the settings same as DARTS (Liu et al., 2018b) for differentiable NAS or SPOS (Guo et al., 2020) for One-shot NAS.

Original: Therefore, Eq.(1) and Eq.(2) can be unified as follows:
Revised: Therefore, Eq.(1) and Eq.(2) can be unified as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the Transformer module is optimized together with the supernet by minimizing losses on the training dataset.
Revised: where W N denotes the super-network weights and W T is the Transformer wights which is optimized together with the super-network by minimizing losses on the training dataset.

Revised: As a result, we unify the differentiable NAS and One-shot NAS into a mono-level optimization.

Original: As a result, the indicator token (IT) can access the knowledge about the effect of different operations to the network during this period by exploring the interaction among them.
Revised: The indicator token (IT) can access the knowledge about the effect of different operations to the network during this period by exploring the interaction among candidate operations.

Original: Architecture Selection.
Revised: Architecture Selection.

Original: The architecture selection process is performed after the training is completed.
Revised: The architecture selection process is performed after the training is completed.

Original: We propagate all images of the validation datasets to obtain the self-attention weights.
Revised: We propagate all images of the validation datasets to obtain the self-attention weights.

Original: Specifically, the first row of the result of the Softmax( · ) matrix, i.e., the cross-attention between the query of the indicator token (IT) and the keys of other operational tokens, denotes the importance of each candidate operation.
Revised: Specifically, the first row of the result of the Softmax( · ) matrix, i.e., the cross-attention between the query of the indicator token (IT) and the keys of other operational tokens, denotes the importance of each candidate operation.

Original: In this case, we can select the optimal operation on each searchable layer according to this importance indicator:
Revised: In this case, we can select the optimal operation on each searchable layer according to this importance indicator:

Original: [Equation]
Revised: [Equation]

Original: Becausethe first entry of the attention weights sequence a  indicates the self-attention of the query of indicator token  and the value of itself, we only need to index the last n cross-attention weights a [ a 1 , a 2 , ..., a n ] , where a i represent the importance of the i -th operations.
Revised: Because a 0 indicates the self-attention of the query of indicator token (IT) and the value of itself, we only need to index the last n cross-attention weights a [ a 1 , a 2 , ..., a n ] , where a i represent the importance of the i -th operations.

Original: Finally, the operation associated with the largest attention weight  each layer is selected to construct the target network.
Revised: Finally, the operation associated with the largest attention weight for each layer is selected to construct the target network.

Original: 3. I N - DEPTH A NALYSIS
Revised: 3. I N - DEPTH A NALYSIS

Original: Fourier Analysis.
Revised: Fourier Analysis.

Original: The self-attention layer actually acts like low-pass filter with the aim of reducing high-frequency signals.
Revised: The self-attention layer actually acts like low-pass filter with the aim of reducing high-frequency signals.

Original: Given the indicator token (IT) query vector q it , a sequence of operational token key vectors K = { k 1 , k 2 , ..., k n } , and the corresponding value vectors V = { v 1 , v 2 , ..., v n } .
Revised: Given the indicator token (IT) query vector q it , a sequence of operational token key vectors K = { k 1 , k 2 , ..., k n } , and the corresponding value vectors V = { v 1 , v 2 , ..., v n } .

Original: The  output of self-attention is:
Revised: The indicator token (IT) output of self-attention is:

Original: [Equation]
Revised: [Equation]

Original: where h ( q it ) =
Revised: where h ( q it ) =

Original: (cid:80) nj =1 exp( q it · k j ) .
Revised: (cid:80) nj =1 exp( q it · k j ) .

Original: Further, we can rewrite Eq.(7) as follows:
Revised: Further, we can rewrite Eq.(7) as follows:

Original: [Equation]
Revised: [Equation]

Revised: where h ′ ( q it ) =

Revised: (cid:80) nj =1 exp( − 12 ||

Original: where h ′ ( q it  ) is the normalized function, ∗ denotes the high-dimensional convolution, G ( · ) is the Gaussian kernel, and S ( · ) is high-dimensional sparse signal.
Revised: q it − k j || 2 ) is the normalized factor, ∗ denotes the high-dimensional convolution, G ( · ) is the Gaussian kernel, and S ( · ) is high-dimensional sparse signal.

Original: Because Gaussian filter is low-pass, the indicator token output o ( q it )of the self-attention layer contains redundant information based on
Revised: Because Gaussian filter is low-pass, the indicator token output o ( q it ) contains redundant information based on

Original: Shannon sampling theorem (Shannon, 1949).
Revised: Shannon sampling theorem (Shannon, 1949).

Original: In other words, not all operational tokens are equally important, but the one with the largest attention weight contributes mostly to network performance.
Revised: In other words, not all operational tokens are equally important, but the one with the largest attention weight contributes most to network performance.

Original: [Table]
Revised: [Table]

Original: Complexity Analysis.
Revised: Complexity Analysis.

Original: Compared with the complexity of O ( n 2 d ) in the classical ViT, our method can be approximated to O ( d ) under the condition of n 2 ≪ d .
Revised: Compared with the complexity of O ( n 2 d ) in the classical ViT, our method can be approximated to O ( d ) under the condition of n 2 ≪ d .

Original: Because n generally does not exceed ten, and d can be hundreds or thousands in NAS.
Revised: Because n generally does not exceed ten, and d can be hundreds or thousands in NAS.

Original: Therefore, the search efficiency is not significantly impacted by the calculation of attention.
Revised: Therefore, the search efficiency is not significantly impacted by the calculation of attention.

Original: As shown in Table 1, GPU-memory overhead is not much increased than that of the baseline method.
Revised: As shown in Table 1, GPU-memory overhead is not much increased than that of the baseline method.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct the experiments on two popular search spaces, including DARTS search space on CIFAR-10 dataset (Krizhevsky et al., 2009), and ProxylessNAS search space on ImageNet dataset (Krizhevsky et al., 2017).
Revised: We firstly search on two popular search spaces, including DARTS search space on CIFAR-10 dataset (Krizhevsky et al., 2009), and ProxylessNAS search space on ImageNet dataset (Krizhevsky et al., 2017).

Original: Moreover, we also evaluate the robustness and effectiveness of IT-NAS on two benchmarksearch spaces of NAS-Bench-201 and S1-S4.
Revised: Moreover, we also evaluate the robustness and effectiveness of IT-NAS on three benchmark of NAS-Bench-201, S1-S4 and NAS-Bench-1Shot1.

Original: Details are in Appendix A.1, A.2.
Revised: Details are in Appendix A.1, A.2, A.5.

Original:  R ESULTS IN DARTS S EARCH S PACE
Revised: 4. R ESULTS IN DARTS S EARCH S PACE

Original: Unlike DARTS (Liu et al., 2018b) that performs bi-level optimization by alternately updating architecture parameters and network weights, IT-NAS only need to train the super-network along with the Lite-Transformer module on half of the CIFAR-10 training dataset.
Revised: Unlike DARTS (Liu et al., 2018b) that performs bi-level optimization by alternately updating architecture parameters and network weights, IT-NAS only need to train the super-network along with the Lite-Transformer module on half of the CIFAR-10 training dataset.

Original: The search process merely elapses 6 hours in total on Tesla V100 GPU.
Revised: The search process merely elapses 6 hours in total on Tesla V100 GPU.

Original: After the training convergence, we propagate the other half of the CIFAR-10 training dataset as validation dataset to compute the self-attention weights.
Revised: After the training convergence, we propagate the other half of the CIFAR-10 training dataset as validation dataset to compute the self-attention weights.

Original: Then we derive the optimal operation on each edge of the normal cell and reduction cell according to Eq.(6).
Revised: Then we derive the optimal operation on each edge of the normal cell and reduction cell according to Eq.(6).

Original: To evaluate the performance, the target network consisted of 20 cells with initial channel size of 36 is trained on the whole training dataset from scratch.
Revised: To evaluate the performance, the target network consisting of 20 cells with initial channel size of 36 is trained on the whole training dataset from scratch.

Original: The details of super-network training and target network retraining settings are in Appendix A.3.
Revised: The details of super-network training and target network retraining settings are in Appendix A.3.

Original: As shown in Table 2, we can see that our IT-NAS achieves state-of-the-art performance compared with other methods in the DARTS search space.
Revised: As shown in Table 2, we can see that our IT-NAS achieves state-of-the-art performance compared with other methods in the DARTS search space.

Original: We report the average results of 3 independent runs with different random seeds to test the effectiveness and stability of our method.
Revised: We report the average results of 3 independent runs with different random seeds to test the effectiveness and stability of our method.

Original: Our approach achieves the average test error of 2.41% with the standard deviation of 0.02 on CIFAT-10 dataset, demonstrating IT-NAS is very stable.
Revised: Our approach achieves the average test error of 2.41% with the standard deviation of 0.02 on CIFAT-10 dataset, demonstrating IT-NAS is very stable.

Original: Moreover, the best result reaches the error rate of 2.39%, outperforming all the other methods.
Revised: Moreover, the best result reaches the error rate of 2.39%, outperforming all the other methods.

Original: We also transfer the cells searched on CIFAR-10 to other datasets for evaluation.
Revised: We also transfer the cells searched on CIFAR-10 to other datasets for evaluation.

Original: As a result, IT-NAS achieves the accuracy of 83.30% ± 0.06 on CIFARby validating the three searched cells in Figure 5.
Revised: As a result, IT-NAS achieves the accuracy of 83.30% ± 0.06 on CIFARby validating the three searched cells in Figure 15.

Original: Moreover, IT-NAS achieves the Top-1 Accuracy of 75.5% on ImageNet by validating the searched Cell 2 in Figure 5.
Revised: Moreover, IT-NAS achieves the Top-1 Accuracy of 75.5% on ImageNet by validating the searched Cell 2 in Figure 15.

Original: 4. R ESULTS IN P ROXYLESS NAS S EARCH S PACE
Revised: 4. R ESULTS IN P ROXYLESS NAS S EARCH S PACE

Original: We directly search on ImageNet dataset to evaluate our method in the ProxylessNAS  search space.
Revised: We directly search on ImageNet dataset to evaluate our method in the ProxylessNAS (w/o SE module) search space.

Original: Following PC-DARTS (Xu et al., 2019), we randomly sample two subsets from1.3M training
Revised: Following PC-DARTS (Xu et al., 2019), we randomly sample two subsets from

Original: [Table]
Revised: [Table]

Original:  searching with the evolutionary algorithm.
Revised: the search cost includes the additional subnet searching with the evolutionary algorithm.

Original: [Table]
Revised: [Table]

Original:  dataset of ImageNet, with 10% and 2.5% images as training and validation dataset, respectively.
Revised: 1.3M training dataset of ImageNet, with 10% and 2.5% images as training and validation dataset, respectively.

Original: In particular, we train the super-network by uniform sampling single paths following SPOS (Guo et al., 2020) on 8 Tesla V100 GPUs with a total batch size of 512 except for 240 epochs.
Revised: In particular, we train the super-network by uniformly sampling single paths following SPOS (Guo et al., 2020) on 8 Tesla V100 GPUs with a total batch size of 512 except forepochs.

Original: The training process elapses 12 hours totally.
Revised: The training process elapses 12 hours totally.

Original: After that, the optimal sub-network is derived according to the self-attention weights on each searchable layer by propagating the validation dataset once time.
Revised: After that, the optimal sub-network is derived according to the self-attention weights on each searchable layer by propagating the validation dataset once.

Original: We restrict the mobile setting to under 600M FLOPs for fair comparison with other methods.
Revised: We restrict the mobile setting to under 600M FLOPs for fair comparison with other methods.

Original: The target network is retrained from scratch on the whole ImageNet training dataset for 240 epochs with the batch size of 1024 on 8 Tesla V100 GPUs.
Revised: The target network is retrained from scratch on the whole ImageNet training dataset for 240 epochs with the batch size of 1024 on 8 Tesla V100 GPUs.

Original: From Table 3, we can see that IT-NAS achieves the best performance on ImageNet.
Revised: From Table 3, we can see that IT-NAS achieves the best performance on ImageNet.

Original: Besides, the search cost is also the lowest compared with other evolutionary algorithm-based methods, indicating that our approach is effective and efficient.
Revised: Besides, the search cost is also the lowest compared with other evolutionary algorithm-based methods, indicating that our approach is effective and efficient.

Revised: The retrained architecture with SE module in ProxylessNAS space is summarized in Appendix A.

Original: 4. R ESULTS IN B ENCHMARK S EARCH S PACE
Revised: 4. R ESULTS IN B ENCHMARK S EARCH S PACE

Original: S1-S4.
Revised: S1-S4.

Original: We also conduct experiments on the reduced search spaces S1-S4 introduced by RobustDARTS (Zela et al., 2020).
Revised: We also conduct experiments on the reduced search spaces S1-S4 introduced by RobustDARTS (Zela et al., 2020).

Original: As shown in Table 4, compared with other methods to regularize architecture parameters, IT-NAS is able to consistently select the optimal architectures with the impressive performance on all three datasets of different search spaces.
Revised: As shown in Table 4, compared with other methods to regularize architecture parameters, IT-NAS is able to consistently select the optimal architectures with the impressive performance on all three datasets of different search spaces.

Original: This once again confirms the effectiveness of self-attention weights in indicating the importance of candidate operations. 
Revised: This once again confirms the effectiveness of self-attention weights in indicating the importance of candidate operations. [Table] [Figure]

Original: Robustness.
Revised: Robustness.

Original: To evaluate the robustness of IT-NAS, we track the performance of IT-NAS and
Revised: To evaluate the robustness of IT-NAS, we track the performance of IT-NAS and

Original: DARTS over the search epochs on NAS-Bench-201.
Revised: DARTS over the search epochs on NAS-Bench-201.

Original: As plotted in Figure 2, IT-NAS achieves sta-

Original: [Table]

Original: [Figure] c y [Figure] ble and state-of-the-art results on three datasets compared to DARTS.
Revised: As plotted in Figure 2, IT-NAS achieves stable and state-of-the-art results on three datasets compared to DARTS.

Original: Whereas, the performance of
Revised: Whereas, the performance of

Original: DARTS is always inferior to that of our method and the standard deviation fluctuates widely as the search epochs through different trials.
Revised: DARTS is always inferior to that of our method and the standard deviation fluctuates widely as the search epochs through different trials.

Original: Because the bi-level optimization of DARTS suffers from performance collapse issues (Zela et al., 2020; Chen & Hsieh, 2020), the architecture selection process is easily disturbed by the magnitude of architecture parameters.
Revised: Because the bi-level optimization of DARTS suffers from performance collapse issues (Zela et al., 2020; Chen & Hsieh, 2020), the architecture selection process is easily disturbed by the magnitude of architecture parameters.

Original: More seriously, DARTS will downgrade the performance due to the domination of skip-connect operation at the end of the search phase (Chu et al., 2020).
Revised: More seriously, DARTS will downgrade the performance due to the domination of skip-connect operation at the end of the search phase (Chu et al., 2020).

Original: On the contrary, the architecture selection of It-NAS dominated by the self-attention mechanism is more stable than architecture parameters.
Revised: On the contrary, the architecture selection of IT-NAS dominated by the self-attention mechanism is more stable than architecture parameters.

Original: The results are in Table 5.
Revised: The results are in Table 5.

Original: Effectiveness.
Revised: Effectiveness.

Original: The effectiveness of the search results can be expressed by the relationship between the architecture selection indicator and its derived architecture performance.
Revised: The effectiveness of the search results can be expressed by the relationship between the architecture selection indicator and its derived architecture performance.

Original: We conduct the experiments in NAS-Bench-201 search space.
Revised: We conduct the experiments in NAS-Bench-201 search space.

Original: We rank the selection indicator corresponding to all candidate operations on each edge, and also rank the discretization accuracy obtained by retaining the corresponding operation on this edge.
Revised: We rank the selection indicator corresponding to all candidate operations on each edge, and also rank the discretization accuracy obtained by retaining the corresponding operation on this edge.

Original: The results are plotted in Figure 3, the higher the indicator rank, the more important its corresponding operation is, and so is the accuracy ranking.
Revised: The results are plotted in Figure 3, the higher the indicator rank, the more important its corresponding operation is, and so is the accuracy ranking.

Original: We also regress the relationship between the two rankings, from which we can see that the kendall-tau of
Revised: We also regress the relationship between the two rankings, from which we can see that the kendall-tau of

Original: IT-NAS is better than DARTS on all edges, indicating that the architecture selection criteria based on self-attention mechanism is more accurate and effective than architecture parameters.
Revised: IT-NAS is better than DARTS on all edges, indicating that the architecture selection criteria based on self-attention mechanism is more accurate and effective than architecture parameters.

Original: 4. V ISUALIZATION A NALYSIS
Revised: 4. V ISUALIZATION A NALYSIS

Original: Here we intuitively demonstrate whether the optimal operation selected by the self-attention is truly beneficial to the network.
Revised: Here we intuitively demonstrate whether the optimal operation selected by the self-attention is truly beneficial to the network.

Original: Let’s take the super-network with six different operations on the last layerof the ProxylessNAS search space as an example.
Revised: Let’s take the super-network with six different operations on the last layer

Original: We map the gradient information corresponding
Revised: 5 Indicator

Original: [Figure]
Revised: [Figure]

Original: A cc u r a c y
Revised: A cc u r a c y

Original: R a n k i n g
Revised: R a n k i n g

Original: IT-NAS [Figure] [Figure]
Revised: IT-NAS, Kendall=0.95 DARTS, Kendall=-0.

Original: [Figure]
Revised: (a) Edge    5 Indicator Ranking A cc u r a c y R a n k i n g IT-NAS, Kendall=0.53 DARTS, Kendall=-0.

Original: [Figure]
Revised: (b) Edge    5 Indicator Ranking

Original: A cc u r a c y
Revised: A cc u r a c y

Original: R a n k i n g
Revised: R a n k i n g

Original: IT-NAS [Figure] Kendall=0. [Figure]
Revised: IT-NAS, Kendall=0.95 DARTS, Kendall=-0.

Original: = [Figure] [Figure] [Figure]
Revised: (c) Edge

Original: [Figure]
Revised: [Figure]

Original: A cc u r a c y
Revised: A cc u r a c y

Original: R a n k i n g
Revised: R a n k i n g

Original: IT-
Revised: IT-

Original: [Figure]
Revised: [Figure]

Original: =

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] c
Revised: =0.

Original: [Figure]
Revised: [Figure]

Original: R a n k i n g
Revised: =0.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  u r a c y
Revised: A cc u r a c y

Original: R a n k i n g
Revised: R a n k i n g

Original: IT- [Figure] [Figure]
Revised: IT-NAS, Kendall=0.53 DARTS, Kendall=0. (e) Edge

Original: [Figure]
Revised: [Figure]

Revised: Ranking

Original:  u r a c y
Revised: A cc u r a c y

Original: R a n k i n g
Revised: R a n k i n g

Revised: IT-NAS, Kendall=1.00 DARTS, Kendall=-0.

Revised: (f)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: to to
Revised: of the ProxylessNAS search space as an example.

Original: Selvaraju et al., 2017).
Revised: We map the gradient information corresponding to each operation to the input image by using Grad-CAM technique (Selvaraju et al., 2017).

Original: The area with higher class activation mapping represents greater responses.
Revised: The area with higher class activation mapping represents greater responses.

Original: As shown in Figure 4, the operation MBE3 K3 locates the object most accurately, and the operation is also selected by IT-NAS on the last layer as shown in Figure 6.
Revised: As shown in Figure 4, the operation MBE3 K3 locates the object most accurately, and the operation is also selected by IT-NAS on the last layer as shown in Figure 16.

Original: This consistency phenomenon demonstrates that the selfattention mechanism can select the most suitable operation for object perception, that in turn proves the effectiveness of our method from the perspective of gradient information.
Revised: This consistency phenomenon demonstrates that the selfattention mechanism can select the most suitable operation for object perception, that in turn proves the effectiveness of our method from the perspective of gradient information.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose for the first time to integrate Lite-Transformer into NAS for architecture selection.
Revised: In this paper, we propose for the first time to integrate Lite-Transformer into NAS for architecture selection.

Original: The candidate operations are regarded as tokens, and then introduce indicator token (IT) to explore the relationship between other operations and assign attention weights to which for selecting the optimal one.
Revised: The candidate operations are regarded as tokens, and then introduce indicator token (IT) to explore the relationship between other operations and assign attention weights to which for selecting the optimal one.

Original: Comprehensive experiments demonstrate that IT-NAS is morestable and effective compared with other architecture parameter-based or evolutionary-based architecture selection processes, and we also achieve state-of-the-art results in different  search spaces.
Revised: Comprehensive experiments demonstrate that IT-NAS is more effective compared with other architecture parameter-based or evolutionary-based architecture selection processes, and we also achieve stable results in different benchmark search spaces.

Original: This work may inspire researchers to explore more appropriate architecture selection criterion, such as borrowing from pruning or other related research, to promote the progress of the NAS field.
Revised: This work may inspire researchers to explore more appropriate architecture selection criterion, such as borrowing from pruning or other related research, to promote the progress of the NAS field.

Original: A A PPENDIX A. D ATASETS
Revised: A A PPENDIX A. D ATASETS

Original: We conduct the experiments on different image classification datasets for various search spaces.
Revised: We conduct the experiments on different image classification datasets for various search spaces.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: The dataset contains 50K training images and 10K testing images with a fixed resolution of 32x32.
Revised: The dataset contains 50K training images and 10K testing images with a fixed resolution of 32x32.

Original: The training dataset is divided into two parts, one half is used for training network weights and Lite-Transformer modules, and the other half is used as the validation dataset to forward propagation once to obtain the self-attention weights of each candidate operation for selecting the final network architecture.
Revised: The training dataset is divided into two parts, one half is used for training network weights and Lite-Transformer modules, and the other half is used as the validation dataset to forward propagation once to obtain the self-attention weights of each candidate operation for selecting the final network architecture.

Original: CIFAR-100.
Revised: CIFAR-100.

Original: The dataset has the same number of images as CIFAR-10 but is more categorized into 100 fine-grained classes.
Revised: The dataset has the same number of images as CIFAR-10 but is more categorized into 100 fine-grained classes.

Original: There are 500 training images and 100 testing images per class.
Revised: There are 500 training images and 100 testing images per class.

Original: ImageNet.
Revised: ImageNet.

Original: The dataset has 1.28M training images and 50K validation images with 1000 object categories.
Revised: The dataset has 1.28M training images and 50K validation images with 1000 object categories.

Original: We sample 10% of the training datasets to train the super-network weights as well as the
Revised: We sample 10% of the training datasets to train the super-network weights as well as the

Original: Lite-Transformer module weights, and another 2.5% training datasets to select the final architecture based on the self-attention weights.
Revised: Lite-Transformer module weights, and another 2.5% training datasets to select the final architecture based on the self-attention weights.

Original: ImageNet-16-120.
Revised: ImageNet-16-120.

Original: The dataset down-samples the original ILSVRC2012 ImageNet to 16x16 resolution and only selects the first 120 categories.
Revised: The dataset down-samples the original ILSVRC2012 ImageNet to 16x16 resolution and only selects the first 120 categories.

Original: SVHN.
Revised: SVHN.

Original: The dataset is a digit classification benchmark dataset that contains around 600,000 32×
Revised: The dataset is a digit classification benchmark dataset that contains around 600,000 32×

Original: RGB images of printed digits (from 0 to 9) cropped from pictures of house number plates.
Revised: RGB images of printed digits (from 0 to 9) cropped from pictures of house number plates.

Original: SVHN has three sets: 73257 digits for training, 26032 digits for testing and an extra set with 531,131 images that are less difficult and can be used for helping with the training process.
Revised: SVHN has three sets: 73257 digits for training, 26032 digits for testing and an extra set with 531,131 images that are less difficult and can be used for helping with the training process.

Original: A. S EARCH S PACES
Revised: A. S EARCH S PACES

Original: We experiment on four popular search spaces, including DARTS search space, ProxylessNAS search space, NAS-Bench-201 search space, and S1-S4.
Revised: We experiment on four popular search spaces, including DARTS search space, ProxylessNAS search space, NAS-Bench-201 search space, and S1-S4.

Original: DARTS Search Space.
Revised: DARTS Search Space.

Original: The cell-based search space aims to search for the normal cell and reduction cell.
Revised: The cell-based search space aims to search for the normal cell and reduction cell.

Original: Each cell is defined as a directed acyclic graph (DAG) consisting of an ordered sequence of N nodes.
Revised: Each cell is defined as a directed acyclic graph (DAG) consisting of an ordered sequence of N nodes.

Original: The edge between two nodes is mixed up by the searchable candidate operations, including { sep-conv-3 × 3, sep-conv-5 × 5, dil-conv-3 × 3, dil-conv-5 × 5, avg-pool-3 × 3, max-pool-3 × 3, identity and none } .
Revised: The edge between two nodes is mixed up by the searchable candidate operations, including { sep-conv-3 × 3, sep-conv-5 × 5, dil-conv-3 × 3, dil-conv-5 × 5, avg-pool-3 × 3, max-pool-3 × 3, identity and none } .

Original: ProxylessNAS Search Space.
Revised: ProxylessNAS Search Space.

Original: The chain-style search space defines the network macro architecture and directly searches for appropriate operations at each layer.
Revised: The chain-style search space defines the network macro architecture and directly searches for appropriate operations at each layer.

Original: There are a total of 21 searchable layers, and each layer includes 6 candidate bottleneck blocks with different kernel size { 3, 5, 7 } and expansion ratio { 3, 6 } .
Revised: There are a total of 21 searchable layers, and each layer includes 6 candidate bottleneck blocks with different kernel size { 3, 5, 7 } and expansion ratio { 3, 6 } .

Original: Besides, the network depth can also be scaled depending on whether a skip-connect operation is selected for each layer.
Revised: Besides, the network depth can also be scaled depending on whether a skip-connect operation is selected for each layer.

Original: NAS-Bench-201 Search Space.
Revised: NAS-Bench-201 Search Space.

Original: It is also the cell-based search space, but it only needs to search for the normal cell and maintain the reduction cell as a residual block with a stride of two.
Revised: It is also the cell-based search space, but it only needs to search for the normal cell and maintain the reduction cell as a residual block with a stride of two.

Original: There are 4 nodes in the normal cell, resulting in 6 searchable edges, each of which contains 5 candidate operations, including { nor-conv-1 × 1, nor-conv-3 × 3, avg-pool-3 × 3, identity and none } .
Revised: There are 4 nodes in the normal cell, resulting in 6 searchable edges, each of which contains 5 candidate operations, including { nor-conv-1 × 1, nor-conv-3 × 3, avg-pool-3 × 3, identity and none } .

Original: S1-S4.
Revised: S1-S4.

Original: The search space is the reduced original DARTS search space.
Revised: The search space is the reduced original DARTS search space.

Original: Specifically, S1 is a preoptimized space with two different candidate operations on each edge.
Revised: Specifically, S1 is a preoptimized space with two different candidate operations on each edge.

Original: S2 has the candidate operations of { sep-conv-3 × 3, identity } per edge.
Revised: S2 has the candidate operations of { sep-conv-3 × 3, identity } per edge.

Original: S3 has the candidate operations of { sep-conv-3 × 3, identity, none } per edge.
Revised: S3 has the candidate operations of { sep-conv-3 × 3, identity, none } per edge.

Original: S4 has the candidate operations of { sep-conv-3 × 3, noise } per edge.
Revised: S4 has the candidate operations of { sep-conv-3 × 3, noise } per edge.

Revised: NAS-Bench-1Shot1.

Revised: The Benchmark splits NAS-Bench-101 into search space 1, search spaceand search space 3.

Revised: The three search spaces contain 6240, 29160, 363648 architectures respectively.

Revised: Each architecture is consisted of three stacked blocks with max-pooling in between.

Revised: Each block contains three searchable cells, and each of cell includes nine nodes.

Original:  A. I MPLEMENTATION D ETAILS
Revised: [Table] A. I MPLEMENTATION D ETAILS

Original: To train the super-network together with Lite-Transformer module on CIFAR-10 in DARTS search space, we use the SGD optimizer with initial learning rate 0.025, momentum 0.9 and weight decay 3 × 10 − 4 .
Revised: To train the super-network together with Lite-Transformer module on CIFAR-10 in DARTS search space, we use the SGD optimizer with initial learning rate 0.025, momentum 0.9 and weight decay 3 × 10 − 4 .

Original: The super-network is trained for 50 epochs with the batch size of 64.
Revised: The super-network is trained for 50 epochs with the batch size of 64.

Original: After deriving the final architecture, the target network consisted of 20 cells with initial channel size of 36 is trained on the whole training dataset from scratch.
Revised: After deriving the final architecture, the target network consisted of 20 cells with initial channel size of 36 is trained on the whole training dataset from scratch.

Original: Specifically, we employ the SGD optimizer with initial [Table] [Table] and weight 
Revised: Specifically, we employ the SGD optimizer with initial learning rate 0.025, momentum 0.9 and weight decay 3 × 10 − 4 .

Original: [Table] 3 × [Table] The target network is trained for 600 epochs with the batch size of 96.
Revised: The target network is trained for 600 epochs with the batch size of 96.

Original: To train the super-network together with Lite-Transformer module on ImageNet in ProxylessNAS search space [Table] we uniform sample single path from the super-network per step.
Revised: To train the super-network together with Lite-Transformer module on ImageNet in ProxylessNAS search space, we uniform sample single path from the super-network per step.

Original: Specifically, we use
Revised: Specifically, we use

Original: Tesla V100 GPUs and train it for 240 epochs with a total batch size of 512.
Revised: Tesla V100 GPUs and train it for 240 epochs with a total batch size of 512.

Original: The SGD optimizer with initial learning rate 0.25, momentum 0.9 and weight decay 4 × 10 − 5 , and the minimum learning rate is 5 × 10 − 4 .
Revised: The SGD optimizer with initial learning rate 0.25, momentum 0.9 and weight decay 4 × 10 − 5 , and the minimum learning rate is 5 × 10 − 4 .

Original: The derived final network is retrained from scratch on the whole ImageNet training dataset for 240 epochs with the batch size of 1024 on 8 Tesla V100 GPUs.
Revised: The derived final network is retrained from scratch on the whole ImageNet training dataset for 240 epochs with the batch size of 1024 on 8 Tesla V100 GPUs.

Original: The SGD optimizer with initial learning rate 0.5, momentum 0.9 and weight decay 4 × 10 − 5 .
Revised: The SGD optimizer with initial learning rate 0.5, momentum 0.9 and weight decay 4 × 10 − 5 .

Original: A. R ESULTS IN NAS-B ENCH - 
Revised: A. R ESULTS IN NAS-B ENCH -

Original: NAS-Bench-201 is a benchmark for almost up-to-date NAS algorithms, and the diagnostic information about accuracy, loss, and parameters is accessible on three datasets including CIFAR-10,
Revised: NAS-Bench-201 is a benchmark for almost up-to-date NAS algorithms, and the diagnostic information about accuracy, loss, and parameters is accessible on three datasets including CIFAR-10,

Original: CIFAR-100, and ImageNet-16-120, respectively.
Revised: CIFAR-100, and ImageNet-16-120, respectively.

Original: We experiment with the search process on CIFAR10 and then index the accuracy on three different datasets.
Revised: We experiment with the search process on CIFAR10 and then index the accuracy on three different datasets.

Original: We keep the hyper-parameters the same as DARTS and repeat the experiments six times with different random seeds.
Revised: We keep the hyper-parameters the same as DARTS and repeat the experiments six times with different random seeds.

Original: The results in Tableshow that our approach achieves  the  bestresults on CIFAR-10 and ImageNet datasets, and also the similar performancewith GDAS on CIFAR-100. 
Revised: The results in Table 5 show that DrNAS has indeed achieved state-of-the-art results that are almost close to the global optimum. Our approach achieves comparable performance with the second best on almost datasets except obtaining the third performance on CIFAR-100. A. R ESULTS IN NAS-B ENCH -1S HOT We implement IT-NAS separately in three search spaces of NAS-Bench-1Shot1.

Revised: The hyperparameters are kept the same as DARTS.

Original: But considering that GDAS requires more epochs to anneal down the temperature coefficient, the search phase ofGDAS is not as efficient as IT-NAS. 
Revised: All methods, including DARTS, GDAS, PC-DARTS, ENAS, as well as IT-NAS are independently searched three times with different seeds for fair comparison. As shown in Figure 5, IT-NAS achieves almost the lowest test regret in different spaces with a smaller variance after convergence, indicating the effectiveness and robustness of IT-NAS. [Figure]

Revised: Methods

Revised: [Table]

Revised: A.

Revised: R ESULTS IN P ROXYLESS NAS ( WITH SE

Revised: MODULE ) SEARCH SPACE

Revised: We compare the searched results on ProxylessNAS search space that includes Squeeze-Excitation (SE) module.

Revised: The experimental settings of retraining the search architecture are the same as DNA

Revised: Li et al.

Revised: Comparing the results in Table 3, the SE module can indeed promote achieving impressive results under the mobile setting.

Revised: As shown in Table 6, IT-NAS achieves the best performance when compared with other methods with SE module, demonstrating the superiority of

Revised: IT-NAS.

Revised: A. R ANKING C ONSISTENCY A NALYSIS

Revised: Instead of using the discretization accuracy at convergence that proposed in DARTS-PT Wang et al.

Revised: (2021a) to represent the accuracy ranking of an operation, here we leverage the best acc definition proposed in Zero-Cost-PT (Xiang et al., 2021).

Revised: [Equation]

Revised: where V ∗ denotes validation accuracy of a network after full training, A | ε | denotes all possible fully-discretized subnetworks, t is the discretization iteration of an edge e with operation o .

Revised: We plot the relationship between best accuracy ranking and indicator ranking of IT-NAS and

Revised: DARTS in NAS-Bench-201 on CIFAR-10 dataset.

Revised: The indicator can be self-attention weights of ITNAS or architecture parameters of DARTS.

Revised: As shown in Figure 6, we can see that IT-NAS achieves almost positive ranking consistency except on the Edge 2( Even so, IT-NAS also obtains a better ranking than DARTS on Edge 2.)

Revised: Besides, the Kendall tau ranking is higher than DARTS on all edges, demonstrating the effectiveness of self-attention weight in evaluating the importance of candidate operations.

Revised: To compare the ranking results with Zero-Cost-PT that leverage Zero-Cost proxy to predict the performance of candidation operation, we choose to combine Zero-Cost-PT with NASWOT and

Revised: SynFlow proxy to predict the score of each operation per edge.

Revised: The results are plotted in Figure

Revised: 7 and Figure 8.

Revised: The results show that SynFlow obtains generally better accuracy ranking than

Revised: NASWOT when combining with Zero-Cost-PT, but IT-NAS still achieves better ranking on most edges than Zero-PT-NASWOT and Zero-PT-SynFlow.

Revised: Though zero-cost proxies are fast in assessing the importance of operations, its accuracy is inferior to the self-attention mechanism.

Revised: A. M ORE E MPIRICAL S TUDIES ON A CCURACY R ANKING

Revised: Based on A.7, we analyze the self-attention weights or architecture parameters in indicating the best accuracy ranking.

Revised: We plot the accuracy and indicator of all six edges on NAS-Bench-201 in

Revised: Figure 9-Figure 14.

Revised: Taking the first edge as an example, the maximum self-attention weight and the highest accuracy of IT-NAS are both on nor conv 3x3 operation, demonstrating the effectiveness of self-attention weight in evaluating the importance of candidate operations.

Revised: Whereas, the largest architecture parameter of DARTS is skip connect, which does not contribute to the highest accuracy, so that the final derived architecture according to alpha would not obtain the satisfactory accuracy.

Revised: On the other hand, the skip connect weights are always the highest across all edges, showing that the bi-level optimization of DARTS is prone to performance collapse.

Revised: The search results of IT-NAS are more stable due to the mono-level optimization.

Original: A. V ISUALIZATION
Revised: A. V ISUALIZATION

Original: Here we visualize the searched normal cells and reduction cells from three different experiments with different initial random seeds in DARTS search space.
Revised: Here we visualize the searched normal cells and reduction cells from three different experiments with different initial random seeds in DARTS search space.

Original: As shown in Figure 5, the test error and parameters of network based on (a)(b) are 2.39% and 3.54M respectively; the test error and parameters of network based on (c)(d) are 2.39% and 4.21M respectively; the test error and parameters of network based on (e)(f) are 2.44% and 3.27M respectively.
Revised: As shown in Figure 15, the test error and parameters of network based on (a)(b) are 2.39% and 3.54M respectively; the test error and parameters of network based on (c)(d) are 2.39% and 4.21M respectively; the test error and parameters of network based on (e)(f) are 2.44% and 3.27M respectively.

Original: Moreover, as shown in Figure 6, we also visualize the searched chain-style architecture in the ProxylessNAS search space on ImageNet directly.
Revised: Moreover, as shown in Figure 16, we also visualize the searched chain-style architecture in the ProxylessNAS search space on ImageNet directly.

Original: Figure 7, Figure 8, Figure 9 separately shows the searched cells on CIFAR-10, CIFAR-100, and
Revised: Figure 17, Figure 18, Figure 19 separately shows the searched cells on CIFAR-10, CIFAR-100, and

Original: SVHN in the reduced search spaces S1-S4, respectively.
Revised: SVHN in the reduced search spaces S1-S4, respectively.

Original: The corresponding accuracy of each cell can be seen in the Table 4.
Revised: The corresponding accuracy of each cell can be seen in the Table [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: A cc u r a c y

Revised: R a n k i n g

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Figure] [Figure]

Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: l f - a tt e n t i o n

Original: [Figure]
Revised: [Figure]

Revised: i

Original: [Table]
Revised: [Figure] T e [Figure] t

Original: [Figure]
Revised: [Figure]

Revised: cc

Original: [Figure]
Revised: [Figure]

Original: _3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2} sep_conv_3x 1 skip_connect sep_conv_3x c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x sep_conv_3x
Revised: r a c y Accuracy [Figure] [Figure]

Revised: [Figure]

Original: c_{k} sep_conv_3x (b) S c_{k-2} sep_conv_3x 1 sep_conv_3x3skip_connectsep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x
Revised: candidate operations both on the 6th edge of the cell in NAS-Bench-201. [Figure] [Table] [Figure]

Original: [-sep_conv_3x c_{k} c_{k-2} 
-]
Revised: [-sep_conv_3x c_{k} c_{k-2} 
-][Figure] [Table]

Original:  sep_conv_3x 1 skip_connect  c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x3 sep_conv_3x
Revised: [Figure] [Figure] _3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2} sep_conv_3x 1 skip_connect sep_conv_3x c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x sep_conv_3x

Original: c_{k} {+sep_conv_3x (b) S c_{k-2} sep_conv_3x 1 sep_conv_3x3skip_connectsep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2} 
 sep_conv_3x 1 skip_connect c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x 3 sep_conv_3x c_{k}+} sep_conv_3x3sep_conv_3x (c) S c_{k-2} sep_conv_3x 1 sep_conv_3x3sep_conv_3x3sep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2} 

Revised: c_{k} {+sep_conv_3x (b) S c_{k-2} sep_conv_3x 1 sep_conv_3x3skip_connectsep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2} 
 sep_conv_3x 1 skip_connect c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x 3 sep_conv_3x c_{k}+} sep_conv_3x3sep_conv_3x (c) S c_{k-2} sep_conv_3x 1 sep_conv_3x3sep_conv_3x3sep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2}

Original: sep_conv_3x 1 sep_conv_3x c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x 3 sep_conv_3x
Revised: sep_conv_3x 1 sep_conv_3x c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x 3 sep_conv_3x

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: _connect skip_connect sep_conv_3x c_{k} skip_connect
Revised: _connect skip_connect sep_conv_3x c_{k} skip_connect

Original: [Figure]
Revised: [Figure]

Original: c_{k-2} sep_conv_3x 1 sep_conv_3x3sep_conv_3x3sep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2} 

Revised: c_{k-2} sep_conv_3x 1 sep_conv_3x3sep_conv_3x3sep_conv_3x c_{k-1} sep_conv_3x sep_conv_3x3 sep_conv_3x sep_conv_3x c_{k} c_{k-2}

Original: sep_conv_3x  sep_conv_3x  noise c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x3 sep_conv_3x c_{k} sep_conv_3x 
Revised: sep_conv_3x  sep_conv_3x  noise c_{k-1} sep_conv_3x3 sep_conv_3x 2 sep_conv_3x3 sep_conv_3x c_{k} sep_conv_3x (

Original: [Figure]
Revised: [Figure]