Original: GraViT-E: Gradient-based Vision Transformer Search with Entangled Weights
Revised: GraViT-E: Gradient-based Vision Transformer Search with Entangled Weights

Original: Abstract
Revised: Abstract

Original: Differentiable one-shot neural architecture search methods have recently become popular since they can exploit weight-sharing to efficiently search in large architectural search spaces.
Revised: Differentiable one-shot neural architecture search methods have recently become popular since they can exploit weight-sharing to efficiently search in large architectural search spaces.

Original: These methods traditionally perform a continuous relaxation of the discrete search space to search for an optimal architecture.
Revised: These methods traditionally perform a continuous relaxation of the discrete search space to search for an optimal architecture.

Original: However, they suffer from large memory requirements, making their application to parameter-heavy architectures like transformers difficult.
Revised: However, they suffer from large memory requirements, making their application to parameter-heavy architectures like transformers difficult.

Original: Recently, single-path one-shot methods have been introduced which often use weight entanglement to alleviate this issue by sampling the weights of the sub-networks from the largest model, which is itself the supernet.
Revised: Recently, single-path one-shot methods have been introduced which often use weight entanglement to alleviate this issue by sampling the weights of the sub-networks from the largest model, which is itself the supernet.

Original: In this work, we propose a continuous relaxation of weight entanglement-based architectural representation.
Revised: In this work, we propose a continuous relaxation of weight entanglement-based architectural representation.

Original: Our Gradient-based Vision Transformer Search with Entangled Weights (GraViT-E) combines the best properties of both differentiable one-shot NAS and weight entanglement.
Revised: Our Gradient-based Vision Transformer Search with Entangled Weights (GraViT-E) combines the best properties of both differentiable one-shot NAS and weight entanglement.

Original: We observe that our method imparts much better regularization properties and memory efficiency to the trained supernet.
Revised: We observe that our method imparts much better regularization properties and memory efficiency to the trained supernet.

Original: We study three one-shot optimizers on the Vision Transformer search space and observe that our method outperforms existing baselines on multiple datasets while being upto 35% more parameter efficient on ImageNet-1k.
Revised: We study three one-shot optimizers on the Vision Transformer search space and observe that our method outperforms existing baselines on multiple datasets while being upto 35% more parameter efficient on ImageNet-1k.

Original:  Introduction
Revised:  Introduction

Original: One-shot neural architecture search (NAS) methods [18, 27, 4, 8] have recently gained a lot of popularity across multiple computer vision domains like classification [18], semantic segmentation [16, 21] and super resolution [5, 23].
Revised: One-shot neural architecture search (NAS) methods [18, 27, 4, 8] have recently gained a lot of popularity across multiple computer vision domains like classification [18], semantic segmentation [16, 21] and super resolution [5, 23].

Original: Most of these methods adopt the weight sharing paradigm [24] for efficient search.
Revised: Most of these methods adopt the weight sharing paradigm [24] for efficient search.

Original: Differentiable NAS methods based on weight-sharing have often been criticized for their large memory consumption  with the size of the required working memory increasing linearly with the number of operator choices.
Revised: Differentiable NAS methods based on weight-sharing have often been criticized for their large memory consumption , with the size of the required working memory increasing linearly with the number of operator choices.

Original: Hence these methods often (have to) perform search on smaller network proxies.
Revised: Hence these methods often (have to) perform search on smaller network proxies.

Original: One-shot NAS methods, such as SPOS [11], AutoFormer [3] and
Revised: One-shot NAS methods, such as SPOS [11], AutoFormer [3] and

Original: OFA [2, 22] alleviate this problem by using weight-entanglement which further enforces sharing of network weights between the operator choices.
Revised: OFA [2, 22] alleviate this problem by using weight-entanglement which further enforces sharing of network weights between the operator choices.

Original: These methods first train the supernet by sampling a subnetwork and updating its weights in every epoch.
Revised: These methods first train the supernet by sampling a subnetwork and updating its weights in every epoch.

Original: They then use black-box optimizers to select the best architecture from the trained supernet.
Revised: They then use black-box optimizers to select the best architecture from the trained supernet.

Original: The black-box search is based upon the surrogate performance of the architectures inherited from the supernet which could potentially have a poor rank correlation with the actual subnetwork performance, hence biasing the search.
Revised: The black-box search is based upon the surrogate performance of the architectures inherited from the supernet which could potentially have a poor rank correlation with the actual subnetwork performance, hence biasing the search.

Original: Motivated by this observation, we combine the continuous relaxation of the architecture space, which is widely studied for differentiable architecture search [19, 8, 4] with weight-entanglement from AutoFormer [3] to directly search for optimal architectures in the transformer search space in a differentiable manner.
Revised: Motivated by this observation, we combine the continuous relaxation of the architecture space, which is widely studied for differentiable architecture search [19, 8, 4] with weight-entanglement from AutoFormer [3] to directly search for optimal architectures in the transformer search space in a differentiable manner.

Original: We observe that our method (GraViT-E) obtains architectures which outperform the baseline models in terms of accuracy while maintaining parameter and FLOP efficiency.
Revised: We observe that our method (GraViT-E) obtains architectures which outperform the baseline models in terms of accuracy while maintaining parameter and FLOP efficiency.

Original: Further our proposed method can be integrated with any off-the-shelf one-shot optimizer and is general enough to be adapted to any new search space design.
Revised: Further our proposed method can be integrated with any off-the-shelf one-shot optimizer and is general enough to be adapted to any new search space design.

Original: Our Contributions.
Revised: Our Contributions.

Original: We outline our main contributions below:
Revised: We outline our main contributions below:

Original: • We propose a search space for Vision Transformers [9] that combines weight-entanglement with the continuous relaxation of the architectural space as seen in gradient-based one-shot NAS methods.
Revised: • We propose a search space for Vision Transformers [9] that combines weight-entanglement with the continuous relaxation of the architectural space as seen in gradient-based one-shot NAS methods.

Original: • We study popular one-shot optimizers like DARTS [17], DrNAS [4] and GDAS [8] applied in conjunction with weight-entanglement.
Revised: • We study popular one-shot optimizers like DARTS [17], DrNAS [4] and GDAS [8] applied in conjunction with weight-entanglement.

Original: Our method Pareto-dominates the AutoFormer baseline models on CIFAR10, CIFAR100 and ImageNet-1k while improving the accuracy by 0.13% and 2.79 % and 1.5% respectively.
Revised: Our method Pareto-dominates the AutoFormer baseline models on CIFAR10, CIFAR100 and ImageNet-1k while improving the accuracy by 0.13% and 2.79 % and 1.5% respectively.

Original:  Related Work
Revised:  Related Work

Original: Neural Architecture Search [10] aims at automating the design of network architectures.
Revised: Neural Architecture Search [10] aims at automating the design of network architectures.

Original: ENAS [20] was the first method to apply weight-sharing , which forced all the architectures in the search space to share the parameters by defining a computational super-graph, the sub-graphs of which represented individual models.
Revised: ENAS [20] was the first method to apply weight-sharing , which forced all the architectures in the search space to share the parameters by defining a computational super-graph, the sub-graphs of which represented individual models.

Original: Using this weight-sharing, ENAS reduced the computational cost of NAS by 1000 × compared to previous methods.
Revised: Using this weight-sharing, ENAS reduced the computational cost of NAS by 1000 × compared to previous methods.

Original: DARTS [19] combined one-shot architecture search with a gradient-based approach.
Revised: DARTS [19] combined one-shot architecture search with a gradient-based approach.

Original: It formulated the search as a bi-level optimization problem that trained the parameters of the super-net on the training set in the inner loop while training the architectural parameters of the super-net on the validation set in the outer loop.
Revised: It formulated the search as a bi-level optimization problem that trained the parameters of the super-net on the training set in the inner loop while training the architectural parameters of the super-net on the validation set in the outer loop.

Original: These architectural parameters are later used to discretize the super-net to get the most promising model.
Revised: These architectural parameters are later used to discretize the super-net to get the most promising model.

Original: DARTS, however, had prominent failure modes , as outlined in
Revised: DARTS, however, had prominent failure modes , as outlined in

Original: Robust-DARTS [27].
Revised: Robust-DARTS [27].

Original: Several other gradient-based one-shot optimization techniques have since been developed to improve upon DARTS, such as GDAS [8] and DrNAS [4].
Revised: Several other gradient-based one-shot optimization techniques have since been developed to improve upon DARTS, such as GDAS [8] and DrNAS [4].

Original: GDAS samples one path through the entire super-graph in every training step using differentiable Gumbel-softmax sampling .
Revised: GDAS samples one path through the entire super-graph in every training step using differentiable Gumbel-softmax sampling .

Original: DrNAS, on the other hand, re-frames the one-shot search as learning a Dirichlet distribution.
Revised: DrNAS, on the other hand, re-frames the one-shot search as learning a Dirichlet distribution.

Original: Lastly, our work is closely related to AutoFormer [3], which introduced weight-entanglement as a means of restricting memory consumption.
Revised: Lastly, our work is closely related to AutoFormer [3], which introduced weight-entanglement as a means of restricting memory consumption.

Original: Our approach performs a continuous relaxation of the weight entanglement search space by adding architectural parameters to it which allows the use of gradient-based one-shot optimizers.
Revised: Our approach performs a continuous relaxation of the weight entanglement search space by adding architectural parameters to it which allows the use of gradient-based one-shot optimizers.

Original:  Methodology
Revised:  Methodology

Original: Weight Sharing and Entanglement
Revised: Weight Sharing and Entanglement

Original: Weight Sharing.
Revised: Weight Sharing.

Original: We first review weight-sharing [24], which is used in many one-shot optimization techniques.
Revised: We first review weight-sharing [24], which is used in many one-shot optimization techniques.

Original: The search space in weight-sharing NAS methods consists of a single supernet , each edge of which holds several different operators , such as convolutional or pooling layers.
Revised: The search space in weight-sharing NAS methods consists of a single supernet , each edge of which holds several different operators , such as convolutional or pooling layers.

Original: Each of these operators has an associated weight matrix, and these weight matrices are shared: all (exponentially many) architectures that include a certain operator use the same shared weight matrix.
Revised: Each of these operators has an associated weight matrix, and these weight matrices are shared: all (exponentially many) architectures that include a certain operator use the same shared weight matrix.

Original: That is, by updating one path of the supernet, exponentially many other paths are also (partially) updated.
Revised: That is, by updating one path of the supernet, exponentially many other paths are also (partially) updated.

Original: Weight-sharing comes with a set of advantages and disadvantages.
Revised: Weight-sharing comes with a set of advantages and disadvantages.

Original: On one hand, it allows a large number of models to be trained and searched at the expense of training a single supernet [19].
Revised: On one hand, it allows a large number of models to be trained and searched at the expense of training a single supernet [19].

Original: This has drastically increased the speed and efficiency of NAS [1].
Revised: This has drastically increased the speed and efficiency of NAS [1].

Original: On the other hand, the memory requirements for training the supernet using differentiable optimizers like DARTS with weight sharing increases linearly with the number of choice of operators on each edge, making it infeasible to train very large models.
Revised: On the other hand, the memory requirements for training the supernet using differentiable optimizers like DARTS with weight sharing increases linearly with the number of choice of operators on each edge, making it infeasible to train very large models.

Original: Weight Entanglement.
Revised: Weight Entanglement.

Original: While weight-sharing shares the operator parameters between different architectures, weight-entanglement goes one step further and shares the parameters between the different operators on a given edge of the supernet.
Revised: While weight-sharing shares the operator parameters between different architectures, weight-entanglement goes one step further and shares the parameters between the different operators on a given edge of the supernet.

Original: E.g., the parameters of a smaller convolutional kernel are sampled from the largest convolutional kernel on the same edge.
Revised: E.g., the parameters of a smaller convolutional kernel are sampled from the largest convolutional kernel on the same edge.

Original: The weights are hence shared between different sets of operators as well as different architectures sampled from the supernet.
Revised: The weights are hence shared between different sets of operators as well as different architectures sampled from the supernet.

Original: Weight-entanglement has several advantages.
Revised: Weight-entanglement has several advantages.

Original: Primarily, the memory requirement for the entire supernet is exactly the same as for the largest architecture in the search space, making it feasible to
Revised: Primarily, the memory requirement for the entire supernet is exactly the same as for the largest architecture in the search space, making it feasible to

Original: [Figure]
Revised: [Figure]

Original: explore much larger architectures than what standard weight-sharing permits.
Revised: explore much larger architectures than what standard weight-sharing permits.

Original: Weight-entanglement also imparts the supernet with several of the beneficial properties of the once-for-all networks [2, 22].
Revised: Weight-entanglement also imparts the supernet with several of the beneficial properties of the once-for-all networks [2, 22].

Original: 3.2 Our Approach
Revised: 3.2 Our Approach

Original: Our approach combines weight-entanglement with gradient-based one-shot NAS.
Revised: Our approach combines weight-entanglement with gradient-based one-shot NAS.

Original: The operators on any given edge reuse the parameters of the largest operator on the edge, as in weight-entanglement, but we also learn architectural parameters for each of these operators, as in gradient-based one-shot
Revised: The operators on any given edge reuse the parameters of the largest operator on the edge, as in weight-entanglement, but we also learn architectural parameters for each of these operators, as in gradient-based one-shot

Original: NAS.
Revised: NAS.

Original: This allows us to apply one-shot architectural optimizers such as DARTS, GDAS and DrNAS to search spaces that leverage weight-entanglement (in our experiments, the AutoFormer search space [3]).
Revised: This allows us to apply one-shot architectural optimizers such as DARTS, GDAS and DrNAS to search spaces that leverage weight-entanglement (in our experiments, the AutoFormer search space [3]).

Original: See Figure 1 for an overview.
Revised: See Figure 1 for an overview.

Original: Unlike in AutoFormer, we can now acquire the optimal architecture directly by discretizing the supernet using the learned architectural parameters.
Revised: Unlike in AutoFormer, we can now acquire the optimal architecture directly by discretizing the supernet using the learned architectural parameters.

Original: Search Space.
Revised: Search Space.

Original: Our experiments were run on the AutoFormer search space [3].
Revised: Our experiments were run on the AutoFormer search space [3].

Original: This is a transformerbased search space that varies five factors in its building blocks: embedding dimension α e (dimension of the patch embeddings which are fed into the transformer encoders) , Q-K-V dimension (dimensions of the query, key and value vectors of the Multi-head Self Attention layer), number of heads in the
Revised: This is a transformerbased search space that varies five factors in its building blocks: embedding dimension α e (dimension of the patch embeddings which are fed into the transformer encoders) , Q-K-V dimension (dimensions of the query, key and value vectors of the Multi-head Self Attention layer), number of heads in the

Original: MSA α h , MLP ratio α r (the ratio of hidden dimension to the embedding dimension in the multi-layer perceptron), and network depth α n (number of transformer encoder blocks stacked together) .
Revised: MSA α h , MLP ratio α r (the ratio of hidden dimension to the embedding dimension in the multi-layer perceptron), and network depth α n (number of transformer encoder blocks stacked together) .

Original: The
Revised: The

Original: AutoFormer search space also fixes the ratio of Q-K-V dimension to the number of heads hence it is not a part of the search space.
Revised: AutoFormer search space also fixes the ratio of Q-K-V dimension to the number of heads hence it is not a part of the search space.

Original: Combination Mixture Operation For each of the building blocks of the vision transformer
Revised: Combination Mixture Operation For each of the building blocks of the vision transformer

Original: , we single out one or more of the above factors it depends on.
Revised: , we single out one or more of the above factors it depends on.

Original: The main difference we observe from traditional gradient based optimization is that some operations depend on two or more α ′ s which we call combination mixop .
Revised: The main difference we observe from traditional gradient based optimization is that some operations depend on two or more α ′ s which we call combination mixop .

Original: Consider the Multiheaded Self Attention (MSA) block, which depends upon the embedding dimension α e and number of heads α h .
Revised: Consider the Multiheaded Self Attention (MSA) block, which depends upon the embedding dimension α e and number of heads α h .

Original: We define the architectural weights for this operation as the combination of these two choices, i.e., α e,h .
Revised: We define the architectural weights for this operation as the combination of these two choices, i.e., α e,h .

Original: We refer the reader to Appendix A.3 for further details on the combination mixop .
Revised: We refer the reader to Appendix A.3 for further details on the combination mixop .

Original:  Experiments and Results
Revised:  Experiments and Results

Original: Image Classification on CIFAR10 and CIFAR100.
Revised: Image Classification on CIFAR10 and CIFAR100.

Original: As our first experiment, we study three gradient-based one-shot optimizers (DARTS, GDAS and DrNAS) on the AutoFormer search space.
Revised: As our first experiment, we study three gradient-based one-shot optimizers (DARTS, GDAS and DrNAS) on the AutoFormer search space.

Original: For each of these optimizers, we first train the supernet with the optimizer for search and then obtain the final architecture by discretizing the supernet; this architecture is then evaluated by finetuning as well as by retraining from scratch.
Revised: For each of these optimizers, we first train the supernet with the optimizer for search and then obtain the final architecture by discretizing the supernet; this architecture is then evaluated by finetuning as well as by retraining from scratch.

Original: The search and training is performed directly for CIFAR-10 and
Revised: The search and training is performed directly for CIFAR-10 and

Original: CIFAR-100 [15] are shown in Table 1 and Table 2.
Revised: CIFAR-100 [15] are shown in Table 1 and Table 2.

Original: [Table]
Revised: [Table]

Original: Table 1 shows that the models from our methods Pareto-dominate the models found by AutoFormer, performing especially well in terms of model size and FLOPS.
Revised: Table 1 shows that the models from our methods Pareto-dominate the models found by AutoFormer, performing especially well in terms of model size and FLOPS.

Original: Interestingly, as Table 2 in Appendix B, our finetuned models are better than training from scratch indicating that our method imparts some regularization properties to the supernet, which require further investigation.
Revised: Interestingly, as Table 2 in Appendix B, our finetuned models are better than training from scratch indicating that our method imparts some regularization properties to the supernet, which require further investigation.

Original: Further, as observed in Table 1 the models discovered by our method with the DARTS, DrNAS and
Revised: Further, as observed in Table 1 the models discovered by our method with the DARTS, DrNAS and

Original: GDAS optimizers achieve a better trade-off between error-rate, number of parameters and FLOPS than the Autoformer.
Revised: GDAS optimizers achieve a better trade-off between error-rate, number of parameters and FLOPS than the Autoformer.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Ours + DrNAS 98.
Revised: Ours + DrNAS 98.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Ours + GDAS 98.
Revised: Ours + GDAS 98.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For our second experiment, we search for the best archi- on ImageNet-1k tecture on CIFAR10 and CIFAR100 ( img size = 224, patch size = 16)
Revised: For our second experiment, we search for the best archi- on ImageNet-1k tecture on CIFAR10 and CIFAR100 ( img size = 224, patch size = 16)

Original: Table 3 in Appendix C.
Revised: Table 3 in Appendix C.

Original: We choose the best model that we obtained from across all the optimizers on CIFAR-100 (the model obtained using GDAS, in this case), and transfer its weights to a model which is then fine-tuned to perform classification on ImageNet-1k [7].
Revised: We choose the best model that we obtained from across all the optimizers on CIFAR-100 (the model obtained using GDAS, in this case), and transfer its weights to a model which is then fine-tuned to perform classification on ImageNet-1k [7].

Original: We achieve an accuracy of 79 .
Revised: We achieve an accuracy of 79 .

Original: 34% on ImageNet-1k, compared to AutoFormer which achieves an accuracy of 74 .
Revised: 34% on ImageNet-1k, compared to AutoFormer which achieves an accuracy of 74 .

Original: Further, as noted, from Figure 2, we obtain improvements of up to 1.5% on ImageNet-1k compared to
Revised: Further, as noted, from Figure 2, we obtain improvements of up to 1.5% on ImageNet-1k compared to

Original: AutoFormers of comparable model sizes.
Revised: AutoFormers of comparable model sizes.

Original:  Conclusion
Revised:  Conclusion

Original: We present a new framework GraViT-E to efficiently search for the architectural parameters of large models like transformers.
Revised: We present a new framework GraViT-E to efficiently search for the architectural parameters of large models like transformers.

Original: Our method outperforms the AutoFormer search strategy on multiple datasets in terms of performance and parameter-efficiency.
Revised: Our method outperforms the AutoFormer search strategy on multiple datasets in terms of performance and parameter-efficiency.

Original: In the future, we plan to extend our work to support multiple search spaces (e.g. different transformer types [12] and convolutional networks [14, 13]) and multiple one-shot optimizers (e.g., PC-DARTS [26], SNAS [25], FairNAS [6]).
Revised: In the future, we plan to extend our work to support multiple search spaces (e.g. different transformer types [12] and convolutional networks [14, 13]) and multiple one-shot optimizers (e.g., PC-DARTS [26], SNAS [25], FairNAS [6]).

Original: While our work initiates the study of continuous relaxation of the transformer search space for gradient-based one-shot optimizers, we still face some limitations.
Revised: While our work initiates the study of continuous relaxation of the transformer search space for gradient-based one-shot optimizers, we still face some limitations.

Original: For now, our observations are based upon the study of only a lightweight Vision Transformer search space.
Revised: For now, our observations are based upon the study of only a lightweight Vision Transformer search space.

Original: Further we do not study different transformer applications like language modeling, semantic image segmentation, etc.
Revised: Further we do not study different transformer applications like language modeling, semantic image segmentation, etc.

Original: How the weight entanglement scheme hampers or benefits the gradient based search also warrants further investigation.
Revised: How the weight entanglement scheme hampers or benefits the gradient based search also warrants further investigation.

Original: However, since our formulation can take advantage of the continually-improving set of methods for gradient-based one-shot NAS, we hope that our work will lead to the first generallyapplicable robust and efficient architecture optimization method for transformers.
Revised: However, since our formulation can take advantage of the continually-improving set of methods for gradient-based one-shot NAS, we hope that our work will lead to the first generallyapplicable robust and efficient architecture optimization method for transformers.

Original: Acknowledgments
Revised: Acknowledgments

Original: This research was partially supported by the following sources:
Revised: This research was partially supported by the following sources:

Original: TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the German Federal Ministry of Education and Research (BMBF, grant RenormalizedFlows 01IS19077C); the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828; the European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765).
Revised: TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the German Federal Ministry of Education and Research (BMBF, grant RenormalizedFlows 01IS19077C); the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828; the European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765).

Original: Robert Bosch GmbH is acknowledged for financial support.
Revised: Robert Bosch GmbH is acknowledged for financial support.

Original: Funded by the European Union.
Revised: Funded by the European Union.

Original: Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC.
Revised: Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC.

Original: Neither the European Union nor the
Revised: Neither the European Union nor the

Original: ERC can be held responsible for them
Revised: ERC can be held responsible for them

Original: A Appendix A.1 Pareto Fronts
Revised: A Appendix A.1 Pareto Fronts

Original: In Figure 3 we present the parameter-size and error Pareto front for AutoFormer v/s our models with different one-shot optimizers.
Revised: In Figure 3 we present the parameter-size and error Pareto front for AutoFormer v/s our models with different one-shot optimizers.

Original: We observe that GDAS pareto dominates other models for CIFAR and GDAS and DrNAS lie on the Pareto front for CIFAR
Revised: We observe that GDAS pareto dominates other models for CIFAR and GDAS and DrNAS lie on the Pareto front for CIFAR

Original: [Figure]
Revised: [Figure]

Original: A.2 Experimental Details
Revised: A.2 Experimental Details

Original: We use the official source code of AutoFormer available at https://github.com/microsoft/
Revised: We use the official source code of AutoFormer available at https://github.com/microsoft/

Original: Cream/tree/main/AutoFormer for all the AutoFormer experiments on CIFAR10 and CIFAR100.
Revised: Cream/tree/main/AutoFormer for all the AutoFormer experiments on CIFAR10 and CIFAR100.

Original: We closely follow the AutoFormer training pipeline and search space design.
Revised: We closely follow the AutoFormer training pipeline and search space design.

Original: AutoFormer searched on three transformer sizes Autoformer-T (tiny), AutoFormer-S (small) and Autoformer-B (base).
Revised: AutoFormer searched on three transformer sizes Autoformer-T (tiny), AutoFormer-S (small) and Autoformer-B (base).

Original: We currently restrict ourselves to Autoformer-T.
Revised: We currently restrict ourselves to Autoformer-T.

Original: We use a 50%-50% train and validation split for the
Revised: We use a 50%-50% train and validation split for the

Original: CIFAR10 and CIFAR100 dataset for the bi-level optimization.
Revised: CIFAR10 and CIFAR100 dataset for the bi-level optimization.

Original: A.3 Optimization Details.
Revised: A.3 Optimization Details.

Original: We assign an architectural parameter α to each of these variables, α d for embedding dimension, α h for number of heads, α r for MLP-ratio and α n for the number of transformer encoder blocks to make the application of one-shot optimizers on the AutoFormer search space feasible.
Revised: We assign an architectural parameter α to each of these variables, α d for embedding dimension, α h for number of heads, α r for MLP-ratio and α n for the number of transformer encoder blocks to make the application of one-shot optimizers on the AutoFormer search space feasible.

Original: Next, for each of the building blocks of the vision transformer, we single out one or more of the above factors it depends on.
Revised: Next, for each of the building blocks of the vision transformer, we single out one or more of the above factors it depends on.

Original: We observe two categories of mixture operations, first which depend on only one of the above factors ( single mixop ) and a second one which is based upon combination of two (or more) α ′ s which we call combination mixop .
Revised: We observe two categories of mixture operations, first which depend on only one of the above factors ( single mixop ) and a second one which is based upon combination of two (or more) α ′ s which we call combination mixop .

Original: In single mixops , we use a mixture operation which is compatible with the specific one-shot optimizer that we use.
Revised: In single mixops , we use a mixture operation which is compatible with the specific one-shot optimizer that we use.

Original: For the combination mixop , let us consider the Multiheaded
Revised: For the combination mixop , let us consider the Multiheaded

Original: Self Attention (MSA) block, which depends upon the embedding dimension α e and number of heads α h .
Revised: Self Attention (MSA) block, which depends upon the embedding dimension α e and number of heads α h .

Original: We define the architecture weights for this operation as the combination of these two choices, i.e., α e,h , which is defined as below.
Revised: We define the architecture weights for this operation as the combination of these two choices, i.e., α e,h , which is defined as below.

Original: The mixture operation for MSA is then defined as in Equation (1), where O is the cross-product of the two sets of operation choices (here embedding dimension and number of heads).
Revised: The mixture operation for MSA is then defined as in Equation (1), where O is the cross-product of the two sets of operation choices (here embedding dimension and number of heads).

Original: The formulation ensures that the ranking of the possible independent choices ( α e and α h ) and the choice for the combination operation ( α e,h ) is not conflicting with each other.
Revised: The formulation ensures that the ranking of the possible independent choices ( α e and α h ) and the choice for the combination operation ( α e,h ) is not conflicting with each other.

Original: [Equation]
Revised: [Equation]

Original: B Finetuning v/s retraining
Revised: B Finetuning v/s retraining

Original: In Table 2 we compare the accuracy of the obtained models from search on CIFAR100 and CIFAR in terms of their retraining and finetuning accuracies.
Revised: In Table 2 we compare the accuracy of the obtained models from search on CIFAR100 and CIFAR in terms of their retraining and finetuning accuracies.

Original: Surprisingly inheriting the weights from the supernetwork and finetuning obtains much better accuracies compared to retraining from scratch.
Revised: Surprisingly inheriting the weights from the supernetwork and finetuning obtains much better accuracies compared to retraining from scratch.

Original: We hypothesize this to be the regularization effect imparted by the supernet training scheme and needs to be studied further.
Revised: We hypothesize this to be the regularization effect imparted by the supernet training scheme and needs to be studied further.

Original: C CIFAR10 and CIFAR100 with image size=224 and patch size= 
Revised: C CIFAR10 and CIFAR100 with image size=224 and patch size=

Original: For architectures to be transferrable to ImageNet we need to perform train and search on CIFAR and CIFAR10 with the 32 × 32 image padded to 224 × 224 and further using a patch size of 16 × 16 .
Revised: For architectures to be transferrable to ImageNet we need to perform train and search on CIFAR and CIFAR10 with the 32 × 32 image padded to 224 × 224 and further using a patch size of 16 × 16 .

Original: The results obtained on CIFAR10 and CIFAR100 on these patch sizes is as presented in Table 3.
Revised: The results obtained on CIFAR10 and CIFAR100 on these patch sizes is as presented in Table 3.

Original: We observe that similar to the 2 × 2 patch size studies in Section 4.
Revised: We observe that similar to the 2 × 2 patch size studies in Section 4.

Original: GDAS performs the best on
Revised: GDAS performs the best on

Original: CIFAR100 and its weights are then transferred to ImageNet-1k.
Revised: CIFAR100 and its weights are then transferred to ImageNet-1k.

Original: [Table]
Revised: [Table]