Original: GraViT-E: Gradient-based Vision Transformer Search with Entangled Weights
Revised: GraViT-E: Gradient-based Vision Transformer Search with Entangled Weights

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Differentiable one-shot neural architecture search methods have recently becomepopular since they can exploit weight-sharing to efficiently search in large architectural search spaces.
Revised: Differentiable one-shot neural architecture search methods have recently become popular since they can exploit weight-sharing to efficiently search in large architectural search spaces.

Original: These methods traditionally perform a continuous relaxation ofthe discrete search space to search for an optimal architecture.
Revised: These methods traditionally perform a continuous relaxation of the discrete search space to search for an optimal architecture.

Original: However, they sufferfrom large memory requirements, making their application to parameter-heavyarchitectures like transformers difficult.
Revised: However, they suffer from large memory requirements, making their application to parameter-heavy architectures like transformers difficult.

Original: Recently, single-path one-shot methodshave been introduced which often use weight entanglement to alleviate this issueby sampling the weights of the sub-networks from the largest model, which isitself the supernet.
Revised: Recently, single-path one-shot methods have been introduced which often use weight entanglement to alleviate this issue by sampling the weights of the sub-networks from the largest model, which is itself the supernet.

Original: In this work, we propose a continuous relaxation of weightentanglement-based architectural representation.
Revised: In this work, we propose a continuous relaxation of weight entanglement-based architectural representation.

Original: Our Gradient-based Vision Transformer Search with Entangled Weights (GraViT-E) combines the best propertiesof both differentiable one-shot NAS and weight entanglement.
Revised: Our Gradient-based Vision Transformer Search with Entangled Weights (GraViT-E) combines the best properties of both differentiable one-shot NAS and weight entanglement.

Original: We observe thatour method imparts much better regularization properties and memory efficiencyto the trained supernet.
Revised: We observe that our method imparts much better regularization properties and memory efficiency to the trained supernet.

Original: We study three one-shot optimizers on the Vision Transformer search space and observe that our method outperforms existing baselines onmultiple datasets while being upto 35% more parameter efficient on ImageNet-1k.
Revised: We study three one-shot optimizers on the Vision Transformer search space and observe that our method outperforms existing baselines on multiple datasets while being upto 35% more parameter efficient on ImageNet-1k.

Original:  Introduction
Revised:  Introduction

Original: One-shot neural architecture search (NAS) methods [18, 27, 4, 8] have recently gained a lot ofpopularity across multiple computer vision domains like classification [18], semantic segmentation[16, 21] and super resolution [5, 23].
Revised: One-shot neural architecture search (NAS) methods [18, 27, 4, 8] have recently gained a lot of popularity across multiple computer vision domains like classification [18], semantic segmentation [16, 21] and super resolution [5, 23].

Original: Most of these methods adopt the weight sharing paradigm [24]for efficient search.
Revised: Most of these methods adopt the weight sharing paradigm [24] for efficient search.

Original: Differentiable NAS methods based on weight-sharing have often been criticizedfor their large memory consumption, with the size of the required working memory increasinglinearly with the number of operator choices.
Revised: Differentiable NAS methods based on weight-sharing have often been criticized for their large memory consumption with the size of the required working memory increasing linearly with the number of operator choices.

Original: Hence these methods often (have to) perform searchon smaller network proxies.
Revised: Hence these methods often (have to) perform search on smaller network proxies.

Original: One-shot NAS methods, such as SPOS [11], AutoFormer [3] and
Revised: One-shot NAS methods, such as SPOS [11], AutoFormer [3] and

Original: OFA [2, 22] alleviate this problem by using weight-entanglement which further enforces sharing ofnetwork weights between the operator choices.
Revised: OFA [2, 22] alleviate this problem by using weight-entanglement which further enforces sharing of network weights between the operator choices.

Original: These methods first train the supernet by sampling asubnetwork and updating its weights in every epoch.
Revised: These methods first train the supernet by sampling a subnetwork and updating its weights in every epoch.

Original: They then use black-box optimizers to selectthe best architecture from the trained supernet.
Revised: They then use black-box optimizers to select the best architecture from the trained supernet.

Original: The black-box search is based upon the surrogateperformance of the architectures inherited from the supernet which could potentially have a poorrank correlation with the actual subnetwork performance, hence biasing the search.
Revised: The black-box search is based upon the surrogate performance of the architectures inherited from the supernet which could potentially have a poor rank correlation with the actual subnetwork performance, hence biasing the search.

Original: Motivated by thisobservation, we combine the continuous relaxation of the architecture space, which is widely studiedfor differentiable architecture search [19, 8, 4] with weight-entanglement from AutoFormer [3] todirectly search for optimal architectures in the transformer search space in a differentiable manner.
Revised: Motivated by this observation, we combine the continuous relaxation of the architecture space, which is widely studied for differentiable architecture search [19, 8, 4] with weight-entanglement from AutoFormer [3] to directly search for optimal architectures in the transformer search space in a differentiable manner.

Original: We observe that our method (GraViT-E) obtains architectures which outperform the baseline modelsin terms of accuracy while maintaining parameter and FLOP efficiency.
Revised: We observe that our method (GraViT-E) obtains architectures which outperform the baseline models in terms of accuracy while maintaining parameter and FLOP efficiency.

Original: Further our proposed methodcan be integrated with any off-the-shelf one-shot optimizer and is general enough to be adapted toany new search space design.
Revised: Further our proposed method can be integrated with any off-the-shelf one-shot optimizer and is general enough to be adapted to any new search space design.

Original: Our Contributions.
Revised: Our Contributions.

Original: We outline our main contributions below:
Revised: We outline our main contributions below:

Original: • We propose a search space for Vision Transformers [9] that combines weight-entanglementwith the continuous relaxation of the architectural space as seen in gradient-based one-shotNAS methods.•
Revised: • We propose a search space for Vision Transformers [9] that combines weight-entanglement with the continuous relaxation of the architectural space as seen in gradient-based one-shot NAS methods.

Original:  We study popular one-shot optimizers like DARTS [17], DrNAS [4] and GDAS [8] appliedin conjunction with weight-entanglement.
Revised: • We study popular one-shot optimizers like DARTS [17], DrNAS [4] and GDAS [8] applied in conjunction with weight-entanglement.

Original: Our method Pareto-dominates the AutoFormerbaseline models on CIFAR10, CIFAR100 and ImageNet-1k while improving the accuracy by 0.13% and 2.79 % and 1.5% respectively.
Revised: Our method Pareto-dominates the AutoFormer baseline models on CIFAR10, CIFAR100 and ImageNet-1k while improving the accuracy by 0.13% and 2.79 % and 1.5% respectively.

Original:  Related Work
Revised:  Related Work

Original: Neural Architecture Search [10] aims at automating the design of network architectures.
Revised: Neural Architecture Search [10] aims at automating the design of network architectures.

Original: ENAS [20]was the first method to apply weight-sharing , which forced all the architectures in the search space toshare the parameters by defining a computational super-graph, the sub-graphs of which representedindividual models.
Revised: ENAS [20] was the first method to apply weight-sharing , which forced all the architectures in the search space to share the parameters by defining a computational super-graph, the sub-graphs of which represented individual models.

Original: Using this weight-sharing, ENAS reduced the computational cost of NAS by1000 × compared to previous methods.
Revised: Using this weight-sharing, ENAS reduced the computational cost of NAS by 1000 × compared to previous methods.

Original: DARTS [19] combined one-shot architecture search with a gradient-based approach.
Revised: DARTS [19] combined one-shot architecture search with a gradient-based approach.

Original: It formulatedthe search as a bi-level optimization problem that trained the parameters of the super-net on thetraining set in the inner loop while training the architectural parameters of the super-net on thevalidation set in the outer loop.
Revised: It formulated the search as a bi-level optimization problem that trained the parameters of the super-net on the training set in the inner loop while training the architectural parameters of the super-net on the validation set in the outer loop.

Original: These architectural parameters are later used to discretize the super-netto get the most promising model.
Revised: These architectural parameters are later used to discretize the super-net to get the most promising model.

Original: DARTS, however, had prominent failure modes , as outlined in
Revised: DARTS, however, had prominent failure modes , as outlined in

Original: Robust-DARTS [27].
Revised: Robust-DARTS [27].

Original: Several other gradient-based one-shot optimization techniques have since beendeveloped to improve upon DARTS, such as GDAS [8] and DrNAS [4].
Revised: Several other gradient-based one-shot optimization techniques have since been developed to improve upon DARTS, such as GDAS [8] and DrNAS [4].

Original: GDAS samples one paththrough the entire super-graph in every training step using differentiable Gumbel-softmax sampling .
Revised: GDAS samples one path through the entire super-graph in every training step using differentiable Gumbel-softmax sampling .

Original: DrNAS, on the other hand, re-frames the one-shot search as learning a Dirichlet distribution.
Revised: DrNAS, on the other hand, re-frames the one-shot search as learning a Dirichlet distribution.

Original: Lastly, our work is closely related to AutoFormer [3], which introduced weight-entanglement as ameans of restricting memory consumption.
Revised: Lastly, our work is closely related to AutoFormer [3], which introduced weight-entanglement as a means of restricting memory consumption.

Original: Our approach performs a continuous relaxation of theweight entanglement search space by adding architectural parameters to it which allows the use ofgradient-based one-shot optimizers.
Revised: Our approach performs a continuous relaxation of the weight entanglement search space by adding architectural parameters to it which allows the use of gradient-based one-shot optimizers.

Original:  Methodology
Revised:  Methodology

Original: 3. Weight Sharing and Entanglement
Revised: Weight Sharing and Entanglement

Original: Weight Sharing.
Revised: Weight Sharing.

Original: We first review weight-sharing [24], which is used in many one-shot optimizationtechniques.
Revised: We first review weight-sharing [24], which is used in many one-shot optimization techniques.

Original: The search space in weight-sharing NAS methods consists of a single supernet , each edgeof which holds several different operators , such as convolutional or pooling layers.
Revised: The search space in weight-sharing NAS methods consists of a single supernet , each edge of which holds several different operators , such as convolutional or pooling layers.

Original: Each of theseoperators has an associated weight matrix, and these weight matrices are shared: all (exponentiallymany) architectures that include a certain operator use the same shared weight matrix.
Revised: Each of these operators has an associated weight matrix, and these weight matrices are shared: all (exponentially many) architectures that include a certain operator use the same shared weight matrix.

Original: That is, byupdating one path of the supernet, exponentially many other paths are also (partially) updated.
Revised: That is, by updating one path of the supernet, exponentially many other paths are also (partially) updated.

Original: Weight-sharing comes with a set of advantages and disadvantages.
Revised: Weight-sharing comes with a set of advantages and disadvantages.

Original: On one hand, it allows a largenumber of models to be trained and searched at the expense of training a single supernet [19].
Revised: On one hand, it allows a large number of models to be trained and searched at the expense of training a single supernet [19].

Original: This has drastically increased the speed and efficiency of NAS [1].
Revised: This has drastically increased the speed and efficiency of NAS [1].

Original: On the other hand, the memoryrequirements for training the supernet using differentiable optimizers like DARTS with weight sharingincreases linearly with the number of choice of operators on each edge, making it infeasible to trainvery large models.[Figure]
Revised: On the other hand, the memory requirements for training the supernet using differentiable optimizers like DARTS with weight sharing increases linearly with the number of choice of operators on each edge, making it infeasible to train very large models.

Original: Weight Entanglement.
Revised: Weight Entanglement.

Original: While weight-sharing shares the operator parameters between differentarchitectures, weight-entanglement goes one step further and shares the parameters between thedifferent operators on a given edge of the supernet.
Revised: While weight-sharing shares the operator parameters between different architectures, weight-entanglement goes one step further and shares the parameters between the different operators on a given edge of the supernet.

Original: E.g., the parameters of a smaller convolutionalkernel are sampled from the largest convolutional kernel on the same edge.
Revised: E.g., the parameters of a smaller convolutional kernel are sampled from the largest convolutional kernel on the same edge.

Original: The weights are henceshared between different sets of operators as well as different architectures sampled from the supernet.
Revised: The weights are hence shared between different sets of operators as well as different architectures sampled from the supernet.

Original: Weight-entanglement has several advantages.
Revised: Weight-entanglement has several advantages.

Original: Primarily, the memory requirement for the entiresupernet is exactly the same as for the largest architecture in the search space, making it feasible toexplore much larger architectures than what standard weight-sharing permits.
Revised: Primarily, the memory requirement for the entire supernet is exactly the same as for the largest architecture in the search space, making it feasible to [Figure] explore much larger architectures than what standard weight-sharing permits.

Original: Weight-entanglementalso imparts the supernet with several of the beneficial properties of the once-for-all networks [2, 22].
Revised: Weight-entanglement also imparts the supernet with several of the beneficial properties of the once-for-all networks [2, 22].

Original: 3.2 Our Approach
Revised: 3.2 Our Approach

Original: Our approach combines weight-entanglement with gradient-based one-shot NAS.
Revised: Our approach combines weight-entanglement with gradient-based one-shot NAS.

Original: The operators onany given edge reuse the parameters of the largest operator on the edge, as in weight-entanglement,but we also learn architectural parameters for each of these operators, as in gradient-based one-shot
Revised: The operators on any given edge reuse the parameters of the largest operator on the edge, as in weight-entanglement, but we also learn architectural parameters for each of these operators, as in gradient-based one-shot

Original: NAS.
Revised: NAS.

Original: This allows us to apply one-shot architectural optimizers such as DARTS, GDAS and DrNASto search spaces that leverage weight-entanglement (in our experiments, the AutoFormer searchspace [3]). See Figure 1 for an overview.
Revised: This allows us to apply one-shot architectural optimizers such as DARTS, GDAS and DrNAS to search spaces that leverage weight-entanglement (in our experiments, the AutoFormer search space [3]). See Figure 1 for an overview.

Original: Unlike in AutoFormer, we can now acquire the optimalarchitecture directly by discretizing the supernet using the learned architectural parameters.
Revised: Unlike in AutoFormer, we can now acquire the optimal architecture directly by discretizing the supernet using the learned architectural parameters.

Original: Search Space.
Revised: Search Space.

Original: Our experiments were run on the AutoFormer search space [3].
Revised: Our experiments were run on the AutoFormer search space [3].

Original: This is a transformerbased search space that varies five factors in its building blocks: embedding dimension α e (dimensionof the patch embeddings which are fed into the transformer encoders) , Q-K-V dimension (dimensionsof the query, key and value vectors of the Multi-head Self Attention layer), number of heads in the
Revised: This is a transformerbased search space that varies five factors in its building blocks: embedding dimension α e (dimension of the patch embeddings which are fed into the transformer encoders) , Q-K-V dimension (dimensions of the query, key and value vectors of the Multi-head Self Attention layer), number of heads in the

Original: MSA α h , MLP ratio α r (the ratio of hidden dimension to the embedding dimension in the multi-layerperceptron), and network depth α n (number of transformer encoder blocks stacked together) .
Revised: MSA α h , MLP ratio α r (the ratio of hidden dimension to the embedding dimension in the multi-layer perceptron), and network depth α n (number of transformer encoder blocks stacked together) .

Original: The
Revised: The

Original: AutoFormer search space also fixes the ratio of Q-K-V dimension to the number of heads hence it isnot a part of the search space.
Revised: AutoFormer search space also fixes the ratio of Q-K-V dimension to the number of heads hence it is not a part of the search space.

Original: Combination Mixture Operation For each of the building blocks of the vision transformer, wesingle out one or more of the above factors it depends on.
Revised: Combination Mixture Operation For each of the building blocks of the vision transformer , we single out one or more of the above factors it depends on.

Original: The main difference we observe fromtraditional gradient based optimization is that some operations depend on two or more α ′ s which wecall combination mixop .
Revised: The main difference we observe from traditional gradient based optimization is that some operations depend on two or more α ′ s which we call combination mixop .

Original: Consider the Multiheaded Self Attention (MSA) block, which depends uponthe embedding dimension α e and number of heads α h .
Revised: Consider the Multiheaded Self Attention (MSA) block, which depends upon the embedding dimension α e and number of heads α h .

Original: We define the architectural weights for thisoperation as the combination of these two choices, i.e., α e,h .
Revised: We define the architectural weights for this operation as the combination of these two choices, i.e., α e,h .

Original: We refer the reader to Appendix A.3 forfurther details on the combination mixop .[Table]
Revised: We refer the reader to Appendix A.3 for further details on the combination mixop .

Original:  Experiments and Results
Revised:  Experiments and Results

Original: Image Classification on CIFAR10 and CIFAR100.
Revised: Image Classification on CIFAR10 and CIFAR100.

Original: As our first experiment, we study threegradient-based one-shot optimizers (DARTS, GDAS and DrNAS) on the AutoFormer search space.
Revised: As our first experiment, we study three gradient-based one-shot optimizers (DARTS, GDAS and DrNAS) on the AutoFormer search space.

Original: For each of these optimizers, we first train the supernet with the optimizer for search and then obtainthe final architecture by discretizing the supernet; this architecture is then evaluated by finetuning aswell as by retraining from scratch.
Revised: For each of these optimizers, we first train the supernet with the optimizer for search and then obtain the final architecture by discretizing the supernet; this architecture is then evaluated by finetuning as well as by retraining from scratch.

Original: The search and training is performed directly for CIFAR-10 and
Revised: The search and training is performed directly for CIFAR-10 and

Original: CIFAR-100 [15] are shown in Table 1 and Table 2 in the Appendix B.
Revised: CIFAR-100 [15] are shown in Table 1 and Table 2. [Table]

Original: Table 1 shows that the models from our methods Pareto- dominate the models found by AutoFormer, performingespecially well in terms of model size and FLOPS.
Revised: Table 1 shows that the models from our methods Pareto-dominate the models found by AutoFormer, performing especially well in terms of model size and FLOPS.

Original: Interestingly, as Table 2 in Appendix B, our finetuned modelsare better than training from scratch indicating that ourmethod imparts some regularization properties to the supernet, which require further investigation.
Revised: Interestingly, as Table 2 in Appendix B, our finetuned models are better than training from scratch indicating that our method imparts some regularization properties to the supernet, which require further investigation.

Original: Further, as observed in Table 1 the models discovered byour method with the DARTS, DrNAS and GDAS optimizers achieve a better trade-off between error-rate, numberof parameters and FLOPS than the Autoformer. 
Revised: Further, as observed in Table 1 the models discovered by our method with the DARTS, DrNAS and GDAS optimizers achieve a better trade-off between error-rate, number of parameters and FLOPS than the Autoformer. [Table] [Table]

Original: Transfer to ImageNet-1k.
Revised: [Table] Ours + DrNAS 98. [Table] [Table]

Revised: [Table]

Revised: Ours + GDAS 98.

Original:  For our second experiment, we search for the best architecture on CIFAR10 and CIFAR100 ( img size = 224, patch size = 16)
Revised: [Table] [Table] For our second experiment, we search for the best archi- on ImageNet-1k tecture on CIFAR10 and CIFAR100 ( img size = 224, patch size = 16)

Original: Table 3 in Appendix C.
Revised: Table 3 in Appendix C.

Original: We choosethe best model that we obtained from across all the optimizers on CIFAR-100 (the model obtainedusing GDAS, in this case), and transfer its weights to a model which is then fine-tuned to performclassification on ImageNet-1k [7].
Revised: We choose the best model that we obtained from across all the optimizers on CIFAR-100 (the model obtained using GDAS, in this case), and transfer its weights to a model which is then fine-tuned to perform classification on ImageNet-1k [7].

Original: We achieve an accuracy of 79 .
Revised: We achieve an accuracy of 79 .

Original: 34% on ImageNet-1k, compared to AutoFormer which achieves anaccuracy of 74 .
Revised: 34% on ImageNet-1k, compared to AutoFormer which achieves an accuracy of 74 .

Original: Further, as noted, from Figure 2, we obtain improvements of up to 1.5% on 
Revised: Further, as noted, from Figure 2, we obtain improvements of up to 1.5% on ImageNet-1k compared to

Original: ImageNet-1k compared to AutoFormers of comparable model sizes.
Revised: AutoFormers of comparable model sizes.

Original:  Conclusion
Revised:  Conclusion

Original: We present a new framework GraViT-E to efficiently search for the architectural parameters of largemodels like transformers.
Revised: We present a new framework GraViT-E to efficiently search for the architectural parameters of large models like transformers.

Original: Our method outperforms the AutoFormer search strategy on multipledatasets in terms of performance and parameter-efficiency.
Revised: Our method outperforms the AutoFormer search strategy on multiple datasets in terms of performance and parameter-efficiency.

Original: In the future, we plan to extend our workto support multiple search spaces (e.g. different transformer types [12] and convolutional networks[14, 13]) and multiple one-shot optimizers (e.g., PC-DARTS [26], SNAS [25], FairNAS [6]).
Revised: In the future, we plan to extend our work to support multiple search spaces (e.g. different transformer types [12] and convolutional networks [14, 13]) and multiple one-shot optimizers (e.g., PC-DARTS [26], SNAS [25], FairNAS [6]).

Original: While our work initiates the study of continuous relaxation of the transformer search space for144 gradient-based one-shot optimizers, we still face some limitations.
Revised: While our work initiates the study of continuous relaxation of the transformer search space for gradient-based one-shot optimizers, we still face some limitations.

Original: For now, our observations arebased upon the study of only a lightweight Vision Transformer search space.
Revised: For now, our observations are based upon the study of only a lightweight Vision Transformer search space.

Original: Further we do not study146 different transformer applications like language modeling, semantic image segmentation, etc.
Revised: Further we do not study different transformer applications like language modeling, semantic image segmentation, etc.

Original: Howthe weight entanglement scheme hampers or benefits the gradient based search also warrants further148 investigation.
Revised: How the weight entanglement scheme hampers or benefits the gradient based search also warrants further investigation.

Original: However, since our formulation can take advantage of the continually-improving setof methods for gradient-based one-shot NAS, we hope that our work will lead to the first generally-
Revised: However, since our formulation can take advantage of the continually-improving set of methods for gradient-based one-shot NAS, we hope that our work will lead to the first generallyapplicable robust and efficient architecture optimization method for transformers. Acknowledgments

Revised: This research was partially supported by the following sources:

Original: A Appendix
Revised: TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the German Federal Ministry of Education and Research (BMBF, grant RenormalizedFlows 01IS19077C); the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828; the European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765). Robert Bosch GmbH is acknowledged for financial support. Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC.

Revised: Neither the European Union nor the

Revised: ERC can be held responsible for them

Original:  A.1 Pareto Fronts
Revised: A Appendix A.1 Pareto Fronts

Original: In Figure 3 we present the parameter-size and error Pareto front for AutoFormer v/s our models withdifferent one-shot optimizers.
Revised: In Figure 3 we present the parameter-size and error Pareto front for AutoFormer v/s our models with different one-shot optimizers.

Original: We observe that GDAS pareto dominates other models for CIFAR100and GDAS and DrNAS lie on the Pareto front for CIFAR10.
Revised: We observe that GDAS pareto dominates other models for CIFAR and GDAS and DrNAS lie on the Pareto front for CIFAR

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: A.2 Experimental Details

Original: We use the official source code of AutoFormer available at https://github.com/microsoft/
Revised: We use the official source code of AutoFormer available at https://github.com/microsoft/

Original: Cream/tree/main/AutoFormer for all the AutoFormer experiments on CIFAR10 and CIFAR100.
Revised: Cream/tree/main/AutoFormer for all the AutoFormer experiments on CIFAR10 and CIFAR100.

Original: We closely follow the AutoFormer training pipeline and search space design.
Revised: We closely follow the AutoFormer training pipeline and search space design.

Original: AutoFormer searchedon three transformer sizes Autoformer-T (tiny), AutoFormer-S (small) and Autoformer-B (base).
Revised: AutoFormer searched on three transformer sizes Autoformer-T (tiny), AutoFormer-S (small) and Autoformer-B (base).

Original: Wecurrently restrict ourselves to Autoformer-T.
Revised: We currently restrict ourselves to Autoformer-T.

Original: We use a 50%-50% train and validation split for the
Revised: We use a 50%-50% train and validation split for the

Original: CIFAR10 and CIFAR100 dataset for the bi-level optimization.
Revised: CIFAR10 and CIFAR100 dataset for the bi-level optimization.

Original: A.3 Optimization Details.
Revised: A.3 Optimization Details.

Original: We assign an architectural parameter α to each of these variables, α d for embedding dimension, α hfor number of heads, α r for MLP-ratio and α n for the number of transformer encoder blocks to makethe application of one-shot optimizers on the AutoFormer search space feasible.
Revised: We assign an architectural parameter α to each of these variables, α d for embedding dimension, α h for number of heads, α r for MLP-ratio and α n for the number of transformer encoder blocks to make the application of one-shot optimizers on the AutoFormer search space feasible.

Original: Next, for each of thebuilding blocks of the vision transformer, we single out one or more of the above factors it depends on.
Revised: Next, for each of the building blocks of the vision transformer, we single out one or more of the above factors it depends on.

Original: We observe two categories of mixture operations, first which depend on only one of the above factors( single mixop ) and a second one which is based upon combination of two (or more) α ′ s which wecall combination mixop .
Revised: We observe two categories of mixture operations, first which depend on only one of the above factors ( single mixop ) and a second one which is based upon combination of two (or more) α ′ s which we call combination mixop .

Original: In single mixops , we use a mixture operation which is compatible with thespecific one-shot optimizer that we use.
Revised: In single mixops , we use a mixture operation which is compatible with the specific one-shot optimizer that we use.

Original: For the combination mixop , let us consider the Multiheaded
Revised: For the combination mixop , let us consider the Multiheaded

Original: Self Attention (MSA) block, which depends upon the embedding dimension α e and number of headsα h .
Revised: Self Attention (MSA) block, which depends upon the embedding dimension α e and number of heads α h .

Original: We define the architecture weights for this operation as the combination of these two choices, i.e.,α e,h , which is defined as below.
Revised: We define the architecture weights for this operation as the combination of these two choices, i.e., α e,h , which is defined as below.

Original: The mixture operation for MSA is then defined as in Equation (1),where O is the cross-product of the two sets of operation choices (here embedding dimension andnumber of heads).
Revised: The mixture operation for MSA is then defined as in Equation (1), where O is the cross-product of the two sets of operation choices (here embedding dimension and number of heads).

Original: The formulation ensures that the ranking of the possible independent choices ( α eand α h ) and the choice for the combination operation ( α e,h ) is not conflicting with each other.[Equation]
Revised: The formulation ensures that the ranking of the possible independent choices ( α e and α h ) and the choice for the combination operation ( α e,h ) is not conflicting with each other.

Original: [Equation]
Revised: [Equation]

Original: B Finetuning v/s retraining
Revised: B Finetuning v/s retraining

Original: Here we compare the accuracy of the obtained models from search on CIFAR100 and CIFAR10in terms of their retraining and finetuning accuracies.
Revised: In Table 2 we compare the accuracy of the obtained models from search on CIFAR100 and CIFAR in terms of their retraining and finetuning accuracies.

Original: Surprisingly inheriting the weights from thesupernetwork and finetuning obtains much better accuracies compared to retraining from scratch.
Revised: Surprisingly inheriting the weights from the supernetwork and finetuning obtains much better accuracies compared to retraining from scratch.

Original: Wehypothesize this to be the regularization effect imparted by the supernet training scheme and needs tobe studied further.[Table]
Revised: We hypothesize this to be the regularization effect imparted by the supernet training scheme and needs to be studied further.

Original: C CIFAR10 and CIFAR100 with image size=224 and patch size= 
Revised: C CIFAR10 and CIFAR100 with image size=224 and patch size=

Original: For architectures to be transferrable to ImageNet we need to perform train and search on CIFAR100and CIFAR10 with the 32 × 32 image padded to 224 × 224 and further using a patch size of 16 × 16 .
Revised: For architectures to be transferrable to ImageNet we need to perform train and search on CIFAR and CIFAR10 with the 32 × 32 image padded to 224 × 224 and further using a patch size of 16 × 16 .

Original: The results obtained on CIFAR10 and CIFAR100 on these patch sizes is as presented in Table 3.
Revised: The results obtained on CIFAR10 and CIFAR100 on these patch sizes is as presented in Table 3.

Original: We observe that similar to the 2 × 2 patch size studies in Section 4.
Revised: We observe that similar to the 2 × 2 patch size studies in Section 4.

Original: GDAS performs the best on
Revised: GDAS performs the best on

Original: CIFAR100 and its weights are then transferred to ImageNet-1k.
Revised: CIFAR100 and its weights are then transferred to ImageNet-1k.

Original: [Table]
Revised: [Table]