Original: Training a Vision Transformer from scratch in less than 24 hours with 1 GPU
Revised: Training a Vision Transformer from scratch in less than 24 hours with 1 GPU

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Transformers have become central to recent advances in computer vision.
Revised: Transformers have become central to recent advances in computer vision.

Original: However,training a vision Transformer (ViT) model from scratch can be resource intensiveand time consuming.
Revised: However, training a vision Transformer (ViT) model from scratch can be resource intensive and time consuming.

Original: In this paper, we aim to explore approaches to reduce thetraining costs of ViT models.
Revised: In this paper, we aim to explore approaches to reduce the training costs of ViT models.

Original: We introduce some algorithmic improvements toenable training a ViT model from scratch with limited hardware (1 GPU) and time(24 hours) resources.
Revised: We introduce some algorithmic improvements to enable training a ViT model from scratch with limited hardware (1 GPU) and time (24 hours) resources.

Original: First, we propose an efficient approach to add locality tothe ViT architecture.
Revised: First, we propose an efficient approach to add locality to the ViT architecture.

Original: Second, we develop a new image size curriculum learningstrategy, which allows to reduce the number of patches extracted from each imageat the beginning of the training.
Revised: Second, we develop a new image size curriculum learning strategy, which allows to reduce the number of patches extracted from each image at the beginning of the training.

Original: Finally, we propose a new variant of the popular
Revised: Finally, we propose a new variant of the popular

Original: ImageNet1k benchmark by adding hardware and time constraints.
Revised: ImageNet1k benchmark by adding hardware and time constraints.

Original: We evaluateour contributions on this benchmark, and show they can significantly improveperformances given the proposed training budget.
Revised: We evaluate our contributions on this benchmark, and show they can significantly improve performances given the proposed training budget.

Revised: We will share the code in https://github.com/BorealisAI/efficient-vit-training.

Original:  Introduction
Revised:  Introduction

Original: Recently, the Transformer architecture [1] has become a key ingredient of an impressive numberof computer vision models [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].
Revised: Recently, the Transformer architecture [1] has become a key ingredient of an impressive number of computer vision models [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].

Original: However, training large Transformermodels usually come at an enormous cost.
Revised: However, training large Transformer models usually come at an enormous cost.

Original: For example, training a small vision Transformer (ViT)like DeiT-S takes about 3 days on 4 GPUs [4].
Revised: For example, training a small vision Transformer (ViT) like DeiT-S takes about 3 days on 4 GPUs [4].

Original: To reduce the cost, we propose to explore the followingproblem: how to train from scratch a ViT model in less than 24 hours with a single GPU .
Revised: To reduce the cost, we propose to explore the following problem: how to train from scratch a ViT model in less than 24 hours with a single GPU .

Original: We thinkthat progress in this direction could have a large impact on the future of computer vision research andapplications for multiple reasons.
Revised: We think that progress in this direction could have a large impact on the future of computer vision research and applications for multiple reasons.

Original: (1) Speed-up model development.
Revised: (1) Speed-up model development.

Original: New models in ML are oftenevaluated in performance by running and analyzing experiments on them, which is not a scalablemethod when the training cost for each experiment is too high.
Revised: New models in ML are often evaluated in performance by running and analyzing experiments on them, which is not a scalable method when the training cost for each experiment is too high.

Original: By reducing the training cost weshorten the development loop.
Revised: By reducing the training cost we shorten the development loop.

Original: (2) Be more accessible.
Revised: (2) Be more accessible.

Original: Most ViT models are trained from scratch byusing multiple GPUs or TPUs, which unfortunately excludes researchers who do not have access tosuch resources from this area of research.
Revised: Most ViT models are trained from scratch by using multiple GPUs or TPUs, which unfortunately excludes researchers who do not have access to such resources from this area of research.

Original: By using only 1 GPU for our benchmark, we significantlyreduce the training cost of ViTs which allows more researchers to push this research direction forward.
Revised: By using only 1 GPU for our benchmark, we significantly reduce the training cost of ViTs which allows more researchers to push this research direction forward.

Original: (3) Reduce the environmental cost.
Revised: (3) Reduce the environmental cost.

Original: One approach to reduce the training cost is to develop more efficient specialized hardware or moreefficient data representations like half-precision.
Revised: One approach to reduce the training cost is to develop more efficient specialized hardware or more efficient data representations like half-precision.

Original: Another orthogonal approach is to develop moreefficient algorithms.
Revised: Another orthogonal approach is to develop more efficient algorithms.

Original: In this paper, we focus on the second approach.
Revised: In this paper, we focus on the second approach.

Original: A lot of methods ( e.g. pruning[13, 14, 15]) have been developed to reduce the inference cost, but a limited number of works areexploring ideas to reduce the training cost.
Revised: A lot of methods ( e.g. pruning [13, 14, 15]) have been developed to reduce the inference cost, but a limited number of works are exploring ideas to reduce the training cost.

Original: [16, 17] explores how to train ViTs from scratch on smallsize datasets.
Revised: [16, 17] explores how to train ViTs from scratch on smallsize datasets.

Original: [18] explores how to train a BERT model on text data in 24 hours but it uses a serverof 8 GPUs whereas we limit ourselves to a single GPU.
Revised: [18] explores how to train a BERT model on text data in 24 hours but it uses a server of 8 GPUs whereas we limit ourselves to a single GPU.

Original: Primer [19] proposes searching for moreefficient alternatives to Transformer, but it focuses on NLP.
Revised: Primer [19] proposes searching for more efficient alternatives to Transformer, but it focuses on NLP.

Original: We tried to apply the findings from thiswork to ViT, but we did not see any improvement.
Revised: We tried to apply the findings from this work to ViT, but we did not see any improvement.

Original: Hence, it remains unclear to us if the improvementsdeveloped for the NLP domain can also be generalized for computer vision applications.
Revised: Hence, it remains unclear to us if the improvements developed for the NLP domain can also be generalized for computer vision applications.

Original: We define our objective as gaining the highest performance metric within our fixed budget.
Revised: We define our objective as gaining the highest performance metric within our fixed budget.

Original: To reducethe training cost, we propose two algorithmic contributions.
Revised: To reduce the training cost, we propose two algorithmic contributions.

Original: First, we show that adding a localitymechanism in each feed-forward network of a Transformer encoder architecture can significantlyimprove the performance given a fixed resource budget.
Revised: First, we show that adding a locality mechanism in each feed-forward network of a Transformer encoder architecture can significantly improve the performance given a fixed resource budget.

Original: Second, we propose an image size-basedcurriculum learning strategy to reduce the training time per epoch at the beginning of the training.
Revised: Second, we propose an image size-based curriculum learning strategy to reduce the training time per epoch at the beginning of the training.

Original: The training starts with small images, and then larger images are gradually added to the training.
Revised: The training starts with small images, and then larger images are gradually added to the training.

Original: Beyond the algorithmic changes introduced to reduce the training cost, we also formally define ournew benchmark on ImageNet1k by including resource constraints (1 GPU and 24 hour time budget),and evaluate our model on it.
Revised: Beyond the algorithmic changes introduced to reduce the training cost, we also formally define our new benchmark on ImageNet1k by including resource constraints (1 GPU and 24 hour time budget), and evaluate our model on it.

Original:  Proposed method 
Revised:  Proposed method 2.1 Locality in vision Transformer architecture

Original: 2.1 Locality in vision Transformer architecture

Original: In this section, we first explain the vision Transformer (ViT) architecture proposed in [3], and thenwe describe our changes to the architecture to speed-up the training.
Revised: In this section, we first explain the vision Transformer (ViT) architecture proposed in [3], and then we describe our changes to the architecture to speed-up the training.

Original: ViT architecture.
Revised: ViT architecture.

Original: The vanilla Transformer [1] receives as input a 1D sequence of token embeddings.
Revised: The vanilla Transformer [1] receives as input a 1D sequence of token embeddings.

Original: To process 2D images, the ViT model [3, 4] splits each input image into a sequence of non-overlappingflattened 2D patches.
Revised: To process 2D images, the ViT model [3, 4] splits each input image into a sequence of non-overlapping flattened 2D patches.

Original: The patches are mapped to D dimensions with a trainable linear projection.
Revised: The patches are mapped to D dimensions with a trainable linear projection.

Original: The output of this projection is usually named patch embeddings.
Revised: The output of this projection is usually named patch embeddings.

Original: Then, the learnable positionembeddings are added to the patch embeddings to encode positional information of each patch in theimage.
Revised: Then, the learnable position embeddings are added to the patch embeddings to encode positional information of each patch in the image.

Original: The output sequence of embedding vectors z 0 serves as the input to the Transformer encoder.
Revised: The output sequence of embedding vectors z 0 serves as the input to the Transformer encoder.

Original: The Transformer encoder [1] consists of alternating layers of multi-head self-attention (MSA) and
Revised: The Transformer encoder [1] consists of alternating layers of multi-head self-attention (MSA) and

Original: Feed-Forward Networks (FFN).
Revised: Feed-Forward Networks (FFN).

Original: LayerNorm (LN) [20] is applied before every block, and residualconnections after every block.
Revised: LayerNorm (LN) [20] is applied before every block, and residual connections after every block.

Original: For a Transformer encoder with L blocks, the output representation iscomputed following the equations:
Revised: For a Transformer encoder with L blocks, the output representation is computed following the equations:

Original: [Equation]
Revised: [Equation]

Original: The FFN is composed of two linear layers separated by a GELU activation [21].
Revised: The FFN is composed of two linear layers separated by a GELU activation [21].

Original: The first linear layerexpands the dimension from D to 4 D , and the second one reduces the dimension from 4 D back to D .
Revised: The first linear layer expands the dimension from D to 4 D , and the second one reduces the dimension from 4 D back to D .

Original: Locality in ViT architecture.
Revised: Locality in ViT architecture.

Original: The self-attention layer of a ViT captures global dependenciesbetween all of the patches, but it lacks locality inductive bias, in particular a mechanism to allowinformation exchange within a local region.
Revised: The self-attention layer of a ViT captures global dependencies between all of the patches, but it lacks locality inductive bias, in particular a mechanism to allow information exchange within a local region.

Original: In order to introduce locality into vision Transformers,we only adapt the FFNs while the other parts, such as self-attention and position encoding, are notchanged.
Revised: In order to introduce locality into vision Transformers, we only adapt the FFNs while the other parts, such as self-attention and position encoding, are not changed.

Original: We propose to add locality to the ViT architecture by adding a depth-wise convolution layerinto each FFN.
Revised: We propose to add locality to the ViT architecture by adding a depth-wise convolution layer into each FFN.

Original: A 3 × 3 depth-wise convolution is added between the two fully-connected layers inthe FFN (Figure 1).
Revised: A 3 × 3 depth-wise convolution is added between the two fully-connected layers in the FFN (Figure 1).

Original: Before each 3 × 3 depth-wise convolution, a sequence-to-image (Seq2Im) layeris used to convert each flattened patch representation into a 2D patch representation.
Revised: Before each 3 × 3 depth-wise convolution, a sequence-to-image (Seq2Im) layer is used to convert each flattened patch representation into a 2D patch representation.

Original: Similarly, animage-to-sequence (Im2Seq) layer is used to convert each 2D patch representation into a flattenedpatch representation.
Revised: Similarly, an image-to-sequence (Im2Seq) layer is used to convert each 2D patch representation into a flattened patch representation.

Original: We also replace the GELU activation layer [21] by the h-swish [22].
Revised: We also replace the GELU activation layer [21] by the h-swish [22].

Original: Connection with existing works.
Revised: Connection with existing works.

Original: Other works [5, 23, 9, 10, 11] explore adding locality in ViTarchitecture.
Revised: Other works [5, 23, 9, 10, 11] explore adding locality in ViT architecture.

Original: Most of them analyze the impact of locality mechanism on the final accuracy, but tothe best of our knowledge, none of them study the impact of locality mechanism on the trainingspeed.
Revised: Most of them analyze the impact of locality mechanism on the final accuracy, but to the best of our knowledge, none of them study the impact of locality mechanism on the training speed.

Original: The closest work to our architecture is probably LocalViT [23], which also uses convolutionsin the FFN.
Revised: The closest work to our architecture is probably LocalViT [23], which also uses convolutions in the FFN.

Original: There are 3 main differences between LocalViT and our model.
Revised: There are 3 main differences between LocalViT and our model.

Original: First, our architectureuses LayerNorm [20] as a normalization layer, whereas LocalViT uses 2D BatchNorm [24].
Revised: First, our architecture uses LayerNorm [20] as a normalization layer, whereas LocalViT uses 2D BatchNorm [24].

Original: Second,the expanding and squeezing layers are implemented as fully-connected layer in our architecture,whereas LocalViT uses convolution layers.
Revised: Second, the expanding and squeezing layers are implemented as fully-connected layer in our architecture, whereas LocalViT uses convolution layers.

Original: Finally, our architecture uses h-swish [22] as activationlayer, whereas LocalViT uses a combination of h-swish and squeeze-and-excitation module [25].
Revised: Finally, our architecture uses h-swish [22] as activation layer, whereas LocalViT uses a combination of h-swish and squeeze-and-excitation module [25].

Original: Weobserve our contributions are important and lead to a more efficient architecture (see subsection 3.2).
Revised: We observe our contributions are important and lead to a more efficient architecture (see subsection 3.2).

Original: 2.2 Image size-based curriculum learning
Revised: 2.2 Image size-based curriculum learning

Original: Training vision Transformers is traditionally done by using mini-batches of 224 × 224 RGB imagessampled uniformly from the training data.
Revised: Training vision Transformers is traditionally done by using mini-batches of 224 × 224 RGB images sampled uniformly from the training data.

Original: Each image is usually decomposed to non-overlapping16 × 16 patches, so the input of vision Transformers is usually a sequence of 196 flattened patches.
Revised: Each image is usually decomposed to non-overlapping 16 × 16 patches, so the input of vision Transformers is usually a sequence of 196 flattened patches.

Original: The complexity of a vanilla Transformer architecture [1] is quadratic to the sequence length ( i.e. number of patches) due to the attention mechanism.
Revised: The complexity of a vanilla Transformer architecture [1] is quadratic to the sequence length ( i.e. number of patches) due to the attention mechanism.

Original: In this section, we explore an approach to reducethe sequence length ( i.e. number of patches) to speed-up training.
Revised: In this section, we explore an approach to reduce the sequence length ( i.e. number of patches) to speed-up training.

Original: We develop a small to large imagesize-based curriculum learning strategy [26], where shorter patch sequences are used at the beginningof the training.
Revised: We develop a small to large image size-based curriculum learning strategy [26], where shorter patch sequences are used at the beginning of the training.

Original: The key idea of curriculum learning [26] is to start small and learn easier aspects of the task, thengradually increase the difficulty level.
Revised: The key idea of curriculum learning [26] is to start small and learn easier aspects of the task, then gradually increase the difficulty level.

Original: There are different ways to use curriculum learning, but apopular one is to start training with easy examples, and then gradually adding more difficult examples[27, 28].
Revised: There are different ways to use curriculum learning, but a popular one is to start training with easy examples, and then gradually adding more difficult examples [27, 28].

Original: We use the image size as a proxy of the image difficulty.
Revised: We use the image size as a proxy of the image difficulty.

Original: At the beginning of the training,the vision Transformer model is trained with low resolution images, and then image resolution isgradually increased every few epochs.
Revised: At the beginning of the training, the vision Transformer model is trained with low resolution images, and then image resolution is gradually increased every few epochs.

Original: We do this by resizing the input images.
Revised: We do this by resizing the input images.

Original: Figure 1 showsdifferent image sizes ( i.e. steps of the curriculum learning) for a given image.
Revised: Figure 1 shows different image sizes ( i.e. steps of the curriculum learning) for a given image.

Original: In each epoch, all theimages have the same size, but the image size can increase between epochs.
Revised: In each epoch, all the images have the same size, but the image size can increase between epochs.

Original: Then, a critical questionis how to design a good strategy to increase the image size.
Revised: Then, a critical question is how to design a good strategy to increase the image size.

Original: First, it is important to define the initialimage size i.e. the image size for the first epoch.
Revised: First, it is important to define the initial image size i.e. the image size for the first epoch.

Original: Then, it is important to control when the image sizeincreases.
Revised: Then, it is important to control when the image size increases.

Original: We use a linear rule that increases the image size by M pixels every N epochs.
Revised: We use a linear rule that increases the image size by M pixels every N epochs.

Original: In theexperimental section, we analyze the impact of these hyper-parameters.
Revised: In the experimental section, we analyze the impact of these hyper-parameters.

Original: By construction, all the layers in the vision Transformer architecture, excepts the positional embeddings, can adapt automatically to multiple sequence lengths.
Revised: By construction, all the layers in the vision Transformer architecture, except the positional embeddings, can adapt automatically to multiple sequence lengths.

Original: The positional embeddings are updatedby interpolation after each image size increase.
Revised: The positional embeddings are updated by interpolation after each image size increase.

Original: To avoid to deal with partial patches, we only useimage sizes that can be decomposed in 16 × 16 patches.
Revised: To avoid to deal with partial patches, we only use image sizes that can be decomposed in 16 × 16 patches.

Original: Using multiple image sizes during trainingcan also help to learn better scale invariant representations. 
Revised: Using multiple image sizes during training can also help to learn better scale invariant representations. [Table]

Original:  Experimental result
Revised:  Experimental result

Original: This section gives experimental results for image classification on ImageNet1k, which is well-knownbenchmark widely used to evaluate image classification models.
Revised: This section gives experimental results for image classification on ImageNet1k, which is a well-known benchmark widely used to evaluate image classification models.

Original: We show our results, and analyzethe impact of our contributions.
Revised: We show our results, and analyze the impact of our contributions.

Original: The implementation details can be found in Appendix A.
Revised: The implementation details can be found in Appendix A.

Original: 3.1 Results
Revised: 3.1 Results

Original: We evaluate models trained from scratch and in less than 24 hours, with a single GPU on ImageNet1k.
Revised: We evaluate models trained from scratch and in less than 24 hours, with a single GPU on ImageNet1k.

Original: To the best of our knowledge, there is no existing results for this benchmark so we define somebaselines.
Revised: To the best of our knowledge, there are no existing results for this benchmark so we define some baselines.

Original: We trained DeiT-S and LocalViT-S models for 24 hours with a single GPU by using theirofficial implementations.
Revised: We trained DeiT-S and LocalViT-S models for 24 hours with a single GPU by using their official implementations.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: We observe our model outperformsother models by a large margin.
Revised: We observe our model outperforms other models by a large margin.

Original: There is an improvement of about 11 pt w.r.t DeiT-S, and about 26 ptw.r.t LocalViT.
Revised: There is an improvement of about 11 pt w.r.t DeiT-S, and about 26 pt w.r.t LocalViT.

Original: We also report the DeiT-S results by using the original resources (4 GPUs and 3 days)as an upper-bound.
Revised: We also report the DeiT-S results by using the original resources (4 GPUs and 3 days) as an upper-bound.

Original: We can see there is still a large gap between this upper bound and our model, but 
Revised: We can see there is still a large gap between this upper bound and our model, but we also see a large improvement w.r.t

Original: [Table] we also see a large improvement w.r.t the baseline DeiT-S. These experiments show adding localityin the Transformer mechanism and using an image size curriculum learning help to speed-up trainingin a limited resource setting.
Revised: the baseline DeiT-S. These experiments show adding locality in the Transformer mechanism and using an image size curriculum learning help to speed-up training in a limited resource setting.

Original: The next sections analyze the impact of each of these contributions.
Revised: The next sections analyze the impact of each of these contributions.

Original: 3.2 Analysis of the locality mechanism
Revised: 3.2 Analysis of the locality mechanism

Original: In this section, we analyze solely the impact of our locality mechanism as proposed in subsection 2.1(without using the image size-based curriculum learning) and compare with DeiT-S [4] and LocalViTS [23].
Revised: In this section, we analyze solely the impact of our locality mechanism as proposed in subsection 2. (without using the image size-based curriculum learning) and compare with DeiT-S [4] and LocalViTS [23].

Original: Figure 2 shows the results on the validation set of ImageNet1k for each training epoch.
Revised: Figure 2 shows the results on the validation set of ImageNet1k for each training epoch.

Original: Eachmodel is trained for a maximum of 24 hours on a single GPU.
Revised: Each model is trained for a maximum of 24 hours on a single GPU.

Original: Interestingly, we can see adding thelocality mechanism does not significantly increase the training time per epoch w.r.t DeiT-S, but itincreases the top-1 accuracy by 6 . 2 pt.
Revised: Interestingly, we can see adding the locality mechanism does not significantly increase the training time per epoch w.r.t DeiT-S, but it increases the top-1 accuracy by 6 . 2 pt.

Original: We observe the training time per epoch of LocalViT is about2.4 times our model’s, so its training stops after 6 epochs.
Revised: We observe the training time per epoch of LocalViT is about 2.4 times our model’s, so its training stops after 6 epochs.

Original: An advantage of our locality mechanismis that it can get good performance when using quite small batch sizes, because it uses LayerNorminstead of BatchNorm.
Revised: An advantage of our locality mechanism is that it can get good performance when using quite small batch sizes, because it uses LayerNorm instead of BatchNorm.

Original: We also found using h-swish instead of GELU improves the performance byabout 1 pt.
Revised: We also found using h-swish instead of GELU improves the performance by about 1 pt.

Original: This analysis brings evidence that our locality mechanism can help speed-up training, andas a result reduce the training cost.
Revised: This analysis brings evidence that our locality mechanism can help speed-up training, and as a result reduce the training cost.

Original: 3.3 Image size-based curriculum learning
Revised: 3.3 Image size-based curriculum learning

Original: In this section, we analyze the image size-based curriculum learning introduced in subsection 2.2.
Revised: In this section, we analyze the image size-based curriculum learning introduced in subsection 2.2.

Original: The proposed curriculum learning has three important parameters: the initial image size, the imagesize increase, and the number of epochs between each image size increase.
Revised: The proposed curriculum learning has three important parameters: the initial image size, the image size increase, and the number of epochs between each image size increase.

Original: Figure 2 shows the resultsfor different configurations.
Revised: Figure 2 shows the results for different configurations.

Revised: The best result was obtained by starting training from 32 ×pixel images, and increasing the image size along each axis by 32 pixels after each 5 epochs of training until we reach the final image size ( 224 pixels).

Original: We can see that adding the image size-based curriculum learning to ourmodel on top of the locality mechanism improves the top-1 accuracy by about 4 . 5 pt.
Revised: We can see that adding the image size-based curriculum learning to our model on top of the locality mechanism improves the top-1 accuracy by about 4 . 5 pt.

Original: Using smallimages allows training for more epochs because the training time per epoch is smaller.
Revised: Using small images allows training for more epochs because the training time per epoch is smaller.

Revised:  Conclusion

Revised: Our experimental results on ImageNet1k bring evidence towards a positive impact of the locality mechanism and the image size-based curriculum learning to reduce the training cost of a vision

Revised: Transformer model.

Revised: It is likely that the findings can be generalized to other Transformer architectures but more experiments will have to be performed to validate this idea.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the 
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: paper’s

Original: contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [No] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [No] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] Our work does not seem to have potential negative societal impacts.
Revised: [No] Our work does not seem to have potential negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments...

Original: (a) Did you include the

Original: code,

Original: data and instructions needed to reproduce the main experi -mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The code andinstructions will be released before NeurIPS if this work is accepted to the workshop.
Revised: [No] The code and instructions will be released before NeurIPS if this work is accepted to the workshop.

Original: The data is public.
Revised: The data is public.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Appendix A (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] See Appendix A (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Information is available in Appendix A 
Revised: [Yes] Information is available in Appendix A 4.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Implementation Details
Revised: A Implementation Details

Original: In this appendix, we describe the implementation details of our model training.
Revised: In this appendix, we describe the implementation details of our model training.

Original: The ImageNet1k dataset [29] contains 1.28M training images and 50K validation images from onethousand classes.
Revised: The ImageNet1k dataset [29] contains 1.28M training images and 50K validation images from one thousand classes.

Original: Otherwise specified, we report model performances on the official validation split tofollow [23, 4] protocol.
Revised: Otherwise specified, we report model performances on the official validation split to follow [23, 4] protocol.

Original: We use ImageNet1k because it is a well-known benchmark, which is widelyused to evaluate image classification models.
Revised: We use ImageNet1k because it is a well-known benchmark, which is widely used to evaluate image classification models.

Original: Our implementation is based on the timm library [30]and DeiT [4].
Revised: Our implementation is based on the timm library [30] and DeiT [4].

Original: We use DeiT-S model as the main baseline in our experiments.
Revised: We use DeiT -S model as the main baseline in our experiments.

Original: To get fair comparison,we only compare Transformer with small architectures because they have similar complexity andnumber of parameters.
Revised: To get fair comparison, we only compare Transformer with small architectures because they have similar complexity and number of parameters.

Original: Our training protocol is quite similar to DeiT, with some changes.
Revised: Our training protocol is quite similar to DeiT, with some changes.

Original: The trainingcontinues until the 24 hours time limit is reached.
Revised: The training continues until the 24 hours time limit is reached.

Original: The AdamW optimizer is used with a momentumof 0.9.
Revised: The AdamW optimizer is used with a momentum of 0.9.

Original: The batch size is set to 64 and the learning rate is set to 1 e − 3 .
Revised: The batch size is set to 64 and the learning rate is set to 1 e − 3 .

Original: The model is trained with across-entropy loss and by using label smoothing.
Revised: The model is trained with a cross-entropy loss and by using label smoothing.

Original: We do not perform learning rate warm-up.
Revised: We do not perform learning rate warm-up.

Original: We usea Quadro RTX 5000 with 16GB, hosted in an internal cluster, to run our experiments.
Revised: We use a Quadro RTX 5000 with 16GB, hosted in an internal cluster, to run our experiments.

Original: We resize theimages by using a bilinear interpolation.
Revised: We resize the images by using a bilinear interpolation.

Original: Assets.
Revised: Assets.

Original: In this project, we use the following existing assets:
Revised: In this project, we use the following existing assets:

Original: • timm ( https://github.com/rwightman/pytorch-image-models ), Apache-2.0 li- cense • DeiT ( https://github.com/facebookresearch/deit ), Apache 2.0 license• LocalViT ( https://github.com/ofsoundof/LocalViT ), MIT license• ImageNet1k ( https://image-net.org/download.php )
Revised: • timm ( https://github.com/rwightman/pytorch-image-models ), Apache-2.0 license • DeiT ( https://github.com/facebookresearch/deit ), Apache 2.0 license • LocalViT ( https://github.com/ofsoundof/LocalViT ), MIT license • ImageNet1k ( https://image-net.org/download.php )