Original: Training a Vision Transformer from scratch in less than 24 hours with 1 GPU
Revised: Training a Vision Transformer from scratch in less than 24 hours with 1 GPU

Original: Abstract
Revised: Abstract

Original: Transformers have become central to recent advances in computer vision.
Revised: Transformers have become central to recent advances in computer vision.

Original: However, training a vision Transformer (ViT) model from scratch can be resource intensive and time consuming.
Revised: However, training a vision Transformer (ViT) model from scratch can be resource intensive and time consuming.

Original: In this paper, we aim to explore approaches to reduce the training costs of ViT models.
Revised: In this paper, we aim to explore approaches to reduce the training costs of ViT models.

Original: We introduce some algorithmic improvements to enable training a ViT model from scratch with limited hardware (1 GPU) and time (24 hours) resources.
Revised: We introduce some algorithmic improvements to enable training a ViT model from scratch with limited hardware (1 GPU) and time (24 hours) resources.

Original: First, we propose an efficient approach to add locality to the ViT architecture.
Revised: First, we propose an efficient approach to add locality to the ViT architecture.

Original: Second, we develop a new image size curriculum learning strategy, which allows to reduce the number of patches extracted from each image at the beginning of the training.
Revised: Second, we develop a new image size curriculum learning strategy, which allows to reduce the number of patches extracted from each image at the beginning of the training.

Original: Finally, we propose a new variant of the popular
Revised: Finally, we propose a new variant of the popular

Original: ImageNet1k benchmark by adding hardware and time constraints.
Revised: ImageNet1k benchmark by adding hardware and time constraints.

Original: We evaluate our contributions on this benchmark, and show they can significantly improve performances given the proposed training budget.
Revised: We evaluate our contributions on this benchmark, and show they can significantly improve performances given the proposed training budget.

Original: We will share the code in https://github.com/BorealisAI/efficient-vit-training.
Revised: We will share the code in https://github.com/BorealisAI/efficient-vit-training.

Original:  Introduction
Revised:  Introduction

Original: Recently, the Transformer architecture [1] has become a key ingredient of an impressive number of computer vision models [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].
Revised: Recently, the Transformer architecture [1] has become a key ingredient of an impressive number of computer vision models [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].

Original: However, training large Transformer models usually come at an enormous cost.
Revised: However, training large Transformer models usually come at an enormous cost.

Original: For example, training a small vision Transformer (ViT) like DeiT-S takes about 3 days on 4 GPUs [4].
Revised: For example, training a small vision Transformer (ViT) like DeiT-S takes about 3 days on 4 GPUs [4].

Original: To reduce the cost, we propose to explore the following problem: how to train from scratch a ViT model in less than 24 hours with a single GPU .
Revised: To reduce the cost, we propose to explore the following problem: how to train from scratch a ViT model in less than 24 hours with a single GPU .

Original: We think that progress in this direction could have a large impact on the future of computer vision research and applications for multiple reasons.
Revised: We think that progress in this direction could have a large impact on the future of computer vision research and applications for multiple reasons.

Original: (1) Speed-up model development.
Revised: (1) Speed-up model development.

Original: New models in ML are often evaluated in performance by running and analyzing experiments on them, which is not a scalable method when the training cost for each experiment is too high.
Revised: New models in ML are often evaluated in performance by running and analyzing experiments on them, which is not a scalable method when the training cost for each experiment is too high.

Original: By reducing the training cost we shorten the development loop.
Revised: By reducing the training cost we shorten the development loop.

Original: (2) Be more accessible.
Revised: (2) Be more accessible.

Original: Most ViT models are trained from scratch by using multiple GPUs or TPUs, which unfortunately excludes researchers who do not have access to such resources from this area of research.
Revised: Most ViT models are trained from scratch by using multiple GPUs or TPUs, which unfortunately excludes researchers who do not have access to such resources from this area of research.

Original: By using only 1 GPU for our benchmark, we significantly reduce the training cost of ViTs which allows more researchers to push this research direction forward.
Revised: By using only 1 GPU for our benchmark, we significantly reduce the training cost of ViTs which allows more researchers to push this research direction forward.

Original: (3) Reduce the environmental cost.
Revised: (3) Reduce the environmental cost.

Original: One approach to reduce the training cost is to develop more efficient specialized hardware or more efficient data representations like half-precision.
Revised: One approach to reduce the training cost is to develop more efficient specialized hardware or more efficient data representations like half-precision.

Original: Another orthogonal approach is to develop more efficient algorithms.
Revised: Another orthogonal approach is to develop more efficient algorithms.

Original: In this paper, we focus on the second approach.
Revised: In this paper, we focus on the second approach.

Original: A lot of methods ( e.g. pruning [13, 14, 15]) have been developed to reduce the inference cost, but a limited number of works are exploring ideas to reduce the training cost.
Revised: A lot of methods ( e.g. pruning [13, 14, 15]) have been developed to reduce the inference cost, but a limited number of works are exploring ideas to reduce the training cost.

Original: [16, 17] explores how to train ViTs from scratch on smallsize datasets.
Revised: [16, 17] explores how to train ViTs from scratch on smallsize datasets.

Original: [18] explores how to train a BERT model on text data in 24 hours but it uses a server of 8 GPUs whereas we limit ourselves to a single GPU.
Revised: [18] explores how to train a BERT model on text data in 24 hours but it uses a server of 8 GPUs whereas we limit ourselves to a single GPU.

Original: Primer [19] proposes searching for more efficient alternatives to Transformer, but it focuses on NLP.
Revised: Primer [19] proposes searching for more efficient alternatives to Transformer, but it focuses on NLP.

Original: We tried to apply the findings from this work to ViT, but we did not see any improvement.
Revised: We tried to apply the findings from this work to ViT, but we did not see any improvement.

Original: Hence, it remains unclear to us if the improvements developed for the NLP domain can also be generalized for computer vision applications.
Revised: Hence, it remains unclear to us if the improvements developed for the NLP domain can also be generalized for computer vision applications.

Original: We define our objective as gaining the highest performance metric within our fixed budget.
Revised: We define our objective as gaining the highest performance metric within our fixed budget.

Original: To reduce the training cost, we propose two algorithmic contributions.
Revised: To reduce the training cost, we propose two algorithmic contributions.

Original: First, we show that adding a locality mechanism in each feed-forward network of a Transformer encoder architecture can significantly improve the performance given a fixed resource budget.
Revised: First, we show that adding a locality mechanism in each feed-forward network of a Transformer encoder architecture can significantly improve the performance given a fixed resource budget.

Original: Second, we propose an image size-based curriculum learning strategy to reduce the training time per epoch at the beginning of the training.
Revised: Second, we propose an image size-based curriculum learning strategy to reduce the training time per epoch at the beginning of the training.

Original: The training starts with small images, and then larger images are gradually added to the training.
Revised: The training starts with small images, and then larger images are gradually added to the training.

Original: Beyond the algorithmic changes introduced to reduce the training cost, we also formally define our new benchmark on ImageNet1k by including resource constraints (1 GPU and 24 hour time budget), and evaluate our model on it.
Revised: Beyond the algorithmic changes introduced to reduce the training cost, we also formally define our new benchmark on ImageNet1k by including resource constraints (1 GPU and 24 hour time budget), and evaluate our model on it.

Original:  Proposed method 2.1 Locality in vision Transformer architecture
Revised:  Proposed method 2.1 Locality in vision Transformer architecture

Original: In this section, we first explain the vision Transformer (ViT) architecture proposed in [3], and then we describe our changes to the architecture to speed-up the training.
Revised: In this section, we first explain the vision Transformer (ViT) architecture proposed in [3], and then we describe our changes to the architecture to speed-up the training.

Original: ViT architecture.
Revised: ViT architecture.

Original: The vanilla Transformer [1] receives as input a 1D sequence of token embeddings.
Revised: The vanilla Transformer [1] receives as input a 1D sequence of token embeddings.

Original: To process 2D images, the ViT model [3, 4] splits each input image into a sequence of non-overlapping flattened 2D patches.
Revised: To process 2D images, the ViT model [3, 4] splits each input image into a sequence of non-overlapping flattened 2D patches.

Original: The patches are mapped to D dimensions with a trainable linear projection.
Revised: The patches are mapped to D dimensions with a trainable linear projection.

Original: The output of this projection is usually named patch embeddings.
Revised: The output of this projection is usually named patch embeddings.

Original: Then, the learnable position embeddings are added to the patch embeddings to encode positional information of each patch in the image.
Revised: Then, the learnable position embeddings are added to the patch embeddings to encode positional information of each patch in the image.

Original: The output sequence of embedding vectors z 0 serves as the input to the Transformer encoder.
Revised: The output sequence of embedding vectors z 0 serves as the input to the Transformer encoder.

Original: The Transformer encoder [1] consists of alternating layers of multi-head self-attention (MSA) and
Revised: The Transformer encoder [1] consists of alternating layers of multi-head self-attention (MSA) and

Original: Feed-Forward Networks (FFN).
Revised: Feed-Forward Networks (FFN).

Original: LayerNorm (LN) [20] is applied before every block, and residual connections after every block.
Revised: LayerNorm (LN) [20] is applied before every block, and residual connections after every block.

Original: For a Transformer encoder with L blocks, the output representation is computed following the equations:
Revised: For a Transformer encoder with L blocks, the output representation is computed following the equations:

Original: [Equation]
Revised: [Equation]

Original: The FFN is composed of two linear layers separated by a GELU activation [21].
Revised: The FFN is composed of two linear layers separated by a GELU activation [21].

Original: The first linear layer expands the dimension from D to 4 D , and the second one reduces the dimension from 4 D back to D .
Revised: The first linear layer expands the dimension from D to 4 D , and the second one reduces the dimension from 4 D back to D .

Original: Locality in ViT architecture.
Revised: Locality in ViT architecture.

Original: The self-attention layer of a ViT captures global dependencies between all of the patches, but it lacks locality inductive bias, in particular a mechanism to allow information exchange within a local region.
Revised: The self-attention layer of a ViT captures global dependencies between all of the patches, but it lacks locality inductive bias, in particular a mechanism to allow information exchange within a local region.

Original: In order to introduce locality into vision Transformers, we only adapt the FFNs while the other parts, such as self-attention and position encoding, are not changed.
Revised: In order to introduce locality into vision Transformers, we only adapt the FFNs while the other parts, such as self-attention and position encoding, are not changed.

Original: We propose to add locality to the ViT architecture by adding a depth-wise convolution layer into each FFN.
Revised: We propose to add locality to the ViT architecture by adding a depth-wise convolution layer into each FFN.

Original: A 3 × 3 depth-wise convolution is added between the two fully-connected layers in the FFN (Figure 1).
Revised: A 3 × 3 depth-wise convolution is added between the two fully-connected layers in the FFN (Figure 1).

Original: Before each 3 × 3 depth-wise convolution, a sequence-to-image (Seq2Im) layer is used to convert each flattened patch representation into a 2D patch representation.
Revised: Before each 3 × 3 depth-wise convolution, a sequence-to-image (Seq2Im) layer is used to convert each flattened patch representation into a 2D patch representation.

Original: Similarly, an image-to-sequence (Im2Seq) layer is used to convert each 2D patch representation into a flattened patch representation.
Revised: Similarly, an image-to-sequence (Im2Seq) layer is used to convert each 2D patch representation into a flattened patch representation.

Original: We also replace the GELU activation layer [21] by the h-swish [22].
Revised: We also replace the GELU activation layer [21] by the h-swish [22].

Original: Connection with existing works.
Revised: Connection with existing works.

Original: Other works [5, 23, 9, 10, 11] explore adding locality in ViT architecture.
Revised: Other works [5, 23, 9, 10, 11] explore adding locality in ViT architecture.

Original: Most of them analyze the impact of locality mechanism on the final accuracy, but to the best of our knowledge, none of them study the impact of locality mechanism on the training speed.
Revised: Most of them analyze the impact of locality mechanism on the final accuracy, but to the best of our knowledge, none of them study the impact of locality mechanism on the training speed.

Original: The closest work to our architecture is probably LocalViT [23], which also uses convolutions in the FFN.
Revised: The closest work to our architecture is probably LocalViT [23], which also uses convolutions in the FFN.

Original: There are 3 main differences between LocalViT and our model.
Revised: There are 3 main differences between LocalViT and our model.

Original: First, our architecture uses LayerNorm [20] as a normalization layer, whereas LocalViT uses 2D BatchNorm [24].
Revised: First, our architecture uses LayerNorm [20] as a normalization layer, whereas LocalViT uses 2D BatchNorm [24].

Original: Second, the expanding and squeezing layers are implemented as fully-connected layer in our architecture, whereas LocalViT uses convolution layers.
Revised: Second, the expanding and squeezing layers are implemented as fully-connected layer in our architecture, whereas LocalViT uses convolution layers.

Original: Finally, our architecture uses h-swish [22] as activation layer, whereas LocalViT uses a combination of h-swish and squeeze-and-excitation module [25].
Revised: Finally, our architecture uses h-swish [22] as activation layer, whereas LocalViT uses a combination of h-swish and squeeze-and-excitation module [25].

Original: We observe our contributions are important and lead to a more efficient architecture (see subsection 3.2).
Revised: We observe our contributions are important and lead to a more efficient architecture (see subsection 3.2).

Original: 2.2 Image size-based curriculum learning
Revised: 2.2 Image size-based curriculum learning

Original: Training vision Transformers is traditionally done by using mini-batches of 224 × 224 RGB images sampled uniformly from the training data.
Revised: Training vision Transformers is traditionally done by using mini-batches of 224 × 224 RGB images sampled uniformly from the training data.

Original: Each image is usually decomposed to non-overlapping 16 × 16 patches, so the input of vision Transformers is usually a sequence of 196 flattened patches.
Revised: Each image is usually decomposed to non-overlapping 16 × 16 patches, so the input of vision Transformers is usually a sequence of 196 flattened patches.

Original: The complexity of a vanilla Transformer architecture [1] is quadratic to the sequence length ( i.e. number of patches) due to the attention mechanism.
Revised: The complexity of a vanilla Transformer architecture [1] is quadratic to the sequence length ( i.e. number of patches) due to the attention mechanism.

Original: In this section, we explore an approach to reduce the sequence length ( i.e. number of patches) to speed-up training.
Revised: In this section, we explore an approach to reduce the sequence length ( i.e. number of patches) to speed-up training.

Original: We develop a small to large image size-based curriculum learning strategy [26], where shorter patch sequences are used at the beginning of the training.
Revised: We develop a small to large image size-based curriculum learning strategy [26], where shorter patch sequences are used at the beginning of the training.

Original: The key idea of curriculum learning [26] is to start small and learn easier aspects of the task, then gradually increase the difficulty level.
Revised: The key idea of curriculum learning [26] is to start small and learn easier aspects of the task, then gradually increase the difficulty level.

Original: There are different ways to use curriculum learning, but a popular one is to start training with easy examples, and then gradually adding more difficult examples [27, 28].
Revised: There are different ways to use curriculum learning, but a popular one is to start training with easy examples, and then gradually adding more difficult examples [27, 28].

Original: We use the image size as a proxy of the image difficulty.
Revised: We use the image size as a proxy of the image difficulty.

Original: At the beginning of the training, the vision Transformer model is trained with low resolution images, and then image resolution is gradually increased every few epochs.
Revised: At the beginning of the training, the vision Transformer model is trained with low resolution images, and then image resolution is gradually increased every few epochs.

Original: We do this by resizing the input images.
Revised: We do this by resizing the input images.

Original: Figure 1 shows different image sizes ( i.e. steps of the curriculum learning) for a given image.
Revised: Figure 1 shows different image sizes ( i.e. steps of the curriculum learning) for a given image.

Original: In each epoch, all the images have the same size, but the image size can increase between epochs.
Revised: In each epoch, all the images have the same size, but the image size can increase between epochs.

Original: Then, a critical question is how to design a good strategy to increase the image size.
Revised: Then, a critical question is how to design a good strategy to increase the image size.

Original: First, it is important to define the initial image size i.e. the image size for the first epoch.
Revised: First, it is important to define the initial image size i.e. the image size for the first epoch.

Original: Then, it is important to control when the image size increases.
Revised: Then, it is important to control when the image size increases.

Original: We use a linear rule that increases the image size by M pixels every N epochs.
Revised: We use a linear rule that increases the image size by M pixels every N epochs.

Original: In the experimental section, we analyze the impact of these hyper-parameters.
Revised: In the experimental section, we analyze the impact of these hyper-parameters.

Original: By construction, all the layers in the vision Transformer architecture, except the positional embeddings, can adapt automatically to multiple sequence lengths.
Revised: By construction, all the layers in the vision Transformer architecture, except the positional embeddings, can adapt automatically to multiple sequence lengths.

Original: The positional embeddings are updated by interpolation after each image size increase.
Revised: The positional embeddings are updated by interpolation after each image size increase.

Original: To avoid to deal with partial patches, we only use image sizes that can be decomposed in 16 × 16 patches.
Revised: To avoid to deal with partial patches, we only use image sizes that can be decomposed in 16 × 16 patches.

Original: Using multiple image sizes during training can also help to learn better scale invariant representations.
Revised: Using multiple image sizes during training can also help to learn better scale invariant representations.

Original: [Table]
Revised: [Table]

Original:  Experimental result
Revised:  Experimental result

Original: This section gives experimental results for image classification on ImageNet1k, which is a well-known benchmark widely used to evaluate image classification models.
Revised: This section gives experimental results for image classification on ImageNet1k, which is a well-known benchmark widely used to evaluate image classification models.

Original: We show our results, and analyze the impact of our contributions.
Revised: We show our results, and analyze the impact of our contributions.

Original: The implementation details can be found in Appendix A.
Revised: The implementation details can be found in Appendix A.

Original: 3.1 Results
Revised: 3.1 Results

Original: We evaluate models trained from scratch and in less than 24 hours, with a single GPU on ImageNet1k.
Revised: We evaluate models trained from scratch and in less than 24 hours, with a single GPU on ImageNet1k.

Original: To the best of our knowledge, there are no existing results for this benchmark
Revised: To the best of our knowledge, there are no existing results for this benchmark

Original: so we define some baselines.
Revised: so we define some baselines.

Original: We trained DeiT-S and LocalViT-S models for 24 hours with a single GPU by using their official implementations.
Revised: We trained DeiT-S and LocalViT-S models for 24 hours with a single GPU by using their official implementations.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: We observe our model outperforms other models by a large margin.
Revised: We observe our model outperforms other models by a large margin.

Original: There is an improvement of about 11 pt w.r.t DeiT-S, and about 26 pt w.r.t
Revised: There is an improvement of about 11 pt w.r.t DeiT-S, and about 26 pt w.r.t

Original: LocalViT.
Revised: LocalViT.

Original: We also report the DeiT-S results by using the original resources (4 GPUs and 3 days) as an upper-bound.
Revised: We also report the DeiT-S results by using the original resources (4 GPUs and 3 days) as an upper-bound.

Original: We can see there is still a large gap between this upper bound and our model, but we also see a large improvement w.r.t
Revised: We can see there is still a large gap between this upper bound and our model, but we also see a large improvement w.r.t

Original: the baseline DeiT-S. These experiments show adding locality in the Transformer mechanism and using an image size curriculum learning help to speed-up training in a limited resource setting.
Revised: the baseline DeiT-S. These experiments show adding locality in the Transformer mechanism and using an image size curriculum learning help to speed-up training in a limited resource setting.

Original: The next sections analyze the impact of each of these contributions.
Revised: The next sections analyze the impact of each of these contributions.

Original: 3.2 Analysis of the locality mechanism
Revised: 3.2 Analysis of the locality mechanism

Original: In this section, we analyze solely the impact of our locality mechanism as proposed in subsection 2.
Revised: In this section, we analyze solely the impact of our locality mechanism as proposed in subsection 2.

Original: (without using the image size-based curriculum learning) and compare with DeiT-S [4] and LocalViTS [23].
Revised: (without using the image size-based curriculum learning) and compare with DeiT-S [4] and LocalViTS [23].

Original: Figure 2 shows the results on the validation set of ImageNet1k for each training epoch.
Revised: Figure 2 shows the results on the validation set of ImageNet1k for each training epoch.

Original: Each model is trained for a maximum of 24 hours on a single GPU.
Revised: Each model is trained for a maximum of 24 hours on a single GPU.

Original: Interestingly, we can see adding the locality mechanism does not significantly increase the training time per epoch w.r.t DeiT-S, but it increases the top-1 accuracy by 6 . 2 pt.
Revised: Interestingly, we can see adding the locality mechanism does not significantly increase the training time per epoch w.r.t DeiT-S, but it increases the top-1 accuracy by 6 . 2 pt.

Original: We observe the training time per epoch of LocalViT is about
Revised: We observe the training time per epoch of LocalViT is about

Original: 2.4 times our model’s, so its training stops after 6 epochs.
Revised: 2.4 times our model’s, so its training stops after 6 epochs.

Original: An advantage of our locality mechanism is that it can get good performance when using quite small batch sizes, because it uses LayerNorm instead of BatchNorm.
Revised: An advantage of our locality mechanism is that it can get good performance when using quite small batch sizes, because it uses LayerNorm instead of BatchNorm.

Original: We also found using h-swish instead of GELU improves the performance by about 1 pt.
Revised: We also found using h-swish instead of GELU improves the performance by about 1 pt.

Original: This analysis brings evidence that our locality mechanism can help speed-up training, and as a result reduce the training cost.
Revised: This analysis brings evidence that our locality mechanism can help speed-up training, and as a result reduce the training cost.

Original: 3.3 Image size-based curriculum learning
Revised: 3.3 Image size-based curriculum learning

Original: In this section, we analyze the image size-based curriculum learning introduced in subsection 2.2.
Revised: In this section, we analyze the image size-based curriculum learning introduced in subsection 2.2.

Original: The proposed curriculum learning has three important parameters: the initial image size, the image size increase, and the number of epochs between each image size increase.
Revised: The proposed curriculum learning has three important parameters: the initial image size, the image size increase, and the number of epochs between each image size increase.

Original: Figure 2 shows the results for different configurations.
Revised: Figure 2 shows the results for different configurations.

Original: The best result was obtained by starting training from 32 ×pixel images, and increasing the image size along each axis by 32 pixels after each 5 epochs of training until we reach the final image size ( 224 pixels).
Revised: The best result was obtained by starting training from 32 ×pixel images, and increasing the image size along each axis by 32 pixels after each 5 epochs of training until we reach the final image size ( 224 pixels).

Original: We can see that adding the image size-based curriculum learning to our model on top of the locality mechanism improves the top-1 accuracy by about 4 . 5 pt.
Revised: We can see that adding the image size-based curriculum learning to our model on top of the locality mechanism improves the top-1 accuracy by about 4 . 5 pt.

Original: Using small images allows training for more epochs because the training time per epoch is smaller.
Revised: Using small images allows training for more epochs because the training time per epoch is smaller.

Original:  Conclusion
Revised:  Conclusion

Original: Our experimental results on ImageNet1k bring evidence towards a positive impact of the locality mechanism and the image size-based curriculum learning to reduce the training cost of a vision
Revised: Our experimental results on ImageNet1k bring evidence towards a positive impact of the locality mechanism and the image size-based curriculum learning to reduce the training cost of a vision

Original: Transformer model.
Revised: Transformer model.

Original: It is likely that the findings can be generalized to other Transformer architectures but more experiments will have to be performed to validate this idea.Checklist
Revised: It is likely that the findings can be generalized to other Transformer architectures but more experiments will have to be performed to validate this idea.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [No] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] Our work does not seem to have potential negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The code and instructions will be released before NeurIPS if this work is accepted to the workshop.

Original: The data is public.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Appendix A (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Information is available in Appendix A 4.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Implementation Details
Revised: A Implementation Details

Original: In this appendix, we describe the implementation details of our model training.
Revised: In this appendix we describe the implementation details of our model training

Original: The ImageNet1k dataset [29] contains 1.28M training images and 50K validation images from one thousand classes.
Revised: The ImageNet1k dataset [29] contains 1.28M training images and 50K validation images from one thousand classes.

Original: Otherwise specified, we report model performances on the official validation split to follow [23, 4] protocol.
Revised: Otherwise specified, we report model performances on the official validation split to follow [23, 4] protocol.

Original: We use ImageNet1k because it is a well-known benchmark, which is widely used to evaluate image classification models.
Revised: We use ImageNet1k because it is a well-known benchmark, which is widely used to evaluate image classification models.

Original: Our implementation is based on the timm library [30] and DeiT [4].
Revised: Our implementation is based on the timm library [30] and DeiT [4].

Original: We use DeiT -S model as the main baseline in our experiments.
Revised: We use DeiT-S model as the main baseline in our experiments.

Original: To get fair comparison, we only compare Transformer with small architectures because they have similar complexity and number of parameters.
Revised: To get fair comparison, we only compare Transformer with small architectures because they have similar complexity and number of parameters.

Original: Our training protocol is quite similar to DeiT, with some changes.
Revised: Our training protocol is quite similar to DeiT, with some changes.

Original: The training continues until the 24 hours time limit is reached.
Revised: The training continues until the 24 hours time limit is reached.

Original: The AdamW optimizer is used with a momentum of 0.9.
Revised: The AdamW optimizer is used with a momentum of 0.9.

Original: The batch size is set to 64 and the learning rate is set to 1 e − 3 .
Revised: The batch size is set to 64 and the learning rate is set to 1 e − 3 .

Original: The model is trained with a cross-entropy loss and by using label smoothing.
Revised: The model is trained with a cross-entropy loss and by using label smoothing.

Original: We do not perform learning rate warm-up.
Revised: We do not perform learning rate warm-up.

Original: We use a Quadro RTX 5000 with 16GB, hosted in an internal cluster, to run our experiments.
Revised: We use a Quadro RTX 5000 with 16GB, hosted in an internal cluster, to run our experiments.

Original: We resize the images by using a bilinear interpolation.
Revised: We resize the images by using a bilinear interpolation.

Original: Assets.
Revised: Assets.

Original: In this project, we use the following existing assets:
Revised: In this project, we use the following existing assets:

Original: • timm ( https://github.com/rwightman/pytorch-image-models ), Apache-2.0 license • DeiT ( https://github.com/facebookresearch/deit ), Apache 2.0 license • LocalViT ( https://github.com/ofsoundof/LocalViT ), MIT license • ImageNet1k ( https://image-net.org/download.php )
Revised: • timm ( https://github.com/rwightman/pytorch-image-models ), Apache-2.0 license • DeiT ( https://github.com/facebookresearch/deit ), Apache 2.0 license • LocalViT ( https://github.com/ofsoundof/LocalViT ), MIT license • ImageNet1k ( https://image-net.org/download.php )