Original: AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Revised: AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Original: Abstract
Revised: Abstract

Original: Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition.
Revised: Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition.

Original: A following scenario is to adapt a ViT to various image and video recognition tasks.
Revised: A following scenario is to adapt a ViT to various image and video recognition tasks.

Original: The adaptation is challenging because of heavy computation and memory storage.
Revised: The adaptation is challenging because of heavy computation and memory storage.

Original: Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains.
Revised: Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains.

Original: To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently.
Revised: To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently.

Original: It possesses several benefits more appealing than prior arts.
Revised: It possesses several benefits more appealing than prior arts.

Original: Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT’s transferability without updating its original pre-trained parameters, significantly outperforming the existing 100% fully fine-tuned models on action recognition benchmarks.
Revised: Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT’s transferability without updating its original pre-trained parameters, significantly outperforming the existing 100% fully fine-tuned models on action recognition benchmarks.

Original: Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks.
Revised: Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks.

Original: Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains.
Revised: Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains.

Original: For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something v2 and HMDB51, respectively.
Revised: For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something v2 and HMDB51, respectively.

Original: Code is available at https://github.com/ShoufaChen/AdaptFormer .
Revised: Code is available at https://github.com/ShoufaChen/AdaptFormer .

Original:  Introduction
Revised:  Introduction

Original: There is a growing interest in adopting a general neural model to tackle a large variety of different tasks since it benefits in reducing the need for task-specific model design and training.
Revised: There is a growing interest in adopting a general neural model to tackle a large variety of different tasks since it benefits in reducing the need for task-specific model design and training.

Original: Recently,
Revised: Recently,

Original: Transformer [81] demonstrates great potential in this goal considering its success in various fields, e.g. , natural language processing (NLP) [27, 10, 82, 88], visual recognition [31, 79, 90, 63], dense prediction [83, 11, 98, 96, 86], Generative Adversarial Network (GAN) [52, 48], reinforcement learning (RL) [18, 16, 87], robotics [50, 25], and etc.
Revised: Transformer [81] demonstrates great potential in this goal considering its success in various fields, e.g. , natural language processing (NLP) [27, 10, 82, 88], visual recognition [31, 79, 90, 63], dense prediction [83, 11, 98, 96, 86], Generative Adversarial Network (GAN) [52, 48], reinforcement learning (RL) [18, 16, 87], robotics [50, 25], and etc.

Original: However, existing literature in computer vision tend to focus on the same network with task-specific weights scenario, where a single network is used to train from scratch or fully fine-tune on a specific dataset, making it infeasible to maintain a separate model weight for every dataset when the number of task grows, especially for the increasing model capacity of state-of-the-art models ( e.g. , ViT-G/14 [93] with over 1.8 billion parameters).
Revised: However, existing literature in computer vision tend to focus on the same network with task-specific weights scenario, where a single network is used to train from scratch or fully fine-tune on a specific dataset, making it infeasible to maintain a separate model weight for every dataset when the number of task grows, especially for the increasing model capacity of state-of-the-art models ( e.g. , ViT-G/14 [93] with over 1.8 billion parameters).

Original: Different from prior arts, we step into the direction of developing same network with almost same weights and achieve superior performance than the full-tuning approach by only tuning less than 2% parameters, with the remaining over 98% parameters shared across different tasks.
Revised: Different from prior arts, we step into the direction of developing same network with almost same weights and achieve superior performance than the full-tuning approach by only tuning less than 2% parameters, with the remaining over 98% parameters shared across different tasks.

Original: There are two challenges to learning universal representations using a single model.
Revised: There are two challenges to learning universal representations using a single model.

Original: The first one lies in the pre-training stage, which requires algorithms that can learn well-generalized representations that are ∗ Equal contribution.
Revised: The first one lies in the pre-training stage, which requires algorithms that can learn well-generalized representations that are ∗ Equal contribution.

Original: easy to be applied to many tasks.
Revised: easy to be applied to many tasks.

Original: Recent arts in self-supervised learning [12, 5, 43, 97, 85, 78, 35] can serve as a solution to this challenge.
Revised: Recent arts in self-supervised learning [12, 5, 43, 97, 85, 78, 35] can serve as a solution to this challenge.

Original: The second one, which is our main concern in this work, is to build an effective pipeline that can adapt the model obtained at the pre-training stage to various downstream tasks by tuning parameters as less as possible and keeping the left parameters frozen.
Revised: The second one, which is our main concern in this work, is to build an effective pipeline that can adapt the model obtained at the pre-training stage to various downstream tasks by tuning parameters as less as possible and keeping the left parameters frozen.

Original: While fine-tuning pre-trained models has been widely studied in NLP [6, 46, 69, 70, 58, 56, 47, 92, 62, 42], this topic is seldomly explored in the vision, where full tine-tuning of model parameters is still the dominant strategy for adapting vision transformers.
Revised: While fine-tuning pre-trained models has been widely studied in NLP [6, 46, 69, 70, 58, 56, 47, 92, 62, 42], this topic is seldomly explored in the vision, where full-tuning of model parameters is still the dominant strategy for adapting vision transformers.

Original: However, the full fine-tuning cannot satisfy the goal of universal representation as it assigns an independent set of weights for every task.
Revised: However, the full fine-tuning cannot satisfy the goal of universal representation as it assigns an independent set of weights for every task.

Original: Linear probing is a straightforward approach to maintaining the pre-trained model fixed by only tuning a specific lightweight classification head for every task.
Revised: Linear probing is a straightforward approach to maintaining the pre-trained model fixed by only tuning a specific lightweight classification head for every task.

Original: However, linear probing tends to have an unsatisfactory performance and misses the opportunity of pursuing strong but non-linear features [43], which indeed benefit deep learning.
Revised: However, linear probing tends to have an unsatisfactory performance and misses the opportunity of pursuing strong but non-linear features [43], which indeed benefit deep learning.

Original: More recently, Bahng et.al.
Revised: More recently, Bahng et.al.

Original: , [4] aimed to adapt pre-trained models by modifying raw input pixel space.
Revised: , [4] aimed to adapt pre-trained models by modifying raw input pixel space.

Original: Jia et.al.
Revised: Jia et.al.

Original: , [51] proposed Visual Prompt
Revised: , [51] proposed Visual Prompt

Original: Tuning (VPT) to adapt transformer models for downstream vision tasks, which prepends several learnable parameters ( prompts ) to the patch embeddings and freezes the whole pre-trained backbone.
Revised: Tuning (VPT) to adapt transformer models for downstream vision tasks, which prepends several learnable parameters ( prompts ) to the patch embeddings and freezes the whole pre-trained backbone.

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose a lightweight module, namely AdaptFormer, to adapt vision transformers by updating the weights of AdaptFormer.
Revised: In this work, we propose a lightweight module, namely AdaptFormer, to adapt vision transformers by updating the weights of AdaptFormer.

Original: We introduce learnable parameters from the model perspective, which is different from VPT, which inserts learnable parameters into the token space.
Revised: We introduce learnable parameters from the model perspective, which is different from VPT, which inserts learnable parameters into the token space.

Original: Our AdaptFormer is conceptually simple yet effective.
Revised: Our AdaptFormer is conceptually simple yet effective.

Original: It consists of two fully connected layers, a non-linear activation function, and a scaling factor.
Revised: It consists of two fully connected layers, a non-linear activation function, and a scaling factor.

Original: This module is set in parallel to the feed-forward network (FFN) of the original ViT model, as shown in Figure 2b.
Revised: This module is set in parallel to the feed-forward network (FFN) of the original ViT model, as shown in Figure 2b.

Original: This design is turned out to be effective for model transfer when processing scalable visual tokens for both image and video data (i.e., image data consists of a small scale of visual tokens while video data consists of a large scale).
Revised: This design is turned out to be effective for model transfer when processing scalable visual tokens for both image and video data (i.e., image data consists of a small scale of visual tokens while video data consists of a large scale).

Original: As shown in
Revised: As shown in

Original: Figure 1, compared with the full-tuning strategy, AdaptFormer achieves comparable performance on video recognition with only about 0.1% tunable parameters.
Revised: Figure 1, compared with the full-tuning strategy, AdaptFormer achieves comparable performance on video recognition with only about 0.1% tunable parameters.

Original: Meanwhile, with less than 2% tunable parameters, AdaptFormer surpasses the full-tuning solution by about 10% on top-1 accuracy.
Revised: Meanwhile, with less than 2% tunable parameters, AdaptFormer surpasses the full-tuning solution by about 10% on top-1 accuracy.

Original: Similar approaches are also proposed in fine-tuning pre-trained language models (PLMs) [6, 46, 70, 42].
Revised: Similar approaches are also proposed in fine-tuning pre-trained language models (PLMs) [6, 46, 70, 42].

Original: The key contributions of this paper are summarized as follows: (1) We propose a simple yet effective framework, namely AdaptFormer, for adapting vision transformers to a large variety of downstream visual recognition tasks and avoiding catastrophic interference with each other.
Revised: The key contributions of this paper are summarized as follows: (1) We propose a simple yet effective framework, namely AdaptFormer, for adapting vision transformers to a large variety of downstream visual recognition tasks and avoiding catastrophic interference with each other.

Original: To the best of our knowledge, this is the first work that explores efficient fine-tuning in video action recognition.
Revised: To the best of our knowledge, this is the first work that explores efficient fine-tuning in video action recognition.

Original: (2) We ablate many design choices and demonstrate the superior robustness of AdaptFormer when parameters scale up.
Revised: (2) We ablate many design choices and demonstrate the superior robustness of AdaptFormer when parameters scale up.

Original: (3) Extensive experiments on various downstream tasks demonstrate that
Revised: (3) Extensive experiments on various downstream tasks demonstrate that

Original: AdaptFormer outperforms existing fine-tuning approaches significantly.
Revised: AdaptFormer outperforms existing fine-tuning approaches significantly.

Original: By demonstrating the effectiveness of AdaptFormer on multiple visual benchmarks, we hope our work could inspire the research communities to rethink the fine-tuning mechanism in computer vision and make progress toward a flexible yet universal Transformer model for visual recognition.
Revised: By demonstrating the effectiveness of AdaptFormer on multiple visual benchmarks, we hope our work could inspire the research communities to rethink the fine-tuning mechanism in computer vision and make progress toward a flexible yet universal Transformer model for visual recognition.

Original:  Related Works
Revised:  Related Works

Original: In the proposed AdaptFormer, we mainly introduce a plug-and-play module for efficiently fine-tuning the current vision Transformer models.
Revised: In the proposed AdaptFormer, we mainly introduce a plug-and-play module for efficiently fine-tuning the current vision Transformer models.

Original: In this section, we perform a literature review on related works from two perspectives, i.e. , the vision Transformers, and efficient transfer learning for vision
Revised: In this section, we perform a literature review on related works from two perspectives, i.e. , the vision Transformers, and efficient transfer learning for vision

Original: Transformers.
Revised: Transformers.

Original: 2.1 Transformer in Vision
Revised: 2.1 Transformer in Vision

Original: The Transformer architecture is first introduced in [81] and has re-energized the natural language processing (NLP) field from then on [27, 10].
Revised: The Transformer architecture is first introduced in [81] and has re-energized the natural language processing (NLP) field from then on [27, 10].

Original: Inspired by its huge success, researches in the computer vision filed have also evolved into Transformer era since ViTs [31].
Revised: Inspired by its huge success, researches in the computer vision filed have also evolved into Transformer era since ViTs [31].

Original: The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including image classification [31, 63, 60], object detection [11, 98, 22], semantic/instance segmentation [86], video understanding [8, 2, 33, 57], point cloud modeling [95, 41], 3D Object Recognition [20] and even low-level processing [17, 59, 84].
Revised: The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including image classification [31, 63, 60], object detection [11, 98, 22], semantic/instance segmentation [86], video understanding [8, 2, 33, 57], point cloud modeling [95, 41], 3D Object Recognition [20] and even low-level processing [17, 59, 84].

Original: Furthermore, transformers have advanced the vision recognition performance by a large-scale pretraining [21, 67, 13, 36, 43, 78, 71].
Revised: Furthermore, transformers have advanced the vision recognition performance by a large-scale pretraining [21, 67, 13, 36, 43, 78, 71].

Original: In such a situation, given the pre-trained Transformer models, which are more larger than the previously prevalent CNN backbones, one open question is how to fine-tune the big vision models so that they can be adapted into more specific down-stream tasks.
Revised: In such a situation, given the pre-trained Transformer models, which are more larger than the previously prevalent CNN backbones, one open question is how to fine-tune the big vision models so that they can be adapted into more specific down-stream tasks.

Original: To solve the open question, we propose AdaptFormer to transfer ViTs from the pre-trained pre-texts into the target tasks in a more effective and efficient way.
Revised: To solve the open question, we propose AdaptFormer to transfer ViTs from the pre-trained pre-texts into the target tasks in a more effective and efficient way.

Original: 2.2 Efficient Transfer learning for Transformers
Revised: 2.2 Efficient Transfer learning for Transformers

Original: Transfer learning targets re-adopting a pre-trained model (either via the supervised or the unsupervised manner) as the starting point and further fine-tuning the specific model on a new task.
Revised: Transfer learning targets re-adopting a pre-trained model (either via the supervised or the unsupervised manner) as the starting point and further fine-tuning the specific model on a new task.

Original: In the NLP field, transferring the large pre-trained language models (PLMs) [27, 10] into downstream tasks has been the popular paradigm for a long time.
Revised: In the NLP field, transferring the large pre-trained language models (PLMs) [27, 10] into downstream tasks has been the popular paradigm for a long time.

Original: Conventional arts [27, 10] set all the network parameters as learnable ones and adapt them to the target tasks.
Revised: Conventional arts [27, 10] set all the network parameters as learnable ones and adapt them to the target tasks.

Original: However, with the growth of model sizes and the complexity of the specific tasks, the conventional paradigm is inevitably limited by the huge computational burden.
Revised: However, with the growth of model sizes and the complexity of the specific tasks, the conventional paradigm is inevitably limited by the huge computational burden.

Original: The NLP community has explored several ways for parameter-efficient transfer learning that only set a few parameters learnable and fine-tune them for efficiency.
Revised: The NLP community has explored several ways for parameter-efficient transfer learning that only set a few parameters learnable and fine-tune them for efficiency.

Original: The pioneer works could be mainly categorized from the token [58, 56] and network perspectives [46, 47, 92, 40].
Revised: The pioneer works could be mainly categorized from the token [58, 56] and network perspectives [46, 47, 92, 40].

Original: Basically speaking, the token-related methods [56, 58] typically prepend several learnable prefix vectors/tokens to the projected tokens within the multi-head self-attention layers (MHSA [81]).
Revised: Basically speaking, the token-related methods [56, 58] typically prepend several learnable prefix vectors/tokens to the projected tokens within the multi-head self-attention layers (MHSA [81]).

Original: The philosophy behind it is to assist the pre-trained models in understanding downstream tasks with the guidance of extra token information.
Revised: The philosophy behind it is to assist the pre-trained models in understanding downstream tasks with the guidance of extra token information.

Original: On the other hand, network-related methods [46, 47] integrate shallow modules to improve the model transferability.
Revised: On the other hand, network-related methods [46, 47] integrate shallow modules to improve the model transferability.

Original: The introduced modules adapt the produced representations into the downstream tasks via features fusion.
Revised: The introduced modules adapt the produced representations into the downstream tasks via features fusion.

Original: Recently, with the emergence of a much more large-scale dataset [26, 72, 74, 66, 53], increasing researchers in computer vision have adopted the homologous paradigm, i.e. , first pre-training and then fine-tuning, to advance the vision tasks.
Revised: Recently, with the emergence of a much more large-scale dataset [26, 72, 74, 66, 53], increasing researchers in computer vision have adopted the homologous paradigm, i.e. , first pre-training and then fine-tuning, to advance the vision tasks.

Original: As for the second stage, traditional methods typically adopt the full-tuning arts in the downstream tasks.
Revised: As for the second stage, traditional methods typically adopt the full-tuning arts in the downstream tasks.

Original: Rare attention has been drawn to the field of efficient adaptation, especially in the field of vision Transformers.
Revised: Rare attention has been drawn to the field of efficient adaptation, especially in the field of vision Transformers.

Original: Inspired by Prompting in NLP, [51] introduced the learnable tokens in exploring the efficient adaptation for ViTs.
Revised: Inspired by Prompting in NLP, [51] introduced the learnable tokens in exploring the efficient adaptation for ViTs.

Original: We empirically found that the performance of prompting is hindered by the scale of tokens.
Revised: We empirically found that the performance of prompting is hindered by the scale of tokens.

Original: That is to say, for the tasks where the number of tokens is on a small scale, e.g. , image classification, Prompting is efficient for improving the model transferability.
Revised: That is to say, for the tasks where the number of tokens is on a small scale, e.g. , image classification, Prompting is efficient for improving the model transferability.

Original: However, for larger scale tokens, e.g. , video understanding,
Revised: However, for larger scale tokens, e.g. , video understanding,

Original: Prompting presents limited potential.
Revised: Prompting presents limited potential.

Original: This observation motivates us to introduce AdaptFormer, which is effective in the scenarios of scalable visual tokens.
Revised: This observation motivates us to introduce AdaptFormer, which is effective in the scenarios of scalable visual tokens.

Original:  Approach
Revised:  Approach

Original: We propose AdaptFormer for efficiently transferring large pre-trained vision transformer models to downstream tasks, in both image and video domains.
Revised: We propose AdaptFormer for efficiently transferring large pre-trained vision transformer models to downstream tasks, in both image and video domains.

Original: AdaptFormer attains strong transfer learning abilities by only fine-tuning a small number of extra parameters, circumventing catastrophic interference among tasks.
Revised: AdaptFormer attains strong transfer learning abilities by only fine-tuning a small number of extra parameters, circumventing catastrophic interference among tasks.

Original: We illustrate the overall framework of AdaptFormer in Figure 2b.
Revised: We illustrate the overall framework of AdaptFormer in Figure 2b.

Original: 3.1 Preliminary and Notation
Revised: 3.1 Preliminary and Notation

Original: Vision Transformers (ViTs) are first introduced by [31] into vision recognition.
Revised: Vision Transformers (ViTs) are first introduced by [31] into vision recognition.

Original: A vanilla vision
Revised: A vanilla vision

Original: Transformer basically consists of a patch embedding layer and several consecutively connected encoders, as depicted in Figure 2a.
Revised: Transformer basically consists of a patch embedding layer and several consecutively connected encoders, as depicted in Figure 2a.

Original: Given an image x ∈ R H × W × 3 , the patch embedding layer first splits and flatten the sample x into sequential patches x p ∈ R N ×
Revised: Given an image x ∈ R H × W × 3 , the patch embedding layer first splits and flatten the sample x into sequential patches x p ∈ R N ×

Original: ( P 2 d ) , where ( H , W ) represents the height and width of the input image, ( P, P ) is the resolution of each image patch, d denotes
Revised: ( P 2 d ) , where ( H , W ) represents the height and width of the input image, ( P, P ) is the resolution of each image patch, d denotes

Original: [Figure]
Revised: [Figure]

Original: Each Transformer encoder mainly consists of two types of sub-layers, i.e. , a multi-head self-attention layer (MHSA) and a MLP layer.
Revised: Each Transformer encoder mainly consists of two types of sub-layers, i.e. , a multi-head self-attention layer (MHSA) and a MLP layer.

Original: In MHSA, the tokens are linearly projected and further re-formulated into three vectors, namely Q , K and V .
Revised: In MHSA, the tokens are linearly projected and further re-formulated into three vectors, namely Q , K and V .

Original: The self-attention calculation is performed on Q , K and V by:
Revised: The self-attention calculation is performed on Q , K and V by:

Original: [Equation]
Revised: [Equation]

Original: where x ′ ℓ are the tokens produced by MHSA at the ℓ -th layer.
Revised: where x ′ ℓ are the tokens produced by MHSA at the ℓ -th layer.

Original: The output tokens x ′ ℓ are further sent to a LayerNorm [3] and a MLP block which is consisted of two fully connected layers with a GELU activation [45] in between.
Revised: The output tokens x ′ ℓ are further sent to a LayerNorm [3] and a MLP block which is consisted of two fully connected layers with a GELU activation [45] in between.

Original: This process is formally formulated as follows,
Revised: This process is formally formulated as follows,

Original: [Equation]
Revised: [Equation]

Original: where x ℓ is the output of the ℓ -th encoder block.
Revised: where x ℓ is the output of the ℓ -th encoder block.

Original: At the last transformer layer, the [CLS] is utilized for the final object recognition.
Revised: At the last transformer layer, the [CLS] is utilized for the final object recognition.

Original: We refer the readers to find more details in [31].
Revised: We refer the readers to find more details in [31].

Original: In our work, we replace the MLP layer with our AdaptMLP module for efficient fine-tuning purposes.
Revised: In our work, we replace the MLP layer with our AdaptMLP module for efficient fine-tuning purposes.

Original: 3.2 AdaptFormer
Revised: 3.2 AdaptFormer

Original: We propose a plug-and-play bottleneck module, namely AdaptMLP 2 .
Revised: We propose a plug-and-play bottleneck module, namely AdaptMLP 2 .

Original: We denote the vision Transformer equipped with AdaptMLP as AdaptFormer.
Revised: We denote the vision Transformer equipped with AdaptMLP as AdaptFormer.

Original: Architecture.
Revised: Architecture.

Original: The design principle of AdaptFormer is simple yet effective, which is illustrated in
Revised: The design principle of AdaptFormer is simple yet effective, which is illustrated in

Original: Figure 2b.
Revised: Figure 2b.

Original: Compared to the vanilla full fine-tuning regime, AdaptFormer replaces the MLP block in the transformer encoder with AdaptMLP , which is consisted of two sub-branches.
Revised: Compared to the vanilla full fine-tuning regime, AdaptFormer replaces the MLP block in the transformer encoder with AdaptMLP , which is consisted of two sub-branches.

Original: The MLP layer in the left branch is identical to the original network, while the right branch is an additionally introduced lightweight module for task-specific fine-tuning.
Revised: The MLP layer in the left branch is identical to the original network, while the right branch is an additionally introduced lightweight module for task-specific fine-tuning.

Original: Specifically, the right branch is designed to be a bottleneck structure for limiting the number of parameters purpose, which includes a down-projection layer with parameters W down ∈ R
Revised: Specifically, the right branch is designed to be a bottleneck structure for limiting the number of parameters purpose, which includes a down-projection layer with parameters W down ∈ R

Original: d
Revised: d

Original: × ˆ d , an up-projection layer with parameters W up ∈ R ˆ d × d , where ˆ d is the bottleneck middle dimension and satisfies ˆ d ≪ d .
Revised: × ˆ d , an up-projection layer with parameters W up ∈ R ˆ d × d , where ˆ d is the bottleneck middle dimension and satisfies ˆ d ≪ d .

Original: In addition, there is a ReLU layer [1] between these projection layers for non-linear property.
Revised: In addition, there is a ReLU layer [1] between these projection layers for non-linear property.

Original: This bottleneck module is connected to the original MLP network (left branch) through the residual connection via a scale factor s .
Revised: This bottleneck module is connected to the original MLP network (left branch) through the residual connection via a scale factor s .

Original: For a specific input feature x ′ ℓ , the right branch in AdaptMLP produces the adapted features, ˜ x ℓ , formally via:
Revised: For a specific input feature x ′ ℓ , the right branch in AdaptMLP produces the adapted features, ˜ x ℓ , formally via:

Original: [Equation]
Revised: [Equation]

Original: Then both the features ˜ x ℓ and x ′ ℓ are fused with x ℓ by residual connection,
Revised: Then both the features ˜ x ℓ and x ′ ℓ are fused with x ℓ by residual connection,

Original: [Equation]
Revised: [Equation]

Original: Fine-tuning.
Revised: Fine-tuning.

Original: During the fine-tuning phase, we only choose the newly added parameters to optimize and keep rest ones fixed.
Revised: During the fine-tuning phase, we only choose the newly added parameters to optimize and keep rest ones fixed.

Original: Specifically, the original model parts (blue blocks in Figure 2b) load weights from the pre-trained checkpoint and keeps parameters frozen .
Revised: Specifically, the original model parts (blue blocks in Figure 2b) load weights from the pre-trained checkpoint and keeps parameters frozen .

Original: The newly added parameters (orange blocks) are updated on the specific data domain with the task-specific losses.
Revised: The newly added parameters (orange blocks) are updated on the specific data domain with the task-specific losses.

Original: Inference.
Revised: Inference.

Original: After fine-tuning, we still keep the shared parameters frozen as in the previous finetuning state, and additionally load the weights of the extra parameters that were fine-tuned in the previous stage.
Revised: After fine-tuning, we still keep the shared parameters frozen as in the previous finetuning state, and additionally load the weights of the extra parameters that were fine-tuned in the previous stage.

Original: The single overall model is able to be adapted to multiple tasks with the assistance of lightweight introduced modules.
Revised: The single overall model is able to be adapted to multiple tasks with the assistance of lightweight introduced modules.

Original: 3.3 Discussion
Revised: 3.3 Discussion

Original: Tunable parameters analysis.
Revised: Tunable parameters analysis.

Original: Our AdaptMLP module is lightweight.
Revised: Our AdaptMLP module is lightweight.

Original: The total number of parameters introduced to per layer is 2 × d × ˆ d + ˆ d + d , which includes biases parameters.
Revised: The total number of parameters introduced to per layer is 2 × d × ˆ d + ˆ d + d , which includes biases parameters.

Original: The middle dimension ˆ d is a small value compared with d (AdaptFormer still obtains a decent performance even when ˆ d = 1 , as discussed in Sec. 4.5).
Revised: The middle dimension ˆ d is a small value compared with d (AdaptFormer still obtains a decent performance even when ˆ d = 1 , as discussed in Sec. 4.5).

Original: Since most of the shared parameters are fixed and the number of newly introduced parameters is small ( < 2% of the pre-trained model parameters), the total model size grows slowly when more downstream tasks are added.
Revised: Since most of the shared parameters are fixed and the number of newly introduced parameters is small ( < 2% of the pre-trained model parameters), the total model size grows slowly when more downstream tasks are added.

Original: [Figure]
Revised: [Figure]

Original: Applicability.
Revised: Applicability.

Original: We note that AdaptMLP is a plug-and-play module that can be adaptively inserted into existing popular vision transformer architectures [31, 63, 83, 90, 23, 29] since all of the backbones share the same MLP layers even though they differ in the MHSA architectures (as shown in
Revised: We note that AdaptMLP is a plug-and-play module that can be adaptively inserted into existing popular vision transformer architectures [31, 63, 83, 90, 23, 29] since all of the backbones share the same MLP layers even though they differ in the MHSA architectures (as shown in

Original: Figure 2b).
Revised: Figure 2b).

Original: Compared to our methods, we notice that recent prompt-related approaches insert trainable parameters into the token space, as illustrated in Figure 3.
Revised: Compared to our methods, we notice that recent prompt-related approaches insert trainable parameters into the token space, as illustrated in Figure 3.

Original: They prepend learnable parameters either into the embedded tokens before linear projection [58] or the key and value tokens after linear projection [51].
Revised: They prepend learnable parameters either into the embedded tokens before linear projection [58] or the key and value tokens after linear projection [51].

Original: Therefore, the prompt-related method can not be straightforwardly adapted to special MHSA variants, especially for the one that takes the pyramid spatial information into account [63, 83].
Revised: Therefore, the prompt-related method can not be straightforwardly adapted to special MHSA variants, especially for the one that takes the pyramid spatial information into account [63, 83].

Original: Besides, we empirically observe that prompt-related methods perform not well when the number of patch tokens grows up from image to video scale, as shown in Figure 1.
Revised: Besides, we empirically observe that prompt-related methods perform not well when the number of patch tokens grows up from image to video scale, as shown in Figure 1.

Original: In summary, we present a strategy for tuning a pre-trained vision Transformer on a set of scalable vision recognition tasks ( e.g. image domain and video domain).
Revised: In summary, we present a strategy for tuning a pre-trained vision Transformer on a set of scalable vision recognition tasks ( e.g. image domain and video domain).

Original: It adds limited learnable parameters for tuning while achieving comparable or even better performance than the full-tuning strategy.
Revised: It adds limited learnable parameters for tuning while achieving comparable or even better performance than the full-tuning strategy.

Original: Moreover, AdaptFormer could serve as a generic module for a large variety of recognition tasks.
Revised: Moreover, AdaptFormer could serve as a generic module for a large variety of recognition tasks.

Original: Insights of architecture design.
Revised: Insights of architecture design.

Original: The MLP module is important for ViTs.
Revised: The MLP module is important for ViTs.

Original: As illustrated in [30], MLPs prevent ViTs from producing a rank-1 matrix.
Revised: As illustrated in [30], MLPs prevent ViTs from producing a rank-1 matrix.

Original: Also, MLPs stop the ViT output from degenerations.
Revised: Also, MLPs stop the ViT output from degenerations.

Original: Inspired by the above analysis, we believe an effective ViT adaptation shall focus on its MLPs rather than multi-head self attentions.
Revised: Inspired by the above analysis, we believe an effective ViT adaptation shall focus on its MLPs rather than multi-head self attentions.

Original: Meanwhile, we learn from the inception framework [76] that parallel design is an effective way for feature ensemble.
Revised: Meanwhile, we learn from the inception framework [76] that parallel design is an effective way for feature ensemble.

Original: With the parallel design, the domain-specific features produced by the adapter module can supplement the domain-agnostic features from the fixed branch for a better feature ensemble.
Revised: With the parallel design, the domain-specific features produced by the adapter module can supplement the domain-agnostic features from the fixed branch for a better feature ensemble.

Original: Our following experiments will verify that the parallel performs better than the sequential design.
Revised: Our following experiments will verify that the parallel performs better than the sequential design.

Original: Besides, though many advanced Transformer-based models [63, 83, 34, 90] which have emerged since the success of ViT having different attention mechanisms within the Transformer block, they all share the similar MLPs (feed-forward network) structures.
Revised: Besides, though many advanced Transformer-based models [63, 83, 34, 90] which have emerged since the success of ViT having different attention mechanisms within the Transformer block, they all share the similar MLPs (feed-forward network) structures.

Original: Therefore, our AdaptMLP can be easily plugged into these ViT variants.
Revised: Therefore, our AdaptMLP can be easily plugged into these ViT variants.

Original: Moreover, AdaptMLP can also be applied to more recent attention-free models [77, 61, 19].
Revised: Moreover, AdaptMLP can also be applied to more recent attention-free models [77, 61, 19].

Original:  Experiments
Revised:  Experiments

Original: We evaluate the effectiveness of AdaptFormer by conducting extensive visual recognition experiments in both the image and video domains.
Revised: We evaluate the effectiveness of AdaptFormer by conducting extensive visual recognition experiments in both the image and video domains.

Original: We first describe our experimental settings in Sec. 4.1, covering the pre-trained backbones, baseline methods, downstream tasks and training details.
Revised: We first describe our experimental settings in Sec. 4.1, covering the pre-trained backbones, baseline methods, downstream tasks and training details.

Original: We then compare
Revised: We then compare

Original: AdaptFormer with baseline methods and provide a thorough analysis in Sec.
Revised: AdaptFormer with baseline methods and provide a thorough analysis in Sec.

Original: In addition, we also conduct ablation studies to explore different experimental configurations and explain what makes for the superiority of AdaptFormer in Sec 4.5.
Revised: In addition, we also conduct ablation studies to explore different experimental configurations and explain what makes for the superiority of AdaptFormer in Sec 4.5.

Original: 4.1 Experimental Settings
Revised: 4.1 Experimental Settings

Original: Pre-trained backbone.
Revised: Pre-trained backbone.

Original: We adopt the plain Vision Transformer (ViT) [31], i.e. , ViT-Base (ViT-B/16) as our backbone model and pre-train the model with both supervised and self-supervised approaches.
Revised: We adopt the plain Vision Transformer (ViT) [31], i.e. , ViT-Base (ViT-B/16) as our backbone model and pre-train the model with both supervised and self-supervised approaches.

Original: Specifically, for image , we directly use the ImageNet-21k [26] supervised pre-trained model 3 and
Revised: Specifically, for image , we directly use the ImageNet-21k [26] supervised pre-trained model 3 and

Original: MAE [43] self-supervised model 4 .
Revised: MAE [43] self-supervised model 4 .

Original: For video , we take both supervised and self-supervised pre-trained models from VideoMAE [78].
Revised: For video , we take both supervised and self-supervised pre-trained models from VideoMAE [78].

Original: More details about pre-training approaches and datasets can be found in Appendix.
Revised: More details about pre-training approaches and datasets can be found in Appendix.

Original: Initialization of AdaptFormer.
Revised: Initialization of AdaptFormer.

Original: For the original networks, we directly load the weights pre-trained on the upstream tasks and keep them frozen/untouched during the fine-tuning process.
Revised: For the original networks, we directly load the weights pre-trained on the upstream tasks and keep them frozen/untouched during the fine-tuning process.

Original: For the newly added modules, the weights of down-projection layers are initialized with Kaiming Normal [44], while the biases of the additional networks and the weights of the up-projection layers are configured with zero initialization.
Revised: For the newly added modules, the weights of down-projection layers are initialized with Kaiming Normal [44], while the biases of the additional networks and the weights of the up-projection layers are configured with zero initialization.

Original: The reason for the zero initialization of other layers is that in this way, the initial newly added parameters are initialized such that the new function resembles the original one at the start of the fine-tuning stage.
Revised: The reason for the zero initialization of other layers is that in this way, the initial newly added parameters are initialized such that the new function resembles the original one at the start of the fine-tuning stage.

Original: We empirically found that if the initialization deviates too far from the identity function, the model is not stable to train.
Revised: We empirically found that if the initialization deviates too far from the identity function, the model is not stable to train.

Original: Baseline methods.
Revised: Baseline methods.

Original: We compare AdaptFormer with three commonly used fine-tuning approaches, including (1) Linear probing: adding an extra linear layer on top of the backbone and tuning the added parameters for evaluation.
Revised: We compare AdaptFormer with three commonly used fine-tuning approaches, including (1) Linear probing: adding an extra linear layer on top of the backbone and tuning the added parameters for evaluation.

Original: (2) Full Fine-tuning: setting all the parameters learnable and tuning them together.
Revised: (2) Full Fine-tuning: setting all the parameters learnable and tuning them together.

Original: (3) Visual Prompt Tuning (VPT): [51] fine-tuning the extra token parameters as shown in
Revised: (3) Visual Prompt Tuning (VPT): [51] fine-tuning the extra token parameters as shown in

Original: Figure 3.
Revised: Figure 3.

Original: Downstream tasks.
Revised: Downstream tasks.

Original: We evaluate our AdaptFormer on both image and video recognition tasks to verify its effectiveness.
Revised: We evaluate our AdaptFormer on both image and video recognition tasks to verify its effectiveness.

Original: The specific datasets leveraged in this work are presented in the following.
Revised: The specific datasets leveraged in this work are presented in the following.

Original: • Image domain : CIFAR-100 [54] contains 50,000 training images and 10,000 validation images of resolution 32×32 with 100 labels.
Revised: • Image domain : CIFAR-100 [54] contains 50,000 training images and 10,000 validation images of resolution 32×32 with 100 labels.

Original: Street View House Numbers (SVHN) [37] is a digit classification benchmark dataset.
Revised: Street View House Numbers (SVHN) [37] is a digit classification benchmark dataset.

Original: In total, the dataset comprises over 600,000 labeled images, containing 73, training samples, 26,032 testing samples and 531,131 extra training data.
Revised: In total, the dataset comprises over 600,000 labeled images, containing 73, training samples, 26,032 testing samples and 531,131 extra training data.

Original: The Food-101 [9] dataset consists of 101 food categories with a total of 101k images, including 750 training and 250 testing samples per category.
Revised: The Food-101 [9] dataset consists of 101 food categories with a total of 101k images, including 750 training and 250 testing samples per category.

Original: • Video domain : Something-Something V2 (SSv2) [39] is a large collection of video clips showing the people perform several normal actions in the daily life ( e.g. , moving stuff and opening the door).
Revised: • Video domain : Something-Something V2 (SSv2) [39] is a large collection of video clips showing the people perform several normal actions in the daily life ( e.g. , moving stuff and opening the door).

Original: It consists of 168,913 training samples, 24,777 validation samples and 27,157 testing samples, making a total of 220,847 videos with 174 labels.
Revised: It consists of 168,913 training samples, 24,777 validation samples and 27,157 testing samples, making a total of 220,847 videos with 174 labels.

Original: HMDB51 [55] is composed of 6, videos with 51 categories, making a split of 3.5k/1.5k train/val videos.
Revised: HMDB51 [55] is composed of 6, videos with 51 categories, making a split of 3.5k/1.5k train/val videos.

Original: Implementation details.
Revised: Implementation details.

Original: In this work, we use PyTorch toolkit [68] to conduct all experiments on
Revised: In this work, we use PyTorch toolkit [68] to conduct all experiments on

Original: NVIDIA V100 GPUs.
Revised: NVIDIA V100 GPUs.

Original: Unless otherwise stated, we use 8 × 8 GPUs for video experiments and 1 ×
Revised: Unless otherwise stated, we use 8 × 8 GPUs for video experiments and 1 ×

Original: GPUs for image experiments.
Revised: GPUs for image experiments.

Original: Our default configurations follow the linear probing settings in [21, 43], which do not utilize many common regularization strategies, such as mixup [94], cutmix [91], color jittering and so on.
Revised: Our default configurations follow the linear probing settings in [21, 43], which do not utilize many common regularization strategies, such as mixup [94], cutmix [91], color jittering and so on.

Original: More details can be found in Appendix.
Revised: More details can be found in Appendix.

Original: 4.2 Main Properties and Analysis
Revised: 4.2 Main Properties and Analysis

Original: We compare the performance of different fine-tuning approaches in Table 1 with the backbones pre-trained via the self-supervised paradigms.
Revised: We compare the performance of different fine-tuning approaches in Table 1 with the backbones pre-trained via the self-supervised paradigms.

Original: The results show that AdaptFormer consistently
Revised: The results show that AdaptFormer consistently

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: surpasses linear probing and Visual Prompt tuning (VPT) methods.
Revised: surpasses linear probing and Visual Prompt tuning (VPT) methods.

Original: Specifically, AdaptFormer64 outperforms VPT on image benchmark CIFAR-100, SVHN and Food-101, by 3.46%, 2.87%, and 4.63% respectively.
Revised: Specifically, AdaptFormer64 outperforms VPT on image benchmark CIFAR-100, SVHN and Food-101, by 3.46%, 2.87%, and 4.63% respectively.

Original: On the more challenging video action recognition dataset SomethingSomething V2, the superiority becomes even more significant, i e. , about 15%.
Revised: On the more challenging video action recognition dataset SomethingSomething V2, the superiority becomes even more significant, i e. , about 15%.

Original: Note that even compared with the full fine-tuning strategy, our AdaptFormer still outperforms by about 5% Topaccuracy on SSv2 dataset.
Revised: Note that even compared with the full fine-tuning strategy, our AdaptFormer still outperforms by about 5% Topaccuracy on SSv2 dataset.

Original: To summarize, our AdaptFormer is highly parameter-efficient as well as yielding good performance with parameter size at most 2% times than the full fine-tuning manner.
Revised: To summarize, our AdaptFormer is highly parameter-efficient as well as yielding good performance with parameter size at most 2% times than the full fine-tuning manner.

Original: 4.3 Scaling Tunable Parameters Up
Revised: 4.3 Scaling Tunable Parameters Up

Original: Even though there are only limited parameters introduced, one might also argue that more tunable parameters of AdaptFormer contribute to its higher accuracy compared with VPT [51].
Revised: Even though there are only limited parameters introduced, one might also argue that more tunable parameters of AdaptFormer contribute to its higher accuracy compared with VPT [51].

Original: We conduct experiments to make a comprehensive discussion on this aspect
Revised: We conduct experiments to make a comprehensive discussion on this aspect

Original: As described in Sec. 3.3, the number of tunable parameters can be adjusted by changing the number of introduced tokens for VPT, or the hidden feature dimension for AdaptFormer.
Revised: As described in Sec. 3.3, the number of tunable parameters can be adjusted by changing the number of introduced tokens for VPT, or the hidden feature dimension for AdaptFormer.

Original: As shown in Figure 4, we conduct experiments with a wide range of tunable parameters on both SSv2 and HMDBdatasets.
Revised: As shown in Figure 4, we conduct experiments with a wide range of tunable parameters on both SSv2 and HMDBdatasets.

Original: Since AdaptFormer and VPT share the same number of parameters of classification head on a specific dataset, we only report the tunable parameters on the x-axis, which comes from the visual prompts (VPT) or weight/bias of the down-up fully-connected layers (AdaptFormer), without calculating the parameters of classification head.
Revised: Since AdaptFormer and VPT share the same number of parameters of classification head on a specific dataset, we only report the tunable parameters on the x-axis, which comes from the visual prompts (VPT) or weight/bias of the down-up fully-connected layers (AdaptFormer), without calculating the parameters of classification head.

Original: For VPT, the number of introduced tokens is chosen from {1, 2, 4, 8, 16, 32, 48, 64}.
Revised: For VPT, the number of introduced tokens is chosen from {1, 2, 4, 8, 16, 32, 48, 64}.

Original: Similarly, the number of hidden dimensions in AdaptFormer is in {1, 2, 4, 8, 16, 32}.
Revised: Similarly, the number of hidden dimensions in AdaptFormer is in {1, 2, 4, 8, 16, 32}.

Original: AdaptFormer has a slight performance gain or maintains the accuracy stably when the parameters scale up.
Revised: AdaptFormer has a slight performance gain or maintains the accuracy stably when the parameters scale up.

Original: On the contrary, the performance of VPT decreases dramatically when the parameters exceed the task-specific value.
Revised: On the contrary, the performance of VPT decreases dramatically when the parameters exceed the task-specific value.

Original: Moreover, choosing the most suitable number of token
Revised: Moreover, choosing the most suitable number of token

Original: [Table]
Revised: [Table]

Original: number becomes laborious since it might be task-specific ( i.e. varying from one dataset to the other one).
Revised: number becomes laborious since it might be task-specific ( i.e. varying from one dataset to the other one).

Original: For example, the accuracy of VPT keeps going up when the number of tunable parameters increases up to 300K on SSv2, whereas it begins to drop when the number of tunable parameters exceeds 50K on HMDB-51.
Revised: For example, the accuracy of VPT keeps going up when the number of tunable parameters increases up to 300K on SSv2, whereas it begins to drop when the number of tunable parameters exceeds 50K on HMDB-51.

Original: We further study the optimization procedures of VPT by monitoring the test accuracy of the training stage.
Revised: We further study the optimization procedures of VPT by monitoring the test accuracy of the training stage.

Original: As shown in Figure 5, we gradually increase the number of tokens in VPT and plot the Topaccuracy of each epoch.
Revised: As shown in Figure 5, we gradually increase the number of tokens in VPT and plot the Topaccuracy of each epoch.

Original: The training stages are stable when the number of tokens is less than or equal to 4, e.g. , {1, 2, 4}.
Revised: The training stages are stable when the number of tokens is less than or equal to 4, e.g. , {1, 2, 4}.

Original: However, when the number becomes 8 or larger, e.g. , {8, 16, 32}, the training procedure collapses at about the tenth epoch and achieves poor performance at the end of the training stage.
Revised: However, when the number becomes 8 or larger, e.g. , {8, 16, 32}, the training procedure collapses at about the tenth epoch and achieves poor performance at the end of the training stage.

Original: On the contrary, the optimization procedures of AdaptFormer are stable when the number of parameters varies across a large range, as shown in Table 3a.
Revised: On the contrary, the optimization procedures of AdaptFormer are stable when the number of parameters varies across a large range, as shown in Table 3a.

Original: The top-1 accuracy fluctuates within 1.5% when the number of parameters increases from 0.44M ( dim=16 ) to 4.87M ( dim=256 ).
Revised: The top-1 accuracy fluctuates within 1.5% when the number of parameters increases from 0.44M ( dim=16 ) to 4.87M ( dim=256 ).

Original: 4.4 Multi-Label Classification
Revised: 4.4 Multi-Label Classification

Original: We further conduct experiments on dataset with larger scale and diversity.
Revised: We further conduct experiments on dataset with larger scale and diversity.

Original: Specifically, we evaluate
Revised: Specifically, we evaluate

Original: AdaptFormer on NUS-WIDE [24] for multi-label classification.
Revised: AdaptFormer on NUS-WIDE [24] for multi-label classification.

Original: NUS-WIDE contains 269, images collected from Flicker, which are annotated with 81 visual concepts.
Revised: NUS-WIDE contains 269, images collected from Flicker, which are annotated with 81 visual concepts.

Original: Since some images are not available on Flicker, we only use 220,000 images following [7, 32].
Revised: Since some images are not available on Flicker, we only use 220,000 images following [7, 32].

Original: We utilize mean average precision (mAP) as performance metric.
Revised: We utilize mean average precision (mAP) as performance metric.

Original: Settings and results.
Revised: Settings and results.

Original: Our training settings mainly follow ASL [7].
Revised: Our training settings mainly follow ASL [7].

Original: Specifically, We trained all models for 40 epochs using Adam optimize and 1-cycle learning rate policy [73].
Revised: Specifically, We trained all models for 40 epochs using Adam optimize and 1-cycle learning rate policy [73].

Original: The maximal learning rate is 0.001.
Revised: The maximal learning rate is 0.001.

Original: As shown in Table 2, though AdaptFormer-64 achieves a slightly lower mAP than fine-tuning, it significantly reduces the amount parameters that need to be updated (from 85.86 to 1.25M).
Revised: As shown in Table 2, though AdaptFormer-64 achieves a slightly lower mAP than fine-tuning, it significantly reduces the amount parameters that need to be updated (from 85.86 to 1.25M).

Original: Moreover, AdaptFormer has an clear advantage over other fine-tuning approaches including linear probing and VPT.
Revised: Moreover, AdaptFormer has an clear advantage over other fine-tuning approaches including linear probing and VPT.

Original: 4.5 Ablation Studies
Revised: 4.5 Ablation Studies

Original: We ablate our AdaptFormer to study what properties make for a good AdaptFormer and observe several intriguing properties.
Revised: We ablate our AdaptFormer to study what properties make for a good AdaptFormer and observe several intriguing properties.

Original: The ablation studies conducted in this work are all performed on the
Revised: The ablation studies conducted in this work are all performed on the

Original: SSv2 validation set [39]
Revised: SSv2 validation set [39]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Middle dimension.
Revised: Middle dimension.

Original: The middle dimension controls the number of introduced parameters by AdaptFormer.
Revised: The middle dimension controls the number of introduced parameters by AdaptFormer.

Original: Lower middle dimensions introduce fewer parameters with a possible performance cost.
Revised: Lower middle dimensions introduce fewer parameters with a possible performance cost.

Original: We ablate AdaptFormer on the middle feature dimension to study this effects.
Revised: We ablate AdaptFormer on the middle feature dimension to study this effects.

Original: As shown in Table 3a, the accuracy consistently improves when the middle dimension increases up to 64 and reaches the saturation point when the middle dimension is about 64 on SSv2 dataset.
Revised: As shown in Table 3a, the accuracy consistently improves when the middle dimension increases up to 64 and reaches the saturation point when the middle dimension is about 64 on SSv2 dataset.

Original: We note that our AdaptFormer can achieve a decent performance when the middle dimension reduces even to one, about 50.03% top-1 accuracy.
Revised: We note that our AdaptFormer can achieve a decent performance when the middle dimension reduces even to one, about 50.03% top-1 accuracy.

Original: We conduct more extensive ablation studies on middle dimension in Appendix Table 10 and found that the optimal middle dimension varies per dataset.
Revised: We conduct more extensive ablation studies on middle dimension in Appendix Table 10 and found that the optimal middle dimension varies per dataset.

Original: For example, the accuracy reaches saturation when the middle dimension equals 64 on SSv2, whereas for NUS-WIDE dataset, the mAP slightly improves when the middle dimension increases from 64 to 512.
Revised: For example, the accuracy reaches saturation when the middle dimension equals 64 on SSv2, whereas for NUS-WIDE dataset, the mAP slightly improves when the middle dimension increases from 64 to 512.

Original: However, AdaptFormer with middle dimension as 512 has 0.75 mAP higher (59.82 vs. 59.07 mAP) than the one with 64 at the cost of about 8 times more parameters.
Revised: However, AdaptFormer with middle dimension as 512 has 0.75 mAP higher (59.82 vs. 59.07 mAP) than the one with 64 at the cost of about 8 times more parameters.

Original: Therefore, we choose the middle dimension=64 for both SSv and NUS-WIDE for a better trade-off.
Revised: Therefore, we choose the middle dimension=64 for both SSv and NUS-WIDE for a better trade-off.

Original: Scaling factor.
Revised: Scaling factor.

Original: The scaling factor s is introduced to balance the task-agnostic features (generated by the original frozen branch) and the task-specific features (generated by the tunable bottleneck branch).
Revised: The scaling factor s is introduced to balance the task-agnostic features (generated by the original frozen branch) and the task-specific features (generated by the tunable bottleneck branch).

Original: We evaluate AdaptFormer with multiple s values and the results are summarized in Table 3c.
Revised: We evaluate AdaptFormer with multiple s values and the results are summarized in Table 3c.

Original: Different from the scaling factor in NLP field which prefer s larger than 1 ( e.g. , s = 4 in [42]), we empirically found that the s should be < 1 for vision tasks, otherwise the fine-tuning would become unstable.
Revised: Different from the scaling factor in NLP field which prefer s larger than 1 ( e.g. , s = 4 in [42]), we empirically found that the s should be < 1 for vision tasks, otherwise the fine-tuning would become unstable.

Original: Besides, we found that AdaptFormer achieves optimal performance with s = 0 .
Revised: Besides, we found that AdaptFormer achieves optimal performance with s = 0 .

Original: A larger or smaller s would bring slight performance drop.
Revised: A larger or smaller s would bring slight performance drop.

Original: Thus, we choose s = 0 .
Revised: Thus, we choose s = 0 .

Original: 10 as a default setting.
Revised: 10 as a default setting.

Original: AdaptFormer position.
Revised: AdaptFormer position.

Original: As shown in Table 3b, we further ablate on the specific position to introduce the AdaptMLP block.
Revised: As shown in Table 3b, we further ablate on the specific position to introduce the AdaptMLP block.

Original: We gradually increase the number of AdaptMLP layers with a step of three ( start → end , both included).
Revised: We gradually increase the number of AdaptMLP layers with a step of three ( start → end , both included).

Original: We observe that the performance of AdaptFormer has a positive correlation with the number of added layers.
Revised: We observe that the performance of AdaptFormer has a positive correlation with the number of added layers.

Original: In addition, AdaptFormer prefers the top part (the one far away from the input image) of the network to the bottom part when introducing the same number of layers, e.g. , AdaptFormer with 7 → 12 obtains over 14.5% higher accuracy than 1 → 6 , though both equipped with six AdaptMLP layers.
Revised: In addition, AdaptFormer prefers the top part (the one far away from the input image) of the network to the bottom part when introducing the same number of layers, e.g. , AdaptFormer with 7 → 12 obtains over 14.5% higher accuracy than 1 → 6 , though both equipped with six AdaptMLP layers.

Original: Insertion form.
Revised: Insertion form.

Original: We study the insertion formulation by comparing the parallel and sequential instances which are illustrated in Figure 6.
Revised: We study the insertion formulation by comparing the parallel and sequential instances which are illustrated in Figure 6.

Original: As shown in Table 3b, the parallel AdaptFormer is able to outperform the sequential one by 0.85% top-1 accuracy.
Revised: As shown in Table 3b, the parallel AdaptFormer is able to outperform the sequential one by 0.85% top-1 accuracy.

Original: The reason might be: (1) the parallel design maintains the original feature using an independent branch and aggregating updated context by element-wise scaled sum; (2) the sequential design is equivalent to adding more layers, which might cause optimization difficulty.
Revised: The reason might be: (1) the parallel design maintains the original feature using an independent branch and aggregating updated context by element-wise scaled sum; (2) the sequential design is equivalent to adding more layers, which might cause optimization difficulty.

Original: Therefore, we adopt the parallel design as our default setting due to its superiority.
Revised: Therefore, we adopt the parallel design as our default setting due to its superiority.

Original: [Figure]
Revised: [Figure]

Original: results are shown in Table 3b
Revised: results are shown in Table 3b

Original: [Figure]
Revised: [Figure]

Original: Number of frames.
Revised: Number of frames.

Original: The number of embedded patch tokens increases linearly with the number of video frames for the plain ViT [31].
Revised: The number of embedded patch tokens increases linearly with the number of video frames for the plain ViT [31].

Original: We conduct experiments with the different number of frames, i.e. , {2, 4, 8} and the results are shown in Figure 7.
Revised: We conduct experiments with the different number of frames, i.e. , {2, 4, 8} and the results are shown in Figure 7.

Original: We observe that increasing the number of frames is beneficial for all these three fine-tuning methods.
Revised: We observe that increasing the number of frames is beneficial for all these three fine-tuning methods.

Original: However, AdaptFormer consistently outperforms the linear manner ( e.g. , +30% top-1 accuracy on 8 input frames) and VPT method( e.g. , +14% topaccuracy on 8 input frames).
Revised: However, AdaptFormer consistently outperforms the linear manner ( e.g. , +30% top-1 accuracy on 8 input frames) and VPT method( e.g. , +14% topaccuracy on 8 input frames).

Original: 4.6 Towards Visual Recognition Generalist Agent
Revised: 4.6 Towards Visual Recognition Generalist Agent

Original: In the above experiments, we typically utilize a modality-specific pre-trained checkpoint for the corresponding downstream tasks.
Revised: In the above experiments, we typically utilize a modality-specific pre-trained checkpoint for the corresponding downstream tasks.

Original: For example, we use Kinetics-400 ( video domain ) pretrained model for downstream video action recognition on Something-Something V2 and HMDB51 benchmarks.
Revised: For example, we use Kinetics-400 ( video domain ) pretrained model for downstream video action recognition on Something-Something V2 and HMDB51 benchmarks.

Original: Besides, we use ImageNet-21K ( image domain ) pre-rained model for downstream image classification on CIFAR-100, SVHN and Food-101 benchmarks.
Revised: Besides, we use ImageNet-21K ( image domain ) pre-rained model for downstream image classification on CIFAR-100, SVHN and Food-101 benchmarks.

Original: Our AdaptFormer achieves superior performances in this same network with modality-specific weights scenario.
Revised: Our AdaptFormer achieves superior performances in this same network with modality-specific weights scenario.

Original: [Table]
Revised: [Table]

Original: Next, we take a further step to ask what would happen if using the same network with the modality-agnostic weights for multiple tasks in the multi-modalities downstream tasks?
Revised: Next, we take a further step to ask what would happen if using the same network with the modality-agnostic weights for multiple tasks in the multi-modalities downstream tasks?

Original: We use the model pre-trained on ImagNet-21k to do action recognition on SSv2.
Revised: We use the model pre-trained on ImagNet-21k to do action recognition on SSv2.

Original: As shown in Table 4, AdaptFormer is robust to domain shift caused by modality.
Revised: As shown in Table 4, AdaptFormer is robust to domain shift caused by modality.

Original: The experimental results show that the linear probe approach obtains a very poor accuracy ( i.e. , 6.56% top-1 accuracy) when fine-tuning on SSv2.
Revised: The experimental results show that the linear probe approach obtains a very poor accuracy ( i.e. , 6.56% top-1 accuracy) when fine-tuning on SSv2.

Original: Meanwhile,
Revised: Meanwhile,

Original: VPT [51] achieves a better performance than linear probe but it is not decent ( i.e. , 16.94% topaccuracy).
Revised: VPT [51] achieves a better performance than linear probe but it is not decent ( i.e. , 16.94% topaccuracy).

Original: Our AdaptFormer, compared to the above two methods, attains a promising
Revised: Our AdaptFormer, compared to the above two methods, attains a promising

Original: [Table]
Revised: [Table]

Original: 06% topaccuracy, which is even higher than the full-tuning schedule (+4.56%).
Revised: 06% topaccuracy, which is even higher than the full-tuning schedule (+4.56%).

Original: 4.7 Visualization
Revised: 4.7 Visualization

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: To evaluate the quality of the produced features, we conduct t-SNE [80] visualizations on AdaptFormer and other baseline methods.
Revised: To evaluate the quality of the produced features, we conduct t-SNE [80] visualizations on AdaptFormer and other baseline methods.

Original: The features are extracted from the SSv2 validation set via the ViT-Base backbone.
Revised: The features are extracted from the SSv2 validation set via the ViT-Base backbone.

Original: Figure 8 shows that the linear fine-tuning and the VPT methods tend to output mixed features as shown in Figure 8(a)-(b).
Revised: Figure 8 shows that the linear fine-tuning and the VPT methods tend to output mixed features as shown in Figure 8(a)-(b).

Original: Compared with the above two methods, the full fine-tuning strategy performs well in projecting features.
Revised: Compared with the above two methods, the full fine-tuning strategy performs well in projecting features.

Original: However, it consumes huge computational sources to tune the whole network parameters.
Revised: However, it consumes huge computational sources to tune the whole network parameters.

Original: Figure 8(d) validates that our AdaptFormer facilitates
Revised: Figure 8(d) validates that our AdaptFormer facilitates

Original: ViT-Base in generating more separable representations with fewer learnable parameters.
Revised: ViT-Base in generating more separable representations with fewer learnable parameters.

Original:  Conclusion
Revised:  Conclusion

Original: We present a conceptually simple yet effective framework, AdaptFormer, for efficiently adapting a pre-trained Vision Transformer (ViT) backbone to scalable vision recognition tasks.
Revised: We present a conceptually simple yet effective framework, AdaptFormer, for efficiently adapting a pre-trained Vision Transformer (ViT) backbone to scalable vision recognition tasks.

Original: By introducing
Revised: By introducing

Original: AdaptMLP, our AdaptFormer is able to fine-tune the lightweight modules for producing features adapted to multiple downstream tasks.
Revised: AdaptMLP, our AdaptFormer is able to fine-tune the lightweight modules for producing features adapted to multiple downstream tasks.

Original: The extensive experiments on five datasets, covering both the image and the video domains, validate that our proposed methods are able to increase the
Revised: The extensive experiments on five datasets, covering both the image and the video domains, validate that our proposed methods are able to increase the

Original: ViT’s transferability with little computational cost.
Revised: ViT’s transferability with little computational cost.

Original: We hope our work will inspire future research in exploring more efficient fine-tuning methods for large vision models.
Revised: We hope our work will inspire future research in exploring more efficient fine-tuning methods for large vision models.

Original: One limitation is that
Revised: One limitation is that

Original: AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can work well in tasks beyond recognition, e.g. , object detection and semantic segmentation.
Revised: AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can work well in tasks beyond recognition, e.g. , object detection and semantic segmentation.

Original: We leave it for the future exploration.
Revised: We leave it for the future exploration.

Original: Since our method is specially designed for efficient fine-tuning, we do not foresee obvious undesirable ethical/social impacts at this moment.
Revised: Since our method is specially designed for efficient fine-tuning, we do not foresee obvious undesirable ethical/social impacts at this moment.

Original: Acknowledgment.
Revised: Acknowledgment.

Original: This work is supported by CCF-Tencent Open Fund.
Revised: This work is supported by CCF-Tencent Open Fund.

Original: Ping Luo is supported by the General Research Fund of HK No.27208720, No.17212120, and No.17200622.
Revised: Ping Luo is supported by the General Research Fund of HK No.27208720, No.17212120, and No.17200622.

Original: A Appendix
Revised: A Appendix

Original: In this supplementary material, we will include the details about the pre-training and fine-tuning processes, the extensive experiments of AdaptFormer on hierarchical vision transformers ( e.g. ,
Revised: In this supplementary material, we will include the details about the pre-training and fine-tuning processes, the extensive experiments of AdaptFormer on hierarchical vision transformers ( e.g. ,

Original: AdaptFormer-Swin), and the pseudo-code of AdaptMLP in a PyTorch-like style.
Revised: AdaptFormer-Swin), and the pseudo-code of AdaptMLP in a PyTorch-like style.

Original: A.1 Experimental Settings A.1.1 Pre-training Approaches
Revised: A.1 Experimental Settings A.1.1 Pre-training Approaches

Original: Image.
Revised: Image.

Original: We use MAE [43] as our self-supervised pre-training method in the image domain, a simple yet effective method that first masks nearly 75% patches of the input image and then reconstructs the missing pixels.
Revised: We use MAE [43] as our self-supervised pre-training method in the image domain, a simple yet effective method that first masks nearly 75% patches of the input image and then reconstructs the missing pixels.

Original: Specifically, we directly adopt the checkpoint 5 of ViT-B/16 for convenience, which is pre-trained on ImageNet-1K [26] for 800 epochs.
Revised: Specifically, we directly adopt the checkpoint 5 of ViT-B/16 for convenience, which is pre-trained on ImageNet-1K [26] for 800 epochs.

Original: Video.
Revised: Video.

Original: We use VideoMAE [78] as our self-supervised pre-training method in the video domain, which is an direct extension of MAE to the video domain.
Revised: We use VideoMAE [78] as our self-supervised pre-training method in the video domain, which is an direct extension of MAE to the video domain.

Original: VideoMAE utilizes the plain ViT [31] architecture of joint space-time attention mechanism [2, 64] and an extremely high proportion of masking ratio ( i.e. , 90% to 95%) for pre-training.
Revised: VideoMAE utilizes the plain ViT [31] architecture of joint space-time attention mechanism [2, 64] and an extremely high proportion of masking ratio ( i.e. , 90% to 95%) for pre-training.

Original: We also directly use the publicly available checkpoint 6 , which is pre-trained on Kinetics-400 [15].
Revised: We also directly use the publicly available checkpoint 6 , which is pre-trained on Kinetics-400 [15].

Original: A.1.2 Implementation Details of Fine-tuning
Revised: A.1.2 Implementation Details of Fine-tuning

Original: [Table]
Revised: [Table]

Original: The implementation details are summarized in Table 5.
Revised: The implementation details are summarized in Table 5.

Original: The video experiments are conducted on 64 Tesla V100 GPUs, while the image experiments are performed on 8 Tesla V100 GPUs.
Revised: The video experiments are conducted on 64 Tesla V100 GPUs, while the image experiments are performed on 8 Tesla V100 GPUs.

Original: For the optimizer, different from [89] that adopts LARS, we leverage SGD [75] for stable training on small-scale dataset ( e.g. , CIFAR10).
Revised: For the optimizer, different from [89] that adopts LARS, we leverage SGD [75] for stable training on small-scale dataset ( e.g. , CIFAR10).

Original: The actual learning rate is calculated by: lr = base_lr × batchsize / 256 following the linear lr scaling rule [38].
Revised: The actual learning rate is calculated by: lr = base_lr × batchsize / 256 following the linear lr scaling rule [38].

Original: More detailed training configurations are presented in
Revised: More detailed training configurations are presented in

Original: Table 5, including the batchsize, learning rate schedule and etc.
Revised: Table 5, including the batchsize, learning rate schedule and etc.

Original: The experimental settings of image and video mainly follow the ones utilized in MAE [43] and
Revised: The experimental settings of image and video mainly follow the ones utilized in MAE [43] and

Original: VideoMAE [78], respectively.
Revised: VideoMAE [78], respectively.

Original: We insert an extra BatchNorm layer [49] without affine transformation ( i.e. affine=
Revised: We insert an extra BatchNorm layer [49] without affine transformation ( i.e. affine=

Original: False ) before the final fully connected layer, following the common practice to normalize the pre-trained features [28, 43].
Revised: False ) before the final fully connected layer, following the common practice to normalize the pre-trained features [28, 43].

Original: In addition, there is no flip augmentation during the fine-tuning stage for video data.
Revised: In addition, there is no flip augmentation during the fine-tuning stage for video data.

Original: A.2 More Supplementary Results A.2.1 AdaptFormer with Supervised Pre-training
Revised: A.2 More Supplementary Results A.2.1 AdaptFormer with Supervised Pre-training

Original: In addition to the self-supervised pre-training presented in the main paper
Revised: In addition to the self-supervised pre-training presented in the main paper

Original: [Table]
Revised: [Table]

Original: we also evaluate AdaptFormer with the supervised pre-trained model.
Revised: we also evaluate AdaptFormer with the supervised pre-trained model.

Original: The results in Table 6 show that AdaptFormer still outperforms linear probe and VPT obviously.
Revised: The results in Table 6 show that AdaptFormer still outperforms linear probe and VPT obviously.

Original: In addition, AdaptFormer surpasses full-tuning on four benchmarks (CIFAR100, SVHN, SSv2, HMDB51) with only 1.46% parameters.
Revised: In addition, AdaptFormer surpasses full-tuning on four benchmarks (CIFAR100, SVHN, SSv2, HMDB51) with only 1.46% parameters.

Original: On the remaining benchmark (Food-101), AdaptFormer achieves an almost comparable performance to full-tuning (90.89% v.s. 90.96%).
Revised: On the remaining benchmark (Food-101), AdaptFormer achieves an almost comparable performance to full-tuning (90.89% v.s. 90.96%).

Original: [Table]
Revised: [Table]

Original: A.2.2 AdaptFormer on Swin Transformer
Revised: A.2.2 AdaptFormer on Swin Transformer

Original: Settings.
Revised: Settings.

Original: We further demonstrate the effectiveness of AdaptFormer on hierarchical vision transformers, e.g. , Swin [63, 64].
Revised: We further demonstrate the effectiveness of AdaptFormer on hierarchical vision transformers, e.g. , Swin [63, 64].

Original: We name AdaptFormer applied to Swin as AdaptFormer-Swin , to distinguish plain AdaptFormer (without any suffix) which is applied to the vanilla ViT [31].
Revised: We name AdaptFormer applied to Swin as AdaptFormer-Swin , to distinguish plain AdaptFormer (without any suffix) which is applied to the vanilla ViT [31].

Original: It is noted that we can adopt AdaptMLP to Swin easily without any special modification as Swin and ViT share the same MLP architecture.
Revised: It is noted that we can adopt AdaptMLP to Swin easily without any special modification as Swin and ViT share the same MLP architecture.

Original: However, VPT [51] needs additional handcraft designs to be suitable for the shifted local windows in the prevalent hierarchical vision transformers, which hinders its general applications.
Revised: However, VPT [51] needs additional handcraft designs to be suitable for the shifted local windows in the prevalent hierarchical vision transformers, which hinders its general applications.

Original: We utilize Swin-B [63] and the video counterpart [64] for image and video.
Revised: We utilize Swin-B [63] and the video counterpart [64] for image and video.

Original: Similarly, we also directly use the officially provided checkpoints 7 , which are pre-trained on ImageNet-21K [26] and
Revised: Similarly, we also directly use the officially provided checkpoints 7 , which are pre-trained on ImageNet-21K [26] and

Original: Kinetics-600 [14].
Revised: Kinetics-600 [14].

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: Since VPT is not applicable in Swin, we do not report its performance.
Revised: Since VPT is not applicable in Swin, we do not report its performance.

Original: Table 7 shows
Revised: Table 7 shows

Original: AdaptFormer-Swin performs well compared with other tuning strategies.
Revised: AdaptFormer-Swin performs well compared with other tuning strategies.

Original: For image benchmarks, our method can outperform full-tuning approach with only 1.43% parameters.
Revised: For image benchmarks, our method can outperform full-tuning approach with only 1.43% parameters.

Original: Moreover, AdaptFormerSwin surpasses linear probing by a significant margin, especially on the challenging dataset, SSv2.
Revised: Moreover, AdaptFormerSwin surpasses linear probing by a significant margin, especially on the challenging dataset, SSv2.

Original: The results validate that AdaptFormer is able
Revised: The results validate that AdaptFormer is able

Original: to generally
Revised: to generally

Original: boost the transferability of various vision Transformer variants.
Revised: boost the transferability of various vision Transformer variants.

Original: A.3 Possible Architectures
Revised: A.3 Possible Architectures

Original: We explore other possible architectures utilized in AdaptFormer.
Revised: We explore other possible architectures utilized in AdaptFormer.

Original: Specifically, we further replace the
Revised: Specifically, we further replace the

Original: MLP architectures within the AdaptMLP module by the convolution layer, depthwise convolution layer, and LayerNorm layer.
Revised: MLP architectures within the AdaptMLP module by the convolution layer, depthwise convolution layer, and LayerNorm layer.

Original: For fair comparisons, we carefully design the above modules to meet the comparable number of parameters (˜1.3M).
Revised: For fair comparisons, we carefully design the above modules to meet the comparable number of parameters (˜1.3M).

Original: The experimental results of different adapter modules are shown in Table 8, which validates that the simple MLP modules are simple yet effective compared with the other architectures.
Revised: The experimental results of different adapter modules are shown in Table 8, which validates that the simple MLP modules are simple yet effective compared with the other architectures.

Original: For example, our AdaptMLP module surpasses the AdaptConv module by 0.55% Top1 accuracy on SSv2 dataset.
Revised: For example, our AdaptMLP module surpasses the AdaptConv module by 0.55% Top1 accuracy on SSv2 dataset.

Original: [Table]
Revised: [Table]

Original: A.4 Evaluation on ImageNet-1k datasets
Revised: A.4 Evaluation on ImageNet-1k datasets

Original: [Table]
Revised: [Table]

Original: We point out that in order to evaluate the adaptation performance across datasets, it’s an unreasonable setting to fine-tune the ImageNet-1k dataset with the ImageNet-21k pre-trained weights.
Revised: We point out that in order to evaluate the adaptation performance across datasets, it’s an unreasonable setting to fine-tune the ImageNet-1k dataset with the ImageNet-21k pre-trained weights.

Original: This is because ImageNet-1K is a subset of the ImageNet-21K as introduced in [72].
Revised: This is because ImageNet-1K is a subset of the ImageNet-21K as introduced in [72].

Original: In contrast, in all the previous experiments, there is no overlap between the fine-tuning and pre-trained datasets.
Revised: In contrast, in all the previous experiments, there is no overlap between the fine-tuning and pre-trained datasets.

Original: However, we document the experiments of fine-tuning with the ImageNet-1k dataset for the completeness.
Revised: However, we document the experiments of fine-tuning with the ImageNet-1k dataset for the completeness.

Original: We adopt exactly identical training configurations to conduct experiments in this subsection.
Revised: We adopt exactly identical training configurations to conduct experiments in this subsection.

Original: We experiment with middle dimension = {1, 4, 16, 64} on ImageNet-1K, and the results are shown Table 9.
Revised: We experiment with middle dimension = {1, 4, 16, 64} on ImageNet-1K, and the results are shown Table 9.

Original: Results.
Revised: Results.

Original: Comparing the results of AdaptFormer with different middle dimension ({1, 4, 16, 64}) on ImageNet-1K, we find that AdaptFormer with the smallest number of parameters ( AdaptFormer1 ) achieves the best top-1 accuracy (82.33%).
Revised: Comparing the results of AdaptFormer with different middle dimension ({1, 4, 16, 64}) on ImageNet-1K, we find that AdaptFormer with the smallest number of parameters ( AdaptFormer1 ) achieves the best top-1 accuracy (82.33%).

Original: Furthermore, when the ‘middle dimension‘ increases from 1 to 4 or 16, AdaptFormer has a slight performance drop (AdaptFormer-4 (-0.07%) and
Revised: Furthermore, when the ‘middle dimension‘ increases from 1 to 4 or 16, AdaptFormer has a slight performance drop (AdaptFormer-4 (-0.07%) and

Original: AdaptFormer-16 (-0.09%)).
Revised: AdaptFormer-16 (-0.09%)).

Original: Further increasing the middle dimension to 64 will cause a relatively clear performance drop (-0.47%).
Revised: Further increasing the middle dimension to 64 will cause a relatively clear performance drop (-0.47%).

Original: Discussions.
Revised: Discussions.

Original: Although our AdaptFormer-64 does not have a clear advantage compared with VPT [51], our AdaptFormer-1 outperforms VPT by +0.65%
Revised: Although our AdaptFormer-64 does not have a clear advantage compared with VPT [51], our AdaptFormer-1 outperforms VPT by +0.65%

Original: top-1 accuracy with only 0.02M additional parameters.
Revised: top-1 accuracy with only 0.02M additional parameters.

Original: Besides, the trend of classification accuracy changing with middle dimension on ImageNet1k is different from other datasets in our paper, e.g. , AdaptFormer with middle dimension= achieves better top-1 accuracy than with middle dimension=1 on CIFAR-100.
Revised: Besides, the trend of classification accuracy changing with middle dimension on ImageNet1k is different from other datasets in our paper, e.g. , AdaptFormer with middle dimension= achieves better top-1 accuracy than with middle dimension=1 on CIFAR-100.

Original: We empirically find introducing a small number of parameters (AdaptFormer-1) is sufficient for ImageNet-1K fine-tuning, while more introduced parameters will make a larger change to the original model and make it harder for optimization since ImageNet-1K is a subset of ImageNet-21K.
Revised: We empirically find introducing a small number of parameters (AdaptFormer-1) is sufficient for ImageNet-1K fine-tuning, while more introduced parameters will make a larger change to the original model and make it harder for optimization since ImageNet-1K is a subset of ImageNet-21K.

Original: However, for other datasets (e.g.,
Revised: However, for other datasets (e.g.,

Original: CIFAR-100) with no overlap between the fine-tuning datasets and the pre-trained datasets
Revised: CIFAR-100) with no overlap between the fine-tuning datasets and the pre-trained datasets

Original: , more introduced parameters are needed for learning better domain knowledge.
Revised: , more introduced parameters are needed for learning better domain knowledge.

Original: A.5 Extended experiments on middle dimension
Revised: A.5 Extended experiments on middle dimension

Original: We conduct the extended ablation studies on the middle dimension design in this sub-section.
Revised: We conduct the extended ablation studies on the middle dimension design in this sub-section.

Original: We aim to seek for a trade-off between model capacity (i.e., potential) and adaptation efficiency.
Revised: We aim to seek for a trade-off between model capacity (i.e., potential) and adaptation efficiency.

Original: In fact, the middle dimension has a main influence on the parameter size of adapter.
Revised: In fact, the middle dimension has a main influence on the parameter size of adapter.

Original: The higher dimension brings more parameters while the efficiency and storage are limited.
Revised: The higher dimension brings more parameters while the efficiency and storage are limited.

Original: As shown in Table 10, we evaluate several numbers of middle dimension and found that using 64 is optimal to achieve accuracy, light-weight storage, and efficiency.
Revised: As shown in Table 10, we evaluate several numbers of middle dimension and found that using 64 is optimal to achieve accuracy, light-weight storage, and efficiency.

Original: [Table]
Revised: [Table]

Original: A.6 Analysis on the fine-tuning time and inference latency
Revised: A.6 Analysis on the fine-tuning time and inference latency

Original: To analysis the computational efficiency, we compare the fine-tuning time and inference time on a single NVIDIA A100-40G GPU.
Revised: To analysis the computational efficiency, we compare the fine-tuning time and inference time on a single NVIDIA A100-40G GPU.

Original: We utilize SSv2 video classification for this part.
Revised: We utilize SSv2 video classification for this part.

Original: For fine-tuing, we experiment with batchsize of 32.
Revised: For fine-tuing, we experiment with batchsize of 32.

Original: For inference, we test the latency with multiple batch sizes to get a comprehensive comparison under various inference scenarios.
Revised: For inference, we test the latency with multiple batch sizes to get a comprehensive comparison under various inference scenarios.

Original: All the time is measured in milliseconds averaged over 100 trials.
Revised: All the time is measured in milliseconds averaged over 100 trials.

Original: The results are summarized in Table 11 and Table 12.
Revised: The results are summarized in Table 11 and Table 12.

Original: As shown in Table 11, AdaptFormer only costs less than a half of the fine-tuning time compared with the full-tuning.
Revised: As shown in Table 11, AdaptFormer only costs less than a half of the fine-tuning time compared with the full-tuning.

Original: Moreover, AdaptFormer significantly outperforms linear probing in terms of accuracy with a slight longer fine-tuning time.
Revised: Moreover, AdaptFormer significantly outperforms linear probing in terms of accuracy with a slight longer fine-tuning time.

Original: For inference, AdaptFormer introduce a negligible FLOPs and latency compared with the Linear/Full-tuning.
Revised: For inference, AdaptFormer introduce a negligible FLOPs and latency compared with the Linear/Full-tuning.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A.7 Discussion about ImageNet and Kinetics Pre-training
Revised: A.7 Discussion about ImageNet and Kinetics Pre-training

Original: The type of spatiotemporal attention ( divided vs. joint ) determines whether the performance of the model pre-trained on ImageNet can outperform the model pre-trained on Kinetics.
Revised: The type of spatiotemporal attention ( divided vs. joint ) determines whether the performance of the model pre-trained on ImageNet can outperform the model pre-trained on Kinetics.

Original: A similar phenomenon has been discussed in recent work, Uniformer [57], independently.
Revised: A similar phenomenon has been discussed in recent work, Uniformer [57], independently.

Original: We borrow the experimental results from Table 4(c) in Uniformer paper [57].
Revised: We borrow the experimental results from Table 4(c) in Uniformer paper [57].

Original: Specifically, the divided spatiotemporal attention prefers ImageNet to Kinetics-400 for the pre-training dataset.
Revised: Specifically, the divided spatiotemporal attention prefers ImageNet to Kinetics-400 for the pre-training dataset.

Original: The performance of the divided attention model pre-trained on ImageNet outperforms the model pre-trained on Kinetics-400.
Revised: The performance of the divided attention model pre-trained on ImageNet outperforms the model pre-trained on Kinetics-400.

Original: On the contrary, the joint spatiotemporal attention prefers Kinetics-400 to ImgaeNet.
Revised: On the contrary, the joint spatiotemporal attention prefers Kinetics-400 to ImgaeNet.

Original: The joint attention model attains higher top-1 accuracy with Kinetics-400 pretraining compared to ImageNet (53.8 vs. 52.0).
Revised: The joint attention model attains higher top-1 accuracy with Kinetics-400 pretraining compared to ImageNet (53.8 vs. 52.0).

Original: We adopt the joint spatiotemporal attention for all video-related experiments in this work (introduced in Appendix A.1.1).
Revised: We adopt the joint spatiotemporal attention for all video-related experiments in this work (introduced in Appendix A.1.1).

Original: Therefore, our experimental phenomenon is consistent with the joint attention in [57], i.e., Kinetics pretraining is preferable.
Revised: Therefore, our experimental phenomenon is consistent with the joint attention in [57], i.e., Kinetics pretraining is preferable.

Original: A.8 Implementation
Revised: A.8 Implementation

Original: Algorithm 1 Implementation of AdaptMLP in PyTorch-like style.
Revised: Algorithm 1 Implementation of AdaptMLP in PyTorch-like style.

Original: [Table]
Revised: [Table]

Original: .zeros_(
Revised: .zeros_(

Original: [Table]
Revised: [Table]

Original: bias) nn.init.zeros_(
Revised: bias) nn.init.zeros_(

Original: [Table]
Revised: [Table]

Original: proj
Revised: proj

Original: [Table]
Revised: [Table]

Original: bias)#freezeoriginalMLP for_,pinself.original_mlp.named_parameters():p.requires_grad=
Revised: bias)#freezeoriginalMLP for_,pinself.original_mlp.named_parameters():p.requires_grad=

Original: False def forward(self, x): down = self.down_proj(x) down = self.act(down) down = self.dropout(down) up = self.up_proj(down) output = self.original_mlp(x)
Revised: False def forward(self, x): down = self.down_proj(x) down = self.act(down) down = self.dropout(down) up = self.up_proj(down) output = self.original_mlp(x)

Original: + up * self.scale return output
Revised: + up * self.scale return output

Original: The core part of AdaptFormer is replacing the original MLP with AdaptMLP, which consists of the frozen original MLP and newly introduced Down → ReLU → Up layers, which are tunable at the fine-tuning stage.
Revised: The core part of AdaptFormer is replacing the original MLP with AdaptMLP, which consists of the frozen original MLP and newly introduced Down → ReLU → Up layers, which are tunable at the fine-tuning stage.

Original: Algorithms 1 provides the implementation of AdaptMLP written in PyTorch [68].
Revised: Algorithms 1 provides the implementation of AdaptMLP written in PyTorch [68].

Original: For more implementation details, please refer to the provided source code.
Revised: For more implementation details, please refer to the provided source code.