Original: AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Revised: AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Although the pre-trained Vision Transformers (ViTs) achieved great success incomputer vision, adapting a ViT to various image and video tasks is challengingbecause of its heavy computation and storage burdens, where each model needsto be independently and comprehensively ﬁne-tuned to different tasks, limitingits transferability in different domains.
Revised: Although the pre-trained Vision Transformers (ViTs) achieved great success incomputer vision, adapting a ViT to various image and video tasks is challengingbecause of its heavy computation and storage burdens, where each model needsto be independently and comprehensively fine-tuned to different tasks, limitingits transferability in different domains.

Original: To address this challenge, we propose aneffective adaptation approach for Transformer, namely AdaptFormer, which canadapt the pre-trained ViTs into many different image and video tasks efﬁciently.
Revised: To address this challenge, we propose aneffective adaptation approach for Transformer, namely AdaptFormer, which canadapt the pre-trained ViTs into many different image and video tasks efficiently.

Original: It possesses several beneﬁts more appealing than prior arts.
Revised: It possesses several benefits more appealing than prior arts.

Original: Firstly, AdaptFormerintroduces lightweight modules that only add less than 2% extra parameters toa ViT, while it is able to increase the ViT’s transferability without updating itsoriginal pre-trained parameters, signiﬁcantly outperforming the existing 100% fullyﬁne-tuned models on action recognition benchmarks.
Revised: Firstly, AdaptFormerintroduces lightweight modules that only add less than 2% extra parameters toa ViT, while it is able to increase the ViT’s transferability without updating itsoriginal pre-trained parameters, significantly outperforming the existing 100% fullyfine-tuned models on action recognition benchmarks.

Original: Secondly, it can be plug-andplay in different Transformers and scalable to many visual tasks.
Revised: Secondly, it can be plug-andplay in different Transformers and scalable to many visual tasks.

Original: Thirdly, extensiveexperiments on ﬁve image and video datasets show that AdaptFormer largelyimproves ViTs in the target domains.
Revised: Thirdly, extensiveexperiments on five image and video datasets show that AdaptFormer largelyimproves ViTs in the target domains.

Original: For example, when updating just 1.5% extraparameters, it achieves about 10% and 19% relative improvement compared to thefully ﬁne-tuned models on Something-Something v2 and HMDB51, respectively.
Revised: For example, when updating just 1.5% extraparameters, it achieves about 10% and 19% relative improvement compared to thefully fine-tuned models on Something-Something v2 and HMDB51, respectively.

Original:  Introduction
Revised:  Introduction

Original: There is a growing interest in adopting a general neural model to tackle a large variety of differenttasks since it beneﬁts in reducing the need for task-speciﬁc model design and training.
Revised: There is a growing interest in adopting a general neural model to tackle a large variety of differenttasks since it benefits in reducing the need for task-specific model design and training.

Original: Recently,
Revised: Recently,

Original: Transformer [75] demonstrates great potential in this goal considering its success in various ﬁelds,e.g. , natural language processing (NLP) [26, 10, 76, 82], visual recognition [29, 73, 84, 59], denseprediction [77, 11, 92, 90, 80], Generative Adversarial Network (GAN) [49, 45], reinforcementlearning (RL) [18, 16, 81], robotics [47, 24], and etc.
Revised: Transformer [75] demonstrates great potential in this goal considering its success in various fields,e.g. , natural language processing (NLP) [26, 10, 76, 82], visual recognition [29, 73, 84, 59], denseprediction [77, 11, 92, 90, 80], Generative Adversarial Network (GAN) [49, 45], reinforcementlearning (RL) [18, 16, 81], robotics [47, 24] and etc.

Original: However, existing literature in computer visiontend to focus on the same network with task-speciﬁc weights scenario, where a single network isused to train from scratch or fully ﬁne-tune on a speciﬁc dataset, making it infeasible to maintain aseparate model weight for every dataset when the number of task grows, especially for the increasingmodel capacity of state-of-the-art models ( e.g. , ViT-G/14 [87] with over 1.8 billion parameters).
Revised: However, existing literature in computer visiontend to focus on the same network with task-specific weights scenario, where a single network isused to train from scratch or fully fine-tune on a specific dataset, making it infeasible to maintain aseparate model weight for every dataset when the number of task grows, especially for the increasingmodel capacity of state-of-the-art models ( e.g. , ViT-G/14 [87] with over 1.8 billion parameters).

Original: Different from prior arts, we step into the direction of developing same network with almost sameweights and achieve superior performance than the full-tuning approach by only tuning less than2% parameters, with the remaining over 98% parameters shared across different tasks.
Revised: Different from prior arts, we step into the direction of developing same network with almost sameweights and achieve superior performance than the full-tuning approach by only tuning less than2% parameters, with the remaining over 98% parameters shared across different tasks.

Original: There aretwo challenges to learning universal representations using a single model.
Revised: There aretwo challenges to learning universal representations using a single model.

Original: The ﬁrst one lies in thepre-training stage, which requires algorithms that can learn well-generalized representations that areeasy to be applied to many tasks.
Revised: The first one lies in thepre-training stage, which requires algorithms that can learn well-generalized representations that areeasy to be applied to many tasks.

Original: Recent arts in self-supervised learning [12, 5, 40, 91, 79, 72, 32]can serve as a solution to this challenge.
Revised: Recent arts in self-supervised learning [12, 5, 40, 91, 79, 72, 32]can serve as a solution to this challenge.

Original: The second one, which is our main concern in this work, is to build an effective pipeline that can adapt the model obtained at the pre-training stage to variousdownstream tasks by tuning parameters as less as possible and keeping the left parameters frozen.
Revised: The second one, which is our main concern in this work, is to build an effective pipeline that can adapt the model obtained at the pre-training stage to variousdownstream tasks by tuning parameters as less as possible and keeping the left parameters frozen.

Original: While ﬁne-tuning pre-trained models has been widely studied in NLP [6, 43, 65, 66, 55, 53, 44, 86,58, 39], this topic is seldomly explored in the vision, where full tine-tuning of model parameters isstill the dominant strategy for adapting vision transformers.
Revised: While fine-tuning pre-trained models has been widely studied in NLP [6, 43, 65, 66, 55, 53, 44, 86,58, 39], this topic is seldomly explored in the vision, where full tine-tuning of model parameters isstill the dominant strategy for adapting vision transformers.

Original: However, the full ﬁne-tuning cannotsatisfy the goal of universal representation as it assigns an independent set of weights for every task.
Revised: However, the full fine-tuning cannotsatisfy the goal of universal representation as it assigns an independent set of weights for every task.

Original: Linear probing is a straightforward approach to maintaining the pre-trained model ﬁxed by onlytuning a speciﬁc lightweight classiﬁcation head for every task.
Revised: Linear probing is a straightforward approach to maintaining the pre-trained model fixed by onlytuning a specific lightweight classification head for every task.

Original: However, linear probing tends tohave an unsatisfactory performance and misses the opportunity of pursuing strong but non-linearfeatures [40], which indeed beneﬁt deep learning.
Revised: However, linear probing tends tohave an unsatisfactory performance and misses the opportunity of pursuing strong but non-linearfeatures [40], which indeed benefit deep learning.

Original: More recently, Bahng et.al.
Revised: More recently, Bahng et.al.

Original: , [4] aimed to adaptpre-trained models by modifying raw input pixel space.
Revised: , [4] aimed to adaptpre-trained models by modifying raw input pixel space.

Original: Jia et.al.
Revised: Jia et.al.

Original: , [48] proposed Visual Prompt
Revised: , [48] proposed Visual Prompt

Original: Tuning (VPT) to adapt transformer models for downstream vision tasks, which prepends severallearnable parameters ( prompts ) to the patch embeddings and freezes the whole pre-trained backbone.
Revised: Tuning (VPT) to adapt transformer models for downstream vision tasks, which prepends severallearnable parameters ( prompts ) to the patch embeddings and freezes the whole pre-trained backbone.

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose a lightweight module,namely AdaptFormer, to adapt vision transformers by updating the weights of AdaptFormer.
Revised: In this work, we propose a lightweight module,namely AdaptFormer, to adapt vision transformers by updating the weights of AdaptFormer.

Original: Weintroduce learnable parameters from the modelperspective, which is different from VPT, whichinserts learnable parameters into the token space.
Revised: Weintroduce learnable parameters from the modelperspective, which is different from VPT, whichinserts learnable parameters into the token space.

Original: Our AdaptFormer is conceptually simple yeteffective.
Revised: Our AdaptFormer is conceptually simple yeteffective.

Original: It consists of two fully connectedlayers, a non-linear activation function, and ascaling factor.
Revised: It consists of two fully connectedlayers, a non-linear activation function, and ascaling factor.

Original: This module is set in parallel tothe feed-forward network (FFN) of the original
Revised: This module is set in parallel tothe feed-forward network (FFN) of the original

Original: ViT model, as shown in Figure 2b.
Revised: ViT model, as shown in Figure 2b.

Original: This designis turned out to be effective for model transferwhen processing scalable visual tokens for bothimage and video data (i.e., image data consistsof a small scale of visual tokens while videodata consists of a large scale).
Revised: This designis turned out to be effective for model transferwhen processing scalable visual tokens for bothimage and video data (i.e., image data consistsof a small scale of visual tokens while videodata consists of a large scale).

Original: As shown in Figure 1, compared with the full-tuning strategy,
Revised: As shown in Figure 1, compared with the full-tuning strategy,

Original: AdaptFormer achieves comparable performanceon video recognition with only about 0.1% tunable parameters.
Revised: AdaptFormer achieves comparable performanceon video recognition with only about 0.1% tunable parameters.

Original: Meanwhile, with less than 2% tunableparameters, AdaptFormer surpasses the full-tuning solution by about 10% on top-1 accuracy.
Revised: Meanwhile, with less than 2% tunableparameters, AdaptFormer surpasses the full-tuning solution by about 10% on top-1 accuracy.

Original: Similarapproaches are also proposed in ﬁne-tuning pre-trained language models (PLMs) [6, 43, 66, 39].
Revised: Similarapproaches are also proposed in fine-tuning pre-trained language models (PLMs) [6, 43, 66, 39].

Original: The key contributions of this paper are summarized as follows: (1) We propose a simple yet effectiveframework, namely AdaptFormer, for adapting vision transformers to a large variety of downstreamvisual recognition tasks and avoiding catastrophic interference with each other.
Revised: The key contributions of this paper are summarized as follows: (1) We propose a simple yet effectiveframework, namely AdaptFormer, for adapting vision transformers to a large variety of downstreamvisual recognition tasks and avoiding catastrophic interference with each other.

Original: To the best ofour knowledge, this is the ﬁrst work that explores efﬁcient ﬁne-tuning in video action recognition.(2) We ablate many design choices and demonstrate the superior robustness of AdaptFormer whenparameters scale up.
Revised: To the best ofour knowledge, this is the first work that explores efficient fine-tuning in video action recognition.(2) We ablate many design choices and demonstrate the superior robustness of AdaptFormer whenparameters scale up.

Original: (3) Extensive experiments on various downstream tasks demonstrate that
Revised: (3) Extensive experiments on various downstream tasks demonstrate that

Original: AdaptFormer outperforms existing ﬁne-tuning approaches signiﬁcantly.
Revised: AdaptFormer outperforms existing fine-tuning approaches significantly.

Original: By demonstrating theeffectiveness of AdaptFormer on multiple visual benchmarks, we hope our work could inspire theresearch communities to rethink the ﬁne-tuning mechanism in computer vision and make progresstoward a ﬂexible yet universal Transformer model for visual recognition.
Revised: By demonstrating theeffectiveness of AdaptFormer on multiple visual benchmarks, we hope our work could inspire theresearch communities to rethink the fine-tuning mechanism in computer vision and make progresstoward a flexible yet universal Transformer model for visual recognition.

Original:  Related Works
Revised:  Related Works

Original: In the proposed AdaptFormer, we mainly introduce a plug-and-play module for efﬁciently ﬁne-tuningthe current vision Transformer models.
Revised: In the proposed AdaptFormer, we mainly introduce a plug-and-play module for efficiently fine-tuningthe current vision Transformer models.

Original: In this section, we perform a literature review on relatedworks from two perspectives, i.e. , the vision Transformers, and efﬁcient transfer learning for vision
Revised: In this section, we perform a literature review on relatedworks from two perspectives, i.e. , the vision Transformers, and efficient transfer learning for vision

Original: Transformers.
Revised: Transformers.

Original: 2.1 Transformer in Vision
Revised: 2.1 Transformer in Vision

Original: The Transformer architecture is ﬁrst introduced in [75] and has re-energized the natural languageprocessing (NLP) ﬁeld from then on [26, 10].
Revised: The Transformer architecture is first introduced in [75] and has re-energized the natural languageprocessing (NLP) field from then on [26, 10].

Original: Inspired by its huge success, researches in thecomputer vision ﬁled have also evolved into Transformer era since ViTs [29].
Revised: Inspired by its huge success, researches in thecomputer vision filed have also evolved into Transformer era since ViTs [29].

Original: The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including imageclassiﬁcation [29, 59, 57], object detection [11, 92, 21], semantic/instance segmentation [80], videounderstanding [8, 2, 31, 54], point cloud modeling [89, 38], 3D Object Recognition [19] and evenlow-level processing [17, 56, 78].
Revised: The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including imageclassification [29, 59, 57], object detection [11, 92, 21], semantic/instance segmentation [80], videounderstanding [8, 2, 31, 54], point cloud modeling [89, 38], 3D Object Recognition [19] and evenlow-level processing [17, 56, 78].

Original: Furthermore, transformers have advanced the vision recognitionperformance by a large-scale pretraining [20, 63, 13, 33, 40, 72, 67].
Revised: Furthermore, transformers have advanced the vision recognitionperformance by a large-scale pretraining [20, 63, 13, 33, 40, 72, 67].

Original: In such a situation, given thepre-trained Transformer models, which are more larger than the previously prevalent CNN backbones,one open question is how to ﬁne-tune the big vision models so that they can be adapted into morespeciﬁc down-stream tasks.
Revised: In such a situation, given thepre-trained Transformer models, which are more larger than the previously prevalent CNN backbones,one open question is how to fine-tune the big vision models so that they can be adapted into morespecific down-stream tasks.

Original: To solve the open question, we propose AdaptFormer to transfer ViTsfrom the pre-trained pre-texts into the target tasks in a more effective and efﬁcient way.
Revised: To solve the open question, we propose AdaptFormer to transfer ViTsfrom the pre-trained pre-texts into the target tasks in a more effective and efficient way.

Original: 2.2 Efﬁcient Transfer learning for Transformers
Revised: 2.2 Efficient Transfer learning for Transformers

Original: Transfer learning targets re-adopting a pre-trained model (either via the supervised or the unsupervisedmanner) as the starting point and further ﬁne-tuning the speciﬁc model on a new task.
Revised: Transfer learning targets re-adopting a pre-trained model (either via the supervised or the unsupervisedmanner) as the starting point and further fine-tuning the specific model on a new task.

Original: In the NLPﬁeld, transferring the large pre-trained language models (PLMs) [26, 10] into downstream tasks hasbeen the popular paradigm for a long time.
Revised: In the NLPfield, transferring the large pre-trained language models (PLMs) [26, 10] into downstream tasks hasbeen the popular paradigm for a long time.

Original: Conventional arts [26, 10] set all the network parametersas learnable ones and adapt them to the target tasks.
Revised: Conventional arts [26, 10] set all the network parametersas learnable ones and adapt them to the target tasks.

Original: However, with the growth of model sizes andthe complexity of the speciﬁc tasks, the conventional paradigm is inevitably limited by the hugecomputational burden.
Revised: However, with the growth of model sizes andthe complexity of the specific tasks, the conventional paradigm is inevitably limited by the hugecomputational burden.

Original: The NLP community has explored several ways for parameter-efﬁcient transferlearning that only set a few parameters learnable and ﬁne-tune them for efﬁciency.
Revised: The NLP community has explored several ways for parameter-efficient transferlearning that only set a few parameters learnable and fine-tune them for efficiency.

Original: The pioneerworks could be mainly categorized from the token [55, 53] and network perspectives [43, 44, 86, 37].
Revised: The pioneerworks could be mainly categorized from the token [55, 53] and network perspectives [43, 44, 86, 37].

Original: Basically speaking, the token-related methods [53, 55] typically prepend several learnable preﬁxvectors/tokens to the projected tokens within the multi-head self-attention layers (MHSA [75]).
Revised: Basically speaking, the token-related methods [53, 55] typically prepend several learnable prefixvectors/tokens to the projected tokens within the multi-head self-attention layers (MHSA [75]).

Original: Thephilosophy behind it is to assist the pre-trained models in understanding downstream tasks with theguidance of extra token information.
Revised: Thephilosophy behind it is to assist the pre-trained models in understanding downstream tasks with theguidance of extra token information.

Original: On the other hand, network-related methods [43, 44] integrateshallow modules to improve the model transferability.
Revised: On the other hand, network-related methods [43, 44] integrateshallow modules to improve the model transferability.

Original: The introduced modules adapt the producedrepresentations into the downstream tasks via features fusion.
Revised: The introduced modules adapt the producedrepresentations into the downstream tasks via features fusion.

Original: Recently, with the emergence of a much more large-scale dataset [25, 68, 70, 62, 50], increasingresearchers in computer vision have adopted the homologous paradigm, i.e. , ﬁrst pre-training andthen ﬁne-tuning, to advance the vision tasks.
Revised: Recently, with the emergence of a much more large-scale dataset [25, 68, 70, 62, 50], increasingresearchers in computer vision have adopted the homologous paradigm, i.e. , first pre-training andthen fine-tuning, to advance the vision tasks.

Original: As for the second stage, traditional methods typicallyadopt the full-tuning arts in the downstream tasks.
Revised: As for the second stage, traditional methods typicallyadopt the full-tuning arts in the downstream tasks.

Original: Rare attention has been drawn to the ﬁeld ofefﬁcient adaptation, especially in the ﬁeld of vision Transformers.
Revised: Rare attention has been drawn to the field ofefficient adaptation, especially in the field of vision Transformers.

Original: Inspired by Prompting in NLP,[48] introduced the learnable tokens in exploring the efﬁcient adaptation for ViTs.
Revised: Inspired by Prompting in NLP,[48] introduced the learnable tokens in exploring the efficient adaptation for ViTs.

Original: We empiricallyfound that the performance of prompting is hindered by the scale of tokens.
Revised: We empiricallyfound that the performance of prompting is hindered by the scale of tokens.

Original: That is to say, for thetasks where the number of tokens is on a small scale, e.g. , image classiﬁcation, Prompting is efﬁcientfor improving the model transferability.
Revised: That is to say, for thetasks where the number of tokens is on a small scale, e.g. , image classification, Prompting is efficientfor improving the model transferability.

Original: However, for larger scale tokens, e.g. , video understanding,
Revised: However, for larger scale tokens, e.g. , video understanding,

Original: Prompting presents limited potential.
Revised: Prompting presents limited potential.

Original: This observation motivates us to introduce AdaptFormer, whichis effective in the scenarios of scalable visual tokens.
Revised: This observation motivates us to introduce AdaptFormer, whichis effective in the scenarios of scalable visual tokens.

Original:  Approach
Revised:  Approach

Original: We propose AdaptFormer for efﬁciently transferring large pre-trained vision transformer modelsto downstream tasks, in both image and video domains.
Revised: We propose AdaptFormer for efficiently transferring large pre-trained vision transformer modelsto downstream tasks, in both image and video domains.

Original: AdaptFormer attains strong transfer learning abilities by only ﬁne-tuning a small number of extra parameters, circumventing catastrophicinterference among tasks.
Revised: AdaptFormer attains strong transfer learning abilities by only fine-tuning a small number of extra parameters, circumventing catastrophicinterference among tasks.

Original: We illustrate the overall framework of AdaptFormer in Figure 2b.
Revised: We illustrate the overall framework of AdaptFormer in Figure 2b.

Original: 3.1 Preliminary and Notation
Revised: 3.1 Preliminary and Notation

Original: Vision Transformers (ViTs) are ﬁrst introduced by [29] into vision recognition.
Revised: Vision Transformers (ViTs) are first introduced by [29] into vision recognition.

Original: A vanilla vision
Revised: A vanilla vision

Original: Transformer basically consists of a patch embedding layer and several consecutively connectedencoders, as depicted in Figure 2a.
Revised: Transformer basically consists of a patch embedding layer and several consecutively connectedencoders, as depicted in Figure 2a.

Original: Given an image x 2 R H ⇥ W ⇥ 3 , the patch embedding layer ﬁrstsplits and ﬂatten the sample x into sequential patches x p 2 R N ⇥ ( P 2 d ) , where ( H , W ) representsthe height and width of the input image, ( P, P ) is the resolution of each image patch, d denotesthe output channel, and N = HW/P 2 is the number of image tokens.
Revised: Given an image x ∈ R H × W × 3 , the patch embedding layer firstsplits and flatten the sample x into sequential patches x p ∈ R N × ( P 2 d ) , where ( H , W ) representsthe height and width of the input image, ( P, P ) is the resolution of each image patch, d denotesthe output channel, and N = HW/P 2 is the number of image tokens.

Original: The overall combinationof a prepended [CLS] token and the image tokens x p are further fed into Transformer encoders forattention calculation.
Revised: The overall combinationof a prepended [CLS] token and the image tokens x p are further fed into Transformer encoders forattention calculation.

Original: [Figure]
Revised: [Figure]

Original: Transformer mainly consists tokens are linearly projected and further formulatedinto three vectors namely is performed on and

Original: by:
Revised: by:

Original: [Equation]
Revised: [Equation]

Original: where x 0 ` are the tokens produced by MHSA at the ` -th layer.
Revised: where x ′ ℓ are the tokens produced by MHSA at the ℓ -th layer.

Original: The output tokens x 0 ` are further sent toa LayerNorm [3] and a MLP block which is consisted of two fully connected layers with a GELUactivation [42] in between.
Revised: The output tokens x ′ ℓ are further sent toa LayerNorm [3] and a MLP block which is consisted of two fully connected layers with a GELUactivation [42] in between.

Original: This process is formally formulated as follows,
Revised: This process is formally formulated as follows,

Original: [Equation]
Revised: [Equation]

Original:  where x ` is the output of the ` -th encoder block.
Revised: (2) where x ℓ is the output of the ℓ -th encoder block.

Original: At the last transformer layer, the [CLS] is utilizedfor the ﬁnal object recognition.
Revised: At the last transformer layer, the [CLS] is utilizedfor the final object recognition.

Original: We refer the readers to ﬁnd more details in [29].
Revised: We refer the readers to find more details in [29].

Original: In our work, wereplace the MLP layer with our AdaptMLP module for efﬁcient ﬁne-tuning purposes.
Revised: In our work, wereplace the MLP layer with our AdaptMLP module for efficient fine-tuning purposes.

Original: 3.2 AdaptFormer
Revised: 3.2 AdaptFormer

Original: We propose a plug-and-play bottleneck module, namely AdaptMLP 1 .
Revised: We propose a plug-and-play bottleneck module, namely AdaptMLP 1 .

Original: We denote the vision Transformer equipped with AdaptMLP as AdaptFormer.
Revised: We denote the vision Transformer equipped with AdaptMLP as AdaptFormer.

Original: Architecture.
Revised: Architecture.

Original: The design principle of AdaptFormer is simple yet effective, which is illustrated in
Revised: The design principle of AdaptFormer is simple yet effective, which is illustrated in

Original: Figure 2b.
Revised: Figure 2b.

Original: Compared to the vanilla full ﬁne-tuning regime, AdaptFormer replaces the MLP block inthe transformer encoder with AdaptMLP , which is consisted of two sub-branches.
Revised: Compared to the vanilla full fine-tuning regime, AdaptFormer replaces the MLP block inthe transformer encoder with AdaptMLP , which is consisted of two sub-branches.

Original: The MLP layer inthe left branch is identical to the original network, while the right branch is an additionally introducedlightweight module for task-speciﬁc ﬁne-tuning.
Revised: The MLP layer inthe left branch is identical to the original network, while the right branch is an additionally introducedlightweight module for task-specific fine-tuning.

Original: Speciﬁcally, the right branch is designed to be abottleneck structure for limiting the number of parameters purpose, which includes a down-projectionlayer with parameters W down 2 R d ⇥ ˆ d , an up-projection layer with parameters W up 2 R ˆ d ⇥ d , whereˆ d is the bottleneck middle dimension and satisﬁes ˆ d ⌧ d .
Revised: Specifically, the right branch is designed to be abottleneck structure for limiting the number of parameters purpose, which includes a down-projectionlayer with parameters W down ∈ R d × ˆ d , an up-projection layer with parameters W up ∈ R ˆ d × d , whereˆ d is the bottleneck middle dimension and satisfies ˆ d ≪ d .

Original: In addition, there is a ReLU layer [1]between these projection layers for non-linear property.
Revised: In addition, there is a ReLU layer [1]between these projection layers for non-linear property.

Original: This bottleneck module is connected to theoriginal MLP network (left branch) through the residual connection via a scale factor s .
Revised: This bottleneck module is connected to theoriginal MLP network (left branch) through the residual connection via a scale factor s .

Original: For a speciﬁcinput feature x 0 ` , the right branch in AdaptMLP produces the adapted features, ˜ x ` , formally via:
Revised: For a specificinput feature x ′ ℓ , the right branch in AdaptMLP produces the adapted features, ˜ x ℓ , formally via:

Original: [Equation]
Revised: [Equation]

Original: Then both the features ˜ x ` and x 0 ` are fused with x ` by residual connection,
Revised: Then both the features ˜ x ℓ and x ′ ℓ are fused with x ℓ by residual connection,

Original: [Equation]
Revised: [Equation]

Original: Fine-tuning.
Revised: Fine-tuning.

Original: During the ﬁne-tuning phase, the original model parts (blue blocks in Figure 2b)load weights from the pre-trained checkpoint and keeps untouched, avoiding interaction amongdownstream tasks.
Revised: During the fine-tuning phase, the original model parts (blue blocks in Figure 2b)load weights from the pre-trained checkpoint and keeps untouched, avoiding interaction amongdownstream tasks.

Original: The newly added parameters (orange blocks) are updated on the speciﬁc datadomain with the task-speciﬁc losses.
Revised: The newly added parameters (orange blocks) are updated on the specific datadomain with the task-specific losses.

Original: Inference.
Revised: Inference.

Original: After ﬁne-tuning, we still keep the shared parameters frozen as in the previous ﬁnetuning state, and additionally load the weights of the extra parameters that were ﬁne-tuned in theprevious stage.
Revised: After fine-tuning, we still keep the shared parameters frozen as in the previous finetuning state, and additionally load the weights of the extra parameters that were fine-tuned in theprevious stage.

Original: The single overall model is able to be adapted to multiple tasks with the assistance oflightweight introduced modules.
Revised: The single overall model is able to be adapted to multiple tasks with the assistance oflightweight introduced modules.

Original: 3.3 Discussion
Revised: 3.3 Discussion

Original: Tunable parameters analysis.
Revised: Tunable parameters analysis.

Original: Our AdaptMLP module is lightweight.
Revised: Our AdaptMLP module is lightweight.

Original: The total number of parameters introduced to per layer is 2 ⇥ d ⇥ ˆ d + ˆ d + d , which includes biases parameters.
Revised: The total number of parameters introduced to per layer is 2 × d × ˆ d + ˆ d + d , which includes biases parameters.

Original: The middledimension ˆ d is a small value compared with d (AdaptFormer still obtains a decent performance evenwhen ˆ d = 1 , as discussed in Sec. 4.4).
Revised: The middledimension ˆ d is a small value compared with d (AdaptFormer still obtains a decent performance evenwhen ˆ d = 1 , as discussed in Sec. 4.4).

Original: Since most of the shared parameters are ﬁxed and the numberof newly introduced parameters is small ( < 2% of the pre-trained model parameters), the total modelsize grows slowly when more downstream tasks are added.
Revised: Since most of the shared parameters are fixed and the numberof newly introduced parameters is small ( < 2% of the pre-trained model parameters), the total modelsize grows slowly when more downstream tasks are added.

Original: [Figure]
Revised: [Figure]

Original: Applicability.
Revised: Applicability.

Original: We note that AdaptMLP is a plug-and-playmodule that can be adaptively inserted into existing popular vision transformer architectures [29, 59, 77, 84, 22, 28]since all of the backbones share the same MLP layers eventhough they differ in the MHSA architectures (as shown in
Revised: We note that AdaptMLP is a plug-and-playmodule that can be adaptively inserted into existing popular vision transformer architectures [29, 59, 77, 84, 22, 28]since all of the backbones share the same MLP layers eventhough they differ in the MHSA architectures (as shown in

Original: Figure 2b).
Revised: Figure 2b).

Original: Compared to our methods, we notice that recentprompt-related approaches insert trainable parameters intothe token space, as illustrated in Figure 3.
Revised: Compared to our methods, we notice that recentprompt-related approaches insert trainable parameters intothe token space, as illustrated in Figure 3.

Original: They prependlearnable parameters either into the embedded tokens beforelinear projection [55] or the key and value tokens after linear projection [48].
Revised: They prependlearnable parameters either into the embedded tokens beforelinear projection [55] or the key and value tokens after linear projection [48].

Original: Therefore, the prompt-related methodcan not be straightforwardly adapted to special MHSA variants, especially for the one that takes the pyramid spatialinformation into account [59, 77].
Revised: Therefore, the prompt-related methodcan not be straightforwardly adapted to special MHSA variants, especially for the one that takes the pyramid spatialinformation into account [59, 77].

Original: Besides, we empiricallyobserve that prompt-related methods perform not well whenthe number of patch tokens grows up from image to videoscale, as shown in Figure 1.
Revised: Besides, we empiricallyobserve that prompt-related methods perform not well whenthe number of patch tokens grows up from image to videoscale, as shown in Figure 1.

Original: In summary, we present a strategy for tuning a pre-trained vision Transformer on a set of scalablevision recognition tasks ( e.g. image domain and video domain).
Revised: In summary, we present a strategy for tuning a pre-trained vision Transformer on a set of scalablevision recognition tasks ( e.g. image domain and video domain).

Original: It adds limited learnable parametersfor tuning while achieving comparable or even better performance than the full-tuning strategy.
Revised: It adds limited learnable parametersfor tuning while achieving comparable or even better performance than the full-tuning strategy.

Original: Moreover, AdaptFormer could serve as a generic module for a large variety of recognition tasks.
Revised: Moreover, AdaptFormer could serve as a generic module for a large variety of recognition tasks.

Original:  Experiments
Revised:  Experiments

Original: We evaluate the effectiveness of AdaptFormer by conducting extensive visual recognition experimentsin both the image and video domains.
Revised: We evaluate the effectiveness of AdaptFormer by conducting extensive visual recognition experimentsin both the image and video domains.

Original: We ﬁrst describe our experimental settings in Sec. 4.1, coveringthe pre-trained backbones, baseline methods, downstream tasks and training details.
Revised: We first describe our experimental settings in Sec. 4.1, coveringthe pre-trained backbones, baseline methods, downstream tasks and training details.

Original: We then compare
Revised: We then compare

Original: AdaptFormer with baseline methods and provide a thorough analysis in Sec.
Revised: AdaptFormer with baseline methods and provide a thorough analysis in Sec.

Original: In addition, we alsoconduct ablation studies to explore different experimental conﬁgurations and explain what makes forthe superiority of AdaptFormer in Sec 4.4.
Revised: In addition, we alsoconduct ablation studies to explore different experimental configurations and explain what makes forthe superiority of AdaptFormer in Sec 4.4.

Original: 4.1 Experimental Settings
Revised: 4.1 Experimental Settings

Original: Pre-trained backbone.
Revised: Pre-trained backbone.

Original: We adopt the plain Vision Transformer (ViT) [29], i.e. , ViT-Base (ViT-B/16)as our backbone model and pre-train the model with both supervised and self-supervised approaches.
Revised: We adopt the plain Vision Transformer (ViT) [29], i.e. , ViT-Base (ViT-B/16)as our backbone model and pre-train the model with both supervised and self-supervised approaches.

Original: Speciﬁcally, for image , we directly use the ImageNet-21k [25] supervised pre-trained model 2 and
Revised: Specifically, for image , we directly use the ImageNet-21k [25] supervised pre-trained model 2 and

Original: MAE [40] self-supervised model 3 .
Revised: MAE [40] self-supervised model 3 .

Original: For video , we take both supervised and self-supervised
Revised: For video , we take both supervised and self-supervised

Original: pre-trained
Revised: pre-trained

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Initialization of AdaptFormer.
Revised: Initialization of AdaptFormer.

Original: For the original networks, we directly load the weights pre-trainedon the upstream tasks and keep them frozen/untouched during the ﬁne-tuning process.
Revised: For the original networks, we directly load the weights pre-trainedon the upstream tasks and keep them frozen/untouched during the fine-tuning process.

Original: For the newlyadded modules, the weights of down-projection layers are initialized with Kaiming Normal [41],while the biases of the additional networks and the weights of the up-projection layers are conﬁguredwith zero initialization.
Revised: For the newlyadded modules, the weights of down-projection layers are initialized with Kaiming Normal [41],while the biases of the additional networks and the weights of the up-projection layers are configuredwith zero initialization.

Original: Baseline methods.
Revised: Baseline methods.

Original: We compare AdaptFormer with three commonly used ﬁne-tuning approaches,including (1) Linear probing: adding an extra linear layer on top of the backbone and tuning the addedparameters for evaluation.
Revised: We compare AdaptFormer with three commonly used fine-tuning approaches,including (1) Linear probing: adding an extra linear layer on top of the backbone and tuning the addedparameters for evaluation.

Original: (2) Full Fine-tuning: setting all the parameters learnable and tuning themtogether.
Revised: (2) Full Fine-tuning: setting all the parameters learnable and tuning themtogether.

Original: (3) Visual Prompt Tuning (VPT): [48] ﬁne-tuning the extra token parameters as shown in
Revised: (3) Visual Prompt Tuning (VPT): [48] fine-tuning the extra token parameters as shown in

Original: Figure 3.
Revised: Figure 3.

Original: Downstream tasks.
Revised: Downstream tasks.

Original: We evaluate our AdaptFormer on both image and video recognition tasks toverify its effectiveness.
Revised: We evaluate our AdaptFormer on both image and video recognition tasks toverify its effectiveness.

Original: The speciﬁc datasets leveraged in this work are presented in the following.
Revised: The specific datasets leveraged in this work are presented in the following.

Original: • Image domain : CIFAR-100 [51] contains 50,000 training images and 10,000 validation imagesof resolution 32 × 32 with 100 labels.
Revised: • Image domain : CIFAR-100 [51] contains 50,000 training images and 10,000 validation imagesof resolution 32×32 with 100 labels.

Original: Street View House Numbers (SVHN) [34] is a digit classiﬁcationbenchmark dataset.
Revised: Street View House Numbers (SVHN) [34] is a digit classificationbenchmark dataset.

Original: In total, the dataset comprises over 600,000 labeled images, containing 73,257training samples, 26,032 testing samples and 531,131 extra training data.
Revised: In total, the dataset comprises over 600,000 labeled images, containing 73,257training samples, 26,032 testing samples and 531,131 extra training data.

Original: The Food-101 [9] datasetconsists of 101 food categories with a total of 101k images, including 750 training and 250 testingsamples per category.• Video domain : Something-Something V2 (SSv2) [36] is a large collection of video clipsshowing the people perform several normal actions in the daily life ( e
Revised: The Food-101 [9] datasetconsists of 101 food categories with a total of 101k images, including 750 training and 250 testingsamples per category. • Video domain : Something-Something V2 (SSv2) [36] is a large collection of video clipsshowing the people perform several normal actions in the daily life ( e

Original: g. , moving stuff and openingthe door).
Revised: g. , moving stuff and openingthe door).

Original: It consists of 168,913 training samples, 24,777 validation samples and 27,157 testingsamples, making a total of 220,847 videos with 174 labels.
Revised: It consists of 168,913 training samples, 24,777 validation samples and 27,157 testingsamples, making a total of 220,847 videos with 174 labels.

Original: HMDB51 [52] is composed of 6,849videos with 51 categories, making a split of 3.5k/1.5k train/val videos.
Revised: HMDB51 [52] is composed of 6,849videos with 51 categories, making a split of 3.5k/1.5k train/val videos.

Original: Implementation details.
Revised: Implementation details.

Original: In this work, we use PyTorch toolkit [64] to conduct all experiments on
Revised: In this work, we use PyTorch toolkit [64] to conduct all experiments on

Original: NVIDIA V100 GPUs.
Revised: NVIDIA V100 GPUs.

Original: Unless otherwise stated, we use 8 ⇥ 8 GPUs for video experiments and 1 ⇥ 

Revised: Unless otherwise stated, we use 8 × 8 GPUs for video experiments and 1 ×

Original: GPUs for image experiments.
Revised: GPUs for image experiments.

Original: Our default conﬁgurations follow the linear probing settings in [20, 40],which do not utilize many common regularization strategies, such as mixup [88], cutmix [85], colorjittering and so on.
Revised: Our default configurations follow the linear probing settings in [20, 40],which do not utilize many common regularization strategies, such as mixup [88], cutmix [85], colorjittering and so on.

Original: More details can be found in Appendix.
Revised: More details can be found in Appendix.

Original: 4.2 Main Properties and Analysis
Revised: 4.2 Main Properties and Analysis

Original: We compare the performance of different ﬁne-tuning approaches in Table 1 with the backbones pretrained via the self-supervised paradigms.
Revised: We compare the performance of different fine-tuning approaches in Table 1 with the backbones pretrained via the self-supervised paradigms.

Original: The results show that AdaptFormer consistently surpasseslinear probing and Visual Prompt tuning (VPT) methods.
Revised: The results show that AdaptFormer consistently surpasseslinear probing and Visual Prompt tuning (VPT) methods.

Original: Speciﬁcally, AdaptFormer outperforms VPTon image benchmark CIFAR-100, SVHN, and Food-101, by 9.50%, 5.95%, and 10.04% respectively.
Revised: Specifically, AdaptFormer outperforms VPTon image benchmark CIFAR-100, SVHN, and Food-101, by 9.50%, 5.95%, and 10.04% respectively.

Original: On the more challenging video action recognition dataset Something-Something V2, the superioritybecomes even more signiﬁcant, i.e. , about 34.96%.
Revised: On the more challenging video action recognition dataset Something-Something V2, the superioritybecomes even more significant, i.e. , about 34.96%.

Original: Note that even compared with the full ﬁne-tuning strategy, our AdaptFormer still outperforms by about 10% Top-1 accuracy on SSv2 dataset.
Revised: Note that even compared with the full fine-tuning [Figure] [Figure] strategy, our AdaptFormer still outperforms by about 10% Top-1 accuracy on SSv2 dataset.

Original: Tosummarize, our AdaptFormer is highly parameter-efﬁcient, as well as yielding good performancewith parameter size at most 2% times than the full ﬁne-tuning manner.
Revised: Tosummarize, our AdaptFormer is highly parameter-efficient, as well as yielding good performancewith parameter size at most 2% times than the full fine-tuning manner.

Original: 4.3 Scaling Tunable Parameters Up
Revised: 4.3 Scaling Tunable Parameters Up

Original: Even though there are only limited parameters introduced, one might also argue that more tunableparameters of AdaptFormer contribute to its higher accuracy compared with VPT [48].
Revised: Even though there are only limited parameters introduced, one might also argue that more tunableparameters of AdaptFormer contribute to its higher accuracy compared with VPT [48].

Original: We conductexperiments to make a comprehensive discussion on this aspect.
Revised: We conductexperiments to make a comprehensive discussion on this aspect.

Original: As described in Sec. 3.3, the number of tunable parameters can be adjusted by changing the number ofintroduced tokens for VPT, or the hidden feature dimension for AdaptFormer.
Revised: As described in Sec. 3.3, the number of tunable parameters can be adjusted by changing the number ofintroduced tokens for VPT, or the hidden feature dimension for AdaptFormer.

Original: As shown in Figure 4,we conduct experiments with a wide range of tunable parameters on both SSv2 and HMDB-51datasets.
Revised: As shown in Figure 4,we conduct experiments with a wide range of tunable parameters on both SSv2 and HMDB-51datasets.

Original: Since AdaptFormer and VPT share the same number of parameters of classiﬁcation headon a speciﬁc dataset, we only report the tunable parameters on the x-axis, which comes from thevisual prompts (VPT) or weight/bias of the down-up fully-connected layers (AdaptFormer), withoutcalculating the parameters of classiﬁcation head.
Revised: Since AdaptFormer and VPT share the same number of parameters of classification headon a specific dataset, we only report the tunable parameters on the x-axis, which comes from thevisual prompts (VPT) or weight/bias of the down-up fully-connected layers (AdaptFormer), withoutcalculating the parameters of classification head.

Original: For VPT, the number of introduced tokens is chosenfrom {1, 2, 4, 8, 16, 32, 48, 64}.
Revised: For VPT, the number of introduced tokens is chosenfrom {1, 2, 4, 8, 16, 32, 48, 64}.

Original: Similarly, the number of hidden dimensions in AdaptFormer is in {1,2, 4, 8, 16, 32}.
Revised: Similarly, the number of hidden dimensions in AdaptFormer is in {1,2, 4, 8, 16, 32}.

Original: AdaptFormer has a slight performance gain or maintains the accuracy stably whenthe parameters scale up.
Revised: AdaptFormer has a slight performance gain or maintains the accuracy stably whenthe parameters scale up.

Original: On the contrary, the performance of VPT decreases dramatically when theparameters exceed the task-speciﬁc value.
Revised: On the contrary, the performance of VPT decreases dramatically when theparameters exceed the task-specific value.

Original: Moreover, choosing the most suitable number of tokennumber becomes laborious since it might be task-speciﬁc ( i.e. varying from one dataset to the otherone).
Revised: Moreover, choosing the most suitable number of tokennumber becomes laborious since it might be task-specific ( i.e. varying from one dataset to the otherone).

Original: For example, the accuracy of VPT keeps going up when the number of tunable parametersincreases up to 300K on SSv2, whereas it begins to drop when the number of tunable parametersexceeds 50K on HMDB-51.
Revised: For example, the accuracy of VPT keeps going up when the number of tunable parametersincreases up to 300K on SSv2, whereas it begins to drop when the number of tunable parametersexceeds 50K on HMDB-51.

Original: We further study the optimization procedures of VPT by monitoring the test accuracy of the trainingstage.
Revised: We further study the optimization procedures of VPT by monitoring the test accuracy of the trainingstage.

Original: As shown in Figure 5, we gradually increase the number of tokens in VPT and plot the Top-1accuracy of each epoch.
Revised: As shown in Figure 5, we gradually increase the number of tokens in VPT and plot the Top-1accuracy of each epoch.

Original: The training stages are stable when the number of tokens is less than or equalto 4, e.g. , {1, 2, 4}.
Revised: The training stages are stable when the number of tokens is less than or equalto 4, e.g. , {1, 2, 4}.

Original: However, when the number becomes 8 or larger, e.g. , {8, 16, 32}, the trainingprocedure collapses at about the tenth epoch and achieves poor performance at the end of the trainingstage.
Revised: However, when the number becomes 8 or larger, e.g. , {8, 16, 32}, the trainingprocedure collapses at about the tenth epoch and achieves poor performance at the end of the trainingstage.

Original: On the contrary, the optimization procedures of AdaptFormer are stable when the number ofparameters varies across a large range, as shown in Table 2a.
Revised: On the contrary, the optimization procedures of AdaptFormer are stable when the number ofparameters varies across a large range, as shown in Table 2a.

Original: The top-1 accuracy ﬂuctuates within1.5% when the number of parameters increases from 0.44M ( dim=16 ) to 4.87M ( dim=256 ).
Revised: The top-1 accuracy fluctuates within1.5% when the number of parameters increases from 0.44M ( dim=16 ) to 4.87M ( dim=256 ).

Original: 4.4 Ablation Studies
Revised: 4.4 Ablation Studies

Original: We ablate our AdaptFormer to study what properties make for a good AdaptFormer and observeseveral intriguing properties.
Revised: We ablate our AdaptFormer to study what properties make for a good AdaptFormer and observeseveral intriguing properties.

Original: The ablation studies conducted in this work are all performed on the
Revised: The ablation studies conducted in this work are all performed on the

Original: SSv2 validation set [36].
Revised: SSv

Original: Middle dimension.
Revised: Middle dimension.

Original: The middle dimension controls the number of introduced parameters by AdaptFormer.
Revised: The middle dimension controls the number of introduced parameters by AdaptFormer.

Original: Lower middle dimensions introduce fewer parameters with a possible performance cost.
Revised: Lower middle dimensions introduce fewer parameters with a possible performance cost.

Original: Weablate AdaptFormer on the middle feature dimension to study this effects.
Revised: Weablate AdaptFormer on the middle feature dimension to study this effects.

Original: As shown in Table 2a,the accuracy consistently improves when the middle dimension increases up to 64 and reaches thesaturation point when the middle dimension is about 64.
Revised: As shown in Table 2a,the accuracy consistently improves when the middle dimension increases up to 64 and reaches thesaturation point when the middle dimension is about 64.

Original: We note that our AdaptFormer can achieve adecent performance when the middle dimension reduces even to one, about 50.03% top-1 accuracy.
Revised: We note that our AdaptFormer can achieve adecent performance when the middle dimension reduces even to one, about 50.03% top-1 accuracy.

Original: [Table]
Revised: [Table]

Original: Scaling factor.
Revised: Scaling factor.

Original: The scaling factor s is introduced to balance the task-agnostic features (generatedby the original frozen branch) and the task-speciﬁc features (generated by the tunable bottleneckbranch).
Revised: The scaling factor s is introduced to balance the task-agnostic features (generatedby the original frozen branch) and the task-specific features (generated by the tunable bottleneckbranch).

Original: We evaluate AdaptFormer with multiple s values and the results are summarized in Table 2c.
Revised: We evaluate AdaptFormer with multiple s values and the results are summarized in Table 2c.

Original: Different from the scaling factor in NLP ﬁeld which prefer s larger than 1 ( e.g. , s = 4 in [39]), weempirically found that the s should be < 1 for vision tasks, otherwise the ﬁne-tuning would becomeunstable.
Revised: Different from the scaling factor in NLP field which prefer s larger than 1 ( e.g. , s = 4 in [39]), weempirically found that the s should be < 1 for vision tasks, otherwise the fine-tuning would becomeunstable.

Original: Besides, we found that AdaptFormer achieves optimal performance with s = 0 .
Revised: Besides, we found that AdaptFormer achieves optimal performance with s = 0 .

Original: A largeror smaller s would bring slight performance drop.
Revised: A largeror smaller s would bring slight performance drop.

Original: Thus, we choose s = 0 .
Revised: Thus, we choose s = 0 .

Original: 10 as a default setting.
Revised: 10 as a default setting.

Original: AdaptFormer position.
Revised: AdaptFormer position.

Original: As shown in Table 2b, we further ablate on the speciﬁc position to introducethe AdaptMLP block.
Revised: As shown in Table 2b, we further ablate on the specific position to introducethe AdaptMLP block.

Original: We gradually increase the number of AdaptMLP layers with a step of three( start ! end , both included).
Revised: We gradually increase the number of AdaptMLP layers with a step of three( start → end , both included).

Original: We observe that the performance of AdaptFormer has a positivecorrelation with the number of added layers.
Revised: We observe that the performance of AdaptFormer has a positivecorrelation with the number of added layers.

Original: In addition, AdaptFormer prefers the top part (the onefar away from the input image) of the network to the bottom part when introducing the same numberof layers, e.g. , AdaptFormer with 7 ! 12 obtains over 14.5% higher accuracy than 1 ! 6 , thoughboth equipped with six AdaptMLP layers.
Revised: In addition, AdaptFormer prefers the top part (the onefar away from the input image) of the network to the bottom part when introducing the same numberof layers, e.g. , AdaptFormer with 7 → 12 obtains over 14.5% higher accuracy than 1 → 6 , thoughboth equipped with six AdaptMLP layers.

Original: Insertion form.
Revised: Insertion form.

Original: We study the insertion formulation by comparing the parallel and sequentialinstances which are illustrated in Figure 6.
Revised: We study the insertion formulation by comparing the parallel and sequentialinstances which are illustrated in Figure 6.

Original: As shown in Table 2b, the parallel AdaptFormer is ableto outperform the sequential one by 0.85% top-1 accuracy.
Revised: As shown in Table 2b, the parallel AdaptFormer is ableto outperform the sequential one by 0.85% top-1 accuracy.

Original: The reason might be: (1) the paralleldesign maintains the original feature using an independent branch and aggregating updated contextby element-wise scaled sum; (2) the sequential design is equivalent to adding more layers, whichmight cause optimization difﬁculty.
Revised: The reason might be: (1) the paralleldesign maintains the original feature using an independent branch and aggregating updated contextby element-wise scaled sum; (2) the sequential design is equivalent to adding more layers, whichmight cause optimization difficulty.

Original: Therefore, we adopt the parallel design as our default setting dueto its superiority.
Revised: Therefore, we adopt the parallel design as our default setting dueto its superiority.

Original: [Figure]
Revised: [Figure]

Original: Com parison results are shown in Table 2b.
Revised: [Figure]

Original: Number of frames.
Revised: Number of frames.

Original: The number of embedded patch tokens increases linearly with the number ofvideo frames for the plain ViT [29].
Revised: The number of embedded patch tokens increases linearly with the number ofvideo frames for the plain ViT [29].

Original: We conduct experiments with the different number of frames,i.e. , {2, 4, 8} and the results are shown in Figure 7.
Revised: We conduct experiments with the different number of frames,i.e. , {2, 4, 8} and the results are shown in Figure 7.

Original: We observe that increasing the number of framesis beneﬁcial for all these three ﬁne-tuning methods.
Revised: We observe that increasing the number of framesis beneficial for all these three fine-tuning methods.

Original: However, AdaptFormer consistently outperforms the linear manner ( e.g. , +30% top-1 accuracy on 8 input frames) and VPT method( e.g. , +14% top-1accuracy on 8 input frames).
Revised: However, AdaptFormer consistently outperforms the linear manner ( e.g. , +30% top-1 accuracy on 8 input frames) and VPT method( e.g. , +14% top-1accuracy on 8 input frames).

Original: 4.5 Towards Visual Recognition Generalist Agent
Revised: 4.5 Towards Visual Recognition Generalist Agent

Original: In the above experiments, we typically utilize a modality-speciﬁc pre-trained checkpoint forthe corresponding downstream tasks.
Revised: In the above experiments, we typically utilize a modality-specific pre-trained checkpoint forthe corresponding downstream tasks.

Original: For example, we use Kinetics-400 ( video domain ) pretrained model for downstream video action recognition on Something-Something V2 and HMDB51 benchmarks.
Revised: For example, we use Kinetics-400 ( video domain ) pretrained model for downstream video action recognition on Something-Something V2 and HMDB51 benchmarks.

Original: Besides, we use ImageNet-21K ( image domain ) pre-rained model for downstream image classiﬁcation on CIFAR-100, SVHN and Food-101 benchmarks.
Revised: Besides, we use ImageNet-21K ( image domain ) pre-rained model for downstream image classification on CIFAR-100, SVHN and Food-101 benchmarks.

Original: Our AdaptFormerachieves superior performances in this same network with modality-speciﬁc weights scenario.
Revised: Our AdaptFormerachieves superior performances in this same network with modality-specific weights scenario.

Original: [Table]
Revised: [Table]

Original: Next, we take a further step to ask what wouldhappen if using the same network with themodality-agnostic weights for multiple tasks inthe multi-modalities downstream tasks?
Revised: Next, we take a further step to ask what wouldhappen if using the same network with themodality-agnostic weights for multiple tasks inthe multi-modalities downstream tasks?

Original: We use the model pre-trained on ImagNet-21kto do action recognition on SSv2.
Revised: We use the model pre-trained on ImagNet-21kto do action recognition on SSv2.

Original: As shownin Table 3, AdaptFormer is robust to domainshift caused by modality.
Revised: As shownin Table 3, AdaptFormer is robust to domainshift caused by modality.

Original: The experimental results show that the linear probe approach obtainsa very poor accuracy ( i.e. , 6.56% top-1 accuracy) when ﬁne-tuning on SSv2.
Revised: The experimental results show that the linear probe approach obtainsa very poor accuracy ( i.e. , 6.56% top-1 accuracy) when fine-tuning on SSv2.

Original: Meanwhile,
Revised: Meanwhile,

Original: VPT [48] achieves a better performance than linear probe but it is not decent ( i.e. , 16.94% top-1accuracy).
Revised: VPT [48] achieves a better performance than linear probe but it is not decent ( i.e. , 16.94% top-1accuracy).

Original: Our AdaptFormer, compared to the above two methods, attains a promising 46.06% top-1accuracy, which is even higher than the full-tuning schedule (+
Revised: Our AdaptFormer, compared to the above two methods, attains a promising 46.06% top-1accuracy, which is even higher than the full-tuning schedule (+4.56%). [Equation] Visualization

Original: [Figure]
Revised: [Figure]

Original: To evaluate the quality of the produced features, we conduct t-SNE [74] visualizations on AdaptFormer and other baseline methods.
Revised: To evaluate the quality of the produced features, we conduct t-SNE [74] visualizations on AdaptFormer and other baseline methods.

Original: The features are extracted from the SSv2 validation set viathe ViT-Base backbone.
Revised: The features are extracted from the SSv2 validation set viathe ViT-Base backbone.

Original: Figure 8 shows that the linear ﬁne-tuning and the VPT methods tend tooutput mixed features as shown in Figure 8(a)-(b).
Revised: Figure 8 shows that the linear fine-tuning and the VPT methods tend tooutput mixed features as shown in Figure 8(a)-(b).

Original: Compared with the above two methods, the fullﬁne-tuning strategy performs well in projecting features.
Revised: Compared with the above two methods, the fullfine-tuning strategy performs well in projecting features.

Original: However, it consumes huge computationalsources to tune the whole network parameters.
Revised: However, it consumes huge computationalsources to tune the whole network parameters.

Original: Figure 8(d) validates that our AdaptFormer facilitates
Revised: Figure 8(d) validates that our AdaptFormer facilitates

Original: ViT-Base in generating more separable representations with fewer learnable parameters.
Revised: ViT-Base in generating more separable representations with fewer learnable parameters.

Original:  Conclusion
Revised:  Conclusion

Original: We present a conceptually simple yet effective framework, AdaptFormer, for efﬁciently adapting apre-trained Vision Transformer (ViT) backbone to scalable vision recognition tasks.
Revised: We present a conceptually simple yet effective framework, AdaptFormer, for efficiently adapting apre-trained Vision Transformer (ViT) backbone to scalable vision recognition tasks.

Original: By introducing
Revised: By introducing

Original: AdaptMLP, our AdaptFormer is able to ﬁne-tune the lightweight modules for producing featuresadapted to multiple downstream tasks.
Revised: AdaptMLP, our AdaptFormer is able to fine-tune the lightweight modules for producing featuresadapted to multiple downstream tasks.

Original: The extensive experiments on ﬁve datasets, covering boththe image and the video domains, validate that our proposed methods are able to increase the
Revised: The extensive experiments on five datasets, covering boththe image and the video domains, validate that our proposed methods are able to increase the

Original: ViT’s transferability with little computational cost.
Revised: ViT’s transferability with little computational cost.

Original: We hope our work will inspire future researchin exploring more efﬁcient ﬁne-tuning methods for large vision models.
Revised: We hope our work will inspire future researchin exploring more efficient fine-tuning methods for large vision models.

Original: One limitation is that
Revised: One limitation is that

Original: AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can workwell in tasks beyond recognition, e.g. , object detection and semantic segmentation.
Revised: AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can workwell in tasks beyond recognition, e.g. , object detection and semantic segmentation.

Original: We leave it forthe future exploration.
Revised: We leave it forthe future exploration.

Original: Since our method is specially designed for efﬁcient ﬁne-tuning, we do notforesee obvious undesirable ethical/social impacts at this moment. 
Revised: Since our method is specially designed for efficient fine-tuning, we do notforesee obvious undesirable ethical/social impacts at this moment. Checklist For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Revised: contributions and scope?

Revised: [Yes]

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] Shown in Conclusion Section.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] Shown inConclusion Section.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A]

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [N/A]

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Revised: imental results (either in the supplemental material or as a URL)?

Revised: [Yes] As a URL

Revised: shown in the abstract.

Revised: (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters, how theywere chosen)?

Revised: [Yes] Shown in supplementary materials.

Revised: (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experiments multiple times)?

Revised: [N/A]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [Yes] Please see Section 4.14.

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a)

Revised: If your work uses existing assets, did you cite the creators?

Revised: [Yes]

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] Shown in supplementary materials.

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: [Table]

Revised: using/curating?

Revised: [Yes] We used publicly available datasets whose licenses allow research

Revised: [Table]

Revised: usage.

Revised: [Table]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Revised: [No] To the best of our knowledge, the data we usedcontains no personally identifiable information or offensive content.

Revised: [Table]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [Table]

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: [Table]

Revised: applicable?

Revised: [N/A]

Revised: [Table]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: [Table]

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: [Table]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Revised: spent on participant compensation?

Revised: [N/A]

Revised: [Table]

Revised: A Appendix

Revised: In this supplementary material, we will include the details about the pre-training and fine-tuningprocesses, the extensive experiments of AdaptFormer on hierarchical vision transformers ( e.g. ,

Revised: AdaptFormer-Swin), and the pseudo-code of AdaptMLP in a PyTorch-like style.

Revised: A.1 Experimental SettingsA.1.1 Pre-training Approaches

Revised: Image.

Revised: We use MAE [40] as our self-supervised pre-training method in the image domain, a simpleyet effective method that first masks nearly 75% patches of the input image and then reconstructs themissing pixels.

Revised: Specifically, we directly adopt the checkpoint 4 of ViT-B/16 for convenience, which ispre-trained on ImageNet-1K [25] for 800 epochs.

Revised: Video.

Revised: We use VideoMAE [72] as our self-supervised pre-training method in the video domain,which is an direct extension of MAE to the video domain.

Revised: VideoMAE utilizes the plain ViT [29]architecture of joint space-time attention mechanism [2, 60] and an extremely high proportionof masking ratio ( i.e. , 90% to 95%) for pre-training.

Revised: We also directly use the publicly availablecheckpoint 5 , which is pre-trained on Kinetics-400 [15].

Revised: A.1.2 Implementation Details of Fine-tuning

Revised: [Table]

Revised: The implementation details are summarized in Table 4.

Revised: The video experiments are conducted on64 Tesla V100 GPUs, while the image experiments are performed on 8 Tesla V100 GPUs.

Revised: Forthe optimizer, different from [83] that adopts LARS, we leverage SGD [71] for stable training onsmall-scale dataset ( e.g. , CIFAR10).

Revised: The actual learning rate is calculated by: lr = base_lr × batchsize/ 256 following the linear lr scaling rule [35].

Revised: More detailed training configurations are presented in

Revised: Table 4, including the batchsize, learning rate schedule and etc.

Revised: The experimental settings of image and video mainly follow the ones utilized in MAE [40] and

Revised: VideoMAE [72], respectively.

Revised: We insert an extra BatchNorm layer [46] without affine transformation ( i.e. affine=

Revised: False ) before the final fully connected layer, following the common practice tonormalize the pre-trained features [27, 40].

Revised: In addition, there is no flip augmentation during thefine-tuning stage for video data.

Revised: A.2 More Supplementary ResultsA.2.1 AdaptFormer with Supervised Pre-training

Revised: In addition to the self-supervised pre-training presented in the main paper

Revised: [Table]

Revised: we also evaluate AdaptFormer with the supervised pre-trained model.

Revised: The results in Table 5 show that AdaptFormerstill outperforms linear probe and VPT obviously.

Revised: In addition, AdaptFormer surpasses full-tuningon four benchmarks (CIFAR100, SVHN, SSv2, HMDB51) with only 1.46% parameters.

Revised: On theremaining benchmark (Food-101), AdaptFormer achieves an almost comparable performance tofull-tuning (90.89% v.s. 90.96%).

Revised: [Table]

Revised: A.2.2 AdaptFormer

Revised: on Swin

Revised: Transformer

Revised: Settings.

Revised: We further demonstrate the effectiveness of AdaptFormer on hierarchical vision transformers, e.g. , Swin [59, 60].

Revised: We name AdaptFormer applied to Swin as AdaptFormer-Swin , to distinguishplain AdaptFormer (without any suffix) which is applied to the vanilla ViT [29].

Revised: It is noted that wecan adopt AdaptMLP to Swin easily without any special modification as Swin and ViT share thesame MLP architecture.

Revised: However, VPT [48] needs additional handcraft designs to be suitable forthe shifted local windows in the prevalent hierarchical vision transformers, which hinders its generalapplications.

Revised: We utilize Swin-B [59] and the video counterpart [60] for image and video.

Revised: Similarly, we alsodirectly use the officially provided checkpoints 6 , which are pre-trained on ImageNet-21K [25] and

Revised: Kinetics-600 [14].

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Results.

Revised: Since VPT is not applicable in Swin, we do not report its performance.

Revised: Table 6 shows

Revised: AdaptFormer-Swin performs well compared with other tuning strategies.

Revised: For image benchmarks, ourmethod can outperform full-tuning approach with only 1.43% parameters.

Revised: Moreover, AdaptFormerSwin surpasses linear probing by a significant margin, especially on the challenging dataset, SSv2.

Revised: The results validate that AdaptFormer is able to generally boost the transferability of various vision

Revised: Transformer variants.

Revised: A.3 Multi-Label Classification

Revised: We further conduct experiments on dataset with larger scale and diversity.

Revised: Specifically, we evaluate

Revised: AdaptFormer on NUS-WIDE [23] for multi-label classification.

Revised: NUS-WIDE contains 269,648images collected from Flicker, which are annotated with 81 visual concepts.

Revised: Since some images arenot available on Flicker, we only use 220,000 images following [7, 30].

Revised: We utilize mean averageprecision (mAP) as performance metric.

Revised: Settings: Our training settings mainly follow ASL [7].

Revised: Specifically, We trained all models for 40epochs using Adam optimize and 1-cycle learning rate policy [69].

Revised: The maximal learning rate is0.001.

Revised: [Table]

Revised: A.4 Possible Architectures

Revised: We explore other possible architectures utilized in AdaptFormer.

Revised: Specifically, we further replace the

Revised: MLP architectures within the AdaptMLP module by the convolution layer, depthwise convolutionlayer, and LayerNorm layer.

Revised: For fair comparisons, we carefully design the above modules to meet thecomparable number of parameters ( ˜ 1.3M).

Revised: The experimental results of different adapter modules areshown in Table 8, which validates that the simple MLP modules are simple yet effective comparedwith the other architectures.

Revised: For example, our AdaptMLP module surpasses the AdaptConv moduleby 0.55% Top1 accuracy on SSv2 dataset.

Revised: [Table]

Revised: A.5 Evaluation on ImageNet-1k datasets

Revised: To further conduct the evaluation on the ImageNet dataset, we directly load the weights pre-trainedon ImageNet-21K, and evaluate the AdaptFormer on ImageNet-1k.

Revised: The results in Table 9 validatethat the performance of AdaptFormer is comparable to the full fine-tuning strategy with only 1.5%tunable parameters.

Revised: Meanwhile, our AdaptFormer outperforms the linear probe and VPT by 0.81%and 0.14% , respectively.

Revised: A.6 Extended experiments on middle dimension

Revised: We conduct the extended ablation studies on the middle dimension design in this sub-section.

Revised: Weaim to seek for a trade-off between model capacity (i.e., potential) and adaptation efficiency.

Revised: In fact,the middle dimension has a main influence on the parameter size of adapter.

Revised: The higher dimension

Revised: [Table]

Revised: brings more parameters while the efficiency and storage are limited.

Revised: As shown in Table 10, weevaluate several numbers of middle dimension and found that using 64 is optimal to achieve accuracy,light-weight storage, and efficiency.

Revised: [Table]

Revised: A.7 Analysis on the fine-tuning time and inference latency

Revised: To analysis the computational efficiency, we compare the fine-tuning time and inference time on asingle NVIDIA A100-40G GPU.

Revised: We utilize SSv2 video classification for this part.

Revised: For fine-tuing,we experiment with batchsize of 32.

Revised: For inference, we test the latency with multiple batch sizesto get a comprehensive comparison under various inference scenarios.

Revised: All the time is measured inmilliseconds averaged over 100 trials.

Revised: The results are summarized in Table 11 and Table 12.

Revised: Asshown in Table 11, AdaptFormer only costs less than a half of the fine-tuning time compared with thefull-tuning.

Revised: Moreover, AdaptFormer significantly outperforms linear probing in terms of accuracywith a slight longer fine-tuning time.

Revised: For inference, AdaptFormer introduce a negligible FLOPs andlatency compared with the Linear/Full-tuning.

Revised: [Table]

Revised: The core part of AdaptFormer is replacing the original MLP with AdaptMLP, which consists of thefrozen original MLP and newly introduced Down → ReLU → Up layers, which are tunable at thefine-tuning stage.

Revised: Algorithms 1 provides the implementation of AdaptMLP written in PyTorch [64].

Revised: For more implementation details, please refer to the provided source code.

Revised: [Table]

Revised: [Table]