Original: AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Revised: AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Although the pre-trained Vision Transformers (ViTs) achieved great success incomputer vision, adapting a ViT to various image and video tasks is challengingbecause of its heavy computation and storage burdens, where each model needsto be independently and comprehensively fine-tuned to different tasks, limitingits transferability in different domains.
Revised: Although the pre-trained Vision Transformers (ViTs) achieved great success incomputer vision, adapting a ViT to various image and video tasks is challengingbecause of its heavy computation and storage burdens, where each model needsto be independently and comprehensively fune-tuned to different tasks, limitingits transferability in different domains.

Original: To address this challenge, we propose aneffective adaptation approach for Transformer, namely AdaptFormer, which canadapt the pre-trained ViTs into many different image and video tasks efficiently.
Revised: To address this challenge, we propose aneffective adaptation approach for Transformer, namely AdaptFormer, which canadapt the pre-trained ViTs into many different image and video tasks efficiently.

Original: It possesses several benefits more appealing than prior arts.
Revised: It possesses several benefits more appealing than prior arts.

Original: Firstly, AdaptFormerintroduces lightweight modules that only add less than 2% extra parameters toa ViT, while it is able to increase the ViT’s transferability without updating itsoriginal pre-trained parameters, significantly outperforming the existing 100% fullyfine-tuned models on action recognition benchmarks.
Revised: Firstly, AdaptFormerintroduces lightweight modules that only adds less than 2% extra parameters toa ViT, while it is able to increase the ViT’s transferability without updating itsoriginal pre-trained parameters, significantly outperforming the existing 100% fullyfine-tuned models on action recognition benchmarks.

Original: Secondly, it can be plug-andplay in different Transformers and scalable to many visual tasks.
Revised: Secondly, it can be plug-andplay in different Transformers and scalable to many visual tasks.

Original: Thirdly, extensiveexperiments on five image and video datasets show that AdaptFormer largelyimproves ViTs in the target domains.
Revised: Thirdly, extensiveexperiments on five image and video datasets show that AdaptFormer largelyimproves ViTs in the target domains.

Original: For example, when updating just 1.5% extraparameters, it achieves about 10% and 19% relative improvement compared to thefully fine-tuned models on Something-Something v2 and HMDB51, respectively.
Revised: For example, when updating just 1.5% extraparameters, it achieves about 10% and 19% relative improvement compared to thefully fine-tuned models on Something-Something v2 and HMDB51, respectively.

Revised: The deliverables are released at anonymous-adaptformer.github.io .

Original:  Introduction
Revised:  Introduction

Original: There is a growing interest in adopting a general neural model to tackle a large variety of differenttasks since it benefits in reducing the need for task-specific model design and training.
Revised: There is a growing interest in adopting a general neural model to tackle a large variety of differenttasks since it benefits in reducing the need for task-specific model design and training.

Original: Recently,
Revised: Recently,

Original: Transformer [75] demonstrates great potential in this goal considering its success in various fields,e.g. , natural language processing (NLP) [26, 10, 76, 82], visual recognition [29, 73, 84, 59], denseprediction [77, 11, 92, 90, 80], Generative Adversarial Network (GAN) [49, 45], reinforcementlearning (RL) [18, 16, 81], robotics [47, 24] and etc.
Revised: Transformer [60] demonstrates great potential in this goal considering its success in various fields,e.g. , natural language processing (NLP) [22, 9, 61, 67], visual recognition [24, 58, 68, 49], denseprediction [62, 10, 76, 74, 65], Generative Adversarial Network (GAN) [40, 37], reinforcementlearning (RL) [15, 13, 66], robotics [38, 20], and etc.

Original: However, existing literature in computer visiontend to focus on the same network with task-specific weights scenario, where a single network isused to train from scratch or fully fine-tune on a specific dataset, making it infeasible to maintain aseparate model weight for every dataset when the number of task grows, especially for the increasingmodel capacity of state-of-the-art models ( e.g. , ViT-G/14 [87] with over 1.8 billion parameters).
Revised: However, existing literature in computer visiontend to focus on the same network with task-specific weights scenario, where a single network isused to train from scratch or fully fine-tune on a specific dataset, making it infeasible to maintain aseparate model weight for every dataset when the number of task grows, especially for the increasingmodel capacity of state-of-the-art models ( e.g. , ViT-G/14 [71] with over 1.8 billion parameters).

Original: Different from  prior arts, we step into the direction of developing same network with almost sameweights and achieve superior performance than the full-tuning approach by only tuning less than2% parameters, with the remaining over 98% parameters shared across different tasks.
Revised: Different from the prior arts, we step into the direction of developing same network with almost sameweights and achieve superior performance than the full-tuning approach by only tuning less than2% parameters, with the remaining over 98% parameters shared across different tasks.

Original: There aretwo challenges to learning universal representations using a single model.
Revised: There aretwo challenges to learning universal representations using a single model.

Original: The first one lies in thepre-training stage, which requires algorithms that can learn well-generalized representations that areeasy to be applied to many tasks.
Revised: The first one lies in thepre-training stage, which requires algorithms that can learn well-generalized representations that areeasy to be applied to many tasks.

Original: Recent arts in self-supervised learning [12, 5, 40, 91, 79, 72, 32]can serve as a solution to this challenge.
Revised: Recent arts in self-supervised learning [11, 5, 32, 75, 64] can serveas a solution to this challenge.

Original: The second one, which is our main concern in this work, is to build an effective pipeline that can adapt the model obtained at the pre-training stage to variousdownstream tasks by tuning parameters as less as possible and keeping the left parameters frozen.
Revised: The second challenge, which is our main concern in this work, is to build an effective pipeline that can adapt the model obtained at the pre-training stage to a various ofdownstream tasks by tuning parameters as less as possible and keeping the left parameters frozen.

Original: While fine-tuning pre-trained models has been widely studied in NLP [6, 43, 65, 66, 55, 53,  44, 86,58, 39], this topic is seldomly explored in the vision, where full tine-tuning of model parameters isstill the dominant strategy for adapting vision transformers.
Revised: While fine-tuning pre-trained models has been widely studied in NLP [6, 35, 52, 53, 46, 44, 36, 70,48, 31], this topic is seldomly explored in the vision, where full tine-tuning of model parameters isstill the dominant strategy for adapting vision transformers.

Original: However, the full fine-tuning cannotsatisfy the goal of universal representation as it assigns an independent set of weights for every task.
Revised: However, the full fine-tuning cannotsatisfy the goal of universal representation as it assigns an independent set of weights for every task.

Original: Linear probing is a straightforward approach to maintaining the pre-trained model fixed by onlytuning a specific lightweight classification head for every task.
Revised: Linear probing is a straightforward approach to maintaining the pre-trained model fixed by onlytuning a specific lightweight classification head for every task.

Original: However, linear probing tends tohave an unsatisfactory performance and misses the opportunity of pursuing strong but non-linearfeatures [40], which indeed benefit deep learning.
Revised: However, linear probing tends tohave an unsatisfactory performance and misses the opportunity of pursuing strong but non-linearfeatures [32], which indeed benefit deep learning.

Original: More recently, Bahng et.al.
Revised: More recently, Bahng et.al.

Original: , [4] aimed to adaptpre-trained models by modifying raw input pixel space.
Revised: , [4] aimed to adaptpre-trained models by modifying raw input pixel space.

Original: Jia et.al.
Revised: Jia et.al.

Original: , [48] proposed Visual Prompt
Revised: , [39] proposed Visual Prompt

Original: Tuning (VPT) to adapt transformer models for downstream vision tasks, which prepends severallearnable parameters ( prompts ) to the patch embeddings and freezes the whole pre-trained backbone.
Revised: Tuning (VPT) to adapt transformer models for downstream vision tasks, which prepends severallearnable parameters ( prompts ) to the patch embeddings and freezes the whole pre-trained backbone.

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose a lightweight module,namely AdaptFormer, to adapt vision transformers by updating the weights of AdaptFormer.
Revised: In this work, we propose a lightweight module,namely AdaptFormer, to adapt vision transformers by updating the weights of AdaptFormer.

Original: Weintroduce learnable parameters from the modelperspective, which is different from VPT, whichinserts learnable parameters into the token space.
Revised: Weintroduce learnable parameters from the modelperspective, which is different from VPT, whichinserts learnable parameters into the token space.

Original: Our AdaptFormer is conceptually simple yeteffective.
Revised: Our AdaptFormer is conceptually simple yeteffective.

Original: It consists of two fully connectedlayers, a non-linear activation function, and ascaling factor.
Revised: It consists of two fully connectedlayers, a non-linear activation function, and ascaling factor.

Original: This module is set in parallel tothe feed-forward network (FFN) of the original
Revised: This module is set in parallel tothe feed-forward network (FFN) of the original

Original: ViT model, as shown in Figure 2b.
Revised: ViT model, as shown in Figure 2b.

Original: This designis turned out to be effective for model transferwhen processing scalable visual tokens for bothimage and video data (i.e., image data consistsof a small scale of visual tokens while videodata consists of a large scale).
Revised: This designis turned out to be effective for model transferwhen processing scalable visual tokens for bothimage and video data (i.e., image data consistsof a small scale of visual tokens while videodata consists of a large scale).

Original: As shown in Figure 1, compared with the full-tuning strategy,
Revised: As shown in Figure 1, compared with the full-tuning strategy,

Original: AdaptFormer achieves comparable performanceon video recognition with only about 0.1% tunable parameters.
Revised: AdaptFormer achieves comparable performanceon video recognition with only about 0.1% tunable parameters.

Original: Meanwhile, with less than 2% tunableparameters, AdaptFormer surpasses the full-tuning solution by about 10% on top-1 accuracy.
Revised: Meanwhile, with less than 2% tunableparameters, AdaptFormer surpasses the full-tuning solution by about 10% on top-1 accuracy.

Original: Similarapproaches are also proposed in fine-tuning pre-trained language models (PLMs) [6, 43, 66, 39].
Revised: Similarapproaches are also proposed in fine-tuning pre-trained language models (PLMs) [6, 35, 53, 31].

Original: The key contributions of this paper are summarized as follows: (1) We propose a simple yet effectiveframework, namely AdaptFormer, for adapting vision transformers to a large variety of downstreamvisual recognition tasks and avoiding catastrophic interference with each other.
Revised: The key contributions of this paper are summarized as follows: (1) We propose a simple yet effectiveframework, namely AdaptFormer, for adapting vision transformers to a large variety of downstreamvisual recognition tasks and avoiding catastrophic interference with each other.

Original: To the best ofour knowledge, this is the first work that explores efficient fine-tuning in video action recognition.(2) We ablate many design choices and demonstrate the superior robustness of AdaptFormer whenparameters scale up.
Revised: To the best ofour knowledge, this is the first work that explores efficient fine-tuning in video action recognition.(2) We ablate many design choices and demonstrate the superior robustness of AdaptFormer whenparameters scale up.

Original: (3) Extensive experiments on various downstream tasks demonstrate that
Revised: (3) Extensive experiments on various downstream tasks demonstrate that

Original: AdaptFormer outperforms existing fine-tuning approaches significantly.
Revised: AdaptFormer outperforms existing fine-tuning approaches significantly.

Original: By demonstrating theeffectiveness of AdaptFormer on multiple visual benchmarks, we hope our work could inspire theresearch communities to rethink the fine-tuning mechanism in computer vision and make progresstoward a flexible yet universal Transformer model for visual recognition.
Revised: By demonstrating theeffectiveness of AdaptFormer on multiple visual benchmarks, we hope our work could inspire theresearch communities to rethink the fine-tuning mechanism in computer vision and make progresstoward a flexible yet universal Transformer model for visual recognition.

Original:  Related Works
Revised:  Related Works

Original: In the proposed AdaptFormer, we mainly introduce a plug-and-play module for efficiently fine-tuningthe current vision Transformer models.
Revised: In the proposed AdaptFormer, we mainly introduce a plug-and-play module for efficiently fine-tuningthe current vision Transformer models.

Original: In this section, we perform a literature review on relatedworks from two perspectives, i.e. , the vision Transformers, and efficient transfer learning for vision
Revised: In this section, we perform a literature review on relatedworks from two perspectives, i.e. , the vision Transformers, and efficient transfer learning for vision

Original: Transformers.
Revised: Transformers.

Original: 2.1 Transformer in Vision
Revised: 2.1 Transformer in Vision

Original: The Transformer architecture is first introduced in [75] and has re-energized the natural languageprocessing (NLP) field from then on [26, 10].
Revised: The Transformer architecture is first introduced in [60] and has re-energized the natural languageprocessing (NLP) field from then on [22, 9].

Original: Inspired by its huge success, researches in thecomputer vision filed have also evolved into Transformer era since ViTs [29].
Revised: Inspired by its huge success, researches in the computervision filed have also evolved into Transformer era since ViTs [24].

Original: The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including imageclassification [29, 59, 57], object detection [11, 92, 21], semantic/instance segmentation [80], videounderstanding [8, 2, 31, 54], point cloud modeling [89, 38], 3D Object Recognition [19] and evenlow-level processing [17, 56, 78].
Revised: The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including imageclassification [24, 49], object detection [10, 76, 18], semantic/instance segmentation [65], videounderstanding [7, 2, 25, 45], point cloud modeling [73, 30], 3D Object Recognition [16] and evenlow-level processing [14, 47, 63].

Original: Furthermore, transformers have advanced the vision recognitionperformance by a large-scale pretraining [20, 63, 13, 33, 40, 72, 67].
Revised: Furthermore, transformers have advanced the vision recognitionperformance by a large-scale pretraining [17, 12, 26, 32, 57, 54].

Original: In such a situation, given thepre-trained Transformer models, which are more larger than the previously prevalent CNN backbones,one open question is how to fine-tune the big vision models so that they can be adapted into morespecific down-stream tasks.
Revised: In such a situation, given thepre-trained Transformer models, which are more larger than the previously prevalent CNN backbones,one open question is how to fine-tune the big vision models so that they can be adapted into morespecific down-stream tasks.

Original: To solve the open question, we propose AdaptFormer to transfer ViTsfrom the pre-trained pre-texts into the target tasks in a more effective and efficient way.
Revised: To solve the open question, we propose AdaptFormer to transfer ViTsfrom the pre-trained pre-texts into the target tasks in a more effective and efficient way.

Original: 2.2 Efficient Transfer learning for Transformers
Revised: 2.2 Efficient Transfer learning for Transformers

Original: Transfer learning targets re-adopting a pre-trained model (either via the supervised or the unsupervisedmanner) as the starting point and further fine-tuning the specific model on a new task.
Revised: Transfer learning targets re-adopting a pre-trained model (either via the supervised or the unsupervisedmanner) as the starting point and further fine-tuning the specific model on a new task.

Original: In the NLPfield, transferring the large pre-trained language models (PLMs) [26, 10] into downstream tasks hasbeen the popular paradigm for a long time.
Revised: In the NLPfield, transferring the large pre-trained language models (PLMs) [22, 9] into downstream tasks hasbeen the popular paradigm for a long time.

Original: Conventional arts [26, 10] set all the network parametersas learnable ones and adapt them to the target tasks.
Revised: Conventional arts [22, 9] set all the network parametersas learnable ones and adapt them to the target tasks.

Original: However, with the growth of model sizes andthe complexity of the specific tasks, the conventional paradigm is inevitably limited by the hugecomputational burden.
Revised: However, with the growth of model sizes andthe complexity of the specific tasks, the conventional paradigm is inevitably limited by the hugecomputational burden.

Original: The NLP community has explored several ways for parameter-efficient transferlearning that only set a few parameters learnable and fine-tune them for efficiency.
Revised: The NLP community has explored several ways for parameter-efficient transferlearning that only set a few parameters learnable and fine-tune them for efficiency.

Original: The pioneerworks could be mainly categorized from the token [55, 53] and network perspectives [43, 44, 86, 37].
Revised: The pioneerworks could be mainly categorized from the token [46, 44] and network perspectives [35, 36, 70, 29].

Original: Basically speaking, the token-related methods [53, 55] typically prepend several learnable prefixvectors/tokens to the projected tokens within the multi-head self-attention layers (MHSA [75]).
Revised: Basically speaking, the token-related methods [44, 46] typically prepend several learnable prefixvectors/tokens to the projected tokens within the multi-head self-attention layers (MHSA [60]).

Original: Thephilosophy behind it is to assist the pre-trained models in understanding downstream tasks with theguidance of extra token information.
Revised: Thephilosophy behind it is to assist the pre-trained models in understanding downstream tasks with theguidance of extra token information.

Original: On the other hand, network-related methods [43, 44] integrateshallow modules to improve the model transferability.
Revised: On the other hand, network-related methods [35, 36] integrateshallow modules to improve the model transferability.

Original: The introduced modules adapt the producedrepresentations into the downstream tasks via features fusion.
Revised: The introduced modules adapt the producedrepresentations into the downstream tasks via features fusion.

Original: Recently, with the emergence of a much more large-scale dataset [25, 68, 70, 62, 50], increasingresearchers in computer vision have adopted the homologous paradigm, i.e. , first pre-training andthen fine-tuning, to advance the vision tasks.
Revised: Recently, with the emergence of a much more large-scale dataset [21, 55, 56, 50, 41], increasingresearchers in computer vision have adopted the homologous paradigm, i.e. , first pre-training andthen fine-tuning, to advance the vision tasks.

Original: As for the second stage, traditional methods typicallyadopt the full-tuning arts in the downstream tasks.
Revised: As for the second stage, traditional methods typicallyadopt the full-tuning arts in the downstream tasks.

Original: Rare attention has been drawn to the field ofefficient adaptation, especially in the field of vision Transformers.
Revised: Rare attention has been drawn to the field ofefficient adaptation, especially in the field of vision Transformers.

Original: Inspired by Prompting in NLP,[48] introduced the learnable tokens in exploring the efficient adaptation for ViTs.
Revised: Inspired by Prompting in NLP,[39] introduced the learnable tokens in exploring the efficient adaptation for ViTs.

Original: We empiricallyfound that the performance of prompting is hindered by the scale of tokens.
Revised: We empiricallyfound that the performance of prompting is hindered by the scale of tokens.

Original: That is to say, for thetasks where the number of tokens is on a small scale, e.g. , image classification, Prompting is efficientfor improving the model transferability.
Revised: That is to say, for thetasks where the number of tokens is on a small scale, e.g. , image classification, Prompting is efficientfor improving the model transferability.

Original: However, for larger scale tokens, e.g. , video understanding,
Revised: However, for larger scale tokens, e.g. , video understanding,

Original: Prompting presents limited potential.
Revised: Prompting presents limited potential.

Original: This observation motivates us to introduce AdaptFormer, whichis effective in the scenarios of scalable visual tokens.
Revised: This observation motivates us to introduce AdaptFormer, whichis effective in the scenarios of scalable visual tokens.

Original:  Approach
Revised:  Approach

Original: We propose AdaptFormer for efficiently transferring large pre-trained vision transformer modelsto downstream tasks, in both image and video domains.
Revised: We propose AdaptFormer for efficiently transferring large pre-trained vision transformer modelsto downstream tasks, in both image and video domains.

Original: AdaptFormer attains strong transfer learning abilities by only fine-tuning a small number of extra parameters, circumventing catastrophicinterference among tasks.
Revised: AdaptFormer attains strong transfer learning abilities by only fine-tuning a small number of extra parameters, circumventing catastrophicinterference among tasks.

Original: We illustrate the overall framework of AdaptFormer in Figure 2b.
Revised: We illustrate the overall framework of AdaptFormer in Figure 2b.

Original: 3.1 Preliminary and Notation
Revised: 3.1 Preliminary and Notation

Original: Vision Transformers (ViTs) are first introduced by [29] into vision recognition.
Revised: Vision Transfomers (ViTs) are first introduced by [24] into vision recognition.

Original: A vanilla vision
Revised: A vanilla vision

Original: Transformer basically consists of a patch embedding layer and several consecutively connectedencoders, as depicted in Figure 2a.
Revised: Transformer basically consists of a patch embedding layer and several consecutively connectedencoders, as depicted in Figure 2a.

Original: Given an image x ∈ R H × W × 3 , the patch embedding layer firstsplits and flatten the sample x into sequential patches x p ∈ R N ×
Revised: Given an image x ∈ R H × W × 3 , the patch embedding layer firstsplits and flatten the sample x into sequential patches x p ∈ R N ×

Original: ( P 2 d ) , where ( H , W ) representsthe height and width of the input image, ( P, P ) is the resolution of each image patch, d denotesthe output channel, and N = HW/P 2 is the number of image tokens.
Revised: ( P 2 d ) , where ( H , W ) representsthe height and width of the input image, ( P, P ) is the resolution of each image patch, d denotesthe output channel, and N = HW/P 2 is the number of image tokens.

Original: The overall combinationof a prepended [CLS] token and the image tokens x p are further fed into Transformer encoders forattention calculation.
Revised: The overall combinationof a prepended [CLS] token and the image tokens x p are further fed into Transformer encoders forattention calculation.

Original: [Figure]
Revised: [Figure]

Original: by:
Revised: by:

Original: [Equation]
Revised: [Equation]

Original: where x ′ ℓ are the tokens produced by MHSA at the ℓ -th layer.
Revised: where x ′ ℓ are the tokens produced by MHSA at the ℓ -th layer.

Original: The output tokens x ′ ℓ are further sent toa LayerNorm [3] and a MLP block which is consisted of two fully connected layers with a GELUactivation [42] in between.
Revised: The output tokens x ′ ℓ are further sent toa LayerNorm [3] and a MLP block which is consisted of two fully connected layers with a GELUactivation [34] in between.

Original: This process is formally formulated as follows,
Revised: This process is formally formulated as follows,

Original: [Equation]
Revised: [Equation]

Original: (2) where x ℓ is the output of the ℓ -th encoder block.
Revised: ) where x ℓ is the output of the ℓ -th encoder block.

Original: At the last transformer layer, the [CLS] is utilizedfor the final object recognition.
Revised: At the last transformer layer, the [CLS] is utilizedfor the final object recognition.

Original: We refer the readers to find more details in [29].
Revised: We refer the readers to find more details in [24].

Original: In our work, wereplace the MLP layer with our AdaptMLP module for efficient fine-tuning purposes.
Revised: In our work, wereplace the MLP layer with our AdaptMLP module for efficient fine-tuning purposes.

Original: 3.2 AdaptFormer
Revised: 3.2 AdaptFormer

Original: We propose a plug-and-play bottleneck module, namely AdaptMLP 1 .
Revised: We propose a plug-and-play bottleneck module, namely AdaptMLP 1 .

Original: We denote the vision Transformer equipped with AdaptMLP as AdaptFormer.
Revised: We denote the vision Transformer equipped with AdaptMLP as AdaptFormer.

Original: Architecture.
Revised: Architecture.

Original: The design principle of AdaptFormer is simple yet effective, which is illustrated in
Revised: The design principle of AdaptFormer is simple yet effective, which is illustrated in

Original: Figure 2b.
Revised: Figure 2b.

Original: Compared to the vanilla full fine-tuning regime, AdaptFormer replaces the MLP block inthe transformer encoder with AdaptMLP , which is consisted of two sub-branches.
Revised: Compared to the vanilla full fine-tuning regime, AdaptFormer replaces the MLP block inthe transformer encoder with AdaptMLP , which is consisted of two sub-branches.

Original: The MLP layer inthe left branch is identical to the original network, while the right branch is an additionally introducedlightweight module for task-specific fine-tuning.
Revised: The MLP layer inthe left branch is identical to the original network, while the right branch is an additionally introducedlightweight module for task-specific fine-tuning.

Original: Specifically, the right branch is designed to be abottleneck structure for limiting the number of parameters purpose, which includes a down-projectionlayer with parameters W down
Revised: Specifically, the right branch is designed to be abottleneck structure for limiting the number of parameters purpose, which includes a down-projectionlayer with parameters W down

Original: ∈ R
Revised: ∈ R

Original: d × ˆ d , an up-projection layer with parameters W up ∈ R ˆ d × d , whereˆ d is the bottleneck middle dimension and satisfies ˆ d ≪ d .
Revised: d × ˆ d , an up-projection layer with parameters W up ∈ R ˆ d × d , whereˆ d is the bottleneck middle dimension and satisfies ˆ d ≪ d .

Original: In addition, there is a ReLU layer [1]between these projection layers for non-linear property.
Revised: In addition, there is a ReLU layer [1]between these projection layers for non-linear property.

Original: This bottleneck module is connected to theoriginal MLP network (left branch) through the residual connection via a scale factor s .
Revised: This bottleneck module is connected to theoriginal MLP network (left branch) through the residual connection via a scale factor s .

Original: For a specificinput feature x ′ ℓ , the right branch in AdaptMLP produces the adapted features, ˜ x ℓ , formally via:
Revised: For a specificinput feature x ′ ℓ , the right branch in AdaptMLP produces the adapted features, ˜ x ℓ , formally via:

Original: [Equation]
Revised: [Equation]

Original: Then both the features ˜ x ℓ and x ′ ℓ are fused with x ℓ by residual connection,
Revised: Then both the features ˜ x ℓ and x ′ ℓ are fused with x ℓ by residual connection,

Original: [Equation]
Revised: [Equation]

Original: Fine-tuning.
Revised: Fine-tuning.

Original: During the fine-tuning phase, the original model parts (blue blocks in Figure 2b)load weights from the pre-trained checkpoint and keeps untouched, avoiding interaction amongdownstream tasks.
Revised: During the fine-tuning phase, the original model parts (blue blocks in Figure 2b)load weights from the pre-trained checkpoint and keeps untouched, avoiding interaction amongdownstream tasks.

Original: The newly added parameters (orange blocks) are updated on the specific datadomain with the task-specific losses.
Revised: The newly added parameters (orange blocks) are updated on the specific datadomain with the task-specific losses.

Original: Inference.
Revised: Inference.

Original: After fine-tuning, we still keep the shared parameters frozen as in the previous finetuning state, and additionally load the weights of the extra parameters that were fine-tuned in theprevious stage.
Revised: After fine-tuning, we still keep the shared parameters frozen as in the previous finetuning state, and additionally load the weights of the extra parameters that were fine-tuned in theprevious stage.

Original: The single overall model is able to be adapted to multiple tasks with the assistance oflightweight introduced modules.
Revised: The single overall model is able to be adapted to multiple tasks with the assistance oflightweight introduced modules.

Original: 3.3 Discussion
Revised: 3.3 Discussion

Original: Tunable parameters analysis.
Revised: Tunable parameters analysis.

Original: Our AdaptMLP module is lightweight.
Revised: Our AdaptMLP module is lightweight.

Original: The total number of parameters introduced to per layer is 2 × d × ˆ d + ˆ d + d , which includes biases parameters.
Revised: The total number of parameters introduced to per layer is 2 × d × ˆ d + ˆ d + d , which includes biases parameters.

Original: The middledimension ˆ d is a small value compared with d (AdaptFormer still obtains a decent performance evenwhen ˆ d = 1 , as discussed in Sec. 4.4).
Revised: The middledimension ˆ d is a small value compared with d (AdaptFormer still obtains a decent performance evenwhen ˆ d = 1 , as discussed in Sec. 4.4).

Original: Since most of the shared parameters are fixed and the numberof newly introduced parameters is small ( < 2% of the pre-trained model parameters), the total modelsize grows slowly when more downstream tasks are added.
Revised: Since most of the shared parameters are fixed and the numberof newly introduced parameters is small ( < 2% of the pre-trained model parameters), the total modelsize grows slowly when more downstream tasks are added.

Original: [Figure]
Revised: [Figure]

Original: Applicability.
Revised: Applicability.

Original: We note that AdaptMLP is a plug-and-playmodule that can be adaptively inserted into existing popular vision transformer architectures [29, 59, 77, 84, 22, 28]since all of the backbones share the same MLP layers eventhough they differ in the MHSA architectures (as shown in
Revised: We note that AdaptMLP is a plug-and-playmodule that can be adaptively inserted into existing popular vision transformer architectures [24, 49, 62, 68, 19, 23]since all of the backbones share the same MLP layers eventhough they differ in the MHSA architectures (as shown in

Original: Figure 2b).
Revised: Figure 2b).

Original: Compared to our methods, we notice that recentprompt-related approaches insert trainable parameters intothe token space, as illustrated in Figure 3.
Revised: Compared to our methods, we notice that recentprompt-related approaches insert trainable parameters intothe token space, as illustrated in Figure 3.

Original: They prependlearnable parameters either into the embedded tokens beforelinear projection [55] or the key and value tokens after linear projection [48].
Revised: They prependlearnable parameters either into the embedded tokens beforelinear projection [46] or the key and value tokens after linear projection [39].

Original: Therefore, the prompt-related methodcan not be straightforwardly adapted to special MHSA variants, especially for the one that takes the pyramid spatialinformation into account [59, 77].
Revised: Therefore, the prompt-related methodcan not be straightforwardly adapted to special MHSA variants, especially for the one that takes the pyramid spatialinformation into account [49, 62].

Original: Besides, we empiricallyobserve that prompt-related methods perform not well whenthe number of patch tokens grows up from image to videoscale, as shown in Figure 1.
Revised: Besides, we empiricallyobserve that prompt-related methods perform not well whenthe number of patch tokens grows up from image to videoscale, as shown in Figure 1.

Original: In summary, we present a strategy for tuning a pre-trained vision Transformer on a set of scalablevision recognition tasks ( e.g. image domain and video domain).
Revised: In summary, we present a strategy for tuning a pre-trained vision Transformer on a set of scalablevision recognition tasks ( e.g. image domain and video domain).

Original: It adds limited learnable parametersfor tuning while achieving comparable or even better performance than the full-tuning strategy.
Revised: It adds limited learnable parametersfor tuning while achieving comparable or even better performance than the full-tuning strategy.

Original: Moreover, AdaptFormer could serve as a generic module for a large variety of recognition tasks.
Revised: Moreover, AdaptFormer could serve as a generic module for a large variety of recognition tasks.

Original:  Experiments
Revised:  Experiments

Original: We evaluate the effectiveness of AdaptFormer by conducting extensive visual recognition experimentsin both the image and video domains.
Revised: We evaluate the effectiveness of AdaptFormer by conducting extensive visual recognition experimentsin both the image and video domains.

Original: We first describe our experimental settings in Sec. 4.1, coveringthe pre-trained backbones, baseline methods, downstream tasks and training details.
Revised: We first describe our experimental settings in Sec. 4.1, coveringthe pre-trained backbones, baseline methods, downstream tasks and training details.

Original: We then compare
Revised: We then compare

Original: AdaptFormer with baseline methods and provide a thorough analysis in Sec.
Revised: AdaptFormer with baseline methods and provide a thorough analysis in Sec.

Original: In addition, we alsoconduct ablation studies to explore different experimental configurations and explain what makes forthe superiority of AdaptFormer in Sec 4.4.
Revised: In addition, we alsoconduct ablation studies to explore different experimental configurations and explain what makes forthe superiority of AdaptFormer in Sec 4.4.

Original: 4.1 Experimental Settings
Revised: 4.1 Experimental Settings

Original: Pre-trained backbone.
Revised: Pre-trained backbone.

Original: We adopt the plain Vision Transformer (ViT) [29], i.e. , ViT-Base (ViT-B/16)as our backbone model and pre-train the model with both supervised and self-supervised approaches.
Revised: We adopt the plain Vision Transformer (ViT) [24], i.e. , ViT-Base (ViT-B/16)as our backbone model and pre-train the model with both supervised and self-supervised approaches.

Original: Specifically, for image , we directly use the ImageNet-21k [25] supervised pre-trained model 2 and
Revised: Specifically, for image , we directly use the ImageNet-21k [21] supervised pre-trained model 2 and

Original: MAE [40] self-supervised model 3 .
Revised: MAE [32] self-supervised model 3 .

Original: For video , we take both supervised and self-supervised
Revised: For video , we take both supervised and self-supervised

Original: pre-trained
Revised: pre-trained

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: models from VideoMAE [57]. More details about pre-training approaches and datasets can be foundin Appendix.

Original: Initialization of AdaptFormer.
Revised: Initialization of AdaptFormer.

Original: For the original networks, we directly load the weights pre-trainedon the upstream tasks and keep them frozen/untouched during the fine-tuning process.
Revised: For the original networks, we directly load the weights pre-trainedon the upstream tasks and keep them frozen/untouched during the fine-tuning process.

Original: For the newlyadded modules, the weights of down-projection layers are initialized with Kaiming Normal [41],while the biases of the additional networks and the weights of the up-projection layers are configuredwith zero initialization.
Revised: For the newlyadded modules, the weights of down-projection layers are initialized with Kaiming Normal [33],while the biases of the additional networks and the weights of the up-projection layers are configuredwith zero initialization.

Original: Baseline methods.
Revised: Baseline methods.

Original: We compare AdaptFormer with three commonly used fine-tuning approaches,including (1) Linear probing: adding an extra linear layer on top of the backbone and tuning the addedparameters for evaluation.
Revised: We compare AdaptFormer with three commonly used fine-tuning approaches,including (1) Linear probing: adding an extra linear layer on top of the backbone and tuning the addedparameters for evaluation.

Original: (2) Full Fine-tuning: setting all the parameters learnable and tuning themtogether.
Revised: (2) Full Fine-tuning: setting all the parameters learnable and tuning themtogether.

Original: (3) Visual Prompt Tuning (VPT): [48] fine-tuning the extra token parameters as shown in
Revised: (3) Visual Prompt Tuning (VPT): [39] fine-tuning the extra token parameters as shown in

Original: Figure 3.
Revised: Figure 3.

Original: Downstream tasks.
Revised: Downstream tasks.

Original: We evaluate our AdaptFormer on both image and video recognition tasks toverify its effectiveness.
Revised: We evaluate our AdaptFormer on both image and video recognition tasks toverify its effectiveness.

Original: The specific datasets leveraged in this work are presented in the following.
Revised: The specific datasets leveraged in this work are presented in the following.

Original: • Image domain : CIFAR-100 [51] contains 50,000 training images and 10,000 validation imagesof resolution 32×32{+32×
+} with 100 labels.
Revised: • Image domain : CIFAR-100 [42] contains 50,000 training images and 10,000 validation images of resolution {+32×
+} with 100 labels.

Original: Street View House Numbers (SVHN) [34] is a digit classificationbenchmark dataset.
Revised: Street View House Numbers (SVHN) [27] is a digit classificationbenchmark dataset.

Original: In total, the dataset comprises over 600,000 labeled images, containing 73,257training samples, 26,032 testing samples and 531,131 extra training data.
Revised: In total, the dataset comprises over 600,000 labeled images, containing 73,257training samples, 26,032 testing samples and 531,131 extra training data.

Original: The Food-101 [9] datasetconsists of 101 food categories with a total of 101k images, including 750 training and 250 testingsamples per category.
Revised: The Food-101 [8] datasetconsists of 101 food categories with a total of 101k images, including 750 training and 250 testingsamples per category.

Original: • Video domain : Something-Something V2 (SSv2) [36] is a large collection of video clipsshowing the people perform several normal actions in the daily life ( e
Revised: • Video domain : Something-Something V2 (SSv2) [28] is a large collection of video clipsshowing the people perform several normal actions in the daily life ( e

Original: g. , moving stuff and openingthe door).
Revised: g. , moving stuff and openingthe door).

Original: It consists of 168,913 training samples, 24,777 validation samples and 27,157 testingsamples, making a total of 220,847 videos with 174 labels.
Revised: It consists of 168,913 training samples, 24,777 validation samples and 27,157 testingsamples, making a total of 220,847 videos with 174 labels.

Original: HMDB51 [52] is composed of 6,849videos with 51 categories, making a split of 3.5k/1.5k train/val videos.
Revised: HMDB51 [43] is composed of 6,849videos with 51 categories, making a split of 3.5k/1.5k train/val videos.

Original: Implementation details.
Revised: Implementation details.

Original: In this work, we use PyTorch toolkit [64] to conduct all experiments on
Revised: In this work, we use PyTorch toolkit [51] to conduct all experiments on

Original: NVIDIA V100 GPUs.
Revised: NVIDIA V100 GPUs.

Original: Unless otherwise stated, we use 8 × 8 GPUs for video experiments and 1 × 

Revised: Unless otherwise stated, we use 8 × 8 GPUs for video experiments and 1 ×

Original: GPUs for image experiments.
Revised: GPUs for image experiments.

Original: Our default configurations follow the linear probing settings in [20, 40],which do not utilize many common regularization strategies, such as mixup [88], cutmix [85], colorjittering and so on.
Revised: Our default configurations follow the linear probing settings in [17, 32],which do not utilize many common regularization strategies, such as mixup [72], cutmix [69], colorjittering and so on.

Original: More details can be found in Appendix.
Revised: More details can be found in Appendix.

Original: 4.2 Main Properties and Analysis
Revised: 4.2 Main Properties and Analysis

Original: We compare the performance of different fine-tuning approaches in Table 1 with the backbones pretrained via the self-supervised paradigms.
Revised: We compare the performance of different fine-tuning approaches in Table 1 with the backbones pretrained via the self-supervised paradigms.

Original: The results show that AdaptFormer consistently surpasseslinear probing and Visual Prompt tuning (VPT) methods.
Revised: The results show that AdaptFormer consistently surpasseslinear probing and Visual Prompt tuning (VPT) methods.

Original: Specifically, AdaptFormer outperforms VPTon image benchmark CIFAR-100, SVHN, and Food-101, by 9.50%, 5.95%, and 10.04% respectively.
Revised: Specifically, AdaptFormer outperforms VPTon image benchmark CIFAR-100, SVHN, and Food-101, by 9.50%, 5.95%, and 10.04% respectively.

Original: On the more challenging video action recognition dataset Something-Something V2, the superioritybecomes even more significant, i.e. , about 34.96%.
Revised: On the more challenging video action recognition dataset Something-Something V2, the superioritybecomes even more significant, i.e. , about 34.96%.

Original: Note that even compared with the full fine-tuning
Revised: Note that even compared with the full-finetuning

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: strategy, our AdaptFormer still outperforms by about 10% Top-1 accuracy on SSv2 dataset.
Revised: strategy, our AdaptFormer still outperforms by about 10% Top-1 accuracy on SSv2 dataset.

Original: Tosummarize, our AdaptFormer is highly parameter-efficient, as well as yielding good performancewith parameter size at most 2% times than the full fine-tuning manner.
Revised: Tosummarize, our AdaptFormer is highly parameter-efficient, as well as yielding good performancewith parameter size at most 2% times than the full fine-tuning manner.

Original: 4.3 Scaling Tunable Parameters Up
Revised: 4.3 Scaling Tunable Parameters Up

Original: Even though there are only limited parameters introduced, one might also argue that more tunableparameters of AdaptFormer contribute to its higher accuracy compared with VPT [48].
Revised: Even though there are only limited parameters introduced, one might also argue that more tunableparameters of AdaptFormer contribute to its higher accuracy compared with VPT [39].

Original: We conductexperiments to make a comprehensive discussion on this aspect.
Revised: We conductexperiments to make a comprehensive discussion on this aspect.

Original: As described in Sec. 3.3, the number of tunable parameters can be adjusted by changing the number ofintroduced tokens for VPT, or the hidden feature dimension for AdaptFormer.
Revised: As described in Sec. 3.3, the number of tunable parameters can be adjusted by changing the number ofintroduced tokens for VPT, or the hidden feature dimension for AdaptFormer.

Original: As shown in Figure 4,we conduct experiments with a wide range of tunable parameters on both SSv2 and HMDB-51datasets.
Revised: As shown in Figure 4,we conduct experiments with a wide range of tunable parameters on both SSv2 and HMDB-51datasets.

Original: Since AdaptFormer and VPT share the same number of parameters of classification headon a specific dataset, we only report the tunable parameters on the x-axis, which comes from thevisual prompts (VPT) or weight/bias of the down-up fully-connected layers (AdaptFormer), withoutcalculating the parameters of classification head.
Revised: Since AdaptFormer and VPT share the same number of parameters of classification headon a specific dataset, we only report the tunable parameters on the x-axis, which comes from thevisual prompts (VPT) or weight/bias of the down-up fully-connected layers (AdaptFormer), withoutcalculating the parameters of classification head.

Original: For VPT, the number of introduced tokens is chosenfrom {1, 2, 4, 8, 16, 32, 48, 64}.
Revised: For VPT, the number of introduced tokens is chosenfrom {1, 2, 4, 8, 16, 32, 48, 64}.

Original: Similarly, the number of hidden dimensions in AdaptFormer is in {1,2, 4, 8, 16, 32}.
Revised: Similarly, the number of hidden dimensions in AdaptFormer is in {1,2, 4, 8, 16, 32}.

Original: AdaptFormer has a slight performance gain or maintains the accuracy stably whenthe parameters scale up.
Revised: AdaptFormer has a slight performance gain or maintains the accuracy stably whenthe parameters scale up.

Original: On the contrary, the performance of VPT decreases dramatically when theparameters exceed the task-specific value.
Revised: On the contrary, the performance of VPT decreases dramatically when theparameters exceed the task-specific value.

Original: Moreover, choosing the most suitable number of tokennumber becomes laborious since it might be task-specific ( i.e. varying from one dataset to the otherone).
Revised: Moreover, choosing the most suitable number of tokennumber becomes laborious since it might be task-specific ( i.e. varying from one dataset to the otherone).

Original: For example, the accuracy of VPT keeps going up when the number of tunable parametersincreases up to 300K on SSv2, whereas it begins to drop when the number of tunable parametersexceeds 50K on HMDB-51.
Revised: For example, the accuracy of VPT keeps going up when the number of tunable parametersincreases up to 300K on SSv2, whereas it begins to drop when the number of tunable parametersexceeds 50K on HMDB-51.

Original: We further study the optimization procedures of VPT by monitoring the test accuracy of the trainingstage.
Revised: We further study the optimization procedures of VPT by monitoring the test accuracy of the trainingstage.

Original: As shown in Figure 5, we gradually increase the number of tokens in VPT and plot the Top-1accuracy of each epoch.
Revised: As shown in Figure 5, we gradually increase the number of tokens in VPT and plot the Top-1accuracy of each epoch.

Original: The training stages are stable when the number of tokens is less than or equalto 4, e.g. , {1, 2, 4}.
Revised: The training stages are stable when the number of tokens is less than or equalto 4, e.g. , {1, 2, 4}.

Original: However, when the number becomes 8 or larger, e.g. , {8, 16, 32}, the trainingprocedure collapses at about the tenth epoch and achieves poor performance at the end of the trainingstage.
Revised: However, when the number becomes 8 or larger, e.g. , {8, 16, 32}, the trainingprocedure collapses at about the tenth epoch and achieves poor performance at the end of the trainingstage.

Original: On the contrary, the optimization procedures of AdaptFormer are stable when the number ofparameters varies across a large range, as shown in Table 2a.
Revised: On the contrary, the optimization procedures of AdaptFormer are stable when the number ofparameters varies across a large range, as shown in Table 2a.

Original: The top-1 accuracy fluctuates within1.5% when the number of parameters increases from 0.44M ( dim=16 ) to 4.87M ( dim=256 ).
Revised: The top-1 accuracy fluctuates within1.5% when the number of parameters increases from 0.44M ( dim=16 ) to 4.87M ( dim=256 ).

Original: 4.4 Ablation Studies
Revised: 4.4 Ablation Studies

Original: We ablate our AdaptFormer to study what properties make for a good AdaptFormer and observeseveral intriguing properties.
Revised: We ablate our AdaptFormer to study what properties make for a good AdaptFormer and observeseveral intriguing properties.

Original: The ablation studies conducted in this workare all performed on the
Revised: The ablation studies conducted in this work

Original: SSv

Original: Middle dimension.
Revised: Middle dimension.

Original: The middle dimension controls the number of introduced parameters by AdaptFormer.
Revised: The middle dimension controls the number of introduced parameters by AdaptFormer.

Original: Lower middle dimensions introduce fewer parameters with a possible performance cost.
Revised: Lower middle dimensions introduce fewer parameters with a possible performance cost.

Original: Weablate AdaptFormer on the middle feature dimension to study this effects.
Revised: Weablate AdaptFormer on the middle feature dimension to study this effects.

Original: As shown in Table 2a,the accuracy consistently improves when the middle dimension increases up to 64 and reaches thesaturation point when the middle dimension is about 64.
Revised: As shown in Table 2a,the accuracy consistently improves when the middle dimension increases up to 64 and reaches thesaturation point when the middle dimension is about 64.

Original: We note that our AdaptFormer can achieve adecent performance when the middle dimension reduces even to one, about 50.03% top-1 accuracy.
Revised: We note that our AdaptFormer can achieve adecent performance when the middle dimension reduces even to one, about 50.03% top-1 accuracy.

Original: [Table]
Revised: [Table]

Original: Scaling factor.

Original: The scaling factor s is introduced to balance the task-agnostic features (generatedby the original frozen branch) and the task-specific features (generated by the tunable bottleneckbranch).

Original: We evaluate AdaptFormer with multiple s values and the results are summarized in Table 2c.

Original: Different from the scaling factor in NLP field which prefer s larger than 1 ( e.g. , s = 4 in [39]), weempirically found that the s should be < 1 for vision tasks, otherwise the fine-tuning would becomeunstable.

Original: Besides, we found that AdaptFormer achieves optimal performance with s = 0 .

Original: A largeror smaller s would bring slight performance drop.

Original: Thus, we choose s = 0 .

Original: 10 as a default setting.

Original: AdaptFormer position.
Revised: AdaptFormer position.

Original: As shown in Table 2b, we further ablate on the specific position to introducethe AdaptMLP block.
Revised: As shown in Table 2b, we further ablate on the specific position to introducethe AdaptMLP block.

Original: We gradually increase the number of AdaptMLP layers with a step of three( start → end , both included).
Revised: We gradually increase the number of AdaptMLP layers with a step of three( start → end , both included).

Original: We observe that the performance of AdaptFormer has a positivecorrelation with the number of added layers.
Revised: We observe that the performance of AdaptFormer has a positivecorrelation with the number of added layers.

Original: In addition, AdaptFormer prefers the top part (the onefar away from the input image) of the network to the bottom part when introducing the same numberof layers, e.g. , AdaptFormer with 7 → 12 obtains over 14.5% higher accuracy than 1 → 6 , thoughboth equipped with six AdaptMLP layers.
Revised: In addition, AdaptFormer prefers the top part (the onefar away from the input image) of the network to the bottom part when introducing the same numberof layers, e.g. , AdaptFormer with 7 → 12 obtains over 14.5% higher accuracy than 1 → 6 , thoughboth equipped with six AdaptMLP layers.

Original: Insertion form.
Revised: Insertion form.

Original: We study the insertion formulation by comparing the parallel and sequentialinstances which are illustrated in Figure 6.
Revised: We study the insertion formulation by comparing the parallel and sequentialinstances which are illustrated in Figure 6.

Original: As shown in Table 2b, the parallel AdaptFormer is ableto outperform the sequential one by 0.85% top-1 accuracy.
Revised: As shown in Table 2b, the parallel AdaptFormer is ableto outperform the sequential one by 0.85% top-1 accuracy.

Original: The reason might be: (1) the paralleldesign maintains the original feature using an independent branch and aggregating updated contextby element-wise scaled sum; (2) the sequential design is equivalent to adding more layers, whichmight cause optimization difficulty.
Revised: The reason might be: (1) the paralleldesign maintains the original feature using an independent branch and aggregating updated contextby element-wise scaled sum; (2) the sequential design is equivalent to adding more layers, whichmight cause optimization difficulty.

Original: Therefore, we adopt the parallel design as our default setting dueto its superiority.
Revised: Therefore, we adopt the parallel design as our default setting dueto its superiority.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Number of frames.
Revised: Number of frames.

Original: The number of embedded patch tokens increases linearly with the number ofvideo frames for the plain ViT [29].
Revised: The number of embedded patch tokens increases linearly with the number ofvideo frames for the plain ViT [24].

Original: We conduct experiments with the different number of frames,i.e. , {2, 4, 8} and the results are shown in Figure 7.
Revised: We conduct experiments with the different number of frames,i.e. , {2, 4, 8} and the results are shown in Figure 7.

Original: We observe that increasing the number of framesis beneficial for all these three fine-tuning methods.
Revised: We observe that increasing the number of framesis beneficial for all these three fine-tuning methods.

Original: However, AdaptFormer consistently outperforms the linear manner ( e.g. , +30% top-1 accuracy on 8 input frames) and VPT method( e.g. , +14% top-1accuracy on 8 input frames).
Revised: However, AdaptFormer consistently ourperformsthe linear manner ( e.g. , +30% top-1 accuracy on 8 input frames) and VPT methhod ( e.g. , +14% top-1accuracy on 8 input frames).

Original: 4.5 Towards Visual Recognition Generalist Agent
Revised: 4.5 Towards Visual Recognition Generalist Agent

Original:  In the above experiments, we typically utilize a modality-specific pre-trained checkpoint forthe corresponding downstream tasks.
Revised: [Table] In the above experiments, we typically utilize amodality-specific pre-trained checkpoint for thecorresponding downstream tasks.

Original: For example, we use Kinetics-400 ( video domain ) pretrained model for downstream video action recognition on Something-Something V2 and HMDB51 benchmarks.
Revised: For example,we use Kinetics-400 ( video domain ) pre-trainedmodel for downstream video action recognition on Something-Something V2 and HMDB51 benchmarks.

Original: Besides, we use ImageNet-21K ( image domain ) pre-rained model for downstream image classification on CIFAR-100, SVHN and Food-101 benchmarks.
Revised: Besides, we use ImageNet21K ( image domain ) pre-rained model fordownstream image classification on CIFAR-100, SVHN and Food-101 benchmarks.

Original: Our AdaptFormerachieves superior performances in this same network with modality-specific weights scenario.[Table]
Revised: Our AdaptFormer achieves superior performances in thissame network with modality-specific weights scenario.

Original: Next, we take a further step to ask what wouldhappen if using the same network with themodality-agnostic weights for multiple tasks inthe multi-modalities downstream tasks?
Revised: Next, we take a further step to ask whatwould happen if using the same network with the modality-agnostic weights for multiple tasks in themulti-modalities downstream tasks?

Original: We use the model pre-trained on ImagNet-21kto do action recognition on SSv2.
Revised: We use the model pre-trained on ImagNet-21k to do action recognition on SSv2 and HMDB-51.

Original: As shownin Table 3, AdaptFormer is robust to domainshift caused by modality.
Revised: Asshown in Table 3, AdaptFormer is robust to domain shift caused by modality.

Original: The experimental results show that the linear probe approach obtainsa very poor accuracy ( i.e. , 6.56% top-1 accuracy) when fine-tuning on SSv2.
Revised: The experimentalresults show that the linear probe approach obtains a very poor accuracy ( i.e. , 6.56% top-1 accuracy)when fine-tuning on SSv2.

Original: Meanwhile, VPT [48] achieves a better performance than linear probe but it is not decent ( i.e. , 16.94% top-1accuracy).
Revised: Meanwhile, VPT [39] achieves a better performance than linear probebut it is not decent ( i.e. , 16.94% top-1 accuracy).

Original: Our AdaptFormer, compared to the above two methods, attains a promising 46.06% top-1accuracy, which is even higher than the full-tuning schedule (+4.56%). 
Revised: Our AdaptFormer, compared to the above twomethods, attains a promising 46.06% top-1 accuracy, which is even higher than the full-tuningschedule (+4.56%). 4.6 Visualization

Original: [Equation]
Revised: [Figure]

Original: Visualization
Revised: [Figure] [Figure] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original:  To evaluate the quality of the produced features, we conduct t-SNE [74] visualizations on AdaptFormer and other baseline methods.
Revised: [Figure] To evaluate the quality of the produced features, we conduct t-SNE [59] visualizations on AdaptFormer and other baseline methods.

Original: The features are extracted from the SSv2 validation set viathe ViT-Base backbone.
Revised: The features are extracted from the SSv2 validation set viathe ViT-Base backbone.

Original: Figure 8 shows that the linear fine-tuning and the VPT methods tend tooutput mixed features as shown in Figure 8(a)-(b).
Revised: Figure 8 shows that the linear fine-tuning and the VPT methods tend tooutput mixed features as shown in Figure 8(a)-(b).

Original: Compared with the above two methods, the fullfine-tuning strategy performs well in projecting features.
Revised: Compared with the above two methods, the fullfine-tuning strategy performs well in projecting features.

Original: However, it consumes huge computationalsources to tune the whole network parameters.
Revised: However, it consumes huge computationalsources to tune the whole network parameters.

Original: Figure 8(d) validates that our AdaptFormer facilitates
Revised: Figure 8(d) validates that our AdaptFormer facilitates

Original: ViT-Base in generating more separable representations with fewer learnable parameters.
Revised: ViT-Base in generating more separable representations with fewer learnable parameters.

Original:  Conclusion
Revised:  Conclusion

Original: We present a conceptually simple yet effective framework, AdaptFormer, for efficiently adapting apre-trained Vision Transformer (ViT) backbone to scalable vision recognition tasks.
Revised: We present a conceptually simple yet effective framework, AdaptFormer, for efficiently adapting apre-trained Vision Transformer (ViT) backbone to scalable vision recognition tasks.

Original: By introducing
Revised: By introducing

Original: AdaptMLP, our AdaptFormer is able to fine-tune the lightweight modules for producing featuresadapted to multiple downstream tasks.
Revised: AdaptMLP, our AdaptFormer is able to fine-tune the lightweight modules for producing featuresadapted to multiple downstream tasks.

Original: The extensive experiments on five datasets, covering boththe image and the video domains, validate that our proposed methods are able to increase the
Revised: The extensive experiments on five datasets, covering boththe image and the video domains, validate that our proposed methods are able to increase the

Original: ViT’s transferability with little computational cost.
Revised: ViT’s transferability with little computational cost.

Original: We hope our work will inspire future researchin exploring more efficient fine-tuning methods for large vision models.
Revised: We hope our work will inspire future researchin exploring more efficient fine-tuning methods for large vision models.

Original: One limitation is that
Revised: One limitation is that

Original: AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can workwell in tasks beyond recognition, e.g. , object detection and semantic segmentation.
Revised: AdaptFormer is only employed in recognition tasks in this work, it’s unclear whether it can workwell in tasks beyond recognition, e.g. , object detection and semantic segmentation.

Original: We leave it forthe future exploration.
Revised: We leave it forthe future exploration.

Original: Since our method is specially designed for efficient fine-tuning, we do notforesee obvious undesirable ethical/social impacts at this moment.
Revised: Since our method is specially designed for efficient fine-tuning, we do notforesee obvious undesirable ethical/social impacts at this moment.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Shown in Conclusion Section.
Revised: [Yes] Shown in the supplementary

Original:  (c) Did you discuss any potential negative societal impacts of your work?
Revised: materials. (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Shown inConclusion Section.
Revised: [Yes] Shown inthe supplementary materials.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] As a URL shown in the abstract.
Revised: [Yes] As a URLshown in the abstract.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Shown in supplementary materials.
Revised: [Yes] Shown in supplementary materials.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experi- ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please see Section 4.14.
Revised: [Yes] Please see Section 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Shown in supplementary materials.
Revised: [Yes] Shown in supplementary materials.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained frompeople whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from

Original: [Table]
Revised: people whose data

Original: using/curating?
Revised: you’reusing/ curating

Original: [Yes] We used publicly available datasets whose licenses allow research [Table] usage.
Revised: Yes ] We used publicly available datasets whose licenses allow researchusage.

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] To the best of our knowledge, the data we usedcontains no personally identifiable information or offensive content.[Table]
Revised: [No] To the best of our knowledge, the data we used contains no personally identifiable information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table]
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]

Original: [Table]

Original: A Appendix

Original: In this supplementary material, we will include the details about the pre-training and fine-tuningprocesses, the extensive experiments of AdaptFormer on hierarchical vision transformers ( e.g. ,

Original: AdaptFormer-Swin), and the pseudo-code of AdaptMLP in a PyTorch-like style.

Original: A.1 Experimental SettingsA.1.1 Pre-training Approaches

Original: Image.

Original: We use MAE [40] as our self-supervised pre-training method in the image domain, a simpleyet effective method that first masks nearly 75% patches of the input image and then reconstructs themissing pixels.

Original: Specifically, we directly adopt the checkpoint 4 of ViT-B/16 for convenience, which ispre-trained on ImageNet-1K [25] for 800 epochs.

Original: Video.

Original: We use VideoMAE [72] as our self-supervised pre-training method in the video domain,which is an direct extension of MAE to the video domain.

Original: VideoMAE utilizes the plain ViT [29]architecture of joint space-time attention mechanism [2, 60] and an extremely high proportionof masking ratio ( i.e. , 90% to 95%) for pre-training.

Original: We also directly use the publicly availablecheckpoint 5 , which is pre-trained on Kinetics-400 [15].

Original: A.1.2 Implementation Details of Fine-tuning

Original: [Table]

Original: The implementation details are summarized in Table 4.

Original: The video experiments are conducted on64 Tesla V100 GPUs, while the image experiments are performed on 8 Tesla V100 GPUs.

Original: Forthe optimizer, different from [83] that adopts LARS, we leverage SGD [71] for stable training onsmall-scale dataset ( e.g. , CIFAR10).

Original: The actual learning rate is calculated by: lr = base_lr × batchsize/ 256 following the linear lr scaling rule [35].

Original: More detailed training configurations are presented in

Original: Table 4, including the batchsize, learning rate schedule and etc.

Original: The experimental settings of image and video mainly follow the ones utilized in MAE [40] and

Original: VideoMAE [72], respectively.

Original: We insert an extra BatchNorm layer [46] without affine transformation ( i.e. affine=

Original: False ) before the final fully connected layer, following the common practice tonormalize the pre-trained features [27, 40].

Original: In addition, there is no flip augmentation during thefine-tuning stage for video data.

Original: A.2 More Supplementary ResultsA.2.1 AdaptFormer with Supervised Pre-training

Original: In addition to the self-supervised pre-training presented in the main paper

Original: [Table]

Original: we also evaluate AdaptFormer with the supervised pre-trained model.

Original: The results in Table 5 show that AdaptFormerstill outperforms linear probe and VPT obviously.

Original: In addition, AdaptFormer surpasses full-tuningon four benchmarks (CIFAR100, SVHN, SSv2, HMDB51) with only 1.46% parameters.

Original: On theremaining benchmark (Food-101), AdaptFormer achieves an almost comparable performance tofull-tuning (90.89% v.s. 90.96%).

Original: [Table]

Original: A.2.2 AdaptFormer

Original: on Swin

Original: Transformer

Original: Settings.

Original: We further demonstrate the effectiveness of AdaptFormer on hierarchical vision transformers, e.g. , Swin [59, 60].

Original: We name AdaptFormer applied to Swin as AdaptFormer-Swin , to distinguishplain AdaptFormer (without any suffix) which is applied to the vanilla ViT [29].

Original: It is noted that wecan adopt AdaptMLP to Swin easily without any special modification as Swin and ViT share thesame MLP architecture.

Original: However, VPT [48] needs additional handcraft designs to be suitable forthe shifted local windows in the prevalent hierarchical vision transformers, which hinders its generalapplications.

Original: We utilize Swin-B [59] and the video counterpart [60] for image and video.

Original: Similarly, we alsodirectly use the officially provided checkpoints 6 , which are pre-trained on ImageNet-21K [25] and

Original: Kinetics-600 [14].

Original: [Table]

Original: [Table]

Original: [Table]

Original: Results.

Original: Since VPT is not applicable in Swin, we do not report its performance.

Original: Table 6 shows

Original: AdaptFormer-Swin performs well compared with other tuning strategies.

Original: For image benchmarks, ourmethod can outperform full-tuning approach with only 1.43% parameters.

Original: Moreover, AdaptFormerSwin surpasses linear probing by a significant margin, especially on the challenging dataset, SSv2.

Original: The results validate that AdaptFormer is able to generally boost the transferability of various vision

Original: Transformer variants.

Original: A.3 Multi-Label Classification

Original: We further conduct experiments on dataset with larger scale and diversity.

Original: Specifically, we evaluate

Original: AdaptFormer on NUS-WIDE [23] for multi-label classification.

Original: NUS-WIDE contains 269,648images collected from Flicker, which are annotated with 81 visual concepts.

Original: Since some images arenot available on Flicker, we only use 220,000 images following [7, 30].

Original: We utilize mean averageprecision (mAP) as performance metric.

Original: Settings: Our training settings mainly follow ASL [7].

Original: Specifically, We trained all models for 40epochs using Adam optimize and 1-cycle learning rate policy [69].

Original: The maximal learning rate is0.001.

Original: [Table]

Original: A.4 Possible Architectures

Original: We explore other possible architectures utilized in AdaptFormer.

Original: Specifically, we further replace the

Original: MLP architectures within the AdaptMLP module by the convolution layer, depthwise convolutionlayer, and LayerNorm layer.

Original: For fair comparisons, we carefully design the above modules to meet thecomparable number of parameters ( ˜ 1.3M).

Original: The experimental results of different adapter modules areshown in Table 8, which validates that the simple MLP modules are simple yet effective comparedwith the other architectures.

Original: For example, our AdaptMLP module surpasses the AdaptConv moduleby 0.55% Top1 accuracy on SSv2 dataset.

Original: [Table]

Original: A.5 Evaluation on ImageNet-1k datasets

Original: We point out that in order to evaluate the adaptation performance across datasets, it’s an unreasonablesetting to fine-tune the ImageNet-1k dataset with the ImageNet-21k pre-trained weights.

Original: This isbecause ImageNet-1K is a subset of the ImageNet-21K as introduced in [68].

Original: In contrast, in all theprevious experiments, there is no overlap between the fine-tuning and pre-trained datasets.

Original: However,we document the experiments of fine-tuning with the ImageNet-1k dataset for the completeness.

Original: We adopt exactly identical training configurations to conduct experiments in this subsection.

Original: Weexperiment with middle dimension = {1, 4, 16, 64} on ImageNet-1K, and the results areshown Table 9.

Original: Results.

Original: Comparing the results of AdaptFormer with different middle dimension ({1, 4, 16, 64})on ImageNet-1K, we find that AdaptFormer with the smallest number of parameters ( AdaptFormer1 ) achieves the best top-1 accuracy (82.33%).

Original: Furthermore, when the ‘middle dimension‘ increases

Original: [Table]

Original: from 1 to 4 or 16, AdaptFormer has a slight performance drop (AdaptFormer-4 (-0.07%) and

Original: AdaptFormer-16 (-0.09%)).

Original: Further increasing the middle dimension to 64 will cause a relativelyclear performance drop (-0.47%).

Original: Discussions.

Original: Although our AdaptFormer-64 does not have a clear advantage compared with VPT [48],our AdaptFormer-1 outperforms VPT by +0.65%

Original: top-1 accuracy with only 0.02M additional parameters.

Original: Besides, the trend of classification accuracy changing with middle dimension on ImageNet1k is different from other datasets in our paper, e.g. , AdaptFormer with middle dimension=64achieves better top-1 accuracy than with middle dimension=1 on CIFAR-100.

Original: We empirically findintroducing a small number of parameters (AdaptFormer-1) is sufficient for ImageNet-1K fine-tuning,while more introduced parameters will make a larger change to the original model and make it harderfor optimization since ImageNet-1K is a subset of ImageNet-21K.

Original: However, for other datasets (e.g.,

Original: CIFAR-100) with no overlap between the fine-tuning datasets and the pre-trained datasets

Original: , moreintroduced parameters are needed for learning better domain knowledge.

Original: A.6 Extended experiments on middle dimension

Original: We conduct the extended ablation studies on the middle dimension design in this sub-section.

Original: Weaim to seek for a trade-off between model capacity (i.e., potential) and adaptation efficiency.

Original: In fact,the middle dimension has a main influence on the parameter size of adapter.

Original: The higher dimensionbrings more parameters while the efficiency and storage are limited.

Original: As shown in Table 10, weevaluate several numbers of middle dimension and found that using 64 is optimal to achieve accuracy,light-weight storage, and efficiency.

Original: [Table]

Original: A.7 Analysis on the fine-tuning time and inference latency

Original: To analysis the computational efficiency, we compare the fine-tuning time and inference time on asingle NVIDIA A100-40G GPU.

Original: We utilize SSv2 video classification for this part.

Original: For fine-tuing,we experiment with batchsize of 32.

Original: For inference, we test the latency with multiple batch sizesto get a comprehensive comparison under various inference scenarios.

Original: All the time is measured inmilliseconds averaged over 100 trials.

Original: The results are summarized in Table 11 and Table 12.

Original: As shown in Table 11, AdaptFormer only costs less than a half of the fine-tuning time compared with thefull-tuning.

Original: Moreover, AdaptFormer significantly outperforms linear probing in terms of accuracywith a slight longer fine-tuning time.

Original: For inference, AdaptFormer introduce a negligible FLOPs andlatency compared with the Linear/Full-tuning.

Original: [Table]

Original: [Table]

Original: A.8 Discussion about ImageNet and Kinetics Pre-training

Original: The type of spatiotemporal attention ( divided vs. joint ) determines whether the performance of themodel pre-trained on ImageNet can outperform the model pre-trained on Kinetics.

Original: A similar phenomenon has been discussed in recent work, Uniformer [54], independently.

Original: We borrowthe experimental results from Table 4(c) in Uniformer paper [54].

Original: Specifically, the divided spatiotemporal attention prefers ImageNet to Kinetics-400 for the pre-training dataset.

Original: The performance of thedivided attention model pre-trained on ImageNet outperforms the model pre-trained on Kinetics-400.

Original: On the contrary, the joint spatiotemporal attention prefers Kinetics-400 to ImgaeNet.

Original: The jointattention model attains higher top-1 accuracy with Kinetics-400 pretraining compared to ImageNet(53.8 vs. 52.0).

Original: We adopt the joint spatiotemporal attention for all video-related experiments in this work (introducedin Appendix A.1.1).

Original: Therefore, our experimental phenomenon is consistent with the joint attentionin [54], i.e., Kinetics pretraining is preferable.

Original: A.9 Implementation

Original: The core part of AdaptFormer is replacing the original MLP with AdaptMLP, which consists of thefrozen original MLP and newly introduced Down → ReLU → Up layers, which are tunable at thefine-tuning stage.

Original: Algorithms 1 provides the implementation of AdaptMLP written in PyTorch [64].

Original: For more implementation details, please refer to the provided source code.

Original: [Table]
Revised: [N/A]