Original: T INY A DAPTERS FOR V ISION T RANSFORMERS
Revised: T INY A DAPTERS FOR V ISION T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Vision Transformers (ViTs) have become one of the dominant architectures in computer vision and pretrained ViT models are commonly adapted to new tasks via fine-tuning of its parameters.
Revised: Vision Transformers (ViTs) have become one of the dominant architectures in computer vision and pretrained ViT models are commonly adapted to new tasks via fine-tuning of its parameters.

Original: Recent works in NLP proposed a variety of parameter-efficient transfer learning methods such as adapters to avoid the prohibitive storage cost of fine-tuning.
Revised: Recent works in NLP proposed a variety of parameter-efficient transfer learning methods such as adapters to avoid the prohibitive storage cost of fine-tuning.

Original: In this work, we start from the observation that adapters perform poorly when the dimension of adapters is small and we propose a training algorithm that addresses this issue.
Revised: In this work, we start from the observation that adapters perform poorly when the dimension of adapters is small and we propose a training algorithm that addresses this issue.

Original: We start from large adapters which can be trained easily and iteratively reduce the size of every adapter.
Revised: We start from large adapters which can be trained easily and iteratively reduce the size of every adapter.

Original: We introduce a scoring function that can compare neuron importance across layers and consequently allow automatic estimation of the hidden dimension of every adapter.
Revised: We introduce a scoring function that can compare neuron importance across layers and consequently allow automatic estimation of the hidden dimension of every adapter.

Original: Our method outperforms existing approaches in terms of the trade-off between accuracy and trained parameters across domain adaptation benchmarks.
Revised: Our method outperforms existing PET methods in terms of the trade-off between accuracy and trained parameters across two benchmarks.

Original: We will release our code publicly upon acceptance.
Revised: We will release our code publicly upon acceptance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers for vision problems have recently attracted growing attention due to their remarkable performance Liu et al.
Revised: Transformers for vision problems have recently attracted growing attention due to their remarkable performance Liu et al.

Original: (2021b)
Revised: (2021b)

Original: ; Touvron et al.
Revised: ; Touvron et al.

Original: ; Dosovitskiy et al.
Revised: ; Dosovitskiy et al.

Original: In particular, the image representation learned with large-scale pretrained Vision Transformers (ViTs) has shown a promising ability for learning new tasks Dosovitskiy et al.
Revised: In particular, the image representation learned with large-scale pretrained Vision Transformers (ViTs) has shown a promising ability for learning new tasks Dosovitskiy et al.

Original: The commonly adopted strategy to learn new tasks consists in fully or partially fine-tuning a pretrained network.
Revised: The commonly adopted strategy to learn new tasks consists in fully or partially fine-tuning a pretrained network.

Original: However, in the case of multiple tasks, this approach requires training numerous separate instances of the models and induces significant storage costs.
Revised: However, in the case of multiple tasks, this approach requires training numerous separate instances of the models and induces significant storage costs.

Original: Parameter-Efficient Training (PET) approaches have recently been developed in order to adapt largepretrained models to new tasks with less training power Hu et al.
Revised: Parameter-Efficient Training (PET) approaches have recently been developed in order to adapt largepretrained models to new tasks with less training power Hu et al.

Original: ; He et al.
Revised: ; He et al.

Original: Among these, adapters Houlsby et al.
Revised: Among these, adapters Houlsby et al.

Original: and their variants Mahabadi et al.
Revised: and their variants Mahabadi et al.

Original: ; He et al.
Revised: ; He et al.

Original: Karimi Mahabadi et al.
Revised: Karimi Mahabadi et al.

Original: (2021) are frequently utilized and adapted to many architectures for Natural
Revised: (2021) are frequently utilized and adapted to many architectures for Natural

Original: Language Processing (NLP) tasks.
Revised: Language Processing (NLP) tasks.

Original: In a nutshell, adapters are tiny modules injected into transformer blocks, which enable efficient adaption of the data representation to the downstream task.
Revised: In a nutshell, adapters are tiny modules injected into transformer blocks, which enable efficient adaption of the data representation to the downstream task.

Original: Adapters offer similar performance to full fine-tuning ( i.e. updating all parameters) while requiring a very limited number of  parametersto be trained Houlsby et al.
Revised: Adapters offer similar performance to full fine-tuning ( i.e. updating all parameters) while requiring a very limited number of trainable parameters Houlsby et al.

Original: ; Pfeiffer et al. (2021).
Revised: ; Pfeiffer et al. (2021).

Original: When it comes to vision tasks, PET approaches are mostly explored for convolutional neural networks Berriel et al.
Revised: When it comes to vision tasks, PET approaches are mostly explored for convolutional neural networks Berriel et al.

Original: ; Rebuffi et al.
Revised: ; Rebuffi et al.

Original: (2018; 2017); Mallya et al.
Revised: (2018; 2017); Mallya et al.

Original: On the contrary, several PET approaches have been proposed in the context of NLP tasks.
Revised: On the contrary, several PET approaches have been proposed in the context of NLP tasks.

Original: In this case, adapters are commonly implemented with Multi-Layer-Perceptron (MLP) layers equipped with residual connections Houlsby et al.
Revised: In this case, adapters are commonly implemented with Multi-Layer-Perceptron (MLP) layers equipped with residual connections Houlsby et al.

Original: Multi-layer adapters offer sufficient representation power to adapt to the new tasks but require a significant number of trainable parameters subject to their bottleneck dimension, which provides a simple trade-off between performance and parameter efficiency Houlsby et al.
Revised: Multi-layer adapters offer sufficient representation power to adapt to the new tasks but require a significant number of trainable parameters subject to their bottleneck dimension, which provides a simple trade-off between performance and parameter efficiency Houlsby et al.

Original: Nevertheless, they suffer from two main weaknesses.
Revised: Nevertheless, they suffer from two main weaknesses.

Original: First, the performance drops when the size of the multi-layer adapters is too small (as also confirmed by our experiments).
Revised: First, the performance drops when the size of the multi-layer adapters is too small (as also confirmed by our experiments).

Original: Therefore, these adapters cannot be employed in scenarios where the available storage is limited.
Revised: Therefore, these adapters cannot be employed in scenarios where the available storage is limited.

Original: Second, the hyper-parametrization ofmultilayer adapters is complex, since the hidden layer dimensions must be specified for every adapters in every layer.
Revised: Second, the hyper-parametrization of adapters is complex, since the hidden layer dimensions must be specified for every adapters in every layer.

Original: Furthermore, the choice of the hyper-parameters depends on the downstream task.
Revised: As well, the hidden dimension depends on the downstream task.

Original: [Figure] In this work, we propose a training scheme named TINA and illustrated in Fig. 1, that addresses these two limitations of adapters.
Revised: In this work, we propose a training scheme named TINA (Fig. 1), that addresses these two limitations of adapters.

Original: Our strategy allows a more efficient parameter allocation since additional parameters are predominantly allocated to layers really requiring adaptation to the new task.
Revised: Our strategy allows a more efficient parameter allocation since additional parameters are predominantly allocated to layers really requiring adaptation to the new task.

Original: More specifically, we start from adapters with hidden spaces of high dimension and we iteratively reduce the dimension by identifying the neurons that can be dropped in every adapter.
Revised: More specifically, we start from adapters with hidden spaces of high dimension and we iteratively reduce the dimension by identifying the neurons that can be dropped in every adapter.

Original: To identify in which layer  adaptation must be primarily performed, we introduce a novel scoring criterion that allows comparison of neuron importance across different adapters.
Revised: To identify in which layer [Figure] adaptation must be primarily performed, we introduce a novel scoring criterion that allows comparison of neuron importance across different adapters.

Original: In short, the contributions of this work are:
Revised: In short, the contributions of this work are:

Original: • We propose a novel iterative training scheme for learning tiny adapters in the case of ViTs.
Revised: • We propose a novel iterative training scheme for learning tiny adapters in the case of ViTs.

Original: • We propose a novel scoring function to compare neuron importance across adapters.
Revised: • We propose a novel scoring function to compare neuron importance across adapters, which can be included in various magnitude pruning methods.

Original: This allows us to automatically estimate the hidden dimension of adapters for ViTs, leading to more efficient parameter allocation.
Revised: This allows us to automatically estimate the hidden dimension of adapters for ViTs, leading to more efficient parameter allocation.

Original: Finally, we compare the proposed approach with multiple PET methods designed for NLP using a total of 10 datasets.
Revised: Finally, we compare the proposed approach with multiple PET methods designed for NLP using a total of 10 datasets.

Original: From these experiments, we draw several conclusions: (i) we observe a discrepancy between the rankings obtained in vision and NLP benchmarks; (ii) we demonstrate that our approach obtains the best performance in terms of accuracy among methods with similar numbers of parameters; (iii) our ablation study validates the positive impact of our adaptive strategy to automatically estimate thedimension of the hidden state of each adapter.
Revised: From these experiments, we draw several conclusions: (i) we observe a discrepancy between the rankings obtained in vision and NLP benchmarks; (ii) we demonstrate that our approach obtains the best performance in terms of accuracy among methods with similar numbers of parameters; (iii) our ablation study validates the positive impact of our adaptive strategy to automatically estimate the hidden dimension of adapters.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Vision Transformers.
Revised: Vision Transformers.

Original: Initially proposed for NLP tasks, Transformers Vaswani et al.
Revised: Initially proposed for NLP tasks, Transformers Vaswani et al.

Original: (2017) have been recently proposed for vision tasks.
Revised: (2017) have been recently proposed for vision tasks.

Original: The image is split into a sequence of patches that are projected into token embeddings, and fed to transformer encoders that employ self-attention to learn rich image representations Dosovitskiy et al.
Revised: The image is split into a sequence of patches that are projected into token embeddings, and fed to transformer encoders that employ self-attention to learn rich image representations Dosovitskiy et al.

Original: With sufficient training data, ViTs are able to outperform ConvNets on various image classification benchmarks Guo et al.
Revised: With sufficient training data, ViTs are able to outperform ConvNets on various image classification benchmarks Guo et al.

Original: Nevertheless, their high number of parameters induces high storage cost, which hinders complete finetuning for each new task and naturally motivates this study.
Revised: Nevertheless, their high number of parameters induces high storage cost, which hinders complete finetuning for each new task and naturally motivates this study.

Original: Several variants of ViTs have been proposed in the literature, out of which Swin Liu et al.
Revised: Several variants of ViTs have been proposed in the literature, out of which Swin Liu et al.

Original: (2021b) is probably the most popular thanks to its high performance on various vision tasks.
Revised: (2021b) is probably the most popular thanks to its high performance on various vision tasks.

Original: Swin employs local attention applied to different regions of the input image from one layer to another, unlike the initial ViT architecture Dosovitskiy et al.
Revised: Swin employs local attention applied to different regions of the input image from one layer to another, unlike the initial ViT architecture Dosovitskiy et al.

Original: (2020) which uses global attention.
Revised: (2020) which uses global attention.

Original: Despite we evaluate our approach using Swin, our tiny adapters can be added to any ViT architecture.
Revised: Despite we evaluate our approach using Swin, our tiny adapters can be added to any ViT architecture.

Original: Network Pruning.
Revised: Network Pruning.

Original: When referred to deep neural networks, pruning consists of reducing the number of parameters of a pre-trained model Han et al.
Revised: When referred to deep neural networks, pruning consists of reducing the number of parameters of a pre-trained model Han et al.

Original: (2015a)
Revised: (2015a)

Original: ; Cun et al.
Revised: ; Cun et al.

Original: It can be roughly categorized into two groups: (i) unstructured pruning, which removes the least significant weights (according to certain criteria like weight magnitude Han et al.
Revised: It can be roughly categorized into two groups: (i) unstructured pruning, which removes the least significant weights (according to certain criteria like weight magnitude Han et al.

Original: (2015b) or gradient magnitude Molchanov et al.
Revised: (2015b) or gradient magnitude Molchanov et al.

Original: without a specific structure to be followed; (ii) structured pruning, which focuses in removing model sub-structures, like channels Tartaglione et al.
Revised: without a specific structure to be followed; (ii) structured pruning, which focuses in removing model sub-structures, like channels Tartaglione et al.

Original: ; He et al.
Revised: ; He et al.

Original: (2017) or attention heads Michel et al.
Revised: (2017) or attention heads Michel et al.

Original: A key focus of these works resides in identifying redundant connections whose removal brings the least perturbation to the overall performance.
Revised: A key focus of these works resides in identifying redundant connections whose removal brings the least perturbation to the overall performance.

Original: Despite within TINA a selection phase for the most relevant units inside adapters occurs, it differs from pruning approaches in two key aspects.
Revised: This work does not introduce a new pruning method, although TINA does use iterative pruning. In this regard TINA is unique in two key aspects.

Original: First, pruning reduces the cardinality of the parameters of a network trained on a specific task, while TINA reduces the number of parameters injected via adapters to adapt the model to a new task without changing the original model parameters.
Revised: First, pruning methods traditionally reduce the cardinality of the parameters of a network trained on a specific task, while TINA reduces the number of parameters injected via adapters to adapt the model to a new task without changing the original model parameters.

Original: Second, we leverage on the adapter’s structure to design an on-purpose importance metric that accounts simultaneously parameters from the two layers in adapters, while pruning typically considers parameters of every layer independently.
Revised: Second, we leverage on the adapter’s structure to design an on-purpose importance metric that accounts simultaneously parameters from the two layers in adapters, while pruning typically considers parameters of every layer independently.

Original: Efficient Transformers Finetuning.
Revised: Efficient Transformers Finetuning.

Original: The lack of the typical CNN inductive biases in ViTs makes them difficult to be finetuned on new tasks without using (very) large datasets.
Revised: The lack of the typical CNN inductive biases in ViTs makes them difficult to be finetuned on new tasks without using (very) large datasets.

Original: For this reason, Liu et al.
Revised: For this reason, Liu et al.

Original: (2021a) addresses this by adding a regularization term during finetuning to extract additional information in self-supervised manner by the model.
Revised: (2021a) addresses this by adding a regularization term during finetuning to extract additional information in self-supervised manner by the model.

Original: However, it requires updating all the parameters and storing one copy of the fine-tuned model per task.
Revised: However, it requires updating all the parameters and storing one copy of the fine-tuned model per task.

Original: This causes substantial storage and deployment costs and hinders the applicability of large-scale models to real-world applications.
Revised: This causes substantial storage and deployment costs and hinders the applicability of large-scale models to real-world applications.

Original: In order to address this issue, three types of approaches have been proposed for NLP tasks: (i) only updating newly added parameters (added either to the input or model)
Revised: In order to address this issue, three types of approaches have been proposed for NLP tasks: (i) only updating newly added parameters (added either to the input or model)

Original: Pfeiffer et al.
Revised: Pfeiffer et al.

Original:  Houlsby et al.
Revised: ; Houlsby et al.

Revised: (2019)Chen et al.

Original: ; (ii) sparsely updating a small number of parameters of the model Hu et al.
Revised: (ii) sparsely updating a small number of parameters of the model Hu et al.

Original: (2021); Ben Zaken et al.
Revised: Ben Zaken et al.

Original: (2022); (iii) performing low-rank factorization for the weights to be updated Karimi Mahabadi et al.
Revised: (2022); (iii) performing low-rank factorization for the weights to be updated Karimi Mahabadi et al.

Revised: Inspired by prompting techniques in NLP, Jia et al.

Revised: (2022) introduced VPT , a method that learns prompts for efficient finetuning of ViTs.

Revised: Although prompting achieves good performance, it lacks flexibility since prompt sizes must be equal to the dimensions of the corresponding layers in the pretrained ViT.

Revised: Thus, prompts are not suitable for downstream tasks that strongly differ form the pre-training task Chen et al.

Original: Among these approaches, adapters, which belong to the first category, showed promising results in comparison with full model fine-tuning.
Revised: Among these approaches, adapters, belonging to the first category, showed promising results in comparison with full model fine-tuning.

Original: Our work falls into the first category of approaches.
Revised: Our work falls into the first category of approaches.

Original: More precisely, TINA is based on the adapters proposed in Houlsby et al.
Revised: More precisely, TINA is based on the adapters proposed in Houlsby et al.

Original: , but we provide a specific training algorithm that enables small size adapters’ training.
Revised: , but we provide a specific training algorithm that enables small size adapters’ training.

Original: Furthermore, while the other state-of-the-art approaches adopt fixed size adapters in every layer, TINA dynamically estimates the adapter size for each layer and even removes them, if necessary.
Revised: Furthermore, while the other state-of-the-art approaches adopt fixed size adapters in every layer Chen et al. In our preliminary results, we show different layers require different adapter sizes to adapt efficiently to the new tasks (see Appendix A.2.1). TINA dynamically estimates the adapter size for each layer and even removes them, if necessary.

Original: In this manner, we reduce the number of trainable parameters and improve the performance.
Revised: In this manner, we reduce the number of trainable parameters and improve the performance.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: In this section, we start with the description of adapters Houlsby et al.
Revised: In this section, we start with the description of adapters Houlsby et al.

Original: (2019) and their practical benefits.
Revised: (2019) and their practical benefits.

Original: Then, we introduce our proposed method to estimate the hidden dimension for each adapter layer that can effectively train ViTs reaching higher performance with fewer parameters.
Revised: Then, we introduce our proposed method to estimate the hidden dimension for each adapter layer that can effectively train ViTs reaching higher performance with fewer parameters.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: We assume that we have at our disposal a pretrained ViT network and that our goal is to adapt this network to learn a classifier for a new task.
Revised: We assume that we have at our disposal a pretrained ViT network and that our goal is to adapt this network to learn a classifier for a new task.

Original: We have access to an annotated dataset for the targeted downstream task.
Revised: We have access to an annotated dataset for the targeted downstream task.

Original: To this aim, we employ small modules, referred to as adapters that are added to the layers of the pre-trained model in order to adapt it to the downstream task while keeping the weights of the original model frozen.
Revised: To this aim, we employ small modules, referred to as adapters that are added to the layers of the pre-trained model in order to adapt it to the downstream task while keeping the weights of the original model frozen.

Original: Therefore, adapting the network consists in jointly training the linear classifier (commonly referred to as head ) and the parameters of the adapters.
Revised: Therefore, adapting the network consists in jointly training the linear classifier (commonly referred to as head ) and the parameters of the adapters.

Original: For the sake of simplicity, we no longer mention the linear classifier parameters when describing our training procedure, but it has to be kept in mind that they are jointly learned with the adapters.
Revised: For the sake of simplicity, we no longer mention the linear classifier parameters when describing our training procedure, but it has to be kept in mind that they are jointly learned with the adapters.

Original: Formally, ViT architectures such as the original ViT Dosovitskiy et al.
Revised: Formally, ViT architectures such as the original ViT Dosovitskiy et al.

Original: (2020) or Swin Liu et al.
Revised: (2020) or Swin Liu et al.

Original: (2021b) employ layers composed of two main sub-layers: a multi-head self-attention layer and a feedforward layer, each of them preceded by layer normalization.
Revised: (2021b) employ layers composed of two main sub-layers: a multi-head self-attention layer and a feedforward layer, each of them preceded by layer normalization.

Original: A residual connection is applied across each of the sub-layers.
Revised: A residual connection is applied across each of the sub-layers.

Original: We insert two adapters after each of these sub-layers.
Revised: We insert two adapters after each of these sub-layers.

Original: The adapter is always applied directly to the output of the target sub-layer, as show in Fig.
Revised: The adapter is always applied directly to the output of the target sub-layer, as show in Fig.

Original: 2a.
Revised: 2a.

Original: The internal structure of adapters is visually summarized in Fig. 2b.
Revised: The internal structure of adapters is visually summarized in Fig. 2b.

Original: Considering the i th adapter added to our pretrained
Revised: Considering the i th adapter added to our pretrained

Original: ViT, let h i ∈
Revised: ViT, let h i ∈

Original: R m i denote its input of size m i
Revised: R m i denote its input of size m i

Original: Following Houlsby et al.
Revised: Following Houlsby et al.

Original: (2019), adapters employ a first fully connected layers that down-project h i into z i ∈ R n i with some non-linear activation ϕ ( · ) .
Revised: (2019), adapters employ a first fully connected layers that down-project h i into z i ∈ R n i with some non-linear activation ϕ ( · ) .

Original: This layer is parametrized by a linear projection matrix W down i ∈
Revised: This layer is parametrized by a linear projection matrix W downi ∈

Original: R m i × n i .
Revised: R m i × n i .

Original: Then, a second fully connected layer with parameters W upi ∈ R n i × m i up-samples z i producing as output r i ∈
Revised: Then, a second fully connected layer with parameters W upi ∈ R n i × m i up-samples z i producing as output r i ∈

Original: R m i .
Revised: R m i .

Original: Finally, a residual skip-connection is employed inside the adapter module such that, if r i is always close to 0, the adapter module degenerates to an identity function.
Revised: Finally, a residual skip-connection is employed inside the adapter module such that, if r i is always close to 0, the adapter module degenerates to an identity function.

Original: To summarize, given the input
Revised: To summarize, given the input

Original: vector h
Revised: vector h

Original: i , the output vector h ′
Revised: i , the output vector h ′

Original: i is calculated as:
Revised: i is calculated as:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: The total number of parameters in adapters is equal to 2 · n i · m i .
Revised: The total number of parameters in adapters is equal to 2 · n i · m i .

Original: Since m i is fixed, we generally specify n i such that n i ≪
Revised: Since m i is fixed, we generally specify n i such that n i ≪

Original: m i to obtain a low number of trainable parameters.
Revised: m i to obtain a low number of trainable parameters.

Original: We define the compression rate σ i of an adapter as follows σ i
Revised: We define the compression rate σ i of an adapter as follows σ i

Original: = m i n i .
Revised: = m i n i .

Original: In previous works Houlsby et al.
Revised: In previous works Houlsby et al.

Original: ; R ¨ uckl ´ e et al. (2021), adapters have a unique hidden dimension n i for every adapter Pfeiffer et al.
Revised: (2019); R¨uckl´e et al. (2021), adapters have a unique hidden dimension n i for every adapter Pfeiffer et al.

Original: ; Houlsby et al.
Revised: ; Houlsby et al.

Original: This choice, however,  can potentially be sub-optimal as early layers may focus on general patterns and late layers on taskspecific ones
Revised: This choice, however, [Figure] can potentially be sub-optimal as early layers may focus on general patterns and late layers on taskspecific ones

Original: Zhang et al.
Revised: Zhang et al.

Original: Allowing the model to dynamically adjust the adapter’s hidden dimension n i (or equivalently, σ i ), and identifying where they should be injected, can help adaptation to the downstream task proficiently.
Revised: Allowing the model to dynamically adjust the adapter’s hidden dimension n i (or equivalently, σ i ), and identifying where they should be injected, can help adaptation to the downstream task proficiently.

Original: 3. O VERALL P ROCEDURE OF TINA
Revised: 3. O VERALL P ROCEDURE OF TINA

Original: Let W V
Revised: Let W V

Original: iT be the initial parameters of the ViT model which are frozen through the whole adaptation process.
Revised: iT be the initial parameters of the ViT model which are frozen through the whole adaptation process.

Original: Our goal is to learn W ada , the set containing the adapter parameters W adai of every
Revised: Our goal is to learn W ada , the set containing the adapter parameters W adai of every

Original: i th
Revised: i th

Original: ViT sub-layer.
Revised: ViT sub-layer.

Original: In previous works Houlsby et al.
Revised: In previous works Houlsby et al.

Original: ; Rebuffi et al.
Revised: ; Rebuffi et al.

Original: , W adai is straightforwardly learned with stochastic-gradient-descent-based optimization.
Revised: , W adai is straightforwardly learned with stochastic-gradient-descent-based optimization.

Original: However, in our experiments (see Sec. 4) we show that this approach does not perform well in the case of tiny adapters (small n i values).
Revised: However, in our experiments (see Sec. 4) we show that this approach does not perform well in the case of tiny adapters (small n i values).

Original: To develop our training scheme, we start from the observation that, with the existing optimizers, sequentially training and pruning a large network is a successful strategy to find small networks with good performance, while directly training small networks usually suffers from optimization issues Frankle & Carbin (2018).
Revised: To develop our training scheme, we start from the observation that, with the existing optimizers, sequentially training and pruning a large network is a successful strategy to find small networks with good performance, while directly training small networks usually suffers from optimization issues Frankle & Carbin (2018).

Original: Therefore, we propose a training procedure where we start from large adapters and  iteratively reduce their dimensions. The overall training procedure is detailed in Algorithm 1.
Revised: Therefore, we propose to start from large adapters and adopt an iterative pruning strategy that iteratively reduce their dimensions -detailed in Algorithm 1-.

Original: Algorithm 1 TINA 1: procedure TINA ( W V iT , W ada , ρ , σtarget ) 2: 
Revised: 1: procedure TINA ( W V iT , W ada , ρ , σ ) 2: Learn W ada [Table]

Original: Learn W ada ▷ W V iT is maintained frozen 3: while σ < σ target do ▷ while compression target is not met 4:
Revised: iT is maintained frozen 3: while σ < σ target do ▷ while compression target is not met 4:

Original: W ada ← top (1 − ρ ) fraction of neurons in W ada  5:
Revised: W ada ← top (1 − ρ ) fraction of neurons in W ada ▷ according to their score (Sec. 4.2) 5:

Original: Fine-tune the remaining W ada on the target dataset ▷ W V iT is maintained frozen 6: end while 7: return W ada 8: end procedure
Revised: Fine-tune the remaining W ada on the target dataset ▷ W V iT is maintained frozen 6: end while 7: return W ada 8: end procedure

Original: We initialize every adapter with a hidden dimension proportional to its input dimensions.
Revised: We initialize every adapter with a hidden dimension proportional to its input dimensions.

Original: We start from constant compression rates σ i for every layer and equal to σ 0 .
Revised: We start from constant compression rates σ i for every layer and equal to σ 0 .

Original: In our first training stage (line 2), we learn the adapter parameters W ada via cross-entropy minimization using stochastic gradient de- scent.
Revised: In our first training stage (line 2), we learn the adapter parameters W ada via cross-entropy minimization using stochastic gradient descent.

Original: Then, we propose to estimate a score that measures the importance of each adapter’s neurons.
Revised: Then, we propose to estimate a score that measures the importance of each adapter’s neurons.

Original: The estimation of this score is detailed in Sec. 3.3.
Revised: The estimation of this score is detailed in Sec. 3.3.

Original: This score is used to select the neurons that have the smallest impact on the adapter outputs.
Revised: This score is used to select the neurons that have the smallest impact on the adapter outputs.

Original: More precisely, we remove the bottomρ fraction  of neurons from W ada (line 4).
Revised: More precisely, we remove the bottom fraction ρ of neurons from W ada (line 4).

Original: The remaining ones will constitute the new adapter configuration and the hidden space sizes n i are updated accordingly.
Revised: The remaining ones will constitute the new adapter configuration and the hidden space sizes n i are updated accordingly.

Original: If the achieved average compression rate σ is still lower than the target σ target , another compression iteration follows; otherwise the achieved configuration will be returned and the method stops.
Revised: If the achieved average compression rate σ is still lower than the target σ target , another compression iteration follows; otherwise the achieved configuration will be returned and the method stops.

Original: Note that the number of training cycles  is given by
Revised: Note that the number of training cycles C is given by: C =

Original: [Equation]
Revised: [Equation]

Original:  , where ⌈·⌉ denotes the ceiling function.
Revised: (cid:25) , where ⌈·⌉ denotes the ceiling function.

Original: Therefore, our training scheme stops after a deterministic number of iterations that can be computed in advance.
Revised: Therefore, our training scheme stops after a deterministic number of iterations that can be computed in advance.

Original: Besides, while we employ a stopping criteria based on a specific target compression rate, a target performance on a validation dataset could be used.
Revised: Besides, while we employ a stopping criteria based on a specific target compression rate, a target performance on a validation dataset could be used.

Original: 3. I MPORTANCE S CORE IN T INY A DAPTIVE A DAPTERS (TINA)
Revised: 3. I MPORTANCE S CORE IN T INY A DAPTIVE A DAPTERS (TINA)

Original: In this section, we present the importance score function that we use in our training algorithm.
Revised: In this section, we present the importance score function that we use in our training algorithm.

Original: Our design of the scoring function is motivated by the observation that, if an entire row in W downi and an entire column in W upi are equal to zero, then our adapter is strictly equivalent to an adapter with a smaller dimension n i .
Revised: Our design of the scoring function is motivated by the observation that, if an entire row in W downi and an entire column in W upi are equal to zero, then our adapter is strictly equivalent to an adapter with a smaller dimension n i .

Original: Therefore, we propose as scoring function to employ the sum of the Lnorm of the corresponding row in W downi and the corresponding column in W upi .
Revised: Therefore, we propose a novel scoring function to employ the sum of the Lnorm of the corresponding row in W downi and the corresponding column in W upi .

Original: More precisely, our importance score is formulated as follows:
Revised: More precisely, our importance score is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: (2) where [ · , · ] denotes the matrix indexing operator.
Revised: where [ · , · ] denotes the matrix indexing operator.

Original: Importantly, I ij is normalized by the total number in the sums to allow fair comparison across adapters with different input and hidden layer sizes.

Original: This importance score can be interpreted as a “look-ahead” strategy, where we observe, besides the output of a specific j -th neuron in the hidden space, also the impact of such an output in the next layer.
Revised: This importance score can be interpreted as a ”look-ahead” strategy, where we observe, besides the output of a specific j -th neuron in the hidden space, also the impact of such an output in the next layer.

Original: Note that this formulation is based only on the magnitude of parameters  and not on the magnitude of activations.
Revised: Note that this formulation is based only on the magnitude of parameters belonging to the same neuron of down-sampling, and its corresponding up-sampling neuron, and not on the magnitude of activations.

Original: This makes the importance score more computation efficient since activation-based scoring would depend on the input images, and consequently, statistics should be gathered at the batch or  dataset level.
Revised: This makes the importance score more computationally efficient since activation-based scoring would depend on the input images, and consequently, statistics should be gathered at the batch or at the dataset level.

Original: This would induce important additional computation.
Revised: This would induce important additional computation.

Original: Furthermore, this choice is empirically supported by many works in the literature, like Chauvin (1988); Han et al.
Revised: Furthermore, this choice is empirically supported by many works in the literature, like Chauvin (1988); Han et al.

Original: (2015b)
Revised: (2015b)

Original: ; Molchanov et al.
Revised: ; Molchanov et al.

Original: ; Renda et al.
Revised: ; Renda et al.

Original: (2020).We also provide more details that motivates equation 2 in the Appendix, at Sec.
Revised: (2020).We also provide more details that motivates equation 2 in the Appendix, at Sec.

Original: A.4.
Revised: A.4.

Revised: Importantly, I ij is normalized by the total number in the sums to allow fair comparison across adapters with different input and hidden layer sizes.

Original:  E XPERIMENTS4. E XPERIMENTS S ETUP
Revised:  E XPERIMENTS

Original: In this section, we provide the details about the datasets and our experimental setup.
Revised: We provide the details about the datasets and our experimental setup.

Original: Datasets.
Revised: Datasets.

Original: We evaluate our methods using the protocol previously adopted by Liu et al.
Revised: We evaluate our methods using the protocol previously adopted by Liu et al.

Original: (2021a), which consists of ten datasets for image classification tasks divided into two benchmarks.
Revised: (2021a), which consists of ten datasets for image classification tasks divided into two benchmarks.

Original: The first benchmark is known as DomainNet Pan et al.
Revised: The first benchmark is known as DomainNet Pan et al.

Original: and is commonly used for domain adaptation tasks.

Original: It contains six different visual domains, which makes the fine-tuning experiments non-trivial.
Revised: It contains six different visual domains, which makes the fine-tuning experiments non-trivial.

Original: Since DomainNet does not have a labeled testing set, we use the validation dataset for testing, as in Pan et al.
Revised: Since DomainNet does not have a labeled testing set, we use the validation dataset for testing, as in Pan et al.

Original: The second benchmark contains CIFAR-10/CIFAR-100 Krizhevsky et al., Oxford Flowers102 Nilsback & Zisserman (2008) and SVHN Netzer et al.
Revised: The second benchmark contains CIFAR-10/CIFAR-100 Krizhevsky et al., Oxford Flowers102 Nilsback & Zisserman (2008) and SVHN Netzer et al.

Original: (2011), which are widely used as low-regime training datasets.
Revised: (2011), which are widely used as low-regime training datasets.

Original: Contrarily to DomainNet, these datasets are not single task oriented, but contain a larger variety of domains/tasks.
Revised: Contrarily to DomainNet, these datasets are not single task oriented, but contain a larger variety of domains/tasks.

Original: For this reason, we refer to them as belonging to the Multi-task benchmark.
Revised: We refer to them as belonging to the Multi-task benchmark.

Original: Implementation Details.
Revised: Implementation Details.

Original: We follow the training protocol adopted by Liu et al.
Revised: We follow the training protocol adopted by Liu et al.

Original: (2021a).
Revised: (2021a).

Original: We conduct our experiments with the official pretrained model Swin-T ( ∼ 27M parameters)
Revised: We conduct our experiments with the official pretrained model Swin-T ( ∼ 27M parameters)

Original: Liu et al.
Revised: Liu et al.

Original: (2021b) trained on ImageNet-1K.
Revised: (2021b) trained on ImageNet-1K.

Original: In all our experiments, we use the AdamW Loshchilov & Hutter (2019) optimizer with a cosine decay learning-rate scheduler for 80 epochs, preceded by 20 epochs of linear warm-up.
Revised: In all our experiments, we use the AdamW Loshchilov & Hutter (2019) optimizer with a cosine decay learning-rate scheduler for 80 epochs, preceded by 20 epochs of linear warm-up.

Original: In all the experiments, the images are resized to the same fixed resolution (224 × 224).[Table]
Revised: In all the experiments, the images are resized to the same fixed resolution (224 × 224).

Original: Contrarily to Liu et al.
Revised: Contrarily to Liu et al.

Original: (2021a) that trains every baseline for 100 epochs only, we train them for 500 epochs to allow fair comparison with our approach that requires multiple training cycles.
Revised: (2021a) that trains every baseline for 100 epochs only, we train them forepochs to allow fair comparison with our approach that requires multiple training cycles C .

Original: With
Revised: With

Original: TINA, ρ is set to 50% , namely we half the number of neurons in the adapters, at each 100 epochs.
Revised: TINA, ρ is set to 50% , namely we half the number of neurons in the adapters, at each 100 epochs.

Original: We use PyTorch Paszke et al.

Original: (2019) to implement our experiments.

Original: The source code will be published upon acceptance.

Original: 4. M AIN RESULTS
Revised: 4. M AIN RESULTS

Original: We compare our proposed method TINA with multiple PETs methods that were initially for NLP transformers.
Revised: We compare our proposed method TINA with multiple PETs methods that were initially for NLP transformers.

Original: We include the following methods.
Revised: We include the following methods

Original: • Full fine-tuning : it fine-tunes all parameters of the model.
Revised: • Full fine-tuning : it fine-tunes all parameters of the model.

Original: • Att/MLP fine-tune : we only tune the Attention/MLP layers and the classification head.
Revised: • Att/MLP fine-tune : we only tune the Attention/MLP layers and the classification head.

Original: Linear-probe : all parameters are frozen except for the task-specific classification layer.
Revised: Linear-probe : all parameters are frozen except for the task-specific classification layer.

Original: • Adapters
Revised: • Adapters

Revised: [Table]

Revised: [Table]

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019): we add adapters with σ = 32 to have adapters with hidden dimensionality proportional to the input dimension
Revised: (2019): we add adapters with σ = 32 to have adapters with hidden dimensionality proportional to the input dimension

Original: m i .
Revised: m i .

Original: We also include variants where the size of every adapter is fixed over all the layers: n i = 47 , and n i = 23 .
Revised: We also include variants where the size of every adapter is fixed over all the layers: n i = 47 , and n i = 23 .

Original: These baselines are considered to emphasize the effect of parameter allocation throughout the layers on the final performance.
Revised: These baselines are considered to emphasize the effect of parameter allocation throughout the layers on the final performance.

Original: • BitFit Ben Zaken et al.
Revised: • BitFit Ben Zaken et al.

Original: (2022): all the weights are frozen, the biases are learned.
Revised: (2022): all the weights are frozen, the biases are learned.

Original: By not storing intermediate activations, this method enables substantial memory savings.
Revised: By not storing intermediate activations, this method enables substantial memory savings.

Original: • PHM-Adapter Zhang et al.
Revised: • PHM-Adapter Zhang et al.

Original: (2021): the weights of the adapters are learned using parameterized hyper-complex multiplication layers (PHM) layers.
Revised: (2021): the weights of the adapters are learned using parameterized hyper-complex multiplication layers (PHM) layers.

Original: • Compacter Karimi Mahabadi et al.
Revised: • Compacter Karimi Mahabadi et al.

Original: (2021): adapter weights are learned using PHM layers.
Revised: (2021): adapter weights are learned using PHM layers.

Original: • LoRa Hu et al.
Revised: • LoRa Hu et al.

Original: (2021): trainable low-rank matrices are employed to approximate the updates for the model.
Revised: (2021): trainable low-rank matrices are employed to approximate the updates for the model.

Original: Updates are applied mainly to the key k and the query q of the attention blocks.
Revised: Updates are applied mainly to the key k and the query q of the attention blocks.

Revised: • AdaptFormer Chen et al.

Revised: (2022): introduces a small module like Adapters, but only after MLP network with a scaling parameters s applied to the output of the injected modules.

Revised: • VPT Jia et al.

Revised: (2022): fine-tuning learnable parameters (i.e. prompts) injected into the embedding space.

Original: Discussion.
Revised: Discussion.

Original: Table 1 reports the number of trained parameters and the average accuracy across datasets in the DomainNet benchmark, while Table 2 contains the results achieved for the Multitask benchmark.
Revised: Table 1 reports the number of trained parameters and the average accuracy across datasets in the DomainNet benchmark, while Table 2 contains the results achieved for the Multitask benchmark.

Original: For both, the number of trained parameters is reported in millions, and the average top-1 accuracy on the datasets is reported in the rightest column.
Revised: For both, the number of trained parameters is reported in millions, and the average top-1 accuracy on the datasets is reported in the rightest column.

Original: We observe that full fine-tuning  has generally the highest accuracy, but it requires a huge number of parameters to be finetuned for each dataset.
Revised: We observe that full fine-tuning [Figure] has generally the highest accuracy, but it requires a huge number of parameters to be finetuned for each dataset.

Original: Among the vanilla fine-tuning baselines, we observe that tuning the parameters of the attention/
Revised: Among the vanilla fine-tuning baselines, we observe that tuning the parameters of the attention/

Original: MLP layer turns out to be surprisingly effective.
Revised: MLP layer turns out to be surprisingly effective.

Original: Nevertheless, it still requires a high number of task-specific parameters, compared to other PET approaches.
Revised: Nevertheless, it still requires a high number of task-specific parameters, compared to other PET approaches.

Original: Linear probing does not perform well illustrating the need to change the feature representations when adapting to new tasks.
Revised: Linear probing does not perform well illustrating the need to change the feature representations when adapting to new tasks.

Original: PHM , LoRA , and compacter are effective methods to get on-par performance with full-model finetune while adjusting less than 2% of the parameters.
Revised: PHM , LoRA , and Compacter are effective methods to get on-par performance with full-model finetune while adjusting less than 2% of the parameters.

Original: Contrarily to what is observed for NLP tasks
Revised: Contrarily to what is observed for NLP tasks

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019), PETs on visual tasks do not reach full-finetune performance on any dataset.
Revised: (2019), PETs on visual tasks do not reach full fine-tuning performance on any dataset with low number of trainable parameters (smaller than 2%).

Revised: VPT does not perform well, indicating that injecting tokens into the embedding space do not help much if the pre-training dataset is different from the downstream task.

Original: Generally speaking, all PET methods maintain similar performance rankings on all the tasks.
Revised: Generally speaking, all PET methods maintain similar performance rankings on all the tasks.

Original: This suggests that the choice of the best adaptation strategy does not depend on the downstream task.
Revised: This suggests that the choice of the best adaptation strategy does not depend on the downstream task.

Original: Adapters outperform all PET methods in terms of accuracy ( 69 . 39% for DomainNet, 92 .
Revised: Adapters outperform all PET methods in terms of accuracy ( 69 . 39% for DomainNet, 92 .

Original: 91% for
Revised: 91% for

Original: Multi-task) but just with a higher number of trainable parameters (1.37M, 4 .
Revised: Multi-task) but just with a higher number of trainable parameters (1.37M, 4 .

Original: 90% of the total) for σ = 32 .
Revised: 90% of the total) for σ = 32 .

Original: When comparing adapter with uniform and proportional parameter distribution, we observe
Revised: Adapters outperform AdaptFormer with fewer parameters (92.91% with 1.37M parameters, versus 92.27% with 2.98M parameters).

Original: [Table] [Figure]
Revised: This result indicates that adapting the representations after both MSA and MLP blocks, as done in Adapters (see Fig2a), allows better adaptation than acting only on the MLP block via a parallel branch (as in AdaptFormer ).

Original:  that allocating parameters proportionally to the layer dimension performs better.
Revised: When comparing adapter with uniform and proportional parameter distribution, we observe that allocating parameters proportionally to the layer dimension performs better.

Original: Indeed, adapters with σ = 32 outperform adapters with n i = 47 ∀ i ( 70 .
Revised: Indeed, adapters with σ = 32 outperform adapters with n i = 47 ∀ i ( 70 .

Original: 65% vs 69 .
Revised: 65% vs 69 .

Original: 39% in DomainNet, 93 .
Revised: 39% in DomainNet, 93 .

Original: 53% vs 92 .
Revised: 53% vs 92 .

Original: 91% in Multi-task).
Revised: 91% in Multi-task).

Original: This suggests that the last layers, which have higher dimensionality, are more task-specific, and consequently require more adaptation.
Revised: This suggests that the last layers, which have higher dimensionality, are more taskspecific, and consequently require more adaptation.

Original: We also observe that reducing the size of adapters ( n i = 23 ) hurts the performance with a drop which, despite being marginal for Multi-task ( 0 .
Revised: We also show that reducing the size of adapters ( n i = 23 ) hurts the performance with a drop which, despite being marginal for Multi-task ( 0 .

Original: 23% ) is more consistent in DomainNet ( 1 . 01% ).
Revised: 23% ) is more consistent in DomainNet ( 1 . 01% ).

Original: This emphasizes that training tiny adapters in a vanilla fashion, leads to unsatisfying performance and motivates our specific training procedure.
Revised: This emphasizes that training tiny adapters in a vanilla fashion, leads to unsatisfying performance and motivates our specific training procedure.

Original: TINA versus Vanilla training.
Revised: TINA versus Vanilla training.

Original: From Table 1 we observe that, in the DomainNet benchmark, TINA outperforms methods with similar trained parameters, in all the compression ranges.
Revised: From Table 1 we observe that, in the DomainNet benchmark, TINA outperforms methods with similar trained parameters, in all the compression ranges.

Original: In particular, in the most challenging one (with 0.30M parameters), TINA outperforms the closest approach, BitFit, which trains 0.34M parameters, showing a gain in average accuracy larger than 2% .
Revised: In particular, in the most challenging one (with 0.30M parameters), TINA outperforms the closest approach, BitFit, which trains 0.34M parameters, showing a gain in average accuracy larger than 2% .

Original: Looking at the Multi-task benchmark (Table 2), we observe that TINA significantly reduces the number of parameters by 4 × (0.40M, 1 . 43% ) while outperforming all PET methods in the Multitask benchmark.
Revised: Looking at the Multi-task benchmark (Table 2), we observe that TINA significantly reduces the number of parameters by 4 × (0.40M, 1 . 43% ) while outperforming all PET methods in the Multitask benchmark.

Original: In particular, TINA outperforms adapters- n i = 47 despite having less parameters, demonstrating that our iterative training procedure improves the parameter efficiency.
Revised: In particular, TINA outperforms adapters- n i = 47 despite having less parameters, demonstrating that our iterative training procedure improves the parameter efficiency.

Original: To further emphasize on the performance gap between the two approaches, we introduce Fig. 3, illustrating the performance as a function of the number of trainable parameters.
Revised: To further emphasize on the performance gap between the two approaches, we introduce Fig. 5, illustrating the performance as a function of the number of trainable parameters.

Original: We observe the significant performance gap between vanilla adapters compared to adapters trained with TINA approach.
Revised: We observe the significant performance gap between vanilla adapters compared to adapters trained with TINA approach.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A BLATION STUDY4.3. I MPORTANCE SCORE FOR TINA: LOCAL VERSUS GLOBAL
Revised: A BLATION STUDY

Revised: Importance score for TINA.

Original: Next, we move on to our design choice of dimensionality reduction inside adapters throughout the adaptation cycles.
Revised: Next [Table] we move on to our design choice of dimensionality reduction inside adapters throughout the adaptation cycles.

Original: We observe the contribution of various components of TINA with different setups.
Revised: We observe the contribution of various components of TINA with different setups.

Original: • Vanilla  L 1 : we select same percentage of neurons for each adapter applied to down-sampling layer only.
Revised: • Vanilla Adapters corresponds to injecting adapters with a compression rate σ . • Vanilla L 1 : we select same percentage of neurons for each adapter applied to down-sampling layer only.

Original: • Local neuron selection : we select the same percentage of neurons for each adapter ( ρ = 50% , 75% , etc...). • Global neuron selection : we select a specific amount of neurons ρ to be removed.
Revised: • Local neuron selection : we select the same percentage of neurons for each adapter ( ρ = 50% , 75% , etc...), applied to down-sampling, and up-sampling layers independently.• Global neuron selection : we select a specific amount of neurons ρ to be removed.

Original: The amount of selected neurons per adapter is calculated using  equation 2, if scaling is applied, otherwise we do not normalize.
Revised: The amount of selected neurons per adapter is calculated using I as in equation 2, if scaling is applied. We also evaluate our scoring function without the scaling factor n i + m i .

Revised: This variant of our score is denoted by I 0 .

Revised: Global with L 1 ( a ) : we select a specific amount of neurons ρ to be removed according to the magnitude of the neuron activation.

Original: • TINA :We use our proposed method asreported in alg.
Revised: • TINA : our proposed method as in Alg. 1.

Original: In order to have a fair comparison between different methods (baseline versus iterative neuron selection) in table 3.
Revised: To compare the different methods we proceed as follows. When using an iterative method, we always start from the baseline model where σ =32 . When using a non-iterative method: we start with adapters of σ 0 = σ target / (1 − ρ ) , and prune once only after the first cycle.

Original: Each column corresponds to an extra training cycle.
Revised: Training continues for

Original: • Baseline corresponds to injecting adapters with a compression rate σ .
Revised: C − 1 cycles to guarantee fair comparison with iterative methods. Results are reported in Table 3.

Original: Discussion.
Revised: Discussion.

Original: Table 3 summarizes the performance of local and global neuron selection for adapters.
Revised: Table 3 summarizes the performance of local and global neuron selection for adapters.

Revised: Firstly, we notice the drop of performance for vanilla adapters, as we reduce the number of parameters -higher values of σ -.

Original: We notice the advantage of using global over local neuron selection.
Revised: Secondly, we notice the advantage of using global over local neuron selection.

Original: Local neuron selection method still has all the adapters injected but with low dimensionality n i .
Revised: Local neuron selection method still has all the adapters injected but with low dimensionality n i .

Original: On the contrary, global neuron selection method emphasizes on finetuning adapters at specific layers, where other adapters are removed completely - more details are provided in Sec. 4.3.2-.
Revised: On the contrary, global neuron selection method emphasizes on finetuning adapters at specific layers, where other adapters are removed completely.

Original: At the last cycles, TINA identifies important adapters for the adaptation, and focuses on finetuning them.
Revised: At the last cycles, TINA identifies important adapters for the adaptation, and focuses on finetuning them.

Original: Thus, the model performance improves by adjusting specific latent representations of the model tailored for the downstream task while training adapters with fewer parameters.
Revised: Thus, the model performance improves by adjusting specific latent representations of the model tailored for the downstream task while training adapters with fewer parameters.

Original: In Table 3, for σ = 64 we observe that TINA (using either local/global neuron selection) achieves better performance than a vanilla training of adapters on VGG-Flowers dataset with a performance gap of 6 .
Revised: In Table 3, for σ = 64 we observe that TINA (using either local/global neuron selection) achieves better performance than a vanilla training of adapters on VGG-Flowers dataset with a performance gap of 6 .

Original: Interestingly, this gap in performance expands, as we compare smaller adapters σ = 256 , 512 .
Revised: Interestingly, this gap in performance expands, as we compare smaller adapters σ = 256 , 512 .

Original: When comparing to a vanilla L 1 importance scoring, we see the beneficial effect of considering parameters both in the downsampling and in the upsampling for the adapters, with consistent gains in performance for all the explored compression rates, ranging from 0 .
Revised: When comparing to a vanilla L 1 importance scoring, we see the beneficial effect of considering parameters both in the downsampling and in the upsampling for the adapters, with consistent gains in performance for all the explored compression rates, ranging from 0 .

Original: 5% to more than 3% .
Revised: 5% to more than 3% .

Original: We can observe that the performance gap is especially clear with high compression rates.
Revised: We can observe that the performance gap is especially clear with high compression rates.

Revised: Finally, scaling the importance score as in equation 2 boosts the performance of Global for every σ value by about 1%.

Original: 4.3. P ARAMETER -A LLOCATION ANALYSIS WITH TINA
Revised: Parameter allocation analysis with TINA.

Original: Fig.
Revised: Fig.

Original: 4 illustrates the distribution of the removed and remaining neurons using TINA on VGGFlowers, and CIFAR-10.
Revised: 4 illustrates the distribution of the removed and remaining neurons using TINA on VGG-Flowers, and CIFAR-10.

Original: We observe the difference between local neuron selection that removes uniformly neurons from adapters and global neuron selection .
Revised: We observe the difference between local neuron selection that removes uniformly neurons from adapters and global neuron selection .

Original: We witness that the latter totally removes some adapters from the model (see layers 4 , 5 , 7 , 8 on VGG-Flowers) and allocates many parameters to some other adapters.
Revised: We witness that the latter totally removes some adapters from the model (see layers 4 , 5 , 7 , 8 on VGG-Flowers) and allocates many parameters to some other adapters.

Revised: We provide normalized plots for VGG-Flowers, and CIFAR100 illustrating the distribution of removed neurons (Fig. 9, and 10)

Original: Furthermore, global neuron selection adapts differently to each dataset as shown in Fig.
Revised: Furthermore, global neuron selection adapts differently to each dataset as shown in Fig.

Original: The distribution of removed neurons is different for CIFAR-10, where fewer adapters have been removed completely with respect to VGG-Flowers.
Revised: The distribution of removed neurons is different for CIFAR-10, where fewer adapters have been removed completely with respect to VGG-Flowers.

Original: For VGG-Flowers, only adapters at late stages are kept,which may indicate that early layer’s representations are suitable for this dataset.
Revised: For VGG-Flowers, only adapters at late stages are kept,

Original: However for CIFAR-10, remaining adapters are widespread through all layers of ViT, even low level features need tuning to the new task.

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: N u m b e r o f N e u r o n s n i
Revised: N u m b e r o f N e u r o n s n i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Removed
Revised: Removed

Original: Remaining (a) Local pruning: all datasets.
Revised: Remaining (a) Local pruning: all datasets.

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: N u m b e r o f N e u r o n s n i
Revised: N u m b e r o f N e u r o n s n i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Removed
Revised: Removed

Original: Remaining (b) Global pruning: VGG-Flowers .
Revised: Remaining (b) Global pruning: VGG-Flowers .

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: N u m b e r o f N e u r o n s n i
Revised: N u m b e r o f N e u r o n s n i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Removed
Revised: Removed

Original: Remaining (
Revised: Remaining (

Original: [Figure]
Revised: [Figure]

Original: pruning
Revised: pruning

Original: [Figure]
Revised: [Figure]

Original: CIFAR-10.
Revised: CIFAR-10.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: which may indicate that early layer’s representations are suitable for this dataset. However for CIFAR-10, remaining adapters are widespread through all layers of ViT.

Original: V I T V ARIANTS WITH TINA
Revised: ViT variants with TINA.

Original: We analyze the performance of TINA using different ViT backbones.
Revised: We analyze the performance of TINA using different ViT backbones.

Original: We inject TINA  into Swin-S, ViT Dosovitskiy et al.
Revised: TINA is injected into Swin-S, ViT Dosovitskiy et al.

Original: and Convolutional Vision Transformers that improves ViTsin performance and efficiency by introducing convolutions into ViT to yield the best of both designs Wu et al.
Revised: and CVT Wu et al. (2021), that improves ViTs by introducing convolutions into ViT to yield the best of both designs .

Original: We train the 3 baselines Finetuning , Adapters [Table] and TINA for 3 cycles.
Revised: We train the 3 baselines Finetuning , Adapters , and TINA for 3 cycles.

Original: We report the best score for the last cycle in Table 4.
Revised: We report the best score for the last cycle in Table 4.

Original: TINA achieves on-par performance with respect to full model finetuning with a gap of 1.2%, 1.2% and 1.4% for ViT-B/16, Swin-S and CvT, respectively.
Revised: TINA achieves on-par performance with respect to full model finetuning with a gap of 1.2%, 1.2% and 1.4% for ViT-B/16, Swin-S and CvT, respectively.

Original: By finetuning less than 1.5% of parameters including the head classifier.
Revised: By finetuning less than 1.5% of parameters including the head classifier.

Original: TINA outperforms vanilla adapters with 4 times fewer parameters on all ViT backbones (ViT, Swin-T and CvT) models.
Revised: TINA outperforms vanilla adapters with 4 times fewer parameters on all ViT backbones (ViT, Swin-T and CvT) models.

Original: These experiments show that TINA do generalize to various ViT backbones.
Revised: These experiments show that TINA do generalize to various ViT backbones.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we propose TINA, a training algorithm to learn TINy Adapters for the problem of ViT finetuning.
Revised: In this work we propose TINA, a training algorithm to learn TINy Adapters for the problem of ViT finetuning.

Original: Rather than directly training adapters with few parameters, we propose to start with large and over-parametrized adapters, and then, iteratively select the more important neurons in every adapter.
Revised: Rather than directly training adapters with few parameters, we propose to start with large and over-parametrized adapters, and then, iteratively select the more important neurons in every adapter.

Original: Our training procedure estimates the hidden dimension for each adapter which reduces the number of trainable parameters and even removes adapters at certain layers when necessary.
Revised: Our training procedure estimates the hidden dimension for each adapter which reduces the number of trainable parameters and even removes adapters at certain layers when necessary.

Original: Experimentally we demonstrate the greater performance of our training scheme with respect to vanilla adapters and show that our method achieves good performance with unprecedentedly low numbers of trainable parameters.
Revised: Experimentally we demonstrate the greater performance of our training scheme with respect to vanilla adapters and show that our method achieves good performance with unprecedentedly low numbers of trainable parameters.

Original: Our ablation study validates the positive impact of our adaptive strategy to estimate the hidden dimension of each adapter.
Revised: Our ablation study validates the positive impact of our adaptive strategy to estimate the hidden dimension of each adapter.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this appendix, we provide: (1) additional experimental results to further analyze the proposed
Revised: In this appendix, we provide: (1) additional experimental results to further analyze the proposed

Original: TINA approach.
Revised: TINA approach.

Original: (2) justify in more detail our choice for the design of the importance score (3) provide details regarding the datasets used in our experiments. 
Revised: (2) justify in more detail our choice for the design of the importance score (3) provide details regarding the datasets used in our experiments. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: A. L OCAL VERSUS G LOBAL N EURONS S ELECTION
Revised: [Table] [Table] [Table] Adapters

Revised: Full-ﬁnetuning

Revised: TINA

Revised: Top-1 Accuracy versus compression rate σ

Revised: on CIFAR

Revised: [Figure]

Revised: We notice that at σ = 2 , 4 , 8 , TINA outperforms full finetuning.

Revised: A. L OCAL VERSUS G LOBAL N EURONS S ELECTION

Revised: [Table]

Original: Here below
Revised: Here below

Original: [Table]
Revised: [Table]

Original: we provide extra experiments for adapters with different sizes for VGG-Flowers,
Revised: we provide extra experiments for adapters with different sizes for VGG-Flowers,

Original: CIFAR-10, and CIFAR-100.
Revised: CIFAR-10, and CIFAR-100.

Original: [Table]

Original: Table 5 reports the performance of TINA algorithm with respect to vanilla training - Baseline -.
Revised: Tables 5and 6 report the performance of TINA algorithm with respect to vanilla training - Baselineon VGG-Flowers, CIFAR-10 and CIFAR-100.

Original: TINA outperforms vanilla adapters on all the adapters with a significant performance gap.
Revised: TINA outperforms vanilla adapters on all the adapters with a significant performance gap.

Original: Interestingly, this gap in performance expands, as we compare smaller adapters σ = 256 , ..., 4096 (higher compression ratio).
Revised: Interestingly, this gap in performance expands, as we compare smaller adapters σ = 256 , ..., 4096 (higher compression ratio).

Original: Furthermore,  resultsin Table 5 emphasizes on the usefulness of each component of the importance score for TINA.
Revised: Furthermore, these results emphasizes on the usefulness of each component of the importance score for TINA.

Original: We notice that applying global neuron selection, normalization and iterative training outperforms local neuron selection on almost all adapter sizes for VGG-Flowers, and CIFAR-100.{+CIFAR
 (Table 6).+}
Revised: We notice that applying global neuron selection, normalization and iterative training outperforms local neuron selection on almost all adapter sizes for VGG-Flowers (Table 5) and {+CIFAR
 (Table 6).+}

Original: This indicates that each component of the importance score of TINA is important to boost the performance and reduce the parameters. 
Revised: This indicates that each component of the importance score of TINA is important to boost the performance and reduce the parameters. [Table]

Original: A. V ANILLA VERSUS TINA T RAINING FOR A DAPTERS
Revised: V ANILLA VERSUS TINA T RAINING FOR A DAPTERS

Original: To validate the greater optimization performance of TINA, in Figs.
Revised: To validate the greater optimization performance of TINA, in Figs.

Original: 5a, 5b, we show the training loss curves of vanilla, and TINA training of adapters for CIFAR-100, and SVHN, respectively.
Revised: 6a, 6b, we show the training loss curves of vanilla, and TINA training of adapters for CIFAR-100, and SVHN, respectively.

Original: At the end of training, the two models ( i.e. vanilla training and TINA) have similar numbers of training parameters.
Revised: At the end of training, the two models ( i.e. vanilla training and TINA) have similar numbers of training parameters.

Original:  We notice that the loss of the training using TINA algorithm is much smoother than vanilla training resulting in adapters that generalize well on the downstream task as previously shown in Fig.
Revised: [Figure] We notice that the loss of the training using TINA algorithm is much smoother than vanilla training resulting in adapters that generalize well on the downstream task as previously shown in Fig.

Original: Furthermore, we notice spikes in the training loss of TINA, due to the removal of neurons after each cycle.
Revised: Furthermore, we notice spikes in the training loss of TINA, due to the removal of neurons after each cycle.

Original: Eventually, sequential training and neuron selection is a successful strategy to find small networks while maintaining good performance, since directly training small networks does not provide similar results Frankle & Carbin (2018).[Figure]
Revised: Eventually, sequential training and neuron selection is a successful strategy to find small networks while maintaining good performance, since directly training small networks does not provide similar results Frankle & Carbin (2018).

Original: A.2. I MPACT OF THE PARAMETER ALLOCATION
Revised: A.2. I MPACT OF THE PARAMETER ALLOCATION

Original: In this ablation study, we validate the idea that every layer of a ViT needd to be adapted differently in order to learn a new task.
Revised: In this ablation study [Table] we validate the idea that every layer of a ViT needd to be adapted differently in order to learn a new task.

Original: The Swin-B vision transformer model that we use in all our experiments consists of 4 stages.
Revised: The Swin-B vision transformer model that we use in all our experiments consists of 4 stages.

Original: Therefore, we propose to evaluate the performance when we vary the adapter size in each stage.
Revised: Therefore, we propose to evaluate the performance when we vary the adapter size in each stage.

Original: The results are reported in table 6.
Revised: The results are reported in table 7.

Original: First, it shows that the size of adapters has a strong effect on the model performance.
Revised: First, it shows that the size of adapters has a strong effect on the model performance.

Original: In general, the best performances are achieved when using adapters with higher number of parameters.
Revised: In general, the best performances are achieved when using adapters with higher number of parameters.

Original: Furthermore, bigger sizes of adapters are not sufficient for better performance, but subject to which stage they are injected into.
Revised: Furthermore, bigger sizes of adapters are not sufficient for better performance, but subject to which stage they are injected into.

Original: We observe that adding adapters to late stages ( i.e. III and IV), boosts the performance better than injecting them into early stages: adapters with σ i = 

Revised: We observe that adding adapters to late stages ( i [Table] e. III and IV), boosts the performance better than injecting them into early stages: adapters with σ i =

Original: added to (III, IV) stages rather than (I, II) improves the performance from 95.29% , 73.38% to 97.40% , 87.04% on CIFAR-10, and VGG-Flowers, respectively. 
Revised: added to (III, IV) stages rather than (I, II) improves the performance from 95.29% , 73.38% to 97.40% , 87.04% on CIFAR-10, and VGG-Flowers, respectively. [Table] 128 ∞ ∞ 96.

Original: A. I LLUSTRATIONS OF L OCAL VERSUS G LOBAL N EURON S ELECTION
Revised: A. I LLUSTRATIONS OF L OCAL VERSUS G LOBAL N EURON S ELECTION

Original: Figures6, and 7 represent extra illustrations  at different cycles for  local and global neuron selection methods for VGG-Flowers and CIFAR-10 datasets.
Revised: Figures 7 and 8 show additional illustrations of the distribution of the removed and remaining neurons using TINA on VGG-Flowers [Figure] and CIFAR-10. We show the learned adapters at different cycles for both local and global neuron selection methods.

Revised: We also complete these visualizations (Figures 9 and 10) with histograms where we show the percentages of remaining neurons.

Original: Overall, these experiments show that our method is capable of obtaining different parameter allocations that are specific to every task.
Revised: Overall, these experiments show that our method is capable of obtaining different parameter allocations that are specific to every task [Figure] [Figure] [Figure]

Revised: [Figure]

Original: A.3. S ERIES A DAPTERS V ERSUS P ARALLEL A DAPTERS
Revised: f N e u [Figure] n s n i Adapter

Revised: [Figure]

Revised: i

Revised: Adapter index

Revised: i

Revised: Removed

Revised: Remaining

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: o

Revised: [Figure]

Revised: N e u

Revised: [Figure]

Revised: o n s n i

Revised: [Figure]

Revised: i

Revised: Adapter index

Revised: i

Revised: Removed

Revised: Remaining

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: N e u r o n s n i

Revised: Adapter index

Revised: i

Revised: Adapter index

Revised: i

Revised: Removed

Revised: Remaining

Revised: [Figure]

Revised: [Figure]

Revised: Remaining

Revised: [Figure]

Revised: [Figure]

Revised: Adapter index

Revised: i

Revised: Adapter index

Revised: i

Revised: Remaining (

Revised: [Figure]

Revised: pruning: CIFAR-10.

Revised: A.3. I MPACT OF ρ

Revised: In this section, we investigate the effect of the hyper-parameter ρ .

Revised: In figure 11.

Revised: We notice that higher values of ρ hurts the performance, because we remove many parameters after each cycle, but we reduce the size of adapters significantly.

Revised: On the other hand, if ρ is small (i.e 25%), we maintain good performance on VGG-Flowers dataset, but it requires higher training cycles C to reach the target compression rate σ target .

Revised: We have a trade-off between the performance, and training budget in order to reach the σ target .

Revised: Removing too much parameters at each cycles, hurts performance.

Revised: Maintaining good performance requires higher number of training cycles C .

Revised: A.3. S ERIES A DAPTERS V ERSUS P ARALLEL A DAPTERS

Original: In this experiment, we would like to investigate two different designs for adapters as illustrated in
Revised: In this experiment, we would like to investigate two different designs for adapters as illustrated in

Original: Figure 8.
Revised: Figure 12.

Original: Traditional adapters known as series adapters (SA)
Revised: Traditional adapters known as series adapters (SA)

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019), and another variant parallel adapters (PA) He et al.
Revised: (2019), and another variant parallel adapters (PA) He et al.

Original: The parallel configuration PA acts as a perturbation on the base network, the serial configuration more directly changes the hidden representations fed into the next layer.
Revised: The parallel configuration PA acts as a perturbation on the base network, the serial configuration more directly changes the hidden representations fed into the next layer.

Original: In this way, the parallel adapters are less prone to the loss of the ‘knowledge’ [Table] [Figure] [Table]
Revised: In this way, the parallel adapters are less prone to the loss of the ‘knowledge’ stored in the base model.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: stored in the base model.

Original: Also, they are more efficient during training, where we can benefit from parallel operations.
Revised: Also, they are more efficient during training, where we can benefit from parallel operations.

Original: On the other hand, SA increases the latency of the model at inference time.
Revised: On the other hand, SA increases the latency of the model at inference time.

Original: [Table]

Original: According to results in table 7, SA outperforms PA with 1.5% on Quickdraw dataset.
Revised: According to results in Table 8, SA outperforms PA with 1.5% on Quickdraw dataset.

Original: It indicates that acting directly on the feature allows more efficient adaptation to the new task.
Revised: It indicates that acting directly on the feature allows more efficient adaptation to the new task.

Original: Furthermore, we notice that both PA, and SA benefits from the higher sizes of adapters, where over-parametrization eases the optimization process and improves performance.
Revised: Furthermore, we [Figure]

Original: A. M AGNITUDE ASSUMPTION AS IMPORTANCE SCORE
Revised: i

Original: In this section we provide more insights on the importance score employed within TINA.
Revised: [Figure] a i n i n g n e u r o n s (

Original: In particular, under Gaussian input assumption for adapters and imposing weight decay at training time, we
Revised: [Figure]

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: N u m b e r o f N e u r o n s n i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Revised: Remaining

Original:  Adapterindex
Revised: [Figure] Adapter

Revised: [Figure]

Original: i
Revised: i

Original: Removed
Revised: [Figure]

Original: Remaining (a) Local pruning: all datasets.
Revised: a i n i n g n e u r o n s [Figure] Adapter index i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: N u m b e r o f N e u r o n s n i
Revised: Remaining ( [Figure] Flowers .

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Adapter index
Revised: [Figure]

Original: i
Revised: i

Original: Removed Remaining (b) Global pruning: VGG-Flowers . Adapter index
Revised: [Figure]

Original: i
Revised: i

Original: N u m b e r o f N e u r o n s n i
Revised: [Figure]

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Adapter index
Revised: Adapter index

Original: i
Revised: i

Original: Removed Remaining (c)
Revised: Remaining [Figure]

Original: Global pruning: CIFAR-10. 
Revised: pruning: CIFAR-10. [Figure]

Revised: A Performance on VGG-Flowers with various values for p (amount of neurons remo

Original: [Figure]
Revised: [Figure]

Original: notation, as we will always refer to the same adapter.
Revised: notice that both PA, and SA benefits from the higher sizes of adapters, where over-parametrization eases the optimization process and improves performance. A. M AGNITUDE ASSUMPTION AS IMPORTANCE SCORE

Revised: In this section we provide more insights on the importance score employed within TINA.

Revised: In particular, under Gaussian input assumption for adapters and imposing weight decay at training time, we will see that, towards a better choice of parameters to be removed, considering just W down

Revised: i is suboptimal, and W up i should be accounted as well.

Revised: We drop the adapter index i for abuse of notation, as we will always refer to the same adapter.

Original: Let us have an m -dimensional input h , whose elements are distributed according to a Gaussian
Revised: Let us have an m -dimensional input h , whose elements are distributed according to a Gaussian

Original: N ( µ k , Σ k ) .
Revised: N ( µ k , Σ k ) .

Original: We assume the adapter has already been trained; hence we consider, in the downsampling phase all the w downjk as constants.
Revised: We assume the adapter has already been trained; hence we consider, in the downsampling phase all the w downjk as constants.

Original: Form the property of linear expectations, we know that, before reaching the non-linear activation, the post-synaptic potential is still a Gaussian random variable, having average
Revised: Form the property of linear expectations, we know that, before reaching the non-linear activation, the post-synaptic potential is still a Gaussian random Parallel Adapter Series Adapter LayerNorm LayerNorm

Revised: MLP

Original: [Equation]
Revised: Adapter LayerNorm Adapter LayerNorm

Revised: Input X

Revised: LayerNorm

Revised: MSA

Revised: Adapter

Revised: LayerNorm

Revised: MLP

Revised: Adapter

Revised: Input X h h' h'

Revised: MSA

Revised: Adapter

Revised: Upsample

Revised: Updated

Revised: Frozen

Revised: Downsample

Revised: [Table]

Revised: variable, having average µ

Revised: [Equation]

Revised: (cid:88)

Original:  and variance 
Revised: [Equation] (3) and variance Σ [Equation] =

Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original:  where Σ ab indicates an element of the covariance matrix for the input of the adapter.
Revised: W [Equation] (cid:35) (4) where Σ ab indicates an element of the covariance matrix for the input of the adapter.

Original: For sake of easier tractation, if we assume Σ kk ′ = 0 ∀ k ̸
Revised: For sake of easier tractation, if we assume Σ kk ′ = 0 ∀ k ̸

Original: = k ′ , equation 4 simply reduces to
Revised: = k ′ , equation 4 simply reduces to

Original: [Equation]
Revised: [Equation]

Original: In transformers, the commonly-used activation function is the Gaussian error linear unit (GELU), whose analytical expression is
Revised: In transformers, the commonly-used activation function is the Gaussian error linear unit (GELU), whose analytical expression is

Original: [Equation]
Revised: [Equation]

Original: where erf ( · ) is the error function.
Revised: where erf ( · ) is the error function.

Original: For values close to zero, or larger than zero, it can be approximated to the identity function, while for values much lower than zero, it asymptotically tends to zero.
Revised: For values close to zero, or larger than zero, it can be approximated to the identity function, while for values much lower than zero, it asymptotically tends to zero.

Original: Let us focus on the first scenario: we can approximate the post-synaptic potential to the output of the non-linearity, saying that the output
Revised: Let us focus on the first scenario: we can approximate the post-synaptic potential to the output of the non-linearity, saying that the output

Original: [Equation]
Revised: [Equation]

Original: At this point, the signal undergoes an up-sampling: following-up on the same approach adopted for the down-sampling, we find that the output r still follows an Gaussian distribution having average
Revised: At this point, the signal undergoes an up-sampling: following-up on the same approach adopted for the down-sampling, we find that the output r still follows an Gaussian distribution having average

Original: [Equation]
Revised: [Equation]

Original: and variance

Original: [Equation]
Revised: [Equation]

Original: In order to assess the impact of removing a whole neuron in the embedding space, we can write the
Revised: In order to assess the impact of removing a whole neuron in the embedding space [Equation] we can write the

Original: KL-divergence of the distribution for r l with and without the a -th neuron in the embedding space
Revised: KL-divergence of the distribution for r l with and without the a -th neuron in the embedding space

Original: [Equation]
Revised: [Equation]

Original: According to equation 10, we can rewrite equation 11 as
Revised: According to equation [Equation] we can rewrite equation 11 as

Original: [Equation]
Revised: [Equation]

Original: Let us now investigate which is the a -th neuron which, when removed, causes the least perturbation at the output r l (or in other words, such that D KL ( r l , r l, ¯ a ) is as low as possible).Looking at the argument of the logarithm, we ask ( W upal ) [Equation] mk
Revised: Let us now investigate which is the a -th neuron which, when removed, causes the least perturbation at the output r l (or in other words, such that D KL ( r l , r l, ¯ a ) is as low as possible).

Original: [Equation]

Original: [Equation]

Original:  W downak [Equation] 2 Σ kk Σ upl = [-
-] and, since we can safely assume Σ upl , we need to select a such that ( W upal ) 2 (cid:80) mk =1 (cid:0) W downak (cid:1) 

Revised: Looking at the argument of the logarithm, we ask ( W upal ) 2 (cid:80) mk =1 ( W downak ) 2 Σ kk Σ upl = [-
-]0 and, since we can safely assume Σ upl , we need to select a such that ( W upal ) 2 (cid:80) mk =1 (cid:0) W downak (cid:1)

Original: Σ kk = 0 .
Revised: Σ kk = 0 .

Original: Considering that also Σ kk > 0 ∀ k , we satisfy the condition if either:
Revised: Considering that also Σ kk > 0 ∀ k , we satisfy the condition if either:

Original: • W downak = 0 ∀ k , namely the L 1 norm for W down − a is zero; • W upal = 0 .
Revised: • W downak = 0 ∀ k , namely the L 1 norm for W down − a is zero; • W up al = 0 .

Original: Considering though that this condition needs to be satisfied for all the l outputs of the adapters, we ask W upal = 0 ∀ l or, in other words, the L 1 norm for W upa − is also zero.
Revised: Considering though that this condition needs to be satisfied for all the l outputs of the adapters, we ask W upal = 0 ∀ l or, in other words, the L 1 norm for W upa − is also zero.

Original: We observe that, when either of the two conditions are met, the KL divergence is zero as
Revised: We observe that, when either of the two conditions are met, the KL divergence is zero as

Original: [Equation]
Revised: [Equation]

Original: We can also assume that, if either W downak = 0 ∀ k or W upal = 0 ∀ l , the norm of the non-zero parameters associated to some neuron a are small when training with any weight penalty regularizer (as the contribution to the output is zero, the signal is either not forward or back-propagated, leaving the weight penalty term the only update for these parameters).
Revised: We can also assume that, if either W downak = 0 ∀ k or W upal = 0 ∀ l , the norm of the non-zero parameters associated to some neuron a are small when training with any weight penalty regularizer (as the contribution to the output is zero, the signal is either not forward or back-propagated, leaving the weight penalty term the only update for these parameters).

Original: This further motivates our proposed importance score given in equation 2.
Revised: This further motivates our proposed importance score given in equation 2.

Original: A. D ETAILS ABOUT D ATASETS
Revised: A. D ETAILS ABOUT D ATASETS

Original: In Table 8, we report different statistics that capture the diversity of the datasets we use in our experiments.
Revised: In Table 9, we report different statistics that capture the diversity of the datasets we use in our experiments.

Original: [Table]
Revised: [Table]

Original: A. B ASELINES D ETAILS : PHM, C OMPACTER , AND L O RA
Revised: A. B ASELINES D ETAILS : PHM, C OMPACTER , AND L O RA

Original: In order to have, a comparable number of trainable parameters for the baselines listed in Table 1 and
Revised: In order to have, a comparable number of trainable parameters for the baselines listed in Table 1 and

Original: Table 2.
Revised: Table 2.

Original: We set Compacter n = 12 , and for PHM , n = 12 .
Revised: We set Compacter n = 12 , and for PHM , n = 12 .

Original: In LoRa [Table] the budget is expressed with a rank parameter r .
Revised: In LoRa, the budget is expressed with a rank parameter r .

Original: We evaluate different rank values for
Revised: We evaluate different rank values for

Original: LoRa and reported in our experiments the configuration leading to the best performance.
Revised: LoRa and reported in our experiments the configuration leading to the best performance.

Original: For this preliminary experiment, we considered the Quickdraw dataset as a proxy to select the best r .
Revised: For this preliminary experiment, we considered the Quickdraw dataset as a proxy to select the best r .

Original: The result is presented in Table 9.
Revised: The result is presented in Table 10.

Original: Rank r
Revised: Rank r

Original: = 8 gives the best performance with a satisfying number of trainable parameters
Revised: = 8 gives the best performance with a satisfying number of trainable parameters.

Original: [Table]
Revised: [Table]

Original: [Table]