From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This work addresses the problem of parameter-efficient fine-tuning in computer vision, proposing a method that iteratively reduces the size of adapters via pruning. The paper is well-written, easy to follow, and the proposed method is intuitive and interesting. The reviewers raised multiple concerns such as incremental novelty, computational inefficiencies beyond parameter reduction, and inferior results to naive fine-tuning. The author response provided useful clarifications, as well as new experiments including additional baselines like VPT and AdaptFormer. The AC agrees with the majority of the reviewers that the paper is not ready for ICLR, due to the following reasons. First, the idea of identifying the right dimensions of adapters in each layer has been explored before, e.g., methods such as LoRA (low-rank adapters), which are based on low-rank factorization, a typical approach for model compression. The authors adopt pruning instead, which is also common in model compression. Knowledge distillation, as suggested by one of the reviewers, could be yet be another alternative. Therefore, the method is incremental in nature. If the novelty focus is on the score function, then it should be evaluated more comprehensively in the context of model compression approaches. Second, while the approach does have advantages in terms of number of parameters, it does introduce other computational inefficiencies, such as multiple rounds of pre-training as pointed out by one of the reviewers. In the space of parameters vs accuracy, from prompting to adapters to fine-tuning, it is not clear whether the proposed method would provide significant value to practitioners. 
justification_for_why_not_higher_score: See the two reasons above
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: [ Reviews ]
comment: Dear Reviewers:

We sincerely appreciate your patient reading and insightful comments, which have greatly aided us in enhancing our article. We have responded to the reviewers' questions, and added the requested experiments.

The end of discussion phase is approaching.  So, if you have any questions or recommendations, please let us know as soon as possible.

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: General response 
comment: We thank all reviewers for providing constructive thoughtful feedback!

We are deeply encouraged by positive comments from the reviewers such as "simple, easy and effective method" (R #sXuV, R #QrJw), "Extensive experiments" (R #fTqj), "TINA achieves or exceeds the performance..." (R #sXuV).

----------------------------------------------------------------------

We appreciate these positive comments and would like to answer reviewer's questions, and provide more experiments and analyses to further improve our paper. We summarise the additional experiments/analyses we made in the rebuttal per the suggestions from the reviewers.

-   **[Experiment added]** We evaluate our method with respect to new PET methods (PVT [1], and AdaptFormer[2]).

-   **[Experiment added]** We show that outperforming full model finetuning is subject to the initial starting size of adapters. With a high number of trainable parameters (10%-15% of the original size), we outperform full model finetuning.

-   **[Experiment added]** We added a new baseline Adapter with size $n_i=1$.

-   **[Experiment added]** We added a pruning baseline based on activation magnitude (Table 3) in section 4.3.

-   **[Experiment added]** We added an experiment to study the trade-off of $rho$ versus number of cycles $C$ in Appendix 3.1.

----------------------------------------------------------------------

In addition, we revised our paper as listed in the following points, and we colorize them _blue_ in the revised version.

-   We corrected the grammar mistakes, and made writing of the method, results more clear, and simple.

-   We added in the related work section, the newly evaluated baselines PVT[1], and AdapFormer[2].

-   We modified Table 3 to make it self-contained.

-   We modified Figure 4 and its caption for a better understanding.

-   We added experiments with varying initial sizes of adapters -Figures 3, and 5-.

-   We added the new experiments, and their discussion.

-   We added new Figures 9 and 10 of the normalised plots for the adapter's layers.

-   We added a new section: Analysis of $\rho$ and $C$ trade-off, and Figure 11.

* * * * *

**References**:

[1] Jia et al. Visual prompt tuning, ECCV-2022.

[2] Chen et al. Adapting vision transformers for scalable visual recognition, Neurips-2022

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: Answers to reviewer #qoFV
comment: Thanks for the suggestions, and the points raised to further improve our paper. Below we address each question and review suggestions in a point by point fashion.

**1- Novelty of TINA**
- Please refer to Reviewer #QrJw, question 2.
- In Table 3, we compare the importance score -Eq. 2- with respect other pruning baselines such as:
<ul>
<li>Vanilla L1: consider only the magnitude of the down-sampling weights. 
<li>Local: prune both down-sampling and up-sampling layers, independently, with respect to the magnitude of their neurons.
</ul>
- **We added another pruning baseline, 'global (activation)'**: pruning with respect to the magnitude of the activations.
- We have updated our paper to clarify more these points - see section 4.2-.


**2- Clarifications about ablation study**
- For ablation study (Table 3), we always use the same architecture. 
- We changed the label from “Baseline” to “Vanilla Adapters”.
- When using an **iterative method**, we always start from the baseline model where $\sigma=32$.
- When using a **non-iterative method**: we start with adapters of size $\sigma_{0}=\frac{\sigma_{target}}{1-\rho}$, and we just prune once. 
- We have updated our paper, to further clarify these points –see Table 3, and section 4.3-.

**Clarifications about Section 4.2**
- TINA and Vanilla adapters are equivalent until the end of the first training cycle. 
- As requested, we changed the label to $Adapters$ in the paper -Table 1, and 2-. 
 
**3- Clear identification of baselines**
- Thanks for the suggestions. To further clarify these points, we have added new Figures 3 and 5 to show the performance gap between vanilla adapters, and TINA. In the first row vanilla adapters (Table 3) correspond to (adapters non-pruned) with different $\sigma$. We remark that these baselines were already included in Ablation study, in Table 3 (also in Appendix, Tables 5 and 6).

- The conclusion of these experiments is that **for vanilla adapters, as we reduce the number of parameters (larger $\sigma$ values), the performance drops** (see Table 3, Figure 3, and Appendix, Figure 5). Sequentially training and pruning a large network is a successful strategy to find small networks with good performance, while directly training small networks usually suffers from optimization issues [1, 2].

- **We added a new baseline Adapters-$n_i=1$** (Tables 1 and 2) to emphasise more on the main motivation of our work “starting directly from smaller adapters performs poorly”.
- We have updated  our paper to clarify more these points -see section 4.3-.


**4 - Normalised plots**
- Thanks for your feedback, we generated the plots as suggested. Nevertheless it turns out that when normalising scores, our perception of the impact of small layers is biased. We have added normalised Figures 9, 10 in Appendix A.3 and kept the original one in the main paper.

**5 - TINA versus Full fine-tuning**
- Please refer to reviewer #fTqj, answer 3.
- We added a new figure, to clarify this - see Figure 3, and in Appendix, Figure 5-.

------------------------------------------------------------------------------------------
We also provided a detailed description of the extra experiments, and changes in a general comment on top of all reviews;

**References**
- [1] Frankle et al. “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”, ICLR2019.
- [2] Pavlo Molchanov et al. “Pruning Convolutional Neural Networks for Resource Efficient Inference”, ICLR2017.
- [3] Renda et al, “Comparing Rewinding and Fine-tuning in Neural Network Pruning”, ICLR2020.

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: Answers to #QrJw
comment: Thanks for the suggestions, and the points raised to further improve our paper. Below we address each question and minor questions in a point by point fashion.

**1- Pre-training complexity**
- We acknowledge this shortcoming of TINA. Nevertheless, our experiments show that our approach can scale to datasets up to 120K samples (Quickdraw, and Real datasets).
- Small or medium datasets correspond to scenarios that are very common in vision tasks. TINA has many possible use cases despite this limitation.

**2- Comparison with standard methods & novelty**
- We do not share the reviewer opinion that the novelty is limited -see sections 1, 3.3-. TINA is not a pruning method: TINA is a training strategy for adapters that combines a pruning method and our novel scoring function. **This scoring function takes into consideration the impact on the up-sampling layer for removing neurons in the down-sampling layer- see section 3.3-.**
- We have updated  our paper to emphasis on novelty and comparison with standard approaches - see section 4.2-. In Table 3, we compare our importance score -Eq. 2- with respect to the classically used L1 score. This comparison is performed with different pruning strategies: Local, Global and a newly added baseline, “Global (activation)” which prunes the neurons according to the magnitude of their activations-. **Our importance score outperforms all the other considered approaches.**

**3- Comparison with knowledge distillation**
- Knowledge distillation methods require having multiple networks (teacher/student) - in other words, for each task, we need a student network-. Thus, the storage cost is higher than employing adapters, due to the fact that student networks are reasonably bigger than adapters. 
- We are not aware of any PET method that uses knowledge distillation. We would be grateful if the reviewer could provide a specific reference. 

**4- Adding more datasets**
- Thanks for the suggestion, we are aware that evaluating on more datasets is valuable. Nevertheless, our method has already been tested on ten different datasets. Evaluating further our method and baselines on 19 more datasets (VTAB) would be computationally and environmentally costly.

-----------------------------------------------------------------------------------------
We also provided a detailed description of the extra experiments, and changes in a general comment on top of all reviews;

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: Answers to reviewer #sXuV
comment: Thanks for the suggestions, and the points raised to further improve our paper. Below we address each question and minor questions in a point by point fashion.

**1- Motivation on the importance score**
- Magnitude pruning is widely adopted by well-cited works in the literature due its good performance [1,2,3]. In our work, we leverage on this, to propose our new importance score metric. The theoretical justification of magnitude pruning is far beyond the scope of this paper.
- In Appendix A.4 we show that weight magnitude alone is suboptimal in adapters as it does not take into account the effect of the up-sampling layer. We propose a score which accounts for this.


**Deeper adapters**
This is an excellent question. We leverage on the existing designs of Adapters [4, 6], which are all single-layer in order to maintain a low number of trainable parameters. It would be interesting to investigate other designs, but it is out of the scope of this work.


**2- Other comparison metrics?**
- In the literature [4, 6], efficient finetuning refers to the number of parameters required to obtain a certain performance level on a given task. The metric suggested does not correspond to our objective.
- We target efficient fine-tuning of ViTs on small-medium datasets, which is a real world scenario. Our advantage with respect to full model fine-tuning is storage. Given $T$ downstream tasks, fine-tuning a ViT model (\~27M parameters) requires storing all the $T$ models, whereas using TINA we only need to store adapters (~0.5M), which have a really small size with respect to the full ViT model (about 54 times less storage cost).


-----------------------------------------------------------------------------
We also provided a detailed description of the extra experiments, and changes in a general comment on top of all reviews;


**References**
- [1] Franckle et al. “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”, ICLR2019.
- [2] Pavlo Molchanov et al. “Pruning Convolutional Neural Networks for Resource Efficient Inference”, ICLR2017.
- [3] Renda et al, “Comparing Rewinding and Fine-tuning in Neural Network Pruning”, ICLR2020.
- [4] Houlsby et al. “Parameter-Efficient Transfer Learning for NLP”, ICML2019.
- [5] Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization.
- [6] Mahabadi et al. “Compacter: Efficient Low-Rank Hypercomplex Adapter Layers”, Neurips2021.

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: Answers to reviewer #fTqj [Part 2]
comment: **d. Hyperparameter $\rho$**

- We do not have any $\beta$ parameter in Algorithm 1. We have a hyper-parameter $\rho$, which represents the amount of neurons to be removed. In all our experiments $\rho$ is set to 50\% (Implementation details, section 4). -Smaller values for $\rho$ require more training cycles $C$ to reach the target compression rate $\sigma_{target}$. On the other hand, higher values of $\rho$ require less training cycles, but it hurts the performance as we remove too many parameters after each cycle-.
    
    **Impact of $\rho$**
    We added a section in Appendix 3.1, to study the impact of $\rho$.

**e. Typos correction**
Thanks for pointing out these typos. We have corrected them accordingly.

-----------------------------------------------------------------
**Performance of TINA:**

**a. Compared to vanilla adapters**
- Compared to vanilla adapters, our gain is about 1.26% with 1.37M trainable parameters on DomainNet, and **a gain of 1.7% with 50% less parameters on the Multi-task dataset.**
- It is worth mentioning that as we reduce the adapter size, vanilla adapters performance drops significantly with respect to TINA with a gap of 10% at $\sigma=512$ (see Table 3), and Figures 3, 5. 
- We added a new baseline, where we consider very small adapters with size $n_i=1$ (see Table 1, and 2). In these experiments **TINA outperforms vanilla adapters ($n_i=1$), with a gain of 2.3% on DomainNet, and more than 3% on the Multi-task benchmark.**

**Compared to full finetuning**
- We added a new experiment with smaller $\sigma$. We show in Figures 3 and 5 that **starting from smaller sizes of $\sigma$ -higher number of trainable parameters-, we outperform full fine-tuning with fewer parameters on VGG-Flowers, and CIFAR-100.**
- In the reported experiments, our objective was to have trainable parameters in the range of 1% - 5%, as done in the literature of PETs [1, 2]. Within this range, we agree that TINA does not outperform full finetuning in terms of accuracy, but it outperforms PET methods in the considered trainable parameters range. 
- We clarified this point in the paper (see abstract).


**b. Training cycles $C$ VS Performance**
- We do not agree with the reviewer’s analysis of the experiments. The objective of this work is to get adapters with a target size while maintaining good performance. Thus, stopping at early cycles would not achieve this constraint. Furthermore, training directly smaller adapters leads to lower performance.
- The trade-off between trainable parameters and performance is well known in the literature of PETs [1, 2]. It is worth mentioning that by training TINA for $C$ cycles, to reach the target size, **it achieves better performance than vanilla adapters having the same size** (see Table 1, 2 and 3), and Figures 3, and 5. 



**c. With respect to $\sigma$**

- For TINA baseline, as shown in Table 3, the performance is approximately stable. On the other hand, in the case of vanilla adapters (aka, baseline) the trend is really clear: the performance drops significantly as we increase $\sigma$. TINA at $\sigma =512$ has a gain of 10% over vanilla adapters.
- We also provided a detailed description of the extra experiments, and changes in a general comment on top of all reviews; 
-----------------------------------------------------------------------------
We also provided a detailed description of the extra experiments, and changes in a general comment on top of all reviews;


**References**
- [1] Houlsby et al. “Parameter-Efficient Transfer Learning for NLP”, ICML2019.
- [2] Mahabadi et al. “Compacter: Efficient Low-Rank Hypercomplex Adapter Layers”, Neurips2021.
- [3] Structured Pruning of Deep Convolutional Neural Networks, NIPS2017

From: ICLR.cc/2023/Conference/Paper1114/Authors
title: Answers to reviewer #fTqj [Part 1]
comment: Thanks for the suggestions, and the points raised to further improve our paper. Below we address each question and minor questions in a point by point fashion.

**Comparison with literature**

Thanks for pointing out the new PETs methods for ViTs (PVT, and AdaptFormer), and their interesting results.

1 - Visual Prompt Tuning (VPT) [1]:
- VPT is different from TINA. VPT emphasises on adjusting the embedding space of the transformer, by **injecting new tokens (i.e. prompts)** which will be fine-tuned for the downstream task. PVT cannot scale-up to a higher number of trainable parameters. Prompting is hindered by the dimensionality of tokens, tokens should have the same dimension as the transformer layer.
- We have included the new baseline PVT (with 10, and 100 tokens).
- **TINA outperforms PVT with fewer parameters**. Furthermore, a different number of prompts $p$ is employed for each dataset to achieve the best performance ($p$ is considered as a hyper-parameter). The optimal $p$ is found with hyper-parameter optimization strategies. It is worth mentioning that we did not fine-tune the hyperparameters specifically for every experiment and dataset.
- We updated related work (section 2), and the main results (section 4.1), accordingly.

2 - AdaptFormer[2]:
- **AdaptFormer has a fixed hidden dimension** for each adapter in the transformer. In our study, we delve deeper into the effect of the added adapters for tasks, as TINA self-adjusts the adapter’s size.
- We added 2 variants of _AdaptFormer_ (with a hidden size of 64, and 256). We show in Table 1 that **TINA outperforms AdaptFormer while using fewer parameters**.
- AdaptFormer investigated the optimal _middle dimension_ $\tilde{d}$ (as indicated in [2]) for Adapters (in [2] Table 3, and Table 9). TINA has the advantage over AdaptFormer that **the middle dimension for each adapter per task is found dynamically, and even removes the entire adapter if necessary**. Thus, we reduce the latency at the inference time, compared to AdaptFormer, where adapters are injected into each layer. 
- We show in our paper -see Table 7- that the size of the hidden dimension $n_i$ of adapters per layer has an effect on the performance.

-----------------------------------------------------------------

**Clarifying the text**

**a.** What is the ceiling function in Section 3.2? 
- A real number's ceiling function is the smallest integer number larger than or equal to the given number.
- The function making use of the ceiling function represents the number of training cycles $C$ to reach the target compression rate $\sigma^{target}$ -section 3.2-.

**b. Usage of Eq. 2**
- Eq. 2 represents our importance score for selecting the neurons to be removed, it corresponds to line 4 in Algorithm 1. This is **a novel score**, we introduced to select neurons taking also into consideration the effect of the up-sampling layer in the adapter.
- We have introduced $\mathcal{I}_{0}$ -see section 4.2- (which is the importance score without scaling), to show the effect of scaling with respect to the sum of weights $n_i + m_i$.
We have edited the paper, in order to make it more clear (see section 3.3).

**c. Difference between neurons magnitude pruning, and magnitude of activations pruning**

**From a methodological perspective:** The difference between the two methods is the following:
* Magnitude pruning means removing neurons with respect to the norm (i.e L1, or L2), by removing the neurons which have lower magnitude -see section 3.3-.
* Pruning with respect to magnitude of activation means ranking neurons according to their activations values, and removing the neurons which have the lowest activation magnitude.

  
**From a quantitative perspective:** We observe in our new experiments (see Table 3) that ranking neurons with respect to their activations is suboptimal: activation-based scoring depends on the input, and consequently, statistics are gathered at the batch level. Gathering statistics at full dataset level is typically infeasible for high computational complexity (see section 3.3).
    
**Reducing number of parameters VS removing neurons**

Removing neurons is different from simply reducing the number of parameters. Pruning can be done in two different ways [3]:
* Removing a neuron corresponds to removing an entire row of $W_{i}^{down}$ -see Figure 2-. The corresponding column in $W_{i}^{up}$ can then be dropped.
* Removing parameters means that any parameter can be removed, in an unstructured way.

We have updated section 3.2 and Table 3 accordingly, to emphasise more on this point.

From: ICLR.cc/2023/Conference/Paper1114/Reviewer_fTqj
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a score function that can reduce the tunable parameters of adapter for ViT-based parameter-efficient tuning.
strength_and_weaknesses: Strength
1.	The motivation for further reducing the parameters of adapter is good.
2.	Extensive experiments are conducted on 10 datasets.
3.	The proposed TINA performs better than adapter with fewer trainable parameters.
Weakness
1.	Some important literatures [1, 2] working on PET in the vision domain are ignored. How do you address the second weakness of adapter? Your method seems more complex than adapter.
2.	Writing is not clear and many grammar mistakes and typos.
a.	What is the ceiling function in Section 3.2?
b.	Where do you use equation 2?
c.	What is the difference between the magnitude of parameters and the magnitude of activations? Is not reducing parameter equal to selecting neurons (i.e., activations) as shown in the caption of Table 3? 
d.	How to you set the value for \beta in Algorithm 1?
Grammar mistakes Typos:
a.	we propose as scoring function …
b.	Publication venue of reference “Towards a unified view of parameter-efficient transfer learning”
c.	…computation efficient, …, at the batch or dataset level.
3.	Performance is not significant and cannot justify the model design.
a.	Compared with adapter the improvements are within 1%. While comparing with full-tuning, it significantly underperforms full-tuning. Note that [1,2] can outperform full-tuning. Hence, the claimed “outperform” in the abstract is exaggerated in terms of accuracy. 
b.	The number of cycles seems useless as increasing the cycle will decrease the performance (see Tables 1 and 2) while just reduce a very small number of parameters.
c.	According to Table 3, larger sigma leads to smaller number of tunable parameters. Why here the performance is higher? It is not consistent with the claim in the abstract "adapters perform poorly when the dimension of adapters is small"

[1] Jia et al. Visual prompt tuning, ECCV-2022
[2] Chen et al. Adapting vision transformers for scalable visual recognition, NIPS-2022
clarity,_quality,_novelty_and_reproducibility: The proposed score function can be a novel method but it is not clearly introduced and supported with significant results.
summary_of_the_review: This work can be a potential work but is not ready for publication in its current form. More important references need to be considered and the writing should be improved.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1114/Reviewer_sXuV
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper focusses on parameter-efficient training for adapting Vision Transformers to new tasks. The key argument is that adapters typically need many parameters to achieve good performance and therefore the paper proposes TINA,  a method for training tiny adapters. The method works by iteratively reducing the dimension of the adapters by identifying the neurons that can be dropped in every adapter - similar to neural pruning approaches. This allows for more efficient parameter allocation. The results show that the method achieves similar performance to existing approaches on common datasets such as Cifar and SVHN, but with much fewer finetuned parameters.


strength_and_weaknesses: Strengths

1. With the advent of large models, efficient finetuning for downstream tasks is becoming an increasingly important topic. The proposed method represents a simple and effective approach for achieving this finetuning.

2. The proposed method is able to achieve similar performance to existing methods (vanilla adapters, LoRa, Compactor) with far fewer parameters. When compared to finetuning attention blocks, for example, the approach achieves similar performance with only one-tenth of the parameters. More generally, TINA is able to match or exceed the performance of all other existing methods on Cifar, Flowers and SVHN.

Weaknesses

1. Using the weight magnitude for the importance score doesn’t have a good theoretical explanation. The description in appendix A.4 provides some insights but isn’t entirely convincing. What happens when there is more than 1 layer in the adapter? And, how does it compare to using the gradient magnitude?

2. There is no description of the real-world performance gains of the proposed approach. While the number of finetuned parameters is significantly reduced, it would still require backpropagating through the whole network. Perhaps a better comparison metric would be the total FLOPs used for finetuning.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly with sufficient technical and implementation details for reproducing the results. While the proposed pruning-based approach is not really new in the sense that it has been extensively studied for network pruning, it is as far as I can tell the first time it’s been used for training adapter modules.
summary_of_the_review: Overall, the paper achieves some good results for parameter-efficient finetuning of ViTs. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1114/Reviewer_QrJw
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes parameter efficient tuning for vision transformers. The method starts with large adapters and gradually iteratively reduces their size with pruning. The paper also introduces a scoring function that weights the importance of neurons. The method outperforms existing approaches with higher accuracy and lower parameters.
strength_and_weaknesses: Strength
---
- The method is simple and easily applicable to smaller datasets.

- A pruning technique is applied for parameter tuning.


Weakness
---
- The method needs several rounds of pre-training. With these additional computational budgets, it may not be feasible for a large dataset.

- The novelty is somewhat limited. At the same time, it is not clear how the score function is useful. Specifically, the method should be compared with some state-of-the-art pruning methods. 

- Experiments should be more thorough. For example, knowledge distillation baseline methods can be used as baselines. 

- It is encouraged to try more datasets such as VTAB.




clarity,_quality,_novelty_and_reproducibility: The paper is easy to read, but the novelty is somewhat limited as pruning is well-known and the scoring function is not well evaluated.
summary_of_the_review: Overall, novelty is somewhat limited and experiments can be further improved to thoroughly evaluate the proposed method.

-- POST REBUTTAL
I appreciate the authors for the response. My concern about the baselines is not fully addressed. The main concern is that this method may not be scalable. And it is not clear how it can be applied to a large-scale dataset. Therefore, I keep my original score.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1114/Reviewer_qoFV
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes TINA, an adaptation mechanism that starts from large adapter layers and interactively reduce their capacity by target domain. New dense layers are inject after each block in a VIT-family backbone (after every MLP and every Multi-head Self-attention block). In practice, the authors adopt a pretrained SWIN-T as their ablations baseline.
The pretrained backbone is kept frozen while the new layers are trained jointly with the classification head. The adapter composition has a bottleneck structure (m neurons are mapped down to n and back to m). TINA focus on dynamically finding the number of neurons used in the bottleneck (n) in a downstream task adaptation.
The proposed pruning of n is based on a score function proposed targets eliminating neurons corresponding to zero valued columns/rows pairs on the adapter's bottleneck structure.
strength_and_weaknesses: The strongest point of the paper is its core idea of pruning the adaptation mechanism per target task. The center pieces of the proposal (iterative pruning and score function) seams to be easy to replicate from the description presented.

Its weaknesses are due to lack of clarity on the experimental settings and also in stating the novelty of methodology proposed. Next I mention some suggestions for improvement.

clarity,_quality,_novelty_and_reproducibility: The proposed heuristic for pruning adapters is clear, thus I consider it can be reproduced. But description of the ablations performed and the clarification of the comparisons need  improvement.

Here are some suggestions.

- It is not clear if the score function adopted was previous proposed in one of the cited papers or if is commonly used on pruning literature. This point must be clarified in order to certify its novelty or previous use. If novel, should be further compared to pre-existent pruning methodologies. 
- On the ablations, it is not clear if the baseline used for (non-pruned) adapters corresponds to the same architecture later adopted in TINA. My suggestion is that tables should have a row with a clear indication of the starting baseline results (prior to prune). For instance, in the first paragraph of Section 4.2, it is said that adapters are tested with sigma=32, but it is not clear which line of Table 1 corresponds to this setting. I am guessing it is the line marked as TINA (sigma=32), used as baseline for TINA before pruning. If so, naming it as TUNA is misleading to the reader, as this line has the strongest results, but not pruning.

In my perception, it is not surprising (nor a new result) that adapters with proportional parameter distribution perform better, as it is not surprising that the last layers, which have higher dimensionality, are more task-specific, and consequently require more adaptation. The authors pointed that this contrasts  the adapter's original proposition adopting a unique size. My question is how much is affected by the use of SWINT and not VIT in the ablations presented, as SWINT doubles the channel layers in between some blocks groups, while the original VIT structure do have uniform number of channels through its backbone. In my understanding, there is no reason for testing uniform number of channels as the increasing number of channels is imposed by the authors choice of SWINT. Thus, my suggestion is that  adapters (not-pruned) with channels proportional to those on its corresponding block to be used as baseline, and clearly pointed in the presented tables. Ideally, the paper should show this baseline at different choices of sigma to sustain the motivation claim that small adapters trained from scratch underperform the proposed mechanism. 

Figure 4 is hard to interpret. I suggest a normalized per layer plot, showing the ratio of total neurons preserved and removed. 

Results are not convincing as TINA underperforms when compared to versions finetuned (tables 2 and 4) and adopting adapters (Table 4).
summary_of_the_review: The core idea is clear, but the comparisons and presentations need improvement. Thus, I am not convinced of its novelty nor on the impact of the presented ablations.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold