Paper: Parameter-Efficient Long-Tailed Recognition (ICLR-2024)

Reviewer comment (Experiments):
Many previous works adopt ResNet as backbone. However, no experment in this paper adopts ResNet as well, which may raise questions about its generalisability.

Author response:
**Concern #2: (Weakness #2): Adopt ResNet as backbone.**

**Response:** We have also considered this problem. However, there is no PEFT method specifically designed for ResNet, making it challenging to integrate our method with ResNet. Despite this constraint, we have explored some straightforward strategies, including 1) incorporating a scaling and shifting (SSF) [6] module after the backbone, and 2) fine-tuning solely the bias terms of ResNet. The results are presented below. In comparison to zero-shot CLIP and previous methods (detailed in Tables 1&2 in the main paper), our method achieves significantly superior performance with lower computational costs.

- ImageNet-LT

  |                          | Backbone  | Learnable Params. | #Epochs | Overall  | Many     | Medium   | Few      |
  | ------------------------ | --------- | ----------------- | ------- | -------- | -------- | -------- | -------- |
  | Zero-shot CLIP           | ResNet-50 | -                 | -       | 57.6     | 58.6     | 56.9     | 56.9     |
  | PEL w/ SSF               | ResNet-50 | 0.002M            | 10      | 66.9     | 72.0     | 66.5     | 54.1     |
  | PEL w/ bias tuning       | ResNet-50 | 0.034M            | 10      | 67.8     | 72.0     | 67.3     | 57.6     |
  | PEL w/ bias tuning & SSF | ResNet-50 | 0.036M            | 10      | **68.3** | **72.5** | **67.8** | **58.2** |

- Places-LT

  |                          | Backbone  | Learnable Params. | #Epochs | Overall  | Many     | Medium   | Few      |
  | ------------------------ | --------- | ----------------- | ------- | -------- | -------- | -------- | -------- |
  | Zero-shot CLIP           | ResNet-50 | -                 | -       | 35.2     | 33.1     | 34.6     | 40.4     |
  | PEL w/ SSF               | ResNet-50 | 0.002M            | 10      | 46.7     | 47.5     | 48.7     | 40.7     |
  | PEL w/ bias tuning       | ResNet-50 | 0.034M            | 10      | 47.9     | **48.2** | 49.8     | 42.9     |
  | PEL w/ bias tuning & SSF | ResNet-50 | 0.036M            | 10      | **48.1** | 48.1     | **50.0** | **43.9** |



[1] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models.

[2] Visual Prompt Tuning.

[3] Parameter-Efficient Transfer Learning for NLP.

[4] LoRA: Low-rank adaptation of large language models

[5] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition.

[6] Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning.

Reviewer comment (Experiments):
Why only use image cropping for test-time ensembling? Have you tried other data augmentation methods?

Author response:
**Concern #3 (Question #1): Why only use image cropping for test-time ensembling? Have you tried other data augmentation methods?**

**Response:** We utilize image cropping tailored for Vision Transformer (ViT), which also uses image cropping to obtain $n\times n$ image patches. ViT carries the inherent risk of pattern separation. In response, we intuitively generate diverse cropped patches to remedy this issue. The detailed explanations are given in Appendix C. The results in Figure 12 demonstrate the effectiveness of our method: 1) When our cropped patches align with those of ViT, pattern separation persists, and the performance improvement is limited; 2) Conversely, when our cropped patches differ from ViT, the pattern separation can be alleviated, and the performance is enhanced.

Aside from TTE, we explore additional augmentation techniques such as TTE + Flipping and Random Augmentations [6] for multiple times. The results below demonstrate that TTE is more effective than other methods.

| Augmentation methods | Augmentations times | Overall  | Many     | Medium   | Few      |
| -------------------- | ------------------- | -------- | -------- | -------- | -------- |
| TTE (Ours)           | 5                   | **78.3** | **81.3** | **77.4** | **73.4** |
| TTE + Flipping       | 10                  | **78.3** | **81.3** | 77.3     | 73.3     |
| Random Augmentation  | 5                   | 76.7     | 79.7     | 75.5     | 71.9     |
| Random Augmentation  | 10                  | 77.3     | 80.3     | 76.3     | 72.2     |
| Random Augmentation  | 15                  | 77.7     | 80.8     | 76.6     | 72.3     |
| Random Augmentation  | 20                  | 77.8     | 80.9     | 76.7     | 72.7     |

Reviewer comment (Reproducibility):
What kind of textual prompts are used for semantic-aware initialization?

Author response:
**Concern #4 (Question #2): What kind of textual prompts are used for semantic-aware initialization?**

**Response:** We employ the most widely used textual prompt, "a photo of a [CLASS]", where [CLASS] can be replaced by the class names. Additionally, we compare different prompting methods, including 1) the original class name ("[CLASS]") and 2) prompt ensembling [7] which applies different templates to class names. The experiments are conducted on ImageNet-LT. The results below show that these prompts have similar performance, and using "a photo of a [CLASS]" is adequate for generalization.

Moreover, Reviewer BCXb has proposed an intriguing situation, i.e., when the class names are OOD or unseen for CLIP. In response to this, we explore an alternative approach by incorporating class descriptions (which can be crafted manually or generated using large language models). The results show that using class descriptions can also enhance the performance compared to random initialization.

| Prompting methods                    | Overall  | Many     | Medium   | Few      |
| ------------------------------------ | -------- | -------- | -------- | -------- |
| Random initialization                | 76.1     | 80.8     | 75.9     | 63.2     |
| "[CLASS]"                            | 78.2     | **81.4** | 77.3     | 72.3     |
| "a photo of a [CLASS]"               | **78.3** | 81.3     | **77.4** | **73.4** |
| Prompt ensembling                    | **78.3** | 81.3     | **77.4** | 73.3     |
| Class descriptions (w/o class names) | 77.4     | 81.3     | 76.9     | 68.2     |



[6] Deep residual learning for image recognition.

[7] Learning Transferable Visual Models From Natural Language Supervision.

Reviewer comment (Novelty):
The related work seems insufficient. The authors should include references to existing studies concerning parameter-efficient fine-tuning approaches. Although some PEFT methods are presented in Section 3.3, these methods are listed without detailed explanations.

Author response:
This is a good suggestion. We will add more explanations as follows:

Parameter-efficient fine-tuning (PEFT) aims to adapt the pre-trained models to downstream tasks by learning a small number of parameters. Over the past few years, various PEFT approaches have been proposed for both computer vision (CV) and natural language processing (NLP) tasks [1]. These approaches can be broadly categorized into three groups [1]: 1) Prompt Tuning, such as VPT-shallow [3] and VPT-deep [3], is inspired by prompting strategies in NLP. It prepends learnable patches to images or sentences to serve as the contexts for downstream tasks; 2) Adapter Tuning, such as Adapter [4] and AdaptFormer [6], incorporates lightweight modules sequentially or parallelly to the pre-trained models; 3) Parameter Tuning, like BitFit [2] and LoRA [5], selectively modifies specific parts of the pre-trained models, such as the weight part or the bias part. Recent work [7] offers a unified view of different PEFT methods and proposes that they are intrinsically similar. Nevertheless, different PEFT modules may excel in distinct tasks due to variations in their design approaches and integration positions.

Thanks for the thoughtful comments. We will update the related works in the next version.

Reviewer comment (Novelty):
The technical contribution of this paper is not clear. The choice of AdaptFormer, based primarily on its state-of-the-art results, appears unrelated to long-tailed learning and should not be considered a contribution to the field. Besides, the initialization from CLIP text embeddings can not be regarded as significant technical contribution. Logit-adjusted (LA) loss and test-time ensembling also has been proposed.

Author response:
We summarize the contribution of this work as follows.

PEL is a general framework, with all its components scalable and efficient. Its design allows updates in alignment with the development of the related field. What is more, we aim to **reveal fundamental insights** and **inspire future research** regarding long-tailed learning with pre-trained models.

1) Full Fine-tuning fails in long-tailed learning, and PEFT emerges as a superior alternative. This contradicts common sense on balanced datasets, where PEFT typically strives to match the performance of full fine-tuning [1-5].
2) We discover that most elaborately designed losses fail when applied to CLIP, except Logit-Adjusted loss (explained in Appendix B). This discovery motivates future research to design more suitable losses.
3) We point out the importance and advantages of classifier initialization, particularly for tail classes. This also inspires more advanced research in the future.
4) We first propose to employ TTE to remedy the inherent limitations of Vision Transformer. As a remedial measure, TTE is intuitive and effective (explained in Appendix C).

To the best of our knowledge, this is the first work that systematically studies the parameter-efficient adaptation of pre-trained models on long-tailed tasks. We believe our findings and efforts can contribute to the long-tailed learning community.

Reviewer comment (Experiments):
In Table 4, the authors seem only conduct experiments with existing PEL methods, the performance of the proposed method is missing.

Author response:
There may be some misunderstandings. PEL is a framework that can adopt any existing PEFT methods. Table 4 is an ablation study on different PEFT methods. The results show that PEL with AdaptFormer achieves superior performance currently. If some new PEFT methods are proposed in the future, PEL can seamlessly incorporate them to enhance its performance.

Moreover, we apologize for the confusion in Table 4. It should be clarified that zero-shot CLIP, full fine-tuning, and classifier fine-tuning are not PEFT methods. The remaining methods are PEFT methods that can be integrated into PEL. We will update Table 4 in the next version.

Reviewer comment (Experiments):
The authors should compare the computational cost with PEL methods in Table 4, instead of only comparing with non-PEL methods.

Author response:
Thank you for the suggestion. We compute the time cost per epoch for different parameter-efficient fine-tuning methods. The experiments are conducted using a single NVIDIA A100 GPU. The results indicate that the time cost for each PEFT method is very close.

| Methods     | ImageNet-LT | Places-LT  |
| ----------- | ----------- | ---------- |
| LN tuning   | 2 min 49 s  | 1 min 30 s |
| BitFit      | 2 min 51 s  | 1 min 33 s |
| VPT-shallow | 2 min 46 s  | 1 min 31 s |
| VPT-deep    | 2 min 56 s  | 1 min 37 s |
| Adapter     | 2 min 49 s  | 1 min 33 s |
| LoRA        | 2 min 58 s  | 1 min 31 s |
| AdaptFormer | 2 min 58 s  | 1 min 38 s |

Reviewer comment (Experiments):
The paper should broaden its scope by comparing various long-tailed learning strategies, rather than focusing on results from Logit adjustment or cosine classifer. The current benchmark could undermine the generalizability and confidence of the results.

Author response:
We have compared different classifiers and losses in Tables 9 and 10 in Appendix A. Moreover, we give an analysis of Logit Adjusted loss in Appendix B. Please refer to the appendix for detailed explanations.

To enhance clarity, we will add a concise outline in the paper to summarize the experiments conducted in the appendix.

Reviewer comment (Theory):
The paper lacks intuitive explanations. The authors claim that their method can prevent overfitting and decrease training epochs without extra data, but how is PEFT helping? More empirical or theoretical explanations are suggested.

Author response:
As illustrated in Figure 4 in the main paper, the utilization of PEFT enhances the capability to extract more separable representations. This is because the PEFT module is integrated into the backbone, and enhances the feature extraction process of the pre-trained model. In this way, it is more effective than solely fine-tuning the classifier.

Moreover, PEFT freezes the pre-trained model while introducing a small number of learnable parameters for adaptation. This approach ensures the retention of the generalization ability of the pre-trained model. In contrast to full fine-tuning, PEFT can mitigate overfitting, as analyzed in Table 8 in Appendix A.

Reviewer comment (Experiments):
Why are the training epochs chosen as 10/20? How would the results differ if the training period were extended?

Author response:
We set the training epochs to 10 for ImageNet-LT, Places-LT, and CIFAR-100-LT. This is because PEL can converge rapidly by learning a small number of parameters, along with a classifier initialization. 

For iNaturalist 2018, we train 20 epochs considering it has much more data. We have presented an analysis of different training epochs in Table 7 in Appendix A. The results show that extending the training epochs can result in improved performance. However, this improvement comes at the expense of increased computational overhead.

We also conduct experiments across different epochs on ImageNet-LT and Places-LT, and the results are presented below. Increasing the training epochs does not yield significant improvements. In practice, the selection of the optimal training epoch depends on the data scale and the task complexity. Generally, 10-20 epochs are appropriate for most cases.

- Results of PEL on ImageNet-LT by training different epochs.

  | #Epochs | Overall  | Many | Medium   | Few      |
  | ------- | -------- | ---- | -------- | -------- |
  | 5       | 77.5     | 80.9 | 77.2     | 69.0     |
  | 10      | **78.3** | 81.3 | **77.4** | **73.4** |
  | 20      | **78.3** | 81.8 | 77.1     | 72.8     |
  | 30      | 78.0     | 82.2 | 76.6     | 71.1     |

- Results of PEL on Places-LT by training different epochs.

  | #Epochs | Overall  | Many     | Medium   | Few      |
  | ------- | -------- | -------- | -------- | -------- |
  | 5       | 51.3     | **51.7** | 53.0     | 46.8     |
  | 10      | **52.2** | **51.7** | **53.1** | 50.9     |
  | 20      | 51.8     | 51.3     | 52.3     | **51.6** |
  | 30      | 51.3     | 51.4     | 51.5     | 50.5     |

Reviewer comment (Experiments):
In Table 3, PEL incurs larger (4x) computation cost than LPT. Besides, LPT is not compared in Table 1. Will PEL degenerate on more real-world long-tailed datasets? It is suggested to add more datasets to enhance the confidence of the results.

Author response:
We have discussed this in Paragraph *Results on iNaturalist 2018* in Section 4.2: Although LPT uses fewer learnable parameters, we can reduce the parameters of PEL to reach a comparable quantity (i.e., reduce the bottleneck dimension r to 64, more details are given in Figure 6). In this case, PEL achieves an accuracy of 77.7% (without TTE) / 79.0% (with TTE), which still outperforms LPT. In fact, due to the large number of classes of iNaturalist 2018, the classifier already contains 6.25M parameters. Therefore, the parameter quantity of PEL does not lead to too much cost.

Reviewer comment (Novelty):
None of the 4 components in PEL is new, and the big featuring of existing techniques seems a bit ad-hoc, which raises novelty concerns.

Author response:
**Concern #1 (Weakness #1): The novelty of PEL.**

**Response:** Since it is the first work that systematically studies the parameter-efficient adaptation of pre-trained models on long-tailed tasks, we aim to **reveal fundamental insights** and **inspire future research**.

1) Full Fine-tuning fails in long-tailed learning, and PEFT emerges as a superior alternative. This contradicts common sense on balanced datasets, where PEFT typically strives to match the performance of full fine-tuning [1-5].
2) We discover that most elaborately designed losses fail when applied to CLIP, except Logit-Adjusted loss (explained in Appendix B). This discovery motivates future research to design more suitable losses.
3) We point out the importance and advantages of classifier initialization, particularly for tail classes. This also inspires more advanced research in the future.
4) We first propose to employ TTE to remedy the inherent limitations of Vision Transformer. As a remedial measure, TTE is intuitive and effective (explained in Appendix C).

Furthermore, PEL is a general framework, with all of the components efficient and scalable. We believe our findings and efforts can contribute to the long-tailed learning community.



[1] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models.

[2] Visual Prompt Tuning.

[3] Parameter-Efficient Transfer Learning for NLP.

[4] LoRA: Low-rank adaptation of large language models

[5] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition.

Reviewer comment (Experiments):
Each of the 4 components is ablated separately --- Tables 1-3 validate the need of test time ensembling, while Figs 9-10 show that both PEFT and semantic-aware initialization are helpful. Tables 9-10 respectively show that the adopted classifier and loss function are the right choice. However, there's no clue about which of the 4 existing techniques play a key role, and which is more like icing on the cake. To tell which components are more important, a systematic ablation is required but is missing in the current paper, e.g., baseline vs baseline+component 1 vs baseline+components 1&2 ... vs baseline+components 1&2&3&4.

Author response:
**Concern #2 (Weakness #2): A systematic ablation of components.**

**Response:** Thanks for your valuable comments. We ablate the components in PEL separately, including 1) the **PEFT** module, 2) the Logit-Adjusted (**LA**) loss, 3) Semantic-Aware Initialization (abbreviated as **SAI**), and 4) Test-Time Ensembling (**TTE**). The results are reported below. 

The results demonstrate the effectiveness of each component. Specifically, 1) PEFT enhances performance on both head and tail classes; 2) without the LA loss, predictions tend to be biased to head classes; 3) Semantic-Aware Initialization (SAI) consistently improves the generalization, particularly on tail classes; 4) Test-Time Ensembling further boost the performance across both head and tail classes.

- Ablation study on ImageNet-LT. The baseline involves learning a classifier using CE loss.

  | PEFT         | LA           | SAI          | TTE          | Overall  | Many     | Medium   | Few      |
  | ------------ | ------------ | ------------ | ------------ | -------- | -------- | -------- | -------- |
  |              |              |              |              | 60.9     | 82.6     | 56.7     | 13.8     |
  | $\checkmark$ |              |              |              | 68.9     | **84.7** | 66.3     | 33.7     |
  | $\checkmark$ | $\checkmark$ |              |              | 74.9     | 79.7     | 74.7     | 61.7     |
  | $\checkmark$ | $\checkmark$ | $\checkmark$ |              | 77.0     | 80.2     | 76.1     | 71.5     |
  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | **78.3** | 81.3     | **77.4** | **73.4** |

- Ablation study on Places-LT. The baseline involves learning a classifier using CE loss.

  | PEFT         | LA           | SAI          | TTE          | Overall  | Many     | Medium   | Few      |
  | ------------ | ------------ | ------------ | ------------ | -------- | -------- | -------- | -------- |
  |              |              |              |              | 34.3     | 53.6     | 30.5     | 7.5      |
  | $\checkmark$ |              |              |              | 38.9     | **55.6** | 35.3     | 16.4     |
  | $\checkmark$ | $\checkmark$ |              |              | 48.7     | 50.6     | 50.8     | 40.5     |
  | $\checkmark$ | $\checkmark$ | $\checkmark$ |              | 51.5     | 51.3     | 52.2     | 50.5     |
  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | **52.2** | 51.7     | **53.1** | **50.9** |

Reviewer comment (Experiments):
Semantic-aware initialization is claimed to accelerate convergence by initializing the classifier weights with text prompt features. What if the class is OOD/unseen for CLIP? Will such class prompt features still be a good initialization?

Author response:
**Concern #3 (Question #1): What if the class is OOD/unseen for CLIP? Will such class prompt features still be a good initialization?** 

**Response:** We posit that CLIP has seen sufficient language corpus, considering its pre-training on web-scale datasets. However, it is noteworthy that CLIP may fail to recognize specific class names. This probably stems from its limited vocabulary size (CLIP contains approximately 49K of vocabulary) or encountering uncommon or novel concepts. In this case, the initialization may regress to random initialization.

One solution is to design some descriptions for this class, which can be crafted manually or generated using large language models. We follow [6] to generate the descriptions. Following are some examples:

| Class    | Descriptions                                                 |
| -------- | ------------------------------------------------------------ |
| Hen      | a chicken; red, brown, or white feathers; a small body; a small head; a beak; two legs; two wings; a tail |
| Trombone | a brass musical instrument; long, coiled tubing; a mouthpiece; a slide; valves or keys; a bell |

We conduct experiments on ImageNet-LT by assuming that the class names are unseen. We solely use the descriptions for initialization. The results shown below demonstrate that the descriptions can enhance generalizations, particularly for tail classes.

|                       | Overall | Many | Medium | Few  |
| --------------------- | ------- | ---- | ------ | ---- |
| Random initialization | 76.1    | 80.8 | 75.9   | 63.2 |
| Class descriptions    | 77.4    | 81.3 | 76.9   | 68.2 |
| Class names           | 78.3    | 81.3 | 77.4   | 73.4 |

Reviewer comment (Evaluation):
In Table 10, it seems that different loss functions are all about the tradeoff between tail and head class performance. For example, the adopted LA loss, despite achieving the best performance on Median/Few splits, is suboptimal on the Many splits. Any comments?

Author response:
**Concern #4 (Question #2): Different loss functions are all about the tradeoff between tail and head class performance.** 

**Response:** The phenomenon you highlighted does indeed exist. In the context of long-tailed learning, there exists a competition for decision boundaries between head and tail classes. By optimizing re-balancing losses, the model tends to predict samples as tail classes, leading to a performance deterioration of head classes, which is called the *seesaw dilemma*.

Most recently, some works have proposed to address this issue through multi-objective optimization [7] [8]. Although the solution may not be fully mature, the topic remains intriguing and warrants exploration in the future.



[6] Visual classification via description from large language models.

[7] Long-Tailed Learning as Multi-Objective Optimization. In https://arxiv.org/abs/2310.20490.

[8] Pareto Deep Long-Tailed Recognition: A Conflict-Averse Solution. In https://openreview.net/forum?id=b66P1u0k15.