Paper: BLG: BALANCED LANGUAGE DISTRIBUTION AS GUIDANCE FOR ROBUST LONG-TAILED VISION CLASSIFICATION (ICLR-2024)

Reviewer comment (Novelty):
Lack of innovation. The approach in this paper provides a more balanced prototype for visual pre-training models to guide the learning of visual feature extractors and designs supervised comparative learning loss. However a similar approach has appeared in previous long-tail methods [1]. The differences in this paper are: 1. A more robust pre-training model, CLIP, is utilized. 2. A text-based prototype design approach is used to replace the target anchor. These innovations are more limited.

Author response:
Thanks for your kind suggestion and for providing related work to help refine our work.

1. Reply to #W1. 

This paper introduces a targeted contrastive learning loss (TSC) aimed at aligning class features with target features on the vertices of a regular simplex. We identify four key distinctions between our approach and TSC:

**(1) Learning Targets and Complexity:**

TSC employs SGD to learn targets, posing challenges in finding the optimal solution on a hypersphere [1,2]. This method imposes a demanding requirement that is difficult to fulfill perfectly. In contrast, our approach is more straightforward, avoiding complex assumptions and calculations to obtain balanced prototypes. Our prototypes, derived from textual features, carry rich semantic information, enhancing the discriminative power of image features. Unlike TSC, which constructs targets without class semantics, we leverage a more abstract and balanced text modality. **Therefore, our methods differ fundamentally, despite sharing a general motivation**. Additionally, TSC requires prior knowledge of the number of classes, whereas our method is more flexible.

**(2) Implementation Complexity:**

TSC faces the challenge of assigning class indices to each unlabeled anchor point, adding complexity to the method. In contrast, our approach constructs prototypes in a more natural and intuitive manner.


**(3) Incorporation of Components:**

TSC relies solely on supervised contrastive learning loss, while our method incorporates various components. TSC updates the model by minimizing the distance between visual features and assigned targets. In contrast, our method transforms balanced textual prototypes into a linear classifier, employing optimal transport for unsupervised alignment of visual features and prototypes. Furthermore, we introduce a supervised contrastive learning loss based on the transport matrix to enhance supervision signals and intra-class information. **Therefore, our contribution extends beyond a supervised contrastive learning loss. From the ablation study, our method can still perform best even without the proposed supervised contrastive loss.**


**(4) Modality and Multimodality:**

TSC operates as an image-only method, while our approach is multimodal. To assess TSC's impact, we integrated its method into our work and conducted a performance comparison. Indicating our proposed three losses as $L_{CE}$, $L_{OT}$ and $L_{SCT}$, we compare the performance replacing $L_{SCT}$ of $L_{TSC}$ based on ViT-B/16 and Places-LT dataset. As shown in the table below, **using textual prototypes to initialize anchor points and applying TSC in Phase B led to a significant performance drop, demonstrating that our method is not a mere substitution of anchor points in TSC with textual features.**

| Methods            | Many-Shot | Medum-Shot | Few-Shot | Overall |
| ------------------ | --------- | ---------- | -------- | ------- |
| OURS               | 49.8      | 52.6       | 54.2     | 51.9    |
| $L_{CE}$ + $L_{SCT}$ | 48.9 | 50.8   | 50.5 | 50.1    |
| $L_{CE}$ + $L_{OT}$ + $L_{SCT}$ | 48.9 | 51.5   | 53.8 | 50.5    |


In summary, these four aspects highlight the substantial differences between our method and TSC, reinforcing the unique contributions of our approach.

[1] https://arxiv.org/pdf/physics/0609231.pdf
[2] https://arxiv.org/abs/cond-mat/0506616

Reviewer comment (Theory):
In the unsupervised prototype-guided feature learning part, the choice of cosine similarity as the distance metric is not well justified compared to other possible metric methods such as minimum entropy.

Author response:
3. Reply to #W3.

 A cost matrix serves as a representation of the cost function in a discrete optimal transport problem, where goods or resources are distributed among a finite number of sources and destinations. This matrix is structured with a row for each source and a column for each destination, where each entry denotes the cost of transporting one unit of goods or resources from the corresponding source to the respective destination. In practical terms, various measurements capable of computing the distance between two distributions can function as a cost matrix. Examples include cosine similarity, Euclidean distance, or a matrix derived through learning in a specific manner (reverse OT problem).

It is worth noting that minimum entropy might be unsuitable as a measurement for the cost matrix in this context. For our purposes, we opt for cosine similarity due to its universality and superior overall performance compared to other distance metrics. The comparison results of cosine similarity and Euclidean distance, employed as a cost matrix based on Places-LT and ViT/B-16, are presented in the following table:

| Methods            | Many-Shot | Medum-Shot | Few-Shot | Overall |
| ------------------ | --------- | ---------- | -------- | ------- |
| Cosine Similarity               | 49.8      | 52.6       | 54.2     | 51.9    |
| Euclidean Distance | 49.0 | 52.1   | 54.5 | 51.7    |

Reviewer comment (Novelty):
For the modules of learnable classifier, unsupervised prototype-guided feature learning, and supervised contrastive loss, they seem to represent incremental improvement and not complementary approaches, so it is unclear why they should be used together to train models.

Author response:
Reply to #W4. 

**Our motivation stems from the observation that the output quality of CLIP's textual encoder remains robust even in the presence of imbalanced datasets.** We aim to leverage this resilience to guide the learning and enhancement of discriminative visual features, addressing CLIP's performance on imbalanced image classification tasks. In our practical implementation, after fine-tuning CLIP, we freeze the encoders and introduce an additional residual structure with a linear layer. This augmentation aims to refine biased visual features, making them as balanced and separable as their textual counterparts through the application of balanced sampling.

**Throughout the implementation process, we utilize textual features to initialize a learnable classifier.** Using cross-entropy loss, we optimize the classifier with refined visual features as input. The objective here is to ensure that the refined visual features maintain consistency with their corresponding textual features, ultimately contributing to the final classification. **Simultaneously, we treat the classifier parameters as prototypes for visual features.** In each input mini-batch, we employ optimal transport to align the two distributions in an unsupervised manner. This approach serves to avoid introducing additional prototype parameters while ensuring that the classifier parameters (prototypes) adapt closely to the current input data during the learning process. **This adaptation aims to achieve superior performance and a more compact classification surface.** Besides, experiment results also show the proposed prototype-guided matching loss helps the training convergence.

**Considering that the optimal transport matching process is instance-level and unsupervised, we then incorporate a supervised contrastive learning loss based on the transport plan.** This step introduces more class-level information and supervision signals into the learning process. 

**In summary, the learnable classifier, prototypes-guided biased visual feature refinement, and supervised contrastive loss are designed to be progressive and complementary components.** The learnable classifier focuses on classification, prototypes-guided feature learning closely connects the optimization of classifier parameters (prototypes) with input data, ensuring adaptability and the supervised contrastive loss introduces class information and supervision signals to compensate for the limitation of relying solely on unsupervised information in prototypes-guided feature learning.

Reviewer comment (Novelty):
The proposed method doesn't seem to have a specific design tailored for long-tail data. The approach of using textual features as guidance for better image features can be applied to situations with limited image feature quality for various reasons, such as long-tail, few-shot, noisy data, generated data, low-resolution data, and so forth.

Author response:
Thanks for your kind suggestion. We will try our method on more scenarios, such as few-shot learning and noisy data. Due to the rebuttal time limit, we may not give comprehensive results now.

Reviewer comment (Evaluation):
The experimental results show that the proposed method underperforms in “many” classes of LT dataset. Does this imply that the method is primarily effective for situations with the few-shot scenario (i.e., “medium” and “few” classes in long-tail datasets)?

Author response:
It involves a trade-off in performance among three different shot divisions in long-tail. Despite this, our method boasts several key advantages:

(1) **We view imbalanced recognition as a task primarily concerned with enhancing the classification performance of tail classes, minimizing the performance gap with head classes, and achieving superior overall classification performance.** While BALLAD and VL-LTR excel in overall performance by mainly improving head class results, they fall short in addressing the tail. For instance, when based on RN50 and additional texts for training, VL-LTR achieves 77.8% accuracy in the Many-shot in ImageNet-LT but only 50.8% accuracy in the Few-shot, resulting in a substantial 27.0% performance gap. In contrast, our method exhibits a significantly lower performance gap of 8.7%.

(2) **Our approach outperforms in medium-shot, few-shot, and overall performance, effectively narrowing the performance gap between these divisions and many-shot.** This aligns with our motivation to attain a more balanced visual feature distribution, resulting in a more balanced classification performance. Notably, we achieve a more substantial performance improvement in few-shot scenarios compared to previous methods. On ImageNet-LT, our method showcases a 10% increase in the few-shot category. Similarly, on Places-LT, the improvement exceeds 12%.

**Furthermore, we wish to draw a distinction between the few-shot division in long-tailed recognition and few-shot learning in the general sense.** The former serves as a means and perspective for performance evaluation after data division under the long-tail problem, indicating performance on minority class samples. In contrast, the latter focuses more on the disparity between base classes and novel classes, involving learning with limited samples after the introduction of new classes to achieve improved performance. In the former, each arrival constitutes a completely imbalanced dataset, requiring the model to learn and classify the entire dataset. Ultimately, our evaluation considers performance under three different divisions and overall performance.

Reviewer comment (Experiments):
Balanced sampling is a fundamental operation in long-tail methods. Why is random sampling used when fine-tuning the CLIP in the initial stage? Is it to intentionally obtain a CLIP encoder with strong biases caused by imbalance?

Author response:
We employ a random sampling strategy when fine-tuning CLIP encoders in the first stage, driven by three key considerations.

**Firstly, as indicated in related works [1,2], the feature extractor's impact on classification performance may not be significant.** The pivotal factor in imbalanced classification accuracy lies in the learning of the classifier. Consequently, utilizing random sampling during encoder training or fine-tuning, even with skewed visual features, remains a viable approach. Following this, methods often choose to fix encoders and employ new sampling strategies to train the classifier. Based on this concept, we adopt a random sampling strategy during encoder fine-tuning in the first stage. In the second stage, we shift to a balanced sampling strategy to refine visual features and train a balanced classifier. This decision aligns with our motivation.

**Secondly, experiment results indicate that using a balanced sampling strategy to fine-tune encoders in the first stage is not the optimal approach.** Updated experiment results on sampling strategy in Appendix H support this observation. Notably, employing random sampling in the first stage and balanced sampling in the second stage yields the best results, outperforming the strategy of using balanced sampling in both stages. This finding is consistent with conclusions drawn in previous related work, such as BALLAD.

**Finally, a new t-SNE visualization result in Appendix H compares visual features extracted from the same class under two different sampling strategies.** Even when a balanced sampling strategy is used in Phase A, it does not significantly enhance tail classes' feature distribution distinctiveness. The visual feature distribution obtained by balanced sampling does not markedly differ from that obtained by random sampling on tail samples. However, for head samples, visual features of the head class are less densely clustered than those obtained by random sampling. This underscores the necessity of employing different sampling strategies in distinct stages from an alternative perspective.

[1] Decoupling representation and classifier for long-tailed recognition.
[2] Improving calibration for long-tailed recognition.

Reviewer comment (Novelty):
Reference [1] also leverages CLIP's text features to enhance the discriminative power of image features. In [1], directly using text features and image features in concurrent training a linear classifier can achieve significant improvements in few-shot tasks. However, compared to the method in this paper, the method in [1] is much simpler, with much faster computation speeds and far less computational overhead.  
[1] https://arxiv.org/abs/2301.06267

Author response:
Thank you for introducing this related work; we noticed it before and have conducted experiments using a similar approach. Initially, we anticipated that incorporating textual features as input would enhance our classifier's performance. **However, in practical implementation, we observed that this method did not yield positive results for our specific task and, in fact, led to some performance degradation, potentially due to overfitting.**

As illustrated in the table below, we introduced text features as input while maintaining other structures unchanged to facilitate classifier learning based on the ViT/B-16 and Places-LT dataset. The results indicate that our performance under few-shot scenarios is slightly worse than the outcomes achieved when text prompts are included as part of the input. This observation suggests that utilizing text prompts as input may offer some assistance in learning tail classes. This finding aligns with the propose of [1].

| Methods            | Many-Shot | Medum-Shot | Few-Shot | Overall |
| ------------------ | --------- | ---------- | -------- | ------- |
| OURS               | 49.8      | 52.6       | 54.2     | 51.9    |
| OURS + Text Inputs | 49.3      | 51.4       | 54.4     | 51.3    |

**However, it is noteworthy that while this approach may benefit the learning of tail classes, it exerts a significant negative impact on many-shot and medium-shot performance, ultimately resulting in an overall decrease in performance**. Given that the primary objective of long-tailed vision recognition tasks is to achieve balanced performance across each class, these methods may not be directly applicable.

We posit two possible reasons for this discrepancy. **Firstly, our method targets imbalanced image classification tasks rather than few-shot learning.** Consequently, our dataset is much larger, containing more head class samples. In datasets with ample data, adding text prompts as part of the input may not yield additional performance gains. **Secondly, the semantic information provided by the text prompt as input is relatively simple.** In the context of large and imbalanced vision datasets, relying solely on text prompts as input may not be sufficient to support robust and balanced decision boundary learning. Consequently, we believe there is a need for improved alignment methods to achieve better discriminability and balance between classes.


[1] https://arxiv.org/abs/2301.06267

Reviewer comment (Novelty):
The overall structure of this paper, which deals with challenges in contrastive learning methods due to class imbalance and suggests remedies, evokes thoughts of Suh and Seo (2023). Nevertheless, the current paper does not include any discourse on this topic.

Author response:
Thank you for sharing these papers. We noticed with them before, as they share a similar motivation to ours: learning balanced and discriminative visual features for long-tailed recognition. Despite these similarities, there are significant differences between our approaches.

**The first paper** maximizes mutual information between visual latent features and ground truth labels, employing a contrastive learning method based on image modality. In contrast, our method leverages complementary textual information, to facilitate the learning of robust visual features. **Notably, our approach does not explicitly model or utilize label frequency**, as seen in the first paper, where it is used to compute logit adjustment terms and construct class-wise queues. Instead, our focus is on achieving high separability between classes based on textual features. **Moreover, our method does not rely on the teacher-student network structure**. **Furthermore, our proposed supervised contrastive learning loss is only part of our contribution**, which serves as a complementary for the OT-based prototype-guided matching. As shown in Table 3, our method can still perform best even without the proposed $\mathcal{L}_{\text{SCT}}$.

**The second paper** employs k-positive samples as anchors for contrastive loss, deviating from using augmented instances. This approach, dependent on the number of samples in the minority class, cannot guarantee the distance between k-positive samples from different classes, potentially causing tail classes to be pushed closer together. **In contrast, our method is independent of the number of samples and consistently identifies prototypes with strong discriminability based on textual features.** Additionally, our approach is multimodal, utilizing optimal transport to align visual features and prototypes in an unsupervised manner. We introduce class-level supervision signals based on the transport matrix to further enhance feature learning.

**In summary**, while these referenced papers effectively aim to learn more discriminative visual distributions, our work distinguishes itself through a **focus on multimodal data, the incorporation of optimal transport for balanced prototype-guided visual representation learning and class-level supervision signals based on transport plan**. We will discuss more detailed differences in the related works .

Reviewer comment (Novelty):
Moreover, while one could mention Kang et al. (2021) as a seminal work on achieving a balanced and discriminative feature space in long-tailed classification scenarios, this is also not discussed.

Author response:
Thank you for sharing these papers. We noticed with them before, as they share a similar motivation to ours: learning balanced and discriminative visual features for long-tailed recognition. Despite these similarities, there are significant differences between our approaches.

**The first paper** maximizes mutual information between visual latent features and ground truth labels, employing a contrastive learning method based on image modality. In contrast, our method leverages complementary textual information, to facilitate the learning of robust visual features. **Notably, our approach does not explicitly model or utilize label frequency**, as seen in the first paper, where it is used to compute logit adjustment terms and construct class-wise queues. Instead, our focus is on achieving high separability between classes based on textual features. **Moreover, our method does not rely on the teacher-student network structure**. **Furthermore, our proposed supervised contrastive learning loss is only part of our contribution**, which serves as a complementary for the OT-based prototype-guided matching. As shown in Table 3, our method can still perform best even without the proposed $\mathcal{L}_{\text{SCT}}$.

**The second paper** employs k-positive samples as anchors for contrastive loss, deviating from using augmented instances. This approach, dependent on the number of samples in the minority class, cannot guarantee the distance between k-positive samples from different classes, potentially causing tail classes to be pushed closer together. **In contrast, our method is independent of the number of samples and consistently identifies prototypes with strong discriminability based on textual features.** Additionally, our approach is multimodal, utilizing optimal transport to align visual features and prototypes in an unsupervised manner. We introduce class-level supervision signals based on the transport matrix to further enhance feature learning.

**In summary**, while these referenced papers effectively aim to learn more discriminative visual distributions, our work distinguishes itself through a **focus on multimodal data, the incorporation of optimal transport for balanced prototype-guided visual representation learning and class-level supervision signals based on transport plan**. We will discuss more detailed differences in the related works .

Reviewer comment (Evaluation):
Could you offer some informed speculation about why there is a tendency for textual features to be balanced?

Author response:
**Firstly**, we argue that text features are a more high-level abstract semantic representation, which inherently has better discriminability. Therefore, they can serve as queries in the Vision Language Model to perform tasks such as classifying the input image features. This discriminability, in our work, is manifested as balance. That is, the language encoder is less affected by the tail class. As shown in the visualization in Appendix A, textual features always maintain good separability and balance, regardless of whether the language encoder of CLIP has been fine-tuned or not. **Secondly**, when we construct the text prompt for the input image, we use the class name information. This kind of information already provides high separability for textual features. From the text features, we can often directly infer which object a sentence is describing by the classname. Therefore, they can serve as prototypes with good classification properties. We aim to make visual features better cluster around the corresponding prototypes. **In addition**, when we use balanced textual features to build prototypes, each class uses the same number of text prompts. From this perspective, the prompts between different classes are balanced. The head class does not have more prompts than the tail class. **Finally**, when we fine-tune CLIP, we only use the InfoNCE loss in the image-retrieval direction, because our goal is to use text to classify images, which can also reduce the negative impact on textual features during Phase A to some extent.

Reviewer comment (Presentation):
Since comparing performance between different architectures does not hold much significance, it would be better to provide results for RN50 and ViT-B/16 in separate groups.

Author response:
Thanks for your kind reminder. We will update our tables for a clear comparison in the revision.

Reviewer comment (Experiments):
Does the proposed approach result in any additional training expenses? For instance, what are the costs associated with setting up an optimal transport plan?

Author response:
Yes. The introduction of OT loss brings some lightweight additional computational burden. The added computational burden mainly comes from the need to calculate the optimal transport (OT) distance when aligning visual features and prototypes. We have comprehensively compared our method with BALLAD and VL-LTR in Appendix G. As shown in Table 14, our method is comparable to other methods in terms of the training cost. Compared with BALLAD, we added 10 epochs in the training phase and required more time, but we needed less inference time and had a significant improvement in performance (51.9% vs 49.5%). Compared with VL-LTR, we require fewer epochs for training, less training time, and no additional data. Still, we achieve better performance (51.9% vs 50.1%) than VL-LTR.