Paper: When Hard Negative Sampling Meets Supervised Contrastive Learning (ICLR-2024)

Reviewer comment (Experiments):
To enhance the credibility of our research, you should consider using the same base models (such as ViT or Swin) as other studies for our baseline in Table 1 and Table 2.

Author response:
Thank you for your suggestion regarding the baseline models in Tables 1 and 2. We do have ViT and MAE as baselines in Table 2. We agree that using widely recognized base models such as the Vision Transformer (ViT) provides a familiar benchmarking standard, enhancing the comparability of our results with existing studies. However, the aim of this paper is to address the issue of how cross-entropy can result in sub-optimal generalization rather than building a dedicated method for few-shot learning. Few-shot learning methods, as cited in our paper, are limited to scenarios such as 1-shot and 5-shot settings and tend to perform better in fine-grained or large-scale datasets. Therefore, our main comparison is between our proposed method and cross-entropy loss rather than directly comparing with these few-shot learning methods. The reason we included these results is to give readers a sense of the performance level of our method. We are also proposing an alternative fine-tuning method to cross-entropy, not a few-shot learning method. Direct comparison with the latter may not be very meaningful. Nevertheless, we have attached above in answer of question 1, which consists of results for MAE and ResNet50, for your comparison. We will update these results in the revised version of our paper.

Reviewer comment (Experiments):
In order to provide a more comprehensive analysis, the results in Table 1 and Table 2 should include the performance of contrastive learning without the hard negative mining method.

Author response:
We appreciate your insight into providing a more comprehensive analysis. Including the performance of contrastive learning without the hard negative mining method in Tables 1 and 2 is indeed crucial for a holistic understanding of our methodology's impact. We will incorporate these additional results to offer a clearer comparison, highlighting the effectiveness of hard negative mining in enhancing the performance of contrastive learning. Nevertheless, our ablation study section provides empirical results that demonstrate the increased performance largely comes from hard negative sampling.

Here is the result for contrastive learning without hard negative sampling for you to compare the performance:

| Model | FT method         | Alpha | CIFAR 100 | CUB-200 | Caltech256 | Oxford-Flowers | Pet   | iNat2017 | Places365 | ImageNet-1k |
|-------|-------------------|-------|-----------|---------|-------------|----------------|-------|----------|-----------|-------------|
| ViT-B | CE                | 0     | 87.13     | 76.93   | 90.92       | 90.86          | 93.81 | 65.26    | 54.06     | 77.91       |
| MAE   | CE                | 0     | 87.67     | 78.46   | 91.82       | 91.67          | 94.05 | 70.5     | 57.9      | 83.60       |
| BEIT-3 | CE                | 0     | 92.96     | 98      | 98.53       | 94.94          | 94.49 | 72.31    | 59.81     | 85.4        |
| BEIT-3 | SCL               | 0     | 93.04     | 98.07   | 98.61       | 94.98          | 94.41 | 72.59    | 59.94     | 85.63       |
| BEIT-3 | CE + SCL          | 0.9   | 93.15     | 98.29   | 98.93       | 95.13          | 94.78 | 73.62    | 60.92     | 85.8        |
| BEIT-3 | CE + CL(SCHaNe)| 0.9   | **93.56** |**98.93** | **99.41** | **95.43** | **95.62** | **75.72**    | **62.22** | **86.14** |

The data presented in the table above illustrates that the application of supervised contrastive learning generally results in a modest enhancement in performance, with the notable exception of the Oxford-Pet dataset. Our proposed method, SCHaNe, on the other hand, shows a significant and more consistent improvement across all evaluated benchmarks.

Reviewer comment (Writing):
The representation of Formula 3 needs to be clarified to ensure better understanding, as it is currently not very clear.

Author response:
Thank you for pointing out the need for clarity in the representation of Formula 3. We understand that the comprehensibility of our mathematical formulations is paramount for the clear communication of our research methodology. To address this, we will revise Formula 3 to ensure that it is presented in a more understandable format. We will also provide a detailed explanation in the accompanying text to aid in its interpretation, ensuring that our approach is transparent and comprehensible to our readers.

Reviewer comment (Experiments):
To provide a more complete comparison, Figure 3 and Figure 4 should include the results of BEiT-3-CE + contrastive learning, in addition to the results of our proposed method.

Author response:
We recognize the importance of comprehensive comparisons in Figures 3 and 4. Including the results of BEiT-3-CE combined with contrastive learning, in addition to our proposed method, will indeed provide a more complete perspective. This addition will allow us to demonstrate the comparative effectiveness of our approach against a relevant benchmark, thereby highlighting the unique contributions and improvements our method offers. We will add them in the updated version.

Reviewer comment (Evaluation):
In each comparative experiment, it is important to clearly indicate the improvement achieved by the hard negative mining method on top of the performance obtained with CE + contrastive learning. This will help demonstrate the added value of our approach.

Author response:
Your suggestion to indicate the improvements achieved by the hard negative mining method in each comparative experiment is well-taken. We will ensure that the incremental gains obtained from our hard negative mining approach, over and above the performance of CE combined with contrastive learning, are explicitly presented in the updated version. Please see the attached result in the answer to question 3.

We appreciate your attention to this detail in our work and hope that our response adequately addresses your concerns, highlighting the deliberate and thorough approach we have taken in addressing the complexities of combining contrastive learning with CE loss fine-tuning.

Reviewer comment (Experiments):
Why did you choose BEiT-3 as your base model? There are other base models like DINOv2, CLIP, etc.

Author response:
We conducted a thorough evaluation of different models, including CLIP, before selecting BEiT-3 as the base model for our study. Our decision was mainly based on BEiT-3's status as a Multimodal Large Language Model and its outstanding performance in image classification tasks. Nevertheless, I attached the results of other models, MAE [2] and ResNet50 [3]. The result demonstrates the observed improvements are not limited to BEiT-3 and show the wide applicability of our proposed loss objective. Furthermore, our selection of BEiT-3 was driven by a combination of the model's alignment with our research goals, the feasibility of conducting extensive experiments within our resource constraints, and its state-of-the-art performance. BEiT-3 was the most suitable platform for our study, guaranteeing robust and significant results, while DINOv2 and CLIP are also commendable but outside the scope of the paper. We further describe our rationale below.

| Model    | FT method | Alpha | CIFAR 100 | CUB-200 | Caltech256 | Oxford-Flowers | Pet    |
|----------|-----------|-------|-----------|---------|-------------|----------------|--------|
| MAE      | CE        | 0     | 87.67     | 78.46   | 91.82       | 91.67          | 94.05  |
| MAE      | SCHaNe    | 0.9   | 90.29     | 81.30   | 93.11       | 92.82          | 94.88  |
| ResNet50 | CE        | 0     | 96.27     | 84.62   | 81.38       | 95.71          | 93.24  |
| ResNet50 | SCHaNe    | 0.9   | 96.92     | 84.98   | 83.05       | 96.33          | 93.51  |
| BEiT-3    | CE        | 0     | 92.96     | 98      | 98.53       | 94.94          | 94.49  |
| BEiT-3    | SCHaNe| 0.9   | **93.56**    | **98.93**  | **99.41**      | **95.43**         | **95.62** |

**a**. Model Suitability and Performance:
The implementation of a multimodal model like BEiT-3 has enabled us to delve into the possibilities of contrastive learning in diverse domains, paving the way for expanding the impact of our research. In this paper, we also offer valuable insights to the academic community regarding the performance of Multimodal Large Language Models in image-only tasks. This is achieved through comprehensive experimentation on few-shot learning and full-dataset evaluations, addressing a gap in the existing literature, and these evaluations are notably absent in the BEiT-3 paper. Lastly, BEiT-3 base model, at the time of its release, achieved a Top-1 accuracy of 85.4\% on the ImageNet-1k dataset [1], setting a new benchmark in image classification.

**b**. Experimental Workload Considerations: 
The scope of our experiments significantly influenced our model selection. Our study involved evaluating four few-shot datasets in both 1-shot and 5-shot settings with two different loss functions, amounting to 16 unique cases. Additionally, we conducted evaluations across eight image benchmark datasets with four different models or methods, leading to 24 cases. Each case required at least three runs. Considering these requirements, our total experimental workload comprised over 120 runs, including several large and computationally intensive datasets such as ImageNet-1k, iNaturalist 2017, and Places365. This extensive experimental requirement necessitated a choice that was feasible within our academic budget and timeline constraints.

**c**. Temporal and Computational Constraints:
The release of DINOv2 [4] in April 2023, subsequent to the initiation of our project in January 2023, presented a significant timing challenge. Starting experiments with DINOv2 in late April would have been impractical, given our academic budget and the sheer volume of experiments required. Thus, BEiT-3 was a more viable option due to its earlier release date, proven performance and fewer parameters with respect to DINOv2 (1100M for DINOv2 vs 86M, BEiT3).

Reviewer comment (Experiments):
In general, fine-tuning with cross-entropy (CE) loss is not strongly coupled with the batch size, while contrastive learning can be affected by the batch size. This can limit the flexibility of fine-tuning with CE when combined with contrastive learning. What are your thoughts on this issue?

Author response:
**a**. Mitigating Batch Size Dependency via Cross-Entropy Integration:
We recognize that contrastive learning is sensitive to batch size variations, unlike fine-tuning with cross-entropy (CE) loss. To address this, we have incorporated the CE component into our overall training objective. This integration strategically reduces the reliance of our contrastive learning approach on large batch sizes, thereby enhancing its adaptability and effectiveness in various computational settings.

**b**. Efficacy of Our Method with Standard Batch Sizes:
The integration of fine-tuning with CE loss into our contrastive learning framework was carefully designed to navigate the challenges posed by hardware constraints, particularly regarding batch size. We meticulously calibrated a batch size that is not only feasible within the limits of standard computational resources but also sufficient to ensure the effectiveness of contrastive learning. For instance, with a commonly available 24GB GPU, a batch size of 64 is achievable and, in our evaluation, has demonstrated superior performance compared to using vanilla CE loss alone, as shown in the table below. In environments with more substantial GPU capacities, like in our experiments, we can employ batch sizes of 128 or higher, further enhancing performance. Importantly, our methodology maintains its superiority over traditional CE loss approaches even at these modest batch sizes, offering a compelling alternative for various deep learning applications.

| CIFAR-FS |         |       |            |       |
|----------|---------|-------|------------|-------|
| Model    | FT method | Alpha | Batch Size | ACC@1 |
| BEIT-3    | CE        | 0     | 64         | 83.68 |
| BEIT-3    | SCHaNe | 0.9   | 32         | 81.2  |
| BEIT-3    | SCHaNe | 0.9   | 64         | 84.59 |
| BEIT-3    | SCHaNe | 0.9   | 128        | 87.00 |

**c**. Adaptive and Hybrid Strategies:
To further mitigate batch size constraints, we employed techniques like gradient accumulation, which enables us to simulate the effects of larger batch sizes.

**d**. Utilizing Batch Size-Independent Techniques:
Future work could involve investigating methods to reduce the batch size dependency in contrastive learning, such as using memory banks or momentum contrast methods. These techniques allow for an expanded set of negative samples without the need for larger batch sizes, thus ensuring the effectiveness of the contrastive learning component.

**e**. Future Research:
Our paper is pioneering this direction of research by combining cross-entropy and contrastive learning. We concur that optimizing the interaction between batch size, contrastive learning, and CE loss fine-tuning remains an open area for further investigation. Our research contributes to this field, and we believe that continued exploration could yield significant advancements in the fine-tuning of deep learning models.

Reviewer comment (Experiments):
The paper could benefit from a more detailed comparison with existing methods. While the authors compare the proposed method with the strong baseline BEiT-3, they do not compare it with other similar state-of-the-art methods [1][2] in the field.

Author response:
We appreciate your insight into providing a more comprehensive comparison, and we are happy to do so. Please see the attached result. It is worth noting that [1] is not a fair comparison with their proposed approach, mainly because their method is fully unsupervised. The original reason for us not comparing with [2] is that our aim is to provide an alternative of cross-entropy during the fine-tuning phase instead of solely boosting the performance of supervised contrastive learning. More analysis of [2] is provided in the answer of weakness 2. Nevertheless, we evaluate the methods from [1] and [2] in five image benchmarks and compare them with our proposed objective, as shown in the table below.

| Model | FT method         | CIFAR 100 | CUB-200 | Caltech256 | Oxford-Flowers | Pet   |
|-------|-------------------|-----------|---------|-------------|----------------|-------|
| BEIT-3 | CE                | 92.96     | 98      | 98.53       | 94.94          | 94.49 |
| BEIT-3 | [1]               | 84.19     | 87.69   | 89.12       | 85.94          | 85.54 |
| BEIT-3 | [2]               | 91.08     | 95.41   | 97.15       | 92.03          | 91.62 |
| BEIT-3 | SCHaNe  | **93.56**  | **98.93**| **99.41**  | **95.43**    | **95.62** |

The SCHaNe fine-tuning method consistently outperforms other methods across all datasets. Method [1] consistently shows the lowest performance across all datasets.
Method [2] perform slightly worse than BEIT3 with Cross-Entropy (CE) across five tested dataset.

Reviewer comment (Experiments):
One of the main weaknesses I've identified is that the primary baseline used in this paper is Cross-Entropy (CE) loss, not Supervised Contrastive Learning (SupCon). If the paper's central claim is that 'introducing importance weights for negative samples based on their dissimilarity plays an important role,' then I believe SupCon should be the main baseline for comparison. Although SCHaNe outperforms SupCon in the few-shot setting as shown in Table 3, the inclusion of SupCon results in other settings—such as in Table 1, Table 2, and Figure 4—could strengthen the paper.

Author response:
Thank you for your insightful feedback regarding the baseline selection in our paper. You rightly point out that Supervised Contrastive Learning (SupCon) should be a central baseline for comparison, given our paper's focus on the importance of weighting negative samples based on dissimilarity. However, our aim is to provide an alternative to cross-entropy during the fine-tuning phase instead of solely boosting the performance of supervised contrastive learning. I attached the current result we have for you to review:

| Model | FT method       | Alpha | CIFAR 100 | CUB-200 | Caltech256 | Oxford-Flowers | Pet   | iNat2017 | Places365 | ImageNet-1k |
|-------|-----------------|-------|-----------|---------|-------------|----------------|-------|----------|-----------|-------------|
| ViT-B | CE              | 0     | 87.13     | 76.93   | 90.92       | 90.86          | 93.81 | 65.26    | 54.06     | 77.91       |
| MAE   | CE              | 0     | 87.67     | 78.46   | 91.82       | 91.67          | 94.05 | 70.5     | 57.9      | 83.60       |
| BEIT-3 | CE              | 0     | 92.96     | 98      | 98.53       | 94.94          | 94.49 | 72.31    | 59.81     | 85.4        |
| BEIT-3 | SCL             | 0     | 93.04     | 98.07   | 98.61       | 94.98          | 94.41 | 72.59    | 59.94     | 85.63       |
| BEIT-3 | CE + SCL        | 0.9   | 93.15     | 98.29   | 98.93       | 95.13          | 94.78 | 73.62    | 60.92     | 85.8        |
| BEIT-3 | CE + CL(SCHaNe) | 0.9   | **93.56**   | **98.93**   | **99.41**   | **95.43**   | **95.62** | **75.72**    | **62.22**     | **86.14**       |

The data presented in the table illustrates that the application of supervised contrastive learning generally results in a modest enhancement in performance, with the notable exception of the Oxford-Pet dataset. Our proposed method, SCHaNe, on the other hand, shows a significant and more consistent improvement across all evaluated benchmarks. We will update these results in the new version of our paper and provide more comparisons in the cameral-ready version.

Reviewer comment (Experiments):
BEiT-3 is primarily utilized as the main architecture. The Future Work section suggests that extending this method to various architectures may be promising, but I believe that evaluating the proposed method across different architectures should be included in this paper.

Author response:
We appreciate your suggestion to evaluate our proposed method across various architectures. While BEiT-3 serves as the main architecture in our current study due to its relevance and performance, we understand the importance of demonstrating the versatility of our method. In light of your feedback, we have expanded our experimental scope to include additional architectures (as explained below), ensuring a more robust validation of our approach.

**a**. Flexibility and Compatibility:
Our method is designed with flexibility and compatibility in mind. The core principles and techniques of our approach, particularly the hard negative mining strategy within the framework of contrastive learning, are model-agnostic. Most importantly, there is no computation overhead and a complicated pipeline for our proposed method. This means they can be applied to a wide range of pre-trained models, not just BEiT-3.

**b**. Integration with Other Pre-trained Models:
We have explored the integration of our method with other popular pre-trained models, such as Vision Transformer (ViT) and ResNet50.

| Model       | FT method       | Alpha | CIFAR 100 | CUB-200 | Caltech256 | Oxford-Flowers | Pet   |
|-------------|-----------------|-------|-----------|---------|-------------|----------------|-------|
| MAE         | CE              | 0     | 87.67     | 78.46   | 91.82       | 91.67          | 94.05 |
| MAE         | CE + CL(SCHaNe) | 0.9   | 90.29     | 81.30   | 93.11       | 92.82          | 94.88 |
| ResNet50    | CE              | 0     | 96.27     | 84.62   | 81.38       | 95.71          | 93.24 |
| ResNet50    | CE + CL(SCHaNe) | 0.9   | 96.92     | 84.98   | 83.05       | 96.33          | 93.51 |
| BEIT-3       | CE              | 0     | 92.96     | 98      | 98.53       | 94.94          | 94.49 |
| BEIT-3       | CE + CL(SCHaNe) | 0.9   | **93.56**     | **98.93**   | **99.41**       | **95.43**   | **95.62** |

These models, known for their robustness in various image-related tasks, have shown promising improvements when augmented with our approach. The results, which we will include in the revised manuscript, indicate a consistent enhancement in performance across these models, demonstrating the versatility of our method. We appreciate your insightful query, as it underscores the potential impact and versatility of our work within the field.

Reviewer comment (Experiments):
The proposed method appears to be limited in its applicability to various tasks, such as dense prediction tasks. While this may not be a significant drawback, explicitly stating this limitation could enhance the paper. Moreover, the paper claims that “Our SCHaNe objective function can be applied using a wide range of encoders, such as BERT for natural language processing tasks,” yet there are no experiments provided to substantiate this claim.

Author response:
Thank you for highlighting the need to clarify the applicability and limitations of our proposed method. We agree that explicitly stating the limitations, especially concerning dense prediction tasks, would enhance the comprehensiveness of our paper. We will include a section detailing these limitations to ensure a balanced and transparent presentation of our work.

Regarding the claim of SCHaNe's applicability with a range of encoders, including BERT for natural language processing tasks, we acknowledge the absence of experimental validation for this statement. To address this, we will rephrase our claim to reflect the current scope of our experiments more accurately, thereby maintaining the integrity of our assertions.

Reviewer comment (Presentation):
The presentation of the paper could be improved. For instance, the notation in Equations 2-4 is confusing. If I understand correctly, $\beta$ should vary with the index, but it might be misunderstood as a constant since it lacks an index. In Equation 2, $z$ denotes the label, which is not the case in Equations 3 and 4. Regarding Figure 4, while the trend is important, we cannot directly compare the accuracies across various downstream tasks in an 'apple-to-apple' manner.

Author response:
Thank you for highlighting the need for enhanced clarity in the presentation of Equations 2-4. We acknowledge the importance of precise and unambiguous mathematical notation for effective communication of our research. In response to your feedback, we will revise Equations 2-4 to eliminate any potential confusion, particularly regarding variable indexing and their consistent representation. This revision will include clear indexing where appropriate and a detailed explanation in the accompanying text to facilitate a better understanding of these equations. These updates, aiming for greater clarity and accuracy, will be prominently featured in the revised version of our paper.

We appreciate your observation regarding the limitations in directly comparing accuracies across different downstream tasks in Figure 4. To address this and provide a more straightforward comparison, we will include a numerical table in the appendix. This table will present the accuracies in a format that allows for easier and more direct comparisons across tasks, thereby complementing the visual trends depicted in Figure 4. This addition aims to enhance the interpretability of our results and facilitate an 'apple-to-apple' comparison, as you rightly suggest.