Paper: Enhancing Instance-Level Image Classification with Set-Level Labels (ICLR-2024)

Reviewer comment (Presentation):
I have some questions about the method part, Sec. 2.1. In Fig.2 (a), are input samples all belongs to the same set-level labels? I am confused by this figure and Fig.1(a) CIFAR images. What I believe is correct is that each batch contains samples belong to different set-level labels, and the coarse label is assigned to each sample for pre-training.

Author response:
**A:** In the revision, we have added set-level label and fine-grained labels in Figure 2. This highlights that the input to the model is a set of images from CIFAR-100 and the corresponding set-level label is the most frequent superclass, i.e., fruit and vegetables. In Figure 2(a), we only show one input set (not a batch of input sets) due to limited space and the corresponding most frequent superclass of the input set, i.e., 'fruit and vegetables'. Figure 2(b) shows the fine-grained labels of the support set are apple, pear, orange, and mushroom.

To clarify, during pretraining in our FACILE framework, each batch indeed consists of multiple input sets. Each of these sets is associated with one set-level label. For the FSP (Fully Supervised Pretraining) baseline model used for comparison, we assign the corresponding coarse-grained label to each sample within an input set.

Reviewer comment (Reproducibility):
How is SupCon trained? It is superised that supervised contrastive learning perform a lot worse than basic CE approach in most setups. There is not much information about training details.

Author response:
**A:** Please find the training details of SupCon in Appendix A and G.2. When we use set-input data with SupCon, the conventional way of augmenting each instance inside a set could create challenges for SupCon training. It is typically trained to maximize agreement between differently augmented views of the same data point using labeled data. In our application, when using set-input data, the conventional data augmentation methods applied to each instance within a set and the set-input models that extract permutation invariant representations for all instances inside a set may present unique challenges for the SupCon model especially when the model is trained from scratch. However, you can see from Tables 1, 2, 4, 5, 7, and 10 that FACILE-FSP and FACILE-SupCon performance similarly. It's important to note that our framework is flexible and allows for any supervised learning algorithm to be used during the pretraining stage.

Reviewer comment (Reproducibility):
There should be more training details about the framework in Sec.2.1.

Author response:
**A:** We have documented these details to facilitate full reproducibility. The primary experimental setups, including configurations and parameters, are described in the Experiment section of the paper. For more in-depth information, including additional configurations and nuanced methodology, we direct readers to Appendix sections A, D, and G.

Reviewer comment (Presentation):
Fig. 3 is referenced in text before Fig. 2

Author response:
**A:** Thanks for pointing this out! We have reorganized the figures in the revised version.

Reviewer comment (Presentation):
Most of the refernece hyperlinks other than page 1 is not working.

Author response:
**A:** Thank you for bringing this up. We have fixed the reference issue in the revised version.

Reviewer comment (Experiments):
The use of coarse-grained labels like TCGA or NCT is an interesting choice. These are indeed umbrella terms for various subcollections, and traditionally they may not provide a strong learning signal. It could be beneficial to delve into why these particular labels were chosen and what advantages they offer in this context.

Author response:
**A:** Thank you very much for this suggestion. We have included these explanation and advantages in each experiment in the revision.

Our theorem suggested that the relatedness, quantified by the relative Lipschitzness relative to $\mathcal{E}$, is an important factor for excess risk bound. It is intuitive that tasks in different domains do not share the same embedding space (and thus a large Lipschitz constant). Therefore, we deliberately choose pretraining tasks in source domain that are closely related to the target downstream task. 

In response to the use of coarse-grained labels like those in TCGA, please note that the TCGA consortium has categorized an extensive array of cancer types and subregions, covering a diverse range of tissues.  The strategic use of TCGA's whole slide level labels is rooted in their potential to enrich tissue-level classification. While these labels may appear broad, they encapsulate a wealth of underlying heterogeneity inherent to different cancer regions and tissue types and are routinely used in the cancer research community.

Reviewer comment (Novelty):
Your team's approach to pretraining with coarse labels and then fine-tuning on a support set is a solid and proven method. However, it would enrich the work to articulate what sets this particular application or implementation apart in terms of novelty.

Author response:
**A:** Thank you for the opportunity to clarify the novelty of our work.  We have edited the introduction to highlight the novelty accordingly. We summarized novelty (and contribution) in the last three paragraphs of the introduction of our paper. Our work introduces FACILE, a novel generic learning framework that innovatively uses coarse-grained labels for pretraining, before fine-tuning on fine-grained tasks. This method diverges from standard fine-tuned supervised learning (FSL) and transfer learning (TL) by leveraging coarse labels from the source domain rather than heavily relying on detailed annotations. Our theoretical analysis underpins the empirical benefits of FACILE, providing evidence of how coarse-grained labels expedite the learning process with a significant increase in the convergence rate for fine-grained tasks. Furthermore, our experiments across different data domains---ranging from natural images to complex histopathology images---demonstrate FACILE's superior performance and versatility. Notably, we record a remarkable 13% performance improvement over existing baselines in a challenging histopathology benchmark. These elements collectively distinguish our work's novelty and significance in advancing the state of the art in representation learning (more specifically, in few-shot learning and model fine-tuning).

Reviewer comment (Experiments):
The comparison with SimCLR and simSIAM provides useful insights, but considering the advancements in the field, benchmarking against more recent self-supervised learning methods like DINO or DINOv2 might offer a more comprehensive evaluation.

Author response:
**A:** Thank you for raising this important point. The primary aim of FACILE is not to engage in self-supervised learning that requires extensive data and computational resources. Instead, our focus is on leveraging coarse-grained labels to improve performance in downstream tasks. In this context, foundation models such as BYOL or DINO V2 serve as robust starting points. We can enhance these models by fine-tuning them with domain-specific data, effectively tailoring their capabilities to the nuances of the target domain.

To this end, we conducted two additional experiments. First, we sought to enhance model performance with coarse-grained labels of anomaly detection datasets (similar to Deep Set [5] and Set Transformer [6]). A total of 11,788 input sets of size 10 are constructed from the CUB200 training dataset by including one example that lacks an attribute common to the other examples in the input set. This setup creates a challenging scenario for models, as they must identify the outlier among otherwise similar instances. In downstream tasks, we evaluate the finetuned feature encoder composed of the CLIP [3] image encoder ViT-B/16 and appended fully-connected layer on the classification of species of the CUB200 test dataset. The batch normalization and ReLU are applied to the fully-connected layer.

The model training approach in this experiment centered around the CLIP image encoder, enhanced with an additional fully-connected layer. FACILE-FSP and FACILE-SupCon incorporate this setup, utilizing the CLIP-based feature encoder and focusing on fine-tuning the fully-connected layer through the FACILE pretraining step. In contrast, the SimSiam approach leverages the CLIP image encoder as a backbone while fine-tuning the projector and predictor components. Similarly, the SimCLR method also uses the CLIP encoder as its foundation but focuses solely on finetuning the projector. These varied strategies reflect our efforts to optimize the feature encoder for accurately identifying anomalies and improving classification performance in related tasks. The training details can be found in the revised appendix.

|pretrain method|NC              | LR             | RC             |
|---------------|----------------|----------------|----------------|
|CLIP (ViT-B/16)|$83.84 \pm 1.10$|$81.01 \pm 1.23$|$82.75 \pm 1.17$|
|SimCLR         |$84.03 \pm 1.08$|$83.49 \pm 1.14$|$86.30 \pm 1.03$|
|SimSiam        |$84.02 \pm 1.10$|$83.90 \pm 1.13$|$85.68 \pm 1.07$|
|FACILE-SupCon  |$87.49 \pm 0.99$|$86.57 \pm 1.07$|$88.01 \pm 0.99$|
|FACILE-FSP     |$\mathbf{88.74 \pm 0.94}$|$\mathbf{ 88.45 \pm 0.96 }$|$\mathbf{ 88.36 \pm 0.95 }$|

Pretraining on input sets from CUB200. Testing with 5-shot 20-way meta-test sets; average F1 and CI are reported.

Second, similar to the first experiment, we finetune a fully-connected layer that is appended after ViT-B/14 of DINO V2 [2]. This methodology is applied across various models to assess their performance on histopathology image datasets. By adopting the DINO V2 architecture, known for its robustness and effectiveness in visual representation learning, we aim to harness its potential for the specialized domain of histopathology. We refer interested readers to the revised appendix for details of pretraining.

Notably, our methods, FACILE-SupCon and FACILE-FSP, demonstrated remarkably superior results in comparison to other baseline models when applied to histopathology image datasets. This outcome highlights the effectiveness of these methods in leveraging coarse-grained labels specific to histopathology, thereby greatly enhancing the model performance of downstream tasks. It also shows that DINO V2 exhibits limitations in its generalization performance on histopathology images. While DINO V2 provides a strong starting point, there is a clear need for further finetuning or prompt learning to optimize its performance for the unique challenges presented by histopathology datasets.

Reviewer comment (Experiments):
To further validate the generalizability of the method, it could be insightful to include results against standardized few-shot learning benchmarks, such as Mini-Imagenet 5-way (1-shot) or SST-2 Binary classification.

Author response:
**A:** Thank you for the suggestion. We did explore the possibility of using benchmarks from the mini-ImageNet and SST-2 datasets. However, we did not find the corresponding coarse-grained labels for these datasets. However, we did contrast our model's performance against some published results including [1] and published models including CLIP [3] and DINO V2 [2].

Reviewer comment (Experiments):
Adding ablation studies that feature additional pretrained models—or even models pretrained without the coarse labels—could help underscore the specific benefits of using coarse-grained label-based pretraining in your approach.

Author response:
**A:** Thank you very much for your valuable suggestion. Please see our two new experiments in response to question 3. The two new experiments provide an ablation study about pretrained models. These additional analyses aim to clearly highlight the specific advantages of using some pretrained (foundation) models.

Reviewer comment (Reproducibility):
Your methodology would be even more robust if additional training details are shared. Information on image augmentations, learning schedules, and optimizer settings could offer valuable insights and help in the reproducibility of your results.

Author response:
**A:** Thank you for the suggestion! We have provided a comprehensive description of our training details, including image augmentation techniques, learning schedules, and optimizer configurations, etc., in the experiment section, appendix section A, D, and G of our paper. We have taken care to present a thorough description of the procedures to ensure that our results can be accurately replicated. For further in-depth technical specifics and hyperparameter settings, we direct readers to appendix A, D, and G, which contain extended information on the training of both the FACILE framework and the baseline models.

Reviewer comment (Presentation):
Do you think you could pictorially diagram the approach adding the relevant details? It is unclear to me if the method essentially pretrains using coarse-labels and then fine-tunes on the test set using the support set or is there more to the method.

Author response:
**A:** Thank you for raising this point. In the revision, we have added set-level label and fine-grained labels in figure 2. This highlights the input to the model is a set of images from CIFAR-100 and corresponding label set-level label is the most frequent superclass, i.e., fruit and vegetable. Figure 2(b) shows the fine-grained labels of support set are apple, pear, orange, and mushroom.

Reviewer comment (Experiments):
Why are the methods for pretraining SupCon and FSP chosen for pre-training? Adding rationale for this might help motivate the choice of pretraining method.

Author response:
**A:** Thank you for the suggestions. We have included the explanation in the revision. Our FACILE framework is designed to be a generic algorithm that is compatible with any supervised learning method in its pretraining stage. We chose SupCon (Supervised Contrastive Learning) and FSP (Fully Supervised Pretraining) as they are representative of the two main approaches within supervised learning: contrastive and non-contrastive (traditional supervised) learning, respectively. These methods were selected to showcase the versatility and wide applicability of our framework, demonstrating that it can be integrated with typical supervised learning paradigms. The rationale for using these specific pretraining strategies is to illustrate the effectiveness of FACILE with different underlying learning principles, ensuring that our results can be generalized across various pretraining methodologies.

Reviewer comment (Experiments):
To what extent does the similarity between the pretraining dataset and its coarse labels and the target dataset with its fine labels affect the effectiveness of the method? For instance, can the method perform well when the pretraining dataset is CIFAR-100 while the downstream task involves a medical dataset? In such a scenario, which pretraining method is preferable: supervised pretraining on ImageNet, self-supervised pretraining (ignoring labels entirely), or the proposed method?

Author response:
Notably, our methods, FACILE-SupCon and FACILE-FSP, demonstrated remarkably superior results in comparison to other baseline models when applied to histopathology image datasets. This outcome highlights the effectiveness of these methods in leveraging coarse-grained labels specific to histopathology, thereby greatly enhancing the model performance of downstream tasks. Another critical insight emerged from our research: the current foundation model, DINO V2, exhibits limitations in its generalization performance on histopathology images. This suggests that while DINO V2 provides a strong starting point due to its robust visual representation capabilities, there is a clear need for further fine-tuning or prompt learning to optimize its performance for the unique challenges presented by histopathology datasets. This finding underscores the importance of specialized adaptation in the application of foundation models to specific domains like medical imaging.

> *Question (1):* To what extent does the similarity between the pretraining dataset and its coarse labels and the target dataset with its fine labels affect the effectiveness of the method? For instance, can the method perform well when the pretraining dataset is CIFAR-100 while the downstream task involves a medical dataset? In such a scenario, which pretraining method is preferable: supervised pretraining on ImageNet, self-supervised pretraining (ignoring labels entirely), or the proposed method?

**A:** Our theorem suggested that the relatedness, quantified by the relative Lipschitzness relative to $\mathcal{E}$, is an improtant factor for excess risk bound. It is intuitive that tasks in different domains do not share the same embedding space (and thus a large Lipschitz constant). Therefore, we deliberately choose pretraining tasks in source domain that is closely related the target downstream task.

In this context, although foundation models provide a powerful tool for general ML applications, sigificant work, involving model fine-tuning or prompt learning, is still needed to adopt it to domain-specific task as shown in our new experiment results with CLIP image encoder and DINO V2 model and some public works, e.g., [4]. We propose an alternative finetuning framework towards this effort.

We acknowledge that estimating the extent to which the source and target datasets need to be similar for optimal transfer is a valuable avenue for future research. However, our current work focuses on demonstrating that even when there are label space and label granularity differences---as in pretraining/fine-tuning on TCGA and applying the model to NCT---our method offers a practical advantage by making effective use of coarse-grained, yet related, labels to improve the learning process for fine-grained classification tasks. The effectiveness of our approach is therefore predicated on the relatedness of the set-level annotations to the target task, which is a realistic and common scenario in many machine learning applications.

Reviewer comment (Experiments):
Given the same "related" dataset, if you have both the fine-grained label and coarse-grained label, which pretraining strategy is preferable? (Let’s say your downstream task is classification on a medical image dataset, with fine-grained label A, B, C. The pretraining dataset you have is another medical image dataset (thus more related than ImageNet). You have both coarse label D, E, F and fine-grained label G, H, I, J, K, L. In this case, is fully supervised pretraining on G, H, I, J, K, L more beneficial or is set level coarse pretraining on D, E, F more beneficial?)

Author response:
We thank the reviewer for the insightful query regarding the optimal pretraining strategy using fine-grained or coarse-grained labels in related datasets.

* Our empirical evaluation in Section 3.4 contrasts models pretrained on the NCT dataset (fine-grained labels) against those using the TCGA dataset (coarse-grained labels). This comparison clearly demonstrates the effectiveness of models leveraging coarse-grained labels, as evidenced by our results and those in [1].
* Theoretically, we quantify dataset relatedness via the Lipschitzness condition (Definition 3). Intuitively, source tasks with lower Lipschitz constants yield better excess risk bounds.
* As highlighted in the Introduction section, obtaining coarse-grained labels is often less resource-intensive, particularly in fields like medical image analysis where label scarcity is a challenge. This underscores the practical value of utilizing more readily available coarse-grained labels. We also note that in some applications coarse-grained labels can be extracted from metadata or otherwise inferred and so the cost of adding coarse-grained labels for some applications is minimal.
* We recognize the importance of future research in identifying the most beneficial source data in advance for optimal transfer, even within the same domain. The choice between fine-grained and coarse-grained labels for pretraining involves complex considerations, including dataset specifics, label acquisition costs, and the goals of downstream tasks.


References:

[1] Yang, Jiawei, Hanbo Chen, Jiangpeng Yan, Xiaoyu Chen, and Jianhua Yao. "Towards better understanding and better generalization of few-shot classification in histology images with contrastive learning." arXiv preprint arXiv:2202.09059 (2022).

[2] Oquab, Maxime, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

[3] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." In International conference on machine learning, pp. 8748-8763. PMLR, 2021.

[4] Zhang, Jingwei, Saarthak Kapse, Ke Ma, Prateek Prasanna, Joel Saltz, Maria Vakalopoulou, and Dimitris Samaras. "Prompt-MIL: Boosting Multi-Instance Learning Schemes via Task-specific Prompt Tuning." arXiv preprint arXiv:2303.12214 (2023).

[5] Zaheer, Manzil, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R. Salakhutdinov, and Alexander J. Smola. "Deep sets." Advances in neural information processing systems 30 (2017).

[6] Lee, Juho, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. "Set transformer: A framework for attention-based permutation-invariant neural networks." In International conference on machine learning, pp. 3744-3753. PMLR, 2019.