Paper: A Data-Driven Solution for the Cold Start Problem in Biomedical Image Classification (ICLR-2024)

Reviewer comment (Experiments):
More explanations on selecting various building blocks in the active learning paradigm are required. For instance, is there a particular reason for selecting contrastive learning to pre-train the encoder? Aside from contrastive learning, there are many self-supervised learning methods, such as reconstruction and masked auto-encoder; why does the study focus on evaluating contrastive learning methods here?

Author response:
Masked autoecnoders (MAE) focus on reconstruction and holistic feature learning, while contrastive learning focuses on discriminative feature learning to distinguish between different data points. Considering this, the main reason to focus on using self-supervised learning is its potential to provide the best pre-training backbone weights for classification tasks (as a discriminative task). Also. the outperformance of SSL pretraining over generative models in biomedical data was previously explored by Chandra et al. 2021 and Boushehri et al. 2022. Additionally, we avoided models such as MAE, which use vision transformers in their backbone, as we did not have enough data for training large models such as ViTs. Now, we address these points in the “Related Works” section, where we write:
“Prioritizing classification tasks that necessitate discriminative feature learning, we chose self-supervised learning over generative models like Masked Autoencoders (MAE) (He et al. 2022). This aligns with the findings by Chandra et al. (2021) and Boushehri et al. (2022), where the effectiveness of discriminative SSL in biomedical data is demonstrated. Moreover, methods like MAE's dependence on large-scale Vision Transformers (ViTs) (Dosovitskiy et al. 2021) was impractical for our dataset size.”

Reviewer comment (Experiments):
In addition to FPS, many sample selection metrics like entropy, marginal entropy, etc., are proposed in prior arts. Why is FPS selected? What are its advantages compared to other methods?

Author response:
The reason to use FPS is that it does not require any prediction values, and only using the feature space vectors is enough for the algorithm to sample data points. In contrast, methods such as entropy and marginal entropy do need access to a softmax layer to be calculated. This requirement eliminates the possibility of using them in a cold-start scenario.

Reviewer comment (Novelty):
The biggest concern I have is about the technical novelty. All of the modules have been established in the prior works.

Author response:
The primary innovation lies in the application of a three-stage pipeline to the cold start problem in the medical domain, an area that has been notably overlooked in existing literature. Our work is not merely an adaptation of existing methods but a savvy approach specifically tailored to address real-world challenges in healthcare. We have strategically combined and refined modules previously used in disparate contexts into a cohesive and efficient pipeline. This novel integration includes SSL pretraining, FPS sampling, and model soups hyperparameter tuning, all meticulously adapted to the unique complexities of biomedical data analysis. Our approach transcends previous, incremental improvements, offering a groundbreaking solution to a critical problem in health informatics, thereby setting a new standard in the field.

Reviewer comment (Experiments):
Self-supervised learning should be pre-trained on a large-scale dataset to learn diverse representations. However, the datasets in this work seem very small. I have doubt about the power of SSL in this setting.

Author response:
The effect of the scale of training data on the performance of SSL depends on a combination of factors, like the inherent complexity, diversity, and representativeness of the data. In medical settings, SSL is effective when trained on smaller scales and with lesser diversity compared to conventional large-scale datasets (Huang et al., 2023).

Reviewer comment (Experiments):
How about using a large-scale pre-trained SSL on biomedical images instead? It is more interesting to study the gap between the pre-trained SSL and the public weights of SSL.

Author response:
Biomedical images often exhibit a high degree of heterogeneity between datasets and contain modality-relevant features that are significantly different from those found in general image datasets. Moreover, models pre-trained on general datasets might not capture the nuanced features critical for accurate biomedical image analysis. This point was explored in our primary experiments and showed much poorer performance compared to pretraining on the target unlabeled dataset.

Reviewer comment (Evaluation):
What is the advantage/effect of SSL on cold-start problem?

Author response:
In addressing the cold-start problem, where no labeled data is initially available, self-supervised learning (SSL) plays a critical role by enabling the extraction of meaningful data representations. This process allows us to identify and prioritize the most informative points for initial labeling in the absence of labels or any prior knowledge about data.

Reviewer comment (Experiments):
Can we directly use semi-supervised learning as one-stage pipeline instead of multi-stage pipeline proposed in this work?

Author response:
In a cold-start scenario, direct application of semi-supervised learning is impractical due to its reliance on some initial labeled data, which is absent in such situations. Our multi-stage pipeline addresses this by enabling the strategic selection of data points for initial annotation, effectively setting the initial annotation budget for semi-supervised learning, and ensuring more efficient use of limited annotation resources.

Reviewer comment (Experiments):
The motivation is not convincing to me. If the aim is to reduce labeling effort, could we also try active learning, few-shot learning for this work?

Author response:
Both active learning and few-shot learning require an initial set of annotated data to begin. It is shown that the initial annotation set significantly impacts the final performance of these learning paradigms. Our method provides the most informative initial data annotations to benefit both learning paradigms from a very initial state. We hope the additional explanation we added to the introduction addresses this kind of misunderstanding.

Reviewer comment (Novelty):
What is the benefit of the multi-stage pipeline proposed here?

Author response:
In a cold-start scenario, direct application of semi-supervised learning is impractical due to its reliance on some initial labeled data, which is absent in such situations. Our multi-stage pipeline addresses this by enabling the strategic selection of data points for initial annotation, effectively setting the initial annotation budget for semi-supervised learning, and ensuring more efficient use of limited annotation resources.

Reviewer comment (Writing):
Typo in the first paragraph: ”How many labels should be “How many labels...

Author response:
We fixed the mentioned typo.

Reviewer comment (Presentation):
Figure 10 is also hard to understand for the efficiency of model soups on the validation set.

Author response:
Update on the caption of Figure 10: “Figure 10: Comprehensive comparison of model soups performance against varying learning rates across multiple datasets and sampling strategies. The datasets are represented column-wise, while the different sampling techniques are delineated row-wise. Within each experimental setup, we trained the classifier head with five distinct initial weight configurations (depicted on the x-axis) combined with various learning rates (indicated by different shapes in the plot). The results consistently demonstrate that model soups, in most instances, effectively match or surpass the performance achieved by the most optimal learning rate.”

Reviewer comment (Novelty):
Elaborate on Significance: It would be helpful to briefly explain the significance of addressing the "cold start" problem in the biomedical domain. Why is this problem important, and how does your proposed solution contribute to solving it?

Author response:
Thanks for highlighting this point. Now, we emphasize the importance of our work by rewriting the first paragraph in the Introduction section:
“...conundrum known as the cold start problem. The cold start problem refers to the initial phase of training where, in the absence of any labels or prior knowledge about the data, we are tasked with identifying and selecting the most informative data points for annotation, a crucial step that lays the groundwork for any subsequent semi-supervised or fully supervised training.
This is especially critical in the biomedical domain. The scarcity of expert time for manual annotation makes the cold start problem even more daunting, as it becomes a bottleneck in advancing medical AI applications (Yakimovich et al., 2021).  Specifically in active learning and few-shot learning paradigms, previous works by (Boushehri et al. 2021), (ye et al. 2022), and (Jin et al. 2022) demonstrated that careful selection of the initial annotation budget significantly accelerates and facilitates reaching peak performance in models trained on biomedical images with limited annotations like few-shot learning and active learning. Biomedical images significantly differ from natural images…”

Reviewer comment (Evaluation):
Table 1: While furthest point sampling (FPS) outperformed other sampling methods, it exhibited relatively low performance compared to using the full dataset. Providing a discussion or explanation for this performance gap would be valuable.

Author response:
Our table presents the classification results when the model is trained with only 100 annotated data. The fully annotated datasets contain 18345, 25331, 3662, and 32266 data. Thus, it is expected to observe a performance gap, as models typically learn more effectively with more data. The point is that our method narrows down this gap in the initial stage by cleverly selecting the most informative data to be annotated and used for training the classifier. To clarify this point, we mention the limited annotation budget and add the following text to the explanation of Table 1:
“… four datasets (for results on other metrics see Appendix A.1). The key insight from our table is that by judiciously selecting limited annotations, we can significantly narrow this gap from the initial state of active learning. This demonstrates that strategic data selection in the initial phase of active learning can approach the optimal performance achievable with a fully annotated dataset.”

Reviewer comment (Presentation):
Figure 3: The imbalanced distribution of cell types makes it challenging to visualize all cell types.

Author response:
To enhance the clarity and utility of our UMAP visualizations, we will provide an interactive exploration of these drawings on our GitHub (in addition to the code).

Reviewer comment (Evaluation):
Table 2: Regarding the Retinopathy dataset, all sampling methods produced identical results, which were indistinguishable from those obtained through random sampling. Could you explain this?

Author response:
We have already discussed this interesting outcome of our experiments in both text and caption, where we wrote: 
In text:“Our analysis shows how well each approach captures the diversity of classes within the dataset, which is crucial in imbalanced and critical-class scenarios. For instance, in the case of the retinopathy dataset, the latent distribution forms a clustered space where each cluster exhibits a heterogeneous distribution of all classes (see Appendix A.2). As a result, all sampling techniques excel in achieving optimal class coverage during initial data selection. Conversely, the Matek dataset, characterized by a high-class imbalance, features non-uniformly sized homogeneous clusters in the latent space (see Appendix 7). This poses a challenge for most sampling techniques to achieve comprehensive class coverage.”
In the caption: “Table 2: Comparative class coverage analysis across diverse data distributions with annotation budget of 100. Overall datasets, FPS provides the best coverage. All sampling techniques excel in the retinopathy dataset as classes are distributed in clusters (see Figure A.2). Values are mean and standard deviation from five independent runs.”