Paper: Distribution Aware Active Learning via Gaussian Mixtures (ICLR-2024)

Reviewer comment (Novelty):
As mentioned in the weakness, the fundamental problem of this paper is that it is merely a special case of a solved general problem, which makes the novelty of the paper very limited. Without detailed discussion of the difference from the existing work as listed, the novelty of the paper does not stay on a safe ground.

Author response:
## Rebuttal 4 (ID: 3WW6ljwKDV)
### **Further Comment on the novelty of the proposed method**

The novelty of our work lies in the **incorporation of GMM into training and selection methods that aim to minimize the discrepancy between $X_L$ and $X_{UL}$,** thereby mitigating the risk of overfitting to small $X_L$. Particularly, our approach involves the **estimation of latent distributions of $X_L$ and $X_{UL}$ using GMM**, a novel feature not observed in existing AL works. GMM is an useful tool for active learning since it can estimate the original data distribution even with the absence of certain categories in $X_L$, by leveraging the rich data in $X_{UL}$. Our proposed GMM-aided methods pose distinctive contributions that differ from prior works, which either train the task model using only $X_L$ or indirectly estimate the latent distribution without using probabilistic models like GMM. **Also, as demonstrated in Table 1 in the main paper, our GMM-aided components can be easily integrated into existing works, potentially enhancing performance.**


Again, we appreciate the reviewer for the time and effort. Your concerns are clear, and we feel we have successfully addressed all the issues you raised. Please let us know in case there are remaining questions/concerns; we hope to be able to have an opportunity to further answer them.

Reviewer comment (Theory):
The author needs to show why the GMM model is a good approximation of the data distribution in active learning as there can be many outliers and unknown classes. How about if the data is a long-tail distribution. The assumption of the paper is a bit too strong.

Author response:
## Rebuttal 2 (ID: ZwyELKg60l)
### **Rationale behind the choice of GMM as an approximation model**
Compared to other clustering algorithms such as K-means clustering, agglomerative clustering, and DBSCAN, GMM has the advantage of considering the desired number of modes along with the variance; it allows GMM to capture complex patterns in data with non-linear or non-spherical cluster shapes. Additionally, GMM is effective when clusters have overlapping boundaries due to its ability to allow for soft assignment, which takes into account ambiguity or uncertainty. Due to these advantages, GMM has been proven to be effective for the approximation of general real-world data (e.g., ImageNet, Cityscapes) in existing works [3,4]. Adopting GMM thus enables us to directly and accurately calculate the latent distribution as probabilistic models, providing a quicker and more reliable acquisition of distributional information compared to previous distributional alignment based AL approaches [5, 6] that indirectly infer the latent distribution by relying on pairwise distances or the output of additional modules.

[3] Lee et al., "Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-Learning." ICLR 2020.  
[4] Liang et al., “GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models”, NeurIPS 2022.  
[5] Fu et al., "Agreement-Discrepancy-Selection: Active Learning with Progressive Distribution Alignment", AAAI 2021.  
[6] Shui et al., "Deep Active Learning: Unified and Principled Method for Query and Training", AISTATS 2020.

### **Insights into the advantage of our method in realistic settings (e.g., unseen classes & long-tail distribution)**

1) Proposed likelihood informativeness ($I_{Like}$ in equation 4) can be calculated without any modification of our algorithm, even when the number of categories in $X_L$ and $X_{UL}$ differs (i.e., the number of modes in two GMMs is different). This is because we compare the likelihood score computed from each GMM. In a realistic imbalanced data setting, the likelihood difference between two GMMs ($\psi_L$, $\psi_{UL}$) is large around minor/unseen classes (i.e., large likelihood from $\psi_{UL}$ & low likelihood from $\psi_L$). Thus, the proposed informativeness score $I_{Like}$ prioritizes samples from minor/unseen classes to be selected, effectively addressing the imbalance problem.
2) Our semi-supervised learning with GMM also works well in an imbalanced setting. This is because, even if there are no or few samples for certain classes in $X_L$, GMM can capture modes for all classes based on information from $X_{UL}$. Therefore, our GMM-aided semi-supervised learning does not suffer from data imbalance, but rather encourages distributional alignment with the aid of GMM statistics from $X_{UL}$.

Reviewer comment (Novelty):
The paper did not add any new insights to this field. The results are merely a small extension and special cases of existing approaches.

Author response:
## Rebuttal 5 (ID: YYOHQQeclc)
We appreciate your response. We would like to make the following points clearer:       


  
### **`Generality/applicability of our work`** 
 
 We respectfully reiterate that **our algorithm is applicable to general AL settings**, and the **use of GMM does not limit the applicability** of our work. We also do not assume any special cases. To support these, we have provided comprehensive responses, conducted extensive experiments under the same/fair setting as other AL baselines in the manuscript, and showed general superiority in various datasets and AL scenarios **(balanced, imbalanced, long-tail, new classes)**, including the ones that the reviewer requested.

 

### **`Open-set annotation`**
 
 In the original review, we believe that the reviewer didn’t mention the open-set annotation setting but instead suggested the “new classes” and “long-tailed” setup. This is why we have focused on the experiments mentioned above, and have explained why our scheme still works well with clear insights.
 
 **Open-set annotation is a recently proposed new AL setting [CVPR’22] that deviates from the typical AL setting**, and there is still limited literature on it. It is assumed that the human annotator is not able to label certain unknown classes that are irrelevant to the main task. **Open-set annotation and conventional AL settings target different applications with different focuses**: For example, as provided in [CVPR’22], during AL for sports image classification, a large portion of images in the unlabeled set could be irrelevant to the main task (e.g., house, dogs). Open-set annotation setting will become important here. **However, the conventional AL setting (w/o open-set annotations) is still important in many cases**: In healthcare applications such as disease prediction, which is a primary application of AL, all images in the unlabeled set are directly related to the main task (e.g., X-ray image from the patient), which requires us to consider the conventional AL setup. 
 
 To handle the new open-set setting, an orthogonal approach to existing AL strategies is required: As we can see from [CVPR’22], all existing AL methods do not work well in this setting. Although this setup has not been actively studied, and unexplored, we can potentially address the challenge by orthogonally introducing an OOD sample detector as in [CVPR’22] to exclude OOD samples from the unlabeled set, and then apply our GMM-based algorithm. 
 
 [CVPR’22] Active Learning for Open-set Annotation

 
### **`Novelty`**
 
 We would like to highlight that **the key philosophy of our work does not lie in distributional alignment (DA)**. DA (in Section 3.4) is just one of our components, and our contribution spans from GMM modeling (in Section 3.2) to regularization (in Section 3.3) and informativeness metrics (in Section 3.5), encompassing the entire AL scheme. **The core of this work is the use of GMM to better capture the dataset distribution and build a unified framework** for "distribution-aware AL" that facilitates effective regularization, effective DA-based learning, and effective informativeness-based sampling. We successfully integrate GMM into both the training and selection stages, aiming to identify and mitigate the discrepancies between the labeled and unlabeled sets.  


 During experiments, we also showed that **GMM can enhance the performance of various existing DA methods** and provided related analyses in Figure 5 of the manuscript and Section A.6 of the Appendix.  Moreover, as illustrated in Figure 12, **each proposed component exhibits a progressive enhancement** in performance. Furthermore, our compatibility analysis in Table 1 demonstrates that **our components can be easily plugged into the existing methods, leading to notable performance improvements**.
 
Importantly, our work introduces a mathematical modeling of datasets employing GMM to enhance AL performance effectively.

Reviewer comment (Evaluation):
Some intuitive explanation to demonstrate the superiority would be great. For example, with each active learning cycle, for each class, how the selected samples from the proposed method are different from the ones from baselines? Or use the selected samples to verify the claim of 'high quality and diverse'.

Author response:
### **Intuitive explanation on the superiority of our method over other baselines.**

In response to the reviewer’s comment, in **Fig. 20 of Section A.10 in Appendix**, we illustrate the candidate set selected by various AL methods under imbalanced CIFAR10. From the t-SNE visualization of candidate sets, we can intuitively compare the properties of various AL methods. For instance, Snapshot_AL [1] that calculates uncertainty using the output of the classifier exhibits redundancy by selecting multiple samples from narrow regions. Coreset [2] that considers only diversity can select diverse samples in broad areas but it lacks the ability to specify certain informative regions. CoreGCN [3] that favors samples expected to be from unlabeled set, is observed to select samples from dense areas of $X_{L}$, as well as outlier samples that hardly represent $X_{UL}$. In contrast, the candidate set from the proposed method demonstrates desired properties: **1) it is diverse enough to encompass a broad area** while **2) represents the distribution of $X_{UL}$ well. Our method compares GMMs of $X_L, X_{UL}$ to select samples that are unfamiliar in $X_L$, but representative of $X_{UL}$**; especially in the imbalanced setting, this comparison leads to select minor classes that are rare in $X_L$, thereby contributing to the significant performance gain as seen in Fig.10 of the main paper. 

Additionally, we reported the class count entropy of $X_L$ under an imbalanced setting in Figure 11 (b) of the original manuscript (i.e., higher entropy corresponds to diverse $X_L$). The results show that the proposed method actively selects minor classes, exhibiting the highest entropy. 

[1] Jung et al., “A Simple Yet Powerful Deep Active Learning With Snapshots Ensembles”, ICLR 2023.  
[2] Sener et al., “Active Learning for Convolutional Neural Networks: A Core-Set Approach”, ICLR 2018.  
[3] Caramalau et al., “Sequential graph convolutional network for active learning”, CVPR 2021.  

We again appreciate the reviewer for providing helpful comments. We would be grateful if you could let us know if there are any other comments/questions.

Reviewer comment (Theory):
Regarding Equation (3), the optimization seems to aim at minimizing the disparity between the unlabeled data, \(X_{UL}\), and the labeled data, \(X_{L}\), within the embedding space. If the model is trained effectively, \(X_{UL}\) and \(X_{L}\) would become indistinguishable in the embedding space, raising the question of whether this is the intended outcome, as it seems important for the embedding space to retain the information that differentiates \(X_{UL}\) from \(X_{L}\).

Author response:
### **Rationale behind Equation (3)**
As the reviewer mentioned, the goal of Equation (3) is to perform adversarial training based on GMM information to reduce the distribution discrepancy between $X_L$ and $X_{UL}$ in latent space, and it offers important advantages in AL. This is because models are susceptible to overfitting due to the limited size of $X_L$ in the early stages of AL (Distinct difference between the distributions of $X_L$ and $X_{UL}$ can be seen in Fig. 1 of the main paper). 
Considering the nature of AL, we believe it is almost impossible for Equation (3) to align $X_L$ with $X_{UL}$ to be indistinguishable in the embedding space during challenging early AL cycles. As pointed out by the reviewer, there might be a moment when it is possible for Equation (3) to align $X_L$ and $X_{UL}$ when the size of $X_L$ grows enough. However, this implies that training the model using only $X_L$ can achieve performance similar to using $X_L$ + $X_{UL}$, overshadowing the need to apply active learning.

Reviewer comment (Experiments):
The combination method outlined in Equation (5) is not entirely convincing as the best approach. An alternative sampling strategy might involve drawing from each of the three separate rankings and then using the highest-ranked samples from each ranking as the data set for annotation. Additionally, it would be insightful to see the results of an ablation study where each component of Equation (5) is used independently as the final selection criterion.

Author response:
### **Ablation study on Equation (5)**
We thank the reviewer for the insightful comments. As can be seen in Fig. 8 (a) and (c) of the main text, two metrics ($I_{Like}$, $I_{Ent}$) exhibit strong redundancy in their candidate set. Therefore, separately considering each metric ($I_{Like}$, $I_{Dis}$, $I_{Ent}$) can lead to redundant and overlapping selections. However, when considering a combined indicator as in our method ($I_{Total}$), one can select diverse yet informative samples as shown in Fig. 8 (d). In response to the reviewer’s comment, we conducted an additional experiment in which data selection is performed based on separate rankings for each informativeness score ($I_{Like}$, $I_{Dis}$, $I_{Ent}$). Through the table below, we confirmed that our integrated indicator achieves higher performance than using each independent indicator.

- **Table 1**: Comparison with the sampling strategy based on separate rankings for each informativeness score on CIFAR10

| # of labeled samples | 1K    | 2K    | 3K    | 4K    | 5K    | 6K    | 7K    | 8K    | 9K    | 10K   |
|----------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| Random               | 51.76 | 63.50  | 72.17 | 76.94 | 80.56 | 82.32 | 84.17 | 85.00    | 86.11 | 87.27 |
| Separate rankings for ($I_{Like}$, $I_{Dis}$, $I_{Ent}$)           | 58.98| 71.79| 78.79| 82.68| 84.96| 86.31| 87.69| 88.63| 89.37| 90.08 |
| DAAL                 | **58.98** | **74.15** | **79.21** | **83.42** | **85.63** | **86.75** | **88.09** | **89.39** | **90.01** | **90.87** |