Paper: Overcoming Data Inequality across Domains with Semi-Supervised Domain Generalization (ICLR-2024)

Reviewer comment (Experiments):
The datasets used in the experiment are all simple and small-scale.

Author response:
>The datasets used in the experiment are all simple and small-scale

While they seem relatively simple and small-scale, PACS, Digits-DG, or Office-home have been used as important benchmark datasets in many of the recent papers published in major ML conferences, such as CVPR([1,2,3]), ICCV([4,5,6]), or ICLR([7,8]).

[1] Shiqi Lin, Zhizheng Zhang, Zhipeng Huang, Yan Lu, Cuiling Lan, Peng Chu, Quanzeng You, Jiang Wang, Zicheng Liu, Amey Parulkar, Viraj Navkal, and Zhibo Chen. "Deep frequency filtering for domain generalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. "Rethinking federated learning with domain shift: A prototype view." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Jin Chen, Zhi Gao, Xinxiao Wu, and Jiebo Luo. "Meta-causal Learning for Single Domain Generalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Sheng Cheng, Tejas Gokhale, and Yezhou Yang. "Adversarial Bayesian Augmentation for Single-Source Domain Generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Xiran Wang, Jian Zhang, and Lei Qi. "Generalizable Decision Boundaries: Dualistic Meta-Learning for Open Set Domain Generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Chaoqi Chen, Luyao Tang, Leitian Tao, Hong-Yu Zhou, Yue Huang, Xiaoguang Han, and Yizhou Yu. "Activate and Reject: Towards Safe Domain Generalization under Category Shift." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[7] Ziqiao Wang, and Yongyi Mao. "Information-Theoretic Analysis of Unsupervised Domain Adaptation." The Eleventh International Conference on Learning Representations. 2022.

[8] Yan Yan, and Yuhong Guo. "Partial Label Unsupervised Domain Adaptation with Class-Prototype Alignment." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
On some datasets, the performance improvement compared to EID is relatively small, and the choice of random seed is not general enough.

Author response:
>On some datasets, the performance improvement compared to EID is relatively small, and the choice of random seed is not general enough

Although the margin of performance improvement may seem relatively small, we would like to emphasize that the major advantage of ProUD over EID lies in its scalability. EID requires separate domain adaptation models and their respective training processes for each unlabeled source domains, to address SSDG. This imposes a significant limitation on its practical application to SSDG problems with a large number of source domains. The same challenge applies to attempts to exploit different combinations of SoTA models for DA and DG. Therefore, our single model-based approach offers a significant practical advantage in handling numerous source domains in a scalable manner.

We would also like to clarify that the choice of random seed has been made with no intention of influencing, or any relation to, the experimental results. Please note that the standard deviations of the average accuracy over different seeds are marginal, and the lower bounds of the average accuracy $(\mu- \sigma)$ still outperform all the baselines.
| Seed |  PACS | Office-Home | Digits-DG |
|:----:|:-----:|:-----------:|:---------:|
| 2022 | 75.08 |    56.33    |   74.37   |
| 2023 | 75.09 |    56.24    |   74.26   |
| 2024 | 75.02 |    56.28    |   74.10   |
| Avg. | 75.06 |    56.28    |   74.24   |
| Std. |  0.03 |     0.04    |    0.56   |

Reviewer comment (Experiments):
There is only one partition for each dataset, and the proportions are not uniform (9:1 and 8:2). The author did not provide an explanation for the choice of different partitions.

Author response:
>There is only one partition for each dataset, and the proportions are not uniform (9:1 and 8:2). The author did not provide an explanation for the choice of different partitions.

For the partitioning of each dataset, we followed EID for fair comparison.

Reviewer comment (Experiments):
The algorithm uses prototypes obtained from soft labels when acquiring pseudo-labels (Equation 1), and prototypes obtained from hard labels are used in subsequent calculations of uncertainty and loss functions (Equation 3). What is the reason for these choices, and what is the difference in effectiveness between the two types of prototypes?

Author response:
>The algorithm uses prototypes obtained from soft labels when acquiring pseudo-labels (Equation 1), and prototypes obtained from hard labels are used in subsequent calculations of uncertainty and loss functions (Equation 3). What is the reason for these choices, and what is the difference in effectiveness between the two types of prototypes?

The alternating use of soft and hard labels is followed by the previous work [1]. Our contribution involves extending the pseudo-labeling method of Liang et al. to be effective in Semi-Supervised Domain Generalization (SSDG) by leveraging domain-aware class prototypes.

[1] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pp. 6028–6039. PMLR, 2020.

Reviewer comment (Experiments):
What is the improvement brought about by data augmentation, and how would the results compare if all algorithms used the same number of data augmentations in the comparison?

Author response:
>What is the improvement brought about by data augmentation, and how would the results compare if all algorithms used the same number of data augmentations in the comparison?

We conducted an additional experiment on the Digits-DG dataset, reducing the number of data augmentations from 3 to 1. The average accuracy decreased by 0.24, yet this performance still outperforms all the baselines.

Reviewer comment (Experiments):
The paper misses many important benchmark datasets in the experiments section, such as VLCS, TerraInc, and DomainNet, as detailed in DomainBed [1]. In particular, DomainBed is an important benchmark.

Author response:
>The paper misses many important benchmark datasets, such as VLCS, TerraInc, and DomainNet. See detail in DomainBed [1]. In particular, DomainBed is an important benchmark.

PACS, Digits-DG, and Office-home have been predominantly used as key benchmarks without including VLCS, TerraInc, or DomainNet, in many recent papers published in major ML conferences,  such as CVPR [6,7,8], ICCV [9,10,11], or ICLR [12,13].

Reviewer comment (Experiments):
The paper fails to compare with many state-of-the-art methods; the latest works compared in the paper are published in 2021. There are many good works from 2022 and 2023 that should be included, such as [2,3,4] and others.

Author response:
>The paper fails to compare lots of state-of-the-art methods. The latest works compared in the paper are published in 2021. There are lots of good works in 2022 and 2023 that should be included, such as [2,3,4] and many more.

SSDG is a new problem setting that has not been extensively discussed previously.  The works [2,3,4] you mentioned are effective SoTA methods in DG. However, those methods are primarily multi-source domain generalization approaches, which are not directly applicable to SSDG, due to the absence of labels on most source domains. Instead, here we present an additional comparison of our method with the latest SoTA methods on single-source DG on the PACS dataset, including the one published in CVPR 2023 [8], thereby demonstrating the effectiveness of our method:

|    Method    |   P  |   A  |   C  |   S  | Avg. |
|:------------:|:----:|:----:|:----:|:----:|:----:|
| RSC+ASR [14] | 54.6 | 76.7 | 79.3 | 61.6 | 68.1 |
|   MCL [8]   | 59.6 | 77.1 | 80.1 | 62.6 | 69.9 |
|     Ours     | 70.5 | 81.0 | 81.3 | 67.4 | 75.1 |

Reviewer comment (Evaluation):
The results in Table 2, Table 3, and Table 4 do not show that ProUD outperforms EID by a large margin, leading to a lack of conviction that the proposed method ProUD is useful.

Author response:
>The results in Table 2, Table 3, and Table 4 cannot even show that ProUD outperforms EID by a large margin. I am not convinced that the proposed method ProUD is a useful method.

Beyond its superior performance, it is important to highlight ProUD's major advantage over EID [15] in terms of scalability. EID requires a separate domain adaptation model and training process for each unlabeled source domain, which becomes impractical for SSDG problems involving many source domains. This limitation also applies to attempts at combining different SoTA models for DA and DG. In contrast, ProUD utilizes a single model capable of handling multiple source domains through its progressive generalization algorithm in a scalable manner.

Reviewer comment (Theory):
The proposed methods need domain labels for domain-aware prototypes. This requirement will restrict the applicability of the methods in practice. It would be good to consider whether there is a way to generalize the method to be domain label-free.

Author response:
>The proposed methods need domain labels for domain-aware prototypes. This will restrict the methods when applied to the application. It would be good to consider whether there is a way to generalize the method to be domain label-free.

SSDG is a new problem setting that assumes the presence of implicit domain labels. This setting is realistic in many scenarios as illustrated in Table 1. For instance, in biomedical imaging, lots of hospitals (domains) obtain medical image data but only few of them undergo additional process for labeling the images. Our method aims to present how to effectively utilize data from unlabeled domains along with implicit domain labels in a scalable manner. Existing methods for semi-supervised learning can also be applied for SSDG, but it is difficult for them to be as effective due to their inability to utilize available domain labels, as presented in our experimental results (Table 2,3, and 4).

Reviewer comment (Novelty):
The methods are simple and trivial. In particular, the PML loss (Equation 8) appears to be quite similar to SupCon [5].

Author response:
>The methods are simple and trivial. In my understanding, PML loss (Equation 8) is pretty similar to SupCon [5].

Contrastive learning itself is a conventional approach, and various studies have derived their own methods for applying this framework to their specific problem settings. The Supervised Contrastive Loss in SupCon [5] is designed to attract features of samples within the same class to each other, yet it lacks consideration for different domains, which is crucial in SSDG. 

On the other hand, PML aims to attract the features of samples towards their corresponding $\bar{C}_k$, which represents the average of the same class prototypes from all source domains. Merging the class prototypes from different domains not only enhances the model’s generalization ability by making features domain-invariant, as visualized in Fig. 2 (further analysis is added in Appendix C of the revised manuscript), but also offers a significant advantage in representing under-represented domains. This approach prevents representation bias caused by the uneven distribution of samples across domains.

Reviewer comment (Theory):
It would be beneficial if the paper could include some theoretical analysis to provide more insights or intuition about ProUD.

Author response:
>It would be good if the paper could introduce some theoretical analysis to give more insights or intuition about the ProUD.

In order to clarify the intuition about the ProUD, we would like to provide a theoretical rationale behind our work. Semi-supervised domain generalization, a specialized form of semi-supervised learning, is particularly relevant in scenarios where labeled and unlabeled data exhibit differing distributions or domains. To address this issue, we utilize the measure of uncertainty inherent in each pseudo-label to modulate the involvement of each sample in the training process. 

Specifically, we employ MixUp between labeled and unlabeled data, applying a reduced mixing ratio in domains with higher uncertainty. By assigning a low mixing ratio to samples with high uncertainty, we prevent these unreliable samples from overly influencing the training process, thereby allowing for a more progressive integration of unlabeled domains. This strategy successfully addresses the scalability issues found in the previous state-of-the-art method [15], as we do not require a distinct DA model for each domain. 

However, directly applying MixUp to domain generalization can be problematic, as it does not distinguish between domain and class information. This entanglement of domain and class knowledge can result in performance degradation, as identified in [16]. To circumvent this issue, we first limit the application of MixUp to images from identical classes (DomainMix) and implement the Prototype Merging Loss, which reduces intra-class distance across varying domains to enhance generalization capabilities, while simultaneously, maintaining a clear separation between data points of different classes for effective discrimination. This approach prevents the entanglement of domain and class knowledge, enabling the effective learning of domain-invariant features that perform class discrimination across datasets from different domains.

Reviewer comment (Novelty):
**[Motivation of this proposed method is unclear.]** It is unclear why the authors propose this method. For example, if there exists a research gap for current studies on SSGD?

Author response:
[The proposed method is lack of novelty.] Essentially, the proposed method is still a combination of DA +DG. The step for assigning pseudo labels for unlabelled source domains can be regarded as DA, while the step for learning domain-invariant representations via a contrastive loss can be regarded as DG. Tools used in each step are also widely used in methods for DA and DG basically 

[Motivation of this proposed method is unclear.] It is unclear why the authors propose this method. For example, if there exists a research gap for current studies on SSGD?

The current state-of-the-art in SSDG [1], requires as many separate domain adaptation models and training processes as there are unlabeled source domains. This requirement imposes a significant limitation on its practical application to SSDG problems with a large number of source domains. The same obstacle applies to any attempts to exploit different combinations of state-of-the-art models for DA and DG. 

However, ProUD’s contribution lies in overcoming the limitations of previous methods that encounter scalability problems. ProUD’s progressive generalization algorithm allows for handling multiple unlabeled source domains simultaneously with a single, scalable model through the effective use of a measure of uncertainty based on domain-aware prototypes. 

Furthermore, our DomainMix method not only modulates the involvement of samples from each unlabeled domain based on their uncertainty but also enhances the diversity of samples from under-represented unlabeled domains by augmenting them through an adaptive mix with labeled samples. The Prototype Merging Loss (PML), another key component of ProUD, constructs mean prototypes ($\bar{C}_k$) by assigning equal weights to the same class prototypes from each domain, irrespective of the number of samples in each domain. This approach offers a significant advantage in representing under-represented domains, preventing representation bias caused by uneven sample distribution across domains.

Reviewer comment (Theory):
**[The propsed method is not explored deeply.]** Firstly, this paper does not provide a theoretical analysis to certify the effectiveness of the proposed method. Furthermore, it is unclear how the accuracy of pseudo-labels affects the final performance. In detail, in Equation (6), you mix labeled and unlabeled samples with the same class up, whose performance may heavily rely on the quality of the pseudo labels. Additionally, in the t-SNE Visualization, I want to know if Equation (8) contributes to the representation most.

Author response:
[The proposed method is not explored deeply.] Firstly, this paper does not provide a theoretical analysis to certify the effectiveness of the proposed method. Furthermore, it is unclear how the accuracy of pseudo-labels affects the final performance. In detail, in Equation (6), you mix labeled and unlabeled samples with the same class up, whose performance may heavily rely on the quality of the pseudo labels. Additionally, in the t-SNE Visualization, I want to know if Equation (8) contributes to the representation most.

To assess the impact of Prototype Merging Loss (PML) on t-SNE visualization, we included an additional t-SNE visualization in Appendix C of the revised draft. This visualization illustrates the performance of our algorithm without the implementation of PML. In contrast to the distinct clustering observed in Fig. 2, the samples are only loosely clustered around their respective prototypes here, resulting in more ambiguous cluster boundaries than those seen with PML. Moreover, several prototypes remain intermingled, demonstrating a lack of distinct discrimination. These observations underscore the vital role of PML, as formulated in Equation (8). PML enhances the algorithm's ability to learn domain-invariant representations by effectively merging prototypes of the same class from diverse domains, while simultaneously ensuring their separation from prototypes of different classes.

Reviewer comment (Experiments):
How do you choose the value of the threshold $\lambda^*$?

Author response:
We chose the value of the threshold $\lambda^∗$ from the following set of values {0.3, 0.35, 0.4, 0.45} by evaluating on validation dataset. However, we found that the value $\lambda^∗$ could be fixed to 0.35, as the same value consistently yielded good results across all datasets and experiments.

Reviewer comment (Theory):
It is not clear why the architecture only learns a single model, and what is the motivation behind it. Although there may be domain-invariant features that can be learned from multiple sources, different features may provide different information depending on the domain.

Author response:
Thank you for your insights. I agree that it is feasible to train a single feature extractor, $g$, along with different classifiers, $h_i$, for distinct source domains. This approach resembles a multi-task learning setting, albeit focused on multi-domains rather than multi-tasks, which may lead to optimal performance for each domain with its corresponding classifier. However, it is important to note that the ultimate goal in SSDG is Domain Generalization, aiming to perform well on “unseen” domains (e.g., after training with domains $A$, $B$, and $C$, we evaluate test performance on domain $D$). If we train multiple classifiers, such as {$h_A$, $h_B$, $h_C$}, we face the issue of deciding which classifier to use for a newly provided domain $D$. 

On the other hand, you might suggest utilizing different domain adaptation models for each unlabeled domain to generate pseudo-labels, so we can apply domain generalization to pseudo-labeled datasets. Indeed, this approach has been proposed in the previous work [1] on SSDG. However, it requires separate domain adaptation models and their respective training processes for each unlabeled source domain, to address SSDG. This imposes a significant limitation on its practical application to SSDG problems with a large number of source domains. The same challenge applies to attempts to exploit different combinations of state-of-the-art (SoTA) models for Domain Generalization (DG) and Domain Adaptation (DA). Therefore, our single model-based approach offers a significant practical advantage in handling numerous source domains in a scalable manner.

Reviewer comment (Theory):
Using the example of two features f_A and f_B, domain A may achieve very good classification from feature f_A (and not f_B), while domain B may achieve very good classification from feature f_B (and not f_A). In such a case, it would make more sense to build different models that can exploit different features across different domains. In the proposed algorithm, this means keeping the feature extractor g the same but building a function h for each different domain (similar strategies are also applied in ZSL).

Author response:
Thank you for your insights. I agree that it is feasible to train a single feature extractor, $g$, along with different classifiers, $h_i$, for distinct source domains. This approach resembles a multi-task learning setting, albeit focused on multi-domains rather than multi-tasks, which may lead to optimal performance for each domain with its corresponding classifier. However, it is important to note that the ultimate goal in SSDG is Domain Generalization, aiming to perform well on “unseen” domains (e.g., after training with domains $A$, $B$, and $C$, we evaluate test performance on domain $D$). If we train multiple classifiers, such as {$h_A$, $h_B$, $h_C$}, we face the issue of deciding which classifier to use for a newly provided domain $D$. 

On the other hand, you might suggest utilizing different domain adaptation models for each unlabeled domain to generate pseudo-labels, so we can apply domain generalization to pseudo-labeled datasets. Indeed, this approach has been proposed in the previous work [1] on SSDG. However, it requires separate domain adaptation models and their respective training processes for each unlabeled source domain, to address SSDG. This imposes a significant limitation on its practical application to SSDG problems with a large number of source domains. The same challenge applies to attempts to exploit different combinations of state-of-the-art (SoTA) models for Domain Generalization (DG) and Domain Adaptation (DA). Therefore, our single model-based approach offers a significant practical advantage in handling numerous source domains in a scalable manner.

Reviewer comment (Theory):
In the data inequality model, some data sources may have a different number of examples. In particular, there could be comparatively fewer data points for a given unlabeled domain. The model appears to give the same weight to each sample from each domain, so if an unlabeled domain t is underrepresented (fewer data N_t), the model would still suffer from a data inequality issue, as the loss would be less influenced by the fewer samples from this domain.

Author response:
We appreciate your constructive feedback. This is an important point that has been under-emphasized in our paper, yet it has been successfully addressed with our method. Our DomainMix method not only modulates the involvement of samples from each unlabeled domain based on their uncertainty, but also enhances the diversity of samples from under-represented unlabeled domains by mixing them with labeled samples. A similar approach has proven effective in previous work on few-shot domain adaptation [2], which paired a few target domain samples with a large number of source domain samples to learn domain-invariant features across both domains. Furthermore, our Prototype Merging Loss (PML) constructs  mean prototypes ($\bar{C}_k$) by assigning equal weights to the same class prototypes from each domain, regardless of the number of samples each domain contains. This offers a significant advantage in representing under-represented domains, as it prevents representation bias due to the uneven sample distribution across domains.

Reviewer comment (Evaluation):
It is unclear why the accuracy is averaged over the last 5 training epochs rather than only on the last epoch.

Author response:
We report the average accuracy over the last five training epochs, following the approach of EID [1], the state-of-the-art baseline for SSDG, to ensure a fair comparison. For our method, the difference between the average accuracy of the last five epochs and the accuracy of the final epoch is negligible.

Reviewer comment (Experiments):
There is no explanation of how the hyper-parameters are chosen for the proposed algorithms (e.g., whether there is a validation step), or how the hyper-parameters are chosen for the baseline algorithms. This is important when evaluating methods on a new dataset, to avoid overfitting due to hyper-parameter selection.

Author response:
We split source domain dataset into training and validation sets, using the validation set for hyper-parameter tuning. The evaluation is conducted on an unseen test domain dataset.

Reviewer comment (Evaluation):
The standard deviation for the average accuracy is not reported. Although an average accuracy over 3 runs (with 3 different seeds) is provided, the variance of this value is not clear, which is important for comparison with other methods. The reported standard deviation is across domain combinations rather than across the 3 seeded runs.

Author response:
The standard deviations of the average accuracy over 3 seeds are marginal, and the lower bounds of the average accuracy $(\mu- \sigma)$ still outperform all the baselines. The detailed experimental results are presented below.
| Seed |  PACS | Office-Home | Digits-DG |
|:----:|:-----:|:-----------:|:---------:|
| 2022 | 75.08 |    56.33    |   74.37   |
| 2023 | 75.09 |    56.24    |   74.26   |
| 2024 | 75.02 |    56.28    |   74.10   |
| Avg. | 75.06 |    56.28    |   74.24   |
| Std. |  0.03 |     0.04    |    0.56   |