Paper: Domain Generalization for Domain-Linked Classes (ICLR-2024)

Reviewer comment (Experiments):
Significance: The practical applicability of the research is questionable, given that the empirical validation is conducted on synthetic datasets, which may not effectively simulate real-world complexities.

Author response:
The reviewer is mistaken. The dataset used – PACS, VLCS and OfficeHome are not synthetic and incorporate real-world images as shown in Fig. 2. Moreover, all contemporary domain generalization works use these dataset for evaluation, and therefore our work cannot be evaluated against an unrealistic benchmark.

Reviewer comment (Experiments):
Could you elaborate on the choice of synthetic datasets for validation? How do these datasets simulate the challenges of real-world applications?

Author response:
The reviewer is mistaken. The dataset used – PACS, VLCS and OfficeHome are not synthetic and incorporate real-world images as shown in Fig. 2. Moreover, all contemporary domain generalization works use these dataset for evaluation, and therefore our work cannot be evaluated against an unrealistic benchmark.

Reviewer comment (Novelty):
The theoretical analysis seems to align closely with established domain generalization theories. Could you elucidate the novel contributions of your theoretical insights?

Author response:
We would like to respectfully disagree. Our main theoretical result is the first result in DG literature which illuminates domain-linked DG challenges; this is extremely important in light of recent calls to develop real world DG techniques. Moreover, existing results also assume that all classes are represented in all domains; this does not reflect real-world challenges.

Furthermore, while existing results are mainly focused on theoretical analysis only, our work leverages the analysis to motivate the proposed approach, whose impacts are further corroborated by in the empirical evaluation where we see that higher number of shared classes helps learning for domain-linked learning. Therefore, while our analysis, like other DG results, uses well-known results in the literature [1,2,3], it is extremely non-trivial and establishes the conditions under which domain-shared classes are better-off than domain linked ones.

Reviewer comment (Evaluation):
The FOND algorithm introduces considerable complexity, especially in the loss function. Can you justify this complexity in relation to the performance gains observed?

Author response:
From our theoretical and experimental results we observe that domain-linked classes are inherently disadvantaged. Therefore, our aim is to transfer useful representations from domain-shared ones. 
To prevent the transfer of domain-specific biases FOND imposes that inter-domain positive and intra-domain negative comparisons should be amplified via the $\beta$ and $\alpha$ functions respectively. Even in the absence of fairness we observed that the inclusion of both these parameters (FOND\F) yields a **+1.5%** performance improvement.
To ensure the transfer of useful representations, the FOND objective enforces that learned representations must benefit both domain-shared and domain-linked classes via the fairness objective. This, to effect, encourages the model to ignore domain-specific representations for domain-linked classes and identify generalizable domain-invariant representations from domain-shared classes. Consequently, we observe a **+10%** performance improvement averaged across all datasets and shared-class settings.
As a result FONDS learning objectives are aligned with our theoretical insights, and are required to accomplish domain-linked generalization.

Reviewer comment (Theory):
How does the FOND algorithm ensure the transfer of useful representations between domain-shared and domain-linked classes? Is there a mechanism to prevent the transfer of domain-specific biases?

Author response:
From our theoretical and experimental results we observe that domain-linked classes are inherently disadvantaged. Therefore, our aim is to transfer useful representations from domain-shared ones. 
To prevent the transfer of domain-specific biases FOND imposes that inter-domain positive and intra-domain negative comparisons should be amplified via the $\beta$ and $\alpha$ functions respectively. Even in the absence of fairness we observed that the inclusion of both these parameters (FOND\F) yields a **+1.5%** performance improvement.
To ensure the transfer of useful representations, the FOND objective enforces that learned representations must benefit both domain-shared and domain-linked classes via the fairness objective. This, to effect, encourages the model to ignore domain-specific representations for domain-linked classes and identify generalizable domain-invariant representations from domain-shared classes. Consequently, we observe a **+10%** performance improvement averaged across all datasets and shared-class settings.
As a result FONDS learning objectives are aligned with our theoretical insights, and are required to accomplish domain-linked generalization.

Reviewer comment (Theory):
Given the focus on domain-linked classes, could the proposed method be adapted to scenarios with fewer or no domain-shared classes?

Author response:
This question motivated our decision to create "low" and "high" shared settings. Our methodology shows how domain-shared classes can be used to boost domain-linked performance and our future work involves settings where domain-shared classes cannot be observed (or added) at all.

Reviewer comment (Writing):
In section 5.2, the description of fairness is somewhat unclear. Is $M$ referring to the model, specifically the neural network? If so, it seems that the fairness loss is intended to reduce the classification loss gap between domain-linked and domain-shared classes, suggesting that minimizing the fairness loss aims to make the classification loss for both types of classes have similar values during training. However, it would be helpful to clarify how exactly this loss relates to fairness.

Author response:
Indeed M refers to the model and conventional fairness objectives aim to make the model's performance independent of protected attributes [1-5] (e.g. gender, race, geography). This is formulated as learning objectives that minimize distance/divergence metrics between the model’s output on a certain class with different protected attributes [1]; note this requires a class to be expressed in different protected attributes. However, the reviewer is correct that fairness in our context cannot be applied to domains since some classes cannot be observed in multiple domains. Therefore, we define a fairness objective that seeks to promote representations that benefit both domain-shared and domain-linked classes. We observe that this regularization yields a +10% domain-linked accuracy performance improvement when tested on novel domains.

Reviewer comment (Experiments):
In section A.3 of the appendix, the hyper-parameter selection and model selection process is not quite clear. It references evaluation settings for domains without distinguishing the source and target domains. Does the selection process use the target domain to evaluate the performance?

Author response:
- Model Selection Data: 20% of each source (training) domain held-out during training
- Model Evaluation Data: 100% of the target (testing) domain

The target domain data must not be used for model/hyper-parameter selection since this data is to remain "unseen" and emulate real-world conditions [6].

Reviewer comment (Novelty):
The simplicity of the proposed methodology, which essentially relies on contrastive learning based on domain-shared classes and aligns the losses between domain-linked and domain-shared classes, does seem straightforward. While leveraging information from domain-shared classes to inform domain-linked classes could be beneficial, it's understandable to question whether such loss matching alone suffices to supply rich information.

Author response:
We would like to respectfully disagree. Drawing from contrastive and fairness learning literature to address domain-linked class DG is highly non-trivial. Fairness objectives minimise divergences between a model’s output on a class regardless of protected attribute [4]; this requires a class to be expressed in different protected attributes. However, we cannot compare model performance for a class across domains since some are domain-linked. To overcome these challenges we use loss values across domain-shared and domain-linked classes (eq 5) as a surrogate for fairness. Although counter-intuitive, successful DG approaches do not seek to capture the variances across different domains but to be **invariant** to them; see [1] and the references therein. Therefore, our method draws discriminative representations from domain-shared classes to benefit domain-linked ones.

Reviewer comment (Theory):
The connection between merely aligning loss magnitudes and achieving fairness metrics seems tenuous. A deeper, more nuanced approach might be necessary to ensure that the model not only aligns superficial loss values but also genuinely captures and transfers the underlying complexities and variances of the classes across different domains.

Author response:
We also respectfully disagree that FOND does not genuinely capture and transfer the underlying complexities. On the contrary, our results and ablations consistently demonstrate how FOND significantly outperforms, even when tested on novel domains, in our rigorous experimental set-up. Moreover, the T-SNE plots demonstrate how our methodology is able to transfer discriminative characteristics for domain-linked classes. We also introduce this task to the community to spark further research, an additional goal of our work.

Reviewer comment (Writing):
The term "domain-linked class," used to describe classes that correspond one-to-one with a specific domain, does not seem particularly intuitive. Just recommend utilizing another word.

Author response:
Thank you for the feedback. We also considered 'domain-specific', but this term is also used in the literature in other contexts. Since this task has never been considered, we chose “domain-linked”,  but we are open to other recommendations.

Reviewer comment (Writing):
The likelihood of encountering domain-linked classes in real-world problems may not be immediately apparent or intuitive. Can you provide a clear, real-world example where such classes prominently emerge?

Author response:
Domain-linked classes occur in real-world applications such as defect detection for quality control applications in manufacturing and food inspection.  In these applications, defect types (classes) vary by products (domain), with certain defects being observed only in certain products. This makes defect detection in new future products extremely challenging where, due to unobserved defects in new contexts, the accuracies of defect detection vary by their type. Specifically, since defects are linked to products, their detection in new products suffers since the model learns spurious correlations between the defect and the product. With its focus on overall accuracy, the current domain generalization methods are inadequate. This is where domain-linked domain generalization can greatly alleviate the performance gap by addressing generalization under these data scarcity issues.

Reviewer comment (Writing):
Is it common in this field to define a dataset comprising both inputs and labels as a domain, as done in this paper?

Author response:
Yes, datasets such as PACS, VLCS, and OfficeHome are widely used as different domains in domain generalization literature [1,2,3]. Since DG assumes that source domain identities and labels are given, the assumption is in complete alignment with the literature.

Reviewer comment (Novelty):
(Same as weaknesses) The likelihood of encountering domain-linked classes in real-world problems may not be immediately apparent or intuitive. Can you provide a clear, real-world example where such classes prominently emerge?

Author response:
Domain-linked classes occur in real-world applications such as defect detection for quality control applications in manufacturing and food inspection.  In these applications, defect types (classes) vary by products (domain), with certain defects being observed only in certain products. This makes defect detection in new future products extremely challenging where, due to unobserved defects in new contexts, the accuracies of defect detection vary by their type. Specifically, since defects are linked to products, their detection in new products suffers since the model learns spurious correlations between the defect and the product. With its focus on overall accuracy, the current domain generalization methods are inadequate. This is where domain-linked domain generalization can greatly alleviate the performance gap by addressing generalization under these data scarcity issues.