Paper: Understanding and Robustifying Sub-domain Alignment for Domain Adaptation (ICLR-2024)

Reviewer comment (Novelty):
While the theory behind sub-domain alignment is certainly compelling, in its practical application, it merely utilizes class labels to segment subdomains. This results in a method that essentially merges class importance weighting with W1 distance to gauge domain discrepancies. While this is a fresh approach, it's not exceptionally innovative. It would enhance the paper if the author explored alternative sub-domain segmentation techniques.

Author response:
We would like to clarify that the primary focus of this work was not on algorithmic novelty, but rather on addressing a critical challenge observed in imbalanced scenarios. Our analysis revealed that subdomain-based methods tend to falter under such conditions. To counter this, we introduced an algorithm with **theoretical guarantee** to enhance the robustness of sub-domain based methods. The effectiveness of our approach is not just theoretical but is also supported by empirical evidence from our study. With the theoretical foundation provided by our work, we expect following works with more advanced algorithms.

Reviewer comment (Experiments):
It would enrich the study if the author included baselines that are specifically designed for label shift scenarios, like the method outlined in [1]. Many of their referenced baselines, such as DANN and ADDA, are primarily constructed for the covariate shift setting.

Author response:
**Baselines.** We have indeed included baselines specifically designed for label shift scenarios, such as Cluster Alignment with a Teacher (CAT). Since we are a theory-focused paper to theoretically understand the effectiveness of sub-domain based methods, our selection also includes theoretically robust methods such as DANN as well as top-performing methods in digit transfer learning tasks, recognized on the Papers with Code website [2]. It's worth noting that most of our domain adaptation accuracy is strong (over 90%), which naturally limits the margin for further improvement by adding additional benchmarks.
We address a different level of imbalance compared to [1]. In [1], the focus is on the original class distributions of digit datasets, highlighting biases across domains (for example, the higher weights of classes 0 and 1 in USPS, and 1 and 2 in SVHN). These scenarios can be considered as **no additional label distribution shifts**, which is significantly different from the experiment setup in our work. In our work, we modified digit datasets to induce label distribution shifts. Here, the parameter $\alpha$ denotes the class imbalance rate, representing a ratio such as 1:$\alpha$ and $\alpha$:1 for the odd:even distribution in the source and target datasets, respectively. And we evaluated both weak ($\alpha=3$) and strong imbalance scenarios (($\alpha=8$)).
In line with your suggestion, we have demonstrated experiments with **no additional label distribution shifts** for a more direct comparison of DARSA with other methods. The results, presented in the subsequent tables, clearly demonstrate that DARSA significantly outperforms [1], reinforcing the effectiveness and robustness of our approach in these specific scenarios.
|                       |     USPS to MNIST         |  SVHN to MNIST  | 
| ------------------:| :------------------------------:| :----------------------:| 
|  DANN     |   	  74.5 	                      |   73.9                 | 
| **WDAN**  [1]     |    65.4	                      |   67.4                 | 
|  WDGRL |      84.8                           |    59.3                  |
|  DSN        |      91.0                           |    82.7                |
|  ADDA       |      90.1                        |    76.0                |
|  CAT          |      80.9                         |   98.1                 |
|  CDAN   |     98.0                         |   89.2                 |
|  pixelDA   |     87.6                           |   71.6                   |
|  DRANet  |     97.8                        |   59.7                   |
|  DARSA          |     97.4                          |    98.6                   |

Table: Performance of DARSA and baseline methods without label shifting.

Reviewer comment (Experiments):
Most of the compared UDA methods were published before 2020. To fully validate the superiority of the proposed DARSA model, more SOTA UDA methods should be included in comparison experiments.

Author response:
**Rationale Behind Baseline Selection:**  Regarding the selection of compared methods, our benchmarks were meticulously chosen to demonstrate the robustness and effectiveness of our proposed algorithm. This includes theoretically solid methods such as DANN, and methods that have top performances in digit transfer learning tasks as recognized on the Papers with Code website  [1]. We also included relevant sub-domain-based methods [2, 3], aligning perfectly with our research's core focus. This strategic selection of baselines was not only comprehensive but also directly relevant to our algorithm's intended application area.

Reviewer comment (Evaluation):
More insightful analyses should be provided. For example, the visualization of the subdomain rebalancing weights should be shown.

Author response:
It appears that the reviewer has neglected the analysis provided in our submitted manuscript. Section 6.2 presents experiments on the Digits datasets to demonstrate the importance of (i) weights re-weighting and (ii) the accuracy of target subdomain weights estimation. As shown in Table 1, while the oracle case provides the best performance, DARSA algorithm demonstrates a very similar level of effectiveness, demonstrating that the weights in the DARSA algorithm provide nearly the same quality of predictions. In addition, it can be seen  from Table 1 that DARSA is robust to minor divergence in weights estimation and varying imbalance rates.

In addition, we would like to clarify that the primary indicator of the effectiveness of our method, DARSA, is the improvement in prediction accuracy, which directly reflects the precision of our estimated weights. However, to directly address your request and provide further transparency, we have included an additional analysis. This analysis involves comparing the difference between the actual ground truth target weight $w_T$ and our estimated target weight $\hat{w_T}$ across epochs. 

As illustrated in the figure 6 in Appendix F, our estimation aligns closely with the ground truth towards the end of the training process. This proximity indicates the effectiveness of our weight estimation approach within the DARSA algorithm.

Reviewer comment (Experiments):
The current experiments are conducted on two datasets, which is not sufficient. It is recommended to conduct on large-scale datasets.

Author response:
**Rationale Behind Datasets Selection:** Regarding the number of datasets used, our focus was on depth and relevance rather than quantity. The datasets selected are highly pertinent to the real-world applications we aim to impact. Specifically, our theoretical analysis indicates that DARSA should perform competitively in scenarios with a manageable number of classes, which is particularly applicable to fields like medical applications. These areas demand high precision and reliability, qualities that our algorithm has been demonstrated to fulfill effectively. 

In addition, as a theory-focused paper, we conducted comprehensive examinations to validate our theoretical propositions. This includes 1) empirically confirming the superiority of the sub-domain-based generalization bound; 2) verifying that the assumptions for Theorem 4.10 are empirically satisfied in real-world datasets; 3) demonstrating the vital role of subdomain weight re-balancing and showing DARSA's robustness to minor weight estimation discrepancies (Section 6.2); and 4) conducting an ablation study to evaluate the impact of each component within our objective function.

Reviewer comment (Experiments):
Most of the compared UDA methods were published before 2020. To fully validate the superiority of the proposed DARSA model, more SOTA UDA methods should be included in comparison experiments.

Author response:
**Rationale Behind Baseline Selection:**  Regarding the selection of compared methods, our benchmarks were meticulously chosen to demonstrate the robustness and effectiveness of our proposed algorithm. This includes theoretically solid methods such as DANN, and methods that have top performances in digit transfer learning tasks as recognized on the Papers with Code website  [1]. We also included relevant sub-domain-based methods [2, 3], aligning perfectly with our research's core focus. This strategic selection of baselines was not only comprehensive but also directly relevant to our algorithm's intended application area.

Reviewer comment (Evaluation):
More insightful analyses should be provided. For example, the visualization of the subdomain rebalancing weights should be shown.

Author response:
It appears that the reviewer has neglected the analysis provided in our submitted manuscript. Section 6.2 presents experiments on the Digits datasets to demonstrate the importance of (i) weights re-weighting and (ii) the accuracy of target subdomain weights estimation. As shown in Table 1, while the oracle case provides the best performance, DARSA algorithm demonstrates a very similar level of effectiveness, demonstrating that the weights in the DARSA algorithm provide nearly the same quality of predictions. In addition, it can be seen  from Table 1 that DARSA is robust to minor divergence in weights estimation and varying imbalance rates.

In addition, we would like to clarify that the primary indicator of the effectiveness of our method, DARSA, is the improvement in prediction accuracy, which directly reflects the precision of our estimated weights. However, to directly address your request and provide further transparency, we have included an additional analysis. This analysis involves comparing the difference between the actual ground truth target weight $w_T$ and our estimated target weight $\hat{w_T}$ across epochs. 

As illustrated in the figure 6 in Appendix F, our estimation aligns closely with the ground truth towards the end of the training process. This proximity indicates the effectiveness of our weight estimation approach within the DARSA algorithm.

Reviewer comment (Experiments):
The current experiments are conducted on two datasets, which is not sufficient. It is recommended to conduct on large-scale datasets.

Author response:
**Rationale Behind Datasets Selection:** Regarding the number of datasets used, our focus was on depth and relevance rather than quantity. The datasets selected are highly pertinent to the real-world applications we aim to impact. Specifically, our theoretical analysis indicates that DARSA should perform competitively in scenarios with a manageable number of classes, which is particularly applicable to fields like medical applications. These areas demand high precision and reliability, qualities that our algorithm has been demonstrated to fulfill effectively. 

In addition, as a theory-focused paper, we conducted comprehensive examinations to validate our theoretical propositions. This includes 1) empirically confirming the superiority of the sub-domain-based generalization bound; 2) verifying that the assumptions for Theorem 4.10 are empirically satisfied in real-world datasets; 3) demonstrating the vital role of subdomain weight re-balancing and showing DARSA's robustness to minor weight estimation discrepancies (Section 6.2); and 4) conducting an ablation study to evaluate the impact of each component within our objective function.

Reviewer comment (Theory):
The theoretical analysis is not instructive. The theoretical analysis has two main theorems, Sub-domain-based Generalization Bound (Theorem 4.5) and Benefits of Sub-domain Alignment (Theorem 4.7 and 4.10). Theorem 4.5 shows that the error of the target domain can be bounded by the weighted error of the source domain and the weighted error of the sub-domain distance. However, some works [1][2] have pointed out similar conclusions. Theorem 4.10 shows that the error bound of Theorem 4.5 is at least as strong as the full domain generalization bound without the sub-domain information. However, the authors may overlook the finiteness of real datasets, which is also important for reliable generalization bound and thus may lead to a different conclusion. Considering that the sub-domains have fewer samples than the whole domain, the finiteness improves more value of the sub-domain generalization bound than the whole-domain generalization bound.

Author response:
> The theoretical analysis is not instructive. The theoretical analysis has two main theorems, Sub-domain-based Generalization Bound (Theorem 4.5) and Benefits of Sub-domain Alignment (Theorem 4.7 and 4.10). Theorem 4.5 shows that the error of the target domain can be bounded by the weighted error of the source domain and the weighted error of the sub-domain distance. However, some works [1][2] have pointed out similar conclusions.

We indeed have already included [1] in our Related Work section. This is one great example of sub-domain-based domain adaptation method that uses class-conditioned domain alignment to address within-domain class imbalance and between-domain class distribution shift. Our theoretical framework provides a foundation for subdomain based methods, addressing their previous lack of rigorous understanding. One of the key messages we hope to deliver is that **Our theoretical framework not only supports our algorithm but can be extended to other methods, contributing to broader impact and value in the field.**  Contrasting with the focus of [2] on **multi-source** domain adaptation, where source domains are aggregated based on their relevance to the target domain (with weights assigned accordingly), our work focuses on **single source** domain adaptation. Specifically, we focus on aligning corresponding sub-domain distributions. For example, we align label 1 in the source domain with label 1 in the target domain. Our weighting logic also differs – rather than relying on the relevancy of source sub-domains, our method gives priority to the important sub-domains within the target domain. This fundamental difference underscores the uniqueness and strength of our methodology in addressing domain adaptation challenges.

> Theorem 4.10 shows that the error bound of Theorem 4.5 is at least as strong as the full domain generalization bound without the sub-domain information. However, the authors may overlook the finiteness of real datasets, which is also important for reliable generalization bound and thus may lead to a different conclusion. Considering that the sub-domains have fewer samples than the whole domain, the finiteness improves more value of the sub-domain generalization bound than the whole-domain generalization bound. 

While the variance increases with fewer samples, this does not imply that the realized error bound is necessarily larger than the expected one because it is equally likely that the realized error bound is smaller than the expected one. We have proved that in expectation, the sub-domain-based generalization bound presents a tighter estimation. Moreover, please note that the classical domain generalization bound have about $O(n^{-1/2})$ concentration while in the sub-domain-based generalization bound, the averaged error have about $O(m^{-1})$ variance reduction where $m$ is the number of class. This leads to $O((mn)^{-1/2})$ concentration, i.e., the variance is in fact reduced by a multiple of $m$ compared to the non-subdomain-based one. Intuitively, this is similar to the classical result in information theory where conditioning on the class random variable can only decrease entropy. In addition, as shown in Section 6.1, our empirical results demonstrate that the subdomain-based generalization bound is empirically much stronger than the non-sub-domain-based bound, corroborating our insights for the effectiveness of sub-domain based methods.

Reviewer comment (Novelty):
Although the paper is based on previous sub-domain alignment works, the previous works are not well connected in theoretical analysis, algorithm, or experiments. The author should give some examples of how the work enhances the understanding and robustifying of previous methods.

Author response:
> Although the paper is based on previous sub-domain alignment works, the previous works are not well connected in theoretical analysis, algorithm, or experiments. The author should give some examples of how the work enhances the understanding and robustifying of previous methods. 

It appears that there is some misunderstanding that our paper primarily build upon previous sub-domain alignment works. Instead, our paper introduces a fresh perspective by analyzing and providing a much-needed theoretical foundation for sub-domain based methods which are popular yet theoretically underexplored. To further clarify, our theoretical analysis does not simply follow the trajectory of existing literature; it carves a new path in understanding how subdomain based methods function and can be optimized. This contributes not only to a deeper comprehension of these methods but also offers tools that can be used to enhance their robustness and efficacy. By doing so, our work extends its impact beyond just supporting our algorithm; it enriches the field of domain adaptation with a solid theoretical base that can be used to re-examine, refine, and strengthen existing subdomain based methods. We believe that such a theoretical foundation is important to the field and merits recognition.

Reviewer comment (Experiments):
The experiments are not sufficient to show the effectiveness of the proposed algorithm. First, the number of datasets used for the experiments is too small. Second, the compared methods are not advanced enough.

Author response:
> The experiments are not sufficient to show the effectiveness of the proposed algorithm. First, the number of datasets used for the experiments is too small. Second, the compared methods are not advanced enough.

**Rationale Behind Baseline Selection:**  Regarding the selection of compared methods, our benchmarks were meticulously chosen to demonstrate the robustness and effectiveness of our proposed algorithm. This includes theoretically solid methods such as DANN and methods that have top performances in digit transfer learning tasks as recognized on the Papers with Code website [1]. We also included relevant sub-domain-based methods [2, 3], aligning perfectly with our research's core focus. This strategic selection of baselines was not only comprehensive but also directly relevant to our algorithm's intended application area.

**Rationale Behind Datasets Selection:** Regarding the number of datasets used, our focus was on depth and relevance rather than mere quantity. The datasets selected are highly pertinent to the real-world applications we aim to impact. Specifically, our theoretical analysis indicates that DARSA should perform competitively in scenarios with a manageable number of classes, which is particularly applicable to fields like medical applications. These areas demand high precision and reliability, qualities that our algorithm has been demonstrated to fulfill effectively. 

In addition, as a theory-focused paper, we conducted comprehensive examinations to validate our theoretical propositions. This includes 1) empirically confirming the superiority of the sub-domain-based generalization bound; 2) verifying that the assumptions for Theorem 4.10 are empirically satisfied in real-world datasets; 3) demonstrating the vital role of subdomain weight re-balancing and showing DARSA's robustness to minor weight estimation discrepancies (Section 6.2); and 4) conducting an ablation study to evaluate the impact of each component within our objective function.

---

[1] https://paperswithcode.com/task/domain-adaptation

[2] Deng, Zhijie, Yucen Luo, and Jun Zhu. "Cluster alignment with a teacher for unsupervised domain adaptation." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[3] Long, Mingsheng, et al. "Conditional adversarial domain adaptation." Advances in neural information processing systems 31 (2018).