Paper: Improving Domain Generalization with Domain Relations (ICLR-2024)

Reviewer comment (Writing):
In section 3.2, it is mentioned that both $m_i$ and $a_{ij}^{g}$ are derived from domain metadata, but the specific relationship or distinction between them is not clearly explained. The experimental results suggest that $a_{ij}^{g}$ is extracted from $m_i$, yet this section lacks explicit clarification.

Author response:
**A1**: Thank you for pointing that out. The fixed domain relation $a_{ij}^g$ can be derived either from domain meta-data $m_i$ or directly captured by domain experts. In the former case, for instance, in spatial distribution shift scenarios like FMoW, we can use geographical information (e.g., longitude and latitude), which may be included in the meta-data $m_i$ or $m_j$, to establish geographical proximity relations. In the latter case, as an example, in drug-target interaction prediction, we employ a fixed relation represented by the protein-protein graph, while $m_i$ represents the protein structure. We have clarified these points in the updated paper in Section 3.2.

Reviewer comment (Reproducibility):
It is unclear whether the two layer neural network $g$ should be trained or directly be used from other previous works. It is mentioned that weight vectors $w_r$ are learnable vectors. Are these weights and the neural network trained by the consistency loss?

Author response:
**A2**: In this paper, the final domain relation $a_{ij}$ contains the learned relation $a_{ij}^l$, which is learned from the domain meta-data (Eqn. (5)) with the two layer neural network. The final domain relation $a_{ij}$ is reflected in the consistency training loss. Thus, the two-layer neural network can only be optimized during the training process only.

Reviewer comment (Theory):
The consistency loss implies that each head is trained on all the domains with different weights. This is similar to the case where each head is trained on all the domains with a weighted combination of the losses of all the domains. For the predictor $f^{(d)}$, the loss is in this form: $l(\frac{\sum_j^{N^{tr}} a_{dj} f^{(d)}(x^j)}{\sum_j^{N_{tr}} a_{dj}})$. Would this loss also help the learning of domain-specific models?

Author response:
**A3**: Yes, this loss is similar to our consistency loss and would help the learning of domain-specific models. We evaluate the performance of D$^3$G with this loss, and present the results in Table R1.

**Table R1**: Performance comparison between D$^3$G and its variant with losses from all the domains.

|      |   DG-15 (Avg. Acc. $\uparrow$)    |        TPT-48 (MSE $\downarrow$)        |                           | FMoW (Worst Acc. $\uparrow$) |              | ChEMBL-STRING (ROC-AUC $\uparrow$) |              |
| :--------------- | :-----------------: | :-----------------------: | :-----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                  |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$ W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
|         D$^3$G with losses from all the domains        |   71.8 $\pm$ 1.8%   |     0.367 $\pm$ 0.041     |     0.249 $\pm$ 0.037     |   27.99 $\pm$ 0.67%   |   38.75 $\pm$ 0.51%   |   78.40 $\pm$ 0.11%   |   76.52 $\pm$ 0.49%   |
| **D$^3$G(Ours)** | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |   **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

The above results show that D$^3$G, utilizing weighted losses mentioned by Reviewer u1oN, performs well across all datasets since our original two-term loss (predictive loss + consistency loss) approximates a weighted loss form. Our original loss essentially upweights the corresponding domain of each head more significantly using a separate predictive loss term, leading to more improvement over the new variant.

Reviewer comment (Theory):
Suppose that each head is trained on all the domains by the loss function (2). In this case, regardless of the values of $a_{ij}$, if both the supervised loss and consistency loss reach their minimum, then the learning of $a_{ij}$ would be useless. Does the proposed method face this issue?

Author response:
**A4**: If each head is trained on all domains and results in minimal predictive loss and consistency loss, it implies that all domains are equally important in this problem, and it suggests that all $a_{ij}$ are identical. This can be considered a special case of our developed method. However, in real-world scenarios, encountering this situation is rare. This issue can be verified through the following experiment:

As suggested by the reviewer u1oN, we train each head on all the training domains using supervised loss and average the predictions of these heads as the test prediction. This experiment has been conducted in our original submission (see the first row in Table 3 in our paper, where no relations are used). We also present the corresponding results in Table R2.

**Table R2**: Performance comparison between D$^3$G and its variant with no related domains.


|      |   FMoW (Worst Acc. $\uparrow$)    |                                     | ChEMBL-STRING (ROC-AUC $\uparrow$) |                       |
|:-----------------|:----------:|:----------:|:---------:|:---------:|
|                  | FMoW-Asia | FMoW-WILDS | PPI>50 |        PPI>100        |
| D3G with no related domains |   26.93 $\pm$ 0.47%   |          35.32 $\pm$ 0.66%          |        76.17 $\pm$ 0.21%        |   73.38 $\pm$ 0.13%   |
| **D3G (Ours)** | **28.12 $\pm$ 0.28%** |        **39.47 $\pm$ 0.57%**        |      **78.67 $\pm$ 0.16%**      | **77.24 $\pm$ 0.30%** |

The results reveal that this baseline slightly outperforms ERM, yet significantly underperforms compared to our D$^3$G. This indicates the importance of using uneven domain relation to weight losses and predictions from different domains/heads.

Reviewer comment (Experiments):
The robustness and generality of the proposed method is unclear. Practically, it is unclear how to design a good similarity definition for metadata of different tasks. The current way of constructing the fixed relations requires specific and expertise knowledge on each task.

Author response:
**A1**: In our experiments, we use cosine similarity for DG-15, TPT-48, and FMoW, and apply PPI scores for biomedical data. Consequently, cosine similarity serves as a reliable backup for defining similarity when domain meta-data is featured.

In situations where domain meta-data is discrete (e.g., label or index) and cosine similarity cannot be applied, we can directly learn domain relations directly from the data. To do this, we compute the fixed relation across different domains by applying optimal transport on the corresponding data [1]. For the learnable part, we utilize the average of features from each domain as input to the two-layer neural network $g$. We report the results in Table R1. We’ve added these results in Appendix H.3 in the updated paper.

**Table R1**: Performance comparison between D$^3$G and its variant that inly utilize domain ids, along with reporting the best baseline performance for each dataset.

|      |   DG-15 (Avg. Acc. $\uparrow$)    |        TPT-48 (MSE $\downarrow$)        |                           | FMoW (Worst Acc. $\uparrow$) |              | ChEMBL-STRING (ROC-AUC $\uparrow$) |              |
| :--------------- | :-----------------: | :-----------------------: | :-----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                  |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$ W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
|         Best baseline        |   66.2 $\pm$ 2.0%   |     0.371 $\pm$ 0.054     |     0.262 $\pm$ 0.034     |   26.99 $\pm$ 1.27%   |   37.64 $\pm$ 0.92%   | 75.42 $\pm$ 0.42% | 73.57 $\pm$ 0.21% |
|         D$^3$G w/o domain meta-data        |   70.2 $\pm$ 3.7%   |     0.381 $\pm$ 0.071     |     0.325 $\pm$ 0.029     |   27.17 $\pm$ 1.09%   |   37.82 $\pm$ 0.31%   |   76.65 $\pm$ 0.22%   |   75.01 $\pm$ 0.36%   |
| **D$^3$G (Ours)** | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |   **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

The findings indicate that this variant exhibits inferior performance compared to the one with domain meta-data, however, it still outperforms the best baselines on most datasets. These outcomes suggest that we can also design a relatively good similarity definition for meta-data of different tasks, even with only labels or indices to distinguish different domains. This further confirms the generality of D$^3$G.

Reviewer comment (Theory):
The limitations of this work are not fully discussed. For example, the generality of the assumptions in the theoretical analysis part.

Author response:
**A2**: We’ve discussed the limitations of our work in our original submission (see Appendix I). We’ve added the reference in the conclusion (see “We discussed our limitations in Appendix I”).

Reviewer comment (Experiments):
The comparison of the proposed method with ensemble models of existing approaches should have been given.

Author response:
Our proposed method shares some similarities with ensemble methods, yet it is more closely aligned with multi-head network structures. As illustrated in Figure 1 in the paper, our approach employs a shared feature extractor and learns a collection of domain-specific heads. Consequently, in our paper, we compare several multi-head models without using domain meta-data, such as DRM [1], LLE [2], and DDN [3]. Our method consistently outperforms these baselines, highlighting the significant role of domain meta-data in generating accurate domain relations and enhancing generalization performance. 

Furthermore, we have also conducted an additional comparison between D$^3$G and one variant, which employs separate feature extractors for each training domain. The results are reported in Table R1 here.

**Table R1**: Performance comparison between D$^3$G and using separate feature extractors for each training domain.

|                  |  DG-15 (Avg. Acc. $\uparrow$)   |        TPT-48 (MSE $\downarrow$)        |                           |   FMoW (Worst Acc. $\uparrow$)    |                       | ChEMBL-STRING (ROC-AUC $\uparrow$) |                       |
| :--------------- | :-----------------: | :-----------------------: | :-----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                  |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$ W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
|         Using separate feature extractors        |   47.3 $\pm$ 3.7%   |     0.563 $\pm$ 0.071     |     0.519 $\pm$ 0.076     |   23.62 $\pm$ 0.47%   |   35.44 $\pm$ 0.58%   |   76.11 $\pm$ 0.31%    |   74.19 $\pm$ 0.74%   |
| **D$^3$G(Ours)** | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |   **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

According to the results, D$^3$G shows superior performance compared to using separate feature extractors for each training domain. This underscores the importance of employing a shared feature extractor to learn a universal representation, while domain-specific heads can further identify domain-specific features. Additionally, our D$^3$G model demonstrates superior efficiency compared to this variant since it utilizes a shared feature extractor while the variant needs $N^{tr}$ feature extractors. We’ve added these results in Appendix H.3 in the updated paper.

Reviewer comment (Experiments):
Does it work when there is no domain metadata available? Can such domain relations be learned from the data themselves?

Author response:
Yes, we do an ablation study for this question by learning domain relations directly from the data. To do this, we compute the fixed relation using optimal transport [4]. For the learnable part, we utilize the average of features from each domain as input to the two-layer neural network $g$. We report the results in Table R2. 

**Table R2**: Performance comparison between D$^3$G and its variant that does not utilize domain meta-data, along with reporting the best baseline performance for each dataset.

|                  |  DG-15 (Avg. Acc. $\uparrow$)   |        TPT-48 (MSE $\downarrow$)        |                           |   FMoW (Worst Acc. $\uparrow$)    |                       | ChEMBL-STRING (ROC-AUC $\uparrow$) |                       |
| :--------------- | :-----------------: | :-----------------------: | :-----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                  |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$ W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
|         Best baseline        |   66.2 $\pm$ 2.0%   |     0.371 $\pm$ 0.054     |     0.262 $\pm$ 0.034     |   26.99 $\pm$ 1.27%   |   37.64 $\pm$ 0.92%   | 75.42 $\pm$ 0.42% | 73.57 $\pm$ 0.21% |
|         D$^3$G w/o domain meta-data        |   70.2 $\pm$ 3.7%   |     0.381 $\pm$ 0.071     |     0.325 $\pm$ 0.029     |   27.17 $\pm$ 1.09%   |   37.82 $\pm$ 0.31%   |   76.65 $\pm$ 0.22%   |   75.01 $\pm$ 0.36%   |
| **D$^3$G(Ours)** | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |   **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

The findings indicate that this variant exhibits inferior performance compared to the one with domain meta-data, however, it still outperforms the best baselines on most datasets. These outcomes suggest that D$^3$G's model design inherently enhances generalization capabilities, which are further augmented by the use of domain meta-data. We’ve added these results in Appendix H.4 in the updated paper.

Reviewer comment (Experiments):
How the domain metadata are used in the comparative methods?

Author response:
As we discussed in the caption of Table 1 (page 8) and in the first paragraph of experiment section, for all baselines, we incorporate domain meta-data as
features during the training and test stages for fair comparison. In detail, we begin by converting domain meta-data into input embeddings, which are then passed through the two-layer neural network. Subsequently, we concatenate this intermediate domain feature with other features and pass this concat feature through another feature extractor. This procedure is consistently applied to all baselines.

Reviewer comment (Theory):
The method requires a user-defined domain similarity function based on meta-data. This might not always be available. Additionally, for specific applications, a user might not be able to credibly define such a function.

Author response:
**Q1**: Availability of domain meta-data and domain similarity function.

**A1**: Most often, domain meta-data is obtained as a by-product during the collection of data, which then serves as the primary criteria for classifying domains. This meta-data typically manifests as an inherent property of the domain, such as latitude and longitude in geographic data or PPI scores in biomedical data.  Furthermore, obtaining meta-data for each testing domain is also not overly expensive since the number of testing domains is relatively small in most scenarios. As for the domain similarity function, we employ cosine similarity across all datasets, except in the case of biomedical data with PPI scores. Therefore, when we do not have specific metrics in certain fields, the alternative is to choose from a range of general distance metrics.

In situations where domain metadata and the domain similarity function are unavailable, we can derive domain distances directly from the data. To do this, we compute the fixed relation using optimal transport [1]. For the learnable part, we utilize the average of features from each domain as input to the two-layer neural network $g$. We report the results in Table R1. 

**Table R1**: Performance comparison between D$^3$G and its variant that does not utilize domain meta-data, along with reporting the best baseline performance for each dataset.

|      |   DG-15 (Avg. Acc. $\uparrow$)    |        TPT-48 (MSE $\downarrow$)        |                           | FMoW (Worst Acc. $\uparrow$) |              | ChEMBL-STRING (ROC-AUC $\uparrow$) |              |
| :--------------- | :-----------------: | :-----------------------: | :-----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                  |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$ W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
|         Best baseline        |   66.2 $\pm$ 2.0%   |     0.371 $\pm$ 0.054     |     0.262 $\pm$ 0.034     |   26.99 $\pm$ 1.27%   |   37.64 $\pm$ 0.92%   | 75.42 $\pm$ 0.42% | 73.57 $\pm$ 0.21% |
|         D$^3$G w/o domain meta-data        |   70.2 $\pm$ 3.7%   |     0.381 $\pm$ 0.071     |     0.325 $\pm$ 0.029     |   27.17 $\pm$ 1.09%   |   37.82 $\pm$ 0.31%   |   76.65 $\pm$ 0.22%   |   75.01 $\pm$ 0.36%   |
| **D$^3$G(Ours)** | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |   **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

The findings indicate that this variant exhibits inferior performance compared to the one with domain meta-data, however, it still outperforms the best baselines on most datasets. These outcomes suggest that D$^3$G's model design inherently enhances generalization capabilities, which are further augmented by the use of domain meta-data. We’ve added these results in Appendix H.4 in the updated paper.

Reviewer comment (Theory):
The theoretical work makes (understandably) quite a few strong assumptions that are unlikely to be true in practice. These results could be improved to show how performance is affected by, for example, noisy domain relations.

Author response:
**Q2**: The theoretical work makes (understandably) quite a few strong assumptions that are unlikely to be true in practice. These results could be improved to show how performance is affected by, for example, noisy domain relations.

**A2**:  Thank you for your suggestion. Our results can be extended to noisy domain relations by incorporating technical tools from the error-in-variable literature. Specifically, in this situation, our assumption will be extended to $\|h^{(i)}-h^{(j)}\|_\infty$ $\le G\cdot$ $\|Z^{(i)}-Z^{(j)}\|$, while our estimated representation $\hat{Z}^{(i)}=Z^{(i)}+\epsilon_i$ includes a zero-mean sub-Gaussian noise term $\epsilon_i$. Then, we can use technical tools from error-in-variable literature to denoise it, we will leave it as a future work.

We acknowledge that our theoretical analysis relies on some assumptions. Nevertheless, we still believe that our current theory provides valuable insights supporting the idea that utilizing good domain relations leads to better generalization. In fact, our proof suggests that using domain relations can help reduce variance while sacrificing a slight amount of bias, resulting in a more favorable bias-variance trade-off.

Reviewer comment (Theory):
Is it possible for the proposed method to utilize negative relationships between domains?

Author response:
**Q3**: Is it possible for the proposed method to utilize negative relationships between domains?

**A3**: Yes, the model can handle and utilize negative domain relations. If our understanding is correct, for a pair of domains $i$ and $j$, the existence of negative domain relations implies that updating domain $j$'s corresponding parameters with data from domain $i$ would negatively impact the performance of domain $j$. In such cases, if the fixed relation indicates these negative relationships or if the learned relation does so, we will set the domain relation $a_{ij}=0$ to prevent information propagation between these two domains.

Reviewer comment (Evaluation):
How will the proposed method behave in a situation where there are no related domains according to the meta-data? Will the method perform as well as a domain-invariant approach? Is this what the consistency loss is for?

Author response:
**Q4**: How will the proposed method behave in a situation where there are no related domains according to the meta-data? Will the method perform as well as a domain-invariant approach? Is this what the consistency loss is for?

**A4**: In situations where there are no related domains according to the meta-data, we will use a uniform domain relation, implying that all $a_ij$ are identical. During the training process, we emphasize the data from each head's corresponding domain with predictive loss, while applying uniform weights to other training domains in the calculation of consistency loss. During inference, our method performs comparably to a domain-invariant approach by averaging the outputs from each head. We analyze this situation in the first row of Table 3 in our paper, with the corresponding results also detailed in Table R2.

**Table R2**: Performance comparison between D$^3$G and its variant with no related domains.


|      |   FMoW (Worst Acc. $\uparrow$)    |                                     | ChEMBL-STRING (ROC-AUC $\uparrow$) |                       |
|:-----------------|:----------:|:----------:|:---------:|:---------:|
|                  | FMoW-Asia | FMoW-WILDS | PPI>50 |        PPI>100        |
| D3G with no related domains |   26.93 $\pm$ 0.47%   |          35.32 $\pm$ 0.66%          |        76.17 $\pm$ 0.21%        |   73.38 $\pm$ 0.13%   |
| **D3G (Ours)** | **28.12 $\pm$ 0.28%** |        **39.47 $\pm$ 0.57%**        |      **78.67 $\pm$ 0.16%**      | **77.24 $\pm$ 0.30%** |

The results reveal that without using domain relation, we can slightly outperform ERM, yet significantly underperforms compared to our D$^3$G. This indicates the importance of using accurate domain relation to weight losses and predictions from different domains/heads.

Reviewer comment (Experiments):
If domain information was unavailable, could a domain discovery method (e.g. clustering) be used to learn domains and domain relations from unlabeled data? In the limit, could the model architecture and inference algorithm be adapted to learn domains and domain relations end-to-end?

Author response:
**Q5**: If domain information was unavailable, could a domain discovery method (e.g. clustering) be used to learn domains and domain relations from unlabeled data? In the limit, could the model architecture and inference algorithm be adapted to learn domains and domain relations end-to-end?

**A5**: Yes, we do an ablation study for this question by clustering all training data into several groups, with the number of groups matching the actual number of domains. Each group is treated as a separate domain, and domain relations are established based on the distances between the clustering centers. The results, as presented in Table R3, show that by clustering data to learn domains, our method's variant surpasses ERM in 5 out of 7 settings, although still lags behind D$^3$G. This indicates that the model architecture and inference algorithm can somehow be adapted to learn domains and domain relations simultaneously in an end-to-end manner. Moreover, the appropriate use of domain meta-data and domain relation will then significantly improve the performance. We’ve added these results in Appendix H.4 in the updated paper.

**Table R3**: Performance comparison between ERM, D$^3$G and its variant that does not utilize domain information.

|      |   DG-15 (Avg. Acc. $\uparrow$)    |        TPT-48 (MSE $\downarrow$)        |                           | FMoW (Worst Acc. $\uparrow$) |              | ChEMBL-STRING (ROC-AUC $\uparrow$) |              |
| :------------------------- | :-----------------: | :-----------------------: | :----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                            |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
| ERM                        |   44.0 $\pm$ 4.6%   |     0.445 $\pm$ 0.029     |    0.328 $\pm$ 0.033     |   26.05 $\pm$ 3.84%   |   34.87 $\pm$ 0.41%   |   74.11 $\pm$ 0.35%    |   71.91 $\pm$ 0.24%   |
| D$^3$G w/o domain split information |   58.1 $\pm$ 4.1%   |     0.429 $\pm$ 0.055     |    0.340 $\pm$ 0.095     |   25.46 $\pm$ 1.35%   |   36.06 $\pm$ 0.39%   |   74.94 $\pm$ 0.79%    |   73.38 $\pm$ 0.91%   |
| **D$^3$G(Ours)**              | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |  **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

Reviewer comment (Experiments):
Could this method be used in data impoverished applications to improve performance?

Author response:
**Q6**: Could this method be used in data impoverished applications to improve performance?

**A6**: Yes, our method can be applied effectively in data-scarce applications. The consistency loss is utilized to establish connections between domains and facilitate training on data insufficient domains. Throughout the training process, for all examples from domain $d$, we use these examples and employ the overall loss not only to update its domain-specific function $f^{(d)}$ but also to influence other domain-specific functions with the consistency loss. Consequently, the utilization of data from one domain effectively updates all parameters, implying that domain-specific models in data-scarce domains are optimized not only by their respective domain data but also by other domains, particularly those that are similar. For example, if training domain $d$ is more similar to training domain $d-1$ than to domain $d+1$, it will have a stronger influence on the optimization of the function $f^{(d-1)}$.

Reviewer comment (Novelty):
The novelty of the proposed method may be somewhat limited as it bears similarities to ensemble methods, which also involve assigning varying weights to different domains. The essential distinctions between the proposed methods and ensemble methods should be clearly elucidated, and it is important to consider baseline models that incorporate ensemble methods for comparison.

Author response:
**Q1**: Comparison between D$^3$G and ensemble methods.

**A1**: We acknowledge that our proposed method shares some similarities with ensemble methods, yet it is more closely aligned with multi-head network structures. As illustrated in Figure 1 in the paper, our approach employs a shared feature extractor and learns a collection of domain-specific heads. Consequently, in our paper, we compare several multi-head models without using domain meta-data, such as DRM [1], LLE [2], and DDN [3]. Our method consistently outperforms these baselines, highlighting the significant role of domain meta-data in generating accurate domain relations and enhancing generalization performance. 

To further address your concern, we have also conducted an additional comparison between D$^3$G and one variant, which employs separate feature extractors for each training domain. The results are reported in Table R1 here.

**Table R1**: Performance comparison between D$^3$G and using separate feature extractors for each training domain.

|      |   DG-15 (Avg. Acc. $\uparrow$)    |        TPT-48 (MSE $\downarrow$)        |                           | FMoW (Worst Acc. $\uparrow$) |              | ChEMBL-STRING (ROC-AUC $\uparrow$) |              |
| :--------------- | :-----------------: | :-----------------------: | :-----------------------: | :-------------------: | :-------------------: | :--------------------: | :-------------------: |
|                  |                     | N(24) $\rightarrow$ S(24) | E(24) $\rightarrow$ W(24) |       FMoW-Asia       |      FMoW-WILDS       |         PPI>50         |        PPI>100        |
|         Using separate feature extractors        |   47.3 $\pm$ 3.7%   |     0.563 $\pm$ 0.071     |     0.519 $\pm$ 0.076     |   23.62 $\pm$ 0.47%   |   35.44 $\pm$ 0.58%   |   76.11 $\pm$ 0.31%    |   74.19 $\pm$ 0.74%   |
| **D$^3$G (Ours)** | **77.5 $\pm$ 2.5%** |   **0.342 $\pm$ 0.019**   |   **0.236 $\pm$ 0.063**   | **28.12 $\pm$ 0.28%** | **39.47 $\pm$ 0.57%** | **78.67 $\pm$ 0.16%**  | **77.24 $\pm$ 0.30%** |

According to the results, D$^3$G shows superior performance compared to using separate feature extractors for each training domain. This underscores the importance of employing a shared feature extractor to learn a universal representation, while domain-specific heads can further identify domain-specific features. Additionally, our D$^3$G model demonstrates superior efficiency compared to this variant since it utilizes a shared feature extractor while the variant needs $N^{tr}$ feature extractors. We’ve added these results in Appendix H.3 in the updated paper.

Reviewer comment (Evaluation):
The evaluation metrics used in this paper are different from previous works conducted on the TPT-48, FMoW, and ChEMBL-STRING datasets. It is unclear whether the metrics used, such as MSE, are suitable and fair to be used to evaluate domain generalization models.

Author response:
**Q2**: Evaluation metrics on the TPT-48, FMoW, and ChEMBL-STRING datasets

**A2**: In this paper, the evaluation metrics we use are those originally selected by the creators of the respective datasets: 
- TPT-48: we follow GRDA [4] and use MSE as our evaluation metric.
- FMoW: we use worse accuracy as our evaluation metric following Wilds [5].
- ChEMBL-STRING: we use ROC-AUC as our evaluation metric following SGNN-EBM [6].

Reviewer comment (Theory):
As stated by the authors, the theoretical analysis of this paper relies on certain assumptions. It is questioning whether these assumptions reflect the real-world datasets. For example, the assumptions that the domain relations accurately capture the similarity between domains, and that they are determined solely by the distance between domain representations, may not always hold true in real-world datasets.

Author response:
**Q3**: As stated by the authors, the theoretical analysis of this paper relies on certain assumptions. It is questioning whether these assumptions reflect the real-world datasets. In the proofs of the theorems, the authors only prove that the proposed estimator outperforms the equally weighted estimator in the minimax sense, but not a theoretically global superiority. Moreover, the theorems only reveal that uneven weights can be better than equal weights, which does not entirely support the key ideas of acquiring appropriate weights for different domains.

**A3**: Thank you for your comment. We acknowledge that our theoretical analysis relies on some assumptions. The aim of our theoretical analysis is to explain some phenomena distilled from our empirical study and to help us understand why D$^3$G can work. In the future, we plan to relax these assumptions.

Furthermore, we would like to note that the comparison in the minimax sense is a common approach in machine learning and statistics literature. For example, you can find it in papers, such as meta-learning [7], reinforcement learning [8], and private learning [9, 10]. Regarding your second point about uneven weights, we want to address it with care. It appears there might be a potential misunderstanding. In our theory, we propose that weighting according to similarity (controlled by the parameter $B$) can yield better results than using equal weights, as opposed to using uneven weights.

Reviewer comment (Theory):
In the proofs of the theorems, the authors only prove that the proposed estimator outperforms the equally weighted estimator in the minimax sense, but not a theoretically global superiority.

Author response:
**Q3**: As stated by the authors, the theoretical analysis of this paper relies on certain assumptions. It is questioning whether these assumptions reflect the real-world datasets. In the proofs of the theorems, the authors only prove that the proposed estimator outperforms the equally weighted estimator in the minimax sense, but not a theoretically global superiority. Moreover, the theorems only reveal that uneven weights can be better than equal weights, which does not entirely support the key ideas of acquiring appropriate weights for different domains.

**A3**: Thank you for your comment. We acknowledge that our theoretical analysis relies on some assumptions. The aim of our theoretical analysis is to explain some phenomena distilled from our empirical study and to help us understand why D$^3$G can work. In the future, we plan to relax these assumptions.

Furthermore, we would like to note that the comparison in the minimax sense is a common approach in machine learning and statistics literature. For example, you can find it in papers, such as meta-learning [7], reinforcement learning [8], and private learning [9, 10]. Regarding your second point about uneven weights, we want to address it with care. It appears there might be a potential misunderstanding. In our theory, we propose that weighting according to similarity (controlled by the parameter $B$) can yield better results than using equal weights, as opposed to using uneven weights.