Paper: Domain Generalization Deep Graph Transformation (ICLR-2024)

Reviewer comment (Theory):
I don't quite understand what is the essence of the optimization of space complexity in this model. The authors claim that, if we traverse all possible combinations of input and output pairs it will require $\Omega(3^N)$ time (BTW the paper uses the asymptotic notation $O$, but actually should be $\Omega$ here), which is true, but usually you don't require to traverse all possible input-output pairs to learn a not-bad model, just like in ordinary supervised classification, the model doesn't need to travers all possible samples from the ditribuition but just a finite subset of them (the training set). In my understanding, the presented paper basically asumed there is a distribution of meta information and the training of the proposed model will require meta information as input. However, if there is indeed an underlying distribution of meta information, then just simply sampling some pairs from all the $\Omega(3^N)$ possible combination might also work well. That's why I don't quite understand the necessity of using this multi-input, multi-output, hypernetwork-based framework here. It is possible that my understanding is wrong. If so feel free to point it out.

Author response:
* It is possible to assign distinct encoders and decoders for each input and output mode, respectively, and sample a subset of input-output pairs to bypass training on all possible input-output combinations. This approach ensures the inclusion of all modes during training. Our baseline model, ED-GNN, employs this method. It demonstrates comparable performance to MultiHyperGNN in prediction tasks (refer to Table 1), but exhibits lower domain generalization performance (see Table 2). This limitation arises due to the absence of meta-information guiding the generalization process.  

* Alternatively, we can indeed use a singular encoder for all input modes and a single decoder for all output modes. In the absence of meta-information, this approach does not reveal relationships among modes. This is crucial as the same input graph may predict different output graphs under varying contexts. By contrast, utilizing meta-information, however, poses two training challenges:

    - Encode the graph into graph embedding and concatenate the meta information into the embedding, but then we cannot enjoy node level embedding for downstream prediction task at the node level.
    - Duplicating meta-information for each node to merge with node embeddings leads to redundancy. Additionally, this does not effectively capture the dependency of node inputs on meta-information due to the nature of concatenation.

* Contrary to the aforementioned approaches, our method efficiently handles all input-output pairs without resorting to subset sampling. This leads to more precise results as we could involve all input and output modes, but at a cost of a much smaller trainable parameter space.

We believe these clarifications underscore the effectiveness and innovation of our approach in comparison to existing methodologies.

Reviewer comment (Theory):
In Theorem 1, it makes no sense to me to assume the genralization error to be iid Gaussian (as the notation is quite confusing here, I don't actually know what does the authors mean here, but I guess it means iid Gaussian. Please correct me if my understanding is wrong). According to Definition 3, the error comes from $X_i ^{(k)} - f_{\hat \gamma_{\mathcal X\to \mathcal Y}}(\cdots)$, and as $f_{\hat \gamma_{\mathcal X\to \mathcal Y}}$ is based on GNNs, there should be inter-node interactions. It is very unlikely ${\boldsymbol{\epsilon}}_i$ can be iid.

Author response:
* Our model assumes that samples (i.e., graphs) within the dataset are independent and identically distributed (iid), as opposed to individual nodes in a graph.

* Notation Explanation: In our notation, $G_i^{(j)}$​ denotes the graph corresponding to the $i$-th sample in mode $j$. Therefore, the generalization error is a vector with the length of the number of nodes in the graph. We concur with the reviewer's point that the covariance matrix of the generalization error, \epsilon, is not diagonal. This acknowledgment is indeed compatible with our theoretical framework and its proofs. To enhance clarity and prevent any misunderstanding: We have revised our notation of this covariance matrix to $\Sigma$. Accordingly, we have updated Equation (16) in our manuscript to reflect this change in notation:

"$m^* = argmax_m -\frac{1}{2}\sum_{i=1}^n(\hat X_i^{(k)}-X_i^{(k)})^T\mathbf\Sigma^{-1}(\hat X_i^{(k)}-X_i^{(k)}) + C$" and also:

" As the covariance matrix $\mathbf\Sigma$ is positive semidefinite, the above objective is equivalent to minimizing the generalization error $\|\mathbf\epsilon_i\|_2^2=\|\hat X_i^{(k)}-X_i^{(k)}\|_2^2$"  

* We sincerely appreciate reviewer's comments to identify the typo in our paper. We assume $\Sigma$ is positive definite to make it an invertible matrix in Eq. (16). We have corrected this in the revision of our paper.

Reviewer comment (Writing):
In Section 3, there's a sentence "where $A^{(j)} \in \mathbb R^{p_j \times p_j}$ is the graph of size $p_j \leq p$". Do you mean $A^{(j)}$ is the **adjancency** of a graph? Also, since in the definition $A^{(j)} \in \mathbb R^{p_j \times p_j}$, do you mean the entries of $A^{(j)}$ can be take any real numbers, even negative numbers?

Author response:
Thank you for your inquiry about our notation. We confirm that $A^{(j)}$ represents the adjacency matrix of the graph in mode j. In this matrix, the elements are binary, with a value of 1 indicating the presence of an edge between nodes, and a value of 0 indicating no edge. We have clarified this in the revised paper: 

"$A^{(j)}\in\mathbb{R}^{p_j\times p_j}$ is the adjacency matrix of size $p_j\le p$"

Reviewer comment (Writing):
In Definition 1, what does "s.t. $\mathcal X^{\mathcal  T} \times \mathcal Y^{\mathcal T} \not \in \mathcal S$" mean? Does that mean, say $\mathcal X^{\mathcal T}$ contains modes that are not in $\mathcal G$?

Author response:
Yes, this means that target domain can be exclusive to source domain.

Reviewer comment (Writing):
In equation (1), what is $\beta_d^{(k)}$? What is the dimensioanlity of $\hat X_i^{(k)}$?

Author response:
* $\beta_d^{(k)}$ is parameter of the decoder of mode $k$ generated by the decoder hypernetwork $\beta_d$. We have highlighted in red the Eq. (2) in the revised paper that illustrates how they are computed.

* Dimensionality of $\hat{X_i}^{(k)}$ is $p$, which involves all nodes (the union of input and output graphs).

Reviewer comment (Writing):
In the definition of $f_{\gamma_{\mathcal X \to \mathcal Y}} = \left\\{f_d^{(k)} \star \left\\{f_e^{(j)}\right\\}\_{j \in \mathcal X}\right\\}_{k \in \mathcal Y}$, what does $\star$ mean here?

Author response:
We appreciate the valuable input from the reviewer and giving us the opportunity to enhance our paper. $\star$ means the function convolution that encode nodes with $f_e^{(i)}$ and decode it with $f_d^{(k)}$.

Reviewer comment (Writing):
In Algorithm 1, what is $i$ in the output? It doesn't seems to be in the input nor is it a loop variable.

Author response:
We have explained the output in the 4-th line of Algorithm 1: {$\hat X_i^{(k)}$}$_{k\in\mathcal{Y}}$. We have also highlighted it in red for the reviewer to review.

Reviewer comment (Writing):
In Theorem 1, ${\boldsymbol{\epsilon}}_i \sim \mathcal N({\boldsymbol{0}}, {\boldsymbol{\sigma}}^2)$, what does ${\boldsymbol{\sigma}}^2$ mean? Do you mean $\sigma^2 {\boldsymbol{I}}$ where ${\boldsymbol{I}}$ is the identity matrix?

Author response:
$\mathbf\sigma^2$ is the covariance matrix of generalization error, which is not a diagonal matrix due to the potential correlation among nodes. To clarify this, we have modified it into $\mathbf\Sigma$.

Reviewer comment (Evaluation):
While space complexity is addressed, the overall computational complexity and potential overfitting risks due to the complex model structure are not discussed in detail.

Author response:
* We have already discussed the computational complexity in Appendix C:

“Regarding computational complexity, similar to space complexity, ED-GNN, MHM and IN have $O(3^N)$ mode combinations to iterate in each training epoch, where $N$ is the total number of modes in the dataset. The computational complexity of EERN and DRAIN is linear to the number of generators (i.e., $O(2^N)$ generators to handle all combinations of modes) or the number of combinations of time points (i.e., $O(N2^N)$ total number of combinations of time points), which are usually much larger than $N^2$. MultiHyperGNN only needs to train on different input-output mode pairs so that we have only $O(N^2)$ such pairs to iterate in each training epoch, achieving a polynomial computational complexity .”

and we have highlighted this part in red for reviewer’s easier reference.

* Overfitting occurs often when the model is too complex while training data is limited. The total number of input-output mode combinations that our model handles is $O(3^N)$. As a result, we have combinatorially many samples to train just two hypernetworks so that the model is very less likely to overfit.  

* We trained our model in the source domain, and evaluated the model in target domain. The source domain and the target domain are exclusive to each other. Our results show a superior performance of our method than comparisons. For instance, for domain generalization, MultiHyperGNN beats DRAIN by 39.09% on average of MSE and 46.23% on average of Pearson correlation coefficient. MultiHyperGNN outperforms EERM by 20.41% on average of MSE and 35.34% on average of Pearson correlation coefficient. This also suggests that our model does not overfit data.

Reviewer comment (Evaluation):
The extent to which the model can generalize to radically different unseen domains is not clearly defined.

Author response:
We explained this in Theorem 1 and its proof:

* Theorem 1 suggests that the most informative meta information that can maximize the predictibility of $\mathcal G^{(k)}$ will perfectly generalize the model to the domain $k$.

* In Eq. (16) of Appendix, we demonstrate that more informative meta information (i.e., $m$), which leads to a higher probability $p(\\{X_i^{(k)}\\}_{i=1}^n|A^{(k)}, \\{G_i^{(j)}\\} _{i=1}^n, m)$ on the left hand side, results in a reduced generalization error on the right hand side. This theoretically suggests that richer meta information can effectively generalize better to the unseen domain.

Reviewer comment (Experiments):
It’s unclear how well the model scales to extremely large graphs or how transferable it is across significantly different domains.

Author response:
* The scalability to large graphs depends on the scalability of GNN (i.e., the encoder and the decoder), which is not the focus of this paper but can be a future work. This is because our work focuses on hypernetwork that generates encoder and decoder whose numbers of parameters are not correlated to the number of nodes or edges in the graphs. In our experiments, the largest graph size our model is trained on is 2,398, which does not cause any scalability issues. Additionally, many existing works in graph transformation also face the same challenge such as [1] and [2]. 

* How transferable the model is across significantly different domains depends on how well the meta information captures relationships among domains. If they are not able to capture relationships among different domains, then the domain generalization would still be very hard. We explained this in Theorem 1 and its proof, which suggests that richer meta information can effectively generalize better to the unseen domain.

* We also conducted ablation studies to show the significance of using informative meta information for domain generalization, HyperGNN is a single-to-multiple version of MultiHyperGNN and its variation HyperGNN-2 limits the meta information input to only the mode type. The results show that HyperGNN performs 40.86% better in MSE and 147.59% better in Pearson correlation coefficient than HyperGNN-2, as shown in the domain generalization performance on the Genes dataset (Table 2). This suggests that detailed meta information significantly enhances the capture of relationships among domains and improves domain generalization performance.

[1] Deep multi-attributed graph translation with node-edge co-evolution. ICDM 2019. 

[2] Deep graph translation. TNNLS 2022.

Reviewer comment (Experiments):
Can you elaborate on the computational efficiency of MultiHyperGNN, especially when dealing with very large graphs?

Author response:
* The scalability to large graphs depends on the scalability of GNN (i.e., the encoder and the decoder), which is not the focus of this paper but can be a future work. This is because our work focuses on hypernetwork that generates encoder and decoder whose numbers of parameters are not correlated to the number of nodes or edges in the graphs. In our experiments, the largest graph size our model is trained on is 2,398, which does not cause any scalability issues. Additionally, many existing works in graph transformation also face the same challenge such as [1] and [2]. 

* How transferable the model is across significantly different domains depends on how well the meta information captures relationships among domains. If they are not able to capture relationships among different domains, then the domain generalization would still be very hard. We explained this in Theorem 1 and its proof, which suggests that richer meta information can effectively generalize better to the unseen domain.

* We also conducted ablation studies to show the significance of using informative meta information for domain generalization, HyperGNN is a single-to-multiple version of MultiHyperGNN and its variation HyperGNN-2 limits the meta information input to only the mode type. The results show that HyperGNN performs 40.86% better in MSE and 147.59% better in Pearson correlation coefficient than HyperGNN-2, as shown in the domain generalization performance on the Genes dataset (Table 2). This suggests that detailed meta information significantly enhances the capture of relationships among domains and improves domain generalization performance.

[1] Deep multi-attributed graph translation with node-edge co-evolution. ICDM 2019. 

[2] Deep graph translation. TNNLS 2022.

Reviewer comment (Evaluation):
Could you discuss any limitations or boundaries in the model's ability to generalize across radically different domains?

Author response:
We explained this in Theorem 1 and its proof:

* Theorem 1 suggests that the most informative meta information that can maximize the predictibility of $\mathcal G^{(k)}$ will perfectly generalize the model to the domain $k$.

* In Eq. (16) of Appendix, we demonstrate that more informative meta information (i.e., $m$), which leads to a higher probability $p(\\{X_i^{(k)}\\}_{i=1}^n|A^{(k)}, \\{G_i^{(j)}\\} _{i=1}^n, m)$ on the left hand side, results in a reduced generalization error on the right hand side. This theoretically suggests that richer meta information can effectively generalize better to the unseen domain.

Reviewer comment (Evaluation):
Could you discuss any limitations or boundaries in the model's ability to generalize across radically different domains?

Author response:
We explained this in Theorem 1 and its proof:

* Theorem 1 suggests that the most informative meta information that can maximize the predictibility of $\mathcal G^{(k)}$ will perfectly generalize the model to the domain $k$.

* In Eq. (16) of Appendix, we demonstrate that more informative meta information (i.e., $m$), which leads to a higher probability $p(\\{X_i^{(k)}\\}_{i=1}^n|A^{(k)}, \\{G_i^{(j)}\\} _{i=1}^n, m)$ on the left hand side, results in a reduced generalization error on the right hand side. This theoretically suggests that richer meta information can effectively generalize better to the unseen domain.

Reviewer comment (Writing):
It is unclear about the practical use case of the proposed model, and hard to understand the problem setups. Can you elaborate on the potential real-world applications in which you proposed domain transformation problem can be applied?

Author response:
**Response 1:** 

*  I would like to highlight the use of graph transformation in cross-tissue gene prediction. Variations in gene expression play a significant role in the susceptibility to complex diseases [1], and these expressions are often specific to certain tissues [2]. Our work addresses the challenge of predicting gene expression in less accessible tissues (such as lung or brain) from data obtained from more accessible tissues (like whole blood) [3, 4]. We frame this as a graph transformation task, leveraging the gene-gene interaction networks from source tissue and target tissues. This application of graph transformation in gene prediction is notably emphasized in the Introduction of our paper (i.e., highlighted in red in the first paragraph of Introduction). 

* Furthermore, graph transformation has broader applications beyond gene prediction [5]. For example, chemists use it to predict chemical reaction outcomes of the output molecular graph from the input molecular graph [6], computer scientists apply it to distinguish between normal and compromised user authentication graphs [7], and urban planners utilize it for forecasting on traffic networks at different time points [8]. These diverse applications, cited in our submission ([5, 6, 7, 8]), demonstrate the versatility of graph transformation. Additionally, those tasks are of one-to-one graph transformation but not multiple-to-multiple because of time and space complexity, but we push it forward to multiple-to-multiple so that it can leverage more of the mode-specific information.

* To evaluate our model, we have incorporated datasets relevant to these tasks: the Genes dataset for cross-tissue gene prediction, the Climate dataset for temporal weather prediction, and the Traffic dataset for traffic flow prediction. The results, as shown in Tables 1 and 2, validate our model's efficacy across these varied applications. Notably, in Genes dataset, our model exhibits an average MSE 4.24% lower than the second-best model. Our model also has the Pearson correlation coefficient 4.74% higher by average than the second best model. In traffic forecasting, the proposed model surpasses the second-best model by 7.62% and 1.97% on average in MSE and Pearson correlation coefficient, respectively, for domain generalization.

[1] Mapping complex disease traits with global gene expression.Nature Reviews Genetics 2009. [2] Understanding tissue-specific gene regulation. Cell reports 2021. 

[3] Predicting tissue-specific gene expression from whole blood transcriptome. Science Advances 2021. 

[4] Prediction of the gene expression in normal lung tissue by the gene expression in blood. BMC medical genomics 2015.

[5] Graph Neural Networks: Graph Transformation. GNNs: Foundations, Frontiers, and Applications. 2021. 

[6] Deep multi-attributed graph translation with node-edge co-evolution. ICDM 2019. 

[7] Deep graph translation. TNNLS 2022. 

[8] Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. IJCNN 2018.

Reviewer comment (Theory):
It would be beneficial for the authors to explain why traditional methods require O(3^N) to solve the problem. Additionally, is it possible that the traditional method, despite its time-consuming nature, performs the best and can serve as a baseline?

Author response:
**Response 2:** We have elaborated on the complexity of our method and traditional methods in Section 4.1 and Appendix C of our paper, respectively, as highlighted in red.

* Traditional approaches require a complexity of $O(3^N)$ to address the problem, given that each of the N modes can serve as either a source, a target, or neither, leading to $3^N$ possible combinations. While these methods could be effective, they incur substantial costs in terms of parameter space and training time complexity. 

* We also compared our method with those that leverage hypernetworks (i.e., DRAIN) or adversarial traing (i.e., EERM) for domain generalization. MultiHyperGNNs shows consistently better performance compared with other models (i.e., Table 1 and Table 2). For instance, for domain generalization, MultiHyperGNN beats DRAIN by 39.09% on average of MSE and 46.23% on average of Pearson correlation coefficient. MultiHyperGNN outperforms EERM by 20.41% on average of MSE and 35.34% on average of Pearson correlation coefficient. 

* For optimal performance, these models necessitate training $O(3^N)$ encoder-decoder pairs to handle all mode combinations. Considering large datasets like the GTEx dataset, which includes over 45 tissue types (i.e., modes), this approach would require training an impractical number of combinations ($3^{45} \approx 3 \times 10^{21}$). Additionally, the lack of aligned training data for some mode combinations further complicates this approach, often rendering it infeasible.

* In contrast, our method processes arbitrary combinations of input and output modes efficiently. We leverage hypernetworks to dynamically generate the necessary encoders and decoders, addressing the challenge of handling the exponentially growing number of input-output mode combinations. This approach significantly reduces the computational burden and makes it feasible to work with large and complex datasets.

Reviewer comment (Experiments):
In the ablation studies, the authors compare with HyperGNN-1 and HyperGNN-2, but do not compare with MultiHyperGNN-1 or MultiHyperGNN-2. I noticed that HyperGNN contains only one decoder, and its structure does not align with the proposed MultiHyperGNN model. Could you please clarify this choice?

Author response:
**Response 4:**: We are grateful for the reviewer's valuable feedback, which has provided us an opportunity to clarify aspects of our study and enhance our paper.

* In the ablation study, HyperGNN serves as a simplified version of MultiHyperGNN, designed to predict multiple output modes from a single input mode. Therefore it contains only one encoder.

* To assess the importance of generating the final prediction layer via hypernetworks, we introduced HyperGNN-1, which learns this layer instead of producing it through hypernetworks. For a fair comparison, we evaluated HyperGNN-1 against HyperGNN rather than MultiHyperGNN. This comparison more directly tests the model's ability to learn all mode combinations, which is a challenging task. Our results, particularly the domain generalization performance on the Genes dataset (shown in Table 2), indicate that HyperGNN significantly outperforms HyperGNN-1, with improvements of 42.74% in MSE and 148.55% in Pearson correlation coefficient on average.

* Further, to test the hypothesis that more informative meta information yields better performance, we developed HyperGNN-2, which limits the meta information to only the mode type. Comparing HyperGNN-2 with HyperGNN, we again observed superior performance of HyperGNN in all settings. Specifically, HyperGNN shows 40.86% better MSE and 147.59% better Pearson correlation coefficient on average than HyperGNN-2, as evidenced by the domain generalization performance on the Genes dataset in Table 2.

* In summary, both HyperGNN-1 and HyperGNN-2 were introduced to validate the necessity of generating prediction layers from hypernetworks and utilizing informative meta information. They were compared with HyperGNN, not MultiHyperGNN. This discussion is highlighted in the second paragraph of Section 5.4.

* Additionally, we are conducting further experiments with MultiHyperGNN-1 and MultiHyperGNN-2 compared with MultiHyperGNN, which learns the prediction layer instead of producing it through hypernetworks and limits the meta information to only the mode type, respectively. The experimemts are based on MultiHyperGNN-GIN on Genes dataset regarding domain generalization (same experimental settings as Table 2). Below are the resutls:

| Model               | Genes-L |        | Genes-LG |        | Genes-S |        |
|-------------------|---------|--------|----------|--------|---------|--------|
|                   | MSE     | PCC    | MSE      | PCC    | MSE     | PCC    |
| MultiHyperGNN | **1.8005**  | **0.6600** | **1.9852**   | **0.6479** | **1.9031**  | **0.6471** |
| MultiHyperGNN-1   | 3.5236  | 0.2795 | 2.9149   | 0.3997 | 2.9512  | 0.4005 |
| MultiHyperGNN-2   | 3.0071  | 0.4114 | 3.0294   | 0.3316 | 3.1411  | 0.3130 |

Specifically, MultiHyperGNN outperforms MultiHyperGNN-1 by 39.70% regarding MSE and 80.98% regarding Pearson correlation coefficient on average, respectively. This indicates the importance of generating the final prediction layer via hypernetworks as training a single prediction layer may not well handel exponential number of mode combinations.

Similarly, MultiHyperGNN outperforms MultiHyperGNN-2 by 38.31% regarding MSE and 85.04% regarding Pearson correlation coefficient on average, respectively. This indicates that more informative meta information could help with better model performance on domain generalization.

Reviewer comment (Theory):
In Section 4.3, the definition of meta-information is very strict. It is necessary to discuss how meta-information in the datasets in experiments can satisfy the condition in the definition of meta-information. The authors mentioned that “An ample amount of meta-information will result in reduced generalization error”, but it is unclear of how to measure the “ample amount”, which makes the conclusion too vague.

Author response:
**Response 5:** We thank the reviewer for their insightful feedback and the opportunity to further clarify and enhance our paper.

* Ample meta information means the most informative meta data for guiding the generating of $\mathcal G^{(k)}$. It guides us how to select meta information, namely we select those that can maximize the predictibility of $\mathcal G^{(k)}$ and don't need to select those that cannot help the predictibility of $\mathcal G^{(k)}$.

* In practice, this theorem can guide us in several ways. First, we can try to solve this optimization. In some situation, we have prior knowledge that some meta information must be very important and some should be irrelevant so can guide our selection. In Eq. (16) of Appendix, we demonstrate that more informative meta information (i.e., $m$), which leads to a higher probability $p(\\{X_i^{(k)}\\}_{i=1}^n|A^{(k)}, \\{G_i^{(j)}\\} _{i=1}^n, m)$ on the left hand side, results in a reduced generalization error on the right hand side. This theoretically suggests that richer meta information can effectively lower generalization error.

* For example, in cross-tissue gene prediction, HyperGNN-2, using only tissue type as meta information, underperforms compared to HyperGNN, which employs richer meta information including tissue type, location, structure, and cell types (as shown in Tables 1 and 2). The optimal meta information selection can be determined through general model selection techniques, such as cross-validation, or domain expertise.

Reviewer comment (Writing):
The notation are overly complex and contain some errors, which hinders readability. For example, in Figure 2, encoder should be represented as f^{(2)}_e. The distinction between \widetilde{A} and \hat{A} in Equations 8 and 9 should be clarified. Moreover, Figure 2 is unclear to me. Whether the decoder's input is a graph or the latent embedding obtained by the encoder, as shown in Equation 11. Why there are two layers of decoder f^{(1)}_d.

Author response:
**Response 6:** 

* For further clarification on the notation, we have added a notation table in Appendix Table 3 as a reference for our readers.

* We have corrected the typo in Figure 1, changing $f_e^{(1)}$ to $f_e^{(2)}$. 

* The paper also addresses the challenge of predicting for graphs with different topologies in the source and target modes. We employ semi-supervised link prediction to infer edges between disjoint nodes. The final topology, $\hat{A}$, learned through semi-supervised link prediction and supervised by $\tilde{A}$ (the output mode's adjacency matrix), is detailed in Eq. (8) and (9), now highlighted in red in the revised paper.

* Yes, the decoder's input is a graph or the latent embedding obtained by the encoder. The decoder's role is two-fold: it first joins the potentially different topologies of source and output modes through semi-supervised link prediction, and then predicts values for all nodes. In this process, the first module of $f_d^{(1)}$ encodes the output mode's graph using node embeddings from the encoder, focusing on nodes present in the source mode. Subsequently, the second module of it performs semi-supervised link prediction for nodes outside the source mode and predicts final values for all nodes in the output mode.

Reviewer comment (Writing):
The introduction part seems to be incomplete. In the third paragraph, the three challenges of the problem are emphasized. The fourth paragraph should focus on writing the core idea of ​​solving these challenges in this work. However, the authors only wrote about their implementation, so I did not understand the innovative ideas of this work from a high-level perspective.

Author response:
We have detailed our strategies in the fourth paragraph of the revised paper, now highlighted in red for easier reference. Here are the specifics:

* **Challenge 1 - High Complexity in the Training Process:** “We introduced a novel framework leveraging a multi-input, multi-output training strategy, significantly reducing the space complexity regarding trainable parameters from exponential to constant during training.”

* **Challenge 2 - Graph Transformation Between Topologically Different Modes:** “We develop an encoder and a decoder for graph transformation between topologically different input and output modes, respectively. Additionally, our model conducts semi-supervised link prediction to complete the output graph, facilitating generalization to all nodes in the output mode.”

* **Challenge 3 - Learning Graph Transformation Involving Unseen Domains and Lack of Training Data:** “We design two novel hypernetworks that produce the encoder and the decoder for domain generalization. Mode-specific meta information serves as the input to guide the hypernetwork to produce the corresponding encoder or decoder, and generalize to unseen target domains.”

Reviewer comment (Experiments):
The paper does not adding ablation studies to understand the contribution of each component of MultiHyperGNN.

Author response:
Thank you for your interest in our ablation studies. We conducted these studies, HyperGNN, HyperGNN-1 and HyperGNN-2 to rigorously test various aspects of our model. Below are the details of our findings:

* **HyperGNN**. To evaluate whether more input domains will result in better predictive performance. HyperGNN is a simpler version of MultiHyperGNN that predicts multiple output modes from a single input mode. In gene prediction tasks, MultiHyperGNN surpassed HyperGNN by an average of 4.24% in MSE and 4.74% in Pearson correlation coefficient. This demonstrates the advantages of using multiple input modes for domain generation in graph transformation.

* **HyperGNN-1**. To evaluate if a single MLP prediction layer is able to predict all output modes rather than being learned from the decoder hypernetwork, unlike HyperGNN, HyperGNN-1 does not generate MLP layers but only GNN layers via a hypernetwork. The domain generalization performance on the Genes dataset (Table 2) shows that HyperGNN significantly outperforms HyperGNN-1, with a 42.74% improvement in MSE and 148.55% in Pearson correlation coefficient. This indicates the benefit of using mode-specific prediction layers.

* **HyperGNN-2**. To determine the importance of using informative meta information for domain generalization, HyperGNN-2 limits the meta information input to only the mode type. The results show that HyperGNN performs 40.86% better in MSE and 147.59% better in Pearson correlation coefficient than HyperGNN-2, as shown in the domain generalization performance on the Genes dataset (Table 2). This suggests that detailed meta information significantly enhances the capture of relationships among domains and improves domain generalization performance.

Reviewer comment (Writing):
While the challenges are listed, a more in-depth discussion on how each challenge is specifically addressed by MultiHyperGNN would enhance understanding.

Author response:
* Challenge 1 - High Complexity in the Training Process: 

    - In our contribution list under paragraph four, we claimed our strategy to address the heavy complexity of traditional methods: “We introduced a novel framework leveraging a multi-input, multi-output training strategy, significantly reducing the space complexity regarding trainable parameters from exponential to constant during training.”

    - In the last paragraph of Section 4.1, we have explained how MultiHyperGNN could reduce space complexity. This part has been highlighted in red in the revised paper for the easier reference for the reviewer.
    
    - We conducted complexity analysis in Appendix C, which is highlighted in red for easier reference. The analysis shows that our method is able to achieve a constant training space complexity, which is the same as DRAIN, but superior than other baselines. Additionally, our method is able to achieve a polynomial (i.e., $O(N^2)$) training computational complexity, which is better than other baselines as well.

* Challenge 2 - Graph Transformation Between Topologically Different Modes:

    - In our contribution list under paragraph four, we have explained our strategy to handle topologically different input and output modes: “We develop an encoder and a decoder for graph transformation between topologically different input and output modes, respectively. Additionally, our model conducts semi-supervised link prediction to complete the output graph, facilitating generalization to all nodes in the output mode.”

    - In the Section 4.2, we introduced the detailed framework of GNN-based encoder and decoder to handle different input and output topologies. We also introduced how we perform semi-supervised link prediction to predict nodes that are not overlapped by the input and output graphs.

    - We also conduced another ablation study by replacing GNN in the encoder and the decoder with MLP, i.e., MultiHyperGNN-MLP, which cannot well learn the graph topologies of input and output modes. The results in Table 1 and Table 2 show that MultiHyperGNN-MLP has a much worse predictive and domain generalization performation compared with MultiHyperGNN.

* Challenge 3 - Learning Graph Transformation Involving Unseen Domains and Lack of Training Data:

    - In our contribution list under paragraph four, we claimed our strategy for domain generalization by leveraging relationships among domains via meta information: “We design two novel hypernetworks that produce the encoder and the decoder for domain generalization. Mode-specific meta information serves as the input to guide the hypernetwork to produce the corresponding encoder or decoder, and generalize to unseen target domains.”

    - In Section 4.3, we introduced detailed design of hypernetworks that take domain-specific meta information as the input to guide the domain generalization. Particularly, to guide the selection of meta information, we introduced a theory (i.e., Theorem 1) and its proof to indicate that informative meta data for guiding the generating of the output mode could help reduce the generalization error. 

    - We also conducted ablation study to address the significance of using informative meta information for domain generalization. HyperGNN-2 limits the meta information input to only the mode type. The results show that HyperGNN performs 40.86% better in MSE and 147.59% better in Pearson correlation coefficient than HyperGNN-2, as shown in the domain generalization performance on the Genes dataset (Table 2). This suggests that detailed meta information significantly enhances the capture of relationships among domains and improves domain generalization performance.

Reviewer comment (Experiments):
Consider adding ablation studies to understand the contribution of each component of MultiHyperGNN.

Author response:
Thank you for your interest in our ablation studies. We conducted these studies, HyperGNN, HyperGNN-1 and HyperGNN-2 to rigorously test various aspects of our model. Below are the details of our findings:

* **HyperGNN**. To evaluate whether more input domains will result in better predictive performance. HyperGNN is a simpler version of MultiHyperGNN that predicts multiple output modes from a single input mode. In gene prediction tasks, MultiHyperGNN surpassed HyperGNN by an average of 4.24% in MSE and 4.74% in Pearson correlation coefficient. This demonstrates the advantages of using multiple input modes for domain generation in graph transformation.

* **HyperGNN-1**. To evaluate if a single MLP prediction layer is able to predict all output modes rather than being learned from the decoder hypernetwork, unlike HyperGNN, HyperGNN-1 does not generate MLP layers but only GNN layers via a hypernetwork. The domain generalization performance on the Genes dataset (Table 2) shows that HyperGNN significantly outperforms HyperGNN-1, with a 42.74% improvement in MSE and 148.55% in Pearson correlation coefficient. This indicates the benefit of using mode-specific prediction layers.

* **HyperGNN-2**. To determine the importance of using informative meta information for domain generalization, HyperGNN-2 limits the meta information input to only the mode type. The results show that HyperGNN performs 40.86% better in MSE and 147.59% better in Pearson correlation coefficient than HyperGNN-2, as shown in the domain generalization performance on the Genes dataset (Table 2). This suggests that detailed meta information significantly enhances the capture of relationships among domains and improves domain generalization performance.