Original: G RAPH D OMAIN A DAPTATION VIA T HEORY -G ROUNDED S PECTRAL R EGULARIZATION
Revised: G RAPH D OMAIN A DAPTATION VIA T HEORY -G ROUNDED S PECTRAL R EGULARIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transfer learning on graphs drawn from varied distributions (domains) is in great demand across many applications.
Revised: Transfer learning on graphs drawn from varied distributions (domains) is in great demand across many applications.

Original: Emerging methods attempt to learn domaininvariant representations using graph neural networks (GNNs), yet the empirical performances vary and the theoretical foundation has been limited.
Revised: Emerging methods attempt to learn domaininvariant representations using graph neural networks (GNNs), yet the empirical performances vary and the theoretical foundation is limited.

Original: This paper targets at designing theory-grounded algorithms for graph domain adaptation (GDA).
Revised: This paper aims at designing theory-grounded algorithms for graph domain adaptation (GDA).

Original: ( i )
Revised: ( i )

Original: As the first attempt, we derive a model-based GDA bound closely related to two
Revised: As the first attempt, we derive a model-based GDA bound closely related to two

Original: GNN spectral properties: spectral smoothness (SS) and maximum frequency response (MFR) .
Revised: GNN spectral properties: spectral smoothness (SS) and maximum frequency response (MFR) .

Original: This is achieved by cross-pollinating between the OT-based (optimal transport) DA and graph filter theories.
Revised: This is achieved by cross-pollinating between the OT-based (optimal transport) DA and graph filter theories.

Original: ( ii ) Inspired by the theoretical results, we propose algorithms regularizing spectral properties of SS and MFR to improve
Revised: ( ii ) Inspired by the theoretical results, we propose algorithms regularizing spectral properties of SS and MFR to improve

Original: GNN transferability.
Revised: GNN transferability.

Original: We further extend the GDA theory into the more challenging conditional-shift scenario, where spectral regularization still applies.
Revised: We further extend the GDA theory into the more challenging scenario of conditional shift, where spectral regularization still applies.

Original: ( iii ) More importantly, our analyses of the theory reveal which regularization would improve performance of what transfer learning scenario, ( iv ) withthe numerical agreement from extensive real-world experiments: SS and MFR regularizations bring more benefits to the scenarios of node transfer and link transfer , respectively.
Revised: ( iii ) More importantly, our analyses of the theory reveal which regularization would improve performance of what transfer learning scenario, ( iv ) with numerical agreement with extensive real-world experiments: SS and MFR regularizations bring more benefits to the scenarios of node transfer and link transfer , respectively.

Original: In a nutshell, our study paves the way toward explicitly constructing and training GNNs that can capture more transferable representations across graph domains.
Revised: In a nutshell, our study paves the way toward explicitly constructing and training GNNs that can capture more transferable representations across graph domains.

Original: Codes will be fully released upon acceptance.
Revised: Codes are released at https://github.com/Shen-Lab/GDA-SpecReg .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many applications call for “transferring” graph representations learned from one distribution (domain) to another, which we refer it as graph domain adaptation (GDA), for instance, temporally evolved social networks (Wang et al., 2021), molecules of different scaffolds (Hu et al., 2019), and protein-protein interaction networks in various species (Cho et al., 2016).
Revised: Many applications call for “transferring” graph representations learned from one distribution (domain) to another, which we refer to as graph domain adaptation (GDA). Examples include temporally-evolved social networks (Wang et al., 2021), molecules of different scaffolds (Hu et al., 2019), and protein-protein interaction networks in various species (Cho et al., 2016).

Original: In general, this setting of transfer learning is challenging due to the data-distribution shift between the training (source) and test (target) domains (i.e. P S ( G, Y )
Revised: In general, this setting of transfer learning is challenging due to the data-distribution shift between the training (source) and test (target) domains (i.e. P S ( G, Y )

Original: ̸
Revised: ̸

Original: = P T ( G, Y ) ).
Revised: = P T ( G, Y ) ).

Original: In particular, such a challenge escalates for graph-structured data that are abstractions of diverse nature (You et al., 2021; 2022).
Revised: In particular, such a challenge escalates for graph-structured data that are abstractions of diverse nature (You et al., 2021; 2022).

Original: Despite the tremendous needs arising from real-world applications, current methods for GDA (as reviewed in Section 2) mostly fall short in delivering competitive target performance with theoretical guarantee .
Revised: Despite the tremendous needs arising from real-world applications, current methods for GDA (as reviewed in Section 2) mostly fall short in delivering competitive target performance with theoretical guarantee .

Original: Inevitably those approaches assuming distribution invariance (or adopting heuristic principles) are restricted in theory (Garg et al., 2020; Verma & Zhang, 2019).
Revised: Inevitably those approaches assuming distribution invariance (or adopting heuristic principles) are restricted in theory (Garg et al., 2020; Verma & Zhang, 2019).

Original: The emerging approaches (Zhang et al., 2019; Wu et al., 2020) straightforwardly apply adversarial training between source and target representations, intentionally founded on the DA theory to bound the target risk (Redko et al., 2020).
Revised: The emerging approaches (Zhang et al., 2019; Wu et al., 2020) straightforwardly apply adversarial training between source and target representations, intentionally founded on the DA theory to bound the target risk (Redko et al., 2020).

Original: However, the generic DA bound in theory is agnostic to graph data and models, which could be operated on to further improve transferability.
Revised: However, the generic DA bound in theory is agnostic to graph data and models, which could be more precisely tailored for graphs.

Original: We therefore set out to explore the following question: How to design algorithms to boost transfer performance across different graph domains, with the grounded theoretical foundation?
Revised: We therefore set out to explore the following question: How to design algorithms to boost transfer performance across different graph domains, with the grounded theoretical foundation?

Original: Our step-by-step answers are as follows.
Revised: Our step-by-step answers are as follows.

Original: ( i ) Derivation of model-based GDA bound.
Revised: ( i ) Derivation of model-based GDA bound.

Original: Building upon the rigorous assurance established in the DA theory (Section 3), we start from directly rewriting the OT-based (optimal transport) DA bound (Redko et al., 2017;
Revised: Building upon the rigorous assurance established in the DA theory (Section 3), we start by directly rewriting the OT-based (optimal transport) DA bound (Redko et al., 2017;

Original: Shen et al., 2018) for graphs (Corollary 1), which is closely coupled with the Lipschitz constant of graph encoders.
Revised: Shen et al., 2018) for graphs (Corollary 1), which is closely coupled with the Lipschitz constant of graph encoders.

Original: The nontrivial challenge here is how to formulate GNN Lipschitz w.r.t the distance metric of non-Euclidean data.
Revised: The nontrivial challenge here is how to formulate GNN Lipschitz w.r.t the distance metric of non-Euclidean data.

Original: Leveraging the graph filter theory (Gama et al., 2020; Arghal et al., 2021), we first state that GNNs can be constructed stably w.r.t. 
Revised: Leveraging the graph filter theory (Gama et al., 2020; Arghal et al., 2021), we first state that GNNs can be constructed stably w.r.t. the

Original: the misalignment of edges and that of node features, multiplied by two spectral properties respectively: spectral smoothness (SS) and maximum frequency response (MFR) (Lemma 1).
Revised: misalignment of edges and that of node features, multiplied by two spectral properties respectively: spectral smoothness (SS) and maximum frequency response (MFR) (Lemma 1).

Original: Subsequently, we utilize SS and MFR to formulate GNN Lipschitz w.r.t graph distances into a general form, and instantiate it as (informally) max (cid:8) O (SS) , O (MFR) (cid:9) w.r.t.
Revised: Subsequently, we utilize SS and MFR to formulate GNN Lipschitz w.r.t graph distances into a general form, and instantiate it as (informally) max (cid:8) O (SS) , O (MFR) (cid:9) w.r.t.

Original: the commonly-used matching distance (Gama et al., 2020; Arghal et al., 2021)
Revised: the commonly-used matching distance (Gama et al., 2020; Arghal et al., 2021)

Original: (Lemma 2).
Revised: (Lemma 2).

Original: This leads to the first model-based GDA bound.
Revised: This leads to the first model-based GDA bound.

Original: ( ii ) Theory-grounded spectral regularization.
Revised: ( ii ) Theory-grounded spectral regularization.

Original: One potential way to tighten the DA bound is to modulate the Lipschitz constant (Section 3).
Revised: One potential way to tighten the DA bound is to modulate the Lipschitz constant (Section 3).

Original: Guided by the theoretical results above, we are wellmotivated to propose spectral regularization (i.e. SSReg and MFRReg) to restrict the target risk bound (Section 4.2).
Revised: Guided by the theoretical results above, we are wellmotivated to propose spectral regularization (i.e. SSReg and MFRReg) to restrict the target risk bound (Section 4.2).

Original: We also extend the GDA theory into the more challenging conditional-shift scenario (Li et al., 2021a;
Revised: We also extend the GDA theory into the more challenging conditional-shift scenario (Li et al., 2021a;

Original: Zhao et al., 2019)
Revised: Zhao et al., 2019)

Original: (Lemma 3), where spectral regularization still applies.
Revised: (Lemma 3), where spectral regularization still applies.

Original: ( iii ) Interpretation on how theory drives practice.
Revised: ( iii ) Interpretation on how theory drives practice.

Original: Our further analyses on theory reveal which regularization would improve performance of what graph transfer scenario : specifically, SSReg and
Revised: Our further analyses on theory reveal which regularization would improve performance of what graph transfer scenario : specifically, SSReg and

Original: MFRReg are respectively beneficial to the scenarios of node transfer and link transfer (Section 4.2), ( iv ) with extensive numerical evidences from un/semi-supervised (cross-species protein-protein interaction) link prediction and (temporally shifted paper topic) 
Revised: MFRReg are respectively beneficial to the scenarios of node transfer and link transfer (Section 4.2), ( iv ) with extensive numerical evidences from un/semi-supervised (cross-species protein-protein interaction) link prediction and (temporally-shifted paper topic) node classification

Original: node classification (Section 5).
Revised: (Section 5).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Self-supervision on graphs.
Revised: Self-supervision on graphs.

Original: Graph self-supervised learning, surging recently, learns empirically more generalizable representations through exploiting vast unlabelled graph data (please refer to (Xie et al., 2021) for a comprehensive review).
Revised: Graph self-supervised learning, surging recently, learns empirically more generalizable representations through exploiting vast unlabelled graph data (please refer to (Xie et al., 2021) for a comprehensive review).

Original: The success of self-supervision largely hinges on big data and, more importantly, heuristically-designed pretext tasks.
Revised: The success of self-supervision largely hinges on big data and, more importantly, heuristically-designed pretext tasks.

Original: The tasks can be predictive (Velickovic et al., 2019;
Revised: The tasks can be predictive (Velickovic et al., 2019;

Original: Hu et al., 2019; Jin et al., 2020; You & Shen, 2022; You et al., 2020b) or contrastive (You et al., 2020a; Zhu et al., 2020b; Qiu et al., 2020; Wei et al., 2022), which does not provide theoretical guarantee of the target performance and, as a result, occasionally leads to “negative transfer” in practice (Hu et al., 2019; You et al., 2020a).
Revised: Hu et al., 2019; Jin et al., 2020; You & Shen, 2022; You et al., 2020b; Chien et al., 2021; Talukder et al., 2022) or contrastive (You et al., 2020a; Zhu et al., 2020b; Qiu et al., 2020; Wei et al., 2022), which does not provide theoretical guarantee of the target performance and, as a result, occasionally leads to “negative transfer” in practice (Hu et al., 2019; You et al., 2020a).

Original: Transferring GNNs with explicit covariate shifts.
Revised: Transferring GNNs with explicit covariate shifts.

Original: To promote target performance, one line of work is to utilize more data and make specific assumptions.
Revised: To promote target performance, one line of work is to utilize more data and make specific assumptions.

Original: One such example is to assume the access to source labels and the explicit covariate shift that P S ( Y | G )
Revised: One such example is to assume the access to source labels and the explicit covariate shift that P S ( Y | G )

Original: =
Revised: =

Original: P T ( Y | G ) and P S ( G )
Revised: P T ( Y | G ) and P S ( G )

Original: ̸
Revised: ̸

Original: = P T ( G ) in a specific way, which enables theoretical tools for certain guarantees.
Revised: = P T ( G ) in a specific way, which enables theoretical tools for certain guarantees.

Original: (Ruiz et al., 2020; Yehudai et al., 2021) study the specific setting of size generalization and use the graphon theory (Lov ´ asz, 2012) to develop size-invariant representations.
Revised: (Ruiz et al., 2020; Yehudai et al., 2021) study the specific setting of size generalization and use the graphon theory (Lov ´ asz, 2012) to develop size-invariant representations.

Original: (Bevilacqua et al., 2021) works on transfer learning in shifting
Revised: (Bevilacqua et al., 2021) works on transfer learning in shifting

Original: d -patterns of subgraphs and adopts the theory of GNN expressiveness (Xu et al., 2018;
Revised: d -patterns of subgraphs and adopts the theory of GNN expressiveness (Xu et al., 2018;

Original: Morris et al., 2019) to demonstrate the existence of negative-transferring GNNs despite their universal approximation capability.
Revised: Morris et al., 2019) to demonstrate the existence of negative-transferring GNNs despite their universal approximation capability.

Original: Accordingly, the study proposes d -pattern classification pre-training to help escape from negative-transferring GNNs.
Revised: Accordingly, the study proposes d -pattern classification pre-training to help escape from negative-transferring GNNs.

Original: These methods are restricted to the designated transfer learning scenarios.
Revised: These methods are restricted to the designated transfer learning scenarios.

Original: Besides, some other works (Fan et al., 2021;
Revised: Besides, some other works (Fan et al., 2021;

Original: Sui et al., 2021; Li et al., 2021b; Kenlay et al., 2021; Chen et al., 2022; Li et al., 2022; Zhang et al.,  2022) adopt the implicit covariate shift assumption with source labels while lacking assurance in theory, e.g. (Wu et al. , 2022a;b) assume the shift could be implicitly modeled with an environment learner.
Revised: Sui et al., 2021; Li et al., 2021b; Kenlay et al., 2021; Chen et al., 2022; Li et al., 2022; Zhang et al., 2022; Jin et al., 2022) adopt the implicit covariate shift assumption with source labels while lacking assurance in theory, e.g. (Wu et al., 2022a;b) assumes that the shift could be implicitly modeled with an environment learner (please refer to (Gui et al., 2022) for a comprehensive review).

Original: Graph domain adaptation.
Revised: Graph domain adaptation.

Original: To deliver a generally applicable guarantee, several methods (Dai et al., 2019;
Revised: To deliver a generally applicable guarantee, several methods (Dai et al., 2019;

Original: Cai et al., 2021; Zhang et al., 2019; Wu et al., 2020; Xu et al., 2022) additionally utilize target graphs to learn domain-invariant representations.
Revised: Cai et al., 2021; Zhang et al., 2019; Wu et al., 2020; Xu et al., 2022) additionally utilize target graphs to learn domain-invariant representations.

Original: According to the DA theory (Ben-David et al., 2007; 2010; Redko et al., 2020; Zhang et al., 2020; Yan et al., 2017), the target risk is guaranteed to be bounded (please refer to (Redko et al., 2020) for a comprehensive review).
Revised: According to the DA theory (Ben-David et al., 2007; 2010; Redko et al., 2020; Zhang et al., 2020; Yan et al., 2017), the target risk is guaranteed to be bounded (please refer to (Redko et al., 2020) for a comprehensive review).

Original: The generic DA bound is not designated for graph distributions or encoders where further improvement could be achieved.
Revised: The generic DA bound is not designated for graph data or encoders where further improvement could be achieved.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem setup.
Revised: Problem setup.

Original: We are given i.i.d.
Revised: We are given i.i.d.

Original: samples (Verma & Zhang, 2019; Zhu et al., 2021; Cong et al., 2021) and their labels { ( G n , Y n ) } N S n =1 from the source distribution P S ( G, Y ) of graphs G ∈ G and labels Y ∈ Y , where G =
Revised: samples (Verma & Zhang, 2019; Zhu et al., 2021; Cong et al., 2021) and their labels { ( G n , Y n ) } N S n =1 from the source distribution P S ( G, Y ) of graphs G ∈ G and labels Y ∈ Y , where G =

Original: { V, E } is associated with the set of nodes V and edges E , together with the node feature X ∈ R | V |× D and adjacency matrices A ∈ R | V |×| V | .
Revised: { V, E } is associated with the set of nodes V and edges E , together with the node feature X ∈ R | V |× D and adjacency matrices A ∈ R | V |×| V | .

Original: We also have access to unlabeled samples { G n } N T n =1 from the marginalized target distribution (cid:82) P T ( G, Y ) dY .
Revised: We also have access to unlabeled samples { G n } N T n =1 from the marginalized target distribution (cid:82) P T ( G, Y ) dY .

Original: With the covariate shift assumption that P S ( G ) ̸ = P T ( G ) , P S ( Y | G )
Revised: With the covariate shift assumption that P S ( G ) ̸ = P T ( G ) , P S ( Y | G )

Original: =
Revised: =

Original: P T ( Y | G ) (Ben-David et al., 2007; 2010), we are expected to train a graph neural network (GNN) h : G → Y with the accessible data and then evaluate on target samples from P T ( G, Y ) .
Revised: P T ( Y | G ) (Ben-David et al., 2007; 2010), we are expected to train a graph neural network (GNN) h : G → Y with the accessible data and then evaluate on target samples from P T ( G, Y ) .

Original: Domain adaptation with optimal transport.
Revised: Domain adaptation with optimal transport.

Original: Studies on transfer learning across distinctly distributed data have proliferated in the past few years, known as domain adaptation (DA) (Redko et al., 2020).
Revised: Studies on transfer learning across distinctly distributed data have proliferated in the past few years, known as domain adaptation (DA) (Redko et al., 2020).

Original: Based on the aforementioned problem setup, we decompose the trained GNN h
Revised: Based on the aforementioned problem setup, we decompose the trained GNN h

Original: = g ◦ f into the feature extractor f : G → R D ′
Revised: = g ◦ f into the feature extractor f : G → R D ′

Original: ( Z = f ( G ) ) and discriminator g : R D ′
Revised: ( Z = f ( G ) ) and discriminator g : R D ′

Original: → Y ( Y = g ( Z ) ).
Revised: → Y ( Y = g ( Z ) ).

Original: Without the loss of generalizability, we consider a binary classification task where
Revised: Without the loss of generalizability, we consider a binary classification task where

Original: Y =
Revised: Y =

Original: [0 , 1] and Y ∈ Y is the probability to belong to class 1.
Revised: [0 , 1] and Y ∈ Y is the probability to belong to class 1.

Original: We denote the labeling function given representations as ˆ g : R D ′
Revised: We denote the labeling function given representations as ˆ g : R D ′

Original: → Y , and the (empirical) source and target risks as ˆ ϵ S ( g, ˆ g ) =
Revised: → Y , and the (empirical) source and target risks as ˆ ϵ S ( g, ˆ g ) =

Original: 1 N S (cid:80) N S n =1 | g ( Z n ) − ˆ g ( Z n ) | and ϵ T ( g, ˆ g )
Revised: 1 N S (cid:80) N S n =1 | g ( Z n ) − ˆ g ( Z n ) | and ϵ T ( g, ˆ g )

Original: =
Revised: =

Original: E P T ( Z ) (cid:8)
Revised: E P T ( Z ) (cid:8)

Original: | g ( Z ) − ˆ g ( Z ) | (cid:9) , respectively.
Revised: | g ( Z ) − ˆ g ( Z ) | (cid:9) , respectively.

Original: Applying DA with optimal transport (OT), if the covariate shift holds on representations that P S ( Y | Z )
Revised: Applying DA with optimal transport (OT), if the covariate shift holds on representations that P S ( Y | Z )

Original: =
Revised: =

Original: P T ( Y | Z ) , the target risk ϵ T ( g, ˆ g ) is bounded as in the following theorem.
Revised: P T ( Y | Z ) , the target risk ϵ T ( g, ˆ g ) is bounded as in the following theorem.

Original: Theorem 1 (Redko et al., 2017; Shen et al., 2018; Li et al., 2021a)
Revised: Theorem 1 (Redko et al., 2017; Shen et al., 2018; Li et al., 2021a)

Original: Suppose that the learned discriminator g is C g -Lipschitz where the Lipschitz norm
Revised: Suppose that the learned discriminator g is C g -Lipschitz where the Lipschitz norm

Original: ∥ g ∥ Lip = max Z 1 ,Z 2 | g ( Z 1 ) − g
Revised: ∥ g ∥ Lip = max Z 1 ,Z 2 | g ( Z 1 ) − g

Original: ( Z 2 )
Revised: ( Z 2 )

Original: | ρ ( Z 1 ,Z 2 ) =
Revised: | ρ ( Z 1 ,Z 2 ) =

Original: C g holds for some distance function ρ (Euclidean distance here).
Revised: C g holds for some distance function ρ (Euclidean distance here).

Original: Let H :
Revised: Let H :

Original: = { g : Z → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H ) =
Revised: = { g : Z → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H ) =

Original: d that g ∈ H , with probability at least 1 − δ the following inequality holds:
Revised: d that g ∈ H , with probability at least 1 − δ the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where ω = min ∥ g ∥ Lip ≤ C g (cid:8) ϵ S ( g, ˆ g )+ ϵ T ( g, ˆ g ) (cid:9) denotes the model discriminative ability (to capture source and target data), and the first Wasserstein distance is defined as (Villani, 2009):
Revised: where ω = min ∥ g ∥ Lip ≤ C g (cid:8) ϵ S ( g, ˆ g )+ ϵ T ( g, ˆ g ) (cid:9) denotes the model discriminative ability (to capture source and target data), and the first Wasserstein distance is defined as (Villani, 2009):

Original: [Equation]
Revised: [Equation]

Original: The thorough tightness justification of the OT-based DA bound can be found in ((Redko et al., 2020), Section 5.3-5.5).
Revised: The thorough tightness justification of the OT-based DA bound can be found in ((Redko et al., 2020), Section 5.3-5.5).

Original: The theorem indicates that the generalization gap depends on both the domain-divergence 2 C g W 1 ( P S ( Z ) , P T ( Z )) and the model discriminability ω .
Revised: The theorem indicates that the generalization gap depends on both the domain-divergence 2 C g W 1 ( P S ( Z ) , P T ( Z )) and the model discriminability ω .

Original: Adversarial training.
Revised: Adversarial training.

Original: Motivated from Theorem 1 (or its variants differing mainly in distribution divergence metrics (Ben-David et al., 2007; 2010; Mansour et al., 2009)), a well-developed practice is to learn domain-invariant representations via jointly optimizing the source risk and the distribution divergence term (conceptually, W 1 ( P S ( Z ) , P T ( Z )) )
Revised: Motivated from Theorem 1 (or its variants differing mainly in distribution divergence metrics (Ben-David et al., 2007; 2010; Mansour et al., 2009)), a well-developed practice is to learn domain-invariant representations via jointly optimizing the source risk and the distribution divergence term (conceptually, W 1 ( P S ( Z ) , P T ( Z )) )

Original: as  (Redko et al., 2017;
Revised: as in (Redko et al., 2017;

Original: Shen et al., 2018):
Revised: Shen et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: where ℓ is the loss function used for training, ˆ W 1 is the empirically calculated first Wasserstein distance with implementation details following (Shen et al., 2018) presented in Appendix E, and γ is the trade-off factor.
Revised: where ℓ is the loss function used for training, ˆ W 1 is the empirically calculated first Wasserstein distance with implementation details following (Shen et al., 2018) presented in Appendix E, and γ is the trade-off factor.

Original: Except for optimizing ˆ W 1 , implementing domain classifiers is another effective way to alleviate the domain discrepancy (Zhang et al., 2019; Wu et al., 2020).
Revised: Besides co-optimizing ˆ W 1 , implementing domain classifiers is another effective way to alleviate the domain discrepancy (Zhang et al., 2019; Wu et al., 2020).

Original:  M ETHODS 4. M ODEL -B ASED DA B OUND FOR G RAPH -S TRUCTURED D ATA
Revised:  M ETHODS 4. M ODEL -B ASED DA B OUND FOR G RAPH -S TRUCTURED D ATA

Original: Notice that the generic DA theory (Theorem 1) is agnostic to data structures and encoders, our first step is to directly rewrite it for graph-structured data ( G ) accompanied with graph feature extractors ( f ) as follows.
Revised: Noticing that the generic DA theory (Theorem 1) is agnostic to data structures and encoders, our first step is to directly rewrite it for graph-structured data ( G ) accompanied with graph feature extractors ( f ) as follows.

Original: The covariate shift assumption is now reframed as P S ( Y | G ) =
Revised: The covariate shift assumption is now reframed as P S ( Y | G ) =

Original: P T ( Y | G ) .
Revised: P T ( Y | G ) .

Original: Corollary 1.
Revised: Corollary 1.

Original: Let’s assume that the learned discriminator is C g -Lipschitz continuous as described in Theorem 1, and the graph feature extractor f (we also refer as GNN) is C f -Lipschitz that∥ f ∥ Lip =
Revised: Let’s assume that the learned discriminator is C g -Lipschitz continuous as described in Theorem 1, and the graph feature extractor f (also referred to as GNN) is C f -Lipschitz that

Original: [Equation]
Revised: [Equation]

Original: = C f for some graph distance measure η .
Revised: = C f for some graph distance measure η .

Original: Let H :
Revised: Let H :

Original: = { h : G → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H ) =
Revised: = { h : G → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H ) =

Original: d that h = g ◦ f ∈ H , with probability at least 1 − δ the following inequality holds:
Revised: d that h = g ◦ f ∈ H , with probability at least 1 − δ the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where the (empirical) source and target risks are ˆ ϵ S
Revised: where the (empirical) source and target risks are ˆ ϵ S

Original: ( h, ˆ h ) =
Revised: ( h, ˆ h ) =

Original: 1 N S (cid:80) N S n =1 | h ( G n ) − ˆ h ( G n )
Revised: 1 N S (cid:80) N S n =1 | h ( G n ) − ˆ h ( G n )

Original: | and ϵ T ( h, ˆ h )
Revised: | and ϵ T ( h, ˆ h )

Original: =
Revised: =

Original: E P T ( G ) (cid:8)
Revised: E P T ( G ) (cid:8)

Original: | h ( G )
Revised: | h ( G )

Original: −
Revised: −

Original: ˆ h ( G ) | (cid:9) , respectively, where ˆ h : G → Y is the labeling function for graphs and ω = min ∥ g ∥ Lip ≤ C g ,
Revised: ˆ h ( G ) | (cid:9) , respectively, where ˆ h : G → Y is the labeling function for graphs and ω = min ∥ g ∥ Lip ≤ C g ,

Original: ∥ f
Revised: ∥ f

Original: ∥ Lip ≤ C f (cid:8)ϵ S
Revised: ∥ Lip ≤ C f (cid:8)

Original: [Table]
Revised: [Table]

Original: , ˆ

Original: [Table]

Original: [Table]

Original: ϵ

Original: [Table] [Table] [Table] The property in Corollary 1 designated for GNNs, as wemainly focus on, is the Lipschitz constant
Revised: The property in Corollary 1 designated for GNNs, as we focus on, is the Lipschitz constant

Original: C f = max G 1 ,G 2 ∥
Revised: C f = max [Equation]

Original: f ( G 1 ) −f
Revised: [Equation] −

Original: ( G 2 ) ∥ 2 η ( G 1 ,G 2 ) .
Revised: [Equation] [Equation] [Equation]

Original: Success in instantiating this conceptual model-related property for graphs provides important insights, on how to rationally construct or train GNNs for better transferability.
Revised: Success in instantiating this conceptual model-related property for graphs provides important insights into how to rationally construct or train GNNs for better transferability.

Original: Other data-relevant properties (e.g. W 1 ( P S ( G ) , P T ( G )) ) are left in the future works.
Revised: Other data-relevant properties (e.g. W 1 ( P S ( G ) , P T ( G )) ) are left to future works.

Original: Instantiating the GNN Lipschitz constant is however nontrivial as in Euclidean data, since the distance metric η ( G 1 , G 2 ) is formulated w.r.t non-Euclidean structures, the computationof which hinges on solving the graph matching problem, whose time complexity is exponential in the number of nodes (Riesen & Bunke, 2009;
Revised: Instantiating the GNN Lipschitz constant is however nontrivial. The distance metric η ( G 1 , G 2 ) is formulated w.r.t non-Euclidean structures and its computation hinges on solving the graph matching problem whose time complexity is exponential in the number of nodes (Riesen & Bunke, 2009;

Original: Riesen et al., 2010).
Revised: Riesen et al., 2010).

Original: Rather than working on the denominator of the Lipschitz norm, we take an alternative perspective of the numerator, i.e. ∥ f ( G 1 ) − f ( G 2 ) ∥ 2 , which is related to the GNN stability (potentially multiplied with the distance term to eliminate the denominator).
Revised: Rather than working on the denominator of the Lipschitz norm, we take an alternative perspective of the numerator, i.e. ∥ f ( G 1 ) − f ( G 2 ) ∥ 2 , which is related to the GNN stability (potentially multiplied with the distance term to eliminate the denominator).

Original: This essentially motivates us to draw a connection between transferability and stability.
Revised: This essentially motivates us to draw a connection between transferability and stability.

Original: Thanks to the permutation invariance property of GNNs, graph matching does not need to be solved here.
Revised: Thanks to the permutation invariance property of GNNs, graph matching does not need to be solved here.

Original: Building upon the graph filter theory (Gama et al., 2020; Arghal et al., 2021), we first state  GNNs can be constructed stably with the lemma below. □ Lemma 1.
Revised: Building upon the graph filter theory (Gama et al., 2020; Arghal et al., 2021), we first state that GNNs can be constructed stably: □ Lemma 1.

Original: Suppose that G is the set for graphs of the size the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).
Revised: Suppose that G is the set for graphs of the size the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).

Original: Given ∀ G 1 , G 2 ∈ G and A 

Revised: Given ∀ G 1 , G 2 ∈ G and A

Original: =
Revised: =

Original: U 1 Λ 1 U T 1 , A 2 =
Revised: U 1 Λ 1 U T 1 , A 2 =

Original: U 2 Λ 2 U T 2 , the eigen decomposition for adjacency matrices
Revised: U 2 Λ 2 U T 2 , the eigen decomposition for adjacency matrices

Original: A 1 and A 2 that Λ 1 = diag([ λ 1 , 1 , ..., λ 1 ,N G ]) , Λ 2 = diag([ λ 2 , 1 , ..., λ 2 ,N G ]) (eigen values are sorted in the descending order).
Revised: A 1 and A 2 that Λ 1 = diag([ λ 1 , 1 , ..., λ 1 ,N G ]) , Λ 2 = diag([ λ 2 , 1 , ..., λ 2 ,N G ]) (eigen values are sorted in the descending order).

Original: A GNN is constructed by composing a graph filter and nonlinear mapping that f ( G 1 ) =
Revised: A GNN is constructed by composing a graph filter and nonlinear mapping that f ( G 1 ) =

Original: r ( σ ( S ( A 1 ) X 1 W ))
Revised: r ( σ ( S ( A 1 ) X 1 W ))

Original: =
Revised: =

Original: r ( σ ( U 1 S (Λ 1 ) U T 1 X 1 W )) where S is the polynomial function that S ( A 1 ) =
Revised: r ( σ ( U 1 S (Λ 1 ) U T 1 X 1 W )) where S is the polynomial function that S ( A 1 ) =

Original: (cid:80) ∞ k =

Revised: (cid:80) ∞ k =

Original: s k
Revised: s k

Original: A k 1 , W ∈
Revised: A k 1 , W ∈

Original: R D × D
Revised: R D × D

Original: ′ is the learnable weight matrix, r is the mean/sum/max readout function to pool node representations, and the pointwise nonlinearity holds as | σ ( b ) − σ ( a ) | ≤ | b − a | , ∀ a, b ∈ R .
Revised: ′ is the learnable weight matrix, r is the mean/sum/max readout function to pool node representations, and the pointwise nonlinearity holds as | σ ( b ) − σ ( a ) | ≤ | b − a | , ∀ a, b ∈ R .

Original: Assuming ∥
Revised: Assuming ∥

Original: X ∥ op ≤ 1 and ∥ W ∥ op ≤ 1 ( ∥·∥ op stands for operator norm), the following inequality holds:
Revised: X ∥ op ≤ 1 and ∥ W ∥ op ≤ 1 ( ∥·∥ op stands for operator norm), the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where τ = ( ∥ U 1 − U 2 ∥ F + 1) 2 − 1 stands for the eigenvector misalignment which can be bounded,
Revised: where τ = ( ∥ U 1 − U 2 ∥ F + 1) 2 − 1 stands for the eigenvector misalignment which can be bounded,

Original: P ∗ = argmin P ∈ Π (cid:8) ∥
Revised: P ∗ = argmin P ∈ Π (cid:8) ∥

Original: X 1 − PX 

Revised: X 1 − PX

Original: ∥
Revised: ∥

Original: F + ∥
Revised: F + ∥

Original: A 1 − PA 2 P T ∥ F (cid:9) , Π is the set of permutation matrices,
Revised: A 1 − PA 2 P T ∥ F (cid:9) , Π is the set of permutation matrices,

Original: O ( ∥ A 1 − P ∗ A 2 P ∗ T ∥
Revised: O ( ∥ A 1 − P ∗ A 2 P ∗ T ∥

Original: 2 F ) is the remainder term with bounded multipliers defined in (Gama et al., 2020), and C λ is the spectral Lipschitz constant that ∀
Revised: 2 F ) is the remainder term with bounded multipliers defined in (Gama et al., 2020), and C λ is the spectral Lipschitz constant that ∀

Original: λ
Revised: λ

Original: i , λ j , |S
Revised: i , λ j , |S

Original: ( λ i )
Revised: ( λ i )

Original: − S ( λ j ) | ≤ C λ | λ i − λ j | .
Revised: − S ( λ j ) | ≤ C λ | λ i − λ j | .

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: The lemma integrates the stability properties w.r.t.
Revised: The lemma integrates the stability properties w.r.t.

Original: node feature and edge perturbations in (Gama et al., 2020) and (Arghal et al., 2021), respectively, where C λ tied to edge perturbations characterizes spectral smoothness (SS) of the underlining graph filter, and max( |S (Λ 2 ) | ) tangled with node feature perturbations defines maximum frequency response (MFR) .
Revised: node feature and edge perturbations in (Gama et al., 2020) and (Arghal et al., 2021), respectively, where C λ tied to edge perturbations characterizes spectral smoothness (SS) of the underlining graph filter, and max( |S (Λ 2 ) | ) tangled with node feature perturbations defines maximum frequency response (MFR) .

Original: We note that in practice for Lemma 1, no padding is actually needed if the summation pooling is adopted as in our experiments (Xu et al., 2018), since the padded nodes do not contribute to the graph embedding (Zhu et al., 2021).
Revised: We note that in practice for Lemma 1, no padding is actually needed if the summation pooling is adopted as in our experiments (Xu et al., 2018), since the padded nodes do not contribute to the graph embedding (Zhu et al., 2021).

Original: Analysis on more sophisticated architectures is left in the future work.
Revised: Analysis on more sophisticated architectures is left to future works.

Original: We next instantiate the GNN Lipschitz constant for the commonly-used matching distance (Gama et al., 2020; Arghal et al., 2021) in the following lemma, achieved by eliminating the distance term in the numerator and denominator of the GNN Lipschitz constant.
Revised: We next instantiate the GNN Lipschitz constant for the commonly-used matching distance (Gama et al., 2020; Arghal et al., 2021) in the following lemma, achieved by eliminating the distance term in the numerator and denominator of the GNN Lipschitz constant.

Original: □ Lemma 2.
Revised: □ Lemma 2.

Original: Suppose that G is the set for graphs of the sizethe size N G after padding with isolated nodes, similar to (Zhu et al., 2021).
Revised: Suppose that G is the set for graphs of the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).

Original: Define the matching distance between G 1 , G 2 ∈ G as η ( G 1 , G 2 ) =
Revised: We define the matching distance between G 1 , G 2 ∈ G as η ( G 1 , G 2 ) =

Original: min P [Equation]{+∈ Π (cid:8) ∥ X 1 − PX 
+}
Revised: min P {+∈ Π (cid:8) ∥ X 1 − PX 
+}

Original: ∥
Revised: ∥

Original: [Equation] + [Equation] PA
Revised: F + ∥

Original: [Equation] P [Equation] ∥ F (cid:9) .
Revised: A 1 − PA 2 P T ∥ F (cid:9) .

Original: Suppose that the edge perturbation is bounded that ∀ G 1 , G 2 ∈ G , ∥ A 1 − P ∗ A 2 P ∗ T ∥ F ≤ ε with the optimal permutation P ∗ , and there exists an eigenvalue λ ∗ ∈ R to achieve the maximum |S ( λ ∗ ) | < ∞ .
Revised: Suppose that the edge perturbation is bounded that ∀ G 1 , G 2 ∈ G , ∥ A 1 − P ∗ A 2 P ∗ T ∥ F ≤ ε with the optimal permutation P ∗ , and there exists an eigenvalue λ ∗ ∈ R to achieve the maximum |S ( λ ∗ ) | < ∞ .

Original: We can then calculate the
Revised: We can then calculate the

Original: Lipschitz constant of GNN as:
Revised: Lipschitz constant of GNN as:

Original: [Equation]
Revised: [Equation]

Original:  where K 1 , K 2 is the supremes of (1 + τ √ N G ) and the remainder multiplier in Lemma 1, respectively, following similar philosophies in ((Gama et al., 2020), Theorem 1).
Revised: (6) where K 1 , K 2 is the supremes of (1 + τ √ N G ) and the remainder multiplier in Lemma 1, respectively, following similar philosophies in ((Gama et al., 2020), Theorem 1).

Original: Proof.
Revised: Proof.

Original: See Appendix B.
Revised: See Appendix B.

Original: [Figure]
Revised: [Figure]

Original: As recap, C λ and |S ( λ ∗ ) | characterize SS and MFR, respectively.
Revised: To recap, C λ and |S ( λ ∗ ) | characterize SS and MFR, respectively.

Original: Thereafter, the direct incorporation of Corollary 1 and Lemma 2 results in the first model-based DA bound for graph data .
Revised: Thereafter, the direct incorporation of Corollary 1 and Lemma 2 results in the first model-based DA bound for graph data .

Original: It builds the foundation to understand and further develop GDA algorithms.
Revised: It builds the foundation to understand and further develop GDA algorithms.

Original: Although in our theory GNN is assumed to be composed of a graph filter and a nonlinear activation layer, in practice such  architecture is ubiquitous, well-known for  simplicity and effectiveness (Wu et al., 2019; Li et al., 2019), and we also provide theory for two-layer GNNs in Appendix I which is easy to be extent to multi-layer architectures.
Revised: Although in our theory GNN is assumed to be composed of a graph filter and a nonlinear activation layer, in practice such an architecture is ubiquitous, well-known for its simplicity and effectiveness (Wu et al., 2019; Li et al., 2019). We also provide theory for two-layer GNNs in Appendix I which is naturally extendable to multi-layer architectures via induction.

Original: 4. T HEORY -G ROUNDED S PECTRAL R EGULARIZATION
Revised: 4. T HEORY -G ROUNDED S PECTRAL R EGULARIZATION

Original: Recalling in the DA inequality (4) (also see Section 3), the gap between the target and source risks is bounded with ( i ) the distribution divergence between source and target ( W 1 term) multiplied by Lipschitz constants, and ( ii ) the discriminative capability of NNs to capture invariant knowledge (the ω term) restricted by Lipschitz constants.
Revised: Recall that, in the DA inequality (4) (see also Section 3), the gap between the target and source risks is bounded with ( i ) the distribution divergence between source and target ( W 1 term) multiplied by Lipschitz constants, and ( ii ) the discriminative capability of NNs to capture invariant knowledge (the ω term) restricted by Lipschitz constants.

Original: This indicates  varying the Lipschitz constant results in the intrinsic trade-off between domain-divergence and discriminability to vary the DA bound.
Revised: This indicates that varying the Lipschitz constant results in the intrinsic trade-off between domain-divergence and discriminability to vary the DA bound.

Original: Therefore,one way to tighten the bound is, in search of a sweet point in the trade-off,  to regularize the Lipschitz constant of NNs.
Revised: Therefore, in search of a sweet spot in the trade-off, one way to tighten the bound is to regularize the Lipschitz constant of NNs.

Original: Equipped with the derived graph model-based DA bound in Section 4.1, we are now ready to propose regularizing the GNN spectral properties of SS or MFR for balancing between domain-divergence and discriminability.
Revised: Equipped with the derived graph model-based DA bound in Section 4.1, we are now ready to propose regularizing the GNN spectral properties of SS or MFR for balancing between domain-divergence and discriminability.

Original: We implement regularization as follows.
Revised: We implement regularization as follows.

Original: Implementations.
Revised: Implementations.

Original: Driven by the analysis, we incorporate spectral regularization (SpecReg ∈ { SSReg, MFRReg } , SSReg for C λ and MFRReg for |S ( λ ∗ ) | in equation (6)) into the traditional domain-invariant representation learning framework (3) as follow:
Revised: Driven by the analysis, we incorporate spectral regularization (SpecReg ∈ { SSReg, MFRReg } , SSReg for C λ and MFRReg for |S ( λ ∗ ) | in equation (6)) into the traditional domain-invariant representation learning framework (3) as follows:

Original: [Equation]
Revised: [Equation]

Original: where γ ′ is the trade-off factor, tuned from { 1, 1e-1, 1e-2, 1e-3 } through validation.
Revised: where γ ′ is the trade-off factor, tuned over { 1, 1e-1, 1e-2, 1e-3 } through validation.

Original: Denote Z = f ( G ) and the spectral signals ˜
Revised: Denote Z = f ( G ) and the spectral signals ˜

Original: Z = U T Z, ˜ X
Revised: Z = U T Z, ˜ X

Original: = U T X , SS and MFR regularization (SSReg, MFRReg) are specifically implemented by regularizing on the spectral outputs w.r.t.
Revised: = U T X , SS and MFR regularization (SSReg, MFRReg) are specifically implemented by regularizing on the spectral outputs w.r.t.

Original: spectral inputs as:
Revised: spectral inputs as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where X [ m, n ] denotes matrix indexing, sum is the matrix summation, ReLU and abs are pointwise rectifier and absolute functions, resectively, and υ is the threshold value, tuned from { 0.1, 1, 10, 100, 1000 } through validation.
Revised: where X [ m, n ] denotes matrix indexing, sum is the matrix summation, ReLU and abs are pointwise rectifier and absolute functions, respectively, and υ is the threshold value tuned over { 0.1, 1, 10, 100, 1000 } through validation.

Original: An overview of the implementation is depicted in Figure 1.
Revised: An overview of the implementation is depicted in Figure 1.

Original: Analysis on complexity of spectral regularization (with eigenvalue decomposition) is presented in Appendix H.
Revised: Analysis on complexity of spectral regularization (with eigendecomposition) is presented in Appendix H.

Original: How could theory further drive practice?
Revised: How could theory further drive practice?

Original: We also notice that in special scenarios, either of the two regularizations (SS and MFR) can be favored than the other.
Revised: We also notice that in special scenarios, either of the two regularizations (SS and MFR) can be favored than the other.

Original: ( i ) Node transfer : When the node features preserve (mostly) the invariant label-related information, while edges are less labelrelevant (noisy) and distribute highly distinctly across source and target domains, we would have large K 1 , ε in Lemma 2 that C λ K 1 + εK 2 > |S ( λ ∗ ) | .
Revised: ( i ) Node transfer : When the node features preserve (mostly) the invariant label-related information, while edges are less labelrelevant (noisy) and distribute highly distinctly across source and target domains, we would have large K 1 , ε in Lemma 2 that C λ K 1 + εK 2 > |S ( λ ∗ ) | .

Original: Thus regularizing SS leads to the decreased domain-divergence term ( C f decreases) and the less increased discriminability term w.r.t.
Revised: Thus regularizing SS leads to the decreased domain-divergence term ( C f decreases) and the less increased discriminability term w.r.t.

Original: node features (GNN is sensitive enough to label-preserving node features as shown in Lemma 1).
Revised: node features (GNN is sensitive enough to label-preserving node features as shown in Lemma 1).

Original: ( ii ) Link transfer : Oppositely, when edges are invariantly label-preserving across source and target domains while node features are relatively noisy, regularizing MFR is more desired.
Revised: ( ii ) Link transfer : Oppositely, when edges are invariantly label-preserving across source and target domains whereas node features are relatively noisy, regularizing MFR is more desired.

Original: This provides us with  guidance to select the regularization in practical applications with domain knowledge.
Revised: This provides us with the guidance to select the regularization in practical applications with domain knowledge.

Original: 4. E XTENSION TO THE S EMI -S UPERVISED S ETTING
Revised: 4. E XTENSION TO THE S EMI -S UPERVISED S ETTING

Original: Results so far do not assume the access to target labels, which is in the unsupervised setting.
Revised: Results so far do not assume the access to target labels, which is in the unsupervised setting.

Original: In this section, we show that our analysis also applies in the semi-supervised setting, where the more challenging conditional shift is considered— P S ( Y | G )
Revised: In this section, we show that our analysis also applies in the semi-supervised setting, where the more challenging conditional shift is considered— P S ( Y | G )

Original: ̸ = P T ( Y | G ) —in addition to the access to a small amount of target labelled data (Li et al., 2021a; Zhao et al., 2019).
Revised: ̸ = P T ( Y | G ) —in addition to the access to a small amount of target labelled data (Li et al., 2021a; Zhao et al., 2019).

Original: We provide a finite-sample semi-supervised OT-based GDA bound in the following lemma.
Revised: We provide a finite-sample semi-supervised OT-based GDA bound in the following lemma.

Original: □ Lemma 3.
Revised: □ Lemma 3.

Original: Under the assumption of Corollary 1, we further assume that there exists a small amount of i.i.d.
Revised: Under the assumption of Corollary 1, we further assume that there exists a small amount of i.i.d.

Original: samples with labels { ( G n , Y n ) } N ′ T n =1 from the target distribution P T ( G, Y ) ( N ′ T ≪
Revised: samples with labels { ( G n , Y n ) } N ′ T n =1 from the target distribution P T ( G, Y ) ( N ′ T ≪

Original: N S )
Revised: N S )

Original: and bring in the conditional shift assumption that domains have different labeling function ˆ h S ̸
Revised: and bring in the conditional shift assumption that domains have different labeling function ˆ h S ̸

Original: = ˆ h T and max G 1 ,G 2 | ˆ h D ( G 1 ) − ˆ h D ( G 2 )
Revised: = ˆ h T and max G 1 ,G 2 | ˆ h D ( G 1 ) − ˆ h D ( G 2 )

Original: | η ( G 1 ,G 2 )
Revised: | η ( G 1 ,G 2 )

Original: =
Revised: =

Original: C h ≤ C f C g
Revised: C h ≤ C f C g

Original: (D ∈ { S , T } )
Revised: (D ∈ { S , T } )

Original: for some constant C h and distance measure η .
Revised: for some constant C h and distance measure η .

Original: Let H :
Revised: Let H :

Original: = { h : G → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H )
Revised: = { h : G → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H )

Original: =
Revised: =

Original: d , with probability at least 1 − δ the following inequality holds:
Revised: d , with probability at least 1 − δ the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where ω = min [Equation] | ϵ [Equation]
Revised: where ω = min (cid:0) | ϵ S ( h, ˆ h S ) − ϵ S ( h, ˆ h T ) | , | ϵ T ( h, ˆ h S ) − ϵ T ( h, ˆ h T ) | (cid:1) .

Original: h

Original: [Equation]

Original: ˆ h

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix C.
Revised: See Appendix C.

Original: Since the main form of inequality (10) is consistent with inequality (4), similar discussion can be made following the thoughts in Section 4.2, to demonstrate that we can properly regularize the spectral properties of GNNs (SS and MFR) to tighten the target risk and seek a sweet point between the domain-divergence and discriminability.
Revised: Since the main form of inequality (10) is consistent with inequality (4), similar discussion can be made following the thoughts in Section 4.2, to demonstrate that we can properly regularize the spectral properties of GNNs (SS and MFR) to tighten the target risk and seek a sweet spot between the domain-divergence and discriminability.

Original: Following optimization (7), the semi-supervised training procedure is implemented as:
Revised: Following optimization (7), the semi-supervised training procedure is implemented as:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: e n c e ( % ) b e t w ee n a d
Revised: e n c e ( % ) b e t w ee n a d

Original: j m a t
Revised: j m a t

Original: r
Revised: r

Original: i c e s o f S
Revised: i c e s o f S

Original: I a n d C
Revised: I a n d C

Original: P P
Revised: PP

Original: I
Revised: I

Original: [Figure] We evaluate our proposed algorithms, SSReg and MFRReg, in two real-world applications of graph transfer learning : ( i ) link prediction of protein-protein interactions (PPIs) across different species (Szklarczyk et al., 2021), and ( ii ) node classification of paper topics across different time periods (Wu et al., 2020;
Revised: Co-expression Physical We evaluate our proposed algorithms, SSReg and MFRReg, in two real-world applications of graph transfer learning : ( i ) link prediction of protein-protein interactions (PPIs) across different species (Szklarczyk et al., 2021), and ( ii ) node classification of paper topics across different time periods (Wu et al., 2020;

Original: Hu et al., 2020).
Revised: Hu et al., 2020).

Original: 5. P REDICTING P ROTEIN -P ROTEIN I NTERACTIONS IN V ARIOUS S PECIES
Revised: 5. P REDICTING P ROTEIN -P ROTEIN I NTERACTIONS IN V ARIOUS S PECIES

Original: Datasets.
Revised: Datasets.

Original: PPI networks have proven important to understand functional genomics and analyze biological pathways (Sharan et al., 2007;
Revised: PPI networks have proven important to understand functional genomics and analyze biological pathways (Sharan et al., 2007;

Original: Navlakha & Kingsford, 2010).
Revised: Navlakha & Kingsford, 2010).

Original: But in most species the coverage of experimental PPI data remains low (Sledzieski et al., 2021).
Revised: But in most species the coverage of experimental PPI data remains low (Sledzieski et al., 2021).

Original: We utilize protein sequences together with freely accessible computational
Revised: We utilize protein sequences together with freely accessible computational

Original: PPIs via whole-genome comparisons (Szklarczyk et al., 2021) to predict experimental PPIs, i.e. the graph is built with nodes represented as protein sequences and edges as computational PPIs.
Revised: PPIs via whole-genome comparisons (Szklarczyk et al., 2021) to predict experimental PPIs, i.e. the graph is built with nodes represented as protein sequences and edges as computational PPIs.

Original: We collect PPIs of species from the STRING database (Szklarczyk et al., 2021) where[Table] [Table]
Revised: We collect PPIs of species from the STRING database (Szklarczyk et al., 2021) where

Original: PPIs in the neighborhood , fusion and co-occurrence channels are defined computational
Revised: PPIs in the neighborhood , fusion and co-occurrence channels are defined computational, and in the

Original: [Table]
Revised: [Table]

Original: and in the co-expression and experiments (we refer to it as physical to prevent confusion) channels (to be predicted) are experimental which deals with expensive functional genomics experiments (Parkinson et al., 2009) or direct lab assays (Br¨uckner et al., 2009).
Revised: [Table] co-expression and experiments (we refer to it as physical to prevent confusion) channels (to be predicted) are experimental which deals with expensive functional genomics experiments (Parkinson et al., 2009) or direct lab assays (Br¨uckner et al., 2009).

Original: More details of PPI data are shown in
Revised: More details of PPI data are shown in

Original: Appendix D, and ablations in Appendix G.
Revised: Appendix D, and ablations in Appendix G.

Original: Configurations.
Revised: Configurations.

Original: We train models on PPIs of Homo Sapiens (human) which contain abundant
Revised: We train models on PPIs of Homo Sapiens (human) which contain abundant

Original: PPI evidences (Ewing et al., 2007) and evaluate the models in four other species: Mus Musculus (mouse), Danio Rerio (zebrafish), Drosophila Melanogaster (fruit fly) and Saccharomyces Cerevisiae (yeast), which are repeated for three times for statistical significance.
Revised: PPI evidences (Ewing et al., 2007) and evaluate the models in four other species: Mus Musculus (mouse), Danio Rerio (zebrafish), Drosophila Melanogaster (fruit fly) and Saccharomyces Cerevisiae (yeast), which are repeated for three times for statistical significance.

Original: In evaluation, we follow (Cho et al., 2016) to perform negative sampling for experimental PPIs, to ensure that the known interactions compose only 5%.
Revised: In evaluation, we follow (Cho et al., 2016) to perform negative sampling for experimental PPIs, to ensure that the known interactions compose only 5%.

Original: We adopt Transformer (Tay et al., 2020) (v.s. HRNN (Karimi et al., 2019; 2020a)) to embed protein sequences and then GIN (Xu et al., 2018)
Revised: We adopt Transformer (Tay et al., 2020) (v.s. HRNN (Karimi et al., 2019; 2020a)) to embed protein sequences and then GIN (Xu et al., 2018)

Original: (v.s. GAT (Veliˇckovi´c et al., 2017)) to perform message passing, with comparisons in Appendix E. The compared state-of-the-art (SOTA) approaches are Mashup (Cho et al., 2016) and D-SCRIPT (Sledzieski et al., 2021) designed for PPI prediction as well as GraphCL (You et al., 2020a) for general graph (self-supervised) representation learning.
Revised: (v.s. GAT (Veliˇckovi´c et al., 2017)) to perform message passing, with comparisons in Appendix E. The compared state-of-the-art (SOTA) approaches are Mashup (Cho et al., 2016) and D-SCRIPT (Sledzieski et al., 2021) designed for PPI prediction as well as GraphCL (You et al., 2020a) for general graph (self-supervised) representation learning.

Original: See Appendix E for details.
Revised: See Appendix E for details.

Original: Hypotheses.
Revised: Hypotheses.

Original: We hypothesize that co-expression interactions are more associated with links (than physical interactions do, link transfer ), based upon the existing findings that two genes are poten- tially similar in expression profiles if they are with similar promoter regions (might indicate locating in the neighborhood)
Revised: We hypothesize that co-expression interactions are more associated with links (than physical interactions do, link transfer ), based upon the existing findings that two genes are poten- tially similar in expression profiles if they are with similar promoter regions (might indicate locating in the neighborhood)

Original: (Park et al., 2002), involved in fusion events (Fernebro et al., 2006), or sharing co-occurrence patterns (Larmuseau et al., 2019).
Revised: (Park et al., 2002), involved in fusion events (Fernebro et al., 2006), or sharing co-occurrence patterns (Larmuseau et al., 2019).

Original: For physical interactions, we hypothesize that they rely more on node features (sequences, node transfer ) considering the recent breakthrough that protein structure information (and ultimately physical interactions and functions) can be recovered from sequences with high accuracy (Jumper et al., 2021).
Revised: For physical interactions, we hypothesize that they rely more on node features (sequences, node transfer ) considering the recent breakthrough that protein structure information (and ultimately physical interactions and functions) can be recovered from sequences with high accuracy (Jumper et al., 2021).

Original: We calculate the homophily ratios (Zhu et al., 2020a; Pei et al., 2020) of 3 out of 4 species in Figure 2 as numerical evidence to support the hypotheses (see Appendix F for the computing procedure, where zebrafish is excluded due to the overly sparse ( < 200) physical interactions as shown in Appendix D)
Revised: We calculate the homophily ratios (Zhu et al., 2020a; Pei et al., 2020) of 3 out of 4 species in Figure 2 as numerical evidence to support the hypotheses (see Appendix F for the computing procedure [Figure] where zebrafish is excluded due to the overly sparse ( < 200) physical interactions as shown in Appendix D)

Original: [Figure]
Revised: [Figure]

Original: Results.
Revised: Results.

Original: The results of unsupervised transfer are shown in Table 1 and 2.
Revised: The results of unsupervised transfer are shown in Table 1 and 2.

Original: We put semi-supervised results in Appendix
Revised: We put semi-supervised results in Appendix

Original: G with the similar findings.
Revised: G with the similar findings.

Original: Through comparing between: ( i ) methods w/o and w/ DA techniques, ( ii ) methods w/o and w/ spectral regularization, and ( iii ) our proposed methods and SOTA competitors, we have the following findings.
Revised: Through comparing between: ( i ) methods w/o and w/ DA techniques, ( ii ) methods w/o and w/ spectral regularization, and ( iii ) our proposed methods and SOTA competitors, we have the following findings.

Original: ( i ) Vanilla DA in general provides benefits though occasionally degrades performance.
Revised: ( i ) Vanilla DA in general provides benefits though occasionally degrades performance.

Original: With the assistance of domain-invariant representation learning, GNNs generally lead to a better performance with exceptions, i.e., with
Revised: With the assistance of domain-invariant representation learning, GNNs generally lead to a better performance with exceptions, i.e., with

Original: Wasserstein distance guided DA (DAW, as in optimization (3) (Redko et al., 2017;
Revised: Wasserstein distance guided DA (DAW, as in optimization (3) (Redko et al., 2017;

Original: Shen et al., 2018)), 11 out of 16 metrics turn higher than w/o DA in the unsupervised setting, and so doout of 16 in semi-supervised.
Revised: Shen et al., 2018)), 11 out of 16 metrics turn higher than w/o DA in the unsupervised setting, and so doout of 16 in semi-supervised.

Original: The occasional performance degrade fits our analysis in Section 4.2, that the optimized source risk and distribution divergence of representations could lead to either improved or deteriorated performances without guarantee.
Revised: The occasional performance degrade fits our analysis in Section 4.2, that the optimized source risk and distribution divergence of representations could lead to either improved or deteriorated performances without guarantee.

Original: Similar phenomenon happens to DA with domain classifiers (DA-C, (Ganin et al., 2016)).
Revised: Similar phenomenon happens to DA with domain classifiers (DA-C, (Ganin et al., 2016)).

Original: Comparison between DAW and DA-C shows that, DA-W performs better than DA-C when transferring to graphs with the larger domain gap (i.e. from human to fruit fly and yeast, justified by their biological taxonomic ranks and phylogenetic distances (Alberts et al., 2003), see Appendix D for details).
Revised: Comparison between DAW and DA-C shows that, DA-W performs better than DA-C when transferring to graphs with the larger domain gap (i.e. from human to fruit fly and yeast, justified by their biological taxonomic ranks and phylogenetic distances (Alberts et al., 2002), see Appendix D for details).

Original: ( ii )
Revised: ( ii )

Original: In a principled way, spectral regularization further boosts GNN performance with DA-W and consistently alleviates performance degradation.
Revised: In a principled way, spectral regularization further boosts GNN performance with DA-W and consistently alleviates performance degradation.

Original: Under the regularization on GNN spectral properties, domain-invariant representation learning provides further benefits in a principled way.
Revised: Under the regularization on GNN spectral properties, domain-invariant representation learning provides further benefits in a principled way.

Original: Specifically, MFRReg improves 13 out of 16 co-expression interaction prediction versus DA-W, since it assists GNN to mine from the computational PPIs that co-expression interactions are more correlated with, as hypothesized in Section 5.1.
Revised: Specifically, MFRReg improves 13 out of 16 co-expression interaction prediction versus DA-W, since it assists GNN to mine from the computational PPIs that co-expression interactions are more correlated with, as hypothesized in Section 5.1.

Original: We show in Figure 3 that it is a consequence of a better sweet spot when regularizing MFR, which echos our analysis in Section 4.1, that MFRReg benefits more in the link transfer setting.
Revised: We show in Figure 3 that it is a consequence of a better sweet spot when regularizing MFR, which echos our analysis in Section 4.1, that MFRReg benefits more in the link transfer setting.

Original: Moreover, SSReg improves 13 out of 16 physical interaction prediction which reinforces GNN to dig more from protein sequence embeddings that physical interactions are more relevant to.
Revised: Moreover, SSReg improves 13 out of 16 physical interaction prediction which reinforces GNN to dig more from protein sequence embeddings that physical interactions are more relevant to.

Original: This echos our analysis that SSReg benefits more in the node transfer setting, conforming to our theory-grounded regularization design.
Revised: This echos our analysis that SSReg benefits more in the node transfer setting, conforming to our theory-grounded regularization design.

Original: Besides, we observe that the improvements are more significant in the unsupervised setting than semi-supervised.
Revised: Besides, we observe that the improvements are more significant in the unsupervised setting than semi-supervised.

Original: This might result from, under the guidance of certain target labelled data during transferring, DA-W are less prone to capture superfluous information even without regularization.
Revised: A possible reason is that, under the guidance of certain target labelled data during transferring, DA-W are less prone to capture superfluous information even without regularization.

Original: ( iii ) Integrating protein sequences together with computational PPIs leads to better transfer performance.
Revised: ( iii ) Integrating protein sequences together with computational PPIs leads to better transfer performance.

Original: Comparing with SOTA competitors, we find that utilizing computational PPIs alone (such as Mashup) or utilizing protein sequences alone (such as D-SCRIPT which heavily relies on the pre-trained sequence encoder on a tremendous and diverse population of protein sequences or even structures (Bepler & Berger, 2019)) leads to less competitive results than integrating them together when transferring across species.
Revised: Comparing with SOTA competitors, we find that utilizing computational PPIs alone (such as Mashup) or utilizing protein sequences alone (such as D-SCRIPT which heavily relies on the pre-trained sequence encoder on a tremendous and diverse population of protein sequences or even structures (Bepler & Berger, 2019)) leads to less competitive results than integrating them together when transferring across species.

Original: For self-supervised pre-training methods (such as GraphCL), the existence of domain gap prompts “negative transfer” in the unsupervised setting, which is alleviated under the guidance of certain target labels in the semi-supervised setting.
Revised: For self-supervised pre-training methods (such as GraphCL), the existence of domain gap prompts “negative transfer” in the unsupervised setting, which is alleviated under the guidance of certain target labels in the semi-supervised setting.

Original: 5. C LASSIFYING P APER T OPICS OF D IFFERENT T IME P ERIODS
Revised: 5. C LASSIFYING P APER T OPICS OF D IFFERENT T IME P ERIODS

Original: [Table]
Revised: [Table]

Original: Datasets and Configurations.
Revised: Datasets and Configurations.

Original: We also conduct experiments on the benchmark datasets provided in ArnetMiner (Tang et al., 2008), to classify paper topics in the temporally shifting citation networks.
Revised: We also conduct experiments on the benchmark datasets provided in ArnetMiner (Tang et al., 2008), to classify paper topics in the temporally shifting citation networks.

Original: Papers published between 2000 and 2010 are collected from ACM and those after 2010 are from DBLP, with statistics shown in Appendix D.
Revised: Papers published between 2000 and 2010 are collected from ACM and those after 2010 are from DBLP, with statistics shown in Appendix D.

Original: The Citation database is not examined here without processed data public.
Revised: The Citation database is not examined here without processed data public.

Original: The networks are built with nodes for papers in six categories (to be predicted) and links representing citations.
Revised: The networks are built with nodes for papers in six categories (to be predicted) and links representing citations.

Original: Following (Wu et al., 2020), we train methods on ACM/DBLP and evaluate them on DBLP/ACM, which are repeated for ten times.
Revised: Following (Wu et al., 2020), we train methods on ACM/DBLP and evaluate them on DBLP/ACM, which are repeated for ten times.

Original: The compared SOTAs include graph representation learning w/o and w/ DA techniques.
Revised: The compared SOTAs include graph representation learning w/o and w/ DA techniques.

Original: Our implementation is built upon UDAGCN (Wu et al., 2020).
Revised: Our implementation is built upon UDAGCN (Wu et al., 2020).

Original: See Appendix E for more details.
Revised: See Appendix E for more details.

Original: We also assay our methods on the large-scale OGB benchmark (Hu et al., 2020).
Revised: We also assay our methods on the large-scale OGB benchmark (Hu et al., 2020).

Original: Hypothesis.
Revised: Hypothesis.

Original: Paper topics are verified to have strong homophily with citations mostly (Zhu et al., 2020a; Pei et al., 2020) ( link transfer ), which is adopted as the hypothesis in our study.
Revised: Paper topics are verified to have strong homophily with citations mostly (Zhu et al., 2020a; Pei et al., 2020) ( link transfer ), which is adopted as the hypothesis in our study.

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: The results of unsupervised transfer are shown in Table 3, demonstrating the applicability of the proposed spectral regularization in different applications.
Revised: The results of unsupervised transfer are shown in Table 3, demonstrating the applicability of the proposed spectral regularization in different applications.

Original: Comparing graph representation learning w/o and w/ DA techniques, domain-invariant representations generally improve performances.
Revised: Comparing graph representation learning w/o and w/ DA techniques, domain-invariant representations generally improve performances.

Original: Further built upon the SOTA UDAGCN, applying DA-W to minimize the domain divergence on representations benefits the transfer performance from ACM to DBLP while hurting it from DBLP to ACM, which is consistent with the observation ( i ).
Revised: Further built upon the SOTA UDAGCN, applying DA-W to minimize the domain divergence on representations benefits the transfer performance from ACM to DBLP while hurting it from DBLP to ACM, which is consistent with the observation ( i ).

Original: Similar results (also referring to Appendix G, Tablefor the semi-supervised setting) on the large-scale ogbn-arxiv dataset are shown in Table 3.
Revised: Similar results (also referring to Appendix G, Tablefor the semi-supervised setting) on the large-scale ogbn-arxiv dataset are shown in Table 3.

Original: Via spectral regularization, specifically MFRReg to exploit the invariant topology information across domains that is more related to node labels as hypothesized, our methods achieve better performance than all competitors, which is consistent with the observation ( ii ) in Section 5.1.
Revised: Via spectral regularization, specifically MFRReg to exploit the invariant topology information across domains that is more related to node labels as hypothesized, our methods achieve better performance than all competitors, which is consistent with the observation ( ii ) in Section 5.1.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: To fulfill the practical demands of transfer learning on graph data, we develop theory-grounded spectral regularization for GNNs to learn transferable graph representations.
Revised: To fulfill the practical demands of transfer learning on graph data, we develop theory-grounded spectral regularization for GNNs to learn transferable graph representations.

Original: We first leverage domain adaptation with optimal transport theory to dive into the guaranteed bound for the transfer performance.
Revised: We first leverage domain adaptation with optimal transport theory to dive into the guaranteed bound for the transfer performance.

Original: This bound reveals that varying the Lipschitz constant of the GNNs could lead to a tighter bound by balancing domain divergence and GNN power.
Revised: This bound reveals that varying the Lipschitz constant of the GNNs could lead to a tighter bound by balancing domain divergence and GNN power.

Original: Building on the graph filter theory, we next show that one can regularize GNN spectral properties(of spectral smoothness (SS) and maximum frequency response (MFR)) to regularize the Lipschitz constant, which motivates us to propose spectral regularization.
Revised: Building on the graph filter theory, we next show that one can regularize GNN spectral properties to regularize the Lipschitz constant, which motivates us to propose spectral regularizations.

Original: Numerical results conform to our theoretical analysis that regularizing SS and MFR brings benefits to the scenarios of node transfer and link transfer, respectively, in both the unsupervised and supervised settings. 
Revised: Numerical results conform to our theoretical analysis that regularizing SS and MFR brings benefits to the scenarios of node transfer and link transfer, respectively, in both the unsupervised and supervised settings. A CKNOWLEDGEMENT

Revised: This project was in part supported by the National Institute of General Medical Sciences (R35GM124952 to Y.S.), the National Science Foundation (CCF-1943008 to Y.S.), and the US

Revised: Army Research Office Young Investigator Award (W911NF2010240 to Z.W.).

Revised: Portions of this research were conducted with the advanced computing resources provided by Texas A&M High

Revised: Performance Research Computing.

Original: A  P ROOF FOR L EMMA
Revised: A PPENDIX A P ROOF FOR L EMMA

Original: □ Lemma 1.
Revised: □ Lemma 1.

Original: Suppose that G is the set for graphs of the sizethe size N G after padding with isolated nodes, similar to (Zhu et al., 2021).
Revised: Suppose that G is the set for graphs of the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).

Original: Given ∀ G 1 , G 2 ∈ G and A 

Revised: Given ∀ G 1 , G 2 ∈ G and A

Original: =
Revised: =

Original: U 1 Λ 1 U T 1 , A 2 =
Revised: U 1 Λ 1 U T 1 , A 2 =

Original: U 2 Λ 2 U T 2 , the eigen decomposition for adjacency matrices
Revised: U 2 Λ 2 U T 2 , the eigen decomposition for adjacency matrices

Original: A 1 and A 2 that Λ 1 = diag([ λ 1 , 1 , ..., λ 1 ,N G ]) , Λ 2 = diag([ λ 2 , 1 , ..., λ 2 ,N G ]) (eigen values are sorted in the descending order).
Revised: A 1 and A 2 that Λ 1 = diag([ λ 1 , 1 , ..., λ 1 ,N G ]) , Λ 2 = diag([ λ 2 , 1 , ..., λ 2 ,N G ]) (eigen values are sorted in the descending order).

Original: A GNN is constructed by composing a graph filter and nonlinear mapping that f ( G 1 ) =
Revised: A GNN is constructed by composing a graph filter and nonlinear mapping that f ( G 1 ) =

Original: r ( σ ( S ( A 1 ) X 1 W ))
Revised: r ( σ ( S ( A 1 ) X 1 W ))

Original: =
Revised: =

Original: r ( σ ( U 1 S (Λ 1 ) U T 1 X 1 W )) where S is the polynomial function that S ( A 1 ) =
Revised: r ( σ ( U 1 S (Λ 1 ) U T 1 X 1 W )) where S is the polynomial function that S ( A 1 ) =

Original: (cid:80) ∞ k =

Revised: (cid:80) ∞ k =

Original: s k
Revised: s k

Original: A k 1 , W ∈
Revised: A k 1 , W ∈

Original: R D × D
Revised: R D × D

Original: ′ is the learnable weight matrix, r is the mean/sum/max readout function to pool node representations, and the pointwise nonlinearity holds as | σ ( b ) − σ ( a ) | ≤ | b − a | , ∀ a, b ∈ R .
Revised: ′ is the learnable weight matrix, r is the mean/sum/max readout function to pool node representations, and the pointwise nonlinearity holds as | σ ( b ) − σ ( a ) | ≤ | b − a | , ∀ a, b ∈ R .

Original: Assuming ∥
Revised: Assuming ∥

Original: X ∥ op ≤ 1 and ∥ W ∥ op ≤ 1 , the following inequality holds:
Revised: X ∥ op ≤ 1 and ∥ W ∥ op ≤ 1 , the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where τ = ( ∥ U 1 − U 2 ∥ F + 1) 2 − 1 stands for the eigenvector misalignment which can be bounded,
Revised: where τ = ( ∥ U 1 − U 2 ∥ F + 1) 2 − 1 stands for the eigenvector misalignment which can be bounded,

Original: P[-∗ = argmin P ∈ Π (cid:8) ∥ X 1 − PX 
-]
Revised: P[-∗ = argmin P ∈ Π (cid:8) ∥ X 1 − PX 
-]

Original: ∥ F + ∥
Revised: [Equation]

Original: A 1 − PA 2 P T ∥ F (cid:9) , Π is the set of permutation matrices,
Revised: ∥ F (cid:9) , Π is the set of permutation matrices,

Original: O ( ∥ A 1 − P ∗ A 2 P ∗ T ∥
Revised: O ( ∥ A 1 − P ∗ A 2 P ∗ T ∥

Original: 2 F ) is the remainder term with bounded multipliers defined in (Gama et al., 2020), and C λ is the spectral Lipschitz constant that ∀
Revised: 2 F ) is the remainder term with bounded multipliers defined in (Gama et al., 2020), and C λ is the spectral Lipschitz constant that ∀

Original: λ
Revised: λ

Original: i , λ j , |S
Revised: i , λ j , |S

Original: ( λ i )
Revised: ( λ i )

Original: − S ( λ j ) | ≤ C λ | λ i − λ j | .
Revised: − S ( λ j ) | ≤ C λ | λ i − λ j | .

Original: Proof.
Revised: Proof.

Original: Denote the optimal permutation matrix for G 1 , G 2 as P ∗ , we compute the difference of the
Revised: Denote the optimal permutation matrix for G 1 , G 2 as P ∗ , we compute the difference of the

Original: GNN outputs:
Revised: GNN outputs:

Original: [Equation]
Revised: [Equation]

Original: where (a) is due to the permutation invariance property of graph filters; (b) is achieved with the triangle inequality and the assumption | σ ( b ) − σ ( a ) | ≤ | b − a | , ∀ a, b ∈ R ; (c) and (d) uses the fact that for any two matrices A, B , ∥ AB ∥ F ≤ min( ∥ A ∥ op ∥
Revised: where (a) is due to the permutation invariance property of graph filters; (b) is achieved with the triangle inequality and the assumption | σ ( b ) − σ ( a ) | ≤ | b − a | , ∀ a, b ∈ R ; (c) and (d) use the fact that for any two matrices A, B , ∥ AB ∥ F ≤ min( ∥ A ∥ op ∥

Original: B ∥ F , ∥
Revised: B ∥ F , ∥

Original: A ∥ F ∥ B ∥ op ) , and (c) further applies the triangle inequality; (e) adopts the assumption ∥
Revised: A ∥ F ∥ B ∥ op ) , and (c) further applies the triangle inequality; (e) adopts the assumption ∥

Original: X ∥ op ≤ 1 , ∥
Revised: X ∥ op ≤ 1 , ∥

Original: W ∥ op ≤ 1 which in practice can be guaranteed with normalization, and is easy extent into the case with ∥ X ∥ op ≤ K, ∥ W ∥ op ≤
Revised: W ∥ op ≤ 1 which in practice can be guaranteed with normalization, and easily extended to the case with ∥ X ∥ op ≤ K, ∥ W ∥ op ≤

Original: K, ∀ K > 0 , and because S ( P ∗ A 2 P ∗ T ) =
Revised: K, ∀ K > 0 , and because S ( P ∗ A 2 P ∗ T ) =

Original: ( P ∗ U 2 ) S (Λ 2 )( P ∗ U 2 ) T can be diagonalized, its operator norms equal the spectral radius; (f) is the direct outcome borrowed from (Gama et al., 2020)
Revised: ( P ∗ U 2 ) S (Λ 2 )( P ∗ U 2 ) T can be diagonalized, its operator norms equal the spectral radius; (f) is the direct outcome borrowed from (Gama et al., 2020)

Original: Theorem 1.
Revised: Theorem 1.

Original: We complete the proof.
Revised: We complete the proof.

Original: B P ROOF FOR L EMMA
Revised: B P ROOF FOR L EMMA

Original: □ Lemma 2.
Revised: □ Lemma 2.

Original: Suppose that G is the set for graphs of the sizethe size N G after padding with isolated nodes, similar to (Zhu et al., 2021).
Revised: Suppose that G is the set for graphs of the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).

Original: Define the matching distance between G 1 , G 2 ∈ G as η 
Revised: Define the matching distance between G 1 , G 2 ∈ G as η ( G 1 , G 2 )

Original: [Equation]
Revised: = min P ∈ Π (cid:8) ∥

Original: G 1 , G{+− PX 
+}
Revised: X 1 {+− PX 
+}

Original: [Equation]
Revised: ∥ F + ∥

Original: ) = [Equation] (cid:9) .
Revised: A 1 − PA 2 P T ∥ F (cid:9) .

Original: Suppose that the edge perturbation is bounded that ∀ G 1 , G 2 ∈ G , ∥ A 1 − P ∗ A 2 P ∗ T ∥ F ≤ ε with the optimal permutation P ∗ , and there exists an eigenvalue λ ∗ ∈ R to achieve the maximum |S ( λ ∗ ) | < ∞ .
Revised: Suppose that the edge perturbation is bounded that ∀ G 1 , G 2 ∈ G , ∥ A 1 − P ∗ A 2 P ∗ T ∥ F ≤ ε with the optimal permutation P ∗ , and there exists an eigenvalue λ ∗ ∈ R to achieve the maximum |S ( λ ∗ ) | < ∞ .

Original: We can then calculate the Lipschitz constant of GNN as:
Revised: We can then calculate the Lipschitz constant of GNN as:

Original: [Equation]
Revised: [Equation]

Original: where K 1 , K 2 is the supremes of (1 + τ √ N G ) and the remainder multiplier in Lemma 1, respectively, following similar philosophies in ((Gama et al., 2020), Theorem 1).
Revised: where K 1 , K 2 is the supremes of (1 + τ √ N G ) and the remainder multiplier in Lemma 1, respectively, following similar philosophies in ((Gama et al., 2020), Theorem 1).

Original: Proof.
Revised: Proof.

Original: To calculate the Lipschitz constant C f w.r.t the matching distance, based upon Lemma 1, we assure the following inequality:
Revised: To calculate the Lipschitz constant C f w.r.t the matching distance, based upon Lemma 1, we assure the following inequality:

Original: [Equation]
Revised: [Equation]

Original: the latter inequality of which can be rewritten as:
Revised: the latter inequality of which can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: which is necessary for:
Revised: which is necessary for:

Original: [Equation]

Original: O ( ∥

Original: [Equation]

Original: 1 − P ∗ A 2 P ∗ T ∥

Original: 2 F )

Original: − C f

Original: ∥ A 1 − P ∗

Original: A 2 P ∗ T ∥ F ≤ 0 ,

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to:
Revised: which is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: The bounding of K 1 , K 2 follows (Gama et al., 2020), Theorem 1 and the first minimum solution can be calculated from the quadratic function w.r.t.
Revised: The bounding of K 1 , K 2 follows (Gama et al., 2020) Theorem 1 and the first minimum solution can be calculated from the quadratic function w.r.t.

Original: the edge matching distance ∥
Revised: the edge matching distance ∥

Original: A 1 − P ∗ A 2 P ∗ T ∥ F .
Revised: A 1 − P ∗ A 2 P ∗ T ∥ F .

Original: Let C f takes the larger value between them, we complete the proof.
Revised: Let C f takes the larger value between them, we complete the proof.

Original: C P ROOF FOR L EMMA
Revised: C P ROOF FOR L EMMA

Original: □ Lemma 3.
Revised: □ Lemma 3.

Original: Under the assumption of Corollary 1, we further assume that there exists a small amount of i.i.d.
Revised: Under the assumption of Corollary 1, we further assume that there exists a small amount of i.i.d.

Original: samples with labels { ( G n , Y n ) } N ′ T n =1 from the target distribution P T ( G, Y ) ( N ′ T ≪
Revised: samples with labels { ( G n , Y n ) } N ′ T n =1 from the target distribution P T ( G, Y ) ( N ′ T ≪

Original: N S )
Revised: N S )

Original: and bring in the conditional shift assumption that domains have different labeling function ˆ h S ̸
Revised: and bring in the conditional shift assumption that domains have different labeling function ˆ h S ̸

Original: = ˆ h T and max G 1 ,G 2 | ˆ h D ( G 1 ) − ˆ h D ( G 2 )
Revised: = ˆ h T and max G 1 ,G 2 | ˆ h D ( G 1 ) − ˆ h D ( G 2 )

Original: | η ( G 1 ,G 2 )
Revised: | η ( G 1 ,G 2 )

Original: =
Revised: =

Original: C h ≤ C f C g
Revised: C h ≤ C f C g

Original: (D ∈ { S , T } )
Revised: (D ∈ { S , T } )

Original: for some constant C h and distance measure η .
Revised: for some constant C h and distance measure η .

Original: Let H :
Revised: Let H :

Original: = { h : G → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H )
Revised: = { h : G → Y} be the set of bounded real-valued functions with the pseudo-dimension Pdim ( H )

Original: =
Revised: =

Original: d , with probability at least 1 − δ the following inequality holds:
Revised: d , with probability at least 1 − δ the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where ω = min (cid:0) | ϵ S ( h, ˆ h S ) − ϵ S ( h, ˆ h T ) | , | ϵ T ( h, ˆ h S ) − ϵ T ( h, ˆ h T ) | (cid:1) .
Revised: where ω = min (cid:0) | ϵ S ( h, ˆ h S ) − ϵ S ( h, ˆ h T ) | , | ϵ T ( h, ˆ h S ) − ϵ T ( h, ˆ h T ) | (cid:1) .

Original: Proof.
Revised: Proof.

Original: Before showing the designated lemma [Equation] we first introduce the following inequality to be used that:
Revised: Before showing the designated lemma, we first introduce the following inequality to be used that:

Original: [Equation]
Revised: [Equation]

Original: where (a) results from (Shen et al., 2018)
Revised: where (a) results from (Shen et al., 2018)

Original: Lemma with the assumption 
Revised: Lemma with the assumption max( ∥ h ∥ Lip , max G 1 ,G 2 | ˆ h D

Original: [Equation]
Revised: [Equation]

Original:  { S , T } .
Revised: G 1 ) − ˆ h D ( G [Equation] [Equation] ) ≤ C f C g , D ∈ { S , T } .

Original: Similarly, we obtain:
Revised: Similarly, we obtain:

Original: [Equation]
Revised: [Equation]

Original: We therefore combine them into:
Revised: We therefore combine them into:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. the following holds to bound the target risk ϵ T ( h, ˆ h T ) :
Revised: e. the following holds to bound the target risk ϵ T ( h, ˆ h T ) :

Original: [Equation]
Revised: [Equation]

Original: We next link the bound with the empirical risk and labeled sample size by showing, with probability at least 1 − δ that: 
Revised: We next link the bound with the empirical risk and labeled sample size by showing, with probability at least 1 − δ that: [Equation] N S log

Original: [Equation]
Revised: [Equation]

Original:  and:
Revised: , and:

Original: [Equation]
Revised: [Equation]

Original: which results from (Mohri et al., 2018)
Revised: which results from (Mohri et al., 2018)

Original: Theorem [Equation]
Revised: Theorem 11.8.

Original: Lastly
Revised: Lastly

Original: [Equation]
Revised: [Equation]

Original: we combine the above two inequalities
Revised: we combine the above two inequalities, with probability at least 1 − δ that:

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ that:
Revised: N S + N ′ T

Original: [Equation]
Revised: [Equation]

Original: where (a) is the outcome of applying the union bound with coefficient
Revised: G

Original: [Equation]
Revised: [Equation]

Original: N
Revised: min

Original: [Equation]
Revised: [Equation]

Original: T
Revised: |

Original: [Equation]
Revised: [Equation]

Original:  N 
Revised: ( c ) ≤ N ′ [Equation] where (a) is the outcome of applying the union bound with coefficient

Original: [Equation]
Revised: [Equation]

Original: T respectively; (b) and (c) result from the Cauchy-Schwartz inequality and (c) additionally adopt the assumption
Revised: ; (b) and (c) result from the Cauchy-Schwartz inequality and (c) additionally adopt the assumption

Original: N ′ T ≪ N S , following 
Revised: N ′ T ≪ N S , following the sleight-of-hand in (Li et al., 2021a) Theorem 3.2.

Original: D D ATASET S TATISTICS
Revised: D D ATASET S TATISTICS

Original: For PPI data, we construct the graph with nodes as protein sequences (using protein language modeling (Tay et al., 2020; Karimi et al., 2019) for node features), and edges as computational interactions of neighborhood , fusion and co-occurrence channels (Szklarczyk et al., 2021)(i [Table]
Revised: For PPI data, we construct the graph with nodes as protein sequences (using protein language modeling (Tay et al., 2020; Karimi et al., 2019) for node features), and edges as computational interactions of neighborhood , fusion and co-occurrence channels (Szklarczyk et al., 2021)

Original: e. the edge feature dimension is 3).
Revised: (i.e. the edge feature dimension is 3).

Original: Dataset statistics for PPI and citation networks are shown in Figure 5 and 6, respectively.
Revised: Dataset statistics for PPI and citation networks are shown in Figure 5 and 6, respectively.

Original: For the species involved in PPI networks, we depict their relationship in biological taxonomic ranks in
Revised: For the species involved in PPI networks, we depict their relationship in biological taxonomic ranks in

Original: Figure 4, showing the domain gap between species in concept.
Revised: Figure 4, showing the domain gap between species in concept.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: Sources ( # Node # Edge # Feature # Class DBLP (2000–2010) 5, 7, 7, ACM (2010– ) 7, 11, 7,

Original: E D ETAILED C ONFIGURATIONS
Revised: E D ETAILED C ONFIGURATIONS

Original: The assayed data are released under the MIT license, and to our best knowledge, contain no privacyinfringing contents.
Revised: The assayed data are released under the MIT license, and to our best knowledge, contain no privacyinfringing contents.

Original: Experiments are distributed on computer clusters with Tesla K80 GPU (11 GB memory) and
Revised: Experiments are distributed on computer clusters with Tesla K80 GPU (11 GB memory) and

Original: NVIDIA A100 GPU (40 GB memory).
Revised: NVIDIA A100 GPU (40 GB memory).

Original: E. W ASSERSTEIN -1 D ISTANCE E STIMATION
Revised: E. W ASSERSTEIN -1 D ISTANCE E STIMATION

Original: We follow the most routine procedure (Shen et al., 2018) to estimate Wasserstein-1 distance of graph representations between distributions in adversarial training (3).
Revised: We follow the most routine procedure (Shen et al., 2018) to estimate Wasserstein-1 distance of graph representations between distributions in adversarial training (3).

Original: Specifically, given the encoder f and source and target data distributions P S ( G ) , P T ( G ) , we estimate the distance as:
Revised: Specifically, given the encoder f and source and target data distributions P S ( G ) , P T ( G ) , we estimate the distance as:

Original: [Equation]
Revised: [Equation]

Original: where the critic function ˆ f is instantiated with a multilayer perceptron and satisfies ∥ ˆ f ∥ = sup∥ ˆ
Revised: where the critic function ˆ f is instantiated with a multilayer perceptron and satisfies ∥ ˆ f ∥ = sup

Original: [Equation] − [Equation]
Revised: ∥ ˆ f ( x ) − ˆ f ( y ) ∥

Original: [Equation] ≤ 1 , which is achieved by enforcing the gradient penalty ( ∥∇ x ˆ f ( x ) ∥ 2 − 1) 2 .
Revised: 2 ∥ x − y ∥≤ 1 , which is achieved by enforcing the gradient penalty ( ∥∇ x ˆ f ( x ) ∥ 2 − 1) 2 .

Original: The final estimation is thus implemented as:
Revised: The final estimation is thus implemented as:

Original: [Equation]
Revised: [Equation]

Original: where we follow (Shen et al., 2018; Gulrajani et al., 2017) to set ˆ γ = 10 by default.
Revised: where we follow (Shen et al., 2018; Gulrajani et al., 2017) to set ˆ γ = 10 by default.

Original: E. P REDICTING P ROTEIN -P ROTEIN I NTERACTIONS ACROSS V ARIOUS S PECIES
Revised: E. P REDICTING P ROTEIN -P ROTEIN I NTERACTIONS ACROSS V ARIOUS S PECIES

Original: During collecting PPI data from the STRING database (Szklarczyk et al., 2021), we abandon the interactions in channels of neighborhood transferred , coexpression transferred , and experiments transferred to prevent information leakage across species (especially from supervisions of co-expression and physical interaction prediction).
Revised: During collecting PPI data from the STRING database (Szklarczyk et al., 2021), we abandon the interactions in channels of neighborhood transferred , coexpression transferred , and experiments transferred to prevent information leakage across species (especially from supervisions of co-expression and physical interaction prediction).

Original: For co-expression and physical interactions, we use the high-quality threshold of 0.7 (Szklarczyk et al., 2021) to convert them into binary labels.
Revised: For co-expression and physical interactions, we use the high-quality threshold of 0.7 (Szklarczyk et al., 2021) to convert them into binary labels.

Original: We use Sinkhorn Transformer (Tay et al., 2020)
Revised: We use Sinkhorn Transformer (Tay et al., 2020)

Original: (a variant of Transformer with sparse attention) with depth 4, attention head 4 and bucket size 32 to embed protein sequences, and further adopt GIN (Xu et al., 2018) with depth 3 and MLP depth 2 to perform message passing across proteins.
Revised: (a variant of Transformer with sparse attention) with depth 4, attention head 4 and bucket size 32 to embed protein sequences, and further adopt GIN (Xu et al., 2018) with depth 3 and MLP depth 2 to perform message passing across proteins.

Original: We also try
Revised: We also try

Original: HRNN (Karimi et al., 2019; 2020a) with k-mer 75 to embed protein sequence and GAT (Veli ˇ ckovi ´
Revised: HRNN (Karimi et al., 2019; 2020a) with k-mer 75 to embed protein sequence and GAT (Veli ˇ ckovi ´

Original: c et al., 2017) with depth 3 and attention head 8 to perform message passing, with comparison on human to yeast transfer shown in Table 7, which states in our case that, (Sinkhorn) Transformer and GIN outperform HRNN and GAT, respectively.
Revised: c et al., 2017) with depth 3 and attention head 8 to perform message passing, with comparison on human to yeast transfer shown in Table 7, which states in our case that, (Sinkhorn) Transformer and GIN outperform HRNN and GAT, respectively.

Original: In training, we hold out 20% of human PPIs for validation.
Revised: In training, we hold out 20% of human PPIs for validation.

Original: For the semi-supervised setting, 0.1% of experimental PPIs are in addition available during training.
Revised: For the semi-supervised setting, 0.1% of experimental PPIs are in addition available during training.

Original: We train with convergence assured for 500 epochs with learning rate 0.0001, hidden dimension 256 and batch size 128 which is sampled by random walk, optimized by Adam optimizer.
Revised: We train with convergence assured for 500 epochs with learning rate 0.0001, hidden dimension 256 and batch size 128 which is sampled by random walk, optimized by Adam optimizer.

Original: For domain-invariant representation learning guided by Wasserstein distance (as described in optimization (3)) please refer to (Shen et al., 2018) for implementation, and see (Ganin et al., 2016) for one with domain classifiers.
Revised: For domain-invariant representation learning guided by Wasserstein distance (as described in optimization (3)) please refer to (Shen et al., 2018) for implementation, and see (Ganin et al., 2016) for one with domain classifiers.

Original: We selected the trade-off factor γ in optimization (3) from { 1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6 } through validation.
Revised: We selected the trade-off factor γ in optimization (3) from { 1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6 } through validation.

Original: In evaluation, we follow (Cho et al., 2016) to perform negative sampling for experimental PPIs, to ensure that the known interactions compose only 5%, with the assumption the number of unobserved interactions is only a small fraction.
Revised: In evaluation, we follow (Cho et al., 2016) to perform negative sampling for experimental PPIs, to ensure that the known interactions compose only 5%, with the assumption the number of unobserved interactions is only a small fraction.

Original: [Table]
Revised: [Table]

Original: 66.65 ± 2.

Original: 11.75 ± 1.

Original: Transformer+GAT 60.85 ± 1.

Original: 9.00 ± 1.

Original: 70.51 ± 1.

Original: 14.53 ± 2.

Original: Transformer+GIN 61.18 ± 2.

Original: 10.90 ± 1. 71.15 ± 1. 15.02 ± 1. On baseline implementation, for Mashup (Cho et al., 2016), we apply the official MATLAB code to extract protein representation w.r.t.
Revised: On baseline implementation, for Mashup (Cho et al., 2016), we apply the official MATLAB code to extract protein representation w.r.t.

Original: computational PPIs in the neighborhood, fusion and cooccurrence channels, on top of which we apply a two-layer MLP to train with labels; for D-SCRIPT (Sledzieski et al., 2021), we apply the official PyTorch APIs to perform the sequence embedding projection, inter-protein residue contact and interaction modeling, where we use the same Sinkhorn
Revised: computational PPIs in the neighborhood, fusion and cooccurrence channels, on top of which we apply a two-layer MLP to train with labels; for D-SCRIPT (Sledzieski et al., 2021), we apply the official PyTorch APIs to perform the sequence embedding projection, inter-protein residue contact and interaction modeling, where we use the same Sinkhorn

Original: Transformer as in our methods to generate sequence embeddings rather than the large model pretrained on additional data of protein sequences with structures, for fair comparison and also complexity consideration; for GraphCL (You et al., 2020a), we follow the main idea and implement it with the node masking augmentation, where we randomly replace 20% of animo acids in each protein sequences with mask tokens.
Revised: Transformer as in our methods to generate sequence embeddings rather than the large model pretrained on additional data of protein sequences with structures, for fair comparison and also complexity consideration; for GraphCL (You et al., 2020a), we follow the main idea and implement it with the node masking augmentation, where we randomly replace 20% of animo acids in each protein sequences with mask tokens.

Original: E. C LASSIFYING P APER T OPICS OF D IFFERENT T IME P ERIODS
Revised: E. C LASSIFYING P APER T OPICS OF D IFFERENT T IME P ERIODS

Original: We follow the same experiment setting as in UDAGCN (Wu et al., 2020), which we implement
Revised: We follow the same experiment setting as in UDAGCN (Wu et al., 2020), which we implement

Original: DA-W and spectral regularization on.
Revised: DA-W and spectral regularization on.

Original: The compared SOTAs without DA techniques are DeepWalk (Perozzi et al., 2014), LINE (Tang et al., 2015), GraphSAGE (Hamilton et al., 2017), DNN (MLP on node features alone), and GCN (Kipf & Welling, 2016), and ones with DA techniques are DGRL (Ganin et al., 2016), AdaGCN (Dai et al., 2019), and UDAGCN (Wu et al., 2020).
Revised: The compared SOTAs without DA techniques are DeepWalk (Perozzi et al., 2014), LINE (Tang et al., 2015), GraphSAGE (Hamilton et al., 2017), DNN (MLP on node features alone), and GCN (Kipf & Welling, 2016), and ones with DA techniques are DGRL (Ganin et al., 2016), AdaGCN (Dai et al., 2019), and UDAGCN (Wu et al., 2020).

Original: We notice that the reported results in (Wu et al., 2020) are not directly comparable with (Mao et al., 2021) due to different experimental configurations (e.g. validation partition).
Revised: We notice that the reported results in (Wu et al., 2020) are not directly comparable with (Mao et al., 2021) due to different experimental configurations (e.g. validation partition).

Original: By applying DA-W, we replace the domain classifier with the Wasserstein distance critic (Shen et al., 2018) to learn domain-invariant representations, and further implement our proposed spectral regularization.
Revised: By applying DA-W, we replace the domain classifier with the Wasserstein distance critic (Shen et al., 2018) to learn domain-invariant representations, and further implement our proposed spectral regularization.

Original: F H OMOPHILY R ATIO C OMPUTATION P ROCEDURE
Revised: F H OMOPHILY R ATIO C OMPUTATION P ROCEDURE

Original: Given a graph G = { V, E } with the set of nodes V and edges E as described in Section 2.
Revised: Given a graph G = { V, E } with the set of nodes V and edges E as described in Section 2.

Original: Denote the labeled edge set as E ′ , to quantify the association between E and E ′ , we borrow the idea from (Zhu et al., 2020a; Pei et al., 2020) to calculate the homophily ratio as: 
Revised: Denote the labeled edge set as E ′ , to quantify the association between E and E ′ , we borrow the idea from (Zhu et al., 2020a; Pei et al., 2020) to calculate the homophily ratio as: [Equation] v

Original: [Equation]
Revised: [Equation]

Revised: |{ ( v, w ) : w ∈ N ( v ) and ( v, w ) ∈

Original:  }| |N ( v ) | , where N ( v ) is the set of neighbor nodes of v determined by E .
Revised: E ′ }| |N ( v ) | , where N ( v ) is the set of neighbor nodes of v determined by E .

Original: In cross-species PPI prediction, E ′ is the experimental (co-expression or physical) PPIs.
Revised: In cross-species PPI prediction, E ′ is the experimental (co-expression or physical) PPIs.

Original: To measure the association between computational and experimental PPIs, we define E as the set of edges if there is any computational PPI (neighborhood, fusion or co-occurrence) value greater than the medium threshold of 0.4 (Szklarczyk et al., 2021).
Revised: To measure the association between computational and experimental PPIs, we define E as the set of edges if there is any computational PPI (neighborhood, fusion or co-occurrence) value greater than the medium threshold of 0.4 (Szklarczyk et al., 2021).

Original: To measure the association between protein sequences and experimental PPIs, we first calculate the sequence identity (Karimi et al., 2020b) between all protein pairs, and define E as the set of edges if the sequence identity is greater than 0.3 (Pearson, 2013).
Revised: To measure the association between protein sequences and experimental PPIs, we first calculate the sequence identity (Karimi et al., 2020b) between all protein pairs, and define E as the set of edges if the sequence identity is greater than 0.3 (Pearson, 2013) [Table]

Original: G M ORE E XPERIMENTAL R ESULTS
Revised: G M ORE E XPERIMENTAL R ESULTS

Original: Results of semi-supervised cross-species protein-protein interaction prediction are shown in Tables 8, 9; semi-supervised paper topic classification in Table 10; ablation studies of model architectures and adversarial training strategies in Tables 11, 12. 
Revised: Results of semi-supervised cross-species protein-protein interaction prediction are shown in Tables 8, 9; semi-supervised paper topic classification in Table 10; ablation studies of model architectures and adversarial training strategies in Tables 11, 12. [Table]

Revised: 8 (17.54 ± 4.70) (22.93 ± 2.42) (11.26 ± 0.98) ( 20.75 ± 3.84) (18.12)

Revised: Transformer 71.57 ± 1.

Original: H C OMPLEXITY A NALYSIS OF S PECTRAL R EGULARIZATION
Revised: 69.87 ± 0. 61.76 ± 2. 67.01 ± 0. 6.2 (20.39 ± 0.90) (26.09 ± 1.58) (12.14 ± 1.60) (14.11 ± 1.24) (18.18) Transformer +GIN 77.86 ± 3.

Revised: 80.39 ± 3.

Revised: 66.92 ± 1.

Revised: 67.07 ± 0.

Revised: 3.5 ( 22.17 ± 7.86) ( 30.15 ± 5.26) (8.04 ± 0.53) (14.43 ± 1.92) (18.69) Transformer +GIN+DA-C 75.13 ± 3.

Revised: 79.01 ± 0.

Revised: 66.02 ± 1.

Revised: 68.12 ± 0.

Revised: 4.8 (17.71 ± 1.39) (28.54 ± 1.88) (8.34 ± 0.41) (17.65 ± 1.69) (18.06) Transformer +GIN+DA-W 77.06 ± 0.

Revised: 79.48 ± 1.

Revised: 71.11 ± 2.

Revised: 67.60 ± 2.

Revised: 3.0 (20.63 ± 1.81) (28.96 ± 2.75) ( 16.04 ± 0.94) (15.14 ± 2.09) ( 20.19 )

Revised: Transformer+GIN+DA-W+SSReg 76.65 ± 1.

Revised: 80.19 ± 0.

Revised: 69.36 ± 2.

Revised: 67.58 ± 2.

Revised: 3.2 (20.66 ± 4.17) (27.23 ± 0.75) ( 13.56 ± 2.54) (18.46 ± 2.22) (19.97) Transformer

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: H C OMPLEXITY A NALYSIS OF S PECTRAL R EGULARIZATION

Revised: Time.

Original: Let’s consider a graph of N nodes.
Revised: Let’s consider a graph of N nodes.

Original: The spectral regularization is composed of two steps: extracting spectral signals (with eigenvalue decomposition (EVD) involved, complexity O ( N 3 ) ) and neural network propagation (complexity O ( N ) (You et al., 2020c)).
Revised: The spectral regularization is composed of two steps: extracting spectral signals (with eigenvalue decomposition (EVD) involved, complexity O ( N 3 ) ) and neural network propagation (complexity O ( N ) (You et al., 2020c)).

Original: Thus, in theory, the former step dominates the time complexity.
Revised: Thus, in theory, the former step dominates the time complexity.

Original: In practice, when training on large networks, mini-batch training is commonly adopted (Hamilton et al., 2017; Zeng et al., 2019) to avoid computational burden and empirically for better generalizability (Hamilton et al., 2017), where the node number in each batch is restricted to be much smaller than the whole network.
Revised: In practice, when training on large networks, mini-batch training is commonly adopted (Hamilton et al., 2017; Zeng et al., 2019) to avoid computational burden and empirically for better generalizability (Hamilton et al., 2017), where the node number in each batch is restricted to be much smaller than the whole network.

Original: In our experiments, batch size is set as 128, with ≤ 3 seconds cost brought for each epoch (see Table 13), which is five times less than the cost brought by adversarial training (20 seconds, see Table 13).
Revised: In our experiments, batch size is set as 128, with ≤ 3 seconds cost brought for each epoch (see Table 13), which is five times less than the cost brought by adversarial training (20 seconds, see Table 13).

Original: We also provide the time consumption for the full training in Table 15, including the best-performed SOTA GraphCL.
Revised: We also provide the time consumption for the full training in Table 15, including the best-performed SOTA GraphCL, where the time consumption for GraphCL was around 2 times higher in pretraining and 2 times lower in finetuning compared with training from scratch. Nevertheless, training from scratch did not take longer than 10 hours.

Original: We further provide Table 14 to demonstrate the numerical running time of EVD is acceptable as we see in Table 14, with batch size ≤ 2048.[Table] [Table]
Revised: We further provide Table 14 to demonstrate the numerical running time of EVD is acceptable as we see in Table 14, with batch size ≤ 2048.

Original: [Table] Nevertheless, if it is believed that there are indispensable benefits from the full-batch training, we can always perform the full-batch EVD for only once before training, for the repetitive usage later.[Table]
Revised: Nevertheless, if it is believed that there are indispensable benefits from the full-batch training, we can always perform the full-batch EVD for only once before training, for the repetitive usage later.

Original: [Table] Moreover, we would like to clarify that the spectral properties of graph filters in theory do not have to depend on inputs (sampled graphs or not, an analog is in signal processing, a low-pass system, here “low-pass” is the spectral property, would suppress high frequencies for any input signal).
Revised: Moreover, we would like to clarify that the spectral properties of graph filters in theory do not have to depend on inputs (sampled graphs or not, an analog is in signal processing, a low-pass system, here “low-pass” is the spectral property, would suppress high frequencies for any input signal).

Original: In details, suppose that the graph filter function S ( · ) with a certain domain D 1 (which is a polynomial function resided in a GNN, see Lemma 1) is constructed (or regularized) to be C λ -Lipschitz and bounded by B (i.e. ∀ x ∈ D, | S ( x ) | ≤ B ), then for an arbitrary graph with the adjacency matrix
Revised: In details, suppose that the graph filter function S ( · ) with a certain domain D 1 (which is a polynomial function resided in a GNN, see Lemma 1) is constructed (or regularized) to be C λ -Lipschitz and bounded by B (i.e. ∀ x ∈ D, | S ( x ) | ≤ B ), then for an arbitrary graph with the adjacency matrix

Original: A = U Λ U T with all eigenvalues range in D 2 (i.e. ∀ λ ∈ λ, λ ∈ D 2 ), the following applies.
Revised: A = U Λ U T with all eigenvalues range in D 2 (i.e. ∀ λ ∈ λ, λ ∈ D 2 ), the following applies.

Original: If
Revised: If

Original: D 2 ⊆
Revised: D 2 ⊆

Original: D 1 , the graph filter S ( A ) =
Revised: D 1 , the graph filter S ( A ) =

Original: US (Λ) U T (Ortega et al., 2018) preserves the properties of (i) spectral smoothness (SS) with C λ that ∀
Revised: US (Λ) U T (Ortega et al., 2018) preserves the properties of (i) spectral smoothness (SS) with C λ that ∀

Original: λ
Revised: λ

Original: i , λ j , | S ( λ i ) − S ( λ j )
Revised: i , λ j , | S ( λ i ) − S ( λ j )

Original: | | λ i − λ j | ≤ C λ , and (ii) maximum frequency response (MFR) B that max | S ( λ ) | ≤ B .
Revised: | | λ i − λ j | ≤ C λ , and (ii) maximum frequency response (MFR) B that max | S ( λ ) | ≤ B .

Original: We can see from the above statement that, spectral properties of SS and MFR are formulated on eigenvalues but not eigenvectors.
Revised: We can see from the above statement that, spectral properties of SS and MFR are formulated on eigenvalues but not eigenvectors.

Original: In practice, the spectral regularization is performed in the spectra of sampled networks (with eigenvalues in Λ B range in D B ) and the readers might concern whether the regularized properties in
Revised: In practice, the spectral regularization is performed in the spectra of sampled networks (with eigenvalues in Λ B range in D B ) and the readers might concern whether the regularized properties in

Original: D B still hold in the spectra of ego-graphs of nodes (with eigenvalues in Λ E range in D E ) since whether D E ⊆ D B is unclear.
Revised: D B still hold in the spectra of ego-graphs of nodes (with eigenvalues in Λ E range in D E ) since whether D E ⊆ D B is unclear.

Original: According to the eigenvalue interlacing theorem (Haemers, 1995), since adjacency matrices of ego-graphs are principal submatrices of sampled networks, we have min(Λ B ) ≤ min(Λ E ) ≤ max(Λ E ) ≤ max(Λ B ) that essentially leads to D E ⊆ D B .
Revised: According to the eigenvalue interlacing theorem (Haemers, 1995), since adjacency matrices of ego-graphs are principal submatrices of sampled networks, we have min(Λ B ) ≤ min(Λ E ) ≤ max(Λ E ) ≤ max(Λ B ) that essentially leads to D E ⊆ D B .

Original: Therefore, regularized spectral properties on spectra D E (in practice) should be preserved on spectra D B (in theory).
Revised: Therefore, regularized spectral properties on spectra D E (in practice) should be preserved on spectra D B (in theory).

Original: [Table]
Revised: Memory. Beyond time complexity, memory consumption results from data processing (while all compared methods use the same amount of data) and model propagation. Compared to the best-performed SOTA GraphCL, memory consumption was similar between GraphCL and

Original: [Table]
Revised: Transformer+GIN(+DA-W+SpecReg), since GraphCL also uses the same Transformer+GIN backbone architecture to extract protein representations (Transformer to embed protein sequences as vertex features, and then GIN to conduct message passing along topology, please refer to Appendix E for details). GPU memory taken by Transformer+GIN was around 24GB which is within the capacity of the conventional NVIDIA A100 GPU (40 GB memory). The improving AUPRC of PPI prediction with regard to such affordable computational resources is significant.

Original: [Table]
Revised: The computational resource is usually not the bottleneck for in-silico methods (in our case, an NVIDIA A100 GPU + less than 10 hours), while detecting PPI (usually highly imbalanced Rao et al. via wet laboratories is very costly with expensive reagents for weeks or months. For instance, the routine yeast two-hybrid system requires steps including building vectors, transforming plasmids, cell culture, luciferase assay, etc Br¨uckner et al. (2009), where accurate predictions could play a critical role in accelerating the process.

Original: I L EMMA 4: L IPSCHITZ C ONSTANT OF T WO -L AYER GNN
Revised: I L EMMA 4: L IPSCHITZ C ONSTANT OF T WO -L AYER GNN

Original: □ Lemma 4.
Revised: □ Lemma 4.

Original: Suppose that G is the set for graphs of the size the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).
Revised: Suppose that G is the set for graphs of the size the size N G after padding with isolated nodes, similar to (Zhu et al., 2021).

Original: Following the setting in Lemma 1 and 2, a GNN layer is constructed as f ( · ) ( G ) =
Revised: Following the setting in Lemma 1 and 2, a GNN layer is [Table] [Table]

Original:  σ ( S ( · ) ( A ) XW ( · ) ) , and a two-layer GNN as f ( G ) =
Revised: [Table] constructed as f ( · ) ( G ) = σ ( S ( · ) ( A ) XW ( · ) ) , and a two-layer GNN as f ( G ) =

Original: r ◦ f (2) ◦ f (1) ( G ) , where r is the mean/sum/max readout function to pool node representations.
Revised: r ◦ f (2) ◦ f (1) ( G ) , where r is the mean/sum/max readout function to pool node representations.

Original: Denote the SS and
Revised: Denote the SS and

Original: MFR terms of f ( l ) as t ( l ) SS = C ( l ) λ K 1 + εK 2 , t ( l ) MFR
Revised: MFR terms of f ( l ) as t ( l ) SS = C ( l ) λ K 1 + εK 2 , t ( l ) MFR

Original: =
Revised: =

Original: |S ( l ) ( λ ∗ ) | that C ( l ) f = max { t ( l

Original: [Equation] SS , t ( l [Equation] MFR } , l ∈ { 1 , 2 } per Lemma 2, we can then calculate the Lipschitz constant of GNN as:
Revised: |S ( l ) ( λ ∗ ) | that C ( l ) f = max { t ( l ) SS , t ( l ) MFR } , l ∈ { 1 , 2 } per Lemma 2, we can then calculate the Lipschitz constant of GNN as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The key step is to recognize that the input of the 2nd GNN layer is actually a graph with the same adjacency matrix A , but a different node feature matrix X (1) =
Revised: The key step is to recognize that the input of the 2nd GNN layer is actually a graph with the same adjacency matrix A , but a different node feature matrix X (1) =

Original: f (1) ( G ) .
Revised: f (1) ( G ) .

Original: Denote the optimal permutation matrix for G 1 , G 2 as P ∗ , we thus compute the difference of the GNN outputs:
Revised: Denote the optimal permutation matrix for G 1 , G 2 as P ∗ , we thus compute the difference of the GNN outputs:

Original: [Equation]
Revised: [Equation]

Original: where (a) [Equation] (b) are due to the reuse the inequalities (a)-(f) in Lemma 1.
Revised: where (a), (b) are due to the reuse the inequalities (a)-(f) in Lemma 1.

Original: Next, following the same spirit in Lemma 2, to calculate the Lipschitz constant of f , we assure the inequality:
Revised: Next, following the same spirit in Lemma 2, to calculate the Lipschitz constant of f , we assure the inequality:

Original: [Equation]
Revised: [Equation]

Original: that is:
Revised: that is:

Original: [Equation]
Revised: [Equation]

Original: which is necessary for:
Revised: which is necessary for:

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to:
Revised: which is equivalent to:

Original: [Equation]
Revised: [Equation]