Original: Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift
Revised: Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift

Original: Abstract
Revised: Abstract

Original: Dynamic graph neural networks (DyGNNs) have demonstrated powerful predictive abilities by exploiting graph structural and temporal dynamics.
Revised: Dynamic graph neural networks (DyGNNs) have demonstrated powerful predictive abilities by exploiting graph structural and temporal dynamics.

Original: However, the existing DyGNNs fail to handle distribution shifts, which naturally exist in dynamic graphs, mainly because the patterns exploited by DyGNNs may be variant with respect to labels under distribution shifts.
Revised: However, the existing DyGNNs fail to handle distribution shifts, which naturally exist in dynamic graphs, mainly because the patterns exploited by DyGNNs may be variant with respect to labels under distribution shifts.

Original: In this paper, we propose to handle spatio-temporal distribution shifts in dynamic graphs by discovering and utilizing invariant patterns , i.e., structures and features whose predictive abilities are stable across distribution shifts, which faces two key challenges: 1) How to discover the complex variant and invariant spatio-temporal patterns in dynamic graphs, which involve both time-varying graph structures and node features.
Revised: In this paper, we propose to handle spatio-temporal distribution shifts in dynamic graphs by discovering and utilizing invariant patterns , i.e., structures and features whose predictive abilities are stable across distribution shifts, which faces two key challenges: 1) How to discover the complex variant and invariant spatio-temporal patterns in dynamic graphs, which involve both time-varying graph structures and node features.

Original: 2) How to handle spatio-temporal distribution shifts with the discovered variant and invariant patterns.
Revised: 2) How to handle spatio-temporal distribution shifts with the discovered variant and invariant patterns.

Original: To tackle these challenges, we propose the Disentangled Intervention-based
Revised: To tackle these challenges, we propose the Disentangled Intervention-based

Original: Dynamic graph Attention networks ( DIDA ).
Revised: Dynamic graph Attention networks ( DIDA ).

Original: Our proposed method can effectively handle spatio-temporal distribution shifts in dynamic graphs by discovering and fully utilizing invariant spatio-temporal patterns.
Revised: Our proposed method can effectively handle spatio-temporal distribution shifts in dynamic graphs by discovering and fully utilizing invariant spatio-temporal patterns.

Original: Specifically, we first propose a disentangled spatio-temporal attention network to capture the variant and invariant patterns.
Revised: Specifically, we first propose a disentangled spatio-temporal attention network to capture the variant and invariant patterns.

Original: Then, we design a spatio-temporal intervention mechanism to create multiple interventional distributions by sampling and reassembling variant patterns across neighborhoods and time stamps to eliminate the spurious impacts of variant patterns.
Revised: Then, we design a spatio-temporal intervention mechanism to create multiple interventional distributions by sampling and reassembling variant patterns across neighborhoods and time stamps to eliminate the spurious impacts of variant patterns.

Original: Lastly, we propose an invariance regularization term to minimize the variance of predictions in intervened distributions so that our model can make predictions based on invariant patterns with stable predictive abilities and therefore handle distribution shifts.
Revised: Lastly, we propose an invariance regularization term to minimize the variance of predictions in intervened distributions so that our model can make predictions based on invariant patterns with stable predictive abilities and therefore handle distribution shifts.

Original: Experiments on three real-world datasets and one synthetic dataset demonstrate the superiority of our method over state-of-the-art baselines under distribution shifts.
Revised: Experiments on three real-world datasets and one synthetic dataset demonstrate the superiority of our method over state-of-the-art baselines under distribution shifts.

Original: Our work is the first study of spatio-temporal distribution shifts in dynamic graphs, to the best of our knowledge
Revised: Our work is the first study of spatio-temporal distribution shifts in dynamic graphs, to the best of our knowledge

Original:  Introduction
Revised:  Introduction

Original: Dynamic graphs widely exist in real-world applications, including financial networks [1, 2], social networks [3, 4], traffic networks [5, 6], etc.
Revised: Dynamic graphs widely exist in real-world applications, including financial networks [1, 2], social networks [3, 4], traffic networks [5, 6], etc.

Original: Distinct from static graphs, dynamic graphs can represent temporal structure and feature patterns, which are more complex yet common in reality.
Revised: Distinct from static graphs, dynamic graphs can represent temporal structure and feature patterns, which are more complex yet common in reality.

Original: Dynamic graph neural networks (DyGNNs) have been proposed to tackle highly complex structural and temporal information over dynamic graphs, and have achieved remarkable progress in many predictive tasks [7, 8].
Revised: Dynamic graph neural networks (DyGNNs) have been proposed to tackle highly complex structural and temporal information over dynamic graphs, and have achieved remarkable progress in many predictive tasks [7, 8].

Original: Nevertheless, the existing DyGNNs fail to handle spatio-temporal distribution shifts, which naturally exist in dynamic graphs for various reasons such as survivorship bias [9], selection bias [10, 11], trending [12], etc.
Revised: Nevertheless, the existing DyGNNs fail to handle spatio-temporal distribution shifts, which naturally exist in dynamic graphs for various reasons such as survivorship bias [9], selection bias [10, 11], trending [12], etc.

Original: For example, in financial networks, external factors like period or market would affect the correlations between the payment flows and transaction illegitimacy [13].
Revised: For example, in financial networks, external factors like period or market would affect the correlations between the payment flows and transaction illegitimacy [13].

Original: Trends or communities also affect interaction patterns in coauthor networks [14] and recommendation networks [15].
Revised: Trends or communities also affect interaction patterns in coauthor networks [14] and recommendation networks [15].

Original: If DyGNNs highly rely on spatio-temporal patterns which are variant under distribution shifts, they will inevitably fail to generalize well to the unseen test distributions.
Revised: If DyGNNs highly rely on spatio-temporal patterns which are variant under distribution shifts, they will inevitably fail to generalize well to the unseen test distributions.

Original: To address this issue, in this paper, we study the problem of handling spatio-temporal distribution shifts in dynamic graphs through discovering and utilizing invariant patterns , i.e., structures and features whose predictive abilities are stable across distribution shifts, which remain unexplored in the literature.
Revised: To address this issue, in this paper, we study the problem of handling spatio-temporal distribution shifts in dynamic graphs through discovering and utilizing invariant patterns , i.e., structures and features whose predictive abilities are stable across distribution shifts, which remain unexplored in the literature.

Original: However, this problem is highly non-trivial with the following challenges:
Revised: However, this problem is highly non-trivial with the following challenges:

Original: • How to discover the complex variant and invariant spatio-temporal patterns in dynamic graphs, which include both graph structures and node features varying through time?
Revised: • How to discover the complex variant and invariant spatio-temporal patterns in dynamic graphs, which include both graph structures and node features varying through time?

Original: • How to handle spatio-temporal distribution shifts in a principled manner with discovered variant and invariant patterns?
Revised: • How to handle spatio-temporal distribution shifts in a principled manner with discovered variant and invariant patterns?

Original: To tackle these challenges, we propose a novel DyGNN named Disentangled Intervention-based
Revised: To tackle these challenges, we propose a novel DyGNN named Disentangled Intervention-based

Original: Dynamic Graph Attention Networks ( DIDA 3 ).
Revised: Dynamic Graph Attention Networks ( DIDA 3 ).

Original: Our proposed method handles distribution shifts well by discovering and utilizing invariant spatio-temporal patterns with stable predictive abilities.
Revised: Our proposed method handles distribution shifts well by discovering and utilizing invariant spatio-temporal patterns with stable predictive abilities.

Original: Specifically, we first propose a disentangled spatio-temporal attention network to capture the variant and invariant patterns in dynamic graphs, which enables each node to attend to all its historic neighbors through a disentangled attention message-passing mechanism.
Revised: Specifically, we first propose a disentangled spatio-temporal attention network to capture the variant and invariant patterns in dynamic graphs, which enables each node to attend to all its historic neighbors through a disentangled attention message-passing mechanism.

Original: Then, inspired by causal inference literatures [16, 17], we propose a spatio-temporal intervention mechanism to create multiple intervened distributions by sampling and reassembling variant patterns across neighborhoods and time, such that spurious impacts of variant patterns can be eliminated.
Revised: Then, inspired by causal inference literatures [16, 17], we propose a spatio-temporal intervention mechanism to create multiple intervened distributions by sampling and reassembling variant patterns across neighborhoods and time, such that spurious impacts of variant patterns can be eliminated.

Original: To tackle the challenges that i) variant patterns are highly entangled across nodes and ii) directly generating and mixing up subsets of structures and features to do intervention is computationally expensive, we approximate the intervention process with summarized patterns obtained by the disentangled spatio-temporal attention network instead of original structures and features.
Revised: To tackle the challenges that i) variant patterns are highly entangled across nodes and ii) directly generating and mixing up subsets of structures and features to do intervention is computationally expensive, we approximate the intervention process with summarized patterns obtained by the disentangled spatio-temporal attention network instead of original structures and features.

Original: Lastly, we propose an invariance regularization term to minimize prediction variance in multiple intervened distributions.
Revised: Lastly, we propose an invariance regularization term to minimize prediction variance in multiple intervened distributions.

Original: In this way, our model can capture and utilize invariant patterns with stable predictive abilities to make predictions under distribution shifts.
Revised: In this way, our model can capture and utilize invariant patterns with stable predictive abilities to make predictions under distribution shifts.

Original: Extensive experiments on one synthetic dataset and three real-world datasets demonstrate the superiority of our proposed method over state-of-the-art baselines under distribution shifts.
Revised: Extensive experiments on one synthetic dataset and three real-world datasets demonstrate the superiority of our proposed method over state-of-the-art baselines under distribution shifts.

Original: The contributions of our work are summarized as follows:
Revised: The contributions of our work are summarized as follows:

Original: • We propose Disentangled Intervention-based Dynamic Graph Attention Networks ( DIDA ), which can handle spatio-temporal distribution shifts in dynamic graphs.
Revised: • We propose Disentangled Intervention-based Dynamic Graph Attention Networks ( DIDA ), which can handle spatio-temporal distribution shifts in dynamic graphs.

Original: This is the first study of spatio-temporal distribution shifts in dynamic graphs, to the best of our knowledge.
Revised: This is the first study of spatio-temporal distribution shifts in dynamic graphs, to the best of our knowledge.

Original: • We propose a disentangled spatio-temporal attention network to capture variant and invariant graph patterns.
Revised: • We propose a disentangled spatio-temporal attention network to capture variant and invariant graph patterns.

Original: We further design a spatio-temporal intervention mechanism to create multiple intervened distributions and an invariance regularization term based on causal inference theory to enable the model to focus on invariant patterns under distribution shifts.
Revised: We further design a spatio-temporal intervention mechanism to create multiple intervened distributions and an invariance regularization term based on causal inference theory to enable the model to focus on invariant patterns under distribution shifts.

Original: • Experiments on three real-world datasets and one synthetic dataset demonstrate the superiority of our method over state-of-the-art baselines.
Revised: • Experiments on three real-world datasets and one synthetic dataset demonstrate the superiority of our method over state-of-the-art baselines.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: In this section, we formulate the problem of spatio-temporal distribution shift in dynamic graphs.
Revised: In this section, we formulate the problem of spatio-temporal distribution shift in dynamic graphs.

Original: Dynamic Graph.
Revised: Dynamic Graph.

Original: Consider a graph G with the node set V and the edge set E .
Revised: Consider a graph G with the node set V and the edge set E .

Original: A dynamic graph can be defined as G =
Revised: A dynamic graph can be defined as G =

Original: ( {G t } T t =1 ) , where T is the number of time stamps, G t =
Revised: ( {G t } T t =1 ) , where T is the number of time stamps, G t =

Original: ( V t , E t ) is the graph slice at time stamp t , V =
Revised: ( V t , E t ) is the graph slice at time stamp t , V =

Original: (cid:83)
Revised: (cid:83)

Original: Tt =1 V t , E = (cid:83)
Revised: Tt =1 V t , E = (cid:83)

Original: Tt =1 E t .
Revised: Tt =1 E t .

Original: For simplicity, a graph slice is also denoted as
Revised: For simplicity, a graph slice is also denoted as

Original: G t =
Revised: G t =

Original: ( X t , A t ) , which includes node features and adjacency matrix at time t .
Revised: ( X t , A t ) , which includes node features and adjacency matrix at time t .

Original: We use G t to denote a random variable of G t .
Revised: We use G t to denote a random variable of G t .

Original:                            
Revised:

Original: Distribution                             ( +    ) t= t= t= a a a a a
Revised: Distribution                             ( +    ) t= t= t= a a a a a

Original: InvariantPattern
Revised: InvariantPattern

Original: VariantPattern
Revised: VariantPattern

Original: Disentangled Dynamic Graph Attention Networks
Revised: Disentangled Dynamic Graph Attention Networks

Original:        
Revised:

Original: =   (  |(  = ))
Revised: =   (  |(  = ))

Original: a a a b a b
Revised: a a a b a b

Original: Temporal
Revised: Temporal

Original: Spatial
Revised: Spatial

Original: Spatial-temporal
Revised: Spatial-temporal

Original: Invariance Loss
Revised: Invariance Loss

Original: Prediction tasks.
Revised: Prediction tasks.

Original: For dynamic graphs, the prediction task can be summarized as using past graphs to make predictions, i.e. p ( Y t | G 1 , G 2 , . . . , G t ) =
Revised: For dynamic graphs, the prediction task can be summarized as using past graphs to make predictions, i.e. p ( Y t | G 1 , G 2 , . . . , G t ) =

Original: p ( Y t | G 1: t ) , where label Y t can be node properties or occurrence of links between nodes at time t +1 .
Revised: p ( Y t | G 1: t ) , where label Y t can be node properties or occurrence of links between nodes at time t +1 .

Original: In this paper, we mainly focus on node-level tasks, which are commonly adopted in dynamic graph literatures [7, 8].
Revised: In this paper, we mainly focus on node-level tasks, which are commonly adopted in dynamic graph literatures [7, 8].

Original: Following [18, 19], we factorize the distribution of graph trajectory into ego-graph trajectories, i.e. p ( Y t | G 1: t ) = (cid:81) v p ( y t | G 1: t v ) .
Revised: Following [18, 19], we factorize the distribution of graph trajectory into ego-graph trajectories, i.e. p ( Y t | G 1: t ) = (cid:81) v p ( y t | G 1: t v ) .

Original: An ego-graph induced from node v at time t is defined as G tv =
Revised: An ego-graph induced from node v at time t is defined as G tv =

Original: ( X tv , A tv ) where A tv is the adjacency matrix including all edges in node v ’s L -hop neighbors at time t , i.e. N t v , and X t v includes the features of nodes in N tv .
Revised: ( X tv , A tv ) where A tv is the adjacency matrix including all edges in node v ’s L -hop neighbors at time t , i.e. N t v , and X t v includes the features of nodes in N tv .

Original: The optimization objective is to learn an optimal predictor with empirical risk minimization
Revised: The optimization objective is to learn an optimal predictor with empirical risk minimization

Original: [Equation]
Revised: [Equation]

Original: where f θ is a learnable dynamic graph neural networks, We use G 1: t v , y t to denote the random variable of the ego-graph trajectory and its label, and G 1: t v , y t refer to the respective instances.
Revised: where f θ is a learnable dynamic graph neural networks, We use G 1: t v , y t to denote the random variable of the ego-graph trajectory and its label, and G 1: t v , y t refer to the respective instances.

Original: Spatio-temporal distribution shift.
Revised: Spatio-temporal distribution shift.

Original: However, the optimal predictor trained with the training distribution may not generalize well to the test distribution when there exists a distribution shift problem.
Revised: However, the optimal predictor trained with the training distribution may not generalize well to the test distribution when there exists a distribution shift problem.

Original: In the literature of dynamic graph, researchers are devoted to capture laws of network dynamics which are stable in systems [20, 21, 22, 23, 24].
Revised: In the literature of dynamic graph, researchers are devoted to capture laws of network dynamics which are stable in systems [20, 21, 22, 23, 24].

Original: Following them, we assume the conditional distribution is the same p tr
Revised: Following them, we assume the conditional distribution is the same p tr

Original: ( Y t | G 1: t ) =
Revised: ( Y t | G 1: t ) =

Original: p te ( Y t | G 1: t ) , and only consider the covariate shift problem where p tr ( G 1: t ) ̸
Revised: p te ( Y t | G 1: t ) , and only consider the covariate shift problem where p tr ( G 1: t ) ̸

Original: = p te ( G 1: t ) .
Revised: = p te ( G 1: t ) .

Original: Besides temporal distribution shift which naturally exists in timevarying data [25, 12, 26, 27, 28] and structural distribution shift in non-euclidean data [29, 18, 30], there exists a much more complex spatio-temporal distribution shift in dynamic graphs.
Revised: Besides temporal distribution shift which naturally exists in timevarying data [25, 12, 26, 27, 28] and structural distribution shift in non-euclidean data [29, 18, 30], there exists a much more complex spatio-temporal distribution shift in dynamic graphs.

Original: For example, the distribution of ego-graph trajectories may vary across periods or communities.
Revised: For example, the distribution of ego-graph trajectories may vary across periods or communities.

Original:  Method
Revised:  Method

Original: In this section, we propose Disentangled Intervention-based Dynamic Graph Attention Networks ( DIDA ) to handle spatio-temporal distribution shift in dynamic graphs.
Revised: In this section, we propose Disentangled Intervention-based Dynamic Graph Attention Networks ( DIDA ) to handle spatio-temporal distribution shift in dynamic graphs.

Original: First, we propose a disentangled dynamic graph attention network to extract invariant and variant spatio-temporal patterns.
Revised: First, we propose a disentangled dynamic graph attention network to extract invariant and variant spatio-temporal patterns.

Original: Then we propose a spatio-temporal intervention mechanism to create multiple intervened data distributions.
Revised: Then we propose a spatio-temporal intervention mechanism to create multiple intervened data distributions.

Original: Finally, we optimize the model with invariance loss to make predictions relying on invariant patterns.
Revised: Finally, we optimize the model with invariance loss to make predictions relying on invariant patterns.

Original: 3.1 Handling Spatio-Temporal Distribution Shift
Revised: 3.1 Handling Spatio-Temporal Distribution Shift

Original: Spatio-Temporal Pattern.
Revised: Spatio-Temporal Pattern.

Original: In recent decades of development of dynamic graphs, some scholars endeavor to conclude insightful patterns of network dynamics to reflect how real-world networks evolve through time [31, 32, 33, 34].
Revised: In recent decades of development of dynamic graphs, some scholars endeavor to conclude insightful patterns of network dynamics to reflect how real-world networks evolve through time [31, 32, 33, 34].

Original: For example, the laws of triadic closure describe that two nodes with common neighbors (patterns) tend to have future interactions in social networks [35, 36, 23].
Revised: For example, the laws of triadic closure describe that two nodes with common neighbors (patterns) tend to have future interactions in social networks [35, 36, 23].

Original: Besides structural information, node attributes are also an important part of the patterns, e.g., social interactions can be also affected by gender and age [37].
Revised: Besides structural information, node attributes are also an important part of the patterns, e.g., social interactions can be also affected by gender and age [37].

Original: Instead of manually concluding patterns, we aim at learning the patterns using DyGNNs so that the more complex spatio-temporal patterns with mixed features and structures can be mined in dynamic graphs.
Revised: Instead of manually concluding patterns, we aim at learning the patterns using DyGNNs so that the more complex spatio-temporal patterns with mixed features and structures can be mined in dynamic graphs.

Original: Therefore, we define the spatio-temporal pattern used for node-level prediction as a subset of ego-graph trajectory
Revised: Therefore, we define the spatio-temporal pattern used for node-level prediction as a subset of ego-graph trajectory

Original: [Equation]
Revised: [Equation]

Original: where m tv ( · ) selects structures and attributes from the ego-graph trajectory.
Revised: where m tv ( · ) selects structures and attributes from the ego-graph trajectory.

Original: In [23], the pattern can be explained as an open triad with similar neighborhood, and the model tend to make link predictions to close the triad with ˆ y tu,v = f θ ( P t ( u ) , P t ( v )) based on the laws of triadic closure [38].
Revised: In [23], the pattern can be explained as an open triad with similar neighborhood, and the model tend to make link predictions to close the triad with ˆ y tu,v = f θ ( P t ( u ) , P t ( v )) based on the laws of triadic closure [38].

Original: DyGNNs aim at exploiting predictive spatio-temporal patterns to boost prediction ability.
Revised: DyGNNs aim at exploiting predictive spatio-temporal patterns to boost prediction ability.

Original: However, the predictive power of some patterns may vary across periods or communities due to spatio-temporal distribution shift.
Revised: However, the predictive power of some patterns may vary across periods or communities due to spatio-temporal distribution shift.

Original: Inspired by the causal theory [16, 17], we make the following assumption
Revised: Inspired by the causal theory [16, 17], we make the following assumption

Original: Assumption 1 For a given task, there exists a predictor f ( · ) , for samples ( G 1: t v , y t ) from any distribution, there exists an invariant pattern P tI ( v ) and a variant pattern P tV ( v ) such that y t
Revised: Assumption 1 For a given task, there exists a predictor f ( · ) , for samples ( G 1: t v , y t ) from any distribution, there exists an invariant pattern P tI ( v ) and a variant pattern P tV ( v ) such that y t

Original: = f ( P tI ( v ))
Revised: = f ( P tI ( v ))

Original: + ϵ and P tI ( v ) =
Revised: + ϵ and P tI ( v ) =

Original: G 1: t v \ P tV ( v ) , i.e., y t ⊥ P tV ( v ) | P tI ( v ) .
Revised: G 1: t v \ P tV ( v ) , i.e., y t ⊥ P tV ( v ) | P tI ( v ) .

Original: Assumption 1 shows that invariant patterns P tI ( v ) are sufficiently predictive for label y t and can be exploited across periods and communities without adjusting the predictor, while the influence of variant patterns P tV ( v ) on y t is shielded by the invariant patterns.
Revised: Assumption 1 shows that invariant patterns P tI ( v ) are sufficiently predictive for label y t and can be exploited across periods and communities without adjusting the predictor, while the influence of variant patterns P tV ( v ) on y t is shielded by the invariant patterns.

Original: Training Objective.
Revised: Training Objective.

Original: Our main idea is that to obtain better generalization ability, the model should rely on invariant patterns instead of variant patterns, as the former is sufficient for prediction while the predictivity of the latter could be variant under distribution shift.
Revised: Our main idea is that to obtain better generalization ability, the model should rely on invariant patterns instead of variant patterns, as the former is sufficient for prediction while the predictivity of the latter could be variant under distribution shift.

Original: Along this, our objective can be transformed to
Revised: Along this, our objective can be transformed to

Original: [Equation]
Revised: [Equation]

Original: where f θ 1 ( · ) make predictions based on the invariant patterns, ϕ θ 2 ( · ) aims at finding the invariant patterns.
Revised: where f θ 1 ( · ) make predictions based on the invariant patterns, ϕ θ 2 ( · ) aims at finding the invariant patterns.

Original: Backed by causal theory[16, 17], it can be transformed into
Revised: Backed by causal theory[16, 17], it can be transformed into

Original: [Equation]
Revised: [Equation]

Original: where ‘ do ’ denotes do-calculas to intervene the original distribution [39, 17], S denotes the intervention set and λ is a balancing hyperparameter.
Revised: where ‘ do ’ denotes do-calculas to intervene the original distribution [39, 17], S denotes the intervention set and λ is a balancing hyperparameter.

Original: The idea can be informally described that as in
Revised: The idea can be informally described that as in

Original: Eq.
Revised: Eq.

Original: (3), variant patterns P tV have no influence on the label y t given the invariant patterns P tI , then the prediction would not be varied if we intervene the variant patterns and keep invariant patterns untouched.
Revised: (3), variant patterns P tV have no influence on the label y t given the invariant patterns P tI , then the prediction would not be varied if we intervene the variant patterns and keep invariant patterns untouched.

Original: More details about the connections between objective Eq.(3) and Eq.(4) can be found in
Revised: More details about the connections between objective Eq.(3) and Eq.(4) can be found in

Original: Appendix.
Revised: Appendix.

Original: Remark 1 Minimizing the variance term in Eq.
Revised: Remark 1 Minimizing the variance term in Eq.

Original: (4) help the model to satisfy the constraint of y t ⊥ ˜ P tV ( v ) | ˜
Revised: (4) help the model to satisfy the constraint of y t ⊥ ˜ P tV ( v ) | ˜

Original: P tI ( v ) in Eq.
Revised: P tI ( v ) in Eq.

Original: (3) , i.e., p ( y t | ˜
Revised: (3) , i.e., p ( y t | ˜

Original: P tI ( v ) , ˜ P tV ( v ))
Revised: P tI ( v ) , ˜ P tV ( v ))

Original: =
Revised: =

Original: p ( y t | ˜
Revised: p ( y t | ˜

Original: P tI ( v ))
Revised: P tI ( v ))

Original: 3.2 Disentangled Dynamic Graph Attention Networks
Revised: 3.2 Disentangled Dynamic Graph Attention Networks

Original: Dynamic Neighborhood .
Revised: Dynamic Neighborhood .

Original: To simultaneously consider the spatio-temporal information, we define the dynamic neighborhood as N t ( u ) = { v : ( u, v ) ∈
Revised: To simultaneously consider the spatio-temporal information, we define the dynamic neighborhood as N t ( u ) = { v : ( u, v ) ∈

Original: E t } , which includes all nodes that have interactions with node u at time t .
Revised: E t } , which includes all nodes that have interactions with node u at time t .

Original: Disentangled Spatio-temporal Graph Attention Layer.
Revised: Disentangled Spatio-temporal Graph Attention Layer.

Original: To capture spatio-temporal pattern for each node, we propose a spatio-temporal graph attention to enable each node to attend to its dynamic neighborhood simultaneously.
Revised: To capture spatio-temporal pattern for each node, we propose a spatio-temporal graph attention to enable each node to attend to its dynamic neighborhood simultaneously.

Original: For a node u at time stamp t and its neighbors v ∈ N t ′ ( u ) , ∀ t ′ ≤ t , we calculate the Query-Key-Value vectors as:
Revised: For a node u at time stamp t and its neighbors v ∈ N t ′ ( u ) , ∀ t ′ ≤ t , we calculate the Query-Key-Value vectors as:

Original: [Equation]
Revised: [Equation]

Original: where h tu denotes the representation of node u at the time stamp t , q , k , v represents the query, key and value vector, respectively, and we omit the bias term for brevity.
Revised: where h tu denotes the representation of node u at the time stamp t , q , k , v represents the query, key and value vector, respectively, and we omit the bias term for brevity.

Original: TE( t ) denotes temporal encoding techniques to obtain embeddings of time t so that the time of link occurrence can be considered inherently [40, 41].
Revised: TE( t ) denotes temporal encoding techniques to obtain embeddings of time t so that the time of link occurrence can be considered inherently [40, 41].

Original: Then, we can calculate the attention scores among nodes in the dynamic neighborhood to obtain the structural masks
Revised: Then, we can calculate the attention scores among nodes in the dynamic neighborhood to obtain the structural masks

Original: [Equation]
Revised: [Equation]

Original: where d denotes feature dimension, m I and m V represent the masks of invariant and variant structural patterns.
Revised: where d denotes feature dimension, m I and m V represent the masks of invariant and variant structural patterns.

Original: In this way, dynamic neighbors with higher attention scores in invariant patterns will have lower attention scores in variant ones, which means the invariant and variant patterns have a negative correlation.
Revised: In this way, dynamic neighbors with higher attention scores in invariant patterns will have lower attention scores in variant ones, which means the invariant and variant patterns have a negative correlation.

Original: To capture invariant featural pattern, we adopt a learnable featural mask m f = Softmax ( w f ) to select features from the messages of dynamic neighbors.
Revised: To capture invariant featural pattern, we adopt a learnable featural mask m f = Softmax ( w f ) to select features from the messages of dynamic neighbors.

Original: Then the messages of dynamic neighborhood can be summarized with respective masks,
Revised: Then the messages of dynamic neighborhood can be summarized with respective masks,

Original: [Equation]
Revised: [Equation]

Original: where Agg ( · ) denotes aggregating and summarizing messages from dynamic neighborhood.
Revised: where Agg ( · ) denotes aggregating and summarizing messages from dynamic neighborhood.

Original: To further disentangle the invariant and variant patterns, we design different aggregation functions
Revised: To further disentangle the invariant and variant patterns, we design different aggregation functions

Original: Agg I ( · ) and Agg V ( · ) to summarize specific messages from masked dynamic neighborhood respectively.
Revised: Agg I ( · ) and Agg V ( · ) to summarize specific messages from masked dynamic neighborhood respectively.

Original: Then the pattern summarizations are added up as hidden embeddings to be fed into subsequent layers.
Revised: Then the pattern summarizations are added up as hidden embeddings to be fed into subsequent layers.

Original: [Equation]
Revised: [Equation]

Original: Overall Architecture.
Revised: Overall Architecture.

Original: The overall architecture is a stacking of spatio-temporal graph attention layers.
Revised: The overall architecture is a stacking of spatio-temporal graph attention layers.

Original: Like classic graph message-passing networks, this enables each node to access high-order dynamic neighborhood indirectly, where z tI ( u ) and z tV ( u ) at l -th layer can be a summarization of invariant and variant patterns in l -order dynamic neighborhood.
Revised: Like classic graph message-passing networks, this enables each node to access high-order dynamic neighborhood indirectly, where z tI ( u ) and z tV ( u ) at l -th layer can be a summarization of invariant and variant patterns in l -order dynamic neighborhood.

Original: In practice, the attention can be easily extended to multi-head attention [42] to stable the training process and model multi-faceted graph evolution [43].
Revised: In practice, the attention can be easily extended to multi-head attention [42] to stable the training process and model multi-faceted graph evolution [43].

Original: 3.3 Spatio-Temporal Intervention Mechanism
Revised: 3.3 Spatio-Temporal Intervention Mechanism

Original: Direct Intervention.
Revised: Direct Intervention.

Original: One way of intervening variant pattern distribution as Eq.
Revised: One way of intervening variant pattern distribution as Eq.

Original: (4) is directly generating and altering the variant patterns.
Revised: (4) is directly generating and altering the variant patterns.

Original: However, this is infeasible in practice due to the following reasons: First, since it has to intervene the dynamic neighborhood and features nodewisely, the computational complexity is unbearable.
Revised: However, this is infeasible in practice due to the following reasons: First, since it has to intervene the dynamic neighborhood and features nodewisely, the computational complexity is unbearable.

Original: Second, generating variant patterns including time-varying structures and features is another intractable problem.
Revised: Second, generating variant patterns including time-varying structures and features is another intractable problem.

Original: Approximate Intervention.
Revised: Approximate Intervention.

Original: To tackle the problems mentioned above, we propose to approximate the patterns P t with summarized patterns z t found in Sec. 3.2.
Revised: To tackle the problems mentioned above, we propose to approximate the patterns P t with summarized patterns z t found in Sec. 3.2.

Original: As z tI ( u ) and z tV ( u ) act as summarizations of invariant and variant spatio-temporal patterns for node u at time t , we approximate the intervention process by sampling and replacing the variant pattern summarizations instead of altering original structures and features with generated ones.
Revised: As z tI ( u ) and z tV ( u ) act as summarizations of invariant and variant spatio-temporal patterns for node u at time t , we approximate the intervention process by sampling and replacing the variant pattern summarizations instead of altering original structures and features with generated ones.

Original: To do spatio-temporal intervention, we collect variant patterns of all nodes at all time, from which we sample one variant pattern to replace the variant patterns of other nodes across time.
Revised: To do spatio-temporal intervention, we collect variant patterns of all nodes at all time, from which we sample one variant pattern to replace the variant patterns of other nodes across time.

Original: For example, we can use the variant pattern of node v at time tto replace the variant pattern of node u at time t 1 as
Revised: For example, we can use the variant pattern of node v at time tto replace the variant pattern of node u at time t 1 as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As the invariant pattern summarization is kept the same, the label should not be changed.
Revised: As the invariant pattern summarization is kept the same, the label should not be changed.

Original: Thanks to the disentangled spatio-temporal graph attention, we get variant patterns across neighborhoods and time, which can act as natural intervention samples inside data so that the complexity of the generation problem can also be avoided.
Revised: Thanks to the disentangled spatio-temporal graph attention, we get variant patterns across neighborhoods and time, which can act as natural intervention samples inside data so that the complexity of the generation problem can also be avoided.

Original: By doing Eq.
Revised: By doing Eq.

Original: (9) multiple times, we can obtain multiple intervened data distributions for the subsequent optimization.
Revised: (9) multiple times, we can obtain multiple intervened data distributions for the subsequent optimization.

Original: 3.4 Optimization with Invariance Loss
Revised: 3.4 Optimization with Invariance Loss

Original: Based on the multiple intervened data distributions with different variant patterns, we can next optimize the model to focus on invariant patterns to make predictions.
Revised: Based on the multiple intervened data distributions with different variant patterns, we can next optimize the model to focus on invariant patterns to make predictions.

Original: Here, we introduce invariance loss to instantiate Eq.
Revised: Here, we introduce invariance loss to instantiate Eq.

Original: Let z
Revised: Let z

Original: I and z V be the summarized invariant and variant patterns, we calculate the task loss by only using the invariant patterns
Revised: I and z V be the summarized invariant and variant patterns, we calculate the task loss by only using the invariant patterns

Original: [Equation]
Revised: [Equation]

Original: ) where f ( · ) is the predictor.
Revised: ) where f ( · ) is the predictor.

Original: The task loss let the model utilize the invariant patterns to make predictions.
Revised: The task loss let the model utilize the invariant patterns to make predictions.

Original: Then we calculate the mixed loss as
Revised: Then we calculate the mixed loss as

Original: [Equation]
Revised: [Equation]

Original: where another predictor g ( · ) makes predictions using both invariant patterns z V and variant patterns z I .
Revised: where another predictor g ( · ) makes predictions using both invariant patterns z V and variant patterns z I .

Original: The mixed loss measure the model’s prediction ability when variant patterns are also exposed to the model.
Revised: The mixed loss measure the model’s prediction ability when variant patterns are also exposed to the model.

Original: Then the invariance loss is calculated by
Revised: Then the invariance loss is calculated by

Original: [Equation]
Revised: [Equation]

Original: where ‘do’ denotes the intervention mechanism as mentioned in Section.
Revised: where ‘do’ denotes the intervention mechanism as mentioned in Section.

Original: The invariance loss measures the variance of the model’s prediction ability under multiple intervened distributions.
Revised: The invariance loss measures the variance of the model’s prediction ability under multiple intervened distributions.

Original: The final training objective is
Revised: The final training objective is

Original: [Equation]
Revised: [Equation]

Original: where the task loss L is minimized to exploit invariant patterns while the invariance loss L do helps the model to discover invariant and variant patterns, and λ is a hyperparameter to balance between two objectives.
Revised: where the task loss L is minimized to exploit invariant patterns while the invariance loss L do helps the model to discover invariant and variant patterns, and λ is a hyperparameter to balance between two objectives.

Original: After training, we only adopt invariant patterns to make predictions in the inference stage.
Revised: After training, we only adopt invariant patterns to make predictions in the inference stage.

Original: The overall algorithm is summarized in Table 1.
Revised: The overall algorithm is summarized in Table 1.

Original: Algorithm time as described in Section 3.
Revised: Algorithm time as described in Section 3.

Original: Calculate task loss and mixed loss as Eq.
Revised: Calculate task loss and mixed loss as Eq.

Original: (10) and Eq.
Revised: (10) and Eq.

Original: Sample S variant patterns from collections of z tV , to construct intervention set S 5: for s in S do 6:
Revised: Sample S variant patterns from collections of z tV , to construct intervention set S 5: for s in S do 6:

Original: Replace the nodes’ variant pattern summarizations with s as Section 3.
Revised: Replace the nodes’ variant pattern summarizations with s as Section 3.

Original: Calculate mixed loss as Eq.
Revised: Calculate mixed loss as Eq.

Original: (11) 8: end for 9:
Revised: (11) 8: end for 9:

Original: Calculate invariance loss as Eq.
Revised: Calculate invariance loss as Eq.

Original: Update the model according to Eq.
Revised: Update the model according to Eq.

Original: (13) 11: end for
Revised: (13) 11: end for

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct extensive experiments to verify that our framework can handle spatiotemporal distribution shifts by discovering and utilizing invariant patterns.
Revised: In this section, we conduct extensive experiments to verify that our framework can handle spatiotemporal distribution shifts by discovering and utilizing invariant patterns.

Original: More Details of the settings and other results can be found in Appendix.
Revised: More Details of the settings and other results can be found in Appendix.

Original: Baselines.
Revised: Baselines.

Original: We adopt several representative GNNs and Out-of-Distribution(OOD) generalization methods as our baselines:
Revised: We adopt several representative GNNs and Out-of-Distribution(OOD) generalization methods as our baselines:

Original: • Static GNNs: GAE [44], a representative static GNN with stacking of graph convolutions; VGAE [44] further introduces variational variables into GAE.
Revised: • Static GNNs: GAE [44], a representative static GNN with stacking of graph convolutions; VGAE [44] further introduces variational variables into GAE.

Original: • Dynamic GNNs: GCRN [45],a representative dynamic GNN that first adopts a GCN[44] to obtain node embeddings and then a GRU [46] to model the dynamics; EvolveGCN [13] adopts a LSTM[47] or GRU [46] to flexibly evolve the GCN[44] parameters instead of directly learning the temporal node embeddings; DySAT [43] models dynamic graph using structural and temporal self-attention.
Revised: • Dynamic GNNs: GCRN [45],a representative dynamic GNN that first adopts a GCN[44] to obtain node embeddings and then a GRU [46] to model the dynamics; EvolveGCN [13] adopts a LSTM[47] or GRU [46] to flexibly evolve the GCN[44] parameters instead of directly learning the temporal node embeddings; DySAT [43] models dynamic graph using structural and temporal self-attention.

Original: • OOD generalization methods: IRM [48] aims at learning an invariant predictor which minimizes the empirical risks for all training domains; GroupDRO [49] reduces differences in risk across training domains to reduce the model’s sensitivity to distributional shifts; V-REx [50] puts more weight on training domains with larger errors when minimizing empirical risk.
Revised: • OOD generalization methods: IRM [48] aims at learning an invariant predictor which minimizes the empirical risks for all training domains; GroupDRO [49] reduces differences in risk across training domains to reduce the model’s sensitivity to distributional shifts; V-REx [50] puts more weight on training domains with larger errors when minimizing empirical risk.

Original: 4.1 Real-world Datasets
Revised: 4.1 Real-world Datasets

Original: Settings.
Revised: Settings.

Original: We use 3 real-world dynamic graph datasets, including COLLAB, Yelp and Transaction.
Revised: We use 3 real-world dynamic graph datasets, including COLLAB, Yelp and Transaction.

Original: We adopt the challenging inductive future link prediction task, where the model exploits past graphs to make link prediction in the next time step.
Revised: We adopt the challenging inductive future link prediction task, where the model exploits past graphs to make link prediction in the next time step.

Original: Each dataset can be split into several partial dynamic graphs based on its field information.
Revised: Each dataset can be split into several partial dynamic graphs based on its field information.

Original: For brevity, we use ‘w/ DS’ and ‘w/o DS’ to represent test data with and without distribution shift respectively.
Revised: For brevity, we use ‘w/ DS’ and ‘w/o DS’ to represent test data with and without distribution shift respectively.

Original: To measure models’ performance under spatiotemporal distribution shift, we choose one field as ‘w/ DS’ and the left others are further split into training, validation and test data (‘w/o DS’) chronologically.
Revised: To measure models’ performance under spatiotemporal distribution shift, we choose one field as ‘w/ DS’ and the left others are further split into training, validation and test data (‘w/o DS’) chronologically.

Original: Note that the ‘w/o DS’ is a merged dynamic graph without field information and ‘w/ DS’ is unseen during training, which is more practical and challenging in real-world scenarios.
Revised: Note that the ‘w/o DS’ is a merged dynamic graph without field information and ‘w/ DS’ is unseen during training, which is more practical and challenging in real-world scenarios.

Original: More details on their spatio-temporal distribution shifts are provided in Appendix.
Revised: More details on their spatio-temporal distribution shifts are provided in Appendix.

Original: Here we briefly introduce the real-world datasets as follows
Revised: Here we briefly introduce the real-world datasets as follows

Original: • COLLAB [51] 4 is an academic collaboration dataset with papers that were published during 1990-2006.
Revised: • COLLAB [51] 4 is an academic collaboration dataset with papers that were published during 1990-2006.

Original: Node and edge represent author and coauthorship respectively.
Revised: Node and edge represent author and coauthorship respectively.

Original: Based on the field of co-authored publication, each edge has the field information including "Data Mining", "Database", "Medical Informatics", "Theory" and "Visualization".
Revised: Based on the field of co-authored publication, each edge has the field information including "Data Mining", "Database", "Medical Informatics", "Theory" and "Visualization".

Original: The time granularity is year, includingtime slices in total.
Revised: The time granularity is year, includingtime slices in total.

Original: We use "Data Mining" as ‘w/ DS’ and the left as ‘w/o DS’.
Revised: We use "Data Mining" as ‘w/ DS’ and the left as ‘w/o DS’.

Original: • Yelp [43] 5 is a business review dataset, containing customer reviews on business.
Revised: • Yelp [43] 5 is a business review dataset, containing customer reviews on business.

Original: Node and edge represent customer/business and review behavior respectively.
Revised: Node and edge represent customer/business and review behavior respectively.

Original: We consider interactions in five categories of business including "Pizza", "American (New) Food", "Coffee & Tea ", "Sushi Bars" and "Fast Food" from January 2019 to December 2020.
Revised: We consider interactions in five categories of business including "Pizza", "American (New) Food", "Coffee & Tea ", "Sushi Bars" and "Fast Food" from January 2019 to December 2020.

Original: The time granularity is month, including 24 time slices in total.
Revised: The time granularity is month, including 24 time slices in total.

Original: We use "Pizza" as ‘w/ DS’ and the left as ‘w/o DS’.
Revised: We use "Pizza" as ‘w/ DS’ and the left as ‘w/o DS’.

Original: • Transaction 6 is a secondary market transaction dataset, which records transaction behaviors of users from 10th April 2022 to 10th May 2022.
Revised: • Transaction 6 is a secondary market transaction dataset, which records transaction behaviors of users from 10th April 2022 to 10th May 2022.

Original: Node and edge represent user and transaction respectively.
Revised: Node and edge represent user and transaction respectively.

Original: The transactions have 4 categories, including "Pants", "Outwears", "Shirts" and "Hoodies".
Revised: The transactions have 4 categories, including "Pants", "Outwears", "Shirts" and "Hoodies".

Original: The time granularity is day, including 30 time slices in total.
Revised: The time granularity is day, including 30 time slices in total.

Original: We use "Pants" as ‘w/ DS’ and the left as ‘w/o DS’.
Revised: We use "Pants" as ‘w/ DS’ and the left as ‘w/o DS’.

Original: Results.
Revised: Results.

Original: Based on the results on real-world datasets in Table.
Revised: Based on the results on real-world datasets in Table.

Original: we have the following observations:
Revised: we have the following observations:

Original: • Baselines fail dramatically under distribution shift: 1) Although DyGNN baselines perform well on test data without distribution shift, their performance drops greatly under distribution shift.
Revised: • Baselines fail dramatically under distribution shift: 1) Although DyGNN baselines perform well on test data without distribution shift, their performance drops greatly under distribution shift.

Original: In particular, the performance of DySAT, which is the best-performed DyGNN in ‘w/o DS’, drop by nearly 12%, 12% and 5% in ‘w/ DS’.
Revised: In particular, the performance of DySAT, which is the best-performed DyGNN in ‘w/o DS’, drop by nearly 12%, 12% and 5% in ‘w/ DS’.

Original: In Yelp and Transaction, GCRN and EGCN even underperform static GNNs, GAE and VGAE.
Revised: In Yelp and Transaction, GCRN and EGCN even underperform static GNNs, GAE and VGAE.

Original: This phenomenon shows that the existing DyGNNs may exploit variant patterns and thus fail to handle distribution shift.
Revised: This phenomenon shows that the existing DyGNNs may exploit variant patterns and thus fail to handle distribution shift.

Original: Moreover, as generalization baselines are not specially designed to consider spatio-temporal distribution shift in dynamic graphs, they only have limited improvements in Yelp and Transaction.
Revised: Moreover, as generalization baselines are not specially designed to consider spatio-temporal distribution shift in dynamic graphs, they only have limited improvements in Yelp and Transaction.

Original: In particular, they rely on ground-truth environment labels to achieve OOD generalization, which are unavailable for real dynamic graphs.
Revised: In particular, they rely on ground-truth environment labels to achieve OOD generalization, which are unavailable for real dynamic graphs.

Original: The inferior performance indicates that they cannot generalize well without accurate environment labels, which verifies that lacking environmental labels is also a key challenge for handling distribution shifts of dynamic graphs.
Revised: The inferior performance indicates that they cannot generalize well without accurate environment labels, which verifies that lacking environmental labels is also a key challenge for handling distribution shifts of dynamic graphs.

Original: • Our method can better handle distribution shift than the baselines, especially in stronger distribution shift.
Revised: • Our method can better handle distribution shift than the baselines, especially in stronger distribution shift.

Original: DIDA improves significantly over all baselines in ‘w/ DS’ for all datasets.
Revised: DIDA improves significantly over all baselines in ‘w/ DS’ for all datasets.

Original: Note that
Revised: Note that

Original: https://www.aminer.cn/collaboration.https://www.yelp.com/dataset
Revised: https://www.aminer.cn/collaboration.https://www.yelp.com/dataset

Original: Collected from Alibaba
Revised: Collected from Alibaba

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Yelp has stronger temporal distribution shift since COVID-19 happens in the midway, strongly affecting consumers’ behavior in business, while DIDA outperforms the most competitive baseline
Revised: Yelp has stronger temporal distribution shift since COVID-19 happens in the midway, strongly affecting consumers’ behavior in business, while DIDA outperforms the most competitive baseline

Original: GroupDRO by 9% in ‘w/ DS’.
Revised: GroupDRO by 9% in ‘w/ DS’.

Original: In comparison to similar field information in Yelp (all restaurants) and Transaction (all costumes), COLLAB has stronger spatial distribution shift since the fields are more different to each other, while DIDA outperforms the most competitive baseline DySAT by 5% in ‘w/ DS’.
Revised: In comparison to similar field information in Yelp (all restaurants) and Transaction (all costumes), COLLAB has stronger spatial distribution shift since the fields are more different to each other, while DIDA outperforms the most competitive baseline DySAT by 5% in ‘w/ DS’.

Original: 4.2 Synthetic Dataset
Revised: 4.2 Synthetic Dataset

Original: Settings.
Revised: Settings.

Original: To evaluate the model’s generalization ability under spatio-temporal distribution shift, following [18], we introduce manually designed shifts in dataset COLLAB with all fields merged.
Revised: To evaluate the model’s generalization ability under spatio-temporal distribution shift, following [18], we introduce manually designed shifts in dataset COLLAB with all fields merged.

Original: Denote original features and structures as X t 1 ∈ R N × d and A t ∈ { 0 , 1 } N × N .
Revised: Denote original features and structures as X t 1 ∈ R N × d and A t ∈ { 0 , 1 } N × N .

Original: For each time t , we uniformly sample p ( t ) |E
Revised: For each time t , we uniformly sample p ( t ) |E

Original: t +1 | positive links and (1 − p ( t ))
Revised: t +1 | positive links and (1 − p ( t ))

Original: |E t +1 | negative links in A t +1 .
Revised: |E t +1 | negative links in A t +1 .

Original: Then they are factorized into variant features X t 2 ∈ R N × d with property of structural preservation.
Revised: Then they are factorized into variant features X t 2 ∈ R N × d with property of structural preservation.

Original: Two portions of features are concatenated as X t =
Revised: Two portions of features are concatenated as X t =

Original: [ X t 1 , X t 2 ] as input node features for training and inference.
Revised: [ X t 1 , X t 2 ] as input node features for training and inference.

Original: The sampling probability p ( t ) =
Revised: The sampling probability p ( t ) =

Original: clip ( p + σcos ( t ) , 0 , 1) refers to the intensity of shifts, where the variant features X t 2 constructed with higher p ( t ) will have stronger correlations with future link A t
Revised: clip ( p + σcos ( t ) , 0 , 1) refers to the intensity of shifts, where the variant features X t 2 constructed with higher p ( t ) will have stronger correlations with future link A t

Original: +1 .
Revised: +1 .

Original: We set p test = 0 .
Revised: We set p test = 0 .

Original: 1 , σ test = 0 , σ train = 0 .
Revised: 1 , σ test = 0 , σ train = 0 .

Original: and vary p train in from 0.4 to 0.8 for evaluation.
Revised: and vary p train in from 0.4 to 0.8 for evaluation.

Original: Since the correlations between X t 2 and label A t +1 vary through time and neighborhood, patterns include X tare variant under distribution shifts.
Revised: Since the correlations between X t 2 and label A t +1 vary through time and neighborhood, patterns include X tare variant under distribution shifts.

Original: As static GNNs can not support time-varing features, we omit their results.
Revised: As static GNNs can not support time-varing features, we omit their results.

Original: Results.
Revised: Results.

Original: Based on the results on synthetic dataset in Table.
Revised: Based on the results on synthetic dataset in Table.

Original: 2, we have the following observations:
Revised: 2, we have the following observations:

Original: • Our method can better handle distribution shift than the baselines.
Revised: • Our method can better handle distribution shift than the baselines.

Original: Although the baselines achieve high performance when training, their performance drop drastically in the test stage, which shows that the existing DyGNNs fail to handle distribution shifts.
Revised: Although the baselines achieve high performance when training, their performance drop drastically in the test stage, which shows that the existing DyGNNs fail to handle distribution shifts.

Original: In terms of test results, DIDA consistently outperforms DyGNN baselines by a significantly large margin.
Revised: In terms of test results, DIDA consistently outperforms DyGNN baselines by a significantly large margin.

Original: In particular , DIDA surpasses the best-performed baseline by nearly 13%/10%/5% in test results for different shift levels.
Revised: In particular, DIDA surpasses the best-performed baseline by nearly 13%/10%/5% in test results for different shift levels.

Original: For the general OOD baselines, they reduce the variance in some cases while their improvements are not significant.
Revised: For the general OOD baselines, they reduce the variance in some cases while their improvements are not significant.

Original: Instead, DIDA is specially designed for dynamic graphs and can exploit the invariant spatio-temporal patterns to handle distribution shift.
Revised: Instead, DIDA is specially designed for dynamic graphs and can exploit the invariant spatio-temporal patterns to handle distribution shift.

Original: • Our method can exploit invariant patterns to consistently alleviate harmful effects of variant patterns under different distribution shift levels.
Revised: • Our method can exploit invariant patterns to consistently alleviate harmful effects of variant patterns under different distribution shift levels.

Original: As shift level increases, almost all baselines increase in train results and decline in test results.
Revised: As shift level increases, almost all baselines increase in train results and decline in test results.

Original: This phenomenon shows that as the relationship between variant patterns and labels goes stronger, the existing DyGNNs become more dependent on the variant patterns when training, causing their failure in test stage.
Revised: This phenomenon shows that as the relationship between variant patterns and labels goes stronger, the existing DyGNNs become more dependent on the variant patterns when training, causing their failure in test stage.

Original: Instead, the rise in train results and drop in test results of DIDA are significantly lower than baselines, which demonstrates that DIDA can exploit invariant patterns and alleviate the harmful effects of variant patterns under distribution shift.
Revised: Instead, the rise in train results and drop in test results of DIDA are significantly lower than baselines, which demonstrates that DIDA can exploit invariant patterns and alleviate the harmful effects of variant patterns under distribution shift.

Original: 4.3 Complexity Analysis
Revised: 4.3 Complexity Analysis

Original: We analyze the computational complexity of DIDA as follows.
Revised: We analyze the computational complexity of DIDA as follows.

Original: Denote | V | and | E | as the total number of nodes and edges in the graph, respectively, and d as the dimensionality of the hidden representation.
Revised: Denote | V | and | E | as the total number of nodes and edges in the graph, respectively, and d as the dimensionality of the hidden representation.

Original: The spatio-temporal aggregation has a time complexity of O ( | E | d + | V | d 2 ) .
Revised: The spatio-temporal aggregation has a time complexity of O ( | E | d + | V | d 2 ) .

Original: The disentangled component adds a constant multiplier 2 , which does not affect the time complexity of aggregation.
Revised: The disentangled component adds a constant multiplier 2 , which does not affect the time complexity of aggregation.

Original: Denote | E p | as the number of edges to predict and | S | as the size of the intervention set.
Revised: Denote | E p | as the number of edges to predict and | S | as the size of the intervention set.

Original: Our intervention mechanism has a time complexity of O ( | E p || S | d ) in training, and does not put extra time complexity in inference.
Revised: Our intervention mechanism has a time complexity of O ( | E p || S | d ) in training, and does not put extra time complexity in inference.

Original: Therefore, the overall time complexity of DIDA is
Revised: Therefore, the overall time complexity of DIDA is

Original: O ( | E | d + | V | d 2 + | E p || S | d ) .
Revised: O ( | E | d + | V | d 2 + | E p || S | d ) .

Original: Notice that | S | is a hyper-parameter and is usually set as a small constant.
Revised: Notice that | S | is a hyper-parameter and is usually set as a small constant.

Original: In summary, DIDA has a linear time complexity with respect to the number of nodes and edges, which is on par with the existing dynamic GNNs.
Revised: In summary, DIDA has a linear time complexity with respect to the number of nodes and edges, which is on par with the existing dynamic GNNs.

Original: 4.4 Ablation study
Revised: 4.4 Ablation study

Original: In this section, we conduct ablation studies to verify the effectiveness of the proposed spatio-temporal intervention mechanism and disentangled graph attention in DIDA .
Revised: In this section, we conduct ablation studies to verify the effectiveness of the proposed spatio-temporal intervention mechanism and disentangled graph attention in DIDA .

Original: [Figure]
Revised: [Figure]

Original: and disentangled attention, where ’w/o I’ denotes removing the spatio-temporal intervention mechanism in DIDA and ’w/o I&D’ further removes disentangled attention.

Original: Spatio-temporal intervention mechanism. We remove the intervention mechanism mentioned in Sec 3.3.
Revised: We remove the intervention mechanism mentioned in Sec 3.3.

Original: From Figure 2, we can see that without spatio-temporal intervention, the model’s performance drop significantly especially in the synthetic dataset, which verifies that our intervention mechanism helps the model to focus on invariant patterns to make predictions.
Revised: From Figure 2, we can see that without spatio-temporal intervention, the model’s performance drop significantly especially in the synthetic dataset, which verifies that our intervention mechanism helps the model to focus on invariant patterns to make predictions.

Original: Disentangled graph attention.
Revised: Disentangled graph attention.

Original: We further remove the disentangled attention mentioned in Sec 3.2.
Revised: We further remove the disentangled attention mentioned in Sec 3.2.

Original: From Figure 2, we can see that disentangled attention is a critical component in the model design, especially in Yelp dataset.
Revised: From Figure 2, we can see that disentangled attention is a critical component in the model design, especially in Yelp dataset.

Original: Moreover, without disentangled module, the model is unable to obtain variant and invariant patterns for the subsequent intervention.
Revised: Moreover, without disentangled module, the model is unable to obtain variant and invariant patterns for the subsequent intervention.

Original:  Related Work
Revised:  Related Work

Original: Dynamic Graph Neural Networks.
Revised: Dynamic Graph Neural Networks.

Original: To tackle the complex structural and temporal information in dynamic graphs, considerable research attention has been devoted to dynamic graph neural networks (DyGNNs) [7, 8].
Revised: To tackle the complex structural and temporal information in dynamic graphs, considerable research attention has been devoted to dynamic graph neural networks (DyGNNs) [7, 8].

Original: A classic of DyGNNs first adopt a GNN to aggregate structural information for graph at each time, followed by a sequence model like RNN [52, 53, 54, 45] or temporal self-attention [43] to process temporal information.
Revised: A classic of DyGNNs first adopt a GNN to aggregate structural information for graph at each time, followed by a sequence model like RNN [52, 53, 54, 45] or temporal self-attention [43] to process temporal information.

Original: Another classic of DyGNNs first introduce time-encoding techniques to represent each temporal link as a function of time, followed by a spatial module like GNN or memory module [20, 55, 40, 41] to process structural information.
Revised: Another classic of DyGNNs first introduce time-encoding techniques to represent each temporal link as a function of time, followed by a spatial module like GNN or memory module [20, 55, 40, 41] to process structural information.

Original: To obtain more fine-grained continous node embeddings in dynamic graphs, some work further leverages neural interaction processes [56] and ordinary differential equation [57].
Revised: To obtain more fine-grained continous node embeddings in dynamic graphs, some work further leverages neural interaction processes [56] and ordinary differential equation [57].

Original: DyGNNs have been widely applied in real-world applications, including dynamic anomaly detection [58], event forecasting [59], dynamic recommendation [60], social character prediction [61], user modeling [62], temporal knowledge graph completion [63], etc.
Revised: DyGNNs have been widely applied in real-world applications, including dynamic anomaly detection [58], event forecasting [59], dynamic recommendation [60], social character prediction [61], user modeling [62], temporal knowledge graph completion [63], etc.

Original: In this paper, we consider DyGNNs under spatio-temporal distribution shift, which remains unexplored in dynamic graph neural networks literature.
Revised: In this paper, we consider DyGNNs under spatio-temporal distribution shift, which remains unexplored in dynamic graph neural networks literature.

Original: Out-of-Distribution Generalization.
Revised: Out-of-Distribution Generalization.

Original: Most existing machine learning methods assume that the testing and training data are independent and identically distributed, which is not guaranteed to hold in many real-world scenarios [64].
Revised: Most existing machine learning methods assume that the testing and training data are independent and identically distributed, which is not guaranteed to hold in many real-world scenarios [64].

Original: In particular, there might be uncontrollable distribution shifts between training and testing data distribution, which may lead to sharp drop of model performance.
Revised: In particular, there might be uncontrollable distribution shifts between training and testing data distribution, which may lead to sharp drop of model performance.

Original: To solve this problem, Out-of-Distribution (OOD) generalization problem has recently become a central research topic in various areas [65, 64, 66].
Revised: To solve this problem, Out-of-Distribution (OOD) generalization problem has recently become a central research topic in various areas [65, 64, 66].

Original: Recently, several works attempt to handle distribution shift on graphs [67, 29, 18, 68, 11, 69, 70, 71, 72, 73].
Revised: Recently, several works attempt to handle distribution shift on graphs [67, 29, 18, 68, 11, 69, 70, 71, 72, 73].

Original: Another classic of OOD methods most related to our works handle distribution shifts on time-series data [25, 26, 12, 27, 28, 74].
Revised: Another classic of OOD methods most related to our works handle distribution shifts on time-series data [25, 26, 12, 27, 28, 74].

Original: Current works consider either only structural distribution shift for static graphs or only temporal distribution shift for time-series data.
Revised: Current works consider either only structural distribution shift for static graphs or only temporal distribution shift for time-series data.

Original: However, spatio-temporal distribution shifts in dynamic graphs are more complex yet remain unexplored.
Revised: However, spatio-temporal distribution shifts in dynamic graphs are more complex yet remain unexplored.

Original: To the best of our knowledge, this is the first study of spatio-temporal distribution shifts in dynamic graphs.
Revised: To the best of our knowledge, this is the first study of spatio-temporal distribution shifts in dynamic graphs.

Original: Disentangled Representation Learning.
Revised: Disentangled Representation Learning.

Original: Disentangled representation learning aims to characterize the multiple latent explanatory factors behind the observed data, where the factors are represented by different vectors [75].
Revised: Disentangled representation learning aims to characterize the multiple latent explanatory factors behind the observed data, where the factors are represented by different vectors [75].

Original: Besides its applications in computer vision [76, 77, 78, 79, 80] and recommendation [81, 82, 83, 84, 85, 86], several disentangled GNNs have proposed to generalize disentangled representation learning in graph data recently.
Revised: Besides its applications in computer vision [76, 77, 78, 79, 80] and recommendation [81, 82, 83, 84, 85, 86], several disentangled GNNs have proposed to generalize disentangled representation learning in graph data recently.

Original: DisenGCN [87] and IPGDN [88] utilize the dynamic routing mechanism to disentangle latent factors for node representations.
Revised: DisenGCN [87] and IPGDN [88] utilize the dynamic routing mechanism to disentangle latent factors for node representations.

Original: FactorGCN [89] decomposes the input graph into several interpretable factor graphs.
Revised: FactorGCN [89] decomposes the input graph into several interpretable factor graphs.

Original: DGCL [90, 91] aim to learn disentangled graph-level representations with self-supervision.
Revised: DGCL [90, 91] aim to learn disentangled graph-level representations with self-supervision.

Original: Some works factorize deep generative models based on node, edge, static, dynamic factors [92] or spatial, temporal, graph factors [93] to achieve interpretable dynamic graph generation.
Revised: Some works factorize deep generative models based on node, edge, static, dynamic factors [92] or spatial, temporal, graph factors [93] to achieve interpretable dynamic graph generation.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose Disentangled Intervention-based Dynamic Graph Attention Networks ( DIDA ) to handle spatio-temporal distribution shift in dynamic graphs.
Revised: In this paper, we propose Disentangled Intervention-based Dynamic Graph Attention Networks ( DIDA ) to handle spatio-temporal distribution shift in dynamic graphs.

Original: First, we propose a disentangled dynamic graph attention network to capture invariant and variant spatio-temporal patterns.
Revised: First, we propose a disentangled dynamic graph attention network to capture invariant and variant spatio-temporal patterns.

Original: Then, based on the causal inference literature, we design a spatio-temporal intervention mechanism to create multiple intervened distributions and propose an invariance regularization term to help the model focus on invariant patterns under distribution shifts.
Revised: Then, based on the causal inference literature, we design a spatio-temporal intervention mechanism to create multiple intervened distributions and propose an invariance regularization term to help the model focus on invariant patterns under distribution shifts.

Original: Extensive experiments on three real-world datasets and one synthetic dataset demonstrate that our method can better handle spatio-temporal distribution shift than state-of-the-art baselines.
Revised: Extensive experiments on three real-world datasets and one synthetic dataset demonstrate that our method can better handle spatio-temporal distribution shift than state-of-the-art baselines.

Original: One limitation is that in this paper we mainly consider dynamic graphs in scenarios of discrete snapshots, and we leave studying spatio-temporal distribution shifts in continous dynamic graphs for further explorations.
Revised: One limitation is that in this paper we mainly consider dynamic graphs in scenarios of discrete snapshots, and we leave studying spatio-temporal distribution shifts in continous dynamic graphs for further explorations.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported in part by the National Key Research and Development Program of China
Revised: This work was supported in part by the National Key Research and Development Program of China

Original: No. 2020AAA0106300, National Natural Science Foundation of China (No. 62250008, 62222209, 62102222, 62206149), China National Postdoctoral Program for Innovative Talents No. BX and China Postdoctoral Science Foundation No. 2022M711813.
Revised: No. 2020AAA0106300, National Natural Science Foundation of China (No. 62250008, 62222209, 62102222, 62206149), China National Postdoctoral Program for Innovative Talents No. BX and China Postdoctoral Science Foundation No. 2022M711813.

Original: All opinions, findings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
Revised: All opinions, findings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e
Revised: [Yes] (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/cu- rating?
Revised: and how consent was obtained from people whose data you’re using/cu- rating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Notations Notations
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Descriptions

Original: G =

Original: ( V , E )

Original: A graph with the node

Original: [Table]

Original: and edge set

Original: G t =

Original: ( V t , E t )

Original: Graph slice

Original: [Table]

Original: t

Original: X t , A t

Original: Features and adjacency matrix of a graph at time t

Original: [Table]

Original: : t , Y t ,

Original: [Table]

Original: 1: t , Y t

Original: Graph trajectory, label and their corresponding random variable

Original: [Table]

Original: t

Original: [Table]

Original: 1: t v , y t

Original: Ego

Original: [Table]

Original: [Table]

Original: Function

Original: [Table]

Original: d

Original: The

Original: [Table]

Original: Query,

Original: [Table]

Original: N t ( u )

Original: Dynamic neighborhood of node u at time t m

Original: I , m V , m f

Original: Structural mask of invariant and variant patterns, and featural mask z tI ( u ) , z tV ( u )

Original: Summarizations of invariant and variant patterns for node u at time t

Original: Agg

Original: I ( · ) , Agg V ( · )

Original: Aggregation functions for invariant and variant patterns h t u

Original: Hidden embeddings for node u at time t

Original: [Table]
Revised: [N/A]

Original: B More Details on Section 3.

Original: Background of Assumption 1.

Original: It is widely adopted in out-of-distribution generalization literature [25, 48, 94, 29, 95, 96, 97] about the assumption that the relationship between labels and some parts of features is invariant across data distributions, and these subsets of features with such properties are called invariant features.

Original: In this paper, we use invariant patterns P I to denote the invariant structures and features.

Original: From the causal perspective, we can formulate the data-generating process in dynamic graphs with a structural causal model (SCM) [16, 17], P

Original: V → G ← P I → y and P V ← P

Original: I , where the arrow between variables denotes casual relationship, and the subscript v and superscript t are omitted for brevity.

Original: P V → G ← P I denotes that variant and invariant patterns construct the ego-graph trajectories observed in the data, while P I → y denotes that invariant patterns determine the ground truth label y , no matter how the variant patterns change inside data across different distributions.

Original: Sometimes, the correlations between variant patterns and labels may be built by some exogenous factors like periods and communities.

Original: In some distributions, P V ← P I would open a backdoor path [17] P V ← P I →

Original: y so that variant patterns P V and labels y are correlated statistically, and this correlation is also called spurious correlation.

Original: If the model highly relies on the relationship between variant patterns and labels, it will fail under distribution shift, since such relationship varies across distributions.

Original: Hence, we propose to help the model focus on invariant patterns to make predictions and thus handle distribution shift.

Original: Connnections in Remark 1.

Original: To eliminate the spurious correlation between variant patterns and labels, one way is to block the backdoor path by using do-calculus to intervene variant patterns.

Original: By applying do-calculus on one variable, all in-coming arrows(causal relationship) to it will be removed [17] and the intervened distributions will be created.

Original: In our case, the operator do( P V ) will cut the causal relationship from invariant patterns to variant patterns, i.e. disabling P V ← P

Original: I

Original: and then blocking the backdoor path P V ← P I → y .

Original: Hence, the model can learn the direct causal effects from invariant patterns to labels in the intervened distributions p ( y , G | do ( P V )) , and the risks should be the same across these intervened distributions.

Original: Therefore we can minimize the variance of empirical risks under different intervened distributions to help the model focus on the relationship between invariant patterns and labels.

Original: On the other hand, if we have the optimal predictor f ∗ θ 1 and pattern finder ϕ ∗ θ 2 according to Eq.(3), then the variance term in Eq.(4) is minimized as the variant patterns will not affect the predictions of f ∗ θ 1 ◦ ϕ ∗ θ 2 across different intervened distributions.

Original: C Additional Experiments C.1 Distribution Shifts in Real-world Datasets

Original: We illustrate the distribution shifts in the real-world datasets with two statistics, number of links and average neighbor degrees [98].

Original: Figure 3 shows that the average neighbor degrees are lower in test data compared to training data.

Original: Lower average neighbor degree indicates that the nodes have less affinity to connect with high-degree neighbors.

Original: Moreover, in COLLAB, the test data has less history than training data, i.e. the graph trajectory is not always complete in training and test data distribution.

Original: This phenomenon of incomplete history is

Original: [Figure]

Original: A

Original: [Figure]

Original: a g e N e i g

Original: [Figure]

Original: o r D

Original: [Figure]

Original: [Figure]

Original: t

Original: i

Original: [Figure]

Original: common in real-world scenarios, e.g. not all the users join the social platforms at the same time.

Original: Figure 4 shows that the number of links and its trend also differ in training and test data.

Original: In COLLAB, #links of test data has a slower rising trend than training data.

Original: In Yelp, #links of training and test data both have a drop during time 13-15 and rise again thereafter, due to the outbreak of COVID-19, which strongly affected the consumers’ behavior.

Original: C.2 Spatial or Temporal Intervention

Original: We compare two other versions of DIDA , where DIDA-S only uses spatial intervention and DIDA-T only uses temporal intervention.

Original: For DIDA-S , we put the constraint that the variant patterns used to intervene must come from the same timestamp in Eq.(9) so that the variant patterns across time are forbidden for intervention.

Original: Similarly, we put the constraint that the variant patterns used to intervene must come from the same node in

Original: [Figure]

Original: [Figure]

Original: [-A U C ( % ) (a) COLLAB 10 4 10 3 10 2 1010 
-]
Revised: [-A U C ( % ) (a) COLLAB 10 4 10 3 10 2 1010 
-]

Original: [-A U C ( % ) (b) Yelp 
-]
Revised: [-A U C ( % ) (b) Yelp 
-]

Original: [-A U C ( % ) (c) Transaction 10 6 10 5 10 4 10 3 10 2 10 1 10 
-]
Revised: [-A U C ( % ) (c) Transaction 10 6 10 5 10 4 10 3 10 2 10 1 10 
-]

Original: A U C ( % ) (d) Synthetic

Original: Eq.(9) for DIDA-T .

Original: Figure 5 shows that DIDA improves significantly over the other two ablated versions, which verifies that it is important to take into consideration both the spatial and temporal aspects of distribution shifts.

Original: C.3 Efficiency of Intervention

Original: For DIDA and DIDA without intervention mechanism, we compare their training time for each epoch on COLLAB dataset.

Original: As shown in Figure 6, the intervention mechanism adds few costs in training time (lower than 5%).

Original: Moreover, as DIDA does not use the intervention mechanism in the test stage, it does not add extra computational costs in the inference time.

Original: C.4 Hyperparameter Sensitivity

Original: We analyze the sensitivity of hyperparameter λ in DIDA for each dataset.

Original: From Figure 7, we can see that as λ is too small or too large, the model’s performance drops in most datasets.

Original: It shows that λ acts as a balance between how DIDA exploits the patterns and satisfies the invariance constraint.

Original: C.5 Case Study

Original: [Figure]

Original: C

Original: C

Original: D

Original: [Figure]

Original: ’s

Original: attention scores for invariant patterns, which are shown in percentage and marked red.

Original: Nodes and links represent users and transactions respectively.

Original: Links with smaller attention scores are omitted for brevity.

Original: For each link, the direction denotes a selling behavior, and it is tagged with the trading goods and transaction time on the link.

Original: For example, the link from B to A with tagging ‘Day 12’ and ‘T3P2’ represents that user B sells user A three T-shirts and two pants on day 12.

Original: The trading goods have four types: T, P, H, O represent T-shirt, Pants, Hoddie, and

Original: Outerwear respectively.

Original: Based on the transactions in their dynamic neighborhood, DIDA predicts whether user A will buy something from user B on day 17, and the dashed bounding box ‘T1P1’ refers to ground-truth trading goods.

Original: Figure 8 illustrates DIDA ’s attention scores for invariant patterns.

Original: We use a well-trained DIDA on Transaction dataset and show the scores of the structural mask for invariant patterns, i.e. m I in Eq.(6).

Original: In this case, DIDA predicts whether user A will buy something (‘T1P1’) from user B on day 17, based on the transactions in their dynamic neighborhood.

Original: We have the following observations:

Original: [Table]

Original: [Table]

Original: • Transactions with higher attention scores are more correlated with the transaction to predict.

Original: The transactions with attention scores 28% and 21% include goods ‘T3P2’ and ‘T2’, which are closed to the ground-truth trading goods ‘T1P1’ from B to A on day 17.

Original: On the right side, similarly, the transactions with attention scores 17%, 14% and 11% include goods ‘T3’, ‘T1P3’ and ‘T1’, which are closed to ‘T1P1’ as well.

Original: In contrast, the transactions with other unrelated goods like ‘H1’ and ‘O1’ have even smaller attention scores.

Original: • Dynamic information is critical in the attention scores for invariant patterns.

Original: For the transactions with attention scores 21% and 8% on the left side, they include goods ‘T2’ and ‘P2’ which are both similar to ‘T1P1’.

Original: However, the latter’s attention score is much lower than the former’s.

Original: This is because the latter happens much earlier than the former, and DIDA learns to attend to more recent transactions to capture users’ recent interest.

Original: These observations indicate that DIDA can summarize invariant patterns in dynamic neighborhood to capture the users’ interests in trading (user A as

Original: a

Original: recent T-shirt

Original: /Pants

Original: buyer and user B as a recent T-shirt/Pants seller and make predictions by matching the summarized interests.

Original: D Reproducibility Details D.1 Training & Evaluation

Original: Hyperparameters.

Original: For all methods, the hidden dimension is set to 16, the number of layers is set to 2.

Original: We adopt the Adam optimizer [99] with a learning rate 0 .

Original: 01 , weight decay 5e-7 and set the patience of early stopping on the validation set as 50.

Original: For other hyperparameters, we adopt grid search for the best parameters using the validation split.

Original: For DIDA , we set the number of intervention samples as 1000 for all datasets, and λ as 1e-2,1e-2,1e-4,1e-1 for COLLAB, Yelp, Transaction, and Synthetic dataset respectively.

Original: The training pipeline for DIDA is shown in Algo.

Original: Evaluation.

Original: We randomly sample negative samples from nodes that do not have links, and the negative samples for validation and testing set are kept the same for all comparing methods.

Original: The number of negative samples is the same as positive ones.

Original: We use Area under the ROC Curve (AUC) as the evaluation metric.

Original: We use the inner product of the two learned node representations to predict links and use cross-entropy as the loss function ℓ .

Original: We randomly run the experiments three times, and report the average results and standard deviations.

Original: D.2 Dataset Details

Original: We summarize dataset statistics in Table 3 and describe dataset details as follows.

Original: COLLAB.

Original: [51] 7 We use word2vec [100] to extract 32-dimensional feature from paper abstracts and average to obtain author features.

Original: We use 10,1,5 chronological graph slices for training, validation and test respectively.

Original: The dataset includes 23035 nodes and 151790 links in total.

Original: Yelp.

Original: [43] 8 We use word2vec [100] to extract 32-dimensional feature from reviews and average to obtain user and business features.

Original: We select users and items with interactions of more than 10.

Original: We use 15,1,8 chronological graph slices for training, validation and test respectively.

Original: The dataset includes 13095 nodes and 65375 links in total.

Original: Transaction.

Original: 9 We calculate the distribution of users’ historical transaction categories as the initial 15-dimensional features.

Original: We use 20,2,8 chronological graph slices for training, validation and test respectively.

Original: The dataset includes 29526 nodes and 53448 links in total.

Original: Synthetic.

Original: We use the same features as X t 1 and structures as A t in COLLAB, and introduce features X t 2 with variable correlation with supervision signals.

Original: X t 2 are obtained by training the embeddings X 2 ∈ R N × d with reconstruction loss ℓ ( X 2 X T 2 , ˜

Original: A t +1 ) , where

Original: ˜

Original: A t +1 refers to the sampled links, and ℓ refers to cross-entropy loss function.

Original: The embeddings X t 2 are trained with Adam optimizer, learning rate 1e-1, weight decay 1e-5 and earlystop patience 50.

Original: In this way, we empirically find that the inner product predictor can achieve results of over 99% AUC by using X t 2 to predict the sampled links ˜

Original: A t +1 , so that the generated features can have strong correlations with the sampled links.

Original: By controlling the p mentioned in the Section 4.2, we can control the correlations of X t and labels A t +1 to vary in training and test stage.

Original: D.3 Baseline Details

Original: Backbones.

Original: This class of methods aim at improving the modeling ability for dynamic graphs.

Original: • GAE.

Original: [44] A representative static GNN with stacking of graph convolutions.

Original: • VGAE.

Original: [44] A representative static GNN which introduces variational variables into GAE.

Original: • GCRN.

Original: [45] A representative dynamic GNN that first adopts a GCN to obtain node embeddings and then a GRU to model the dynamics.

Original: • DySAT.

Original: [43] A representative dynamic GNN that models dynamic graph using structural and temporal self-attention.

Original: • EvolveGCN.

Original: [13] A representative dynamic GNN that uses an RNN to evolve the GCN parameters instead of directly learning the temporal node embeddings.

Original: OOD Generalization

Original: methods

Original: This class

Original: of methods

Original: aim at

Original: improving the robustness and generalization ability of models against distribution shift.

Original: For fair comparison, we

Original: randomly

Original: split

Original: the

Original: samples

Original: into different

Original: domains

Original: as the field information is unknown

Original: to all

Original: methods.

Original: Since they are

Original: general

Original: OOD

Original: generalization methods

Original: and are not specifically designed for dynamic graphs, we adopt DySAT as their backbone, which is the best-performed DyGNN on training dataset.

Original: • IRM [48] aims at learning an invariant predictor which minimizes the empirical risks for all training domains.

Original: • VREx [50] reduces differences in risk across training domains to reduce the model’s sensitivity to distribu- tional shifts.

Original: • GroupDRO [49] puts more weight on training domains with larger errors when minimizing empirical risk.

Original: D.4 Details of DIDA

Original: First Layer.

Original: Before stacking of disentangled spatio-temporal graph attention Layers, we use a fully-connected layer FC ( · ) to transform the features into hidden embeddings.

Original: [Equation]

Original: Agg Functions.

Original: We implement the aggregation function for invariant and variant patterns as

Original: [Equation]

Original: FFN.

Original: The FFN includes a layer normalization [101], multi-layer perceptron and skip connection.

Original: [Equation]

Original: where α is a learnable parameter.

Original: Predictor.

Original: For link prediction task, we implement the predictor f ( · ) in Eq.(10) as inner product of hidden embeddings, i.e. f ( z tI ( u ) , z tI ( v )) =

Original: z tI ( u ) · ( z tI ( v )) T , which is conformed to classic link prediction settings.

Original: To implement the predictor g ( · ) in Eq.(11), we adopt the biased training technique following [102], i.e. g ( z tV ( u ) , z tI ( u ) , z tV ( v ) , z tI ( v ))

Original: =

Original: f ( z tI ( u ) , z tI ( v )) · σ ( f ( z tV ( u ) , z tV ( v ))) .

Original: D.5 Configurations

Original: Experiments on COLLAB, Yelp and Synthetic datasets are conducted with:

Original: • Operating System:

Original: Ubuntu 18.04.1 LTS • CPU:

Original: Intel(R)

Original: Xeon(R)

Original: Gold 6240R CPU @

Original: 2.40GHz • GPU: NVIDIA GeForce RTX 3090 with 24 GB of memory • Software: Python 3.8.13, Cuda 11.3, PyTorch [103] 1.11.0, PyTorch Geometric [104] 2.0.3.

Original: Experiments on Transaction dataset are conducted with:

Original: • Operating System: Ubuntu 18.04.5 LTS • CPU:

Original: Intel(R)

Original: Xeon(R)

Original: Platinum 8163 CPU @

Original: 2.50GHz • GPU: NVIDIA Tesla V100 with 16 GB of memory • Software:

Original: Python 3.6.12, Cuda 10.1, PyTorch [103] 1.8.2, PyTorch Geometric [104] 2.0.3.