Original: D Y G2V EC : R EPRESENTATION L EARNING FOR D YNAMIC G RAPHS WITH S ELF -S UPERVISION
Revised: D Y G2V EC : R EPRESENTATION L EARNING FOR D YNAMIC G RAPHS WITH S ELF -S UPERVISION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The challenge in learning from dynamic graphs for predictive tasks lies in extracting fine-grained temporal motifs from an ever-evolving graph.
Revised: The challenge in learning from dynamic graphs for predictive tasks lies in extracting fine-grained temporal motifs from an ever-evolving graph.

Original: Moreover, task labels are often scarce, costly to obtain, and highly imbalanced for large dynamic graphs.
Revised: Moreover, task labels are often scarce, costly to obtain, and highly imbalanced for large dynamic graphs.

Original: Recent advances in self-supervised learning on graphs demonstrate great potential, but focus on static graphs.
Revised: Recent advances in self-supervised learning on graphs demonstrate great potential, but focus on static graphs.

Original: State-of-the-art (SoTA) models for dynamic graphs are not only incompatible with the self-supervised learning (SSL) paradigm but also fail to forecast interactions beyond the very near future.
Revised: State-of-the-art (SoTA) models for dynamic graphs are not only incompatible with the self-supervised learning (SSL) paradigm but also fail to forecast interactions beyond the very near future.

Original: To address these limitations, we present DyG2Vec, an SSL-compatible, efficient model for representation learning on dynamic graphs.
Revised: To address these limitations, we present DyG2Vec, an SSL-compatible, efficient model for representation learning on dynamic graphs.

Original: DyG2Vec uses a window-based mechanism to generate task-agnostic node embeddings that can be used to forecast future interactions.
Revised: DyG2Vec uses a window-based mechanism to generate task-agnostic node embeddings that can be used to forecast future interactions.

Original: DyG2Vec significantly outperforms SoTA baselines on benchmark datasets for downstream tasks while only requiring a fraction of the training/inference time.
Revised: DyG2Vec significantly outperforms SoTA baselines on benchmark datasets for downstream tasks while only requiring a fraction of the training/inference time.

Original: We adapt two SSL evaluation mechanisms to make them applicable to dynamic graphs and thus show that SSL pre-training helps learn more robust temporal node representations, especially for scenarios with few labels.
Revised: We adapt two SSL evaluation mechanisms to make them applicable to dynamic graphs and thus show that SSL pre-training helps learn more robust temporal node representations, especially for scenarios with few labels.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) have recently found great success in representation learning for complex networks of interactions, as present in recommendation systems, transaction networks, and social media (Wu et al., 2020; Zhang et al., 2019; Qiu et al., 2018).
Revised: Graph Neural Networks (GNNs) have recently found great success in representation learning for complex networks of interactions, as present in recommendation systems, transaction networks, and social media (Wu et al., 2020; Zhang et al., 2019; Qiu et al., 2018).

Original: However, most approaches ignore the dynamic nature of graphs encountered in many real-world domains.
Revised: However, most approaches ignore the dynamic nature of graphs encountered in many real-world domains.

Original: Dynamic graphs model complex, time-evolving interactions between entities (Kazemi et al., 2020;
Revised: Dynamic graphs model complex, time-evolving interactions between entities (Kazemi et al., 2020;

Original: Skarding et al., 2021; Xue et al., 2022).
Revised: Skarding et al., 2021; Xue et al., 2022).

Original: Multiple works have revealed that real-world dynamic graphs possess finegrained temporal patterns known as temporal motifs (Toivonen et al., 2007; Paranjape et al., 2017).
Revised: Multiple works have revealed that real-world dynamic graphs possess finegrained temporal patterns known as temporal motifs (Toivonen et al., 2007; Paranjape et al., 2017).

Original: For example, a simple pattern in social networks specifies that two users who share many friends are likely to interact in the future.
Revised: For example, a simple pattern in social networks specifies that two users who share many friends are likely to interact in the future.

Original: A robust representation learning approach must be able to extract such temporal patterns from an ever-evolving dynamic graph in order to make accurate predictions.
Revised: A robust representation learning approach must be able to extract such temporal patterns from an ever-evolving dynamic graph in order to make accurate predictions.

Original: Self-Supervised Representation Learning (SSL) has shown promise in achieving competitive performance for different data modalities on multiple predictive tasks (Liu et al., 2021).
Revised: Self-Supervised Representation Learning (SSL) has shown promise in achieving competitive performance for different data modalities on multiple predictive tasks (Liu et al., 2021).

Original: Given a large corpus of unlabelled data, SSL postulates that unsupervised pre-training is sufficient to learn robust representations that are predictive for downstream tasks with minimal fine-tuning.
Revised: Given a large corpus of unlabelled data, SSL postulates that unsupervised pre-training is sufficient to learn robust representations that are predictive for downstream tasks with minimal fine-tuning.

Original: However, it is important to specify a pre-training objective function that induces good performance for the downstream tasks.
Revised: However, it is important to specify a pre-training objective function that induces good performance for the downstream tasks.

Original: Contrastive SSL methods, despite their early success, rely heavily on negative samples, extensive data augmentation, and large batch sizes (Jing et al., 2022; Garrido et al., 2022).
Revised: Contrastive SSL methods, despite their early success, rely heavily on negative samples, extensive data augmentation, and large batch sizes (Jing et al., 2022; Garrido et al., 2022).

Original: Non-contrastive methods address these shortcomings, incorporating information theoretic principles through architectural innovations or regularization methods.
Revised: Non-contrastive methods address these shortcomings, incorporating information theoretic principles through architectural innovations or regularization methods.

Original: These closely resemble strategies employed in manifold learning and spectral embedding methods (Balestriero & LeCun, 2022).
Revised: These closely resemble strategies employed in manifold learning and spectral embedding methods (Balestriero & LeCun, 2022).

Original: The success of such SSL methods on sequential data (Tong et al., 2022; Eldele et al., 2021; Patrick et al., 2021) suggests that one can learn rich temporal node embeddings from dynamic graphs without direct supervision.
Revised: The success of such SSL methods on sequential data (Tong et al., 2022; Eldele et al., 2021; Patrick et al., 2021) suggests that one can learn rich temporal node embeddings from dynamic graphs without direct supervision.

Original: SSL methods are attractive for dynamic graphs because it is often costly to generate ground truth labels.
Revised: SSL methods are attractive for dynamic graphs because it is often costly to generate ground truth labels.

Original: Contrastive approaches are very sensitive to the quality of the negative samples, and these are challenging to identify in dynamic graphs due to the temporal evolution of interactions and the lack of semantic labels at the contextual level.
Revised: Contrastive approaches are very sensitive to the quality of the negative samples, and these are challenging to identify in dynamic graphs due to the temporal evolution of interactions and the lack of semantic labels at the contextual level.

Original: As a result, it is desirable to explore non-contrastive techniques, but state-of-the-art models for dynamic graphs suffer from shortcomings that make them hard to adapt to SSL paradigms.
Revised: As a result, it is desirable to explore non-contrastive techniques, but state-of-the-art models for dynamic graphs suffer from shortcomings that make them hard to adapt to SSL paradigms.

Original: First, they heavily rely on chronological training or a full history of interactions to construct predictions (Kumar et al., 2019; Xu et al., 2020; Rossi et al., 2020; Wang et al., 2021).
Revised: First, they heavily rely on chronological training or a full history of interactions to construct predictions (Kumar et al., 2019; Xu et al., 2020; Rossi et al., 2020; Wang et al., 2021b).

Original: Second, the encoding modules either use inefficient message-passing procedures (Xu et al., 2020), memory blocks (Kumar et al., 2019; Rossi et al., 2020), or expensive random walkbased algorithms (Wang et al., 2021).
Revised: Second, the encoding modules either use inefficient message-passing procedures (Xu et al., 2020), memory blocks (Kumar et al., 2019; Rossi et al., 2020), or expensive random walkbased algorithms (Wang et al., 2021b) that are designed for edge-level tasks only.

Original: As a result, while SSL pre-training has been applied successfully for static graphs (Thakoor et al., 2022; Hassani & Khasahmadi, 2020; You et al., 2022), there has been limited success in adapting SSL pre-training to dynamic graphs.
Revised: As a result, while SSL pre-training has been applied successfully for static graphs (Thakoor et al., 2022; Hassani & Khasahmadi, 2020; You et al., 2022), there has been limited success in adapting SSL pre-training to dynamic graphs.

Original: In this work, we propose DyG2Vec 1 , a novel encoder-decoder model for continuous-time dynamic graphs that benefits from a window-based architecture that acts a regularizer to avoid over-fitting.
Revised: In this work, we propose DyG2Vec 1 , a novel encoder-decoder model for continuous-time dynamic graphs that benefits from a window-based architecture that acts a regularizer to avoid over-fitting.

Original: DyG2Vec is an efficient attention-based graph neural network that performs message-passing across structure and time to output task-agnostic node embeddings.
Revised: DyG2Vec is an efficient attention-based graph neural network that performs message-passing across structure and time to output task-agnostic node embeddings.

Original: Experimental results for 7 benchmark datasets indicate that DyG2Vec outperforms SoTA baselines on future link prediction and dynamic node classification in terms of performance and speed, particularly in medium- and long-range forecasting.
Revised: Experimental results for 7 benchmark datasets indicate that DyG2Vec outperforms SoTA baselines on future link prediction and dynamic node classification in terms of performance and speed, particularly in medium- and long-range forecasting.

Original: The novelty of our model lies in its compatibility with SoTA SSL approaches.
Revised: The novelty of our model lies in its compatibility with SoTA SSL approaches.

Original: That is, we propose a joint-embedding architecture for DyG2Vec that can benefit from non-contrastive SSL.
Revised: That is, we propose a joint-embedding architecture for DyG2Vec that can benefit from non-contrastive SSL.

Original: We adapt two evaluation protocols (linear and semi-supervised probing) to the dynamic graph setting and demonstrate that the proposed SSL pre-training is effective in the low-label regime.
Revised: We adapt two evaluation protocols (linear and semi-supervised probing) to the dynamic graph setting and demonstrate that the proposed SSL pre-training is effective in the low-label regime.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-supervised representation learning: Multiple works explore learning visual representations without labels (see (Liu et al., 2021) for a survey).
Revised: Self-supervised representation learning: Multiple works explore learning visual representations without labels (see (Liu et al., 2021) for a survey).

Original: The more recent contrastive methods generate random views of images through data augmentations, and then force representations of positive pairs to be similar while pushing apart representations of negative pairs (Chen et al., 2020a;
Revised: The more recent contrastive methods generate random views of images through data augmentations, and then force representations of positive pairs to be similar while pushing apart representations of negative pairs (Chen et al., 2020a;

Original: He et al., 2019).
Revised: He et al., 2019).

Original: With the goal of attaining hard negative samples, such methods typically use large batch sizes Chen et al.
Revised: With the goal of attaining hard negative samples, such methods typically use large batch sizes Chen et al.

Original: (2020a) or memory banks (
Revised: (2020a) or memory banks (

Original: He et al., 2019; Chen et al., 2020b).
Revised: He et al., 2019; Chen et al., 2020b).

Original: Non-contrastive methods such as BYOL (Grill et al., 2020) and VICReg (Bardes et al., 2022) eliminate the need for negative samples through various techniques that avoid representation collapse (Jing et al., 2022).
Revised: Non-contrastive methods such as BYOL (Grill et al., 2020) and VICReg (Bardes et al., 2022) eliminate the need for negative samples through various techniques that avoid representation collapse (Jing et al., 2022).

Original: Recently, several SSL methods have been adapted to pre-train GNNs (Xie et al., 2022).
Revised: Recently, several SSL methods have been adapted to pre-train GNNs (Xie et al., 2022).

Original: Deep Graph
Revised: Deep Graph

Original: Infomax (DGI) (Velickovic et al., 2019) and InfoGCL (Xu et al., 2021)) rely on mutual information maximization or information bottle-necking between patch-level and graph-level summaries.
Revised: Infomax (DGI) (Velickovic et al., 2019) and InfoGCL (Xu et al., 2021)) rely on mutual information maximization or information bottle-necking between patch-level and graph-level summaries.

Original: MVGRL (Hassani & Khasahmadi, 2020) incorporates multiple views, and BGRL (Thakoor et al., 2022) adapts BYOL to graphs to eliminate the need for negative samples, which are often memory-heavy in the graph setting.
Revised: MVGRL (Hassani & Khasahmadi, 2020) incorporates multiple views, and BGRL (Thakoor et al., 2022) adapts BYOL to graphs to eliminate the need for negative samples, which are often memory-heavy in the graph setting.

Original: The experiments demonstrate the high degree of scalability of non-contrastive methods and their effectiveness in leveraging both labeled and unlabeled data.
Revised: The experiments demonstrate the high degree of scalability of non-contrastive methods and their effectiveness in leveraging both labeled and unlabeled data.

Original: Representation learning for dynamic graphs: Early works on representation learning for continuous-time dynamic graphs typically divide the graph into snapshots that are encoded by a static GNN and then processed by an RNN module (Sankar et al., 2020;
Revised: Representation learning for dynamic graphs: Early works on representation learning for continuous-time dynamic graphs typically divide the graph into snapshots that are encoded by a static GNN and then processed by an RNN module (Sankar et al., 2020;

Original: Pareja et al., 2020;
Revised: Pareja et al., 2020;

Original: Kazemi et al., 2020).
Revised: Kazemi et al., 2020).

Original: Such methods fail to learn fine-grained temporal patterns at smaller timescales within each snapshot.
Revised: Such methods fail to learn fine-grained temporal patterns at smaller timescales within each snapshot.

Original: Therefore, several RNN-based methods were introduced that sequentially update node embeddings as new edges arrive.
Revised: Therefore, several RNN-based methods were introduced that sequentially update node embeddings as new edges arrive.

Original: JODIE (Kumar et al., 2019) employs two RNN modules to update the source and destination embeddings respectively of a arriving edge.
Revised: JODIE (Kumar et al., 2019) employs two RNN modules to update the source and destination embeddings respectively of a arriving edge.

Original: DyRep (Trivedi et al., 2019) adds a temporal attention layer to take into account multi-hop interactions when updating node embeddings.
Revised: DyRep (Trivedi et al., 2019) adds a temporal attention layer to take into account multi-hop interactions when updating node embeddings.

Original: TGAT (Xu et al., 2020) includes an attention-based message passing (AMP) architecture to aggregate messages from a historical neighborhood.
Revised: TGAT (Xu et al., 2020) includes an attention-based message passing (AMP) architecture to aggregate messages from a historical neighborhood.

Original: TGN (Rossi et al., 2020) alleviates the expensive neighborhood aggregation of TGAT by using an RNN memory module to encode the history of each node.
Revised: TGN (Rossi et al., 2020) alleviates the expensive neighborhood aggregation of TGAT by using an RNN memory module to encode the history of each node.

Original: CaW (Wang et al., 2021) extracts temporal patterns through an expensive procedure that samples temporal random walks and encodes them with an LSTM.
Revised: CaW (Wang et al., 2021b) extracts temporal patterns through an expensive procedure that samples temporal random walks and encodes them with an LSTM.

Original: This procedure must be performed for every prediction.
Revised: This procedure must be performed for every prediction.

Original: In contrast to prior works, our method operates on a fixed window of history to generate node embeddings.
Revised: In contrast to prior works, our method operates on a fixed window of history to generate node embeddings.

Original: Additionally, we do not recompute embeddings for every prediction, which allows for efficient computation and memory usage.
Revised: Additionally, we do not recompute embeddings for every prediction, which allows for efficient computation and memory usage.

Original:  P ROBLEM FORMULATION
Revised:  P ROBLEM FORMULATION

Original: A Continuous-Time Dynamic Graph (CTDG) G =
Revised: A Continuous-Time Dynamic Graph (CTDG) G =

Original: ( V , E , X ) is a sequence of E = |E| interactions, where X =
Revised: ( V , E , X ) is a sequence of E = |E| interactions, where X =

Original: ( X V , X E ) is the set of input features containing the node features X V ∈ R N × D V and the edge features X E ∈ R E × D E .
Revised: ( X V , X E ) is the set of input features containing the node features X V ∈ R N × D V and the edge features X E ∈ R E × D E .

Original: E
Revised: E

Original: = { e 1 , e 2 , . .
Revised: = { e 1 , e 2 , . .

Original: , e E } is the set of interactions.
Revised: , e E } is the set of interactions.

Original: There are N = |V| nodes, and D V and D E are the dimensions of the node and edge feature vectors, respectively.
Revised: There are N = |V| nodes, and D V and D E are the dimensions of the node and edge feature vectors, respectively.

Original: An edge e i =
Revised: An edge e i =

Original: ( u i , v i , t i , m i ) is an interaction between any two nodes u i , v i ∈ V , with t i ∈ R being a continuous timestamp, and m i ∈
Revised: ( u i , v i , t i , m i ) is an interaction between any two nodes u i , v i ∈ V , with t i ∈ R being a continuous timestamp, and m i ∈

Original: X E an edge feature vector.
Revised: X E an edge feature vector.

Original: For simplicity, we assume that the edges are undirected and ordered by time (i.e., t i ≤ t i +1 ) .
Revised: For simplicity, we assume that the edges are undirected and ordered by time (i.e., t i ≤ t i +1 ) .

Original: A temporal sub-graph G i,j is defined as a set of all the edges in the interval [ t i , t j ] , such that E ij = {
Revised: A temporal sub-graph G i,j is defined as a set of all the edges in the interval [ t i , t j ] , such that E ij = {

Original: e k | t i ≤ t k ≤ t j } .
Revised: e k | t i ≤ t k ≤ t j } .

Original: Any two nodes can interact multiple times throughout the time horizon; therefore, G is a multi-graph.
Revised: Any two nodes can interact multiple times throughout the time horizon; therefore, G is a multi-graph.

Original: Our goal is to learn a model f that maps the input graph to a representation space.
Revised: Our goal is to learn a model f that maps the input graph to a representation space.

Original: The model is a pre-trainable encoder-decoder architecture, f = ( g θ , d γ ) .
Revised: The model is a pre-trainable encoder-decoder architecture, f = ( g θ , d γ ) .

Original: The encoder g θ maps a dynamic graph to node embeddings H ∈ R N × D H ; the decoder d γ performs a task-specific prediction given the embeddings.
Revised: The encoder g θ maps a dynamic graph to node embeddings H ∈ R N × D H ; the decoder d γ performs a task-specific prediction given the embeddings.

Original: The model is parameterized by the encoder/decoder parameters ( θ, γ ) .
Revised: The model is parameterized by the encoder/decoder parameters ( θ, γ ) .

Original: More concretely,
Revised: More concretely,

Original: H =
Revised: [Equation] =

Original: [Equation]
Revised: [Equation]

Original: θ ( G) ,
Revised: ( G

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where Z ∈ R N × D Y is the prediction of task-specific labels (e.g., edge prediction or source node classification labels) of all edges in ¯ E .
Revised: where Z ∈ R N × D Y is the prediction of task-specific labels (e.g., edge prediction or source node classification labels) of all edges in ¯ E .

Original: The node embeddings H must capture the temporal and structural dynamics of each node such that the future can be accurately predicted from the past, e.g., future edge prediction given past edges.
Revised: The node embeddings H must capture the temporal and structural dynamics of each node such that the future can be accurately predicted from the past, e.g., future edge prediction given past edges.

Original: The main distinction of this design is that, unlike previous dynamic graph models (Rossi et al., 2020; Xu et al., 2020; Wang et al., 2021), the encoder must produce embeddings independent of the downstream task specifications.
Revised: The main distinction of this design is that, unlike previous dynamic graph models (Rossi et al., 2020; Xu et al., 2020; Wang et al., 2021b), the encoder must produce embeddings independent of the downstream task specifications.

Original: This special trait can allow the model to be compatible with the SSL paradigm where an encoder is pre-trained separately and then fine-tuned together with a task-specific decoder to predict the labels.
Revised: This special trait can allow the model to be compatible with the SSL paradigm where an encoder is pre-trained separately and then fine-tuned together with a task-specific decoder to predict the labels.

Original: To this end, we present a novel DyG2Vec framework, that can learn rich node embeddings at any timestamp t independent of the downstream task.
Revised: To this end, we present a novel DyG2Vec framework, that can learn rich node embeddings at any timestamp t independent of the downstream task.

Original: DyG2Vec is formulated as a two-stage framework.
Revised: DyG2Vec is formulated as a two-stage framework.

Original: In the first stage, we use a non-contrastive SSL method to learn the model f SSL =
Revised: In the first stage, we use a non-contrastive SSL method to learn the model f SSL =

Original: ( g θ , d ψ ) over various sampled dynamic sub-graphs with self-supervision.
Revised: ( g θ , d ψ ) over various sampled dynamic sub-graphs with self-supervision.

Original: d ψ is an SSL decoder that is only used in the SSL pre-training stage.
Revised: d ψ is an SSL decoder that is only used in the SSL pre-training stage.

Original: In the second stage, a task-specific decoder d γ is trained on top of the pre-trained encoder g θ to compute the outputs for the downstream tasks, e.g., future edge prediction or dynamic node classification (Xu et al., 2020; Wang et al., 2021).
Revised: In the second stage, a task-specific decoder d γ is trained on top of the pre-trained encoder g θ to compute the outputs for the downstream tasks, e.g., future edge prediction or dynamic node classification (Xu et al., 2020; Wang et al., 2021b).

Original: We consider two example downstream tasks: future link prediction (FLP), and dynamic node classification (DNC).
Revised: We consider two example downstream tasks: future link prediction (FLP), and dynamic node classification (DNC).

Original: In each case, there is a prediction horizon of the next K interactions.
Revised: In each case, there is a prediction horizon of the next K interactions.

Original: The test window for FLP starting at time t i is ¯ E = { ( u j , v j , t j , m j ) | j ∈
Revised: The test window for FLP starting at time t i is ¯ E = { ( u j , v j , t j , m j ) | j ∈

Original: [ i, i + K ] } .
Revised: [ i, i + K ] } .

Original: This is augmented by a set of K negative edges.
Revised: This is augmented by a set of K negative edges.

Original: Each negative edge ( u j , v ′ j , t j , m j ) differs from its corresponding positive edge only in the destination node, v ′ j ̸ = v j , which is selected at random from all nodes.
Revised: Each negative edge ( u j , v ′ j , t j , m j ) differs from its corresponding positive edge only in the destination node, v ′ j ̸ = v j , which is selected at random from all nodes.

Original: The FLP task is then binary classification for the test set of 2 K edges.
Revised: The FLP task is then binary classification for the test set of 2 K edges.

Original: In the DNC task, a dynamic label is associated with each node that participates in an interaction.
Revised: In the DNC task, a dynamic label is associated with each node that participates in an interaction.

Original: We are provided with { ( u j , t j ) | j ∈
Revised: We are provided with { ( u j , t j ) | j ∈

Original: [ i, i + K ] } , i.e., the source node and interaction time.
Revised: [ i, i + K ] } , i.e., the source node and interaction time.

Original: The goal is to predict the source node labels for the next K interactions.
Revised: The goal is to predict the source node labels for the next K interactions.

Original: The performance metrics are detailed in Appendix A.4.
Revised: The performance metrics are detailed in Appendix A.4.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We now introduce our novel dynamic graph learning framework DyG2Vec, which can achieve downstream task-agnostic representation.
Revised: We now introduce our novel dynamic graph learning framework DyG2Vec, which can achieve downstream task-agnostic representation.

Original: We first present the SSL pre-training approach with a non-contrastive loss function for dynamic graphs.
Revised: We first present the SSL pre-training approach with a non-contrastive loss function for dynamic graphs.

Original: We then introduce the novel window-based downstream training approach.
Revised: We then introduce the novel window-based downstream training approach.

Original: Finally, we outline the encoder architecture.
Revised: Finally, we outline the encoder architecture.

Original: Given the full dynamic graph G 0 ,E , a set of intervals I is generated by dividing the entire time-span { t 0 , t E } into M =  E/S  intervals with stride S and window length W :
Revised: Given the full dynamic graph G 0 ,E , a set of intervals I is generated by dividing the entire time-span { t 0 , t E } into M = ⌈ E/S ⌉ − 1 intervals with stride S and interval length W (See Appendix A.4 for details).

Original: [Equation]
Revised: Let B ⊂ I be the mini-batch of intervals.

Original: Let B ⊂ I be the mini-batch of intervals selected by the sub-graph sampler m ( G , B ; W ) to construct the mini-batch of input graphs: ˆ G = {G i − W,i | [ i − W, i ) ∈ B } .
Revised: Given B , the sub-graph sampler m ( G , B ; W ) constructs the mini-batch of input graphs: ˆ G = {G i,j | [ i, j ) ∈ B } . The corresponding mini-batch of target graphs is denoted by ¯ G = {G j,j + K | [ i, j ) ∈ B } .

Original: The corresponding mini-batch of 
Revised: In principle, G i,j ∈ ˆ G is an input (history) graph used to predict the target labels of the corresponding target (future) graph G j,j + K ∈ ¯ G . The parameter W controls the size of the history while K controls how far the model is predicting into

Original: 𝒢 = (𝒱,ℰ,𝑋)
Revised: 𝒢 = (𝒱,ℰ,𝑋)

Original: Dynamic Graph
Revised: Dynamic Graph

Original: Temporal Subgraph
Revised: Temporal Subgraph

Original: Sampler
Revised: Sampler

Original: Mini-Batch of sub-graphs
Revised: Mini-Batch of sub-graphs

Original: Views 𝑡′ ∼ 𝒯 𝑡′′ ∼ 𝒯 𝑔 𝜃 𝑔 𝜃
Revised: Views 𝑡′ ∼ 𝒯 𝑡′′ ∼ 𝒯 𝑔 𝜃 𝑔 𝜃

Original: Embedding 𝑯′ 𝑯′′ 𝑝 𝜙 𝑝 𝜙 𝒁′ 𝒁′′
Revised: Embedding 𝑯′ 𝑯′′ 𝑝 𝜙 𝑝 𝜙 𝒁′ 𝒁′′

Original: Prediction 𝑣(𝒁′) 𝑐(𝒁′)
Revised: Prediction 𝑣(𝒁′) 𝑐(𝒁′)

Original: 𝑣(𝒁′′) 𝑐(𝒁′′) 𝑠(𝒁′,𝒁′′)
Revised: 𝑣(𝒁′′) 𝑐(𝒁′′) 𝑠(𝒁′,𝒁′′)

Original: መ𝒢 መ𝒢 ′ 𝐵 𝑚(𝒢,B;W) መ𝒢 ′′ target graphs is denoted by ¯ G = {G i,i + K | [ i − W, i ) ∈ B } .
Revised: መ𝒢 መ𝒢 ′ 𝐵 𝑚(𝒢,B;W) መ𝒢 ′′ the future.

Original: In principle, G i − W,i ∈ ˆ G is an input (history) graph used to predict the target labels of the corresponding target (future) graph G i,i + K ∈ ¯ G . The parameter W controls the size of the history while K controlshow far the model is predicting into the future.
Revised: S controls the stride between intervals.

Original: In practice, we set S = K ; hence, each edge is only predicted once.
Revised: In practice, we set S = K so that each edge is only predicted once in each epoch.

Original: Since ¯ G is only available for training in the downstream task, SSL pre-training only operates on ˆ G , as seen in Figure 2.
Revised: Since ¯ G is only provided for training in the downstream task, SSL pre-training only operates on ˆ G , as seen in Figure 2.

Original: 4. S TAGE 1: P RE - TRAINING  D YNAMIC G RAPHS WITH S ELF -S UPERVISION
Revised: 4. S TAGE 1: P RE - TRAINING ON D YNAMIC G RAPHS WITH S ELF -S UPERVISION

Original: We formulate a joint-embedding architecture (Bromley et al., 1993) for DyG2Vec in which two views of a mini-batch of sub-graphs are generated through random transformations.
Revised: We formulate a joint-embedding architecture (Bromley et al., 1993) for DyG2Vec in which two views of a mini-batch of sub-graphs are generated through random transformations.

Original: The transformations are randomly sampled from a distribution defined by a distortion pipeline.
Revised: The transformations are randomly sampled from a distribution defined by a distortion pipeline.

Original: The encoder maps the views to node embeddings which are processed by the predictor to generate node representations.
Revised: The encoder maps the views to node embeddings which are processed by the predictor to generate node representations.

Original: We minimize an SSL objective (Eq. 3, described below) to optimize the model parameters end-to-end in the pre-training stage.
Revised: We minimize an SSL objective (Eq. 2, described below) to optimize the model parameters end-to-end in the pre-training stage.

Original: See Figure 1 for an overall design of the SSL framework.
Revised: See Figure 1 for an overall design of the SSL framework.

Original: Views : The temporal distortion module generates two views of the input graphs ˆ G ′ =
Revised: Views : The temporal distortion module generates two views of the input graphs ˆ G ′ =

Original: t ′ ( ˆ G ) and ˆ G ′′ =
Revised: t ′ ( ˆ G ) and ˆ G ′′ =

Original: t ′′ ( ˆ G ) where the transformations t ′ and t ′′ are sampled from a distribution T over a predefined set of candidate graph transformations.
Revised: t ′′ ( ˆ G ) where the transformations t ′ and t ′′ are sampled from a distribution T over a predefined set of candidate graph transformations.

Original: In this work, we use edge dropout and edge feature masking (Thakoor et al., 2022) in the transformation pipeline.
Revised: In this work, we use edge dropout and edge feature masking (Thakoor et al., 2022) in the transformation pipeline.

Original: See Appendix A.4 for more details.
Revised: See Appendix A.4 for more details.

Original: Embedding : The encoding model g θ is an attention-based message-passing (AMP) neural network that produces node embeddings H ′ and H ′′ for the views ˆ G ′
Revised: Embedding : The encoding model g θ is an attention-based message-passing (AMP) neural network that produces node embeddings H ′ and H ′′ for the views ˆ G ′

Original: and ˆ G ′′ of the input graphs ˆ G i,j .
Revised: and ˆ G ′′ of the input graphs ˆ G i,j .

Original: We elaborate on the details of the encoder in Sec. 4.3.
Revised: We elaborate on the details of the encoder in Sec. 4.3.

Original: Prediction : The decoding head d γ for our self-supervised learning design consists of a node-level predictor p ϕ that outputs the final representations Z ′ and Z ′′ , where Z = p ϕ ( H ) .
Revised: Prediction : The decoding head d γ for our self-supervised learning design consists of a node-level predictor p ϕ that outputs the final representations Z ′ and Z ′′ , where Z = p ϕ ( H ) .

Original: SSL Objective : In order to learn useful representations, we minimize the VICReg regularizationbased SSL loss function from (Bardes et al., 2022):
Revised: SSL Objective : In order to learn useful representations, we minimize the VICReg regularizationbased SSL loss function from (Bardes et al., 2022):

Original: [Equation]
Revised: [Equation]

Original: In this loss function, the weights λ , µ , and ν control the emphasis placed on each of three regularization terms.
Revised: In this loss function, the weights λ , µ , and ν control the emphasis placed on each of three regularization terms.

Original: The invariance term s encourages representations of the two views to be similar.
Revised: The invariance term s encourages representations of the two views to be similar.

Original: The variance term v is included to prevent the well-known collapse problem (Jing et al., 2022).
Revised: The variance term v is included to prevent the well-known collapse problem (Jing et al., 2022).

Original: The covariance term c promotes maximization of the information content of the representations.
Revised: The covariance term c promotes maximization of the information content of the representations.

Original: More details and complete expressions for s , v and c are provided in Appendix A.3.
Revised: More details and complete expressions for s , v and c are provided in Appendix A.3.

Original: Unlike previous regularization-based SSL approaches (Chen et al., 2020a; Bardes et al., 2022) in computer vision, we do not use a projector network because the embedding dimensions are relatively small in the graph domain.
Revised: Unlike previous regularization-based SSL approaches (Chen et al., 2020a; Bardes et al., 2022) in computer vision, we do not use a projector network because the embedding dimensions are relatively small in the graph domain.

Original: The full pre-training procedure is illustrated in Figure 1.
Revised: The full pre-training procedure is illustrated in Figure 1.

Original: Following the pre-training stage, we replace the SSL decoder with a task-specific downstream decoder d ψ that is trained on top of the frozen pre-trained encoder.
Revised: Following the pre-training stage, we replace the SSL decoder with a task-specific downstream decoder d ψ that is trained on top of the frozen pre-trained encoder.

Original: 𝐵 ⊆
Revised: 𝐵 ⊆

Original: I 𝑊 𝒢 = (𝒱,ℰ,𝑋) 𝑡𝑡 𝑙 𝒕𝒊−𝑾 𝒕 𝒊 𝒕 𝒊+𝑲 𝑡 𝑛 𝑡 ℰ 𝑚(𝒢, B; W)
Revised: I 𝑊 𝒢 = (𝒱,ℰ,𝑋) 𝑡𝑡 𝑙 𝒕 𝒊 𝒕 𝒋 𝒕 𝒋+𝑲 𝑡 𝑛 𝑡 ℰ 𝑚(𝒢, B; W)

Original: SSL Pre-training Task
Revised: SSL Pre-training Task

Original: Downstream Task 𝑡 𝑚 𝒢 𝑖−𝑊,𝑖 ∈ መ 𝒢 𝒢 𝑖−𝑊,𝑖 ∈
Revised: Downstream Task 𝑡 𝑚 𝒢 𝑖,𝑗 ∈ መ 𝒢 𝒢 𝑖,𝑗 ∈

Original: መ 𝒢 𝒢 𝑖,𝑖+𝐾 ∈ ҧ𝒢 𝑾 𝑲
Revised: መ 𝒢 𝒢 𝑗,𝑗+𝐾 ∈ ҧ𝒢 𝑾 𝑲

Original: 4. S TAGE 2: D Y G2V EC D OWNSTREAM T RAINING
Revised: 4. S TAGE 2: D Y G2V EC D OWNSTREAM T RAINING

Original: In the downstream training stage, the DyG2Vec model f = ( g θ , d ψ ) consists of the SSL pre-trained encoder g θ and a task-specific decoder d ψ which is trained using a similar window-based training strategy.
Revised: In the downstream training stage, the DyG2Vec model f = ( g θ , d ψ ) consists of the SSL pre-trained encoder g θ and a task-specific decoder d ψ which is trained using a similar window-based training strategy.

Original: The model is trained to make predictions depending on the downstream tasks (e.g. link prediction or node classification) given the input and target graphs ˆ G and ¯ G as follows:
Revised: The model is trained to make predictions depending on the downstream tasks (e.g. link prediction or node classification) given the input and target graphs ˆ G and ¯ G as follows:

Original: H =
Revised: H =

Original: g θ
Revised: g θ

Original: ( ˆ G ) is the node embeddings returned by the encoder, and Z = d ψ ( H ; ¯ E ) is the prediction output of the decoder.
Revised: ( ˆ G ) is the node embeddings returned by the encoder, and Z = d ψ ( H ; ¯ E ) is the prediction output of the decoder.

Original: Here ¯ E is a set of (partial) edges for which predictions are requested from the decoder.
Revised: Here ¯ E is a set of (partial) edges for which predictions are requested from the decoder.

Original: The model parameters are optimized by training with a loss function L D ( Z , O ) , where L D is defined depending on the downstream task and O contains task-specific labels (See Section 3).
Revised: The model parameters are optimized by training with a loss function L D ( Z , O ) , where L D is defined depending on the downstream task and O contains task-specific labels (See Section 3).

Original: The window-based training strategy has several major advantages.
Revised: The window-based training strategy has several major advantages.

Original: First, the window acts as a regularizer by providing a natural inductive bias towards recent edges, which are often more predictive of the future.
Revised: First, the window acts as a regularizer by providing a natural inductive bias towards recent edges, which are often more predictive of the future.

Original: It also avoids costly time-based neighborhood sampling techniques (Wang et al., 2021).
Revised: It also avoids costly time-based neighborhood sampling techniques (Wang et al., 2021b).

Original: Second, relying on a fixed window-size for message-passing allows for constant memory and computational complexity, which is well-suited to the practical online streaming data scenario.
Revised: Second, relying on a fixed window-size for message-passing allows for constant memory and computational complexity, which is well-suited to the practical online streaming data scenario.

Original: Third, unlike previous works (Xu et al., 2020; Wang et al., 2021) which generate separate node embeddings for each target edge, a generic encoder allows us to use the same set of embeddings for any prediction.
Revised: Third, unlike previous works (Xu et al., 2020; Wang et al., 2021b) which generate separate node embeddings for each target edge, a generic encoder allows us to use the same set of embeddings for any prediction.

Original: This dramatically reduces the training/inference overhead.
Revised: This dramatically reduces the training/inference overhead.

Original: Another advantage of this design is that it allows the model to forecast unseen edges relatively far into the future, in contrast to existing works (Xu et al., 2020; Rossi et al., 2020) that focus on predicting the next occurring edge.
Revised: Another advantage of this design is that it allows the model to forecast unseen edges relatively far into the future, in contrast to existing works (Xu et al., 2020; Rossi et al., 2020) that focus on predicting the next occurring edge.

Original: 4. D Y G2V EC E NCODER A RCHITECTURE
Revised: 4. D Y G2V EC E NCODER A RCHITECTURE

Original: Our encoder combines a self-attention mechanism for message-passing with the Time2Vec module (Kazemi et al., 2019) that provides relative time encoding.
Revised: Our encoder combines a self-attention mechanism for message-passing with the Time2Vec module (Kazemi et al., 2019) that provides relative time encoding.

Original: We also introduce a novel temporal edge encoding that efficiently captures the temporal structural relationship between nodes.
Revised: We also introduce a novel temporal edge encoding that efficiently captures the temporal structural relationship between nodes.

Original: Temporal Attention Embedding : Given a dynamic graph G , the encoder g θ computes the embedding h Li of node i through a series of L multi-head attention (MHA) layers (Vaswani et al., 2017) that aggregate messages from its L -hop neighborhood (Xu et al., 2020; Velickovic et al., 2018).
Revised: Temporal Attention Embedding : Given a dynamic graph G , the encoder g θ computes the embedding h Li of node i through a series of L multi-head attention (MHA) layers (Vaswani et al., 2017) that aggregate messages from its L -hop neighborhood (Xu et al., 2020; Velickovic et al., 2018).

Original: Given a node embedding h l − 1 i at layer l − 1 , we uniformly sample N 1-hop neighborhood interactions of node i , N ( i ) = { e p , . . .
Revised: Given a node embedding h l − 1 i at layer l − 1 , we uniformly sample N 1-hop neighborhood interactions of node i , N ( i ) = { e p , . . .

Original: , e k } ⊆ E .
Revised: , e k } ⊆ E .

Original: The embedding h li at layer l is calculated by:
Revised: The embedding h li at layer l is calculated by:

Original: [Equation]
Revised: [Equation]

Original: Here, W 1 is a learnable mapping matrix, MHA l ( · ) is a multi-head dot-product attention layer, and Φ p ( t p ) represents the edge feature vector of edge
Revised: Here, W 1 is a learnable mapping matrix, MHA l ( · ) is a multi-head dot-product attention layer, and Φ p ( t p ) represents the edge feature vector of edge

Original: e p =
Revised: e p =

Original: ( u p , v p , t p , m p ) ∈ N ( i ) at time t p : 
Revised: ( u p , v p , t p , m p ) ∈ N ( i ) at time t p : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where || denotes concatenation and ϕ ( . ) is a learnable Time2Vec module that helps the model be aware of the relative timespan between a sampled interaction and the most recent interaction of node v p in the input graph.
Revised: where || denotes concatenation and ϕ ( . ) is a learnable Time2Vec module that helps the model be aware of the relative timespan between a sampled interaction and the most recent interaction of node v p in the input graph.

Original: Θ p ( . ) is a temporal edge encoding function, described in more detail below.
Revised: Θ p ( . ) is a temporal edge encoding function, described in more detail below.

Original: In contrast to TGAT’s recursive message passing procedure (Xu et al., 2020), the message passing in our encoder is ‘flat’:
Revised: In contrast to TGAT’s recursive message passing procedure (Xu et al., 2020), the message passing in our encoder is ‘flat’:

Original: at every iteration, the same set of node embeddings is used to propagate messages to neighbors.
Revised: at every iteration, the same set of node embeddings is used to propagate messages to neighbors.

Original: Our encoder performs message passing once to generate a set of node embeddings H used for all target predictions on ¯ G .
Revised: Our encoder performs message passing once to generate a set of node embeddings H used for all target predictions on ¯ G .

Original: Temporal Edge Encoding : Dynamic graphs often follow evolutionary patterns that reflect how nodes interact over time (Kovanen et al., 2011).
Revised: Temporal Edge Encoding : Dynamic graphs often follow evolutionary patterns that reflect how nodes interact over time (Kovanen et al., 2011).

Original: For example, in social networks, two people who share many friends are likely to interact in the future.
Revised: For example, in social networks, two people who share many friends are likely to interact in the future.

Original: Therefore, we incorporate two simple yet effective temporal encoding methods that provide inductive biases to capture common structural and temporal evolutionary behaviour of dynamic graphs.
Revised: Therefore, we incorporate two simple yet effective temporal encoding methods that provide inductive biases to capture common structural and temporal evolutionary behaviour of dynamic graphs.

Original: The temporal edge encoding function is then:
Revised: The temporal edge encoding function is then:

Original: [Equation]
Revised: [Equation]

Original: where we incorporate (i) Temporal Degree Centrality z p ( t p )  : the  current degrees of nodes u p and v p at time t p ; and (ii) Common Neighbors c p ( t p )  : the number of common 1-hop neighbors between nodes u p and v p at time t p .
Revised: where we incorporate (i) Temporal Degree Centrality z p ( t p ) ∈ R 2 : the concatenated current degrees of nodes u p and v p at time t p ; and (ii) Common Neighbors c p ( t p ) ∈ R : the number of common 1-hop neighbors between nodes u p and v p at time t p .

Original: By using the degree centrality as an edge feature, the model is able to learn any bias towards more frequent interactions with high-degree nodes.
Revised: By using the degree centrality as an edge feature, the model is able to learn any bias towards more frequent interactions with high-degree nodes.

Original: The number of common neighbors helps capture temporal motifs, and it is known to often have a strong positive correlation with the likelihood of a future interaction (Yao et al., 2016).
Revised: The number of common neighbors helps capture temporal motifs, and it is known to often have a strong positive correlation with the likelihood of a future interaction (Yao et al., 2016).

Original:  E XPERIMENTAL E VALUATION 5. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL E VALUATION 5. E XPERIMENTAL S ETUP

Original: Baselines : We compare DyG2Vec to five state-of-the-art baseline models: DyRep (Trivedi et al., 2019), JODIE (Kumar et al., 2019), TGAT (Xu et al., 2020),
Revised: Baselines : We compare DyG2Vec to five state-of-the-art baseline models: DyRep (Trivedi et al., 2019), JODIE (Kumar et al., 2019), TGAT (Xu et al., 2020),

Original: TGN (Rossi et al., 2020), and
Revised: TGN (Rossi et al., 2020), and

Original: CaW (Wang et al., 2021).
Revised: CaW (Wang et al., 2021b).

Original: DyRep, JODIE, and TGN sequentially update node embeddings using an RNN.
Revised: DyRep, JODIE, and TGN sequentially update node embeddings using an RNN.

Original: TGAT applies message passing via attention on a sampled temporal subgraph.
Revised: TGAT applies message passing via attention on a sampled temporal subgraph.

Original: CaW samples temporal random walks and learns temporal motifs by counting node occurrences in each walk.
Revised: CaW samples temporal random walks and learns temporal motifs by counting node occurrences in each walk.

Original: Downstream Tasks :
Revised: Downstream Tasks :

Original: We evaluate all models on two temporal tasks: future link prediction (FLP), and dynamic node classification (DNC).
Revised: We evaluate all models on two temporal tasks: future link prediction (FLP), and dynamic node classification (DNC).

Original: In FLP, the goal is to predict the probability of future edges occurring given the source, destination, and timestamp.
Revised: In FLP, the goal is to predict the probability of future edges occurring given the source, destination, and timestamp.

Original: For each positive edge, we sample a negative edge that the model is trained to predict as negative.
Revised: For each positive edge, we sample a negative edge that the model is trained to predict as negative.

Original: The DNC task involves predicting the label of the source node of a future interaction.
Revised: The DNC task involves predicting the label of the source node of a future interaction.

Original: Both tasks are trained using binary cross entropy loss.
Revised: Both tasks are trained using binary cross entropy loss.

Original: Contrary to prior works which only evaluate under the K = 1 setting, we evaluate all models under K ∈ { 1 , 200 , 2000 } .
Revised: Contrary to prior works which only evaluate under the K = 1 setting, we evaluate all models under K ∈ { 1 , 200 , 2000 } .

Original: This is a more challenging evaluation scheme as it tests the forecasting capabilities of the models across multiple time horizons.
Revised: This is a more challenging evaluation scheme as it tests the forecasting capabilities of the models across multiple time horizons.

Original: See Appendix A.4 for details.
Revised: See Appendix A.4 for details.

Original: [Table] For the FLP task [Table] we report both classification and recommendation metrics: Average Precision (AP), Mean Reciprocal-Rank (MRR)
Revised: For the FLP task, we report both classification and recommendation metrics: Average Precision (AP), Mean Reciprocal-Rank (MRR), and Rec@10.

Original: [Table]

Original: and Rec@10.

Original: For the DNC task [Table] we report the area under the curve (AUC) metric due to the prevailing issue of class imbalance in dynamic graphs
Revised: For the DNC task, we report the area under the curve (AUC) metric due to the prevailing issue of class imbalance in dynamic graphs.

Original: [Table] [Figure] 
Revised: [Figure] [Figure] [Table]

Original: Datasets : We use 7 real-world datasets: Wikipedia, Reddit, MOOC, and LastFM (Kumar et al., 2019); SocialEvolution, Enron, and UCI (Wang et al., 2021).
Revised: Datasets : We use 7 real-world datasets: Wikipedia, Reddit, MOOC, and LastFM (Kumar et al., 2019); SocialEvolution, Enron, and UCI (Wang et al., 2021b).

Original: These datasets span a wide range in terms of number of nodes and interactions, time range, and repetition ratio.
Revised: These datasets span a wide range in terms of number of nodes and interactions, time range, and repetition ratio.

Original: We perform the same 70% - 15% - 15% chronological split for all datasets as in (Wang et al., 2021).
Revised: We perform the same 70% - 15% - 15% chronological split for all datasets as in (Wang et al., 2021b).

Original: The datasets are split differently under two settings: Transductive and Inductive.
Revised: The datasets are split differently under two settings: Transductive and Inductive.

Original: More details can be found in Appendix
Revised: More details can be found in Appendix

Original: A.1.
Revised: A.1.

Original: The code and datasets will be publicly available upon publication.
Revised: The code and datasets will be publicly available upon publication.

Original: Training Protocols and Hyper-parameters : We train and evaluate the models under three different settings commonly used in vision SSL works (Grill et al., 2020;
Revised: Training Protocols and Hyper-parameters : We train and evaluate the models under three different settings commonly used in vision SSL works (Grill et al., 2020;

Original: Bardes et al., 2022).
Revised: Bardes et al., 2022).

Original: In the supervised setting, DyG2Vec is initialized with random parameters and trained directly on the downstream tasks and compared to all supervised baselines.
Revised: In the supervised setting, DyG2Vec is initialized with random parameters and trained directly on the downstream tasks and compared to all supervised baselines.

Original: In the self-supervised setting, the encoder is pre-trained using our SSL framework, and the performance is measured under two evaluation protocols: Linear and Semi-supervised Probing.
Revised: In the self-supervised setting, the encoder is pre-trained using our SSL framework, and the performance is measured under two evaluation protocols: Linear and Semi-supervised Probing.

Original: In the linear evaluation setting, the decoder is trained on top of the frozen encoder and compared to the supervised counterpart.
Revised: In the linear evaluation setting, the decoder is trained on top of the frozen encoder and compared to the supervised counterpart.

Original: In the semi-supervised evaluation setting, the decoder is trained on top of the frozen pre-trained encoder on a random portion of the dataset (i.e., a fraction of the intervals I ).
Revised: In the semi-supervised evaluation setting, the decoder is trained on top of the frozen pre-trained encoder on a random portion of the dataset (i.e., a fraction of the intervals I ).

Original: The DyG2Vec encoder performs L = 3 layers of message passing.
Revised: The DyG2Vec encoder performs L = 3 layers of message passing.

Original: We sample N = 20 temporal neighbors at each hop as in Xu et al.
Revised: We sample N = 20 temporal neighbors at each hop as in Xu et al.

Original: Other hyperparameters are discussed in Appendix A.5.
Revised: Other hyperparameters are discussed in Appendix A.5.

Original: For the DNC task, following prior work Rossi et al.
Revised: For the DNC task, following prior work Rossi et al.

Original: (2020), the decoder is trained on top of the frozen encoder that is pre-trained on the future link prediction task unless otherwise explicitly stated.
Revised: (2020), the decoder is trained on top of the frozen encoder that is pre-trained on the future link prediction task unless otherwise explicitly stated.

Original: 5. E XPERIMENTAL R ESULTS
Revised: 5. E XPERIMENTAL R ESULTS

Original: Future Link Prediction : We report the tranductive test AP scores for future link prediction in Table 1.
Revised: Future Link Prediction : We report the tranductive test AP scores for future link prediction in Table 1.

Original: Unlike previous work which focuses on K = 1 , we evaluate all models under K ∈ { 1 , 200 , 2000 } to test their medium- and longer-term forecasting capabilities.
Revised: Unlike previous work which focuses on K = 1 , we evaluate all models under K ∈ { 1 , 200 , 2000 } to test their medium- and longer-term forecasting capabilities.

Original: Unsurprisingly, all methods degrade [Figure]
Revised: Unsurprisingly, all methods degrade as K increases.

Original: [Table] as K increases. Our model significantly outperforms all sequential and message-passing baselines  for K > 1 and is on par with SoTA (CaW) for K = 1 . 
Revised: Our model significantly outperforms all sequential and message-passing baselines on 5/7 of the datasets for K > 1 and is on par with SoTA (CaW) for K = 1 . The gap is particularly large on the UCI and SocialEvol.

Original: The gap is particularly large on the UCI and
Revised: datasets for K = 2000 , where DyG2Vec outperforms the second-

Original: SocialEvol.
Revised: [Figure]

Original: datasets for K = 2000 , where DyG2Vec outperforms the second-best method (CaW) by over 10% and 6% respectively.
Revised: For each setting, DyG2Vec was trained [Table] best method (CaW) by over 10% and 6% respectively.

Original: Interestingly, while SocialEvol.
Revised: Interestingly, while SocialEvol.

Original: is the largest dataset with ∼ 2 M edges, our model is able to achieve this performance while only using the last 1000 edges (See Table 8) to predict any future edge.
Revised: is the largest dataset with ∼ 2 M edges, our model is able to achieve this performance while only using the last 1000 edges (See Table 9) to predict any future edge.

Original: This further cements the findings by Xu et al.
Revised: This further cements the findings by Xu et al.

Original: (2020) that capturing recent interactions may be more important for certain tasks.
Revised: (2020) that capturing recent interactions may be more important for certain tasks.

Original: Our window-based framework offers a good trade-off between capturing recent interactions and recurrent patterns which both have a major influence on future interactions.
Revised: Our window-based framework offers a good trade-off between capturing recent interactions and recurrent patterns which both have a major influence on future interactions.

Original: Appendix A.2 contains results in the Inductive setting which show that DyG2Vec is competitive with CaW  while using a small fraction of the computation (See Figure 3).
Revised: Appendix A.2 contains results in the Inductive setting which show that DyG2Vec is competitive with CaW for 4/7 of the datasets while using a small fraction of the computation (See Figure 3).

Original: Dynamic Node classification : We evaluate DyG2Vec on 3 datasets for node classification where the labels indicate whether a user will be banned from editing/posting after an interaction.
Revised: Dynamic Node classification : We evaluate DyG2Vec on 3 datasets for node classification where the labels indicate whether a user will be banned from editing/posting after an interaction.

Original: This task is challenging both due to its dynamic nature (i.e., nodes can change labels) and the high class imbalance (only 217 of 157K interactions result in a ban).
Revised: This task is challenging both due to its dynamic nature (i.e., nodes can change labels) and the high class imbalance (only 217 of 157K interactions result in a ban).

Original: We measure performance using the AUC metric to deal with the class imbalance.
Revised: We measure performance using the AUC metric to deal with the class imbalance.

Original: Table 2 shows that DyG2Vec outperforms all baselines significantly, especially for  K= 200 .
Revised: Table 2 shows that DyG2Vec outperforms all baselines on 7/9 of the tasks for different K .

Original: Interestingly, the performance of all methods does not always drop as K increases.
Revised: Interestingly, the performance of all methods does not always drop as K increases.

Original: This could be explained by the fact that depending on slightly out-of-date history can help make less noisy and more consistent predictions; thus, increasing performance.
Revised: This could be explained by the fact that depending on slightly out-of-date history can help make less noisy and more consistent predictions; thus, increasing performance.

Original: Training/Inference Speed :
Revised: Training/Inference Speed :

Original: Relying on a fixed window of history to produce task-agnostic node embeddings gives DyG2Vec a significant advantage in speed and memory.
Revised: Relying on a fixed window of history to produce task-agnostic node embeddings gives DyG2Vec a significant advantage in speed and memory.

Original: Figure 3 shows the performance and runtime per epoch of all methods on the three large datasets: LastFM, SocialEvolution and MOOC, with K = 200 .
Revised: Figure 3 shows the performance and runtime per epoch of all methods on the three large datasets: LastFM, SocialEvolution and MOOC, with K = 200 .

Original: DyG2Vec is many orders of magnitude faster than CaW due to the latter’s expensive random walk sampling procedure.
Revised: DyG2Vec is many orders of magnitude faster than CaW due to the latter’s expensive random walk sampling procedure.

Original: RNN-based methods such as TGN have a good runtime on LastFM and MOOC; however, they are significantly slower on SocialEvol.
Revised: RNN-based methods such as TGN have a good runtime on LastFM and MOOC; however, they are significantly slower on SocialEvol.

Original: which has a small number of nodes (74) but a large number of interactions ( ∼ 2 M ).
Revised: which has a small number of nodes (74) but a large number of interactions ( ∼ 2 M ).

Original: This suggests that memorybased methods are slower for settings where a node’s memory is updated frequently.
Revised: This suggests that memorybased methods are slower for settings where a node’s memory is updated frequently.

Original: Furthermore, while TGAT has a similar AMP encoder, DyG2Vec improves the efficiency and performance significantly.
Revised: Furthermore, while TGAT has a similar AMP encoder, DyG2Vec improves the efficiency and performance significantly.

Original: This reveals the significance of the window-based mechanism and the encoder architecture.
Revised: This reveals the significance of the window-based mechanism and the encoder architecture.

Original: Overall, DyG2Vec presents the best trade-off between speed and performance.
Revised: Overall, DyG2Vec presents the best trade-off between speed and performance [Table] [Table]

Original: SSL for Future Link Prediction : Table 3 reports the transductive AP results for DyG2Vec under 3 different settings.
Revised: Table 3 reports the transductive AP results for DyG2Vec under 3 different settings.

Original: Namely, we compare a random frozen encoder (Random-init) and an SSL pretrained encoder (SSL-init) with the supervised baseline.
Revised: Namely, we compare a random frozen encoder (Randominit) and an SSL pre-trained encoder (SSL-init) with the supervised baseline.

Original: The results reveal that our SSL pre-training learns informative node embeddings that are almost on par with the fully supervised baseline.
Revised: The results reveal that our SSL pre-training learns informative node embeddings that are almost on par with the fully supervised baseline.

Original: This supports the capability of the non-contrastive methods to learn generic representations across unlabelled large-scale dynamic graphs, which is in line with the findings for other data modalities (Bardes et al., 2022).
Revised: This supports the capability of the non-contrastive methods to learn generic representations across unlabelled large-scale dynamic graphs, which is in line with the findings for other data modalities (Bardes et al., 2022).

Original: The Random-init baseline is surprisingly good, as observed by recent works (Thakoor et al., 2022), but is outperformed by the SSL pre-trained encoder.
Revised: The Random-init baseline is surprisingly good, as observed by recent works (Thakoor et al., 2022), but is outperformed by the SSL pre-trained encoder.

Original: Semi-supervised Learning on Dynamic Node Classification : The DNC task is challenging due to its highly imbalanced labels.
Revised: Semi-supervised Learning on Dynamic Node Classification : The DNC task is challenging due to its highly imbalanced labels.

Original: Previous works alleviate this issue by pre-training the encoder on future link prediction.
Revised: Previous works alleviate this issue by pre-training the encoder on future link prediction.

Original: In Figure 5, we show that SSL is a more effective pre-training strategy for dynamic graphs than FLP, particularly in the low-label data regime where each model is trained on a portion of the target intervals I .
Revised: In Figure 5, we show that SSL is a more effective pre-training strategy for dynamic graphs than FLP, particularly in the low-label data regime where each model is trained on a portion of the target intervals I .

Original: This further cements the findings that reconstruction-based tasks such as link prediction overemphasize proximity which can be limiting for some downstream tasks (Velickovic et al., 2019; You et al., 2020) [Figure]
Revised: This further cements the findings that reconstruction-based tasks such as link prediction overemphasize proximity which can be limiting for some downstream tasks (Velickovic et al., 2019; You et al., 2020).

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: We perform a detailed study on different instances of our framework with 3 datasets.
Revised: We perform a detailed study on different instances of our framework with 3 datasets.

Original: All ablation results are reported in Figure 6.
Revised: All ablation results are reported in Figure 6.

Original: Window Size : We observe that each dataset has its own optimal window size W due to the inherently different recurring temporal patterns.
Revised: Window Size : We observe that each dataset has its own optimal window size W due to the inherently different recurring temporal patterns.

Original: As observed by Xu et al.
Revised: As observed by Xu et al.

Original: (2020), recent and/or recurrent interactions are often the most predictive of future interactions.
Revised: (2020), recent and/or recurrent interactions are often the most predictive of future interactions.

Original: Therefore, datasets with long range dependencies favor larger window sizes to capture the recurrent patterns while some datasets benefit from an increased bias towards recent interactions.
Revised: Therefore, datasets with long range dependencies favor larger window sizes to capture the recurrent patterns while some datasets benefit from an increased bias towards recent interactions.

Original: Our window-based framework coupled with uniform neighbor sampling strikes a balance between the two.
Revised: Our window-based framework coupled with uniform neighbor sampling strikes a balance between the two.

Revised: Moreover, increasing the window size to 64K for UCI, which is effectively full history as it has 60K edges in total, results in a 4% drop in performance.

Revised: This shows that the fixed window size also contributes to the performance as it helps limit irrelevant information that is not highly predictive of future interactions.

Original: Number of Layers: Most datasets benefit from more embedding layers, and some (e.g. MOOC) more than others.
Revised: Number of Layers: Most datasets benefit from more embedding layers, and some (e.g. MOOC) more than others.

Original: This suggests that these datasets contain higher order temporal correlations among nodes that must be learned using long-range message passing.
Revised: This suggests that these datasets contain higher order temporal correlations among nodes that must be learned using long-range message passing.

Original: Overall, the results show that one can choose to sacrifice some performance to further improve the speed of DyG2Vec by decreasing the window size and the number of layers.
Revised: Overall, the results show that one can choose to sacrifice some performance to further improve the speed of DyG2Vec by decreasing the window size and the number of layers.

Original: Temporal Edge Features : The results show a significant decrease in performance for MOOC when temporal edge features are removed.
Revised: Temporal Edge Features : The results show a significant decrease in performance for MOOC when temporal edge features are removed (i.e. 1-5% drop).

Original: This indicates that such temporal edge features provide useful multi-hop information about the evolution of the dynamic graph (Yao et al., 2016).
Revised: This indicates that such temporal edge features provide useful multi-hop information about the evolution of the dynamic graph (Yao et al., 2016).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce DyG2Vec , a novel window-based encoder-decoder model for dynamic graphs.
Revised: In this paper, we introduce DyG2Vec, a novel window-based encoder-decoder model for dynamic graphs.

Original: We present an efficient attention-based message-passing model that utilizes hierarchical multi-head attention modules to encode node embeddings across time.
Revised: We present an efficient attention-based message-passing model that utilizes hierarchical multi-head attention modules to encode node embeddings across time.

Original: Furthermore, we present a joint-embedding architecture for dynamic graphs in which two views of temporal sub-graphs are encoded to minimize a non-contrastive loss function.
Revised: Furthermore, we present a joint-embedding architecture for dynamic graphs in which two views of temporal sub-graphs are encoded to minimize a non-contrastive loss function.

Original: We evaluate the SSL pre-training of
Revised: We evaluate the SSL pre-training of

Original: DyG2Vec under both linear and semi-supervised protocols and demonstrate the effectiveness of such pre-training on benchmark datasets.
Revised: DyG2Vec under both linear and semi-supervised protocols and demonstrate the effectiveness of such pre-training on benchmark datasets.

Original: Our window-based architecture allows for efficient messagepassing and robust forecasting abilities.
Revised: Our window-based architecture allows for efficient messagepassing and robust forecasting abilities.

Original: We aim to further explore ways to improve the capacity of the dynamic graph models to learn long-range dependencies.
Revised: We aim to further explore ways to improve the capacity of the dynamic graph models to learn long-range dependencies.

Original: Additionally, other SSL paradigms such as masked auto-encoders are worthwhile future explorations for dynamic graphs given their success on sequential tasks.
Revised: Additionally, other SSL paradigms such as masked auto-encoders are worthwhile future explorations for dynamic graphs given their success on sequential tasks.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Dynamic graph neural network techniques have been commonly used for prediction tasks in social networks and recommender systems.
Revised: Dynamic graph neural network techniques have been commonly used for prediction tasks in social networks and recommender systems.

Original: Our techniques, as an efficient and effective variant of dynamic graph representation technique, can be used in those scenarios to further improve the model performance.
Revised: Our techniques, as an efficient and effective variant of dynamic graph representation technique, can be used in those scenarios to further improve the model performance.

Original: However, having such an ability is a double-edged sword.
Revised: However, having such an ability is a double-edged sword.

Original: On one hand, it can be beneficial to greatly improve user experience.
Revised: On one hand, it can be beneficial to greatly improve user experience.

Original: On the other hand, there may be some concerns about the potential use of the model to exploit data relating to user behaviour and thus invade privacy.
Revised: On the other hand, there may be some concerns about the potential use of the model to exploit data relating to user behaviour and thus invade privacy.

Original: Overall, our paper does not include content which has immediate ethical concerns.
Revised: Overall, our paper does not include content which has immediate ethical concerns.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We describe the details of the benchmarks we used in Appendix A.1.
Revised: We describe the details of the benchmarks we used in Appendix A.1.

Original: We also include the full implementation details in Appendix A.4, including our detailed model design for decoder, negative sampler, distortion pipeline for view generations and etc.
Revised: We also include the full implementation details in Appendix A.4, including our detailed model design for decoder, negative sampler, distortion pipeline for view generations and etc.

Original: Besides, we provide a justification for our choice of hyperparameters in Appendix A.5.
Revised: Besides, we provide a justification for our choice of hyperparameters in Appendix A.5.

Original: Additionally, we elaborate on the baselines we consider in our paper as well as their hyper-parameter details in Appendix A
Revised: Additionally, we elaborate on the baselines we consider in our paper as well as their hyper-parameter details in Appendix A.6.

Original: We believe with the code open-sourced upon acceptance and the detailed description of our model and the baselines provided in the paper, our work is fully reproducible
Revised: We believe with the code open-sourced upon acceptance and the detailed description of our model and the baselines provided in the paper, our work is fully reproducible

Original: A. A DDITIONAL R ESULTS 
Revised: A. A DDITIONAL R ESULTS A.2. I NDUCTIVE S ETTING AND R ANKING M ETRICS

Original: Table 5 reports the future link prediction results in the inductive setting.
Revised: Table 5 reports the future link prediction results in the inductive setting.

Original: DyG2Vec is generally competitive with CaW while using 50-100 × less inference and training runtime (See 3 and 7 respectively).
Revised: DyG2Vec is generally competitive with CaW while using 50-100 × less inference and training runtime (See 3 and 7 respectively).

Original: Tables 6 and 7 report the future link prediction ranking performance for K = 1 on the transductive and inductive tasks, respectively.
Revised: Tables 6 and 7 report the future link prediction ranking performance for K = 1 on the transductive and inductive tasks, respectively.

Original: While ranking metrics should provide more finegrained analysis, the results are consistent with the AP results (See Tables 1 and 5). 
Revised: While ranking metrics should provide more finegrained analysis, the results are consistent with the AP results (See Tables 1 and 5). [Table] [Table]

Revised: A.2. E FFECT OF K DURING T RAINING

Original: [Table]
Revised: One of the advantages of the DyG2Vec framework is that, unlike prior work, it is trained to forecast with K > 1 . In fact, the parameter K can be set during training based on how much long-range range forecasting is favored. In our experiments, we found that training with K = 200 achieves a good balance between long and short-range forecasting capabilities. Moreover, the limited history of

Revised: W edges forces the model to be more inductive as it is predicting based on limited long-range historical information.

Revised: To better understand this, we trained both DyG2Vec and CaW under K = { 1 , 200 } and evaluate on K = { 1 , 200 , 2000 } .

Revised: Table 8 shows that training DyG2Vec with short-range prediction ( K = 1 ) improves performance to be on par with CaW on K = 1 and outperform it for K > 1 .

Revised: However, as expected, this comes at a cost of ∼ 2% drop for long-range forecasting ( K > 1 ) when compared to DyG2Vec trained with K = 200 .

Revised: On the other hand, CaW’s performance drops significantly when trained with K = 200 (i

Original: [Table]
Revised: [Table]

Revised: e. over 10% drop for UCI and MOOC).

Revised: We believe this is due to the sampling bias α which may be incorrectly favoring recent edges over edges that occurred further in the past but can help for long-range forecasting.

Revised: Unfortunately, we were unable to address this by re-tuning α .

Revised: An interesting direction for future research would be to study training settings under which all models can have improved forecasting abilities

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: A.2. R UNTIME AND C OMPUTATIONAL C OMPLEXITY

Revised: Figure 7 shows the training time per method for 3 datasets.

Revised: DyG2Vec is orders-of-magnitude faster than CaW and is on par with memory-based methods in terms of speed.

Revised: The main runtime overhead lies in how each of the baselines processes the input graph to predict a batch of K target edges.

Revised: CaW samples M L -hop random walks for each target edge.

Revised: This is followed by an expensive set-based anonymization scheme.

Revised: To achieve good performance, CaW can require relatively long walks (e.g., for Enron, L = 5 ).

Revised: On the other hand, memory-based methods and TGAT sample a different L -hop subgraph for each target edge.

Revised: DyG2Vec samples a single L -hop subgraph within a constant window size W for all target edges.

Revised: Thus, assuming we use sparse operations in Pytorch Geometric (Fey & Lenssen, 2019) for messagepassing, the encoding computational complexities are: DyG2Vec

Revised: = O ( LW ) ; CaW

Revised: = O

Revised: ( LMN s K ) and TGN and variants =

Revised: O ( KLN s ) .

Revised: Here, N s represents the maximum number of sampled nodes in an L -hop subgraph and K is the number of target edges to predict.

Revised: We can see that the main difference is the factor M and the fact that this sampling is done for each of the K target edges.

Revised: The factor N s comes from the complexity of message passing at each hop (assuming sparse operations).

Revised: Note that DyG2Vec is limited to O ( W ) nodes so it does not have this factor.

Original: A. P RELIMINARY : VICR EG
Revised: A. P RELIMINARY : VICR EG

Original: We outline the details of the VICReg (Bardes et al., 2022) method used in our SSL pre-training stage.
Revised: We outline the details of the VICReg (Bardes et al., 2022) method used in our SSL pre-training stage.

Original: Given z ′ ∈ R d and z ′′ ∈ R d , the representations of two random views of an object (e.g., image) generated through random distortions, the objective of non-contrastive SSL is two-fold.
Revised: Given z ′ ∈ R d and z ′′ ∈ R d , the representations of two random views of an object (e.g., image) generated through random distortions, the objective of non-contrastive SSL is two-fold.

Original: First, the output representation of one view should be maximally informative of the input representation of the view.
Revised: First, the output representation of one view should be maximally informative of the input representation of the view.

Original: Second, the representation of one view should be maximally predictable from the representation of the other view.
Revised: Second, the representation of one view should be maximally predictable from the representation of the other view.

Original: These two aspects are formulated by VICReg (Bardes et al., 2022) where a combination of 3 loss terms (i.e. Variance, Covariance, Invariance) is minimized to learn useful representations while also avoiding the well-known problem of collapse (Jing et al., 2022) in the mapping space.
Revised: These two aspects are formulated by VICReg (Bardes et al., 2022) where a combination of 3 loss terms (i.e. Variance, Covariance, Invariance) is minimized to learn useful representations while also avoiding the well-known problem of collapse (Jing et al., 2022) in the mapping space.

Original: More concretely, let Z ′ =
Revised: More concretely, let Z ′ =

Original: [ z ′ 1 , . . .
Revised: [ z ′ 1 , . . .

Original: , z ′ n ] and Z ′′ =
Revised: , z ′ n ] and Z ′′ =

Original: [ z ′′ 1 , . . .
Revised: [ z ′′ 1 , . . .

Original: , z ′′ n ] be the batches composed of n representations of dimension d .
Revised: , z ′′ n ] be the batches composed of n representations of dimension d .

Original: Variance term : The variance regularization term v is the mean over the representation dimension of the hinge function on the standard deviation of the representations along the batch dimension: v ( Z )
Revised: Variance term : The variance regularization term v is the mean over the representation dimension of the hinge function on the standard deviation of the representations along the batch dimension: v ( Z )

Original: = 1 D (cid:80)
Revised: = 1 D (cid:80)

Original: Dj =1 max (0 , γ − S
Revised: Dj =1 max (0 , γ − S

Original: ( Z : ,j , ϵ )) .
Revised: ( Z : ,j , ϵ )) .

Original: Here Z : ,j is the column j of matrix Z .
Revised: Here Z : ,j is the column j of matrix Z .

Original: S is the regularized standard deviation defined by S ( z , ϵ ) =
Revised: S is the regu- larized standard deviation defined by S ( z , ϵ ) =

Original: (cid:112) Var ( z ) + ϵ , γ is a constant value set to 1 in our[Table] [Figure] experiments, and ϵ is a small scalar that helps to prevent numerical instability.
Revised: (cid:112) Var ( z ) + ϵ , γ is a constant value set to 1 in our experiments, and ϵ is a small scalar that helps to prevent numerical instability.

Original: This term avoids dimensional collapse by maximizing the volume of the distribution of the mapped views in all dimensions.
Revised: This term avoids dimensional collapse by maximizing the volume of the distribution of the mapped views in all dimensions.

Original: In other words, it prevents the well-known trivial solution where the representations of the two views of a sample collapse to the same representation (Jing et al., 2022).
Revised: In other words, it prevents the well-known trivial solution where the representations of the two views of a sample collapse to the same representation (Jing et al., 2022).

Original: Covariance term : The covariance regularization terms C decorrelates different dimensions of the representations and prevents them from encoding similar information.
Revised: Covariance term : The covariance regularization terms C decorrelates different dimensions of the representations and prevents them from encoding similar information.

Original: The covariance matrix of
Revised: The covariance matrix of

Original: Z is C ( Z ) =
Revised: Z is C ( Z ) =

Original: 1 n (cid:80) ni
Revised: 1 n (cid:80) ni

Original: =

Revised: =

Original: ( z i − ¯ z )( z i − ¯ z ) T where ¯ z = 1 N (cid:80) i =1 n z i .
Revised: ( z i − ¯ z )( z i − ¯ z ) T where ¯ z = 1 N (cid:80) i =1 n z i .

Original: The covariance regularization term c is then defined as the sum of the squared off-diagonal coefficients of the covariance matrix as follows c ( Z )
Revised: The covariance regularization term c is then defined as the sum of the squared off-diagonal coefficients of the covariance matrix as follows c ( Z )

Original: = 1 d (cid:80) i ̸
Revised: = 1 d (cid:80) i ̸

Original: = j [ C ( Z )] 2 i,j where [ C ( Z )] 2 i,j is the element at row i and column j of the matrix C ( Z ) .
Revised: = j [ C ( Z )] 2 i,j where [ C ( Z )] 2 i,j is the element at row i and column j of the matrix C ( Z ) .

Original: Both the variance and covariance terms helps to maximize the information encoded by the model in the representation space.
Revised: Both the variance and covariance terms helps to maximize the information encoded by the model in the representation space.

Original: Invariance criterion : The invariance criterion s between Z ′ and Z ′′ is defined as the mean squared Euclidean distance between the representation vectors in the two views s ( Z ′ , Z ′′ )
Revised: Invariance criterion : The invariance criterion s between Z ′ and Z ′′ is defined as the mean squared Euclidean distance between the representation vectors in the two views s ( Z ′ , Z ′′ )

Original: = 1 n (cid:80) i ∥
Revised: = 1 n (cid:80) i ∥

Original: z ′ i − z ′′ i ∥ 2 2 .
Revised: z ′ i − z ′′ i ∥ 2 2 .

Original: The invariance term encourages the parametric mapping to ensure that the views of an object remain close in the latent space.
Revised: The invariance term encourages the parametric mapping to ensure that the views of an object remain close in the latent space.

Original: Finally, the SSL loss function L SSL over a batch of representations is a weighted average of the invariance, variance, and covariance terms:
Revised: Finally, the SSL loss function L SSL over a batch of representations is a weighted average of the invariance, variance, and covariance terms:

Original: [Equation]
Revised: [Equation]

Original: In our experiments, we set λ
Revised: In our experiments, we set λ

Original: = µ = 25 and ν = 1 , following Bardes et al.
Revised: = µ = 25 and ν = 1 , following Bardes et al.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: We train our model using the Pytorch framework (Paszke et al., 2019).
Revised: We train our model using the Pytorch framework (Paszke et al., 2019).

Original: The dynamic graph data and
Revised: The dynamic graph data and

Original: GNN encoder architecture are implemented using Pytorch Geometric (Fey & Lenssen, 2019).
Revised: GNN encoder architecture are implemented using Pytorch Geometric (Fey & Lenssen, 2019).

Original: The
Revised: The

Original: ReLU activation function is used for all models.
Revised: ReLU activation function is used for all models.

Original: The code and datasets are publicly available 2 .
Revised: The code and datasets are publicly available 2 .

Revised: Window-based framework : As mentioned in Section 4, the full dynamic graph G 0 ,E is divided into a set of intervals I that is generated by dividing the entire time-span into M = ⌈ E/S ⌉ −intervals with stride S and interval length W :

Revised: [Equation]

Revised: Here, W defines the number of edges in an interval and S defines the stride.

Revised: Note that we include all intervals up to but not including [ E − W, E ) so that the target interval contains at least one edge.

Original: Negative Sampling :
Revised: Negative Sampling :

Original: For future link prediction, we sample an equal number of negative interactions and positive (target) interactions.
Revised: For future link prediction, we sample an equal number of negative interactions and positive (target) interactions.

Original: Negative interactions are sampled by sampling a random negative destination node from the set of all possible nodes.
Revised: Negative interactions are sampled by sampling a random negative destination node from the set of all possible nodes.

Original: For the Recall@10 and MRR metrics, we sample 500 random negative destinations for each positive target edge.
Revised: For the Recall@10 and MRR metrics, we sample 500 random negative destinations for each positive target edge.

Original: Decoder Architecture :
Revised: Decoder Architecture :

Original: Denote by t max the timestamp of the latest interaction, within the provided history, incident to node u .
Revised: Denote by t max the timestamp of the latest interaction, within the provided history, incident to node u .

Original: For future link prediction, to predict a target interaction ( u, v, t ) , our decoder maps the sum of the two node embeddings of u and v and a time embedding of t − t max to an edge probability.
Revised: For future link prediction, to predict a target interaction ( u, v, t ) , our decoder maps the sum of the two node embeddings of u and v and a time embedding of t − t max to an edge probability.

Original: Following Xu et al.
Revised: Following Xu et al.

Original: (2020), the FLP decoder is a 2-layer MLP.
Revised: (2020), the FLP decoder is a 2-layer MLP.

Original: For dynamic node classification, to predict the label of node u for interaction ( u, v, t ) , the decoder maps the source node embedding and time embedding of t − t max to class probabilities.
Revised: For dynamic node classification, to predict the label of node u for interaction ( u, v, t ) , the decoder maps the source node embedding and time embedding of t − t max to class probabilities.

Original: Following
Revised: Following

Original: Xu et al.
Revised: Xu et al.

Original: (2020), the DNC decoder is a 3-layer MLP with a dropout layer with p = 0 .
Revised: (2020), the DNC decoder is a 3-layer MLP with a dropout layer with p = 0 .

Original: The time embedding is calculated using a trainable Time2Vec module (Kazemi et al., 2019).
Revised: The time embedding is calculated using a trainable Time2Vec module (Kazemi et al., 2019).

Original: The time embedding allows the decoder to be time-aware; hence, possibly output different predictions for the same nodes/edges at different timestamps.
Revised: The time embedding allows the decoder to be time-aware; hence, possibly output different predictions for the same nodes/edges at different timestamps.

Original: For SSL pre-training, the predictor p ϕ is a simple 2-layer MLP that maps node embeddings H to node representations Z .
Revised: For SSL pre-training, the predictor p ϕ is a simple 2-layer MLP that maps node embeddings H to node representations Z .

Original: Distortion Pipeline :
Revised: Distortion Pipeline :

Original: Following static graph SSL methods such as Thakoor et al.
Revised: Following static graph SSL methods such as Thakoor et al.

Original: , we use edge dropout and edge feature dropout as distortions.
Revised: , we use edge dropout and edge feature dropout as distortions.

Original: Both distortions are applied with dropout probability p d = 0 .
Revised: Both distortions are applied with dropout probability p d = 0 .

Original: 3 which we have found to work best in a validation experiment exploring the values p d ∈ { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 3 } .
Revised: 3 which we have found to work best in a validation experiment exploring the values p d ∈ { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 3 } .

Original: The edge feature dropout is applied on the temporal edge encodings introduced in Section 4.3, i.e., z p ( t p ) and c p ( t p ) .
Revised: The edge feature dropout is applied on the temporal edge encodings introduced in Section 4.3, i.e., z p ( t p ) and c p ( t p ) .

Original: More advanced temporal distortions could be explored such as time distortions or edge shuffling, but we leave those for future work.
Revised: More advanced temporal distortions could be explored such as time distortions or edge shuffling, but we leave those for future work.

Original: A. H YPER - PARAMETERS
Revised: A. H YPER - PARAMETERS

Original: Hyper-parameters : We use a constant learning rate of 0.0001 for all datasets and tasks.
Revised: Hyper-parameters : We use a constant learning rate of 0.0001 for all datasets and tasks.

Original: DyG2Vec is trained for 100 epochs for both downstream and SSL pre-training.
Revised: DyG2Vec is trained for 100 epochs for both downstream and SSL pre-training.

Original: The model from the last epoch of pre-training is used for downstream training.
Revised: The model from the last epoch of pre-training is used for downstream training.

Original: For downstream evaluation, we pick the model with the best validation AP performance.
Revised: For downstream evaluation, we pick the model with the best validation AP performance.

Original: Overall, we found that DyG2Vec converges within ∼epochs.
Revised: Overall, we found that DyG2Vec converges within ∼epochs.

Original: For downstream training, the window size W is tuned on the validation set from the range { 1 K, 4 K, 8 K, 12 K, 32 K, 65 K } .
Revised: For downstream training, the window size W is tuned on the validation set from the range { 1 K, 4 K, 8 K, 12 K, 32 K, 65 K } .

Original: The best window sizes per dataset can be found in Table 8.
Revised: The best window sizes per dataset can be found in Table 9.

Original: The target window size K is fixed to 200 during training.
Revised: The target window size K is fixed to 200 during training.

Original: The stride is always fixed to be equal to K , so that each edge is only predicted once.
Revised: The stride is always fixed to be equal to K , so that each edge is only predicted once.

Original: One could augment the dataset by changing S
Revised: One could augment the dataset by changing S

Original: but we leave this for future work.
Revised: but we leave this for future work.

Original: The batch size is always set to 1; hence, we only predict one target interval of size K at a time.
Revised: The batch size is always set to 1; hence, we only predict one target interval of size K at a time.

Original: However, the model could be sped up by increasing batch size at the cost of higher memory.
Revised: However, the model could be sped up by increasing batch size at the cost of higher memory.

Original: During SSL pre-training, we use a constant window size of 32K with stride 200 .
Revised: During SSL pre-training, we use a constant window size of 32K with stride 200 .

Original: The DyG2Vec encoder hyperparameters can be found in Table 9.
Revised: The DyG2Vec encoder hyperparameters can be found in Table 10.

Original: [Table]
Revised: [Table]

Original: Following previous work (Rossi et al., 2020; Xu et al., 2020), all dynamic node classification training experiments are performed with L2-decay parameter λ = 0 .
Revised: Following previous work (Rossi et al., 2020; Xu et al., 2020), all dynamic node classification training experiments are performed with L2-decay parameter λ = 0 .

Original: to alleviate over-fitting.
Revised: to alleviate over-fitting.

Original: A. B ASELINES
Revised: A. B ASELINES

Original: Baselines: Following prior work (Rossi et al., 2020; Xu et al., 2020), all baselines are trained with a constant learning rate of 0.0001 using the Adam optimizer (Kingma & Ba, 2014) on batch-sizefor a total of 50 epochs.
Revised: Baselines: Following prior work (Rossi et al., 2020; Xu et al., 2020), all baselines are trained with a constant learning rate of 0.0001 using the Adam optimizer (Kingma & Ba, 2014) on batch-sizefor a total of 50 epochs.

Original: The early stopping strategy is used to stop training if validation AP does not improve for 5 epochs.
Revised: The early stopping strategy is used to stop training if validation AP does not improve for 5 epochs.

Original: For JODIE (Kumar et al., 2019), DyRep (Trivedi et al., 2019), and TGN (Rossi et al., 2020), we use the general framework 3 implemented by Rossi et al.
Revised: For JODIE (Kumar et al., 2019), DyRep (Trivedi et al., 2019), and TGN (Rossi et al., 2020), we use the general framework 3 implemented by Rossi et al.

Original: The node memory dimension is set to 172.
Revised: The node memory dimension is set to 172.

Original: Other encoder hyperparameters are specified in Table 9. [Table]
Revised: Other encoder hyperparameters are specified in Table 10.

Original: For TGAT 4 , we use the default hyperparameters of 2 layer neighbor sampling with 20 neighbors sampled at each hop.
Revised: For TGAT 4 , we use the default hyperparameters of 2 layer neighbor sampling with 20 neighbors sampled at each hop.

Original: Other hyperparameters are specified in Table 9.
Revised: Other hyperparameters are specified in Table 10.

Original: For the CaW 5 method, we tune the time decay parameter α ∈ { 0 . 01 , 0 . 1 , 0 . 3 , 0 .
Revised: For the CaW 5 method, we tune the time decay parameter α ∈ { 0 . 01 , 0 . 1 , 0 . 3 , 0 .

Original: 5 , 1 , 2 , 4 , 10 } × 10 − 6 , and length of the walks m ∈ { 2 , 3 , 4 , 5 } on the validation set.
Revised: 5 , 1 , 2 , 4 , 10 } × 10 − 6 , and length of the walks m ∈ { 2 , 3 , 4 , 5 } on the validation set.

Original: The optimal hyperparameters for each dataset are specified in
Revised: The optimal hyperparameters for each dataset are specified in

Original: Table 10.
Revised: Table 11.

Original: The number of heads for the walking-based attention is fixed to 8. 
Revised: The number of heads for the walking-based attention is fixed to 8. [Table] [Table] A.

Original: [Table]
Revised: [Table]

Original: A. C OMPUTING I NFRASTRUCTURE
Revised: C OMPUTING I NFRASTRUCTURE

Original: All experiments were done on a Ubuntu 20.4 server with 72 Intel(R)
Revised: All experiments were done on a Ubuntu 20.4 server with 72 Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6140 CPU @
Revised: Gold 6140 CPU @

Original: 2.30GHz cores and a RAM of size 755 Gb.
Revised: 2.30GHz cores and a RAM of size 755 Gb.

Original: We use a NVIDIA Tesla V100-PCIE-32GB GPU.
Revised: We use a NVIDIA Tesla V100-PCIE-32GB GPU.

Revised: A. A DDITIONAL R ELATED W ORK

Revised: Self-supervised learning for dynamic graphs: Most adaptations of SSL for dynamic graphs have focused on improving downstream task performance via auxiliary losses rather than learning general pre-trained models.

Revised: Jiang et al.

Revised: (2021) adapt a sub-graph contrastive learning method (Jiao et al., 2020) where a node representation is contrasted in both structure and time.

Revised: That is, for each node in the graph, a GNN encoder is trained to contrast its real temporal subgraph to its fake temporal subgraph .

Revised: This is done by constructing a positive sample, a structural negative sample and a temporal negative sample.

Revised: The positive sample is a time-weighted subgraph representation.

Revised: The margin triplet loss is proposed to maximize the mutual information with the positive sample while maximizing distance with the structural and temporal negative samples.

Revised: Experiments on downstream link prediction task under the freeze setting show improvement over baselines.

Revised: However, their approach comes with several shortcomings.

Revised: First, initial node features are computed as one-hot encodings which makes the method not suitable for the inductive scenario (i.e. predicting on new nodes).

Revised: Second, the use of contrastive learning method is known to result in high memory and computation due to negative sampling (Thakoor et al., 2022).

Revised: This makes the method less desirable for large-scale graphs.

Revised: Third, they do not include results on other downstream tasks (e.g. dynamic node classification).

Revised: Lastly, they do not compare to the SoTA CaW method (Wang et al., 2021b).

Revised: Cong et al.

Revised: (2022) propose the dynamic graph transformer (DGT) which is a transformer-based graph encoder for discrete-time dynamic graphs .

Revised: DGT is composed of two-tower networks that embed the temporal evolution and topological information of the input graph.

Revised: Moreover, a temporalunion graph structure is proposed to efficiently summarize the temporal evolution into one graph.

Revised: DGT is trained to encode the temporal-union graph using two complementary self-supervised pretext tasks.

Revised: Namely, temporal reconstruction and multi-view contrasting.

Revised: The first aims to reconstruct a snapshot given the past and present similar to how language models are trained.

Revised: On the other hand, the latter is trained via non-contrastive learning on two views with randomly masked nodes.

Revised: All together, DGT outperforms SOTA discrete-time baselines on several datasets for link prediction tasks.

Revised: While they operate in a different domain, an interesting direction for future work would be to adapt their pre-training strategy for continuous-time dynamic graphs.

Revised: [Table]

Revised: Tian et al.

Revised: (2021) adapt the TGAT encoder with a self-supervised contrastive framework across time.

Revised: That is, they propose an extension to the classic contrastive learning paradigm by contrasting two nearby temporal views of the same node using a time-dependent similarity metric.

Revised: Moreover, a de-basied contrastive loss is utilized to correct the typical negative sampling bias in contrastive learning.

Revised: Experiments on the fine-tune and mutli-task learning settings show that the simple TGAT encoder can be significantly improved on both future link prediction and dynamic node classification.

Revised: Nonetheless, their approach comes with several shortcomings.

Revised: First, it is built on the TGAT encoder which, as seen in Tables 1 and 5, is a weak encoder; particularly, for large datasets.

Revised: Second, experiments for the FLP task are limited to the Reddit and Wikipedia datasets which are relatively easy.

Revised: Lastly, the authors do not experiment under the standard settings in graph SSL literature such as the freeze and semi-supervised settings.

Revised: Table 12 shows the results for downstream future link prediction under the freeze setting.

Revised: The results show up to 10% gap compared to DyG2Vec, particularly for datasets where the TGAT encoder under-performs (e.g. Enron, UCI).

Revised: More encoders for temporal graphs : Souza et al.

Revised: (2022) is a concurrent work that establishes a series of theoretical results on temporal graph encoders.

Revised: Their analysis exposes several weakness of both memory-based methods (e.g. TGN) and walk-based methods (e.g. CaW).

Revised: Given these insights, they propose PINT, a memory-based method that leverages injective message-passing and novel relative positional encodings.

Revised: The relative positional encodings count how many temporal walks of a given length exist between two nodes.

Revised: Experiments show significant improvement over

Revised: SoTA baselines on the link prediction task.

Revised: An interesting direction for future research would be to evaluate the expressive power of DyG2Vec compared to baselines using their theoretical framework (e.g. temporal WL test).

Revised: Wang et al.

Revised: (2021a) adapt the vanilla transformer architecture to dynamic graphs by designing a twostream encoder that extracts temporal and structural information from the temporal neighborhoods associated with any two interaction nodes.

Revised: Rather than treating link prediction as a binary classification task, the authors leverage a contrastive learning strategy that maximizes the mutual information between the representations of future interaction nodes.

Revised: Experiments show improved performance on future link prediction due to the more robust contrastive training strategy.

Revised: Nonetheless, the paper does not compare to the SoTA CaW method (Wang et al., 2021b).

Revised: Moreover, experiments are limited to the future link prediction task.