Original: T IME - VARYING G
Revised: T IME - VARYING G

Original: RAPH
Revised: RAPH

Original: R
Revised: R

Original: EPRESENTATION L EARNING
Revised: EPRESENTATION L EARNING

Original: VIA
Revised: VIA

Original: H IGHER -O RDER
Revised: H IGHER -O RDER

Original: S KIP -G RAM
Revised: S KIP -G RAM

Original: WITH
Revised: WITH

Original: N EGATIVE
Revised: N EGATIVE

Original: S AMPLING
Revised: S AMPLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Representation learning models for graphs are a successful family of techniques that project nodes into feature spaces that can be exploited by other machine learning algorithms.
Revised: Representation learning models for graphs are a successful family of techniques that project nodes into feature spaces that can be exploited by other machine learning algorithms.

Original: Since many real-world networks are inherently dynamic, with interactions among nodes changing over time, these techniques can be deﬁned both for static and for time-varying graphs.
Revised: Since many real-world networks are inherently dynamic, with interactions among nodes changing over time, these techniques can be deﬁned both for static and for time-varying graphs.

Original: Here, we show how the skip-gram embedding approach can be used to perform implicit tensor factorization on different tensor representations of time-varying graphs.
Revised: Here, we show how the skip-gram embedding approach can be used to perform implicit tensor factorization on different tensor representations of time-varying graphs.

Original: We show that higher-order skip-gram with negative sampling (HOSGNS) is able to disentangle the role of nodes and time, with a small fraction of the number of parameters needed by other approaches.
Revised: We show that higher-order skip-gram with negative sampling (HOSGNS) is able to disentangle the role of nodes and time, with a small fraction of the number of parameters needed by other approaches.

Original: We empirically evaluate our approach using time-resolved face-to-face proximity data, showing that the learned representations outperform state-of-the-art methods when used to solve downstream tasks such as network reconstruction.
Revised: We empirically evaluate our approach using time-resolved face-to-face proximity data, showing that the learned representations outperform state-of-the-art methods when used to solve downstream tasks such as network reconstruction.

Original: Good performance on predicting the outcome of dynamical processes such as disease spreading shows the potential of this new method to estimate contagion risk, providing early risk awareness based on contact tracing data.
Revised: Good performance on predicting the outcome of dynamical processes such as disease spreading shows the potential of this new method to estimate contagion risk, providing early risk awareness based on contact tracing data.

Original: The source code and data are publicly available at [link to anonymized repository].
Revised: The source code and data are publicly available at [link to anonymized repository].

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A great variety of natural and artiﬁcial systems can be represented as networks of elementary structural entities coupled by relations between them.
Revised: A great variety of natural and artiﬁcial systems can be represented as networks of elementary structural entities coupled by relations between them.

Original: The abstraction of such systems as networks helps us understand, predict and optimize their behaviour (Newman, 2003; Albert & Barabási, 2002).
Revised: The abstraction of such systems as networks helps us understand, predict and optimize their behaviour (Newman, 2003; Albert & Barabási, 2002).

Original: In this sense, node and graph embeddings have been established as standard feature representations in many learning tasks (Cai et al., 2018; Goyal & Ferrara, 2018).
Revised: In this sense, node and graph embeddings have been established as standard feature representations in many learning tasks (Cai et al., 2018; Goyal & Ferrara, 2018).

Original: Node embedding methods map nodes into low-dimensional vectors that can be used to solve downstream tasks such as edge prediction, network reconstruction and node classiﬁcation.
Revised: Node embedding methods map nodes into low-dimensional vectors that can be used to solve downstream tasks such as edge prediction, network reconstruction and node classiﬁcation.

Original: Node embeddings have proven successful in achieving low-dimensional encoding of static network structures, but many real-world networks are inherently dynamic (Holme & Saramäki, 2012).
Revised: Node embeddings have proven successful in achieving low-dimensional encoding of static network structures, but many real-world networks are inherently dynamic (Holme & Saramäki, 2012).

Original: Timeresolved networks are also the support of important dynamical processes, such as epidemic or rumor spreading, cascading failures, consensus formation, etc.
Revised: Timeresolved networks are also the support of important dynamical processes, such as epidemic or rumor spreading, cascading failures, consensus formation, etc.

Original: (Barrat et al., 2008).
Revised: (Barrat et al., 2008).

Original: Time-resolved node embeddings have been shown to yield improved performance for predicting the outcome of dynamical processes over networks, such as information diffusion and disease spreading (Sato et al., 2019), providing estimation of infection and contagion risk when used with contact tracing data.
Revised: Time-resolved node embeddings have been shown to yield improved performance for predicting the outcome of dynamical processes over networks, such as information diffusion and disease spreading (Sato et al., 2019), providing estimation of infection and contagion risk when used with contact tracing data.

Original: Since we expect having more data on proximity networks being used for contact tracing and as proxies for epidemic risk (Alsdurf et al., 2020), learning meaningful representations of time-resolved proximity networks can be of extreme importance when facing events such as epidemic outbreaks (Kapoor et al., 2020; Gao et al., 2020).
Revised: Since we expect having more data on proximity networks being used for contact tracing and as proxies for epidemic risk (Alsdurf et al., 2020), learning meaningful representations of time-resolved proximity networks can be of extreme importance when facing events such as epidemic outbreaks (Kapoor et al., 2020; Gao et al., 2020).

Original: The manual and automatic collection of time-resolved proximity graphs for contact tracing purposes presents an opportunity for quick identiﬁcation of possible infection clusters and infection chains.
Revised: The manual and automatic collection of time-resolved proximity graphs for contact tracing purposes presents an opportunity for quick identiﬁcation of possible infection clusters and infection chains.

Original: Even before the COVID-19 pandemic, the use of wearable proximity sensors for collecting time-resolved proximity networks has been largely discussed in the literature and many approaches have been used to describe patterns of activity and community structure, and to study spreading patterns of infectious diseases (Sapienza et al., 2015;
Revised: Even before the COVID-19 pandemic, the use of wearable proximity sensors for collecting time-resolved proximity networks has been largely discussed in the literature and many approaches have been used to describe patterns of activity and community structure, and to study spreading patterns of infectious diseases (Sapienza et al., 2015;

Original: Gauvin et al., 2014; Génois et al., 2015).
Revised: Gauvin et al., 2014; Génois et al., 2015).

Original: Here we propose a representation learning model that performs implicit tensor factorization on different higher-order representations of time-varying graphs.
Revised: Here we propose a representation learning model that performs implicit tensor factorization on different higher-order representations of time-varying graphs.

Original: The main contributions are as follows:
Revised: The main contributions are as follows:

Original: Given that the skip-gram embedding approach implicitly performs a factorization of the shifted pointwise mutual information matrix (PMI) (Levy & Goldberg, 2014), we generalize it to perform implicit factorization of a shifted PMI tensor.
Revised: Given that the skip-gram embedding approach implicitly performs a factorization of the shifted pointwise mutual information matrix (PMI) (Levy & Goldberg, 2014), we generalize it to perform implicit factorization of a shifted PMI tensor.

Original: We then deﬁne the steps to achieve this factorization using higher-order skip-gram with negative sampling (HOSGNS) optimization.
Revised: We then deﬁne the steps to achieve this factorization using higher-order skip-gram with negative sampling (HOSGNS) optimization.

Original: We show how to apply 3rd-order and 4th-order SGNS on different higher-order representations of time-varying graphs.
Revised: We show how to apply 3rd-order and 4th-order SGNS on different higher-order representations of time-varying graphs.

Original: Finally, we show that time-varying graph representations learned via HOSGNS outperform stateof-the-art methods when used to solve downstream tasks, even using a fraction of the number of embedding parameters.
Revised: Finally, we show that time-varying graph representations learned via HOSGNS outperform stateof-the-art methods when used to solve downstream tasks, even using a fraction of the number of embedding parameters.

Original: We report the results of learning embeddings on empirical time-resolved face-to-face proximity data and using such representations as predictors for solving two different tasks: network reconstruction and predicting the outcomes of a SIR spreading process over the time-varying graph.
Revised: We report the results of learning embeddings on empirical time-resolved face-to-face proximity data and using such representations as predictors for solving two different tasks: network reconstruction and predicting the outcomes of a SIR spreading process over the time-varying graph.

Original: We compare these results with state-of-the art methods for time-varying graph representation learning.
Revised: We compare these results with state-of-the art methods for time-varying graph representation learning.

Original:  P RELIMINARIES AND R ELATED W ORK
Revised:  P RELIMINARIES AND R ELATED W ORK

Original: Skip-gram representation learning.
Revised: Skip-gram representation learning.

Original: The skip-gram model was designed to compute word embeddings in WORD 2 VEC (Mikolov et al., 2013), and afterwards extended to graph node embeddings (Perozzi et al., 2014; Tang et al., 2015; Grover & Leskovec, 2016).
Revised: The skip-gram model was designed to compute word embeddings in WORD 2 VEC (Mikolov et al., 2013), and afterwards extended to graph node embeddings (Perozzi et al., 2014; Tang et al., 2015; Grover & Leskovec, 2016).

Original: Levy & Goldberg (2014) established the relation between skip-gram trained with negative sampling (SGNS) and traditional low-rank approximation methods (Kolda & Bader, 2009; Anandkumar et al., 2014), showing the equivalence of SGNS optimization to factorizing a shifted PMI matrix (Church & Hanks, 1990).
Revised: Levy & Goldberg (2014) established the relation between skip-gram trained with negative sampling (SGNS) and traditional low-rank approximation methods (Kolda & Bader, 2009; Anandkumar et al., 2014), showing the equivalence of SGNS optimization to factorizing a shifted PMI matrix (Church & Hanks, 1990).

Original: This equivalence was later retrieved from diverse assumptions (Assylbekov & Takhanov, 2019; Allen et al., 2019; Melamud & Goldberger, 2017; Arora et al., 2016; Li et al., 2015), and exploited to compute closed form expressions approximated in different graph embedding models (Qiu et al., 2018).
Revised: This equivalence was later retrieved from diverse assumptions (Assylbekov & Takhanov, 2019; Allen et al., 2019; Melamud & Goldberger, 2017; Arora et al., 2016; Li et al., 2015), and exploited to compute closed form expressions approximated in different graph embedding models (Qiu et al., 2018).

Original: In this work, we refer to the shifted PMI matrix also as SPMI κ =
Revised: In this work, we refer to the shifted PMI matrix also as SPMI κ =

Original: PMI − log κ , where κ is the number of negative samples.
Revised: PMI − log κ , where κ is the number of negative samples.

Original: Random walk based graph embeddings.
Revised: Random walk based graph embeddings.

Original: Given an undirected, weighted and connected graph
Revised: Given an undirected, weighted and connected graph

Original: G =
Revised: G =

Original: ( V , E ) with nodes
Revised: ( V , E ) with nodes

Original: i, j ∈ V , edges ( i, j ) ∈ E and adjacency matrix A , graph embedding methods are unsupervised models designed to map nodes into dense d -dimensional representations ( d (cid:28) |V| ) (Hamilton et al., 2017).
Revised: i, j ∈ V , edges ( i, j ) ∈ E and adjacency matrix A , graph embedding methods are unsupervised models designed to map nodes into dense d -dimensional representations ( d (cid:28) |V| ) (Hamilton et al., 2017).

Original: A well known family of approaches based on the skip-gram model consists in sampling random walks from the graph and processing node sequences as textual sentences.
Revised: A well known family of approaches based on the skip-gram model consists in sampling random walks from the graph and processing node sequences as textual sentences.

Original: In D EEP W ALK (Perozzi et al., 2014) and NODE 2 VEC (Grover & Leskovec, 2016), the skip-gram model is used to obtain node embeddings from co-occurrences in random walk realizations.
Revised: In D EEP W ALK (Perozzi et al., 2014) and NODE 2 VEC (Grover & Leskovec, 2016), the skip-gram model is used to obtain node embeddings from co-occurrences in random walk realizations.

Original: Although the original implementation of D EEP W ALK uses hierarchical softmax to compute embeddings, we will refer to the SGNS formulation given by Qiu et al.
Revised: Although the original implementation of D EEP W ALK uses hierarchical softmax to compute embeddings, we will refer to the SGNS formulation given by Qiu et al.

Original: Since SGNS can be interpreted as a factorization of the word-context PMI matrix (Levy & Goldberg, 2014), the asymptotic form of the PMI matrix implicitly decomposed in D EEP W ALK can be derived (Qiu et al., 2018).
Revised: Since SGNS can be interpreted as a factorization of the word-context PMI matrix (Levy & Goldberg, 2014), the asymptotic form of the PMI matrix implicitly decomposed in D EEP W ALK can be derived (Qiu et al., 2018).

Original: Given the 1-step transition matrix P = D − 1 A , where D = diag ( d 1 , . . .
Revised: Given the 1-step transition matrix P = D − 1 A , where D = diag ( d 1 , . . .

Original: , d |V| ) and d i = (cid:80)
Revised: , d |V| ) and d i = (cid:80)

Original: j ∈V A ij is the (weighted) node degree, the expected PMI for a node-context pair ( i, j ) occurring in a T -sized window is:
Revised: j ∈V A ij is the (weighted) node degree, the expected PMI for a node-context pair ( i, j ) occurring in a T -sized window is:

Original: [Equation]
Revised: [Equation]

Original: .1) where p ∗ ( i ) =
Revised: .1) where p ∗ ( i ) =

Original: d i vol( G ) is the unique stationary distribution for random walks (Masuda et al., 2017).
Revised: d i vol( G ) is the unique stationary distribution for random walks (Masuda et al., 2017) and vol( G ) =

Revised: (cid:80) i,j ∈V

Original:  We will use this expression in Section 3.2 to build PMI tensors from higher-order graph representations.
Revised: A ij . We will use this expression in Section 3.2 to build PMI tensors from higher-order graph representations.

Original: Time-varying graphs and their algebraic representations.
Revised: Time-varying graphs and their algebraic representations.

Original: Time-varying graphs (Holme &
Revised: Time-varying graphs (Holme &

Original: Saramäki, 2012) are deﬁned as triples H =
Revised: Saramäki, 2012) are deﬁned as triples H =

Original: ( V , E , T ) , i.e. collections of events ( i, j, k ) ∈ E , representing undirected pairwise relations among nodes at discrete times ( i, j ∈ V , k ∈ T ).
Revised: ( V , E , T ) , i.e. collections of events ( i, j, k ) ∈ E , representing undirected pairwise relations among nodes at discrete times ( i, j ∈ V , k ∈ T ).

Original: H can be seen as a temporal sequence of static adjacency matrices { A ( k ) } k ∈T  such that A ( k ) ij = ω
Revised: H can be seen as a temporal sequence of static graphs {G ( k ) } k ∈T , each of those with adjacency matrix A ( k ) such that A ( k ) ij = ω

Original: ( i, j
Revised: ( i, j

Original: , k ) ∈
Revised: , k ) ∈

Original: R is the weight of the event ( i, j, k ) ∈ E .
Revised: R is the weight of the event ( i, j, k ) ∈ E .

Original: We can concatenate the list of time-stamped snapshots [ A (1) , . . .
Revised: We can concatenate the list of time-stamped snapshots [ A (1) , . . .

Original: , A ( |T | ) ] to obtain a single 3rd-order tensor A stat ( H ) ∈ R |V|×|V|×|T | which characterize the evolution of the graph over time.
Revised: , A ( |T | ) ] to obtain a single 3rd-order tensor A stat ( H ) ∈ R |V|×|V|×|T | which characterize the evolution of the graph over time.

Original: This representation has been used to discover latent community structures of temporal graphs (Gauvin et al., 2014) and to perform temporal link prediction (Dunlavy et al., 2011).
Revised: This representation has been used to discover latent community structures of temporal graphs (Gauvin et al., 2014) and to perform temporal link prediction (Dunlavy et al., 2011).

Original: Indeed, beyond the above stacked graph representation, more exhaustive representations are possible.
Revised: Indeed, beyond the above stacked graph representation, more exhaustive representations are possible.

Original: In particular, the multi-layer approach (De Domenico et al., 2013) allows to map the topology of a time-varying graph H into a static network G H =
Revised: In particular, the multi-layer approach (De Domenico et al., 2013) allows to map the topology of a time-varying graph H into a static network G H =

Original: ( V H , E H )
Revised: ( V H , E H )

Original: (the supra-adjacency graph) such that vertices of G H correspond to pairs ( i, k ) ≡ i ( k ) ∈
Revised: (the supra-adjacency graph) such that vertices of G H correspond to pairs ( i, k ) ≡ i ( k ) ∈

Original: V × T of the original time-dependent network.
Revised: V × T of the original time-dependent network.

Original: This representation can be stored in a 4th-order tensor A dyn ( H ) ∈ R |V|×|V|×|T |×|T | equivalent, up to an opportune reshaping, to the adjacency matrix A ( G H ) ∈ R |V||T |×|V||T | associated to G H .
Revised: This representation can be stored in a 4th-order tensor A dyn ( H ) ∈ R |V|×|V|×|T |×|T | equivalent, up to an opportune reshaping, to the adjacency matrix A ( G H ) ∈ R |V||T |×|V||T | associated to G H .

Original: Multi-layer representations for time-varying networks have been used to study time-dependent centrality measures (Taylor et al., 2019) and properties of spreading processes (Valdano et al., 2015).
Revised: Multi-layer representations for time-varying networks have been used to study time-dependent centrality measures (Taylor et al., 2019) and properties of spreading processes (Valdano et al., 2015).

Original: Time-varying graph representation learning.
Revised: Time-varying graph representation learning.

Original: Given a time-varying graph H =
Revised: Given a time-varying graph H =

Original: ( V , E , T ) , we deﬁne as temporal network embedding a model that learns from data, implicitly or explicitly, a mapping function:
Revised: ( V , E , T ) , we deﬁne as temporal network embedding a model that learns from data, implicitly or explicitly, a mapping function:

Original: [Equation]
Revised: [Equation]

Original: which project time-stamped nodes into a latent low-rank vector space that encodes structural and temporal properties of the original evolving graph.
Revised: which project time-stamped nodes into a latent low-rank vector space that encodes structural and temporal properties of the original evolving graph

Original: Many existing methods learn node representations from sequences of static snapshots through incremental updates in a streaming scenario: deep autoencoders (Goyal et al., 2017), SVD (Zhang et al., 2018), skip-gram (Du et al., 2018) and random walk sampling (Béres et al., 2019; Mahdavi et al., 2018; Yu et al., 2018).
Revised: Many existing methods learn node representations from sequences of static snapshots through incremental updates in a streaming scenario: deep autoencoders (Goyal et al., 2017), SVD (Zhang et al., 2018), skip-gram (Du et al., 2018) and random walk sampling (Béres et al., 2019; Mahdavi et al., 2018; Yu et al., 2018).

Original: Another class of models learn dynamic node representations by recurrent/attention mechanisms (Goyal et al., 2020; Li et al., 2018; Sankar et al., 2020; Xu et al., 2020) or by imposing temporal stability among adjacent time intervals (Zhou et al., 2018; Zhu et al., 2016).
Revised: Another class of models learn dynamic node representations by recurrent/attention mechanisms (Goyal et al., 2020; Li et al., 2018; Sankar et al., 2020; Xu et al., 2020) or by imposing temporal stability among adjacent time intervals (Zhou et al., 2018; Zhu et al., 2016).

Original: D Y ANE (Sato et al., 2019) and WEG 2 VEC (Torricelli et al., 2020) project the dynamic graph structure into a static graph, in order to compute embeddings with WORD 2 VEC .
Revised: D Y ANE (Sato et al., 2019) and WEG 2 VEC (Torricelli et al., 2020) project the dynamic graph structure into a static graph, in order to compute embeddings with WORD 2 VEC .

Original: Closely related to these ones are Nguyen et al.
Revised: Closely related to these ones are Nguyen et al.

Original: (2018) and Zhan et al.
Revised: (2018) and Zhan et al.

Original: (2020), which learn node vectors according to time-respecting random walks or spreading trajectory paths.
Revised: (2020), which learn node vectors according to time-respecting random walks or spreading trajectory paths.

Original: Moreover
Revised: Moreover,

Original: Kumar et al.
Revised: Kumar et al.

Original: (2019) proposed an embedding framework for user-item dynamical interactions.
Revised: (2019) proposed an embedding framework for user-item temporal interactions, and

Revised: Malik et al.

Revised: (2020) suggested a tensor-based convolutional architecture for dynamic graphs.

Original: Methods that perform well for predicting outcomes of spreading processes make use of timerespecting supra-adjacency representations such as the one proposed by Valdano et al.
Revised: Methods that perform well for predicting outcomes of spreading processes make use of timerespecting supra-adjacency representations such as the one proposed by Valdano et al.

Original: In this representation, random itineraries correspond to temporal paths of the original time-varying graph.
Revised: In this representation, random itineraries correspond to temporal paths of the original time-varying graph.

Original: The supra-adjacency representation G H that we refer in Section 3.2, also used in D Y ANE, with adjacency matrix A ( G H ) , is deﬁned by two rules:
Revised: The supra-adjacency representation G H that we refer in Section 3.2, also used in D Y ANE, with adjacency matrix A ( G H ) , is deﬁned by two rules:

Original: For each event ( i, j, t 0 ) , if i is also active at time t 1 > t 0 and in no other time-stamp between the two, we add a cross-coupling edge between supra-adjacency nodes j ( t 0 ) and i ( t 1 ) .
Revised: For each event ( i, j, t 0 ) , if i is also active at time t 1 > t 0 and in no other time-stamp between the two, we add a cross-coupling edge between supra-adjacency nodes j ( t 0 ) and i ( t 1 ) .

Original: In addition, if the next interaction of j with other nodes happens at t 2 > t 0 , we add an edge between i ( t 0 ) and j ( t 2 ) .
Revised: In addition, if the next interaction of j with other nodes happens at t 2 > t 0 , we add an edge between i ( t 0 ) and j ( t 2 ) .

Original: The weights of such edges are set to ω ( i, j, t 0 ) .
Revised: The weights of such edges are set to ω ( i, j, t 0 ) .

Original: For every case as described above, we also add self-coupling edges ( i ( t 0 ) , i ( t 1 ) ) and ( j ( t 0 ) , j ( t 2 ) ) , with weights set to 1. 
Revised: For every case as described above, we also add self-coupling edges ( i ( t 0 ) , i ( t 1 ) ) and ( j ( t 0 ) , j ( t 2 ) ) , with weights set to 1. ttti j k j ttti k self-coupling cross-coupling

Revised: Figure 1 shows the differences between a time-varying graph and its time-aware supra-adjacency representation, according to the formulation described above.

Original: D Y ANE computes, given a node i ∈ V , one vector representation for each time-stamped node i ( t )∈
Revised: D Y ANE computes, given a node i ∈ V , one vector representation for each time-stamped node i ( t )

Original:  V ( T ) =
Revised: ∈ V ( T ) =

Original: { ( i, t ) ∈ V × T : ∃ ( i, j, t ) ∈ E} of this supra-adjacency representation.
Revised: { ( i, t ) ∈ V ×T : ∃ ( i, j, t ) ∈ E} of this supra-adjacency representation.

Original: Similar models that learn time-resolved node representations require a quantity O ( |V| × |T | ) of embedding parameters to represent the system in the latent space.
Revised: Similar models that learn time-resolved node representations require a quantity O ( |V| · |T | ) of embedding parameters to represent the system in the latent space.

Original: Compared with these methods, our approach requires a quantity O ( |V| + |T | ) of embedding parameters for disentangled node and time representations.
Revised: Compared with these methods, our approach requires a quantity O ( |V| + |T | ) of embedding parameters for disentangled node and time representations.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: Given a time-varying graph H =
Revised: Given a time-varying graph H =

Original: ( V , E , T ) , we propose a representation learning method that learns disentangled representations for nodes and time slices.
Revised: ( V , E , T ) , we propose a representation learning method that learns disentangled representations for nodes and time slices.

Original: More formally, we learn a function:
Revised: More formally, we learn a function:

Original: [Equation]
Revised: [Equation]

Original: through a number of parameters proportional to O ( |V| + |T | ) .
Revised: through a number of parameters proportional to O ( |V| + |T | ) .

Original: This embedding representation can then be reconciled with the deﬁnition in Eq.
Revised: This embedding representation can then be reconciled with the deﬁnition in Eq.

Original: (2.2) by combining v and t in a single v ( t ) representation using any combination function c : ( v
Revised: (2.2) by combining v and t in a single v ( t ) representation using any combination function c : ( v

Original: , t ) ∈ R d × R d (cid:55)→ v ( t ) ∈ R d .
Revised: , t ) ∈ R d × R d (cid:55)→ v ( t ) ∈ R d .

Original: Starting from the existing skip-gram framework for node embeddings, we propose a higher-order generalization of skip-gram with negative sampling (HOSGNS) applied to time-varying graphs.
Revised: Starting from the existing skip-gram framework for node embeddings, we propose a higher-order generalization of skip-gram with negative sampling (HOSGNS) applied to time-varying graphs.

Original: We show that it allows to implicitly factorize higher-order relations that characterize tensor representations of time-varying graphs, in the same way that the classical SGNS decomposes dyadic relations associated to a static graph.
Revised: We show that it allows to implicitly factorize higher-order relations that characterize tensor representations of time-varying graphs, in the same way that the classical SGNS decomposes dyadic relations associated to a static graph.

Original: Similar approaches have been applied in NLP for dynamic word embeddings (Rudolph & Blei, 2018), and higher-order extensions of the skip-gram model have been proposed to learn context-dependent (Liu et al., 2015) and syntactic-aware (Cotterell et al., 2017) word representations.
Revised: Similar approaches have been applied in NLP for dynamic word embeddings (Rudolph & Blei, 2018), and higher-order extensions of the skip-gram model have been proposed to learn context-dependent (Liu et al., 2015) and syntactic-aware (Cotterell et al., 2017) word representations.

Original: Moreover tensor factorization techniques have been applied to include the temporal dimension in recommender systems (Xiong et al., 2010; Wu et al.,  2019) and face-to-face contact networks (Sapienza et al., 2015; Gauvin et al., 2014).
Revised: Moreover tensor factorization techniques have been applied to include the temporal dimension in recommender systems (Xiong et al., 2010; Wu et al., 2019), knowledge graphs (Lacroix et al., 2020; Ma et al., 2019) and face-to-face contact networks (Sapienza et al., 2015; Gauvin et al., 2014).

Original: But this work is the ﬁrst to merge SGNS with tensor factorization, and then apply it to learn time-varying graph embeddings.
Revised: But this work is the ﬁrst to merge SGNS with tensor factorization, and then apply it to learn time-varying graph embeddings.

Original: 3. H IGHER - ORDER SKIP - GRAM WITH NEGATIVE SAMPLING AS IMPLICIT TENSOR FACTORIZATION
Revised: 3. H IGHER - ORDER SKIP - GRAM WITH NEGATIVE SAMPLING AS IMPLICIT TENSOR FACTORIZATION

Original: Here we address the problem of generalizing SGNS to learn embedding representations from higherorder co-occurrences.
Revised: Here we address the problem of generalizing SGNS to learn embedding representations from higherorder co-occurrences.

Original: We analyze here the 3rd-order case, giving the description of the general
Revised: We analyze here the 3rd-order case, giving the description of the general

Original: N -order case in the Supplementary Information.
Revised: N -order case in the Supplementary Information.

Original: Later in this work we will focus 3rd and 4th order representations since these are the most interesting for time-varying graphs.
Revised: Later in this work we will focus 3rd and 4th order representations since these are the most interesting for time-varying graphs.

Original: We consider a set of training samples D
Revised: We consider a set of training samples D

Original: = { ( i, j, k ) , i ∈ W , j ∈ C , k ∈ T } obtained by collecting co-occurrences among elements from three sets
Revised: = { ( i, j, k ) , i ∈ W , j ∈ C , k ∈ T } obtained by collecting co-occurrences among elements from three sets

Original: W , C and T .
Revised: W , C and T .

Original: While SGNS is limited to pairs of node- context ( i, j ) , here
Revised: While SGNS is limited to pairs of node- context ( i, j ) , here

Original: D is constructed with three (or more) variables, e.g. sampling random walks over a higher-order data structure.
Revised: D is constructed with three (or more) variables, e.g. sampling random walks over a higher-order data structure.

Original: We denote as #( i, j, k ) the number of times the triple ( i, j, k ) appears in
Revised: We denote as #( i, j, k ) the number of times the triple ( i, j, k ) appears in

Original: D .
Revised: D .

Original: Similarly we use # i =
Revised: Similarly we use # i =

Original: (cid:80) j,k #
Revised: (cid:80) j,k #

Original: ( i, j, k ) , # j = (cid:80) i,k
Revised: ( i, j, k ) , # j = (cid:80) i,k

Original: #( i, j, k ) and # k = (cid:80) i,j #( i, j, k ) as the number of times each distinct element occurs in D , with relative frequencies P D ( i, j, k )
Revised: #( i, j, k ) and # k = (cid:80) i,j #( i, j, k ) as the number of times each distinct element occurs in D , with relative frequencies P D ( i, j, k )

Original: =
Revised: =

Original: #( i,j,k ) |D| ,
Revised: #( i,j,k ) |D| ,

Original: P D ( i ) =
Revised: P D ( i ) =

Original: # i |D| , P D ( j ) =
Revised: # i |D| , P D ( j ) =

Original: # j |D| and P D ( k ) =
Revised: # j |D| and P D ( k ) =

Original: # k |D| .
Revised: # k |D| .

Original: Optimization is performed as a binary classiﬁcation task, where the objective is to discern occurrences actually coming from D from random occurrences.
Revised: Optimization is performed as a binary classiﬁcation task, where the objective is to discern occurrences actually coming from D from random occurrences.

Original: We deﬁne the likelihood for a single observation ( i, j, k ) by applying a sigmoid ( σ ( x ) =
Revised: We deﬁne the likelihood for a single observation ( i, j, k ) by applying a sigmoid ( σ ( x ) =

Original: (1 + e − x ) − 1 ) to the higher-order inner product [[ · ]] of corresponding d -dimensional representations:
Revised: (1 + e − x ) − 1 ) to the higher-order inner product [[ · ]] of corresponding d -dimensional representations:

Original: [Equation]
Revised: [Equation]

Original: where embedding vectors w i , c j , t k ∈ R
Revised: where embedding vectors w i , c j , t k ∈ R

Original: d are respectively rows of W ∈ R |W|× d , C ∈ R |C|× d and T ∈ R |T |× d .
Revised: d are respectively rows of W ∈ R |W|× d , C ∈ R |C|× d and T ∈ R |T |× d .

Original: In the 4th-order case we will also have a fourth embedding matrix S ∈ R |S|× d related to a fourth set S .
Revised: In the 4th-order case we will also have a fourth embedding matrix S ∈ R |S|× d related to a fourth set S .

Original: For negative sampling we ﬁx an observed ( i, j, k ) ∈ D and independently sample j
Revised: For negative sampling we ﬁx an observed ( i, j, k ) ∈ D and independently sample j

Original: N and k N to generate κ negative examples ( i, j N , k N ) .
Revised: N and k N to generate κ negative examples ( i, j N , k N ) .

Original: In this way, for a single occurrence ( i, j, k ) ∈ D , the expected contribution to the loss is:
Revised: In this way, for a single occurrence ( i, j, k ) ∈ D , the expected contribution to the loss is:

Original: [Equation]
Revised: [Equation]

Original: where the noise distribution is the product of independent marginal probabilities P N ( j, k )
Revised: where the noise distribution is the product of independent marginal probabilities P N ( j, k )

Original: =
Revised: =

Original: P D ( j ) ·
Revised: P D ( j ) ·

Original: P D ( k ) .
Revised: P D ( k ) .

Original: Thus the global objective is the sum of all the quantities of Eq.
Revised: Thus the global objective is the sum of all the quantities of Eq.

Original: (3.2) weighted with the corresponding relative frequency P D ( i, j, k ) .
Revised: (3.2) weighted with the corresponding relative frequency P D ( i, j, k ) .

Original: The full loss function can be expressed as:
Revised: The full loss function can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: In Supplementary Information we show the formal steps to obtain Eq.
Revised: In Supplementary Information we show the formal steps to obtain Eq.

Original: (3.3) for the N -order case and that it can be optimized with respect to the embedding parameters, satisfying the low-rank tensor approximation of the multivariate shifted PMI tensor into factor matrices W , C , T :
Revised: (3.3) for the N -order case and that it can be optimized with respect to the embedding parameters, satisfying the low-rank tensor approximation of the multivariate shifted PMI tensor into factor matrices W , C , T :

Original: [Equation]
Revised: [Equation]

Original: 3. T IME - VARYING GRAPH EMBEDDING VIA HOSGNS
Revised: 3. T IME - VARYING GRAPH EMBEDDING VIA HOSGNS

Original: While a static graph G =
Revised: While a static graph G =

Original: ( V , E ) is uniquely represented by an adjacency matrix A ( G ) ∈ R |V|×|V| , a time-varying graph H =
Revised: ( V , E ) is uniquely represented by an adjacency matrix A ( G ) ∈ R |V|×|V| , a time-varying graph H =

Original: ( V , E , T ) admits diverse possible higher-order adjacency relations (Section i ( k ) Node [Table]
Revised: ( V , E , T ) admits diverse possible higher-order adjacency relations (Section 2).

Original: [Table]

Original: i w i

Original: [[ ⋅ ]] σ c j t k

Original: Context embeddings matrix,

Original: [Table]

Original: Time

Original: [Table]

Original: matrix,

Original: [Table]

Original: j k

Original: [Table]

Original: [Table]

Original: i w i ⋅ σ c j

Original: Context embeddings matrix, C j

Original: Classical SGNS

Original: HOSGNS

Original: Context-

Original: [Table]

Original: embeddings matrix, S s l l j ( l )

Original: [Table]

Original: on a higher-order representation of H =

Original: ( V , E , T ) <latexit sha1_base64="LirSwMR5Yx1+7KqOq1UjwmeB8Fg=">AAAB/HicbVDLSgMxFL3js9bXaJdugkWoUMqMFNSFUBShywp9QTuUTJq2oZkHSUYYhvorblwo4tYPceffmGlnoa0HAodz7uWeHDfkTCrL+jbW1jc2t7ZzO/ndvf2DQ/PouC2DSBDaIgEPRNfFknLm05ZiitNuKCj2XE477vQu9TuPVEgW+E0Vh9Tx8NhnI0aw0tLALPQ9rCYE86R+U2qX78vN89nALFoVaw60SuyMFCFDY2B+9YcBiTzqK8KxlD3bCpWTYKEY4XSW70eShphM8Zj2NPWxR6WTzMPP0JlWhmgUCP18hebq740Ee1LGnqsn06hy2UvF/7xepEZXTsL8MFLUJ4tDo4gjFaC0CTRkghLFY00wEUxnRWSCBSZK95XXJdjLX14l7YuKXa1cP1SLtdusjhycwCmUwIZLqEEdGtACAjE8wyu8GU/Gi/FufCxG14xspwB/YHz+AA4qk8A=</latexit> combinations w i · c j ) (SGNS) or σ ( − [[ w i , c j , t k , s l ]]) (HOSGNS) with their corresponding embedding vectors (negative sampling).

Original: Starting from these higher-order relations, we can either use them directly or use random walk realizations to build a dataset of higher-order co-occurrences.
Revised: Starting from these higher-order relations, we can either use them directly or use random walk realizations to build a dataset of higher-order co-occurrences.

Original: In the same spirit that random walk realizations give place to dyadic co-occurrences used to learn embeddings in SGNS, we use higher-order co-occurrences to learn embeddings via HOSGNS.
Revised: In the same spirit that random walk realizations give place to dyadic co-occurrences used to learn embeddings in SGNS, we use higher-order co-occurrences to learn embeddings via HOSGNS.

Original: As discussed in Section 3.1, the statistics of higher-order relations can be summarized in multivariate
Revised: As discussed in Section 3.1, the statistics of higher-order relations can be summarized in multivariate

Original: PMI tensors, which derive from proper co-occurrence probabilities among elements.
Revised: PMI tensors, which derive from proper co-occurrence probabilities among elements.

Original: Once such PMI tensors are constructed, we can again factorize them via HOSGNS.
Revised: Once such PMI tensors are constructed, we can again factorize them via HOSGNS.

Original: To show the versatility of this approach, we choose PMI tensors derived from two different types of higher-order relations:
Revised: To show the versatility of this approach, we choose PMI tensors derived from two different types of higher-order relations:

Original: A 3rd-order tensor P ( stat ) ( H ) ∈ R |V|×|V|×|T | which gather relative frequencies of nodes occurrences in temporal edges
Revised: A 3rd-order tensor P ( stat ) ( H ) ∈ R |V|×|V|×|T | which gather relative frequencies of nodes occurrences in temporal edges:

Original: [Equation]
Revised: [Equation]

Original: where vol( H ) =
Revised: where vol H ) =

Original: (cid:80) i,j,k ω ( i, j, k ) is the total weight of interactions occurring in H.
Revised: (cid:80) i,j,k ω ( i, j, k ) is the total weight of interactions occurring in H

Original: These probabilities are associated to the snapshot sequence representation A stat ( H ) =
Revised: These probabilities are associated to the snapshot sequence representation A stat ( H ) =

Original: [ A (1) , . .
Revised: [ A ( , A

Original: A ( |T |) ] and contain information about the topological structure of H .
Revised: ( |T | ] and contain information about the topological structure of H .

Original: A 4th-order tensor P ( dyn ) ( H ) ∈ R |V|×|V|×|T |×|T | , which gatheroccurrence probabilities of time-stamped nodes over
Revised: A 4th-order tensor P ( dyn ) ( H ) ∈ R |V|×|V|×|T |×|T | , which gather

Original:  random walks of the supra -adjacency graph G H (as used in
Revised: occurrence probabilities of time-stamped nodes over random walks of the supra-adjacency graph G H (as used in

Original: D Y ANE)
Revised: D Y ANE).

Original: Using the numerator of Eq.
Revised: Using the numerator of Eq.

Original: (2.1) tensor entries are given by:
Revised: (2.1) tensor entries are given by:

Original: [Equation]
Revised: [Equation]

Original: where ( ik ) and ( jl ) are lexicographic indices of the supra-adjacency matrix A ( G H ) corresponding to nodes i ( k ) and node j ( l ) .
Revised: where ( ik ) and ( jl ) are lexicographic indices of the supra-adjacency matrix A ( G H ) corresponding to nodes i ( k ) and node j ( l ) .

Original: These probabilities encode causal dependencies among temporal nodes and are correlated with dynamical properties of spreading processes.
Revised: These probabilities encode causal dependencies among temporal nodes and are correlated with dynamical properties of spreading processes.

Original: We also combined the two representations in a single tensor that is the average of P ( stat ) and P ( dyn )
Revised: We also combined the two representations in a single tensor that is the average of P ( stat ) and P ( dyn )

Original: [Equation]
Revised: [Equation]

Original: where δ kl = 

Revised: where δ kl =

Original: [ k = l ] is the Kronecker delta.
Revised: [ k = l ] is the Kronecker delta.

Original: Figure 1 summarizes the differences between graph embedding via classical SGNS and time-varying graph embedding via HOSGNS.
Revised: Figure 2 summarizes the differences between graph embedding via classical SGNS and time-varying graph embedding via HOSGNS.

Original: Here, indices ( i, j, k, l ) correspond to (node, context, time, contexttime) in a 4th-order tensor representation of H .
Revised: Here, indices ( i, j, k, l ) correspond to (node, context, time, contexttime) in a 4th-order tensor representation of H .

Original: The above tensors gather empirical probabilities P D ( i, j, k . . . )
Revised: The above tensors gather empirical probabilities P D ( i, j, k . . . )

Original: corresponding to positive examples of observable higher-order relations.
Revised: corresponding to positive examples of observable higher-order relations.

Original: The probabilities of negative examples P N ( i, j, k . . . ) can be obtained as the product of marginal distributions P D
Revised: The probabilities of negative examples P N ( i, j, k . . . ) can be i ( k ) Node

Original: ( i ) , P D ( j ) , P D ( k) . . .
Revised: [Table] [Table] i w i [[ ⋅ ]] σ c j t k

Revised: Context embeddings matrix,

Revised: [Table]

Revised: Time

Revised: [Table]

Revised: matrix,

Revised: [Table]

Revised: j k

Revised: [Table]

Revised: [Table]

Revised: i w i ⋅ σ c j

Revised: Context embeddings matrix, C j

Revised: Classical SGNS

Revised: HOSGNS

Revised: Context-

Revised: [Table]

Revised: embeddings matrix, S s l l j ( l )

Revised: [Table]

Revised: on a higher-order representation of H =

Revised: ( V , E , T ) <latexit sha1_base64="LirSwMR5Yx1+7KqOq1UjwmeB8Fg=">AAAB/HicbVDLSgMxFL3js9bXaJdugkWoUMqMFNSFUBShywp9QTuUTJq2oZkHSUYYhvorblwo4tYPceffmGlnoa0HAodz7uWeHDfkTCrL+jbW1jc2t7ZzO/ndvf2DQ/PouC2DSBDaIgEPRNfFknLm05ZiitNuKCj2XE477vQu9TuPVEgW+E0Vh9Tx8NhnI0aw0tLALPQ9rCYE86R+U2qX78vN89nALFoVaw60SuyMFCFDY2B+9YcBiTzqK8KxlD3bCpWTYKEY4XSW70eShphM8Zj2NPWxR6WTzMPP0JlWhmgUCP18hebq740Ee1LGnqsn06hy2UvF/7xepEZXTsL8MFLUJ4tDo4gjFaC0CTRkghLFY00wEUxnRWSCBSZK95XXJdjLX14l7YuKXa1cP1SLtdusjhycwCmUwIZLqEEdGtACAjE8wyu8GU/Gi/FufCxG14xspwB/YHz+AA4qk8A=</latexit> w i · c j ) (SGNS) or σ ( − [[ w i , c j , t k , s l ]]) (HOSGNS) with their corresponding embedding vectors (negative sampling).

Revised: obtained as the product of marginal distributions P D ( i ) , P D ( j ) , P D ( k ) . . .

Original: Computing exactly the objective function in Eq.
Revised: Computing exactly the objective function in Eq.

Original: (3.3) (or the 4th-order analogous) is computationally expensive, but it can be approximated by a sampling strategy: picking positive tuples according to the data distribution P D and negative ones according to independent sampling P N , HOSGNS objective can be asymptotically approximated through the optimization of the following weighted cross entropy loss:
Revised: (3.3) (or the 4th-order analogous) is computationally expensive, but it can be approximated by a sampling strategy: picking positive tuples according to the data distribution P D and negative ones according to independent sampling P N , HOSGNS objective can be asymptotically approximated through the optimization of the following weighted cross entropy loss:

Original: [Equation]
Revised: [Equation]

Original: where B is the number of the samples drawn in a training step and κ is the negative sampling constant.
Revised: where B is the number of the samples drawn in a training step and κ is the negative sampling constant.

Original: We preliminarily apply additional warm-up steps (explained in Supplementary Information) to speed-up themain training stage.
Revised: We additionally apply the warm-up steps explained in Supplementary Information to speed-up the training convergence.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For the experiments we use time-varying graphs collected by the SocioPatterns collaboration (http://www.sociopatterns.org) using wearable proximity sensors that sense the face-to-face proximity relations of individuals wearing them.
Revised: For the experiments we use time-varying graphs collected by the SocioPatterns collaboration (http://www.sociopatterns.org) using wearable proximity sensors that sense the face-to-face proximity relations of individuals wearing them.

Original: After training the proposed models (HOSGNS applied to
Revised: After training the proposed models (HOSGNS applied to

Original: P ( stat ) , P ( dyn ) or P ( stat | dyn ) ) on each dataset, embedding matrices W , C , T (and S in case of
Revised: P ( stat ) , P ( dyn ) or P ( stat | dyn ) ) on each dataset, embedding matrices W , C , T (and S in case of

Original: P ( stat ) ) are mapped to embedding vectors w i , c j , t k (and s l ) where i, j ∈ V and k, l ∈ T , and we use them to solve different downstream tasks: node classiﬁcation and temporal event reconstruction .
Revised: P ( stat ) ) are mapped to embedding vectors w i , c j , t k (and s l ) where i, j ∈ V and k, l ∈ T , and we use them to solve different downstream tasks: node classiﬁcation and temporal event reconstruction .

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL S

Revised: ETUP

Original: Datasets.
Revised: Datasets.

Original: We used publicly available datasets describing face-to-face proximity of individuals with a temporal resolution of 20 seconds (Cattuto et al., 2010), collected by the SocioPatterns collaboration in a variety of contexts: in a school (“L YON S CHOOL ”), a conference (“SFHH”), a hospital (“LH10”), a highschool (“T HIERS 13”), and in ofﬁces (“I N VS15”) (Génois & Barrat, 2018).
Revised: We performed experiments with both empirical and synthetic datasets describing face-toface proximity of individuals. We used publicly available empirical contact data collected by the SocioPatterns collaboration (Cattuto et al., 2010), with a temporal resolution of 20 seconds, in a variety of contexts: in a school (“L YON S CHOOL ”), a conference (“SFHH”), a hospital (“LH10”), a highschool (“T HIERS 13”), and in ofﬁces (“I N VS15”) (Génois & Barrat, 2018).

Original: This is currently the largest collection of open datasets sensing proximity in the same range and temporal resolution used by modern contact tracing systems.
Revised: This is currently the largest collection of open datasets sensing proximity in the same range and temporal resolution used by modern contact tracing systems.

Revised: In addition, we used social interactions data generated by the agent-based-model OpenABM-Covid19 (Hinch et al., 2020) to simulate an outbreak of COVID-19 in a urban setting.

Original: We built a time-varying graph from each dataset by aggregating the  data  on 600 seconds time windows,and neglecting those snapshots without registered interactions at that time scale.
Revised: We built a time-varying graph from each dataset, and for the empirical data we performed aggregation on 600 seconds time windows, neglecting those snapshots without registered interactions at that time scale.

Original: The weight of the link ( i, j, k ) is the number of events recorded between nodes ( i, j ) in a certain aggregated window k .
Revised: The weight of the link ( i, j, k ) is the number of events recorded between nodes ( i, j ) in a certain aggregated window k .

Revised: For synthetic data we maintained the original temporal resolution and we set links weights to 1.

Original: Table 1 shows statistics for each dataset.
Revised: Table 1 shows statistics for each dataset

Original: [Table]
Revised: [Table]

Original: 0.0148 SFHH

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Baselines.
Revised: Baselines.

Original: We compare our approach with several baseline methods from the literature of timevarying graph embeddings, which learn time-stamped node representations: (1) D Y ANE (Sato et al., 2019), which learns temporal node embeddings with D EEP W ALK , mapping a time-varying graph into a supra-adjacency representation; (2) D YN GEM (Goyal et al., 2017), a deep autoencoder architecture which dynamically reconstructs each graph snapshot initializing model weights with parameters learned in previous time frames; (3) D YNAMIC T RIAD (Zhou et al., 2018), which captures structural information and temporal patterns of nodes, modeling the triadic closure process.
Revised: We compare our approach with several baseline methods from the literature of timevarying graph embeddings, which learn time-stamped node representations: (1) D Y ANE (Sato et al., 2019), which learns temporal node embeddings with D EEP W ALK , mapping a time-varying graph into a supra-adjacency representation; (2) D YN GEM (Goyal et al., 2017), a deep autoencoder architecture which dynamically reconstructs each graph snapshot initializing model weights with parameters learned in previous time frames; (3) D YNAMIC T RIAD (Zhou et al., 2018), which captures structural information and temporal patterns of nodes, modeling the triadic closure process.

Original: Details about hyper-parameters used in each method can be found in the Supplementary Information.
Revised: Details about hyper-parameters used in each method can be found in the Supplementary Information.

Original: 4. D OWNSTREAM TASKS
Revised: 4. D OWNSTREAM TASKS

Original: Node Classiﬁcation.
Revised: Node Classiﬁcation.

Original: The aim of this task is to classify nodes in epidemic states according to a SIR epidemic process with infection rate β and recovery rate µ .
Revised: The aim of this task is to classify nodes in epidemic states according to a SIR epidemic process with infection rate β and recovery rate µ .

Original: We simulated 30 realizations of the SIR process on top of each empirical graph with different combinations of parameters ( β, µ ) .
Revised: We simulated 30 realizations of the SIR process on top of each empirical graph with different combinations of parameters ( β, µ ) .

Original: We used similar combinations of epidemic parameters and the same dynamical process to produce SIR states as described in Sato et al.
Revised: We used similar combinations of epidemic parameters and the same dynamical process to produce SIR states as described in Sato et al.

Original: Then we set a logistic regression to classify epidemic states S-I-R assigned to each active node i ( k ) during the unfolding of the spreading process.
Revised: Then we set a logistic regression to classify epidemic states S-I-R assigned to each active node i ( k ) during the unfolding of the spreading process.

Original: We combine the embedding vectors of HOSGNS using the Hadamard (element-wise) product w i ◦ t k .
Revised: We combine the embedding vectors of HOSGNS using the Hadamard (element-wise) product w i ◦ t k .

Original: We compared with dynamic node embeddings learned from baselines.
Revised: We compared with dynamic node embeddings learned from baselines.

Original: For fair comparison, all models produce time-stamped node representations with dimension d = 128 as input to the logistic regression.
Revised: For fair comparison, all models produce time-stamped node representations with dimension d = 128 as input to the logistic regression.

Original: Temporal Event Reconstruction.
Revised: Temporal Event Reconstruction.

Original: In this task, we aim to determine if an event ( i, j, k ) is in
Revised: In this task, we aim to determine if an event ( i, j, k ) is in

Original: H =
Revised: H =

Original: ( V , E , T ) , i.e., if there is an edge between nodes i and j at time k .
Revised: ( V , E , T ) , i.e., if there is an edge between nodes i and j at time k .

Original: We create a random timevarying graph H ∗ =
Revised: We create a random timevarying graph H ∗ =

Original: ( V , E ∗ , T ) with same active nodes V ( T ) and a number of |E| events that are not part of
Revised: ( V , E ∗ , T ) with same active nodes V ( T ) and a number of |E| events that are not part of

Original: E .
Revised: E .

Original: Embedding representations learned from H are used as features to train a logistic regression to predict if a given event ( i, j, k ) is in E or in E ∗ .
Revised: Embedding representations learned from H are used as features to train a logistic regression to predict if a given event ( i, j, k ) is in E or in E ∗ .

Original: We combine the embedding vectors of HOSGNS as follows: for HOSGNS ( stat ) , we use the Hadamard product w i ◦ c j ◦ t k ; for HOSGNS ( dyn ) and
Revised: We combine the embedding vectors of HOSGNS as follows: for HOSGNS ( stat ) , we use the Hadamard product w i ◦ c j ◦ t k ; for HOSGNS ( dyn ) and

Original: HOSGNS ( stat | dyn ) , we use w i ◦ c j ◦ t k ◦ s k .
Revised: HOSGNS ( stat | dyn ) , we use w i ◦ c j ◦ t k ◦ s k .

Original: For baseline methods, we aggregate vector embeddings to obtain link-level representations with binary operators ( Average , Hadamard , Weighted-L1 , WeightedL2 and Concat ) as already used in previous works (Grover & Leskovec, 2016; Tsitsulin et al., 2018).
Revised: For baseline methods, we aggregate vector embeddings to obtain link-level representations with binary operators ( Average , Hadamard , Weighted-L1 , WeightedL2 and Concat ) as already used in previous works (Grover & Leskovec, 2016; Tsitsulin et al., 2018).

Original: For fair comparison, all models are required produce event representations with dimension d = 192 .
Revised: For fair comparison, all models are required produce event representations with dimension d = 192 .

Original: Tasks were evaluated using train-test split.
Revised: Tasks were evaluated using train-test split.

Original: To avoid information leakage from training to test, we randomly split V and T in train and test sets ( V tr , V ts ) and ( T tr , T ts ) , with proportion 70% − 30% .
Revised: To avoid information leakage from training to test, we randomly split V and T in train and test sets ( V tr , V ts ) and ( T tr , T ts ) , with proportion 70% − 30% .

Original: For node classiﬁcation, only nodes in
Revised: For node classiﬁcation, only nodes in

Original: V tr at times in T tr were included in the train set, and only nodes in
Revised: V tr at times in T tr were included in the train set, and only nodes in

Original: V ts at times in T ts were included in the test set.
Revised: V ts at times in T ts were included in the test set.

Original: For temporal event reconstruction, only events with i, j ∈ V tr and k ∈ T tr were included in the train set, and only events with i, j ∈ V ts and k ∈ T ts were included in the test set.
Revised: For temporal event reconstruction, only events with i, j ∈ V tr and k ∈ T tr were included in the train set, and only events with i, j ∈ V ts and k ∈ T ts were included in the test set.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Revised: In this section we ﬁrst show downstream task performance results for the empirical datasets, leaving results for synthetic datasets to Supplementary Information.

Revised: Synthetic datasets are used here to compare the performance of the different approaches in terms of training complexity, by measuring the number of trainable parameters and the training time with ﬁxed number of training steps.

Original: All approaches were evaluated for both downstream tasks in terms of Macro-F1 scores in all datasets.
Revised: All approaches were evaluated for both downstream tasks in terms of Macro-F1 scores in all datasets.

Original: 5 different runs of the embedding model are evaluated on 30 different train-test splits for both downstream tasks.
Revised: 5 different runs of the embedding model are evaluated on 30 different train-test splits for both downstream tasks.

Original: We report the average score with standard error over all splits.
Revised: We report the average score with standard error over all splits.

Original: In node classiﬁcation, every SIR realization is assigned to a single embedding run to compute prediction scores.
Revised: In node classiﬁcation, every SIR realization is assigned to a single embedding run to compute prediction scores.

Original: In event reconstruction, a different random realization H ∗ is assigned to each train-test subset.
Revised: In event reconstruction, a different random realization H ∗ is assigned to each train-test subset.

Original: Results for the classiﬁcation of nodes in epidemic states are shown in Table 2.
Revised: Results for the classiﬁcation of nodes in epidemic states are shown in Table 2.

Original: We report here a subset of ( β, µ ) but other combinations are available on the Supplementary Information.
Revised: We report here a subset of ( β, µ ) but other combinations are available on the Supplementary Information.

Original: D YN GEM and D YNAMIC T RIAD have low scores, since they are not devised to learn from graph dynamics.
Revised: D YN GEM and D YNAMIC T RIAD have low scores, since they are not devised to learn from graph dynamics.

Original: HOSGNS ( stat ) is not able to capture the graph dynamics due to the static nature of P ( stat ) .
Revised: HOSGNS ( stat ) is not able to capture the graph dynamics due to the static nature of P ( stat ) .

Original: D Y ANE, HOSGNS ( stat | dyn ) and HOSGNS ( dyn ) show good performance in this task, with these two HOSGNS variants outperforming D Y ANE in most of the combinations of datasets and SIR parameters. 
Revised: D Y ANE, HOSGNS ( stat | dyn ) and HOSGNS ( dyn ) show good performance, with these two HOSGNS variants outperforming D Y ANE in most of the combinations of datasets and SIR parameters. [Table]

Original: Results for the temporal event reconstruction task are reported in Table 3.
Revised: Results for the temporal event reconstruction task are reported in Table 3.

Original: Temporal event reconstruction is not performed well by D YN GEM.
Revised: Temporal event reconstruction is not performed well by D YN GEM.

Original: D YNAMIC T RIAD has better performance with Weighted-L and Weighted-L2 operators, while D Y ANE has better performance using Hadamard or Weighted-L2.
Revised: D YNAMIC T RIAD has better performance with Weighted-L and Weighted-L2 operators, while D Y ANE has better performance using Hadamard or WeightedL2.

Original: Since Hadamard product is explicitly used in Eq.
Revised: Since Hadamard product is explicitly used in Eq.

Original: (3.1) to optimize HOSGNS, all HOSGNS variants show best scores with this operator.
Revised: (3.1) to optimize HOSGNS, all HOSGNS variants show best scores with this operator.

Original: HOSGNS ( stat ) outperforms all approaches, setting new state-of-the-art results in this task.
Revised: HOSGNS ( stat ) outperforms all approaches, setting new state-of-the-art results in this task.

Original: The P ( dyn ) representation used as input to HOSGNS ( dyn ) does not focus on events but on dynamics, so the performance for event reconstruction is slightly below D Y ANE, while HOSGNS ( stat | dyn ) is comparable to D Y ANE.
Revised: The P ( dyn ) representation used as input to HOSGNS ( dyn ) does not focus on events but on dynamics, so the performance for event reconstruction is slightly below D Y ANE, while HOSGNS ( stat | dyn ) is comparable to D Y ANE.

Original: Results for HOSGNS models using other operators are available in the Supplementary Information.
Revised: Results for HOSGNS models using other operators are available in the Supplementary Information.

Original: We observe an overall good performance of HOSGNS ( stat | dyn ) in both downstream tasks, being in almost all cases the second highest score, compared to the other two variants which excel in one task but have lower performance in the other. 
Revised: We observe an overall good performance of HOSGNS ( stat | dyn ) in both downstream tasks, being in almost all cases the second highest score, compared to the other two variants which excel in one task but have lower performance in the other. Training Complexity.

Original: We recall that HOSGNS, by learning disentangled representations of nodes and  time intervals, uses a number of  parameters in the order of O ( |V| + |T | ) ,
Revised: We report in Table 4 the number of trainable parameters and training time duration for each considered algorithm, when applied to an empirical graph (L YON S CHOOL ) and to the synthetic ones. The proposed HOSGNS model requires a number of trainable parameters that is orders of magnitude smaller than other approaches, with a training time considerably shorter as the number of nodes increases, given a ﬁxed number of training iterations. For this analysis, HOSGNS [Table]

Revised: sampling was implemented by picking positive and negative examples from a corpus of random walks sampled from a given graph.

Revised: For HOSGNS ( stat ) random walks are sampled from the set of temporal snapshots {G ( k ) } k ∈T with window size T =

Original: [Table]
Revised: [Table]

Original: [Table] while models that learn node-time representations (such as D Y ANE) need a number of parameters that is at least O ( |V| × |T | ) 
Revised: and for HOSGNS ( dyn ) random walks are sampled from the supra-adjacency graph G H with window size T = 10 . With these sampling strategies, positive examples are drawn from the same probability distributions as deﬁned in Eq. (3.5) and Eq.

Original: [Table]
Revised: [Table]

Original: In the Supplementary Information we include plots with two dimensional projections of these embeddings
Revised: We recall that HOSGNS, by learning disentangled representations of nodes and time intervals

Original: [Table]
Revised: [Table]

Revised: uses a number of parameters in the order of O ( |V| + |T | ) , while models that learn node-time representations (such as D Y ANE) need a number of parameters that is at least

Revised: O ( |V| · |T | ) .

Original:  showing that the embedding matrices of HOSGNS approaches successfully capture both the structure and the dynamics of the time-varying graph
Revised: In the Supplementary Information we include plots with two dimensional projections of these embeddings, showing that the embedding matrices of HOSGNS approaches successfully capture both the structure and the dynamics of the time-varying graph

Original: [Table]
Revised: [Table]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper [Table] we introduce higher-order skip-gram with negative sampling (HOSGNS) for timevarying graph representation learning.
Revised: In this paper, we introduce higher- order skip-gram with negative sampling (HOSGNS) for timevarying graph representation learning.

Original: We show that this method is able to disentangle the role of nodes and time, with a small fraction of the number of parameters needed by other methods.
Revised: We show that this method is able to disentangle the role of nodes and time, with a small fraction of the number of parameters needed by other methods.

Original: The embedding representations learned by HOSGNS outperform other methods in the literature and set new state-of-the-art results for predicting the outcome of dynamical processes and for temporal event reconstruction.
Revised: The embedding representations learned by HOSGNS outperform other methods in the literature and set new state-of-the-art results for predicting the outcome of dynamical processes and for temporal event reconstruction.

Original: We show that HOSGNS can be intuitively applied to time-varying graphs, but this methodology can be easily adapted to solve other representation learning problems that involve multi-modal data and multi-layered graph representations.
Revised: We show that HOSGNS can be intuitively applied to time-varying graphs, but this methodology can be easily adapted to solve other representation learning problems that involve multi-modal data and multi-layered graph representations.