Asymptotics of ℓ 2 Regularized Network Embeddings
Anonymous Author(s)
AffiliationAddress email
Abstract
A common approach to solving prediction tasks on large networks, such as nodeclassification or link prediction, begins by learning a Euclidean embedding of thenodes of the network, from which traditional machine learning methods can thenbe applied.
This includes methods such as DeepWalk and node2vec, which learnembeddings by optimizing stochastic losses formed over subsamples of the graphat each iteration of stochastic gradient descent.
In this paper, we study the effects ofadding an ℓ 2 penalty of the embedding vectors to the training loss of these types ofmethods.
We prove that, under some exchangeability assumptions on the graph, thisasymptotically leads to learning a graphon with a nuclear-norm-type penalty, andgive guarantees for the asymptotic distribution of the learned embedding vectors.
In particular, the exact form of the penalty depends on the choice of subsamplingmethod used as part of stochastic gradient descent.
We also illustrate empiricallythat concatenating node covariates to ℓ 2 regularized node2vec embeddings leads tocomparable, when not superior, performance to methods which incorporate nodecovariates and the network structure in a non-linear manner.
Introduction
Network embedding methods [see e.g 6, 23–26, 46, 56, 62] aim to find a latent representation of thenodes of the network within Euclidean space, in order to facilitate the solution of tasks such as nodeclassification and link prediction, by using the produced embeddings as features for machine learningalgorithms designed for Euclidean data.
For example, they can be used for recommender systems insocial networks, or to predict whether two proteins should be linked in a protein-protein interactiongraph.
Generally, such methods obtain state-of-the-art performance on these type of tasks.
A classical approach to learning a network embedding is to use the principal components of the Laplacian of the network [6]; however, this approach is computationally prohibitive for large datasets.
Inorder to scale to large datasets, methods such as DeepWalk [46] and node2vec [24] learn embeddingsvia optimizing a loss formed over stochastic subsamples of the graph.
Letting G =
( V , E ) denotean undirected graph, and writing ω u ∈ R d for the embedding of a vertex u ∈ V with embeddingdimension d such that d ≪ |V| , these methods learn embeddings by iterating over the followingprocess: we take a random walk ( u i ) i ≤ k over the graph, form a loss
[Equation]
and then perform gradient updates of the form ω u ← ω u − η ∇ ω u L for some step size η > 0 .
Here ⟨· , ·⟩ denotes the Euclidean inner product, W is a window size, N ( ·| v ) a negative samplingdistribution for vertex v and σ ( y ) :
= (1 + e − y )
− 1 the sigmoid function.
LINE [56] uses a similarapproach, except that the random walk is replaced by sampling edges uniformly from the edge set ofthe graph.
One can generalize (1) to consider embeddings learned via stochastic updates
[Equation]
where
[Equation]
P , N ⊆
V × V are random subsamples of the graph called positive and negative samples respectively[see e.g 30], and ℓ P ( · ) and ℓ N ( · ) are chosen to force the embeddings of vertex pairs appearing in
P close together, and those in N far away.
Usually one takes ℓ P ( y )
= − log σ ( y ) and ℓ N ( y ) =
− log σ ( − y ) .
For example, in node2vec P is formed by taking concurrent edges in a random walk,and in LINE, P is formed by uniform edge sampling; in both methods N is taken to be the samenegative sampling distribution N ( u | v ) ∝ deg( v ) 3 / 4 [44].
The scheme in (2) attempts to minimize
[Equation]
obtained by averaging over the random process used to create P and N at each iteration of stochasticgradient descent [53, 61]; see Appendix A.1 for a detailed derivation.
In the case where ℓ P ( y )
= − log σ ( y ) and ℓ N ( y ) =
− log σ ( − y ) , with P and N being randomsubsets of E and ( V × V ) \ E (such as in LINE, or node2vec with a window length of 1 ), then we canview (3) as the function obtained via trying to minimize the negative log-likelihood (equivalent tomaximizing the log-likelihood) of a probabilistic modela [Equation] ) independently for u < v
[Equation]
( a uv = 1 indicating ( u, v ) is an edge) using the stochastic gradient descent scheme specified in (2)(see Appendix A.2).
If it is assumed that the embedding vectors are drawn i.i.d from some latentdistribution, then this corresponds to implicitly fitting an exchangeable model to the graph [42]. Herethe distribution of the adjacency matrix ( a uv ) u,v is invariant to joint permutations of the vertices ofthe network, or equivalently (by the Aldous-Hoover theorem [1]) arise from a probabilistic modelλ u ∼ Unif([0 , 1]) independently , a [Equation]
Bernoulli [Equation] independently
[Equation]
5) for some symmetric measurable function W : [0 , 1] 2 → [0 , 1] called a graphon .
We highlight that themodel in (4) is implicitly fitting a dense graph model to the data, even when the observed graph dataare sparse , as is the case with real world networks.
One way of partially addressing this issue is toconsider a sparsified exchangeable graph with W → ρ n W for some sparsifying sequence ρ n → 0 .
A natural example of a prior distribution on embedding vectors is ω
i ∼ Normal(0 , (2 ξ ) − 1 / 2 I d ) forsome constant ξ > 0 , so that the contribution to the negative log-likelihood of each embedding isξ ∥ ω i ∥ 22 .
The full negative log-likelihood of (4) with such a prior distribution is then given by
[Equation]
which depends only on the matrix G ij = ⟨ ω i , ω j ⟩ ; this loss can also arise by considering a weightdecay optimization scheme [see e.g 36].
In either case, letting Ω ∈ R n × d be the matrix whose rowsare the ω i , so G = ΩΩ T , we get that
[Equation]
where ∥ G ∥ ∗ is the nuclear norm, which equals the sum of the singular values of G .
As a result, theprior distribution on the ω i induces a nuclear-norm penalty on the matrix G , which is well known toshrink the singular values of G exactly towards zero, and to make G low rank [see e.g 4, 18, 35, 52],and consequently lower the effective dimension of the embeddings.
From a computation perspective,this is advantageous compared to treating the embedding dimension as a tunable hyperparameter, aswarm-start procedures can be used to efficiently tune the regularization weight ξ (and consequentlythe effective embedding dimension).
In this paper, our interest is in studying the effects of such a regularizer in the scenario whereembeddings are learned via subsampling, in which case the corresponding version of (3) becomes [Equation]
i
[Equation]
where V ( P ∪ N ) is the set of vertices which appear either in P or N (see Appendix A.3).
We notethat for certain sampling schemes (e.g random walk samplers), the probability a vertex is sampled isnot equiprobable across vertices, and so the regularizer will not be the same form as in (7); however,we still want to analyze the extent to which nuclear norm type regularization (and hence effectivedimension reduction) may still arise.
To do so, we study minimizers ( (cid:98) ω 1 , . . .
, (cid:98) ω n ) of (8) assumingthe graph arises from a sparsified exchangeable graph, and obtain guarantees of the form n [Equation] i [Equation]
[Equation]
i
[Equation]
for some functions K : [0 , 1] 2 → R and ψ : [0 , 1] → R d obtained through the process of minimizingthe population objective (Section 3).
We highlight that our theoretical results apply in the regimed ≪ n , as what occurs with chosen embedding dimensions in practice.
This means that (8) isnon-convex in the matrix G ij = ⟨ ω i , ω j ⟩ due to rank constraints, which complicates the theoreticalanalysis.
Our results will recover the motivation above that the regularization acts to effectivelyreduce the rank of the learned embedding vectors.
We also illustrate experimentally that using suchregularization can give performance competitive to architectures such as GraphSAGE (Section 4).
1.1 Related works
Guarantees for embedding methods: We highlight that there is an extensive literature on theembeddings formed by the eigenvectors of the adjacency or Laplacian matrices of a network [e.g 3,38–40, 54, 57].
Under various latent space models for the network, these works give guarantees onquantities of the form max i ∥ (cid:98) ω
i − Qψ i ∥ 22 for some orthogonal matrix Q and vectors ψ i , to discussrecovery of latent variables and/or obtain exact recovery in a community detection task.
Strongerbounds are obtained in this setting as they are able to directly apply matrix eigenvector perturbationmethods to study the embeddings, which we cannot with our approach; as a tradeoff, our approachallows us to study embeddings learned via a variety of subsampling schemes, which these works donot.
We highlight that the second bound in (9) can still be used to give guarantees for weak recoveryof community detection [39].
There are a few works discussing random walk methods like node2vec,albeit circumventing the non-convexity in the problem; [48] discusses the unconstrained minima ofthe loss (1) when d = n , with [66] then examining the best rank r approximation to this matrix whenthe generating model is a stochastic block model with k communities and r ≤ k .
[20] covers thenon-convex regime d ≪ n in the case where ξ n = 0 , i.e without regularization.
Nuclear norm penalties and ℓ 2 regularized embeddings: In the context of matrix factorization (sothe matrix factors are embedding matrices), the effects of Frobenius norm penalties inducing nuclearnorm penalties are well known [e.g 51, 59]; generally, there is an extensive literature on the effectsof nuclear norm penalization in the finite-dimensional setting [e.g 4, 18, 35, 52].
In [65], it is alsoshown that ℓ 2 regularized node2vec gives an improvement in performance on downstream tasks.
Graphon estimation: We mention that our guarantees on the gram matrix formed by the learnedembeddings are similar to those obtained in the graphon estimation literature [see e.g 8, 9, 17, 21,33, 37, 63, 64].
Depending on the choice of sampling scheme and loss, it is possible for the limitingmatrix K
( λ i , λ j ) to be an invertible transformation of W ( λ i , λ j ) , and so we compare our rates ofconvergence in such a scenario (see Remark 6 in Appendix C).
Framework and assumptions of analysis
Given a sequence of graphs G n = ( V n , E n ) , and writing ω n = ( ω 1 , . . .
, ω n ) with ω u ∈ R d denotingthe d -dimensional embedding of vertex u , we study the regularized empirical risk function R n
[Equation]
ω n ) + [Equation] n R reg n ( ω n ) where R n
[Equation]
ω [Equation] :=
[Equation]
for ξ n ≥ 0 .
Here, we define a subsample S ( G ) of a graph G as a collection of vertices V ( S ( G )) anda symmetric subset of V ( S ( G ))
× V ( S ( G )) .
The sampling probabilities are conditional on G n as we will soon assume that the G n also arise from a probabilistic model.
We note (10) arises from (8)whenever P and N are random subsets of E n and V
× V \ E n respectively (see Appendix A.4).
Throughout, ℓ ( y, x ) is either the cross-entropy loss ℓ σ ( y, x ) := − x log σ ( y ) − (1 − x ) log σ ( − y )
orthe squared loss ℓ 2 ( y, x ) :=
( y − x ) 2 .
We now discuss our assumptions on the generative modelof the graph. Recall in the introduction we argued embedding methods are implicitly fitting anexchangeable graph model; consequently, as a first step to analysis, we will assume that the grapharises from a sparsified graphon (to account for the sparsity in graphs observed in the real world).
Assumption 1.
We assume that the sequence of graphs ( G n ) n ≥ 1 have vertex sets V n = [ n ] and arisefrom a graphon process with generating graphon W n = ρ n W with ρ n ≫ log( n ) /n , so thatλ i i.i.d ∼ Unif[0 , 1] for i ∈ [ n ] , a ij | λ i , λ j indep.
∼ Bernoulli( ρ n W ( λ i , λ j )) for i < j.
We moreover suppose that W ∈ [ c, 1 − c ] for some c > 0 , and either a) W is piecewise constant on apartition Q × Q where Q is a partition of [0 , 1] of size κ (so the model is a stochastic block model),or b) W is Hölder ([0 , 1] 2 , β W , L W ) for some exponents and constants β W ∈ (0 , 1] and L W < ∞ .
Remark 1.
Here we use the canonical choice of uniform latent variables for the graphon asguaranteed by the Aldous-Hoover theorem for vertex exchangeable graphs [1]; in principle, ourresults can extend to graphons on higher dimensional latent spaces by the same style of arguments [seee.g 20].
We highlight that our assumptions are somewhat restrictive with regards to the boundednessand and sparsity conditions; while it is common to allow ρ n ≫ log( n ) /n
[e.g 45, 63] - in whichcase the degree structure is regular, not necessarily realistic of real world networks - it is alsocommon to work in the regime where ρ n = Θ(log( n ) /n ) or smaller [e.g 9, 64].
We highlight that ingeneral graphons, regardless of the sparsity factor, tend to not give rise to graphs with power-law orheavy-tailed type degree structures, which is frequent with real world networks [15, 67].
For subsampling schemes used in practice (such as random walk and uniform edge samplers), thesampling probability of vertices and edges depends only on local features of the graph.
Following[20], we can formalize this as follows:
Assumption 2.
There exist sequences of measurable functions ( f n ) n ≥ 1 and (˜ g n ) n ≥ 1 and a sequences n = o (1) , such that
[Equation]
and moreover E [ f n ( λ
[Equation]
, λ
[Equation] [Equation]
This assumption allows us to replace the sampling probabilities in the empirical risk by functionswhich depend on the latent variables and edge assignments in the model, from which the exchangeability in the model can be used to allow for a large sample analysis.
Examples of sampling schemessatisfying this condition are given in Section 3.2.
We additionally impose some regularity conditionson the "averaged" versions of the above functions defined by
[Equation]
Assumption 3.
We assume the the functions ˜ f
n ( l, l ′ , 1) , ˜ f n ( l, l ′ , 0) and ˜ g n ( l ) are uniformly boundedabove by M and away from zero by M − 1 for some constant M ∈ (0 , ∞ ) .
We also suppose that eithera) there exists a partition Q of [0 , 1] into κ parts
such that ˜ f n ( l, l ′ , 1) and ˜ f n ( l, l ′ , 0) are piecewiseconstant on Q × Q , and ˜ g n ( l ) is piecewise constant on Q ; or b) the ˜ f n ( l, l ′ , 1) and ˜ f n ( l, l ′ , 0) are
Hölder ([0 , 1] 2 , β , L ) , and that the ˜ g n ( l ) are Hölder ([0 , 1] , β , L ) .
Assumption 3 will follow as a consequence of Assumptions 1 and 2 for the sampling schemesdiscussed in Section 3.2, with β depending on β W and the hyper-parameters of the sampling scheme.
Theoretical results
Our theoretical results take the following flavour: we identify the correct population versions (inreference to an infinite graphon on a vertex set N ) of R n ( ω n ) and R reg n ( ω n ) in the large samplelimit n → ∞ to give a regularized population risk, and then use this to give guarantees about any minimizers of R n ( ω n ) + ξ n R reg n ( ω n ) being close (in some sense) to the unique minimizer of theregularized population risk.
As network embedding methods are used on very large networks, such alarge sample statistical analysis is appropriate.
We introduce the population versions of R n ( ω n ) and R reg n ( ω n ) respectively as
I
[Equation]
defined over functions K : [0 , 1] 2 → R .
I n [ K ] was first introduced in [20].
The formula given for
I reg n [ K ] holds only for K whose diagonal is well defined; in general, if K admits a decomposition
K ( l, l ′ [Equation] =
[Equation]
(cid:88) i =
[Equation] µ [Equation] [Equation]
[Equation] i ( l [Equation] ) where (cid:90)
[Equation] otherwise, (13) and µ i ( K ) ≥ 0 for all i (understood as a limit in L 2 ([0 , 1] 2 ) ), then we can extend the definition ofthe regularizer to be I reg n [ K ] := (cid:80) ∞ i =1 µ i ( K ) .
Consequently, the penalty corresponds to the traceof K , when viewed as the kernel of a Hilbert-Schmidt operator.
This means that I reg n [ K ] should beviewed as a nuclear-norm penalty on the kernel K , which encourages the µ i ( K ) to be shrunk exactlytowards zero, similar to the finite dimensional scenario; also see e.g. Theorem 5.
3.1 Guarantees on the learned embedding vectors
We begin with a result which guarantees that I n [ K ] + ξ n I reg n [ K ] , once restricted to an appropriatedomain, is the correct population version of R n ( ω n ) + ξ n R reg n ( ω n ) .
Theorem 1.
Suppose that Assumptions 2 and 3 hold, and that ξ n = O (1) .
Define
Z {+≥ 
+}
d ( A ) := (cid:8) K : K ( l, l ′ ) =
⟨ η ( l ) , η ( l ′ ) ⟩ , η : [0 , 1] →
[ − A, A ] d (cid:9) for d ∈ N , A > 0 .
Then we have that
[Equation]
O p
[Equation]
r n ) where r n = s n + ( d p /nρ n ) 1 / 2 + t n , with t n = (log κ/n ) 1 / 2 under Assumption 3a), t n =(log( n )
/n 2 β/ (1+2 β ) ) 1 / 2 under Assumption 3b), p = 3 for the cross-entropy loss and p = 5 forthe squared loss.
See Appendix B for the proof and a discussion of the rates given; we note that it is necessary thatd ≪ n in order for r n → 0 .
The rates can be improved to give p = 1 , under additional restrictions onthe parameter space (Remark 3).
The interpretation of the set Z ≥
d ( A ) is as follows: if ω u ∈ [ − A, A ] dis the embedding of vertex u , in the population limit η ( λ u ) ∈
[ − A, A ] d should give the embeddingof a vertex with latent feature λ u .
As (10) is parameterized through terms of the form ⟨ ω u , ω v ⟩ , thepopulation version of (10) should be parameterized through functions K ( λ u , λ v ) = ⟨ η ( λ u ) , η ( λ v ) ⟩ .
Remark 2.
We note that the assumption that the embedding vectors belong to a hypercube [ − A, A ] dis not restrictive; for example, in practice embedding vectors are usually initialized randomly anduniformly over [ − 1 , 1] d .
Moreover, our results allow for A to grow logarithmically with n , and onlychange the bound by a poly-log factor.
We highlight that if ξ n → ∞ as n → ∞ , then the embeddingvectors will shrink towards 0 as n → ∞ (as seen in Figure 1).
This is because the proof of Theorem 1shows that any minimizer must satisfy n − 1 (cid:80) ni
=1 ∥ ω i ∥ 22 = O p ( ξ − 1 n ) =
o p (1) in such a regime.
We now give convergence guarantees for any sequence of embedding vectors minimizing (10).
Theorem 2.
Suppose that Assumptions 2 and 3 hold and that ξ n = O (1) .
Then for each n , thereexists a unique minimizer to the optimization problem
[Equation]
where
[Equation]
is free of A >
(see Proposition 2 for details).
Moreover, under some regularity conditions on the
K ∗ n (see Theorem 6 in Appendix C), there exists A ′ < ∞ free of n and a sequence of embeddingdimensions d = d ( n ) ≪ n such that, for any sequence of minimizers
[Equation]
with A 1 , A 2 ≥ A ′ , we have that
[Equation]
Moreover, when Assumption 3a) holds, K ∗ n can be computed via a finite dimensional convex program,and is of rank r ≤ κ , in that an expansion of the form (13) holds with µ i ( K ∗ n )
=
for i > r .
The case where ξ n = 0 is proven in [20], which also verifies the convergence on simulated data.
Aproof is given in Appendix C. Under certain circumstances, this allows us to give guarantees aboutthe distribution of the embedding vectors themselves.
Theorem 3.
Suppose that K ∗ n is regular in the sense of Theorem 2, with the conclusions of thetheorem holding.
Moreover suppose that K ∗ n is a kernel of rank r < ∞ , has a decomposition of theform K ∗ n ( l, l ′ )
= (cid:80) ri =1 ϕ n,i ( l ) ϕ n,i ( l ′ ) for some functions ϕ n,i : [0 , 1] → R , and the dimension d ofthe embedding vectors is chosen to be equal to r .
Writing ϕ n ( l ) =
( ϕ n,i ( l )) ri =1 , we have
[Equation]
The assumption that d = r in Theorem 3 is a restrictive one, given that embedding dimensions inpractice are usually chosen to be one of either 128, 256 or 512.
This can be alleviated by insteadgiving a guarantee for the optimal r dimensional projection of the embedding vectors.
Theorem 4.
If instead d > r in Theorem 3, let (cid:101) G ∈ R n × n denote the best rank r approximation to thematrix G ij := ( ⟨ (cid:98) ω
i , (cid:98) ω
j ⟩ ) ij , and write (cid:101) G = ˜Ω˜Ω T for some ˜Ω ∈ R n × r .
Then n − 2 ∥ (cid:101) G − G ∥ 2 F = o p (1) ,and writing (cid:101) ω
i for the rows of (cid:101) Ω , the guarantee in (14) holds with (cid:101) ω
i replacing the (cid:98) ω i .
Informally, this says that the embedding vectors approximately lie on a r -dimensional subspacewhich contains some latent information about the network, depending on the minimizing kernel K ∗ n .
We now highlight that the assumption that K ∗ n is of finite rank r < ∞ is not restrictive even when Wis not a SBM; this is a consequence of the effect of the regularization penalty I reg n [ K ] .
Theorem 5.
Let ℓ ( y, x ) be the squared loss, and suppose that ρ n = 1 , ˜ f n ( l, l ′ , 1) =
˜
f n ( l, l ′ , 0)
=
c
1and ˜ g n ( l ) =
c 2 for some constants c 1 , c 2 >
(see e.g. Algorithm 1 in Section 3.2).
Then if W is
Hölder ([0 , 1] 2 , β, L ) , the minima of I n [ K ] + ξ n I reg n [ K ] over Z ≥ 0 is of finite rank for any ξ n > 0 ,and is also Hölder continuous of exponent β .
We highlight that this result also shows that the regularizer acts to shrink the singular values of a minimizer of I n [ K ]+ ξ n I reg n [ K ] exactly towards zero.
See Appendix D for proofs of Theorems 3, 4 and 5.
3.2 Sampling schemes satisfying Assumption
We now discuss some examples of frequently used sampling schemes which satisfy Assumption 2.
Proofs of the results in this section can be found in
Appendix E. We introduce the notation
[Equation]
Algorithm 1 (Uniform vertex sampling) .
Given a graph G n and number of samples k , we select kvertices from G n uniformly and without replacement, and then return S ( G n ) as the induced subgraphusing these sampled vertices.
Lemma 1.
For Algorithm 1, Assumption 2 holds with f n ( λ i , λ j , a ij ) =
k ( k − 1) , ˜ g n ( λ i ) =
k ands n = 1 /n .
Algorithm 2 (Uniform edge sampling [56]) .
Given a graph G n , number of edges to sample k , andnumber of negative samples l per positive sample,
i)
We form a sample S [-
-]0 ( G n ) by sampling k edges from G n uniformly and without replacement;
ii) We form a sample set of negative samples
S ns ( G n ) by drawing, for each u ∈ V ( S 0 ( G n )) , lvertices v 1 , . .
. , v l i.i.d according to the unigram distribution
[Equation]
and then adjoining ( u, v i )
→ S ns ( G n ) if a uv i = 0 .
[Figure]
return S ( G
Lemma 2.
For Algorithm 2, Assumption 2 holds with s n = (log( n ) /nρ n ) − 1 / 2 ,
[Equation]
Algorithm 3 (Random walk sampling [24, 46]) .
Given a graph G n , a walk length k , number ofnegative samples l per positively sampled vertex and unigram parameter α , wei) Perform a simple random walk on G n of length k , beginning from its stationary distribution, toform a path (˜ v i ) i ≤ k +1 , and report (˜ v i , ˜ v i +1 ) for i
≤ k as part of S 0 ( G n ) ;ii)
For each vertex ˜ v
i , we select l vertices ( η j ) j ≤ l independently and identically according to theunigram distribution
Ug α ( v | G n ) ∝ P (cid:0) ˜ v i = v for some i ≤ k | G n (cid:1) α
and then form S ns ( G n ) as the collection of vertex pairs (˜ v i , η j ) which are not an edge in G n .
We then return S ( G n ) as the union of S 0 ( G n ) and S ns ( G n ) .
Lemma 3.
For Algorithm 3, Assumption 2 holds with s n = (log( n ) /nρ n ) 1 / 2 ,f n ( λ i , λ j , 1) =
[Equation]
Examining the formula for ˜ g n ( λ ) above, we see that for random walk samplers the shrinkage providedto the learned kernel will be greater for vertices with larger degrees.
Indeed, as K ( λ, λ ) =
∥ η ( λ )
∥ 22for K ∈
Z ≥
d ( A ) , the larger ˜ g n ( λ ) is, ∥ η ( λ ) ∥ 22 will be forced closer towards zero.
Experiments
We now examine the performance in using regularized node2vec embeddings for link predictionand node classification tasks, and illustrate comparable, when not superior, performance to morecomplicated encoders for network embeddings.
We perform experiments on the Cora, CiteSeer and
PubMedDiabetes citation network datasets (see Appendix F for more details), which we use as theyare commonly used benchmark datasets - see e.g [25, 27, 32, 62]. [Table]
4.1 Methods used for comparison
[Table] For our experiments, we consider 128 dimensional node2vec embeddings learned with either noregularization, or a ℓ 2 penalty on the embedding vectors with weight ξ ∈ { 1 , 5 } × 10 −{ 3 , 4 , 5 , 6 , 7 , 8 } ;and with or without the node features concatenated.
We compare these against methods whichincorporate the network and covariate structure together in a non-linear fashion.
We consider threemethods - a two layer GCN [31] with 256 dimensional output embeddings trained in an unsupervisedfashion through the node2vec loss, a two layer GraphSAGE architecture [25] with 256 dimensionaloutput embeddings trained through the node2vec loss, and a single layer 256 dimensional GCNtrained using Deep Graph Infomax (DGI) [62].
We highlight that the GCN, GraphSAGE and DGIalways have access to the nodal features during training.
All of our experiments used the Stellargraph[19] implementations for each method.
Further experimental details are given in Appendix F.
4.2 Link prediction experiments
For the link prediction experiments, we create a training graph by removing 10% of both the edgesand non-edges within the network, and use this to learn an embedding of the network.
We thenform link embeddings by taking the entry-wise product of the corresponding node embeddings,
use10% of the held-out edges to build a logistic classifier for the link categories, and then evaluate theperformance on the remaining edges, repeating this process 50 times.
The PR AUC scores are given in Table 1 and visualized in Figure 2.
Provided the regularizationparameter is chosen optimally, we see an improvement in performance compared to using noregularization, with the jump in performance slightly greater when nodal features are not incorporatedinto the embedding.
The optimally regularized version of node2vec, even without including the nodefeatures, is competitive with the GCN and GraphSAGE (being equal or outperforming at least one ofthem across the three datasets), and that the version with concatenated node features is as good as the GCN trained using DGI.
For all the datasets, we observe a sharp decrease in performance after theoptimal weight, suggesting that it needs to be chosen carefully to avoid removing the informativenetwork structure.
As seen in Figure 1, this occurs as the learned embeddings become randomlydistributed at the origin once the regularization weight is too large.
4.3 Node classification experiments
To evaluate performance for the node classification task, we learn a network embedding withoutaccess to the node labels, and then learn/evaluate a one-versus-rest multinomial node classifier using5%/95% stratified training/test splits of the node labels.
We repeat this over 25 training runs of theembeddings, and a further 25 training/test splits for the node classifiers per embedding.
Table 1 and
Figure 2 show the average macro F1 scores and their standard deviation for each method and dataset.
Similar to the link prediction experiments, we see that the optimally regularized node2vec methodsare competitive, if not outperforming, the GCN and GraphSAGE trained through the node2vec loss,and is outperformed by the GCN learned using DGI by no more than two percentage points. In
[Figure] these experiments, the standard deviations correspond partially randomness induced by the training/test splits of the node labels, and suggest that the regularized version of node2vec is no lessrobust to particular choices of training/test splits than the other methods.
Interestingly, we note that the optimal performance on PubMed is given by the regularized node2vecwith node features.
However, as the highest level of performance is performed when the regularizationweight is so large that the learnt embeddings are uninformative (as illustrated by the massive decreasein performance of the regularized method without node features), this suggests that the nodes can beclassified using only the covariate information, and that the network features are not needed.
As thedataset only has three distinct classes corresponding to the academic field, and the node features are
TF-IDF embeddings of the academic papers, this is not too surprising.
Conclusion
In this paper we have theoretically described the effects of performing ℓ 2 regularization on networkembedding vectors learned by schemes such as node2vec, describing the asymptotic distribution ofthe embedding vectors learned through such schemes, and showed that the regularization helps toreduce the effective dimensionality of the learned embeddings by penalizing the singular values ofthe limiting distribution of the embeddings towards zero.
We do so in the non-convex regime d ≪ n ,reflecting how embedding dimensions are chosen in the real world.
We moreover highlight empiricallythat using ℓ 2 regularization with node2vec leads to competitive performance on downstream tasks,when compared to embeddings produced from more recent encoding and training architectures.
We end with a brief discussion of some limitations of our works, and directions for future work.
Generally, graphons are not realistic models for graphs; we suggest that our work could be extendedto frameworks such as graphexes [10, 11, 60] which can produce more realistic degree distributionsfor networks, but have enough underlying latent exchangeability for our arguments to go through.
We ignore aspects of optimization, i.e whether the minima of (10) are obtained in practice, whichwe believe would be an interesting area of future research.
As for our experimental results, we notethat methods such as GraphSAGE are better than node2vec in that they provide embeddings for dataunobserved during training, and also scale better to larger networks.
Consequently, we believe ourexperiments should be used primarily as motivation to investigate better methods for incorporating nodal covariates into network embedding models, and how to regularize embeddings produced byencoder methods such as GCNs or GraphSAGE.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See the first paragraph inthe related works section; Remark 1; the paragraph between Theorem 3 and 4; theconclusion; Remark 6 in the appendix re: our rates of convergence.
(c) Did you discuss any potential negative societal impacts of your work?
[No] The paper
discusses the theoretical effects of a particular type of regularization scheme, and does
not advocate for any methodological changes or gives particular applications; I believe
that making any types of claims about the societal impacts of this work would be
excessively speculative, to be frank.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Yes ( https://neurips.cc/public/EthicsGuidelines ).
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
and the relevant theorem statements; the one theorem in the main body which does not
state them fully (Theorem 2) mentions the corresponding version of the theorem in the
appendix (Theorem 6) which has the full regularity conditions.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix F.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 and Appendix F.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 1 and Figure 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix F.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1 and
Appendix F.
(b) Did you mention the license of the assets?
[Yes] See Appendix F.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
No new assets were created.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
Standard datasets were used, which do not contain information
about/curated from individuals.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] Datsets used were standard citation network
datasets with no identifiable/textual information provided.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] No crowdsourcing/human subjects were used.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] No crowdsourcing/human subjects were
used.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] No crowdsourcing/human subjects were used.