F AST G RAPH L EARNING WITH U NIQUE O PTIMAL S OLUTIONS
A BSTRACT
We consider two popular Graph Representation Learning (GRL) methods: message passing for node classiﬁcation and network embedding for link prediction.
For each, we pick a popular model that we: (i) linearize and (ii) and switch its training objective to Frobenius norm error minimization .
These simpliﬁcations can cast the training into ﬁnding the optimal parameters in closed-form.
We program in TensorFlow a functional form of Truncated Singular Value Decomposition (SVD), such that, we could decompose a dense matrix M , without explicitly computing M .
We achieve competitive performance on popular GRL tasks while providing orders of magnitude speedup.
We open-source our code at http://github.com/samihaija/tf-fsvd
I NTRODUCTION
Many recent graph representation learning (GRL) models are creative and
theoretically-justiﬁed (Kipf & Welling, 2017; Hamilton et al., 2017;
Veli ˇ ckovi ´ c et al., 2018; Qiu et al., 2018; Xu et al., 2019; Abu-El-Haija et al., 2019; Chen et al., 2020).
Unfortunately, however, they contain hyperparameters that need to be tuned (such as learning rate, regularization coefﬁcient, depth and width of the network), and training takes a long time (e.g. minutes) even on smaller datasets.
We circumvent these weaknesses.
We (i) quickly train (ii) competitive GRL models by posing convex objectives and estimating optimal solutions in closed-form, hence (iii) relieving practitioners from hyperparameter tuning or convergence checks.
Our goals remind us of a classical learning technique that has been used for decades.
Speciﬁcally, Singlar Value Decomposition (SVD).
SVD periodically appears within powerful yet simple methods, competing on state-of-the-art.
The common practice is to design a matrix M , such that its decomposition (via SVD), provides an estimate for learning a model given an objective.
For instance, Levy & Goldberg (2014) show that the learning of NLP skipgram models such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), can be approximated by the SVD of a Shifted Positive Pointwise Mutual
Information matrix.
In GRL, Chen et al.
; Qiu et al.
; Abu-El-Haija et al.
(2018) have approximated methods of DeepWalk (Perozzi et al., 2014) and Node2Vec (Grover & Leskovec, 2016) via decomposition of some matrix M .
However, their decomposition requires M to be either (a) exactly calculated or (b) sampled entry-wise, but (a) is unnecessarily expensive for real-world large networks (due to Small
World Phenomenon, (Travers & Milgram, 1969)) and (b) incurs unnecessary estimation errors.
On the other hand, known algorithms in matrix theory can decompose any matrix M without explicitly knowing M .
Speciﬁcally, it is sufﬁcient to provide a function f M ( . )
= (cid:104) M , .
(cid:105) that can multiply M with arbitrary vectors (§4).
We, argue that if the popular frameworks (e.g., TensorFlow) implement a functional SVD , that accept f M ( . ) rather than M , then modern practitioners may ﬁnd it useful.
We review powerful GRL methods (§2.2) that we convexify (§3), allowing us to use (randomized)
SVD for obtaining (approximate) optimum solutions.
Our contributions are:
We implement a functional SVD (§4) of the randomized algorithm of Halko et al.
We approximate embedding and message passing methods via SVD (§3), showing competitive performance with state-of-the-art, yet much faster to train (§5)
We analyze that learning is fast and approximation error can be made arbitrarily small (§A.3.2).
P RELIMINARIES 2. S INGULAR V ALUE D ECOMPOSITION (SVD)
Truncated (top- k ) Singular Value Decomposition (SVD) estimates input matrix M ∈ R r × c with low-rank estimate (cid:102) M that minimizes the Frobenius norm of the error:
[Equation]
subject to: rank ( (cid:102)
[Equation]
while parameterizing (cid:102) M as (cid:102) M = U k S k V (cid:62) k , subject to, columns of U k , V k being orthonormal.
It turns out, the minimizer of Frobenius norm || .
||
F recovers the top- k singular values (stored along diagonal matrix S k ∈ R k × k ), with their corresponding left- and right-singular vectors, respectively, stored as columns of the unitary matrices U k ∈ R r × k and V k ∈ R c × k ( a.k.a , the singular bases ).
SVD has many applications and we utilize two: (1) it is used for embedding and matrix completion; and (2) it can estimate the pseudoinverse of matrix M (a.k.a., Moore-Penrose inverse), as:
[Equation]
where one calculates inverse S − 1 by reciprocating entries of diagonal matrix S .
The ≈ becomes = when k ≥ rank ( M ) , due to (Eckart & Young, 1936; Golub & Loan, 1996).
2. G RAPH R EPRESENTATION L EARNING (GRL)
(i) Many message passing models can be written as:
[Equation]
where L is the number of layers, matrix X ∈ R n × d contains d features per node, W ’s are trainable parameters, σ denote activations (e.g. ReLu), and g is some (possibly trainable) transformation of adjacency matrix.
GCN (Kipf & Welling, 2017) set g to symmetric normalization per renormalization trick , GAT
(Veli ˇ ckovi ´ c et al., 2018) set g ( A ) =
A ◦ MultiHeadedAttention and GIN (Xu et al., 2019) as g ( A ) =
A + (1 + (cid:15) )
I with identity I and (cid:15) > 0 .
For node classiﬁcation, it is common to set σ L = softmax (applied row-wise), specify the size of W L s.t.
H ∈ R n × y where y is number of classes, and optimize cross-entropy objective:
[Equation]
where Y is a binary matrix with one-hot rows indicating node labels.
◦ is Hadamard product.
in semi-supervised node classiﬁcation settings where not all nodes are labeled, before measuring the objective, subset of rows can be kept in Y and H that correspond to labeled nodes.
(ii) Network embedding methods map nodes onto a z -dimensional vector space Z ∈ R n × z .
Modern approaches train skipgram models (e.g. word2vec
(Mikolov et al., 2013)) on sampled random walks.
It has been shown that these skipgram network embedding methods, including DeepWalk (Perozzi et al., 2014)) and node2vec (Grover & Leskovec, 2016), with a learning process of walk sampling followed by positional embedding, can be approximated as a matrix deomposition (Chen et al., 2017; Abu-El-Haija et al., 2018; Qiu et al., 2018).
We point the curious reader to the listed papers for how the decomposition was derived, but show here the derivation of Abu-ElHaija et al.
(WYS, 2018), as it performs well in our experiments:
[Equation]
where h ( Z ) =
h ([ L R ]) =
(1 + exp
( L × R (cid:62) )) − 1 i.e. Z concatenates L , R ∈ R n × z 2 and h is the logistic of their cross-correlation (pairwise dot-products).
M (WYS) =
(cid:80) Ci ( D − 1 A )
i c i
= (cid:80)
Ci =1 T i c
i , where T is the transition matrix, D = diag ( 1 (cid:62) A ) is diagonal degree matrix, and we ﬁx vector c to staircase: c i
=
C − i + 1 .
For instance, c = [4 , 3 , 2 , 1] for context size C = 4 .
O UR P ROPOSED C ONVEX O BJECTIVES 3. N ETWORK E MBEDDING M ODEL
Objective in Eq 5 learns node embeddings Z =
[ L R ] ∈ R n × z using cross-entropy.
The terms: model output (outer product, σ ( L × R (cid:62) ) ), negatives (non-edges, 1 − A ), and positives (expected number of node pairs covisits, M ), are all dense matrices with O ( n 2 ) (cid:29) m nonzero entries.
For instance, even a relatively-small social network with n =100,000 and average degree of 100 (i.e. m = 100 n ) would produce an M occupying ≈ 40GB memory, whereas one can do the entire learning with ≈ 40MB memory using functional SVD (§4).
We start by designing a matrix (cid:99) M (WYS) incorporating positive and negative information ( M (WYS) and 1 − A ) as:
[Equation]
with coefﬁcient λ ≥ 0 weighing negative samples.
We use (cid:98) .
to denote our convexiﬁcation.
Learning: We can directly set Z to the SVD basis ( U , S , V ) .
Matrix (cid:99) M
(WYS) has large entry (cid:99)
M uv when nodes ( u, v ) are well-connected (co-visited many times, during random walks) and small if they are non-edges.
SVD provides a rank k estimator of (cid:99) M as (cid:98) L (cid:98) R (cid:62)
≈ (cid:99) M i.e. with minimum
Frobenius norm of error.
We can set the network embedding model parameters Z
= [ (cid:98) L (cid:98) R ] as:
SVD (
[Equation]
M , k ) ← arg min
[Equation]
[Table]
Learning follows as:
[Equation]
Inference: Given query edges Q = { ( u i , v i ) } i , one can compute model: H Q = (cid:98) R [Equation]
[Equation]
} i
[Equation]
L
[Equation] } i , (9) where the (RHS) set-subscript denotes gathering rows (a.k.a, advanced indexing), and H Q ∈ R | Q | . 3. M ESSAGING P ASSING M ODELS
We can linearize message passing models (Eq. 3) by assuming all σ ’s are identity ( σ ( . )
= . ) .
Let g = g 1 = g 2 = . . .
, speciﬁcally let g ( A ) =
( D + I ) − 1 / 2 ( A + I )
( D + I ) − 1 / 2 per renormalization trick of (Kipf & Welling, 2017).
A linear L -layer message passing network can be:
[Equation]
Concatenation of all layers was proposed in Jumping Knowledge Networks (JKN, Xu et al., 2018).
Learning: We optimize (cid:99) W with:
[Equation]
Loss in Equation 11 can perform well on classiﬁcation tasks, and according to Hui & Belkin (2021), as well as the cross entropy loss deﬁned in Equation 4.
Taking ∇ (cid:99) W of Eq.
11 then setting to zero, yields minimizer:
[Equation]
Rank- k SVD can estimate (cid:16) (cid:99) M (JKN) (cid:17) † and hence (cid:99) W ∗ as:
[Equation]
k
[Equation]
Order of multiplications in Eq.
13 is for efﬁciency.
Further, in the case when only subset of nodes
V = { v } v have labels, the right-most multiplication of Eq.
13 could restricted to the labeled nodes.
Let Y V be a matrix of |V| rows selected from Y according to elements V .
The right-most multiplication of Eq.
13 can modiﬁed to: (cid:98) U (cid:62) k V Y V .
F UNCTIONAL S INGULAR V ALUE D ECOMPOSITION
We do not have to explicitly calculate the matrices (cid:99) M .
Rather, we only need to implement product functions f (cid:99) M ( v ) =
(cid:104) (cid:99) M , v (cid:105) that can multiply (cid:99) M with arbitrary (appropriately-sized) vector v .
We implement a (TensorFlow) functional version of the randomized SVD algorithm of Halko et al.
(2009), that accepts f (cid:99) M rather than (cid:99) M .
We show that it can train our models quickly and with arbitrarily small approximation error (in linear time of graph size, in practice, with less thanpasses over the data) and can yield l2-regularized solutions for classiﬁcation (see Appendix).
We now need the (straightforward f (cid:99) M (WYS) and f (cid:99) M (JKN) .
We leave the second outside this writing.
For the ﬁrst, the non-edges term, (1 − A ) , can be re-written by explicit broadcasting as ( 11 (cid:62) − A ) giving
[Equation]
All matrix-vector products can be efﬁciently computed when A is sparse.
E XPERIMENTAL R ESULTS ( DETAILS IN A PPENDIX )
[Figure]
[Figure] [Table]
Planetoid
[Table]
A CKNOWLEDGMENTS
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) and the Army Contracting Command-Aberdeen Proving Grounds (ACC-APG) under
Contract Number W911NF-18-C-0020.
A A PPENDIX : I MPLEMENTATION OF F UNCTIONAL SVD A. C ALCULATING THE SVD
SVD of M yields its left and right singular orthonormal (basis) vectors, respectively, in columns of
U and V .
Since U and V , respectively, are the eigenvectors of MM (cid:62) and M (cid:62) M , then perhaps the most intuitive algorithms for SVD are variants of the power iteration , including Arnoldi iteration (Arnoldi, 1951) and Lanczos algorithm (Lanczos, 1950).
In practice, randomized algorithms for estimating SVD run faster than these variants, including the algorithm of Halko et al.
(2009) which is implemented in scikit-learn.
None of these methods require individual access to M ’s entries, but rather, require two operations: ability to multiply any vector with M and with M (cid:62) .
Therefore, it is only a practical gap that we ﬁll in this section: we open-source a TensorFlow implementation that accept product and transpose operators.
A. T ENSOR F LOW I MPLEMENTATION
Since we do not explicitly calculate the (cid:99) M matrices displayed in Equations 6 and 10, as doing so consumes quadratic memory O ( n 2 ) , we implement a functional form SVD of the celebrated randomized SVD algorithm of Halko et al.
To run our Algorithm 1, one must specify k ∈ N + (rank of decomposition), as well as functions f, l, s that the program provider promises they operate as:
Product function f that exactly computes f ( v ) =
(cid:104) M , v (cid:105) for any v ∈ R c (recall: M ∈ R r × c ) 2.
Transpose 1 function t .
∀ v ∈ R r , ( t ◦ f )( v ) =
(cid:104) M (cid:62) , v (cid:105) 3.
Shape (constant) function s that knows and returns ( r, c ) .
Once transposes, should return ( c, r ) .
, however, in practice, TensorFlow is optimized for CSR matrices and computationally favors sparse-times-dense 1: input: rank k ∈ N + , product fn f : R c
→ R r , shape fn s , transpose fn t : ( R c → R
) → ( R c → R r ) 2: procedure fSVD( f, t, s, k ) 3: ( r, c ) ← s () 4:
Q ∼ N (0 , 1) c × 2 k (cid:46) IID Gaussian.
Shape: ( c × 2 k ) 5: for i ← 1 to iterations do 6:
Q, ← tf.linalg.qr ( f ( Q ))
(cid:46) ( r × 2 k ) 7:
Q, ← tf.linalg.qr (( t ◦ f )( Q ))
(cid:46) ( c × 2 k ) 8:
Q, ← tf.linalg.qr ( f ( Q ))
(cid:46) ( r × 2 k ) 9:
B ← (( t ◦ f )( Q )) (cid:62) (cid:46) (2 k × c ) 10:
U, s, V (cid:62) ← tf.linalg.svd ( B ) 11:
U ← Q × U (cid:46) ( r × 2 k ) 12: return U [: , : k ] , s [: k ] , V [: , : k ] (cid:62)
A. A NALYSIS A.3. N ORM R EGULARIZATION OF W IDE M ODELS
If (cid:99) M is too wide, then we need not to worry much about overﬁtting , due to the following Theorem.
Theorem 1 (Min. Norm)
If system (cid:99) M (cid:99) W =
Y is underdetermined 2 with rows of (cid:99) M being linearly independent, then there are inﬁnitely many solutions.
Denote solution space (cid:99) W ∗ =
(cid:110)(cid:99) W (cid:12)(cid:12)(cid:12) (cid:99) M (cid:99) W =
Y (cid:111) .
Then, for k ≥ rank ( (cid:99) M ) , matrix (cid:99) W ∗ , deﬁned in Eq.13 satisﬁes: (cid:99) W ∗ = argmin (cid:99) W ∈ (cid:99) W ∗ || (cid:99) W || 2 F
Proof Assume Y =
y is a column vector (the proof can be generalized to matrix Y by repeated column-wise application 3 ).
SVD ( (cid:99) M , k ) , k ≥ rank ( (cid:99) M ) , recovers the solution:
[Equation]
The Gram matrix (cid:99) M (cid:99) M (cid:62) is nonsingular as the rows of (cid:99) M are linearly independent.
To prove the claim let us ﬁrst verify that (cid:99) W ∗ ∈ (cid:99) W ∗ : [Equation] (cid:62)
[Equation]
Let (cid:99) W p ∈ (cid:99) W ∗ .
We must show that || (cid:99) W ∗ || 2 ≤ || (cid:99) W p || 2 .
Since (cid:99) M (cid:99) W p =
y and (cid:99) M (cid:99) W ∗ =
y , their subtraction gives:
[Equation]
It follows that ( (cid:99)
[Equation]
Finally, using Pythagoras Theorem (due to ⊥ ):
[Equation]
As a consequence, solution for classiﬁcation models recovered by SVD follow a strong standard
Gaussian prior, which may be regarded as a form of regularization.
The minimizer for the Frobenius norm is composed, column minimizers argmin
A.3. C OMPUTATIONAL C OMPLEXITY AND A PPROXIMATION E RROR
Theorem 2 (Linear Time)
Functional SVD (Alg. 1) trains our convexiﬁed GRL models in time linear in the graph size.
Proof of Theorem 2 for our two model families:
For rank- k SVD over f (cid:99) M
(WYS) : Let cost of running f (cid:99) M (WYS) =
T mult .
The run-time to compute SVD, as derived in Section 1.4.2 of (Halko et al., 2009), is:
[Equation]
[Equation] Since f (cid:99) M (WYS) can be deﬁned as C (context window size) multiplications with sparse n × n matrix
T with m non-zero entries, then running fSVD( f (cid:99) M (WYS) , k ) costs:
[Equation]
[Equation]
For rank- k SVD over f (cid:99) M (JKN) :
Suppose feature matrix contains d -dimensional rows.
One can calculate (cid:99) M (JKN) ∈ R n × Ld with L sparse multiplies in O ( Lmd ) .
Calculating and running SVD (see Section 1.4.1 of Halko et al., 2009) on (cid:99) M (JKN) costs total of:
[Equation]
Therefore, training time is linear in n and m .
(cid:4)
Contrast with methods of WYS (Abu-El-Haija et al., 2018) and NetMF (Qiu et al., 2018), which require assembling a dense n × n matrix requiring O ( n 2 ) time to decompose.
One wonders: how far are we from the optimal SVD with a linear-time algorithm?
The following bounds the error.
Theorem 3 (Exponentially-decaying Approx.
Error)
Rank- k randomized SVD algorithm of Halko et al.
(2009) gives an approximation error that can be brought down, exponentially-fast, to no more than twice of the approximation error of the optimal (true) SVD.
Proof is in Theorem 1.2 of Halko et al.
(cid:4)
Consequently, compared to (cid:94) NetMF of (Qiu et al., 2018), which incurs unnecessary estimation error, our estimation error can be brought-down exponentially by increasing the iters parameter of
Alg.
B A PPENDIX : E XPERIMENTAL D ETAILS B. D ATASETS
We apply our functional SVD on popular datasets that can be trained using our simpliﬁed (i.e., convexiﬁed) models.
Speciﬁcally either (1) semi-supervised node-classiﬁcation datasets, where features are present, or (2) link-prediction datasets where features are absent.
It is possible to convexify other setups, e.g., link prediction when node features are present, but we leave this as future work.
We run experiments on seven graph datasets:
• Protein-Protein Interactions (PPI): a large graph where every node is a protein and an edge between two nodes indicate that the two proteins interact.
Processed version of PPI was downloaded from (Grover & Leskovec, 2016).
• Three citation networks that are extremely popular: Cora, Citeseer, Pubmed.
Each node is an article and each (directed) edge implies that an article cites another.
Additionally, each article is accompanied with a feature vector (containing NLP-extracted features of the article’s abstract), as well as a label (article type).
• Two collaboration datasets: ca-AstroPh and ca-HepTh, where nodes are researchers and an edge between two nodes indicate that the researchers co-published together at least one article, in the areas Astro-Physics and High Energy Physics.
• ego-Facebook: an ego-centered social network.
[Table]
For citation networks, we processed node features and labels.
For all other datasets, we did not process features during training nor inference.
For train/validation/test partitions: we used the splits of Yang et al.
(2016) for Citeseer, Cora, Pubmed; we used the splits of Abu-El-Haija et al.
(2018) for
PPI, Facebook, ca-AstroPh and ca-HepTh; we used the splits of OGB (Hu et al., 2020) for ogbl-ddi.
All datasets and statistics are summarized in Table 2.
In §B.2 and §B.3, unless otherwise noted, we download authors’ source code from github, modify 4 it to record wall-clock run-time, and run on GPU NVidia Tesla k80.
Thankfully, downloaded code has one script to run each dataset, or hyperparameters are clearly stated in the source paper.
B.
[Table]
We consider a transductive setting where a graph is entirely visible (all nodes and edges).
Additionally, some of the nodes are labeled.
The goal is to recover the labels of unlabeled nodes.
All nodes have feature vectors.
Baselines : We download code of GAT (Veli ˇ ckovi ´
c et al., 2018), MixHop (Abu-El-Haija et al., 2019), GCNII (Chen et al., 2020) and re-ran them, with slight modiﬁcations to record training time.
However, for baselines Planetoid (Yang et al., 2016) and GCN (Kipf & Welling, 2017), we copied them from the GCN paper (Kipf & Welling, 2017).
In these experiments, to train our method, we run our functional SVD twice per graph.
We take the feature matrix X bundled with the datasets, and concatenate to it two matrices, (cid:98) L and (cid:98) R , calculated per Equation 8: the calculation itself invokes our functional SVD (the ﬁrst time) on f (cid:99) M (WYS) with rank = 32 .
Hyperparameters of f (cid:99) M (WYS) are λ (negative coefﬁcient) and C (context window-size).
After concatenating (cid:98) L and (cid:98) R into X , we PCA the resulting matrix to 1000 dimensions, which forms our new X .
Finally, we express our model as the linear L -layer messaging passing network (Eq. 10) and learn its parameters via rank k SVD on f (cid:99) M (JKN) (the second time), as explained in §3.2.
We use the validation partition to tune L , k , λ , and C .
Table 1 (left) summarizes the performance of our approach ( f (cid:99) M (JKN) ) against aforementioned baselines, showing both test accuracy and training time.
While our method is competitive with state-ofthe-art, it trains much faster.
B. ROC-AUC L INK P REDICTION
Given a partial graph: only of a subset of edges are observed.
The goal is to recover unobserved edges.
This has applications in recommender systems: when a user expresses interest in products, the system wants to predict other products the user is interested in.
The task is usually setup by partitioning the edges of the input graph into train and test edges.
Further, it is common to sample negative test edges e.g. uniformly from the graph compliment.
Lastly, a GRL method for link prediction can be trained on the train edges partition, then can be asked to score the test partition edges versus the negative test edges .
The quality of the scoring can be quantiﬁed by a ranking metric, e.g., ROC-AUC.
Baselines : We download code of WYS (Abu-El-Haija et al., 2018) and update it to for TensorFlow2.0.
We download code of Qiu et al.
and denote their methods as NetMF and (cid:94) NetMF, where the ﬁrst computes complete matrix M before SVD decomposition and the second sample M entrywise – the second is faster for larger graphs but sacriﬁces on estimation error and performance.
For node2vec (n2v), we use its PyG implementation (Fey & Lenssen, 2019).
Table 1 (right) summarizes results test ROC-AUC.
For our method (denoted f (cid:99) M (WYS) ), we call our functional SVD (Alg. 1) and pass it f (cid:99) M (WYS) as deﬁned in Eq.
Embeddings are set to the SVD basis (as in, §3.1) and edge score of nodes ( u, v ) is ∝ dot-product of embeddings.
The last row of the table shows results when svd rank = 100.
Lastly, we set the context window hyperparameter (a.k.a, length of walk) as follows.
For WYS, we trained with their default context (as WYS learns the context), but for all others (NetMF, n2v, ours) we used context window of length C =5 for datasets
Facebook and PPI (for us, this sets c = [5 , 4 , 3 , 2 , 1] ) and C =20 for AstroPh and HepTh.
B. S ENSITIVITY A NALYSIS
While in §B.2 we tune the number of layers ( L ) using the performance on the validation partition, in this section, we show impact of varying L on test accuracy.
According to the summary in Figure(left), accuracy of classifying a node improves when incorporating information from further nodes.
We see little gains beyond L > 6 .
Note that L = 0 corresponds to ignoring the adjacency matrix altogether when running f (cid:99) M (JKN) .
Here, we
Further, while in §B.3 we do SVD on f (cid:99) M (WYS) with rank k = 32 or k = 100 , Figure 2 (right) shows test accuracy while sweeping k ≤ 32 .
In general, increasing the rank improves estimation accuracy and test performance.
However, if k is larger than the inherit dimensionality of the data, then this could cause overﬁtting (though perfect memorization of training edges).
The Norm Regularization note (§A.3.1) applies only to pseudoinversion i.e. our classiﬁcation models.