GRASPEL: G RAPH S PECTRAL L EARNING AT S CALE
Anonymous authors Paper under double-blind review
A BSTRACT
Learning meaningful graphs from data plays important roles in many data mining and machine learning tasks, such as data representation and analysis, dimension reduction, data clustering, and visualization, etc.
In this work, for the ﬁrst time we present a highly-scalable spectral approach (GRASPEL) for learning large graphs from data.
By limiting the precision matrix to be a graph Laplacian, our approach aims to estimate ultra-sparse (tree-like) weighted undirected graphs and shows a clear connection with the prior graphical Lasso method.
By interleaving the latest high-performance nearly-linear time spectral methods for graph sparsiﬁcation, coarsening and embedding, ultra-sparse yet spectrally-robust graphs can be learned by identifying and including the most spectrally-critical edges into the graph.
Compared with prior state-of-the-art graph learning approaches, GRASPEL is more scalable and allows substantially improving computing efﬁciency and solution quality of a variety of data mining and machine learning applications, such as spectral clustering (SC), and t-Distributed Stochastic Neighbor Embedding (t-SNE). For example, when comparing with graphs constructed using existing methods, GRASPEL achieved the best spectral clustering efﬁciency and accuracy using graphs learned from the proposed approach.
I NTRODUCTION
Graph construction is playing increasingly important roles in many machine learning and data mining applications.
For example, a key step of many existing machine learning methods requires converting potentially high-dimensional data sets into graph representations: it is a common practice to represent each (high-dimensional) data point as a node, and assign each edge a weight to encode the similarity between the two nodes (data points).
The constructed graphs can be efﬁciently leveraged to represent the underlying structure of a data set or the relationship between data points (Jebara et al., 2009;
Maier et al., 2009; Liu et al., 2018).
However, how to learn meaningful graphs from large data set at scale still remains a challenging problem.
In the past decades, considerable effort has been devoted to the development of graph construction methods.
For example, constructing k -nearest-neighbor (kNN) graphs requires each node to be connected with its top- k nearest neighbors, while in construction of the (cid:15) -neighborhood graphs all the neighbors within the range of distance (cid:15) will be connected; to improve the capability of kNN graph in handling multi-scale data, (Zelnik-Manor & Perona, 2005) introduced a self-tuning technique to adjust the local scaling parameter for similarity measurement; to ﬁnd meaningful similarity measures between nodes, (Bach & Jordan, 2006) propose to learn the similarities from feature vectors in a supervised setting; (Zhu et al., 2014) adopted an information-theoretic deﬁnition of data similarity to capture subtle similarity information; (Jebara & Shchogolev, 2006) proposed to remove spurious edges from kNN graph via b-matching; (Pavan & Pelillo, 2007) introduced a method for removing noisy edges by selecting the maximum cliques; (Premachandran & Kakarala, 2013) proposed to leverage collected consensus information form various neighborhoods to improve the robustness of the kNN graph; (Nie et al., 2014) proposed to learn the adjacency graph by adaptively assigning neighbors.
However, the aforementioned nearest-neighbor (NN) based graph construction methods can only capture local manifold information and may not be able to truthfully reveal the global structure of a given data set (Nie et al., 2016; Liu et al., 2018; Guo, 2015), which can result in over complicated (with too many edges) or sometimes misleading graph representations.
For example, choosing different numbers of nearest neighbors for constructing kNN graphs may lead to drastically different classiﬁcation performance in spectral clustering tasks (Chen et al., 2018).
Several recent graph learning methods leverage emerging graph signal processing (GSP) techniques for estimating sparse graph Laplacians, which show very promising results (Dong et al., 2016; Egilmez et al., 2017; Dong et al., 2019; Kalofolias & Perraudin, 2019).
For example, (Egilmez et al., 2017) addresses the graph learning problem by restricting the precision matrix to be a graph Laplacian and maximizing a posterior estimation of Gaussian Markov Random Field (GMRF), while an L 1 - regularization term is used to promote graph sparsity; (Rabbat, 2017) provides an error analysis for inferring sparse graphs from smooth signals; (Kalofolias & Perraudin, 2019) leverages approximate nearest-neighbor (ANN) graphs to reduce the number of variables for optimization.
However, even the state-of-the-art Laplacian estimation methods for graph learning do not scale well for large data set due to their extremely high algorithm complexity.
For example, solving the optimization problem for Laplacian estimation in (Dong et al., 2016; Kalofolias, 2016; Egilmez et al., 2017; Dong et al., 2019) requires O ( N 2 ) time complexity per iteration for N data entities and nontrivial parameters tuning for controlling graph sparsity which limits their applications to only very small data sets (e. g. with up to a few thousands of data points). The latest graph learning approach (Kalofolias &
Perraudin, 2019) take advantages of ANN graphs but can still run rather slowly for large data sets. This work for the ﬁrst time introduces a spectral method (GRASPEL) for learning ultra-sparse graphs from data by leveraging the latest results in spectral graph theory (Feng, 2016; 2018; Zhao et al., 2018).
There is a clear connection between our approach and the GSP-based Laplacian estimation methods (Dong et al., 2016; Kalofolias, 2016; Egilmez et al., 2017; Kalofolias & Perraudin, 2019; Dong et al., 2019), as well as the classical graphical Lasso framework (Friedman et al., 2008).
Speciﬁcally, by treating p -dimensional data points as p graph signals, GRASPEL learns a graph Laplacian by maximizing its ﬁrst few eigenvalues as well as the smoothness of graph signals across edges, subject to a graph sparsity constraint.
By iteratively interleaving recent nearly-linear time spectral graph sparsiﬁcation, coarsening and embedding methods (Feng, 2016; 2018; Zhao et al., 2018), GRASPEL enjoys a nearly-linear runtime and space complexity. GRASPEL is similar to the original graphical Lasso (Friedman et al., 2008) with the precision matrix replaced by a graph Laplacian.
GRASPEL iteratively identiﬁes and includes the most spectrallycritical edges into the latest graph, so that the ﬁrst few Laplacian eigenvalues and eigenvectors can be most signiﬁcantly perturbed by adding the minimum amount of edges.
The iterative graph learning procedure will be terminated when the graph spectra become sufﬁciently stable (or graph signals become sufﬁciently smooth across the graph and lead to rather small Laplacian quadratic forms). Comparing with state-of-the-art methods, GRASPEL allows more scalable estimation of attractive Gaussian Markov Random Fields (GMRFs) for even very large data set. We show through extensive experiments that GRASPEL can learn high-quality ultra-sparse (tree-like) graphs that can be immediately leveraged to signiﬁcantly improve the efﬁciency and accuracy of spectral clustering (SC) tasks; the proposed approach also leads to the development of a multilevel t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm that shows signiﬁcantly improved runtime over existing methods (Maaten & Hinton, 2008; Van Der Maaten, 2014).
B ACKGROUND OF G RAPH LEARNING VIA L APLACIAN ESTIMATION
Given M observations on N data entities stored in a data matrix X ∈ R N × M , each column of X can be considered as a signal on a graph.
The recent graph learning method (Dong et al., 2016) aims to estimate a graph Laplacian from X while achieving the following desired characteristics:
1) Smoothness of signals on the graph.
The graph signals corresponding to the real-world data should be sufﬁciently smooth on the learned graph structure: the signal values will only change gradually across connected neighboring nodes.
The smoothness of a signal x over a graph G can be measured with Laplacian quadratic form x T Lx where L = D − W denotes the Laplacian matrix of graph G with D and W denoting the degree and the weighted adjacency matrices of G , respectively.
w p,q denotes the weight for edge ( p, q ), while x ( p ) and x ( q ) denote the graph signal values on nodes p and q .
The smaller value of quadratic form indicates the smoother signals across the graph.
It is also possible to quantify the smoothness ( Q ) of a set of signals X over graph G using the following matrix trace (Kalofolias, 2016):
Q ( X, L ) =
Tr ( X T LX ) , where Tr denotes the matrix trace.
2) Sparsity of the estimated graph (Laplacian).
Graph sparsity is another critical consideration in graph learning.
One of the most important motivations of learning a graph is to use it for downstream data mining or machine learning tasks.
Therefore, desired graph learning algorithms should allow better capturing and understanding the global structure (manifold) of the data set, while producing sufﬁciently sparse graphs that can be easily stored and efﬁciently manipulated in the downstream algorithms, such as graph clustering, partitioning, dimension reduction, data visualization, etc.
To this end, the graphical Lasso algorithm (Friedman et al., 2008) has been proposed to learn the structure in an undirected Gaussian graphical model using L 1 regularization to control the sparsity of the precision matrix.
Given a sample covariance matrix S and a regularization parameter β , graphical Lasso targets the following objective function:
[Equation]
over all non-negative deﬁnite precision matrices Θ .
The ﬁrst two terms together can be interpreted as the log-likelihood under a Gaussian Markov Random Field (GMRF).
β (cid:107) Θ (cid:107) 1 is the sparsity promoting regularization term.
This model tries to learn the graph structure by maximizing the penalized log-likelihood.
However, the log-determinant problems are very computationally expensive.
The emerging GSP-based methods infer the graph by adopting the criterion of signal smoothness (Kalofolias, 2016; Dong et al., 2016; Egilmez et al., 2017; Kalofolias & Perraudin, 2019).
However, their extremely high complexities do not allow for learning large-scale graphs involving millions or even hundred thousands of nodes.
Furthermore, these methods usually require nontrivial parameters tuning for controlling graph sparsity.
GRASPEL: A SPECTRAL APPROACH TO GRAPH LEARNING FROM DATA
At high level, GRASPEL gains insight from recent GSP-based Laplacian estimation methods (Dong et al., 2019), aiming to solve the following optimization problem that is similar to the graphical Lasso problem (Friedman et al., 2008):
[Equation]
where L denotes the set of valid Laplacian matrices.
It can be shown that the three terms in (2) are corresponding to log det(Θ) ,
Tr (Θ S ) and β (cid:107) Θ (cid:107) 1 in (1), respectively.
When the precision matrix Θ is restricted to be a graph Laplacian, and each column vector in the original data matrix X is treated as a graph signal vector, there is a close connection between our formulation and the graphical
Lasso problem.
Since graph Laplacians are symmetric and positive deﬁnite (PSD) matrices (or M matrices) with non-positive off-diagonal entries, this formulation will lead to the estimation of attractive GMRFs (Dong et al., 2019).
3. O VERVIEW OF GRASPEL
Step 3B. Add
[Table]
edge New edge v S = s 1 ss 3 s⎡ ⎣
⎢⎢⎢⎢⎢⎢ ⎤ ⎦ ⎥⎥⎥⎥⎥⎥ v P = p 1 pp 3 p⎡ ⎣ ⎢⎢⎢⎢⎢⎢ ⎤ ⎦ ⎥⎥⎥⎥⎥⎥
P
S
G
[Figure]
embedding vectors [Figure] If [Figure]
≻
[Figure]
embedding distortion
[Figure] To achieve good efﬁciency in graph learning that may involve millions of nodes, instead of solving (2) directly, GRASPEL leverages a spectral approach for solving (2) implicitly. Deﬁne spectrally-critical edges to be the ones that can most effectively perturb the graph spectral properties, such as the ﬁrst few Laplacian eigenvalues and eigenvectors. GRASPEL aims to iteratively identify and add the most spectrally-critical edges into the latest graph until no such edges can be found, which consists of the following key steps as illustrated in Figure 1:
Step (1): Initial graph construction.
Similar to (Kalofolias & Perraudin, 2019), we start with constructing an ANN graph that can be achieved in O ( N log N ) time, where each edge weight encodes the similarity (e.g. Gaussian kernel or cosine similarity) of two data entities; next, the ANN graph is converted into an ultra-sparse nearest-neighbor (uNN) graph with O ( N log N ) edges by leveraging a nearly-linear-time spectral sparsiﬁcation algorithm
(Feng, 2018).
Step (2): Spectral graph embedding.
We apply a nearly-linear-time spectral embedding procedure (Zhao et al., 2018) to the current graph so that each node will be associated with a low-dimensional embedding vector (e.g. v s for node S in Figure 1), where the embedding dimension (number of eigenvectors) can be determined based on the largest gaps of the ﬁrst few (e.g. 100 ) Laplacian eigenvalues (Peng et al., 2015).
Step (3): Spectrally-critical edge identiﬁcation.
We quickly check the embedding distortion for each candidate edge (node pair) deﬁned as η = z emb z data , where z emb ( z data ) denotes the distance in the embedding (data) vector space.
The edges with the largest η are considered as the most spectrally-critical edges and will be added into the latest graph.
Step (4): Spectral stability checking.
After repeating the Steps (2)-(3) multiple times for adding new edges, GRASPEL will return the ﬁnal graph once the overall embedding distortion becomes sufﬁciently small or stable: if the ﬁrst few Laplacian eigenvalues do not change much over iterations of adding extra edges, the graph spectra is considered stable (robust) since adding more edges does not signiﬁcantly perturb the key (smallest) eigenvalues. Connection between GRASPEL and the formulation (2) . The original optimization objective function includes the following three components: (a) log det( L ) corresponding to the sum of the logarithmic Laplacian eigenvalues, (b) − αTr ( X T LX ) corresponding to the smoothness of signals across the graph 1 , and (c) − β ∗ | L | 1 corresponding to graph sparsity. Including spectrally-critical edges into the graph will signiﬁcantly impact the ﬁrst few Laplacian eigenvalues and eigenvectors key to graph spectral properties, thereby dramatically improving embedding distortion and the overall smoothness of signals across the graph.
It can be shown that including any additional edge into the graph will monotonically increase (a) while monotonically decreasing both (b) and (c). Therefore, when the spectra of the learned graph is not stable adding any spectrally-critical edges into the graph will dramatically increase (a) while slightly decreasing (b) and (c), since the improved graph signal smoothness will only result in a slight change (increase) to Tr ( X T LX ) . Convergence analysis for the GRASPEL framework . The objective function in (2) will be ef- fectively maximized by including only a small amount of spectrally-critical edges until the ﬁrst few eigenvalues become sufﬁciently stable; when adding extra edges can no longer perturb the ﬁrst few eigenvalues, (b) and (c) will start to dominate the objective function value, indicating that the iterations should be terminated.
The stopping condition can be controlled by properly setting an embedding distortion threshold for η or parameters α and β . Complexity analysis for the GRASPEL framework . To achieve scalable spectral graph embedding key to identiﬁcation of spectrally-critical edges in Steps (2)-(3), we will leverage the latest highperformance spectral graph algorithms. Since all the kernel functions involved in GRASPEL, such as ANN graph construction (Muja & Lowe, 2009; 2014; Malkov & Yashunin, 2018), spectral graph sparsiﬁcation (Feng, 2016; 2018), spectral coarsening and embedding (Zhao et al., 2018), are all nearly-linear O ( N log N ) time algorithms, the entire spectral graph learning approach GRASPEL also has a nearly-linear time complexity.
3. D ETAILED S TEPS IN GRASPEL 3.2. I NITIAL GRAPH CONSTRUCTION
As aforementioned, (approximate) kNN graphs can be used to construct the initial graphs in Step (1), since they can be created very efﬁciently (Muja & Lowe, 2009), while being able to approximate the local data proximity (Roweis & Saul, 2000).
However, traditional kNN graphs have the following drawbacks: a) the kNN graphs with large k (the number of nearest neighbors) has the tendency of increasing the cut-ratio (Qian et al., 2012); b) the optimal k value is usually problem dependent and can be very difﬁcult to ﬁnd.
In this work, we will start creating an (approximate) kNN graph with a relatively small k value (e.g. k = 5 ), and strive to signiﬁcantly improve the graph quality by adding extra spectrally-critical edges through implicitly solving the proposed optimization problem in (2).
In the last, a spectral sparsiﬁcation algorithm (GRASS) 2 has been applied to further simplify the kNN graph into a uNN graph with only O ( N log N ) edges (Wang & Feng, 2017).
3.2. S PECTRAL GRAPH EMBEDDING
Spectral graph embedding directly leverages the ﬁrst few nontrivial eigenvectors for mapping nodes onto low-dimensional space (Belkin & Niyogi, 2003).
The eigenvalue decomposition of Laplacian matrix is usually the computational bottleneck in spectral graph embedding, especially for large graphs (Shi & Malik, 2000; Von Luxburg, 2007; Chen et al., 2011).
To achieve good scalability, we exploit multilevel spectrally-reduced graphs that allow for much faster eigenvector (eigenvalue) computations without loss of accuracy (Zhao et al., 2018).
Speciﬁcally, the multilevel method ﬁrst spectrally coarsens the ﬁne-level graph into much smaller ones with preservation of key spectral properties, and then maps the eigenvectors obtained on the coarse graphs back to the original graph; multilevel eigenvector reﬁnement (smoothing) and orthogonalization procedures can be applied to further improve the approximation accuracy (Zhao et al., 2018).
3.2. S PECTRALLY - CRITICAL EDGE IDENTIFICATION
Once Laplacian eigenvectors are available for the current graph, we can identify spectrally-critical edges by looking at each candidate edge’s embedding distortion ( η ).
To this end, we exploit the following ﬁrst-order spectral perturbation analysis to quantitatively evaluate each candidate edge’s impact on the ﬁrst few eigenvalues.
Denote the edge weight by w p,q and the Laplacian eigenvector corresponding to the eigenvalue
λ
i by u i .
We deﬁne e p ∈ R n to be a standard basis vector with all zero entries except for the p -th being 1 , and e p,q = e p − e q .
The following theorem will allow us to identify the most spectrally-critical edges leveraging the ﬁrst few Laplacian eigenvectors.
Theorem 1 The spectral criticality c p,q or embedding distortion η p,q of a candidate edge ( p, q ) on the Laplacian eigenvalue λ i can be properly estimated by c p,q = w p, [Equation] (cid:0) [Equation]
(cid:1)
[Equation]
Proof: See the appendix.
Edge identiﬁcation with Fiedler vectors.
The idea for identifying spectrally-critical edges is to sort nodes according to the Fiedler vector.
Only the nodes with large embedding distances will be examined as candidate edges.
Therefore, we are able to limit the search within the candidate edge connections between the top and bottom few nodes in the 1 D sorted node vector.
Only the candidate edges with top spectral criticality or embedding distortion values will be added into the latest graph.
Edge identiﬁcation with multiple eigenvectors.
With k eigenvectors for spectral embedding, we can ﬁrst project the graph nodes onto a k -dimensional space and perform spectral clustering to group the nodes into k clusters, where the embedding dimension k can be determined based on the largest gaps of the ﬁrst few (e.g. 100 ) Laplacian eigenvalues (Peng et al., 2015).
Next, we only have to examine the candidate edges that connect nodes between two distant clusters in the embedding space, and sort them based on embedding distortions.
To further reduce the computational cost, we ﬁrst perform spectral graph coarsening (Zhao et al., 2018) and subsequently search for high-distortion candidate edges on the coarsest graph.
Once a small set of top spectrally-critical edges has been identiﬁed, we will ﬁnd their corresponding candidate edges in the original graph.
Since each coarse-level candidate edge may correspond to multiple candidate edges in the original graph, we will sort them based on their embedding distortions, and only add the ones with largest distortions into the latest graph.
The algorithms for spectrally-critical edge identiﬁcation using Fiedler vector and multiple eigenvectors have been described in Algorithm 1 and Algorithm 2 in the Appendix.
3.2. S PECTRAL S TABILITY C HECKING
[Figure] We employ an iterative procedure in GRASPEL to repeatedly add new edges into the graph, thereby improving the approximation quality.
Speciﬁcally, at each iteration new spectrally-critical edges are identiﬁed and added to the current graph; we will ter- minate the iterations when the graph spectra become sufﬁciently stable.
Alternatively, we can check if the embedding distortions of top candidate edges still keep improving; if not, the iterations can be termi- nated, indicating that no additional spectrally-critical edges can be found to signiﬁcantly perturb the ﬁrst few Laplacian eigenvectors and eigenvalues.
In practice, we observe that the number of spectrally-critical edges decreases rapidly within very few (two to four) iterations.
For GRASPEL we adopt the following scheme for checking the spectral stability of each graph learning iteration: 1) in each iteration, we compute and record the several smallest eigenvalues of the latest graph
Laplacian according to the largest gap between eigenvalues (Peng et al., 2015): for example, the ﬁrst (smallest) k nonzero eigenvalues that are critical for spectral clustering will be stored; 2) we check whether a sufﬁciently stable spectra has been reached for graph learning by comparing them with the eigenvalues computed in the previous iteration: if the change is signiﬁcant, more iterations may be needed.
To this end, we record the ﬁrst k Laplacian eigenvalues computed in the previous (current) iteration into vector v p ( v p +1 ), and calculate the spectral variation ratio by:
[Equation]
A greater spectral variation ratio indicates less stable eigenvalues within the latest graph, and thus justiﬁes another iteration for adding more spectrally-critical edges into the graph.
The spectral stability checking results for the USPS data set (see Appendix for details) have been shown in Figure 2; as observed, only four iterations will sufﬁce for achieving a rather stable graph spectra.
E XPERIMENTS
In this section, extensive experiments have been conducted to evaluate the performance of GRASPEL for a variety of public domain data sets.
{+Note that all the graphs learned by GRASPEL have ultrasparse tree-like structures with graph densities (deﬁned as | E | / | V | ) between 1 . 1 to 1 . 3 ; our approach allows learning much sparser graphs when comparing with the latest approach that always produces graphs with densities much greater than 3 . 
+}
(Kalofolias & Perraudin, 2019).
4. G RAPH LEARNING FOR SPECTRAL CLUSTERING
As shown in Algorithm 3 in Appendix, the the classical spectral clustering (SC) algorithm ﬁrst constructs a graph where each edge weight encodes similarities between different data points (entities); then SC calculates the eigenvectors of the graph Laplacian matrix and embeds data points into low- dimensional space (Belkin & Niyogi, 2003); in the last, k-means algorithms are used to partition the data points into multiple clusters.
The performance of SC strongly depends on the quality of the underlying graph (Guo, 2015).
In this section, we apply GRASPEL for graph construction, and show the learned graphs can result in drastically improved efﬁciency and accuracy in SC tasks.
The detailed description of our evaluation metrics, data sets and experiment setup has been provided in Appendix.
Table 1 shows the ACC and NMI results of SC with graphs constructed by different methods with the best numbers highlighted, where graph construction time (Time-C) and spectral clustering time (Time-S) that involves eigendecomposition and kmeans clustering have also been reported.
Note that the high computational and memory cost of recent GSP-based graph learning methods, such as GL-SigRep (Dong et al., 2016), GL-Logdet (Dong et al., 2016) and GLSC (Egilmez et al., 2017) do not allow for processing data sets with more than a few thousands of data entities, thus can not be used for real-world SC tasks.
We observe that GRASPEL can consistently lead to dramatic performance improvement in SC. Speciﬁcally, GRASPEL beats all competitors in clustering accuracy (ACC) across all data sets: GRASPEL achieves more than 35% accuracy gain on USPS and 10% gain on COIL20 over the second-best methods; for the MNIST data set GRASPEL also achieves over 10% accuracy gain over the SC with standard kNN graph and more than 6 X speedup in graph construction time. Note that the graphs learned by GRASPEL are ultra sparse and have tree-like structures, thereby allowing much faster eigendecompositions in SC when comparing with other methods (Wang & Feng, 2017): the SC of the MNIST data set with standard kNN takes over 6000 seconds, which will be dramatically improved to take less than three seconds (over 2000 X speedup) using the graph learned by our method (GRASPEL).
The superior performance of GRASPEL is due to the following reasons: 1) In traditional kNN graphs, all the nodes have the same degrees; as a result, the clustering may strongly favor balanced cut, which may lead to improper cuts in high-density regions of the graph.
In contrast, GRASPEL always learns ultra-sparse (tree-like) graphs that only include edges with the largest impact to graph spectral (structural) properties; as a result, the corresponding cuts will always occur in proper regions of the graph, which enables to handle even unbalanced data.
Our approach is less susceptible to noise in similarity calculations, since it only connects two nodes according to its spectral criticality (embedding distortion).
Based on matrix perturbation theory (Stewart & Sun, 1990), eigenvectors corresponding to small eigenvalues can be more inﬂuenced by noises in similarities, which can lead to accuracy degradation when the number of desired clusters is not small (Hennig et al., 2015).
Although consensus kNN method attempts to use consensus information from kNN graph for discarding noisy edges, the improvement can still be rather limited since it can be very difﬁcult to extract useful consensus information from a given kNN graph with a small neighborhood size. 4. R ESULTS FOR G RAPH RECOVERY We also quantitatively compare graph recovery performance of GRASPEL with state-of-the-art GSP-based graph learning methods, by comparing the graphs learned from observations to the ground
[Table]
[Table]
truth. The experiments are performed for two widely-used synthetic graphs: 1) The Gaussian graph: the coordinates of the vertices are generated uniformly in the unit square randomly.
Edge weights are determined by the Gaussian radial basis function. The ER graph: the graphs generated by following the Erdos-Renyi model (Erd ˝ os & Rényi, 1960). Four widely adopted evaluation metrics in information retrieval have been adopted: Precision, Recall, F-measure and
Normalized Mutual Information
(Dong et al., 2016).
The Precision measures the percentage of correct edges (the edges that are present in the ground-truth graph) in the learned graph.
The Recall measures the percentage of the edges in the ground-truth graph that are also in the learned graph.
F-measure measures the overall quality by taking both Precision and Recall into account.
NMI measures the mutual dependence between the edge sets of the learned graph and the ground-truth graph. The best two F-measure and NMI results have been highlighted in Table 2, showing the effectiveness of GRASPEL in learning graphs that are always close to the ground-truth graphs.
Compared with other graph learning methods that can only deal with a few hundreds or thousands of data entities, GRASPEL has much better (nearly-linear runtime and space) scalability and thus will be more efﬁcient for handling large data sets.
[Figure]
As shown in Figure 3, the graph recovery runtime results of GRASPEL has been compared with state-of-the-art graph learning meth- ods, such as the GL-SigRep, GL-LogDet and GLSC algorithm proposed in (Dong et al., 2016; Egilmez et al., 2017). Since the graph learning method LSGL proposed in (Kalofolias & Perraudin, 2019) can not produce comparable quality of graph recovery results, we did not show the runtime results in the ﬁgure. As observed, the proposed approach has a much better runtime scalability when comparing with stateof-the-art methods.
4. A PPLICATIONS IN T -SNE
The t-Distributed Stochastic Neighbor Embed- ding (t-SNE) has become one of the most popular visualization tools for high-dimensional data analytic tasks (Maaten & Hinton, 2008; Linderman & Steinerberger, 2017).
However, its high computational cost ( O ( N 2 ) where N is the number of data points in the data set) limits its applicability to large scale problems.
An substantially improved t-SNE algorithm has been introduced based on tree approximation (Van Der Maaten, 2014).
However, for large data set the computational cost can still be very high.
A multilevel t-SNE algorithm has been proposed in (Zhao et al., 2018) to dramatically mitigate the computational burden by leveraging spectral graph coarsening as a pre-processing step applied to the original kNN graph. A much smaller set of representative data points can be then selected from the coarsened graph for t-SNE visualization.
However, constructing the original kNN graph can still be costly while the performance may strongly depend on the selection of the number of nearest neighbors.
In this work, we use GRASPEL to learn ultra-sparse graphs that can be further reduced into much smaller ones using the spectral graph reduction (Zhao et al., 2018).
Then more efﬁcient t-SNE visualization can be achieved based on the data points corresponding to the nodes in the coarsened graphs. Figure 4 shows the visualization and runtime results of the standard t-SNE (with tree-based accelera- tion) (Van Der Maaten, 2014) and the multilevel t-SNE algorithm (Zhao et al., 2018) based on graphs learned by GRASPEL.
The runtime for the multi-level t-SNE method covers both the graph learning and t-SNE procedures.
When using a 5 X graph reduction ration, it can be seen that t-SNE runtime for both data sets can be dramatically improved (12.8X and 7X speedups for MNIST and USPS data sets, respectively) without loss of visualization quality.
C ONCLUSION
In this work, we present a scalable spectral approach to graph learning from data.
By replacing the precision matrix with a graph Laplacian, our approach aims to estimate ultra-sparse weighted graphs and has a clear connection with the prior graphical Lasso method.
Compared with prior graph learning approaches that do not scale to large problems, our approach is more scalable for constructing graphs that can immediately lead to substantially improved computing efﬁciency and solution quality for a variety of data mining and machine learning applications, such as spectral clustering (SC), and t-Distributed Stochastic Neighbor Embedding (t-SNE).
A PPENDIX A P ROOF OF T HEOREM
Let L P denote the Laplacian matrix of an undirected graph P , and u i denote the i -th eigenvector of L
P corresponding to the i -th
eigenvalue λ i that satisﬁes:
[Equation]
then we have the following eigenvalue perturbation analysis:
[Equation]
where a perturbation δL P that includes a new edge connection is applied to L P , resulting in perturbed eigenvalues and eigenvectors λ
i + δλ i and u i
+ δu i for i = 1 , ..., n , respectively.
Keeping only the ﬁrst-order terms leads to:
[Equation]
Write δu i in terms of the original eigenvectors u i for for i = 1 , ..., n :
[Equation]
Substituting (7) into (6) leads to:
[Equation]
Multiplying u Ti to both sides of (8) results in:
[Equation]
Since u i for for i = 1 , ..., n are unit-length, mutually-orthogonal eigenvectors, we have:
[Equation]
[Equation]
Substituting (4) into (10), we have:
[Equation]
According to (10), we have:
[Equation]
Substituting (12) into (9) leads to:
[Equation]
Then the eigenvalue perturbation due to δL P is given by:
[Equation]
[Equation]
If each edge weight w p,q encodes the similarity of data vectors x p and x q at nodes p and q , it can be shown that w p,q ∝ 1 z data , where z data denotes the distance between x p and x q ; on the other hand, (cid:0) u Ti e p,q (cid:1) 2 ∝ z emb .
Therefore, as long as we can ﬁnd an edge with large w p,q (cid:0) u Ti e p,q (cid:1) 2 or η p,q = z embp,q z datap,q , including this edge into the current graph will signiﬁcantly perturb the Laplacian eigenvalue
λ i and eigenvector u i .
A PPENDIX B A LGORITHMS
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
A PPENDIX C D ATA S ETS D ESCRIPTION
COIL20:
A data set contains 1 , 440 gray-scale images of 20 objects, and each object on a turntable has 72 normalized gray-scale images taken from different degrees.
The image size is 32x 32 pixels.
PenDigits: A data set consists of 7,494 images of handwritten digits from 44 writers, using the sampled coordination information.
Each digit is represented by 16 attributes.
USPS: A data set includes 9 , 298 scanned hand-written digits on the envelops from U.S. Postal Service with 256 attributes.
MNIST:
A data set consists of 70,000 images of handwritten digits.
Each image has 28-by-28 pixels in size.
This database can be found from Prof.Yann LeCun’s website (http://yann.lecun.com/exdb/mnist/).
A PPENDIX D C OMPARED A LGORITHMS
Standard kNN : the most widely used afﬁnity graph construction method.
Each node is connected to its k nearest neighbors.
Consensus of kNN (cons-kNN) (Premachandran & Kakarala, 2013): adopts the state-of-the-art neighborhood selection methods to construct the afﬁnity graphs.
It selects strong neighborhoods to improve the robustness of the graph by using the consensus information from different neighborhoods in a given kNN graph.
LSGL (Kalofolias & Perraudin, 2019): a method to automatically select the parameters of the model introduced in (Kalofolias, 2016) given a desired graph sparsity level.
GL-SigRep (Dong et al., 2016): construct a graph from signals that are assumed to be smooth with respect to the corresponding graph.
GL-LogDet (Dong et al., 2016): encodes the information about the partial correlations between the variables without the constraint to form a valid Laplacian.
Graph Learning under structural constraints (GLSC) (Egilmez et al., 2017): formulated the problem as to maximum a posterior estimation of Gaussian Markov Random Field (GMRF) when the precision matrix is chosen to be a graph Laplacian.
A PPENDIX E E VALUATION M ETRIC
[Equation]
(15) where n is the number of samples in the data set, y i is the ground-truth label provided by the data sets, and c i is clustering result obtained from the algorithm.
δ ( x, y ) is a delta function deﬁned as: δ ( x, y )
=
for x = y , and δ ( x, y ) =0, otherwise.
map ( • ) is a permutation function that maps each cluster index
c i to a ground truth label, which can be realized using the Hungarian algorithm (Papadimitrou & Steiglitz, 1982).
ACC measures the agreement between the clustering results generated by clustering algorithms and the ground-truth labels.
A higher value of ACC indicates better clustering quality.
For two random variables P and Q , normalized mutual information is deﬁned as (Strehl & Ghosh, 2002):
[Equation]
where I ( P, Q ) denotes the mutual information between P and Q , while H ( P ) and H ( Q ) are entropies of P and Q .
In practice, the NMI metric can be calculated as follows (Strehl & Ghosh, 2002):
[Equation]
where n is the number of data points in the data set, k is the number of clusters, n i is the number of data points in cluster C
i according to the clustering result generated by algorithm, n j is the number of data points in class C j according to the ground truth labels provided by the data set, and n i,j is the number of data points in cluster C
i according to the clustering result as well as in class C j according to the ground truth labels.
The NMI value is in the range of [0, 1], while a higher NMI value indicates a better matching between the algorithm generated result and ground truth result.