Original: L ARGE S CALE G RAPH L EARNING FROM S MOOTH S IGNALS
Revised: L ARGE S CALE G RAPH L EARNING FROM S MOOTH S IGNALS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graphs are a prevalent tool in data science, as they model the inherent structure of the data.
Revised: Graphs are a prevalent tool in data science, as they model the inherent structure of the data.

Original: Typically they are constructed either by connecting nearest samples, or by learning them from data, solving an optimization problem.
Revised: Typically they are constructed either by connecting nearest samples, or by learning them from data, solving an optimization problem.

Original: While graph learning does achieve a better quality, it also comes with a higher computational cost.
Revised: While graph learning does achieve a better quality, it also comes with a higher computational cost.

Original: In particular, the current state-of-the-art model cost is O (cid:0) n 2 (cid:1) for n samples.
Revised: In particular, the current state-of-the-art model cost is O (cid:0) n 2 (cid:1) for n samples.

Original: In this paper, we show how to scale it, obtaining an approximation with leading cost of O ( n log( n )) , with quality that approaches the exact graph learning model.
Revised: In this paper, we show how to scale it, obtaining an approximation with leading cost of O ( n log( n )) , with quality that approaches the exact graph learning model.

Original: Our algorithm uses known approximate nearest neighbor techniques to reduce the number of variables, and automatically selects the correct parameters of the model, requiring a single intuitive input: the desired edge density.
Revised: Our algorithm uses known approximate nearest neighbor techniques to reduce the number of variables, and automatically selects the correct parameters of the model, requiring a single intuitive input: the desired edge density.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs are an invaluable tool in data science, as they can capture complex structures inherent in seemingly irregular high-dimensional data.
Revised: Graphs are an invaluable tool in data science, as they can capture complex structures inherent in seemingly irregular high-dimensional data.

Original: While classical applications of graphs include data embedding, manifold learning, clustering and semi-supervised learning (Zhu et al., 2003; Belkin et al., 2006; Von Luxburg, 2007), they were later used for regularizing various machine learning models, for example for classiﬁcation, sparse coding, matrix completion, or PCA (Zhang et al., 2006;
Revised: While classical applications of graphs include data embedding, manifold learning, clustering and semi-supervised learning (Zhu et al., 2003; Belkin et al., 2006; Von Luxburg, 2007), they were later used for regularizing various machine learning models, for example for classiﬁcation, sparse coding, matrix completion, or PCA (Zhang et al., 2006;

Original: Zheng et al., 2011; Kalofolias et al., 2014; Shahid et al., 2016).
Revised: Zheng et al., 2011; Kalofolias et al., 2014; Shahid et al., 2016).

Original: More recently, graphs drew the attention of the deep learning community.
Revised: More recently, graphs drew the attention of the deep learning community.

Original: While convolutional neural networks (CNNs) were highly successful for learning image representations, it was not obvious how to generalize them for irregular high dimensional domains, were standard convolution is not applicable.
Revised: While convolutional neural networks (CNNs) were highly successful for learning image representations, it was not obvious how to generalize them for irregular high dimensional domains, were standard convolution is not applicable.

Original: Graphs bridge the gap between irregular data and CNNs through the generalization of convolutions on graphs (Defferrard et al., 2016; Kipf & Welling, 2016;
Revised: Graphs bridge the gap between irregular data and CNNs through the generalization of convolutions on graphs (Defferrard et al., 2016; Kipf & Welling, 2016;

Original: Monti et al., 2016; Li et al., 2015).
Revised: Monti et al., 2016; Li et al., 2015).

Original: While clearly graph quality is important in such applications (Henaff et al., 2015; Defferrard et al., 2016), the question of how to optimally construct a graph remains an open problem.
Revised: While clearly graph quality is important in such applications (Henaff et al., 2015; Defferrard et al., 2016), the question of how to optimally construct a graph remains an open problem.

Original: The ﬁrst applications mostly used weighted k -nearest neighbors graphs ( k -NN) (Zhu et al., 2003;
Revised: The ﬁrst applications mostly used weighted k -nearest neighbors graphs ( k -NN) (Zhu et al., 2003;

Original: Belkin et al., 2006; Von Luxburg, 2007), but the last few years more sophisticated methods of learning graphs from data were proposed.
Revised: Belkin et al., 2006; Von Luxburg, 2007), but the last few years more sophisticated methods of learning graphs from data were proposed.

Original: Today, graph learning , or network inference , has become an important problem itself (Wang & Zhang, 2008; Daitch et al., 2009;
Revised: Today, graph learning , or network inference , has become an important problem itself (Wang & Zhang, 2008; Daitch et al., 2009;

Original: Jebara et al., 2009; Lake &
Revised: Jebara et al., 2009; Lake &

Original: Tenenbaum, 2010; Hu et al., 2013;
Revised: Tenenbaum, 2010; Hu et al., 2013;

Original: Dong et al., 2015; Kalofolias, 2016).
Revised: Dong et al., 2015; Kalofolias, 2016).

Original: Hoever, graph learning is computationally too costly for large-scale applications that need graphs between millions of samples.
Revised: Hoever, graph learning is computationally too costly for large-scale applications that need graphs between millions of samples.

Original: Current state of the art models for learning weighted undirected graphs (Dong et al., 2015; Kalofolias, 2016) cost O (cid:0) n 2 (cid:1) per iteration for n nodes, while previous solutions are even more expensive.
Revised: Current state of the art models for learning weighted undirected graphs (Dong et al., 2015; Kalofolias, 2016) cost O (cid:0) n 2 (cid:1) per iteration for n nodes, while previous solutions are even more expensive.

Original: Furthermore, they need parameter tuning to control sparsity, and this adds an extra burden making them prohibitive for applications with more than a few thousands of nodes.
Revised: Furthermore, they need parameter tuning to control sparsity, and this adds an extra burden making them prohibitive for applications with more than a few thousands of nodes.

Original: Large-scale applications can only resort to approximate nearest neighbors (A-NN), e.g. (Dong et al., 2011; Muja & Lowe, 2014; Malkov & Yashunin, 2016), that run with a cost of O ( n log( n )) .
Revised: Large-scale applications can only resort to approximate nearest neighbors (A-NN), e.g. (Dong et al., 2011; Muja & Lowe, 2014; Malkov & Yashunin, 2016), that run with a cost of O ( n log( n )) .

Original: This is low compared to computing a simple k -NN graph, as even the pairwise distance matrix between all samples needs O (cid:0) n 2 (cid:1) computations.
Revised: This is low compared to computing a simple k -NN graph, as even the pairwise distance matrix between all samples needs O (cid:0) n 2 (cid:1) computations.

Original: However, the quality of A-NN is worse than the k -NN, that is already not as good as if we would learn the graph from data.
Revised: However, the quality of A-NN is worse than the k -NN, that is already not as good as if we would learn the graph from data.

Original: In this paper, we propose the ﬁrst scalable graph learning method, with the same leading cost as
Revised: In this paper, we propose the ﬁrst scalable graph learning method, with the same leading cost as

Original: A-NN, and with quality that approaches state-of-the-art graph learning.
Revised: A-NN, and with quality that approaches state -of-the-art graph learning.

Original: Our method leverages A-NN graphs to effectively reduce the number of variables, and the state-of-the-art graph learning model by Kalofolias (2016) in order to achieve the best of both worlds: low cost and good quality.
Revised: Our method leverages A-NN graphs to effectively reduce the number of variables, and the state-of-the-art graph learning model by Kalofolias (2016) in order to achieve the best of both worlds: low cost and good quality.

Original: In
Revised: In

Original: Figure 1 we illustrate the advantage of our solution compared to the current state-of-the-art.
Revised: Figure 1 we illustrate the advantage of our solution compared to the current state-of-the-art.

Original: Note that while the standard model costs the same regardless of the graph density (average node degree) k , our solution beneﬁts from the desired graph sparsity to reduce computation.
Revised: Note that while the standard model costs the same regardless of the graph density (average node degree) k , our solution beneﬁts from the desired graph sparsity to reduce computation.

Original: One of our key contributions is to provide a method to automatically select the parameters of the model by Kalofolias (2016) given a desired graph sparsity level.
Revised: One of our key contributions is to provide a method to automatically select the parameters of the model by Kalofolias (2016) given a desired graph sparsity level.

Original: Like in k -NN, the user can choose the number of neighbors k , without performing grid search over two parameters.
Revised: Like in k -NN, the user can choose the number of neighbors k , without performing grid search over two parameters.

Original: Using our scheme, we can learn a 1-million-nodes graph with a desired sparsity level on a desktop computer inminutes, with a simple Matlab implementation.
Revised: Using our scheme, we can learn a 1-million-nodes graph with a desired sparsity level on a desktop computer inminutes, with a simple Matlab implementation.

Original:  G RAPH L EARNING FROM S MOOTH S IGNALS
Revised:  G RAPH L EARNING FROM S MOOTH S IGNALS

Original: A widely used assumption for data residing on graphs is that values change smoothly across adjacent nodes.
Revised: A widely used assumption for data residing on graphs is that values change smoothly across adjacent nodes.

Original: The smoothness of a set of vectors x 1 , . . .
Revised: The smoothness of a set of vectors x 1 , . . .

Original: , x n ∈ R d on a given weighted undirected graph is usually quantiﬁed by the Dirichlet energy (Belkin & Niyogi, 2001)
Revised: , x n ∈ R d on a given weighted undirected graph is usually quantiﬁed by the Dirichlet energy (Belkin & Niyogi, 2001)

Original: [Equation]
Revised: [Equation]

Original: (1) where W ij ∈ R + denotes the weight of the edge between nodes i and j , L = D − W is the graph
Revised: (1) where W ij ∈ R + denotes the weight of the edge between nodes i and j , L = D − W is the graph

Original: Laplacian, and D ii = (cid:80) j W ij is the diagonal weighted degree matrix.
Revised: Laplacian, and D ii = (cid:80) j W ij is the diagonal weighted degree matrix.

Original: The ﬁrst works for graph learning focused on learning the weights of a ﬁxed k-nearest neighbor pattern (Wang & Zhang, 2008), learning a binary pattern (Jebara et al., 2009) or the whole adjacency matrix (Daitch et al., 2009).
Revised: The ﬁrst works for graph learning focused on learning the weights of a ﬁxed k-nearest neighbor pattern (Wang & Zhang, 2008), learning a binary pattern (Jebara et al., 2009) or the whole adjacency matrix (Daitch et al., 2009).

Original: A more recent family of models is based on minimizing the Dirichlet energy on a graph (Lake & Tenenbaum, 2010;
Revised: A more recent family of models is based on minimizing the Dirichlet energy on a graph (Lake & Tenenbaum, 2010;

Original: Hu et al., 2013; Dong et al., 2015; Kalofolias, 2016).
Revised: Hu et al., 2013; Dong et al., 2015; Kalofolias, 2016).

Original: In the latter, Kalofolias (2016) proposed a uniﬁed model for learning a graph from smooth signals, that reads as follows: 
Revised: In the latter, Kalofolias (2016) proposed a uniﬁed model for learning a graph from smooth signals, that reads as follows: [Equation] Here, Z

Original: [Equation]
Revised: [Equation]

Revised: =

Original: Here, Z ij = (cid:107) x i − x j (cid:107) 2 , ◦ denotes the Hadamard product, and the ﬁrst term is equal to tr (cid:0) X (cid:62) LX (cid:1) .
Revised: [Equation] x i [Equation] x j (cid:107) 2 , ◦ denotes the Hadamard product, and the ﬁrst term is equal to tr (cid:0) X (cid:62) LX (cid:1) .

Original: The optimization is over the set W of valid adjacency matrices (non-negative, symmetric, with zero diagonal).
Revised: The optimization is over the set W of valid adjacency matrices (non-negative, symmetric, with zero diagonal).

Original: The role of matrix function f ( W ) is to prevent W from obtaining a trivial zero value, control sparsity, and impose further structure, depending on the data and the application.
Revised: The role of matrix function f ( W ) is to prevent W from obtaining a trivial zero value, control sparsity, and impose further structure, depending on the data and the application.

Original: Kalofolias obtained state-of-the-art results using
Revised: Kalofolias obtained state-of-the-art results using

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) where 1 = [1 , . . .
Revised: where 1 = [1 , . . .

Original: 1] (cid:62) .
Revised: 1] (cid:62) .

Original: We will call this the log model .
Revised: We will call this the log model .

Original: The previous state of the art was proposed by
Revised: The previous state of the art was proposed by

Original: Hu et al.
Revised: Hu et al.

Original: (2013) and by Dong et al.
Revised: (2013) and by Dong et al.

Original: (2015), using
Revised: (2015), using

Original: [Equation]
Revised: [Equation]

Original: where 1 { condition } = 0 if condition holds, ∞ otherwise.
Revised: where 1 { condition } = 0 if condition holds, ∞ otherwise.

Original: In the sequel we call this the (cid:96) 2 model .
Revised: In the sequel we call this the (cid:96) 2 model .

Original: Since W 1 is the node degrees’ vector, the log model (3) prevents the formation of disconnected nodes due to the logarithmic barrier, while the (cid:96) 2 model (4) controls sparsity by penalizing large degrees due to the ﬁrst term.
Revised: Since W 1 is the node degrees’ vector, the log model (3) prevents the formation of disconnected nodes due to the logarithmic barrier, while the (cid:96) 2 model (4) controls sparsity by penalizing large degrees due to the ﬁrst term.

Original:  C ONSTRAINED EDGE PATTERN
Revised:  C ONSTRAINED EDGE PATTERN

Original: In traditional graph learning, all (cid:0) n 2 (cid:1) possible edges between n nodes are considered, that results in a cost of at least O (cid:0) n 2 (cid:1) computations per iteration.
Revised: In traditional graph learning, all (cid:0) n 2 (cid:1) possible edges between n nodes are considered, that results in a cost of at least O (cid:0) n 2 (cid:1) computations per iteration.

Original: Often, however, we need graphs with a roughly ﬁxed number of edges per node, like in k -NN graphs.
Revised: Often, however, we need graphs with a roughly ﬁxed number of edges per node, like in k -NN graphs.

Original: It is natural to then ask ourselves whether the cost of graph learning can be reduced, reﬂecting the ﬁnal desired graph sparsity.
Revised: It is natural to then ask ourselves whether the cost of graph learning can be reduced, reﬂecting the ﬁnal desired graph sparsity.

Original: In fact, the original problem (2) for the log model (3) can be solved efﬁciently when a constrained set
Revised: In fact, the original problem (2) for the log model (3) can be solved efﬁciently when a constrained set

Original: E allowed ⊆ { ( i, j ) : i < j } of allowed edges is known a priori.
Revised: E allowed ⊆ { ( i, j ) : i < j } of allowed edges is known a priori.

Original: In that case, it sufﬁces to solve the modiﬁed problem
Revised: In that case, it sufﬁces to solve the modiﬁed problem

Original: [Equation]
Revised: [Equation]

Original: where we optimize in the constrained set of adjacency matrices W ∈ (cid:102) W .
Revised: where we optimize in the constrained set of adjacency matrices W ∈ (cid:102) W .

Original: After reducing the set of edges to E allowed , it sufﬁces to solve the modiﬁed problem (5) Following Kalofolias (2016), we can rewrite the problem as minimize
Revised: After reducing the set of edges to E allowed , it sufﬁces to solve the modiﬁed problem (5) Following Kalofolias (2016), we can rewrite the problem as minimize

Original: [Equation]
Revised: [Equation]

Original: 6) with
Revised: 6) with

Original: [Equation]
Revised: [Equation]

Original: where ζ is the Lipschitz constant of f 3 .
Revised: where ζ is the Lipschitz constant of f 3 .

Original: Note that we gather all free parameters of the adjacency matrix (cid:102) W ∈ (cid:102) W m in a vector (cid:101)
Revised: Note that we gather all free parameters of the adjacency matrix (cid:102) W ∈ [Equation]

Original:  w ∈ (cid:102) W v of size only |E allowed | , that is, the number of allowed edges, each counted only once.
Revised: m in a vector (cid:101) w ∈ (cid:102) W v of size only |E allowed | , that is, the number of allowed edges, each counted only once.

Original: Accordingly, in ˜ z
Revised: Accordingly, in ˜ z

Original: = z ( E allowed ) we only keep the corresponding pairwise distances from matrix Z .
Revised: = z ( E allowed ) we only keep the corresponding pairwise distances from matrix Z .

Original: The linear operator K = (cid:101) S = S (: , E allowed ) is also modiﬁed, keeping only the columns corresponding to the edges in E allowed .
Revised: The linear operator K = (cid:101) S = S (: , E allowed ) is also modiﬁed, keeping only the columns corresponding to the edges in E allowed .

Original: In this form, the problem can be solved by the primal dual techniques by Komodakis & Pesquet (2015).
Revised: In this form, the problem can be solved by the primal dual techniques by Komodakis & Pesquet (2015).

Original: The cost of the dual step, operating on the dual variable v (degrees vector) remains O ( n ) .
Revised: The cost of the dual step, operating on the dual variable v (degrees vector) remains O ( n ) .

Original: However, the cost of the primal step, as well as the cost of applying the modiﬁed operator (cid:101) S in order to exchange between the primal and dual spaces is O (cid:0) E allowed (cid:1) instead of O (cid:0) n 2 (cid:1) of the initial algorithm 1 by Kalofolias (2016), reducing the overall complexity.
Revised: However, the cost of the primal step, as well as the cost of applying the modiﬁed operator (cid:101) S in order to exchange between the primal and dual spaces is O (cid:0) E allowed (cid:1) instead of O (cid:0) n 2 (cid:1) of the initial algorithm 1 by Kalofolias (2016), reducing the overall complexity.

Original: In some cases, a pattern of allowed edges E allowed can be induced by constraints of the model, for example sensor networks only assume connections between geographically nearby sensors.
Revised: In some cases, a pattern of allowed edges E allowed can be induced by constraints of the model, for example sensor networks only assume connections between geographically nearby sensors.

Original: In most applications, however, a constrained set is not known beforehand, and we need to approximate the edge support of the ﬁnal learned graph in order to reduce the number of variables.
Revised: In most applications, however, a constrained set is not known beforehand, and we need to approximate the edge support of the ﬁnal learned graph in order to reduce the number of variables.

Original: To this end, we propose using approximate nearest neighbors graphs to obtain a good approximation.
Revised: To this end, we propose using approximate nearest neighbors graphs to obtain a good approximation.

Original: While computing a k -NN graph needs O (cid:0) n 2 d (cid:1) computations, approximate nearest neighbors (A-NN) algorithms (Muja & Lowe, 2009; Dong et al., 2011; Muja & Lowe, 2012; 2014; Malkov & Yashunin, 2016) offer a good compromise between accuracy and speed.
Revised: While computing a k -NN graph needs O (cid:0) n 2 d (cid:1) computations, approximate nearest neighbors (A-NN) algorithms (Muja & Lowe, 2009; Dong et al., 2011; Muja & Lowe, 2012; 2014; Malkov & Yashunin, 2016) offer a good compromise between accuracy and speed.

Original: Speciﬁcally, A-NN methods scale gracefully with the number of nodes n , the fastest ones having an overall complexity of O ( n log( n ) d ) for d -dimensional data.
Revised: Speciﬁcally, A-NN methods scale gracefully with the number of nodes n , the fastest ones having an overall complexity of O ( n log( n ) d ) for d -dimensional data.

Original: When approximating the support of the ﬁnal edges of a graph, we prefer to have false positives than false negatives.
Revised: When approximating the support of the ﬁnal edges of a graph, we prefer to have false positives than false negatives.

Original: We thus start with an initial support with a larger cardinality than that of the desired ﬁnal graph, and let the weight learning step automatically select which edges to set to zero.
Revised: We thus start with an initial support with a larger cardinality than that of the desired ﬁnal graph, and let the weight learning step automatically select which edges to set to zero.

Original: We select a set E allowed with cardinality |E allowed |
Revised: We select a set E allowed with cardinality |E allowed |

Original: = O ( nkr ) , where k is the desired number of neighbors per node and r a small multiplicative factor.
Revised: = O ( nkr ) , where k is the desired number of neighbors per node and r a small multiplicative factor.

Original: By setting the sparsity parameters correctly, the graph learning step will only keep the ﬁnal O ( nk ) edges, setting the less important or wrong edges to zero.
Revised: By setting the sparsity parameters correctly, the graph learning step will only keep the ﬁnal O ( nk ) edges, setting the less important or wrong edges to zero.

Original: The bigger the factor r , the more freedom the learning algorithm has to select the right edges.
Revised: The bigger the factor r , the more freedom the learning algorithm has to select the right edges.

Original: 3. O VERALL THEORETICAL COMPLEXITY
Revised: 3. O VERALL THEORETICAL COMPLEXITY

Original: The cost of learning a kr -A-NN graph is O ( n log( n ) d ) for n nodes and data in R d , while additionally learning the edge weights costs O ( krn ) per iteration.
Revised: The cost of learning a kr -A-NN graph is O ( n log( n ) d ) for n nodes and data in R d , while additionally learning the edge weights costs O ( krn ) per iteration.

Original: The overall complexity is therefore
Revised: The overall complexity is therefore

Original: O ( n log( n ) d ) + O ( nkrI ) for I iterations.
Revised: O ( n log( n ) d ) + O ( nkrI ) for I iterations.

Original: For large n , the dominating cost is asymptotically the one of computing the A-NN and not the cost of learning the weights on the reduced set.
Revised: For large n , the dominating cost is asymptotically the one of computing the A-NN and not the cost of learning the weights on the reduced set.

Original:  A UTOMATIC PARAMETER SELECTION
Revised:  A UTOMATIC PARAMETER SELECTION

Original: A major problem of models (3) and (4) is the choice of meaningful parameters α, β , as grid search increases computation signiﬁcantly.
Revised: A major problem of models (3) and (4) is the choice of meaningful parameters α, β , as grid search increases computation signiﬁcantly.

Original: We show how this burden can be completely avoided for model (3).
Revised: We show how this burden can be completely avoided for model (3).

Original: First, we show that sparsity depends effectively on a single parameter, and then we propose a method to set it automatically for any k .
Revised: First, we show that sparsity depends effectively on a single parameter, and then we propose a method to set it automatically for any k .

Original: Our method is based on predicting the number of edges of any node for a given parameter value, if we relax the symmetricity constraint of W .
Revised: Our method is based on predicting the number of edges of any node for a given parameter value, if we relax the symmetricity constraint of W .

Original: 4. R EDUCTION TO A SINGLE OPTIMIZATION PARAMETER
Revised: 4. R EDUCTION TO A SINGLE OPTIMIZATION PARAMETER

Original: In (Kalofolias, 2016, Proposition 2), it is argued that model (3) effectively has one parameter changing the shape of the edges, the other changing the magnitude.
Revised: In (Kalofolias, 2016, Proposition 2), it is argued that model (3) effectively has one parameter changing the shape of the edges, the other changing the magnitude.

Original: We reformulate this claim as follows:
Revised: We reformulate this claim as follows:

Original: Proposition 1.
Revised: Proposition 1.

Original: Let W ∗
Revised: Let W ∗

Original: Z, α, β ) denote the solution of model (3) for input distances Z and parameters α, β > 0 .
Revised: Z, α, β ) denote the solution of model (3) for input distances Z and parameters α, β > 0 .

Original: Then the same solution can be obtained with ﬁxed parameters α = 1 and β = 1 , by multiplying the input distances by θ =
Revised: Then the same solution can be obtained with ﬁxed parameters α = 1 and β = 1 , by multiplying the input distances by θ =

Original: [Equation]
Revised: [Equation]

Original: and the resulting edges by δ = (cid:113) αβ 
Revised: and the resulting edges by δ = (cid:113) αβ :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Apply (Kalofolias, 2016, Prop. 2), with γ = (cid:113) αβ and divide all operands bythe same constant √ αβ .
Revised: Apply (Kalofolias, 2016, Prop. 2), with γ = (cid:113) αβ and divide all operands by √ αβ .

Original: Proposition 1 shows that thetwo parameter spaces ( α, β ) and ( θ, δ ) are equivalent.
Revised: Proposition 1 shows that the parameter spaces ( α, β ) and ( θ, δ ) are equivalent.

Original: While the ﬁrst one is convenient to deﬁne (3), the second one makes the sparsityanalysis and the application of the model simpler.
Revised: While the ﬁrst one is convenient to deﬁne (3), the second one allows independent tuning of sparsity and scale of the solution.

Original: It allows us to tune a single parameter θ that controls the sparsity of the solution instead of α and β .

Original: The larger θ is, the greater the pairwise distances between nodes and therefore the sparser the edges, as connections between distant nodes are penalized.

Original: This result is also important for graph-based applications that are multiplicative scale invariant. In the latter, multiplying all edges by the same constant does not change the functionality of the graph.
Revised: For some applications the scale of the graph is less important, and multiplying all edges by the same constant does not change its functionality.

Original: In other cases, we want to explicitly normalize the graph to a speciﬁc size, for example setting (cid:107) W (cid:107) 1 , 1 = n as in (Hu et al., 2013), or making sure that λ max = 1 .
Revised: In other cases, we want to explicitly normalize the graph to a speciﬁc size, for example setting (cid:107) W (cid:107) 1 , 1 = n as in (Hu et al., 2013), or making sure that λ max = 1 .

Original: In these cases, the user needs only search for  the value of θ that will obtain the desired sparsity andthen multiply the graph with the appropriate constant.
Revised: Nevertheless, for applications where scale shall be set automatically, we provide the following Theorem that characterizes the connection between δ and the scale of the weights. Theorem 1. All edges of the solution W ∗ ( Z, α, β ) = δW ∗ ( θZ, 1 , 1) with θ = 1 √ αβ , δ = (cid:113) αβ of model (3) are upper bounded by δ : W ∗ ( Z, α, β ) ≤ (cid:113) αβ = δ.

Revised: Proof.

Revised: See supplementary material A.

Original: 4. S ETTING THE REMAINING REGULARIZATION PARAMETER
Revised: 4. S ETTING THE REMAINING REGULARIZATION PARAMETER

Original: The last step for automatizing parameter selection is to ﬁnd a relation between θ and the desired sparsity (the average number of neighbors per node).
Revised: The last step for automatizing parameter selection is to ﬁnd a relation between θ and the desired sparsity (the average number of neighbors per node).

Original: We ﬁrst analyze the sparsity level with respect to θ for each node independently.
Revised: We ﬁrst analyze the sparsity level with respect to θ for each node independently.

Original: Once the independent problems are well characterized, we propose an empirical solution to obtain a global value of θ providing approximately the desired sparsity level.
Revised: Once the independent problems are well characterized, we propose an empirical solution to obtain a global value of θ providing approximately the desired sparsity level.

Original: 4.2. S PARSITY ANALYSIS FOR ONE NODE
Revised: 4.2. S PARSITY ANALYSIS FOR ONE NODE

Original: In order to analyze the sparsity of the graphs obtained by the model (3), we take one step back and drop the symmetricity constraint.
Revised: In order to analyze the sparsity of the graphs obtained by the model (3), we take one step back and drop the symmetricity constraint.

Original: The problem becomes separable and we can focus on only one node.
Revised: The problem becomes separable and we can focus on only one node.

Original: Keeping only one column w of matrix W , we arrive to the simpler optimization problem
Revised: Keeping only one column w of matrix W , we arrive to the simpler optimization problem

Original: [Equation]
Revised: [Equation]

Original: The above problem also has only one parameter θ that controls sparsity, so that larger values of θ yield sparser solutions w ∗ .
Revised: The above problem also has only one parameter θ that controls sparsity, so that larger values of θ yield sparser solutions w ∗ .

Original: Furthermore, it enjoys an analytic solution if we sort the elements of z , as we prove with the next theorem.
Revised: Furthermore, it enjoys an analytic solution if we sort the elements of z , as we prove with the next theorem.

Original: Theorem 1.
Revised: Theorem 2.

Original: Suppose that the input vector z is sorted in ascending order.
Revised: Suppose that the input vector z is sorted in ascending order.

Original: Then the solution of problem (8) has the form
Revised: Then the solution of problem (8) has the form

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: The set I = { 1 , . . .
Revised: The set I = { 1 , . . .

Original: , k } corresponds to the indices of the k smallest distances
Revised: , k } corresponds to the indices of the k smallest distances

Original: z i and b k is the 
Revised: z i and b k is the cumulative sum of the smallest k distances in z , b k = (cid:80) ki

Original: [Table]{+=
 z i .+}
Revised: {+=
 z i .+}

Original: We provide the proof of Theorem 1 after presenting certain intermediate results.
Revised: We provide the proof of Theorem 1 after presenting certain intermediate results.

Original: In order to solve
Revised: In order to solve

Original: Problem (8) we ﬁrst introduce a slack variable l for the inequality constraint, so that the KKT optimality conditions are
Revised: Problem (8) we ﬁrst introduce a slack variable l for the inequality constraint, so that the KKT optimality conditions are

Original: [Equation]
Revised: [Equation]

Original: the optimum of w can be revealed by introducing the term λ ∗ = [Equation] w ∗(cid:62) 1 and rewrite (10) as
Revised: the optimum of w can be revealed by introducing the term λ ∗ = 1 w ∗(cid:62) 1 and rewrite (10) as

Original: [Equation]
Revised: [Equation]

Original: Then, we split the elements of w in two sets, A and I according to the activity of the inequality constraint (11), so that w I > 0 (inactive) and w A = 0 (active).
Revised: Then, we split the elements of w in two sets, A and I according to the activity of the inequality constraint (11), so that w I > 0 (inactive) and w A = 0 (active).

Original: Note that at the minimum, the elements of w will also be sorted in a descending order so that w ∗ =
Revised: Note that at the minimum, the elements of w will also be sorted in a descending order so that w ∗ =

Original: [ w ∗I ; 0 ] , according to Theorem 1.
Revised: [ w ∗I ; 0 ] , according to Theorem 2.

Original: We ﬁrst need a condition for an element of w ∗ to be positive, as expressed in the following lemma:
Revised: We ﬁrst need a condition for an element of w ∗ to be positive, as expressed in the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: An element w ∗ i of the solution w ∗ of problem (8) is in the active set
Revised: An element w ∗ i of the solution w ∗ of problem (8) is in the active set

Original: A if and only if it corresponds to an element of z i for which θz i ≥ λ ∗ .
Revised: A if and only if it corresponds to an element of z i for which θz i ≥ λ ∗ .

Original: Proof.
Revised: Proof.

Original: ( ⇒ )
Revised: ( ⇒ )

Original: : If w i is in the active set we have w i = 0 and l i ≥ 0 , therefore from (14) we have θz
Revised: : If w i is in the active set we have w i = 0 and l i ≥ 0 , therefore from (14) we have θz

Original: i − λ ∗ ≥ 0 .
Revised: i − λ ∗ ≥ 0 .

Original: ( ⇐ ) : Suppose that there exists
Revised: ( ⇐ ) : Suppose that there exists

Original: i ∈
Revised: i ∈

Original: I for which θz i ≥ λ ∗ .
Revised: I for which θz i ≥ λ ∗ .

Original: The constraint being inactive means that w ∗ i > 0 .
Revised: The constraint being inactive means that w ∗ i > 0 .

Original: From (13) we have that l i = 

Revised: From (13) we have that l i =

Original: and (14) gives w ∗ i =
Revised: and (14) gives w ∗ i =

Original: λ ∗ − θz i ≤ 0 , a contradiction.
Revised: λ ∗ − θz i ≤ 0 , a contradiction.

Original: We are now ready to proceed to the proof of Theorem 1.
Revised: We are now ready to proceed to the proof of Theorem 2.

Original: Proof (Theorem 1) .
Revised: Proof (Theorem 2) .

Original: As elements of θz are sorted in an ascending order, the elements of λ ∗ − θz will be in a descending order.
Revised: As elements of θz are sorted in an ascending order, the elements of λ ∗ − θz will be in a descending order.

Original: Furthermore, we know from Lemma 1 that all positive w ∗ i will correspond to θz i < λ ∗ .
Revised: Furthermore, we know from Lemma 1 that all positive w ∗ i will correspond to θz i < λ ∗ .

Original: Then, supposing that |I| =
Revised: Then, supposing that |I| =

Original: k we have the following ordering:
Revised: k we have the following ordering:

Original: [Equation]
Revised: [Equation]

Original: In words, the vector λ ∗ − θz will have sorted elements so that the ﬁrst k are positive and the rest are non-positive.
Revised: In words, the vector λ ∗ − θz will have sorted elements so that the ﬁrst k are positive and the rest are non-positive.

Original: Furthermore, we know that the elements of l in the optimal have to be 0 for all inactive variables w ∗I , therefore w ∗I = λ ∗ − θz I .
Revised: Furthermore, we know that the elements of l in the optimal have to be 0 for all inactive variables w ∗I , therefore w ∗I = λ ∗ − θz I .

Original: The remaining elements of w will be 0 by deﬁnition of the active set:
Revised: The remaining elements of w will be 0 by deﬁnition of the active set:

Original: [Equation]
Revised: [Equation]

Original: What remains is to ﬁnd an expression to compute λ ∗ for any given z .
Revised: What remains is to ﬁnd an expression to compute λ ∗ for any given z .

Original: Keeping z ordered in ascending order, let the cumulative sum of z i be b k = (cid:80) ki
Revised: Keeping z ordered in ascending order, let the cumulative sum of z i be b k = (cid:80) k

Original: =1 z i .
Revised: i = z i .

Original: Then, from the deﬁnition of λ ∗ = 1 w ∗(cid:62) 1 and using the structure of w ∗ we have
Revised: Then, from the deﬁnition of λ ∗ = 1 w ∗(cid:62) 1 and using the structure of w ∗ we have

Original: [Equation]
Revised: [Equation]

Original: which has only one positive solution,
Revised: which has only one positive solution,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 4.2. P ARAMETER SELECTION FOR THE NON - SYMMETRIC CASE
Revised: P ARAMETER SELECTION FOR THE NON - SYMMETRIC CASE

Original: While Theorem 1 gives the form of the solution for a known k , the latter cannot be known a priori, as it is also a function of z .
Revised: While Theorem 2 gives the form of the solution for a known k , the latter cannot be known a priori, as it is also a function of z .

Original: For this, we propose Algorithm 1 that solves this problem, simultaneously ﬁnding k and λ ∗ in O ( k ) iterations.
Revised: For this, we propose Algorithm 1 that solves this problem, simultaneously ﬁnding k and λ ∗ in O ( k ) iterations.

Original: This algorithm will be needed for automatically setting the parameters for the symmetric case of graph learning.
Revised: This algorithm will be needed for automatically setting the parameters for the symmetric case of graph learning.

Original: As k is the number of non-zero edges per node, we can assume it to be small, like for k -NN.
Revised: As k is the number of non-zero edges per node, we can assume it to be small, like for k -NN.

Original: It is thus cheap to incrementally try all values of k from k = 1 until we ﬁnd the correct one, as
Revised: It is thus cheap to incrementally try all values of k from k = 1 until we ﬁnd the correct one, as

Original: Algorithm 1 does.
Revised: Algorithm 1 does.

Original: Once we try a value that satisﬁes λ ∈ ( θz k , θz k +1 ] , all KKT conditions hold and we have found the solution to our problem.
Revised: Once we try a value that satisﬁes λ ∈ ( θz k , θz k +1 ] , all KKT conditions hold and we have found the solution to our problem.

Original: A similar algorithm was proposed in (Duchi et al., 2008) for projecting vectors on the probability simplex, that could be used for a similar analysis for the (cid:96) 2 -degree constraints model (4).
Revised: A similar algorithm was proposed in (Duchi et al., 2008) for projecting vectors on the probability simplex, that could be used for a similar analysis for the (cid:96) 2 -degree constraints model (4).

Original: Most interestingly, using the form of the solution given by Theorem 1 we can solve the reverse problem: If we know the distances vector z and we want a solution w ∗ with exactly k non-zero elements, what should the parameter θ be?
Revised: Most interestingly, using the form of the solution given by Theorem 2 we can solve the reverse problem: If we know the distances vector z and we want a solution w ∗ with exactly k non-zero elements, what should the parameter θ be?

Original: The following theorem answers this question, giving intervals for θ as a function of k , z and its cumulative sum b .
Revised: The following theorem answers this question, giving intervals for θ as a function of k , z and its cumulative sum b .

Original: Theorem 2.
Revised: Theorem 3.

Original: Let θ ∈ 
Revised: Let θ ∈ (cid:18)

Original: [Equation]
Revised: [Equation]

Original: , the solution of eq.
Revised: , the solution of eq.

Original: (8) has exactly k non-zeros.
Revised: (8) has exactly k non-zeros.

Original: Proof.
Revised: Proof.

Original: Seethe supplementary material A.
Revised: See supplementary material B.

Original: The idea of Theorem 2 is illustrated in the left part of Figure 2.
Revised: The idea of Theorem 3 is illustrated in the left part of Figure 2.

Original: For this ﬁgure we have used the distances between one image of MNIST and 999 other images.
Revised: For this ﬁgure we have used the distances between one image of MNIST and 999 other images.

Original: For any given sparsity level k we know what are the intervals of the valid values of θ by looking at the pairwise distances.
Revised: For any given sparsity level k we know what are the intervals of the valid values of θ by looking at the pairwise distances.

Original: Algorithm 1 Solver of the one-node problem, (8).

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: : end if

Original: [Table]
Revised: [Table]

Original: : end for

Original: [Figure]
Revised: [Figure]

Original: 4.2. P ARAMETER SELECTION FOR THE SYMMETRIC CASE
Revised: 4.2. P ARAMETER SELECTION FOR THE SYMMETRIC CASE

Original: In order to approximate the parameter θ that gives the desired sparsity of W , we use the above analysis for each row or column separately, omitting the symmetricity constraint.
Revised: In order to approximate the parameter θ that gives the desired sparsity of W , we use the above analysis for each row or column separately, omitting the symmetricity constraint.

Original: Then, using the arithmetic mean of the bounds of θ we obtain a good approximation of the behaviour of the full symmetric problem.
Revised: Then, using the arithmetic mean of the bounds of θ we obtain a good approximation of the behaviour of the full symmetric problem.

Original: In other words, to obtain a graph with approximately k edges per node, we propose to use the following intervals:
Revised: In other words, to obtain a graph with approximately k edges per node, we propose to use the following intervals:

Original: [Equation]
Revised: [Equation]

Original: where ˆ Z is obtained by sorting each column of Z in increasing order, and B k,j = (cid:80) ki
Revised: where ˆ Z is obtained by sorting each column of Z in increasing order [Equation] and B k,j = (cid:80) k i

Original: =1 ˆ Z i,j .
Revised: = 1 ˆ Z i,j .

Original: The above expression is the arithmetic mean over all minimum and maximum values of θ k,j that would give a k -sparse result W : ,j if we were to solve problem (8) for each of columns separately, according to Theorem 2.
Revised: The above expression is the arithmetic mean over all minimum and maximum values of θ k,j that would give a k -sparse result W : ,j if we were to solve problem (8) for each of columns separately, according to Theorem 3.

Original: Even though the above approach does not take into account the symmetricity constraints, it gives surprisingly good results in the vast majority of the cases.
Revised: Even though the above approach does not take into account the symmetricity constraints, it gives surprisingly good results in the vast majority of the cases.

Original: For the ﬁnal value of θ for a given k , we use the middle of this interval in a logarithmic scale, that is, the geometric mean of the upper and lower limits. 
Revised: For the ﬁnal value of θ for a given k , we use the middle of this interval in a logarithmic scale, that is, the geometric mean of the upper and lower limits. [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we wish to answer questions regarding (1) the approximation quality of our model, (2) the quality of our automatic parameter selection, (3) the beneﬁt from learning versus A-NN for large scale applications and (4) the scalability of the model.
Revised: In our experiments we wish to answer questions regarding (1) the approximation quality of our model, (2) the quality of our automatic parameter selection, (3) the beneﬁt from learning versus A-NN for large scale applications and (4) the scalability of the model.

Original: As this contribution is the ﬁrst to scale a graph learning algorithm to a large amount of nodes, it is difﬁcult to compare with other methods without trying to scale them ﬁrst.
Revised: As this contribution is the ﬁrst to scale a graph learning algorithm to a large amount of nodes [Figure] it is difﬁcult to compare with other methods without trying to scale them ﬁrst.

Original: While this paper focuses on the scaled log model, we also scaled the (cid:96) 2 -model and the "hard" and "soft" models proposed by (Daitch et al., 2009).
Revised: While this paper focuses on the scaled log model [Figure] we also scaled the (cid:96) 2 -model and the "hard" and "soft" models proposed by (Daitch et al., 2009).

Original: We refer the reader to the supplementary material B for further details, and C for details about the datasets we use.
Revised: We refer the reader to the supplementary material C for further details [Figure] and D for details about the datasets we use [Figure]

Original: 5. A PPROXIMATION QUALITY OF LARGE SCALE MODEL
Revised: 5. A PPROXIMATION QUALITY OF LARGE SCALE MODEL

Original: [Figure]
Revised: [Figure]

Original:  When computing E allowed , we use an A-NN graph from the publicly available FLANN library 1 that implements the work of Muja & Lowe (2014).
Revised: [Figure] When computing E allowed , we use an A-NN graph from the publicly available FLANN library 1 that implements the work of Muja & Lowe (2014).

Original: To learn a graph with on average k neighbors per node ( k -NN), we ﬁrst compute an rk -A-NN graph and use its edges as E allowed .
Revised: To learn a graph with on average k neighbors per node ( k -NN), we ﬁrst compute an rk -A-NN graph and use its edges as E allowed .

Original: The graph is then learned on this subset.
Revised: The graph is then learned on this subset.

Original: The size of E allowed does not only affect the time needed to learn the graph, but also its quality.
Revised: The size of E allowed does not only affect the time needed to learn the graph, but also its quality.

Original: A too restrictive choice might prevent the ﬁnal graph from learning useful edges.
Revised: A too restrictive choice might prevent the ﬁnal graph from learning useful edges.

Original: In Figure 3, we study the effect of this restriction on the ﬁnal result.
Revised: In Figure 3, we study the effect of this restriction on the ﬁnal result.

Original: The vertical axis is the relative (cid:96) 1 error between our approximate log model and the actual log model by Kalofolias (2016) when learning a graph between 1000 images of
Revised: The vertical axis is the relative (cid:96) 1 error between our approximate log model and the actual log model by Kalofolias (2016) when learning a graph between 1000 images of

Original: MNIST, averaged over 10 runs.
Revised: MNIST, averaged over 10 runs.

Original: Note that the result will depend on the A-NN algorithm used, while a comparison between different types of A-NN is beyond the scope of this paper.
Revised: Note that the result will depend on the A-NN algorithm used, while a comparison between different types of A-NN is beyond the scope of this paper.

Original: 5. E FFECTIVENESS OF AUTOMATIC PARAMETER SELECTION
Revised: 5. E FFECTIVENESS OF AUTOMATIC PARAMETER SELECTION

Original: eq.
Revised: eq.

Original: Requested versus obtained degree, "spherical" data ( 262 , 000 nodes).
Revised: Requested versus obtained degree, "spherical" data ( 262 , 000 nodes).

Original: As we saw in the middle and right plots of Figure 2, the approximate bounds of θ (red line) given by eq.
Revised: As we saw in the middle and right plots of Figure 2, the approximate bounds of θ (red line) given by eq.

Original: (17) are very effective at predicting sparsity.
Revised: (17) are very effective at predicting sparsity.

Original: The same experiment is repeated for more datasets is in the supplementary material.
Revised: The same experiment is repeated for more datasets is in the supplementary material.

Original: In Figure 4 we see this also in large scale , between 262K nodes from our "spherical" dataset.
Revised: In Figure 4 we see this also in large scale , between 262K nodes from our "spherical" dataset.

Original: Please note, that in the rare cases that the actual sparsity is outside the predicted bounds, we already have a good starting point for ﬁnding a good θ .
Revised: Please note, that in the rare cases that the actual sparsity is outside the predicted bounds, we already have a good starting point for ﬁnding a good θ .

Original: Note also that small ﬂuctuations in the density are tolerated, for example in k -NN or A-NN graphs we always obtain results with slightly more than nk edges due to the fact that W is symmetric.
Revised: Note also that small ﬂuctuations in the density are tolerated, for example in k -NN or A-NN graphs we always obtain results with slightly more than nk edges due to the fact that W is symmetric.

Original: Finally, as we see in Figure 11, the bounds are very robust to outliers, as well as duplicates in the data.
Revised: Finally, as we see in Figure 11, the bounds are very robust to outliers, as well as duplicates in the data.

Original: The curious reader can ﬁnd details in Section C.3.1 of the supplementary material.
Revised: The curious reader can ﬁnd details in Section D.3.1 of the supplementary material.

Original: 5. E DGE QUALITY
Revised: 5. E DGE QUALITY

Original: We ﬁrst compare scalable models of graphs between 60 , 000 images of MNIST.
Revised: We ﬁrst compare scalable models of graphs between 60 , 000 images of MNIST.

Original: MNIST has a relatively uniform sampling between numbers of different labels, except for the digits “1” that are more densely sampled .
Revised: MNIST has a relatively uniform sampling between numbers of different labels, except for the digits “1” that are more densely sampled .

Original: That is, the average intra-class distance between digits “1” is smaller than for other digits (supplementary material C.2).
Revised: That is, the average intra-class distance between digits “1” is smaller than for other digits (supplementary material D.2).

Original: As we see, this affects the results of graph learning.
Revised: As we see, this affects the results of graph learning.

Original: In Figure 5, we plot the histograms of connectivity between images of each label.
Revised: In Figure 5, we plot the histograms of connectivity between images of each label.

Original: We normalize all graphs to (cid:107) W (cid:107) 1 , 1 = 1 .
Revised: We normalize all graphs to (cid:107) W (cid:107) 1 , 1 = 1 .

Original: The last bar is the connectivity wasted on wrong edges (pairs of different[Figure] labels).
Revised: The last bar is the connectivity wasted on wrong edges (pairs of different labels).

Original: The A-NN graph does not take into account the different sampling rates of different digits, while it has many wrong edges.
Revised: The A-NN graph does not take into account the different sampling rates of different digits, while it has many wrong edges.

Original: The (cid:96) 2 model (4) assigns most of its connectivity to the label “1” and neglects digits with larger distance (“2”, “8”) even for 30 edges per node.
Revised: The (cid:96) 2 model (4) assigns most of its connectivity to the label “1” and neglects digits with larger distance (“2”, “8”) even for 30 edges per node.

Original: The log model does not suffer from this problem even for sparse graphs of degree 5 and gives consistent connectivities for all digits.
Revised: The log model does not suffer from this problem even for sparse graphs of degree 5 and gives consistent connectivities for all digits.

Original: The scaled versions of models in (Daitch et al., 2009) perform worse than our large scale log model, but often better than A-NN or (cid:96) -2, as plotted in Figure 13 of the supplementary material.
Revised: The scaled versions of models in (Daitch et al., 2009) perform worse than our large scale log model, but often better than A-NN or (cid:96) -2, as plotted in Figure 13 of the supplementary material.

Original: Figure 6 (left) summarizes the number of wrong edges with respect to k .
Revised: Figure 6 (left) summarizes the number of wrong edges with respect to k .

Original: Models (cid:96) 2 and log that minimize tr( X (cid:62) LX ) = (cid:107) Z ◦ W (cid:107) 1 , 1 have a large advantage compared to models by Daitch et al.
Revised: Models (cid:96) 2 and log that minimize tr( X (cid:62) LX ) = (cid:107) Z ◦ W (cid:107) 1 , 1 have a large advantage compared to models by Daitch et al.

Original: (2009) that minimize (cid:107) LX (cid:107) 2 F a quadratic function of W .
Revised: (2009) that minimize (cid:107) LX (cid:107) 2 F a quadratic function of W .

Original: The ﬁrst induces sparsity of W , while the latter favors small edges.
Revised: The ﬁrst induces sparsity of W , while the latter favors small edges.

Original: This explains the existence of many wrong edges for models "hard" and "soft", while creates problems in controlling sparsity.
Revised: This explains the existence of many wrong edges for models "hard" and "soft", while creates problems in controlling sparsity.

Original: Note that A-NN is more accurate than "hard" and "soft" graphs.
Revised: Note that A-NN is more accurate than "hard" and "soft" graphs.

Original: It seems that (cid:96) 2 is more accurate than the log model, but as shown in Figure 5, the majority of its edges connect digit "1" without connecting sufﬁciently digits such as 2 , 3 , 8 .
Revised: It seems that (cid:96) 2 is more accurate than the log model, but as shown in Figure 5, the majority of its edges connect digit "1" without connecting sufﬁciently digits such as 2 , 3 , 8 .

Original: 5. S EMI - SUPERVISED LEARNING
Revised: 5. S EMI - SUPERVISED LEARNING

Original: On the same graphs, we perform label propagation (Zhu et al., 2003) on MNIST, with 1% known labels.
Revised: On the same graphs, we perform label propagation (Zhu et al., 2003) on MNIST, with 1% known labels.

Original: The results are plotted in Figure 6 (right).
Revised: The results are plotted in Figure 6 (right).

Original: Note that in label propagation, knowledge is not shared between disconnected components, making it impossible to classify nodes in components without known labels.
Revised: Note that in label propagation, knowledge is not shared between disconnected components, making it impossible to classify nodes in components without known labels.

Original: The dashed lines of Figure 6 represent the number of nodes in such components, that occur much less in the log graph.
Revised: The dashed lines of Figure 6 represent the number of nodes in such components, that occur much less in the log graph.

Original: Again, the log model performs best.
Revised: Again, the log model performs best.

Original: To have a fair comparison with A-NN, we weighted its edges with standard exponential decay W i,j = exp( − Z i,j /σ 2 ) and chose every time the best performing σ .
Revised: To have a fair comparison with A-NN, we weighted its edges with standard exponential decay W i,j = exp( − Z i,j /σ 2 ) and chose every time the best performing σ .

Original: Note that it is very difﬁcult for the "soft" and "hard" models to control sparsity (hard graphs have no parameter at all).
Revised: Note that it is very difﬁcult for the "soft" and "hard" models to control sparsity (hard graphs have no parameter at all).

Original: Hence they are less adaptable to a particular problem and in this particular case perform poorly.
Revised: Hence they are less adaptable to a particular problem and in this particular case perform poorly.

Original: 5. M ANIFOLD RECOVERY QUALITY
Revised: 5. M ANIFOLD RECOVERY QUALITY

Original: Graphs can be used to recover a low dimensional manifold from data.
Revised: Graphs can be used to recover a low dimensional manifold from data.

Original: We evaluate the performance of our large scale model for this application on three datasets: " spherical data " ( n = 262 , 144 ) and " spherical data small " ( n = 4 , 096 ) from a known 2-D manifold, and " word2vec " ( n = 10 , 000 ).
Revised: We evaluate the performance of our large scale model for this application on three datasets: " spherical data " ( n = 262 , 144 ) and " spherical data small " ( n = 4 , 096 ) from a known 2-D manifold, and " word2vec " ( n = 10 , 000 ).

Original: 5.5. L ARGE SPHERICAL DATA
Revised: 5.5. L ARGE SPHERICAL DATA

Original: Using 1920 signals from a known spherical manifold (Perraudin et al., 2018)
Revised: Using 1920 signals from a known spherical manifold (Perraudin et al., 2018)

Original: we learn graphs of 262 , 144 nodes and recover a 2D manifold using the ﬁrst 2 non-trivial eigenvectors of their Laplacians.
Revised: we learn graphs of 262 , 144 nodes and recover a 2D manifold using the ﬁrst 2 non-trivial eigenvectors of their Laplacians.

Original: The data is sampled on a 512 × 512 grid on the sphere, so we use graphs of degree close to k = 4 .
Revised: The data is sampled on a 512 × 512 grid on the sphere, so we use graphs of degree close to k = 4 .

Original: We try to recover it using scalable models: (cid:96) 2 ( k = 4 . 70 ), log ( k = 4 . 73 ) and A-NN ( k = 4 . 31 ).
Revised: We try to recover it using scalable models: (cid:96) 2 ( k = 4 . 70 ), log ( k = 4 . 73 ) and A-NN ( k = 4 . 31 ).

Original: While the A-NN graph performs very poorly, the representations recovered by both (cid:96) 2 and log graphs are almost perfectly square (Figure 15).
Revised: While the A-NN graph performs very poorly, the representations recovered by both (cid:96) 2 and log graphs are almost perfectly square (Figure 15).

Original: However, as shown in Figure 7, the log graph gives the best representation.
Revised: However, as shown in Figure 7, the log graph gives the best representation.

Original: We plot subgraphs of the two models with only the nodes that correspond to 2D grids in the middle or the corner of the the original manifold .
Revised: We plot subgraphs of the two models with only the nodes that correspond to 2D grids in the middle or the corner of the the original manifold .

Original: While the (cid:96) 2 graph does not connect all central nodes (green), the log graph is closer to a 2D grid.
Revised: While the (cid:96) 2 graph does not connect all central nodes (green), the log graph is closer to a 2D grid.

Original: Furthermore, while the (cid:96) 2 model haddisconnected nodes that we discarded before computing eigenvectors, the log graph was connected.
Revised: Furthermore, while the (cid:96) 2 model haddisconnected nodes that we discarded before computing eigenvectors, the log graph was connected.

Original: 5.5. S MALL SPHERICAL DATA
Revised: 5.5. S MALL SPHERICAL DATA

Original: We then learn graphs from the “small spherical data” so as to be able to compute graph diameters.
Revised: We then learn graphs from the “small spherical data” so as to be able to compute graph diameters.

Original: This operation has a complexity above O ( n 2 ) and is not possible for large graphs.
Revised: This operation has a complexity above O ( n 2 ) and is not possible for large graphs.

Original: Manifold-like graphs are known to have larger diameter compared to small world graphs (Watts & Strogatz, 1998) for the same average degree.
Revised: Manifold-like graphs are known to have larger diameter compared to small world graphs (Watts & Strogatz, 1998) for the same average degree.

Original: In Figure 8 (left and middle) we plot the diameter of scalable graph models.
Revised: In Figure 8 (left and middle) we plot the diameter of scalable graph models.

Original: The data has 4096 nodes organized as a 64 × 64 grid, therefore a ground-truth 4 -NN has exactly diameter 127 = 2 · 64 − 1 , and a ground-truth 8 -NN diameter 63 .
Revised: The data has 4096 nodes organized as a 64 × 64 grid, therefore a ground-truth 4 -NN has exactly diameter 127 = 2 · 64 − 1 , and a ground-truth 8 -NN diameter 63 .

Original: The learned graphs, given enough information (1920 signals) reach the ideal diameter around degrees 4 or 8, with a phase transition from diameter 127 to diameter 63 between degrees 6 to 7 .
Revised: The learned graphs, given enough information (1920 signals) reach the ideal diameter around degrees 4 or 8, with a phase transition from diameter 127 to diameter 63 between degrees 6 to 7 .

Original: When the information is not enough to recover the exact manifold (using only 40 signals, middle plot of Figure 8), the learned graphs have a higher diameter than the A-NN graph. 
Revised: When the information is not enough to recover the exact manifold (using only 40 signals, middle plot of Figure 8), the learned graphs have a higher diameter than the A-NN graph. [Figure]

Original: . G RAPH BETWEEN WORDS USING WORD 2 VEC REPRESENTATIONS
Revised: G RAPH BETWEEN WORDS USING WORD 2 VEC REPRESENTATIONS

Original: Finally we learn a graph between 10 , 000 words using 300 word2vec features, like the ones used for graph convolutional neural networks (Defferrard et al., 2016).
Revised: Finally we learn a graph between 10 , 000 words using 300 word2vec features, like the ones used for graph convolutional neural networks (Defferrard et al., 2016).

Original: In this application, connectivity of all nodes is important, and (cid:96) 2 graphs failed, always giving many disconnected nodes.
Revised: In this application, connectivity of all nodes is important, and (cid:96) 2 graphs failed, always giving many disconnected nodes.

Original: In Figure 8 we compare the diameter of our large scale log graphs against A-NN and the k -NN graphs used by (Defferrard et al., 2016).
Revised: In Figure 8 we compare the diameter of our large scale log graphs against A-NN and the k -NN graphs used by (Defferrard et al., 2016).

Original: The large scale log graph has signiﬁcantly larger diameter than both k -NN and A-NN graphs.
Revised: The large scale log graph has signiﬁcantly larger diameter than both k -NN and A-NN graphs.

Original: This indicates that our learned graph is closer to a manifold-like structure, unlike the other two types that are closer to a small-world graph.
Revised: This indicates that our learned graph is closer to a manifold-like structure, unlike the other two types that are closer to a small-world graph.

Original: Manifold-like graphs reveal better the structure of data, while small world graphs are related to randomness (Watts & Strogatz, 1998).
Revised: Manifold-like graphs reveal better the structure of data, while small world graphs are related to randomness (Watts & Strogatz, 1998).

Original: In Figure 9, we plot nodes within two hops from the word "use" in the three types of graphs of degree around k = 5 .
Revised: In Figure 9, we plot nodes within two hops from the word "use" in the three types of graphs of degree around k = 5 .

Original: We see that the NN graphs span a larger part of the entire graph just with 2 hops, the A-NN being closer to small-world.
Revised: We see that the NN graphs span a larger part of the entire graph just with 2 hops, the A-NN being closer to small-world.

Original: While the k -NN graph does better in terms of quality, it is signiﬁcantly worse than our large scale log graph, that is actually cheaper to compute.
Revised: While the k -NN graph does better in terms of quality, it is signiﬁcantly worse than our large scale log graph, that is actually cheaper to compute.

Original: Additionally, graph learning seems to assign more meaningful weights as we show in Table 1 of the supplementary material C.4.
Revised: Additionally, graph learning seems to assign more meaningful weights as we show in Table 1 of the supplementary material D.4.

Original:  C OMPUTATION TIME
Revised: 5. C OMPUTATION TIME

Original: In Figure 1 we compare our scaled model to other graph learning models in terms of time as a function of n .
Revised: In Figure 1 we compare our scaled model to other graph learning models in terms of time as a function of n .

Original: For all iterative algorithms, except A-NN, we perform a maximum of 500 iterations.
Revised: For all iterative algorithms, except A-NN, we perform a maximum of 500 iterations.

Original: The scaled (cid:96) 2 needs time per iteration identical to the scaled log model, but typically needs more iterations (more than 5 , 

Revised: The scaled (cid:96) 2 needs time per iteration identical to the scaled log model, but typically needs more iterations (more than 5 ,

Original: compared to 500 ) till convergence.
Revised: compared to 500 ) till convergence.

Original: As we ﬁx the number of iterations, they overlap and we only plot the log model.
Revised: As we ﬁx the number of iterations, they overlap and we only plot the log model.

Original: In Figure 1 (left) we learn graphs between words, using 300 features ( word2vec ).
Revised: In Figure 1 (left) we learn graphs between words, using 300 features ( word2vec ).

Original: The cost is almost linear for our method, but quadratic for the original log model of Kalofolias (2016).
Revised: The cost is almost linear for our method, but quadratic for the original log model of Kalofolias (2016).

Original: The "hard" model corresponds to a close implementation of the original work of (Daitch et al., 2009).
Revised: The "hard" model corresponds to a close implementation of the original work of (Daitch et al., 2009).

Original: The most expensive part of the computation comes from the quadratic program solver.
Revised: The most expensive part of the computation comes from the quadratic program solver.

Original: For the "soft" model, we used the forward backward scheme of (Beck & Teboulle, 2009) and the cost is governed by evaluating the KKT conditions to ﬁnd the optimal support ( O ( n 2 ) ), that is faster than their original algorithm.
Revised: For the "soft" model, we used the forward backward scheme of (Beck & Teboulle, 2009) and the cost is governed by evaluating the KKT conditions to ﬁnd the optimal support ( O ( n 2 ) ), that is faster than their original algorithm.

Original: We also compare to our scaled versions of the "hard" and "soft" models, where we ﬁx the support in the same way as explained in Section 3 (cf. supplementary material B).
Revised: We also compare to our scaled versions of the "hard" and "soft" models, where we ﬁx the support in the same way as explained in Section 3 (cf. supplementary material C).

Original: The bottleneck of the computation comes from the term (cid:107) LX (cid:107) 2 F that requires O ( knd ) operations as opposed to our O ( kn ) for tr (cid:0) X (cid:62) LX (cid:1) .
Revised: The bottleneck of the computation comes from the term (cid:107) LX (cid:107) 2 F that requires O ( knd ) operations as opposed to our O ( kn ) for tr (cid:0) X (cid:62) LX (cid:1) .

Original: To reduce this cost, we randomly projected the original d = 300 input features to a subspace of size d = 20 .
Revised: To reduce this cost, we randomly projected the original d = 300 input features to a subspace of size d = 20 .

Original: While the cost per iteration of the "soft" method is slower than for the "hard" method, the latter typically needs more iterations to reach convergence.
Revised: While the cost per iteration of the "soft" method is slower than for the "hard" method, the latter typically needs more iterations to reach convergence.

Original: In Figure1 (right), we show the scalability of our model to graphs of up to 1 M samples 2 of US census .
Revised: In Figure1 (right), we show the scalability of our model to graphs of up to 1 M samples 2 of US census .

Original: Setting k = 5 , it used 16 minutes to perform 500 iterations of graph learning on a desktop computer running Matlab.
Revised: Setting k = 5 , it used 16 minutes to perform 500 iterations of graph learning on a desktop computer running Matlab.

Original: While this is a proof of scalability, for really large data one will have to consider implementation details like memory management, and using a faster programming language (like
Revised: While this is a proof of scalability, for really large data one will have to consider implementation details like memory management, and using a faster programming language (like

Original: C++).
Revised: C++).

Original: Note that the A-NN implementation we used was compiled C code and thus much faster.
Revised: Note that the A-NN implementation we used was compiled C code and thus much faster.

Original: In
Revised: In

Original: Figure 14 we illustrate the linear scalability of our model w.r.t.
Revised: Figure 14 we illustrate the linear scalability of our model w.r.t.

Original: the size of the set of allowed edges.
Revised: the size of the set of allowed edges.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose the ﬁrst scalable solution to learn a weighted undirected graph from data, based on A-NN and the current state-of-the-art graph learning model.
Revised: We propose the ﬁrst scalable solution to learn a weighted undirected graph from data, based on A-NN and the current state-of-the-art graph learning model.

Original: While it costs roughly as much as A-NN, it achieves quality very close to state-of-the-art.
Revised: While it costs roughly as much as A-NN, it achieves quality very close to state-of-the-art.

Original: Its ability to scale is based on reducing the variables used for learning, while out automatica parameter selection eliminates the need for grid-search in order to achieve a sought graph sparsity.
Revised: Its ability to scale is based on reducing the variables used for learning, while out automatica parameter selection eliminates the need for grid-search in order to achieve a sought graph sparsity.

Original: We assess its quality and scalability by providing an extensive set of experiments on many real datasets.
Revised: We assess its quality and scalability by providing an extensive set of experiments on many real datasets.

Original: The new large scale graphs perform best in various machine learning cases.
Revised: The new large scale graphs perform best in various machine learning cases.

Original: They give better manifolds, are better for semi-supervised learning, and select the right amount of edges without allowing disconnected nodes.
Revised: They give better manifolds, are better for semi-supervised learning, and select the right amount of edges without allowing disconnected nodes.

Original: Learning a graph ofmillion nodes only takes 16 minutes using our simple Matlab implementation on a desktop computer.
Revised: Learning a graph ofmillion nodes only takes 16 minutes using our simple Matlab implementation on a desktop computer.

Original: L
Revised: T HANKS

Original: ARGE S CALE G RAPH L EARNING
Revised: The authors would like to especially thank Pierre Vandergheynst for his helpful comments during the preparation of this work at LTS2. We thank also the ETHZ Cosmology Research Group for allowing us to use their "Spherical convergence maps dataset" in the manifold recovery experiment. learn

Original: FROM S MOOTH S IGNALS: S UPPLEMENTARY M ATERIAL A P ROOF OF T HEOREM
Revised: FROM S MOOTH S IGNALS

Revised: S UPPLEMENTARY M ATERIAL A P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Revised: For any edge i, j the optimality conditions are:

Revised: [Equation]

Revised: Let us select any i, j such that W i,j > 0 .

Revised: Deﬁne η

Revised: i,j so that

Revised: [Equation]

Revised: ) and note that η i,j ≥ 0 , because all elements of W ∗ are non-negative and 1 W ∗ i,j ≥

Revised: [Equation]

Revised: Then we can rewrite the optimality conditions so that they can be solved analytically as a function of the unknown η i,j :

Revised: [Equation]

Revised: since √ 4 + x 2 − x ≤ 2 , ∀ x ≥ 0 .

Revised: This inequality holds for any edge i, j of the learned graph such that W ∗ i,j > 0 , therefore we can write

Revised: [Equation]

Revised: [Equation]

Revised: Hence using proposition 1, we conclude the proof.

Revised: Note that for sparsely connected regions, η i,j becomes smaller, and therefore the edges of the graph come closer to the upper limit.

Revised: We can deduct from equation 19 that the limit is actually reached only in case of duplicate nodes, i.e when Z i,j = 0 , and if the rest of the nodes are sufﬁciently far so that the edge i, j is the only edge for both nodes (so that η i,j = 0 ).

Revised: B P ROOF OF T HEOREM

Original:  From the proof of Theorem 1, we know that (cid:107) w ∗ (cid:107) 

Revised: Proof. From the proof of Theorem 2, we know that (cid:107) w ∗ (cid:107)

Original: =
Revised: =

Original: k if and only if λ ∗ ∈ [ θz k , θz k +1 ) .
Revised: k if and only if λ ∗ ∈ [ θz k , θz k +1 ) .

Original: We can rewrite this condition as
Revised: We can rewrite this condition as

Original: [Equation]
Revised: [Equation]

Original: As θ is constrained to be positive, the only values that satisfy the above inequalities are the ones proposed in the theorem.
Revised: As θ is constrained to be positive, the only values that satisfy the above inequalities are the ones proposed in the theorem.

Original: B S CALING ALGORITHM OF (D AITCH ET AL ., 2009)
Revised: C S CALING ALGORITHM OF (D AITCH ET AL ., 2009)

Original: (Daitch et al., 2009) proposes two convex optimization problems in order to learn graph from smooth signals.
Revised: (Daitch et al., 2009) proposes two convex optimization problems in order to learn graph from smooth signals.

Original: The prior used is (cid:107) LX (cid:107) 2 F , which is a quadratic function of the weight W .
Revised: The prior used is (cid:107) LX (cid:107) 2 F , which is a quadratic function of the weight W .

Original: For the ﬁrst problem, corresponding to the "hard" model, each node is constraint to have a degree of at least 1 :minimize
Revised: For the ﬁrst problem, corresponding to the "hard" model, each node is constraint to have a degree of at least 1 :

Original: w (cid:107) Mw (cid:107)s.t. w ≥ 0 , Sw ≥ 1 , (18) where M is a linear operator such that (cid:107) LX (cid:107) 2 F = (cid:107) Mw (cid:107) 22 .
Revised: [Equation] [Equation] where M is a linear operator such that (cid:107) LX (cid:107) 2 F = (cid:107) Mw (cid:107) 22 .

Original: We refer to the original paper for the construction of M .
Revised: We refer to the original paper for the construction of M .

Original: The second problem, corresponding to the "soft" problem, is a relaxed version of the "hard" one where each every degree below 1 is penalized quadratically.
Revised: The second problem, corresponding to the "soft" problem, is a relaxed version of the "hard" one where each every degree below 1 is penalized quadratically.

Original: It reads:minimize w [Equation] [Table] s
Revised: It reads:

Original: [Equation]
Revised: [Equation]

Original: t [Equation] (19) where µ is a parameter weighting the importance of the constraint.
Revised: where µ is a parameter weighting the importance of the constraint.

Original: In comparizon to the log and the (cid:96) 2 models where changing the regularization parameters has an important effect on the ﬁnal average node degree, we note that µ has little effect on the ﬁnal average node degree.
Revised: In comparizon to the log and the (cid:96) 2 models where changing the regularization parameters has an important effect on the ﬁnal average node degree, we note that µ has little effect on the ﬁnal average node degree.

Original: In their original paper, Daitch etal solve their optimization problems using SDPT3 (Tütüncü et al., 2003), which has a complexity signiﬁcantly higher than O (cid:0) v 2 (cid:1) for v variables.
Revised: In their original paper, Daitch etal solve their optimization problems using SDPT3 (Tütüncü et al., 2003), which has a complexity signiﬁcantly higher than O (cid:0) v 2 (cid:1) for v variables.

Original: So even when the number of edges scales with the number of nodes, i.e. v = O ( n ) , the complexity is not better than O (cid:0) n 2 (cid:1) .
Revised: So even when the number of edges scales with the number of nodes, i.e. v = O ( n ) , the complexity is not better than O (cid:0) n 2 (cid:1) .

Original: To minimize this issue, one of the important contribution of (Daitch et al., 2009) is to solve the problem on subset v of edges.
Revised: To minimize this issue, one of the important contribution of (Daitch et al., 2009) is to solve the problem on subset v of edges.

Original: Nevertheless, instead of keeping the support ﬁxed (as we do), they propose to check the KKT conditions of the dual variable, they can assess if the some edges should be added to the support at end optimization scheme.
Revised: Nevertheless, instead of keeping the support ﬁxed (as we do), they propose to check the KKT conditions of the dual variable, they can assess if the some edges should be added to the support at end optimization scheme.

Original: If so, the optimization is run again with the new edges.
Revised: If so, the optimization is run again with the new edges.

Original: The process is repeated until all KKT conditions are satisﬁed.
Revised: The process is repeated until all KKT conditions are satisﬁed.

Original: This technique allows for ﬁnding the global solution of larger problems, (particularly if SDPT3 is used).
Revised: This technique allows for ﬁnding the global solution of larger problems, (particularly if SDPT3 is used).

Original: We note however that the search of the next relevant support costs O ( n 2 ) again, since they have to compute the KKT conditions for all possible edges.
Revised: We note however that the search of the next relevant support costs O ( n 2 ) again, since they have to compute the KKT conditions for all possible edges.

Original: We started by implementing the original "hard" and "soft" algorithms and obtained similar speed that were comparable as the results reported in the original paper.
Revised: We started by implementing the original "hard" and "soft" algorithms and obtained similar speed that were comparable as the results reported in the original paper.

Original: Unfortunately, we were not able to run theses implementations for more than a few thousands of nodes.
Revised: Unfortunately, we were not able to run theses implementations for more than a few thousands of nodes.

Original: Hence, in order to compare with our algorithm, we had to cope with their scalability issues ﬁrst.
Revised: Hence, in order to compare with our algorithm, we had to cope with their scalability issues ﬁrst.

Original: Essentially, we derived new algorithms thanks to two modiﬁcations.
Revised: Essentially, we derived new algorithms thanks to two modiﬁcations.

Original: First, we used FISTA ((Beck & Teboulle, 2009)) and forwardbackward-based primal-dual (Komodakis & Pesquet, 2015, Algorithm 2) optimization schemes respectively for the soft and hard graph optimization instead of SDPT3.
Revised: First, we used FISTA ((Beck & Teboulle, 2009)) and forwardbackward-based primal-dual (Komodakis & Pesquet, 2015, Algorithm 2) optimization schemes respectively for the soft and hard graph optimization instead of SDPT3.

Original: Second, we removed the support optimization using the KKT conditions and use the same A-NN support as our algorithm.
Revised: Second, we removed the support optimization using the KKT conditions and use the same A-NN support as our algorithm.

Original: After those modiﬁcation, our implementation scaled to approximately a 100 (cid:48) 000 nodes.
Revised: After those modiﬁcation, our implementation scaled to approximately a 100 (cid:48) 000 nodes.

Original: While the theoretical complexity is about the same as for our optimization problem, the running time are signiﬁcantly longer than our method because the term (cid:107) LX (cid:107) 2 F = (cid:107) Mw (cid:107) 2 2 requires d times more computation than tr (cid:0) X T LX (cid:1)
Revised: While the theoretical complexity is about the same as for our optimization problem, the running time are signiﬁcantly longer than our method because the term (cid:107) LX (cid:107) 2 F = (cid:107) Mw (cid:107) 22 requires d times more computation than tr (cid:0) X T LX (cid:1)

Original: = (cid:107) Zw (cid:107) 1 , where d is the number of signals.
Revised: = (cid:107) Zw (cid:107) 1 , where d is the number of signals.

Original: In order to be able to compare to these algorithm in Figure 1, we randomly projected the original d = 300 input features to a subspace of size d = 20 (this does not change signiﬁcantly the computation time of our model).
Revised: In order to be able to compare to these algorithm in Figure 1, we randomly projected the original d = 300 input features to a subspace of size d = 20 (this does not change signiﬁcantly the computation time of our model).

Original: While the cost per iteration of the "soft" method is slower than for the "hard" method, the latter typically needs more iterations to reach convergence.
Revised: While the cost per iteration of the "soft" method is slower than for the "hard" method, the latter typically needs more iterations to reach convergence.

Original: The reason is that our accelerated algorithm uses FISTA that has a global rate of convergence proven to be signiﬁcantly better than the forward-backward based primal-dual we used (Beck & Teboulle, 2009; Komodakis & Pesquet, 2015).
Revised: The reason is that our accelerated algorithm uses FISTA that has a global rate of convergence proven to be signiﬁcantly better than the forward-backward based primal-dual we used (Beck & Teboulle, 2009; Komodakis & Pesquet, 2015).

Original: C E XPERIMENTS C. D ATASETS
Revised: D E XPERIMENTS D. D ATASETS

Original: • MNIST: We use the 60000 images of the MNIST training dataset.
Revised: • MNIST: We use the 60000 images of the MNIST training dataset.

Original: • US Census 1990:
Revised: • US Census 1990:

Original: Dataset available at the UCI machine learning repository, consists of approximately 2.5 million samples of 68 features.
Revised: Dataset available at the UCI machine learning repository, consists of approximately 2.5 million samples of 68 features.

Original: https://archive.ics.uci.edu/ ml/datasets/US+Census+Data+(1990) • word2vec:
Revised: https://archive.ics.uci.edu/ ml/datasets/US+Census+Data+(1990) • word2vec:

Original: The 10 (cid:48) 000 most used words in English ( https://research.
Revised: The 10 (cid:48) 000 most used words in English ( https://research.

Original: googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.
Revised: googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.

Original: html ).
Revised: html ).

Original: It uses the Google word2vec features ( https://code.google.com/ archive/p/word2vec/ ).
Revised: It uses the Google word2vec features ( https://code.google.com/ archive/p/word2vec/ ).

Original: • Spherical data: Simulated cosmological mass maps (i e. the amount of mass in the universe observed from earth in every direction) from Perraudin et al. 
Revised: • Spherical data: Simulated cosmological mass maps (i.e. the amount of mass in the universe observed from earth in every direction) from Perraudin et al. ; Sgier et al.

Original: The dataset consists of two sets of maps that were created using the standard cosmological model with two sets of cosmological parameters.
Revised: The dataset consists of two sets of maps that were created using the standard cosmological model with two sets of cosmological parameters.

Original: This data resides on the sphere, which can be considered as its underlying manifold.
Revised: This data resides on the sphere, which can be considered as its underlying manifold.

Original: We used the slightly smoothed maps (Gaussian kernel of radius 3 arcmin), as proposed in the original paper.
Revised: We used the slightly smoothed maps (Gaussian kernel of radius 3 arcmin), as proposed in the original paper.

Original: While the original data consists of 40 maps of size 12 · 1024 2 , we work only a sup-part of the sphere of size.
Revised: While the original data consists of 40 maps of size 12 · 1024 2 , we work only a sup-part of the sphere of size.

Original: This allow us to build 1920 signals on a sub-part of the sphere ( 512 × 512 grid), i.e. between 262 , 144 nodes.
Revised: This allow us to build 1920 signals on a sub-part of the sphere ( 512 × 512 grid), i.e. between 262 ,nodes.

Original: The actual manifold with the correct coordinates is plotted in Figure 15 up left.
Revised: The actual manifold with the correct coordinates is plotted in Figure 15 up left.

Revised: https://zenodo.org/record/

Original: • Small spherical data: A subset of "Spherical data" containing 4096 nodes, (64x64 grid) and the same 1920 signals.
Revised: • Small spherical data: A subset of "Spherical data" containing 4096 nodes, (64x64 grid) and the same 1920 signals.

Original: • USPS: https://ieeexplore.ieee.org/document/
Revised: • USPS: Handwritten digits from 0 to 9 . https://ieeexplore.ieee.org/ document/

Original: C. MNIST IRREGULAR INTRA - CLASS DISTANCES
Revised: D. MNIST IRREGULAR INTRA - CLASS DISTANCES

Original: Figure 10 illustrates one irregularity of the MNIST dataset.
Revised: Figure 10 illustrates one irregularity of the MNIST dataset.

Original: One could expect that the average distance between two digits of the same class (intra-class) is more or less independent of the class.
Revised: One could expect that the average distance between two digits of the same class (intra-class) is more or less independent of the class.

Original: Nevertheless, for the MNIST dataset, the distance between the 1 is signiﬁcantly smaller than the one of the other digits.
Revised: Nevertheless, for the MNIST dataset, the distance between the 1 is signiﬁcantly smaller than the one of the other digits.

Original: For this reason, the L2 model connects signiﬁcantly more the digits 1 than the others.
Revised: For this reason, the L2 model connects signiﬁcantly more the digits 1 than the others.

Original: [Figure]
Revised: [Figure]

Original: C. A PPROXIMATION ACCURACY AND ROBUSTNESS OF PARAMETER θ
Revised: D. A PPROXIMATION ACCURACY AND ROBUSTNESS OF PARAMETER θ

Original: We already saw in Figure 2 (middle) that for the MNIST dataset (17) predicts very well the sparsity of the ﬁnal graph for any choice of θ .
Revised: We already saw in Figure 2 (middle) that for the MNIST dataset (17) predicts very well the sparsity of the ﬁnal graph for any choice of θ .

Original: This is further illustrated on the USPS and ATT faces datasets in Figure 12.
Revised: This is further illustrated on the USPS and ATT faces datasets in Figure 12.

Original: Note, that in the rare cases that the actual sparsity is outside the predicted bounds, we already have a good starting point for ﬁnding a good θ .
Revised: Note, that in the rare cases that the actual sparsity is outside the predicted bounds, we already have a good starting point for ﬁnding a good θ .

Original: For example, in the COIL dataset, if we want a graph with 15 edges per node we will set θ = 1 . 2 , obtaining instead a graph with 12 edges per node.
Revised: For example, in the COIL dataset, if we want a graph with 15 edges per node we will set θ = 1 . 2 , obtaining instead a graph with 12 edges per node.

Original: This kind of ﬂuctuations are usually tolerated, while even in k -NN we always obtain graphs with more than nk edges due to the fact that W is symmetric.
Revised: This kind of ﬂuctuations are usually tolerated, while even in k -NN we always obtain graphs with more than nk edges due to the fact that W is symmetric.

Original: C.3. R OBUSTNESS TO OUTLIERS AND DUPLICATES
Revised: D.3. R OBUSTNESS TO OUTLIERS AND DUPLICATES

Original: We test the robustness of the bounds of θ by repeating the experiment of Figure 2 (middle) after replacing 10% of the images with outliers or duplicates.
Revised: We test the robustness of the bounds of θ by repeating the experiment of Figure 2 (middle) after replacing 10% of the images with outliers or duplicates.

Original: For this experiment, we ﬁrst added outliers by contaminating 10% of the images with Gaussian noise from N ( (cid:48) , ∞ ) .
Revised: For this experiment, we ﬁrst added outliers by contaminating 10% of the images with Gaussian noise from N ( (cid:48) , ∞ ) .

Original: Note that this is huge noise since the initial images have intensities in the range [0 , 1] .
Revised: Note that this is huge noise since the initial images have intensities in the range [0 , 1] .

Original: The result, plotted in Figure 11 (left), is almost identical to the results without outliers (Figure 2, middle).
Revised: The result, plotted in Figure 11 (left), is almost identical to the results without outliers (Figure 2, middle).

Original: While this might seem surprising, note that θ given by eq.
Revised: While this might seem surprising, note that θ given by eq.

Original: (17) are really dependent on the smallest distances in Z , rather than the largest ones:
Revised: (17) are really dependent on the smallest distances in Z , rather than the largest ones:

Original: ˆ Z is sorted, and B as well.
Revised: ˆ Z is sorted, and B as well.

Original: Adding outliers induces additional nodes distances larger than usually, that never make it to change the ﬁrst k rows of matrices ˆ
Revised: Adding outliers induces additional nodes distances larger than usually, that never make it to change the ﬁrst k rows of matrices ˆ

Original: Z and B , for the columns that correspond to non-outlier nodes.
Revised: Z and B , for the columns that correspond to non-outlier nodes.

Original: Therefore, eq.
Revised: Therefore, eq.

Original: (17) is very robust to outliers .
Revised: (17) is very robust to outliers .

Original: To complete the experiment, instead of adding noise, we replaced 10% of the images with copies of other images already in the dataset.
Revised: To complete the experiment, instead of adding noise, we replaced 10% of the images with copies of other images already in the dataset.

Original: In this case, we have 20% of images in pairs of duplicates , with essentially zero distances with each other in matrix Z .
Revised: In this case, we have 20% of images in pairs of duplicates , with essentially zero distances with each other in matrix Z .

Original: As we see in Figure 11 (right), while the intervals of θ change, they do so following closely the actual measured sparsity . 
Revised: As we see in Figure 11 (right), while the intervals of θ change, they do so following closely the actual measured sparsity . [Figure]

Original: [Figure]
Revised: [Figure]

Original: C. C ONNECTIVITY EXAMPLE OF THE GRAPH OF WORDS
Revised: D. C ONNECTIVITY EXAMPLE OF THE GRAPH OF WORDS

Original: In Table 1, we look in more detail at the graph constructed from the word2vec features.
Revised: In Table [Figure] we look in more detail at the graph constructed from the word2vec features.

Original: We present the connectivity for the word "glucose" and "academy".
Revised: We present the connectivity for the word "glucose" and "academy".

Original: Looking at different words, we observe that the learned graph is able to associate meaningful edge weights to the different words according to the conﬁdence of their similarity. 
Revised: Looking at different words, we observe that the learned graph is able to associate meaningful edge weights to the different words according to the conﬁdence of their similarity. [Table]

Original: C. MNIST CONNECTIVITY FOR (D AITCH ET AL ., 2009) METHODS
Revised: D. MNIST CONNECTIVITY FOR (D AITCH ET AL ., 2009) METHODS

Original: In Figure13, we plot the connectivity across different digits of MNIST for the "hard" and "soft" model.
Revised: In Figure we plot the connectivity across different digits of MNIST for the "hard" and "soft" model.

Original: As the degree is constant over the nodes, the hard model perform similarly to the A-NN (see
Revised: As the degree is constant over the nodes, the hard model perform similarly to the A-NN (see

Original: Figure 5).
Revised: Figure 5).

Original: It seems that due to hard constraint, the "hard" model forces many edges with a small [Figure] [Figure] [Table]
Revised: It seems that due to hard constraint, the "hard" model forces many edges with a small weights.

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: On the other hand, in terms of connextivity, the soft model seems to be between the log and the (cid:96) 2 model.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: s, this effect becomes worse with higher density.

Original: [Figure]

Original: [Table] [Figure] weights. On the other hand, in terms of connextivity, the soft model seems to be between the log
Revised: Note also that these algorithms fail to give reasonable graphs for densities outside a small range, making it very difﬁcult to control sparsity.

Original:  C 
Revised: D. MNIST C OMPUTATIONAL TIME WITH RESPECT TO K

Original: MNIST C OMPUTATIONAL TIME WITH RESPECT TO K

Original: The cost of learning a graph with a subset of allowed edges E allowed is linear to the size of the set as illustrated in Figure 14.
Revised: The cost of learning a graph with a subset of allowed edges E allowed is linear to the size of the set as illustrated in Figure 14.

Original: For this experiment, we use the MNIST data set.
Revised: For this experiment, we use the MNIST data set.

Original: To learn a graph with approximately 10 edges per node, we needed 20 seconds to compute E allowed , and 20 seconds to learn the ﬁnal graph of 60000 nodes (around 250 iterations).
Revised: To learn a graph with approximately 10 edges per node, we needed 20 seconds to compute E allowed , and 20 seconds to learn the ﬁnal graph of 60000 nodes (around 250 iterations).

Original: Note that the time necessary to search for the nearest neighbors is in the same order of magnitude than the learning process.
Revised: Note that the time necessary to search for the nearest neighbors is in the same order of magnitude than the learning process.

Original: [Figure]
Revised: [Figure]