Learning the Structure of Large Networked Systems Obeying Conservation Laws ⇤
Abstract
Many networked systems such as electric networks, the brain, and social networks of opinion dynamics are known to obey conservation laws.
Examples of this phenomenon include the Kirchoff laws in electric networks and opinion consensus in social networks.
Conservation laws in networked systems are modeled as balance equations of the form X = B ⇤ Y , where the sparsity pattern of B ⇤ 2 R p ⇥
p captures the connectivity of the network on p nodes, and Y, X 2 R p are vectors of potentials and injected ﬂows at the nodes respectively.
The node potentials Y cause ﬂows across edges which aim to balance out the potential difference, and the ﬂows X injected at the nodes are extraneous to the network dynamics.
In several practical systems, the network structure is often unknown and needs to be estimated from data to facilitate modeling, management, and control.
To this end, one has access to samples of the node potentials Y , but only the statistics of the node injections X .
Motivated by this important problem, we study the estimation of the sparsity structure of the matrix B ⇤ from n samples of Y under the assumption that the node injections
X follow a Gaussian distribution with a known covariance ⌃ X .
We propose a new ` 1 -regularized maximum likelihood estimator for tackling this problem in the high-dimensional regime where the size of the network may be vastly larger than the number of samples n .
We show that this optimization problem is convex in the objective and admits a unique solution.
Under a new mutual incoherence condition, we establish sufﬁcient conditions on the triple ( n, p, d ) for which exact sparsity recovery of B ⇤ is possible with high probability; and d is the degree of the underlying graph.
We also establish guarantees for the recovery of B ⇤ in the element-wise maximum, Frobenius, and operator norms.
Finally, we complement our theoretical results with experimental validation of the performance of the proposed estimator on synthetic and real world data.
Introduction
Let G =
([ p ] , E ) be a directed graph on the vertex set [ p ] , { 1 , 2 , . .
. , p } with a size m edge set
E ⇢ [ p ] ⇥ [ p ] .
Let D denote the p ⇥ m incidence matrix that encodes the edges of G as follows: each column of D is associated with an edge ( i, j ) 2 E and is a vector of zeros except at the locations i and j where it is  1 and +1 respectively.
Let X 2 R p be a vector of injected ﬂows or signals at the vertices and let f 2 R m be the vector of ﬂows through the edges.
Then, the basic conservation law between these ﬂows may be expressed as Df + X = 0 ; that is, at each vertex, the ﬂow (which is a linear combination of ﬂows at the edges incident on the vertex) has to balance the injections.
In physical systems, edge ﬂows f often arise as a way to balance the differences between certain potentials Y 2 R p at the vertices.
That is, the ﬂows satisfy f =  D T Y ; notice that this implies that the ﬂow at the edge ( i, j ) is given by Y j  Y i .
Thus, the above conservation law yields the following relationship, called a balance equation , between the node potentials and injected ﬂows:
[Equation]
where B ⇤ , DD T 2 R p ⇥ p is the symmetric Laplacian matrix [4, 60].
In an electrical circuit (with unit resistances on edges), Y corresponds to the voltage potentials at the vertices, f corresponds to the edge currents, and X denotes the injected currents at the vertices.
Indeed, this picture can be generalized by assigning weights to the edges of the network (conductances in the case of an electric network), and allowing the ﬂows to be weighted by these weights.
The model in (1) is referred to as generalized Kirchoff’s law, and importantly, this models the relationship between ﬂows and potentials in a range of systems that satisfy conservation laws such as hydraulic networks, opinion consensus in social networks, and transportation/distribution networks (see [60, 40, 55] and references therein).
It can be readily seen that the Laplacian B ⇤ is a positive semi-deﬁnite that encodes the edges of
G .
Speciﬁcally, ( i, j ) 2 E
if and only if B ⇤ ij 6 = 0 .
The Laplacian lies at the heart of spectral graph theory [15], and owing to its fascinating properties has found a range of applications in diverse areas such as image processing, manifold learning, spectral clustering, and bandits [52, 6, 62, 58, 32, 56].
In this paper, we consider a situation where the edge set E of the graph is unknown and needs to be estimated from measurements of the node potentials Y .
Based on the above discussion, we will cast this as a problem of learning an unknown positive deﬁnite B ⇤ (or the sparsity pattern thereof) from measurements of Y .
Further, we suppose that we only have access to the statistics of X , namely, that it is a 0-mean Gaussian random vector with a covariance matrix ⌃ X .
The situations where ⌃ X is unknown and B ⇤ is non-invertible is brieﬂy discussed in the remarks in Section 2.
We list a variety of applications where this learning problem arises naturally.
Topology learning in electric networks : Consider an electric network (or circuit) with p nodes, current injections X , node voltages Y , conductances A ij  0 between nodes i and j , and shunt conductances A ii  0 connecting i -th node to the ground.
The currentbalance equation is given by (1), where B ⇤ is the Laplacian with B ⇤ ij =  A ij and B ⇤ ii
= A ii + P nj =
A ij [21].
In real electric grids, current injections are unknown random variables.
To ensure reliable power supply, learning B ⇤ and its underlying graph from voltage samples is important and has been widely studied [20, 33, 13, 1].
The current-balance equation also appears in Markov chains and ﬂow networks where Kirchoff laws apply [60, 46].
Brain connectivity from graph ﬁlters : The structural connectivity of the human brain is often studied using a network with nodes representing brain regions, and the edge weights representing the density of anatomical connections [45, 25, 61].
Recent studies showed that the weights can be inferred using graph ﬁlters satisfying (1) with B ⇤ = P
[Equation] h l A l [-=
-])
1 , where A is the symmetric adjacency matrix; h l is the ﬁlter coefﬁcient; and X is the latent graph signal.
For brain networks, [51, 36] showed that L = 3 is a reasonable choice.
Additionally, for large L , the graph ﬁlter B ⇤ can be approximated as B ⇤ = 1  ↵ A using matrix power series expansion, thus a feasible choice for analysis.
Graph ﬁlters are also used in social and protein interaction networks [50, 37].
Structural equation models (SEM) :
Structural equation models are used to explain relationships among exploratory variables in several domains; for e.g., psychoanalysis [22], social sciences [16], medical research, and neuroimaging [7, 47, 38].
Using SEMs, [42] provided a causal interpretation of Linear Hawkes Processes.
In SEM with no latent variables, we let y
[Equation] x , where A is the path matrix .
Then the SEM satisiﬁes (1) with B ⇤ =
I  A . 4.
Linear dynamical (diffusion) networks : These network dynamics are satisﬁed by many systems including consensus dynamics, thermal capacitance networks, power swing dynamics, [54].
Further, by lifting approach, these dynamics can be used to study periodic/cyclic behavior in atmospheric systems [53].
Before we detail our topology discovery method, we comment on a few competing approaches that only have limited utility in our setting.
First, penalized (nodal) regression methods [39] are not applicable here since these require samples of both X and Y .
Yet other works have considered the estimation of topology from linear measurements [2, 9, 33, 65, 11, 14, 18, 19], but these assume direct linear access to the graph (via the adjacency or appropriate covariances) or assume low-rankedness in the underlying structure.
A recent line of work [20, 1], proposed estimating B ⇤ by estimating the inverse covariance (or precision) matrix ⇥ ⇤ of Y using the graphical LASSO (GLASSO) [66, 23].
In particular, [20, 1] showed that ⇥ ⇤ has non-zeros corresponding to those pairs of vertices that are connected by paths of length at most two; that is, the ( i, j ) -th entry of ⇥ ⇤ is non zero if and only if ( i, j ) is an edge in G or there is a k 2 [ p ] such that i  k  j is a path in G .
The authors then estimated edges of G by identifying (and eliminating) the pairs of vertices that have two-hop connections in G (see Fig.
1)—for future reference, we call this estimator as GLASSO+2HR (2 hop reﬁnement).
However, this estimator requires strong structural assumptions on G such as trianglefreeness.
Further, the precision matrix ⇥ ⇤ of Y is far more dense than the underlying graph G since ⇥ ⇤ =
B ⇤ ⌃  1 X ( B ⇤ ) T ; this results in sub-optimal data requirements for reliable recovery (see Remark 3).
Finally, if ⌃ X is a diagonal matrix, we can estimate the sparsity pattern of B ⇤ by taking the principal square root of the empirical covariance matrix of Y .
Unfortunately, this method does not allow for any correlation between the node injections which is not the case in practice.
Importantly, this method is numerically unstable unless one has a large number of samples ( n ) of Y so that the empirical covariance matrix is invertible—-a requirement that is at odds with the high-dimensional regime where one typically desires n to be smaller than the number of variables p .
In light of the limitations of previous approaches, we study a natural penalized maximum likelihood estimator for B ⇤ using the samples { Y i } ni =1 .
This estimator is not only statistically efﬁcient but also obviates the restrictive assumptions imposed by the aforementioned methods.
We now summarize the main contributions of the paper:
• We propose a novel ` 1 regularized maximum likelihood estimator (MLE) for B ⇤ from samples of Y .
It is worth noting that the optimization program we propose is not the standard graphical LASSO program [66, 23] as it involves terms that are quadratic in the optimization variable.
Our ﬁrst result shows that, notwithstanding its form, the ` 1 regularized MLE is convex in B and it has a unique minimum even in the high-dimensional regime ( n ⌧ p ) under certain standard conditions.
• Under a new mutual incoherence condition, we provide a sufﬁcient condition on the number of samples n required to recover the exact sparsity of B ⇤ with high probability.
Furthermore, under these sufﬁcient conditions we also establish the consistency of our estimator in the element-wise maximum, Frobenius, and spectral norms.
Formally, we show that if n = ⌦ ( d 2 log p ) then with high probability k [Equation]
log p [Equation] ) , where d is the degree of the underlying graph.
Finally, we complement our theoretical results with experimental results both on the synthetic data sets and data from a benchmark power distribution system.
Our experiments demonstrate the clear beneﬁt of the proposed estimator over baseline and competing methods
estimate advocated in recovers B by ﬁrst estimating ⇥ ⇤ =
B ⇤ ⌃  1 X ( B
) T which is potentially dense when compared to B ⇤ (see Fig 1(a) and Fig 1(b)).
Hence, the GLASSO+2HR estimate in Fig 1(d) might have more edges than the true graph. However, since our ` 1 -MLE directly estimates the sparse matrix B ⇤ , likely, the estimated and the true graph might only differ on a few edges, as shown in Fig 1 (c). Our experimental results in Section 4 validate this claim. Organization of the paper :
In Section 2, we introduce an ` 1 -regularized ML estimation problem for networked systems obeying conservation laws.
In Section 3, we show that this optimization problem is convex in the objective and establish consistency and support recovery rates for our estimator.
In
Section 4, we provide simulation results.
In Section 5, we summarize our paper with future directions.
Notation : For any two subsets T 1 and T 2 of [ p ] ⇥ [ p ] , we denote by A T 1 T 2 the submatrix of A with rows and columns indexed by T 1 and T 2 , respectively.
When T 1 = T 2 we denote the submatrix by A T 1 .
For a matrix A
=
( A i,j ) 2 R p ⇥ p , we use k A k 1 , max i,j |
A ij | to denote the maximum element-wise norm, and k A k F and k
A k 2 to denote the Frobenius norm and the operator norm.
We denote the ` 1 -matrix norm of A deﬁned as ⌫ A = |||
A ||| 1 , max j =1 ,...,p P pj =1 | A ij | .
We use k A k 1 , off =
P i 6 =
j | A ij | to denote the off-diagonal ` 1 norm.
We use vec ( A ) to denote the p 2 -vector formed by stacking the columns of A and use  ( A ) =
( I ⌦ A ) to denote the kronecker product of
A with the identity matrix I .
For symmetric positive deﬁnite matrices A
and A 2 , we use A 1  Ato denote A 1
A 2 is positive deﬁnite.
We deﬁne sign ( A ij ) =
+1 if A ij > 0 and sign ( A ij ) = if
A ij < 0 .
For two-real valued functions f ( · ) and g ( · ) , we write f ( n ) =
O ( g ( n )) if f ( n )  cg ( n ) and f ( n ) =
⌦ ( g ( n )) if f ( n )  c 0 g ( n ) for constants c, c 0 > 0 .
Problem Setup
Consider a p -dimensional random vector X following the Gaussian distribution N (0 , ⌃ X ) with a known covariance matrix ⌃ X [-≻ 
-] 0 (we outline a relaxation of this assumption in Remark 1).
Let
Y =
( B ⇤ )  1 X with a p ⇥ p matrix B ⇤  0 and note that Y ⇠ N (0 , ⇥ ⇤ 1 ) , where ⇥ ⇤ =
B ⇤ ⌃ X B ⇤ .
Deﬁne the sample covariance matrix S = n
P n i =1 Y i Y T i , where { Y 1 , . .
, Y n } are the n (possibly n < p ) i.i.d.
samples of Y .
For some  n > 0 , we consider the ` 1 regularized MLE for estimating B ⇤ :
[Equation]
where k B k 1 , off =
P i 6 =
j | B ij | is the ` 1 -norm applied to the off-diagonal entries of B 2 R p ⇥ p .
The loss function in (2) without the ` 1 penalty is the negative log-likelihood of Y , and maximizing it to estimate B ⇤ yields an unrestricted MLE.
The optimization problem in (2) looks similar to the ` 1 -regularized log-determinant problem, which has a rich, long history in high-dimensional statistics, machine learning, signal processing, and network sciences (see for instance [35, 3, 66, 23]).
The bulk of this literature focuses on estimating ⇥ ⇤ .
The resultant estimator, referred to as the graphical LASSO (or GLASSO), has many nice theoretical properties (e.g., asymptotic consistency and support recovery in the high-dimensional regime) [48, 49, 67].
However, our estimator in (2) is signiﬁcantly different from GLASSO because we are estimating B ⇤ rather than ⇥ ⇤ .
Other studies close to our setup estimate a sparse Cholesky factor of ⇥ ⇤ [27, 17, 28].
Recall that the Cholesky decomposition is given by ⇥ ⇤ = LL T where
L  0 is a lower triangular matrix.
We differ from this line of work on multiple fronts: (i) we do not require B ⇤ to be a lower or upper triangular Cholesky factor; (ii) our method allows for arbitrary correlations between the nodal injections resulting in an extra ⌃  1 X between the factors; and (iii) to the best of our knowledge, ours is the ﬁrst work to provide guarantees on the sample complexity for estimating B ⇤ in the high-dimensional regime.
Remark 1.
( Unknown covariance matrix ⌃ X ).
In problem (2) , we assume that ⌃ X is known.
If this is not the case, we can slightly modify (2) to estimate B ⇤ D instead of B ⇤ , where D is the unique square root of ⌃  1 X satisfying D 2 = ⌃  1 X .
This approach works best if the sparsity of B ⇤ (approximately) equals the sparsity of B ⇤ D , which for instance happens when ⌃ X is (approximately) diagonal.
Remark 2.
( On invertibility of B ⇤ ).
The invertiblity assumption of B ⇤ ensures that B ⇤ is identiﬁable from samples.
This holds in several applications including the ones in (2)-(4) in Section 1.
However, this might not be true if B ⇤ is a Laplacian matrix that has k zero eigenvalues.
One common work around (see e.g., [24, 20, 21]) is to work with a reduced Laplacian matrix by deleting k rows and columns of B ⇤ (we employ this insight in our experiments; see Section 4).
A Convex Estimator and Statistical Guarantees
In this section, we ﬁrst recast the objective in (2) in terms of B for a known ⌃ X .
We then present our main results on the performance of our estimator in (2) when X is Gaussian and non-Gaussian.
We comment on extending our results to other convex loss functions and conclude with an overview of the key steps in proving our results.
Full details are given in the appendix.
We begin by rewriting the problem in (2) in a form that is more suitable to our methods of analysis.
Let D be the unique square root of ⌃  1 X satisfying D 2 =
⌃  1 X (see [8]).
Substituting B = B T and ⇥ =
BD 2 B T in the cost function of (2) yields the following:
[Equation]
where we use the fact that the trace operator is cyclic and the determinant of a matrix product equals the product of matrix determinants.
We dropped constants that have no effect on the estimate.
The symmetry and invertibility of B is sufﬁcient enough to ensure that log( · ) is well-deﬁned.
In other words, the positive-deﬁniteness assumption is not needed for the well-posedness of (3).
Lemma 1 below is the starting point of our analysis.
It establishes two key properties of the estimator in (3) under the positive deﬁniteness of B : (i) loss function in (3) is convex in B and (ii) b B is unique. The following result is proved in Appendix A.3.
Lemma 1.
( Convexity and Uniqueness)
For any  n > 0 and B  0 , if the diagonal elements of the sample covariance matrix S ii > 0 for all i , then (i) the ` 1 -log determinant problem in (3) is convex and (ii) b B in (3) is the unique minima satisfying the sub-gradient condition 2 D
[Equation]
Here b Z belong to the sub-gradient @ k b B k 1 , off with b
Z ij = 0 , for i = j , and b Z ij = sign( b B ij ) when b B ij 6 = 0 and | b
Z ij |  1 when b B ij = 0 , for i 6 = j .
A few comments of Lemma 1 are in order.
( Convexity )
First, we recall that the compositions of two convex functions is in general not convex.
As an example, consider two convex functions f ( x ) =
xand g ( x ) =
x , however, g ( f ( x ))
=  x 2 is not convex.
Therefore, in light of the fact that the loss function is a composite function of B , it is not clear if (3) is convex.
Nonetheless, in Lemma 1 we prove that (3) is a convex program.
Key to our proof is the notion of monotone convex functions.
( Uniqueness ) Second, the uniqueness result is non-trivial in high-dimensions ( n < p )
because the
Hessian is rank deﬁcient, and hence, the loss function in (3) might not be be strictly convex.
However, in Lemma 1 (ii) we show that b B is unique.
Key to our proof is the notion of coercivity and it adapts techniques in [48] to the case where the objective function is quadratic in the optimization variable.
3.1 Statement of Main result
Our ﬁrst result theoretically characterizes the performance of b B in (3) when Y is Gaussian.
Instead, our second result provides such a characterization for b B when Y is non-Gaussian.
At a crude level, our results guarantee that when the number of samples n scales as d 2 log p , our ` 1 -regularized MLE correctly recovers the support of B ⇤ and is close to B ⇤ (measured in Frobenius and operator norms) with high probability.
Here, d is the degree of the graph underlying B ⇤ .
Since we consider an ` 1 regularized log-determinant program for our ML estimator (3), our results might appear similar to that of [48].
However, as also pointed in Section 3.2, our main results, including the assumptions and sufﬁcient conditions needed to derive them, are not subsumed by those in [48], or vice versa (see Remark 3 and Section 4 for more thorough discussion).
We begin with the a few assumptions that are essential to prove our theoretical statements.
Similar subset of assumptions in the context of ` 1 regularized least squares problem appeared in [63, 57, 39, 68], and in the context of ` 1 regularized inverse covariance estimation problem appeared in [48, 67].
We deﬁne the edge set E ( B ⇤ )
= { ( i, j ) : B ⇤ ij 6 = 0 , for all i 6 = j } .
Let E :=
{E ( B ⇤ )
[ ( p, p ) } be the augmented set including the diagonal elements.
Let E c be the complement of E .
[A1] Mutual incoherence condition.
Let  ⇤ be the Hessian of the log-determinant function in (3):
[Equation]
For  ⇤ in (4), there exists some ↵ 2 (0 , 1] such that,
[Equation]
[A2] Hessian regularity condition.
Let d be the maximum number of non zero entries among all the rows in B ⇤ (i.e., the degree of the underlying graph), ⇥ ⇤ =
B ⇤ ⌃  1 X B ⇤ , and D 2 =
⌃  1 X .
Then,
[Equation]
A few comments are in order.
[ A1 ]
Our novel mutual incoherence condition on B ⇤ regulates the inﬂuence of irrelevant variables (elements of Hessian restricted to E c ⇥ E ) on relevant variables (elements of Hessian restricted to E ⇥ E ).
The ↵ -incoherence assumption of the above type is standard in literature, and [48] demonstrates its validity for several graphs, including chain and grid graphs, which we will explore in experimental section.
Notice that the ↵ -incoherence in [48] is imposed on ⇥ ⇤ .
Instead, we require it on B ⇤ .
[ A2 ] This condition is in parallel with bounding the maximum eigenvalue of ⇥ ⇤ 1 condition for estimating sparse ⇥ ⇤ (see for e.g., [49, 30]).
Our problem set-up assumes that the injected ﬂows X i , i 2 [ p ] , are Gaussian. Given that the vector of node potentials Y depends on X via the balance equation (see equation (1)), we have that Y is
Gaussian.
However, we work with sub-Gaussian distributions, a natural generalization to the Gaussian case, which encompasses many well known distributions that occur in practice (for e.g., bounded random variables, gaussians and mixture of gaussians).
We deﬁne this distributional assumption below.
Deﬁnition 1.
(Sub-Gaussian random variable)
A zero mean random variable Z is said to be subGaussian if there exists a constant  > 0 such that for any t 2 R , E [exp( tZ )]  exp   2 t 2 / .
Our ﬁrst main result below provides sufﬁcient conditions on the number of samples n needed for b B in (3) to exactly recover the sparsity structure of B ⇤ and to achieve sign consistency, deﬁned as sign ( b B ij )
=
sign ( B ⇤ ij ) , for all ( i, j ) 2 E .
We recall that ⌫ A = |||
A ||| 1 , max j =1 ,...,p P pj =1 |
A ij | and deﬁne ⌃ ⇤ = ⇥
⇤ 1 to be the covariance matrix of the node potential Y .
Theorem 1.
(Support Recovery: Sub-Gaussian) Let X =
( X 1 , . . .
, X p ) be the vector of injected ﬂows.
Suppose that for all i , X i / p ⌃ X ( ii ) is sub-Gaussian with parameter  and assumptions [ A1-A2 ] hold.
Let the regularization parameter  n =
Cp ⌧ (log 4 p ) /n , where C 0 is given below.
Given n independent samples from Y , if the sample size n > C 21 d 2 ( ⌧ log p + log 4) , the following hold with probability at least 1  1 p ⌧  2 , for some ⌧ > 2 : (a) b B exactly
recovers the sparsity structure of B ⇤ ; that is,
[Equation]
(b) b B satisﬁes the element-wise ` 1 bound k
[Equation]
and
(c) b B satisﬁes sign consistency if |
[Equation]
where C 1 = 192 p 2[(1+4  2 )  ( ⌃ ⇤ ) Tr( ⌃ X ) ⌫ D 2 ⌫ B ⇤ ⌫ B ⇤ 1 ] max { ⌫  ⇤ 1 ⌫ B ⇤ 1 , 2 ⌫ 2  ⇤ 1 ⌫ 3 B ⇤ 1 , 2 ↵  1 d  1 } ,
C 2 =
{+[64 p 2(1 + 4  2 )  ( ⌃ ⇤ ) Tr( ⌃ X ) ⌫  ⇤ 1 ⌫ D 2 ⌫ B ⇤ ⌫ B ⇤ 1 ] , C 
 = C 2 / (4 ⌫  ⇤ 1 ) and  ( · ) is the condition number.+}
The quantities ( ⌫  ⇤ 1 , ⌫ D 2 , ⌫ B ⇤ ,  ( ⌃ ⇤ ) , Tr( ⌃ X )) capture the inherent complexity of the model and do not depend on the number of samples n .
As long as the magnitude of the entries in  ⇤ 1 , D 2 , and B ⇤ scale as O (1 /d ) , the model complexity parameters do not depend on ( p, d ) .
That is, as the size of the network grows with ( p, d ) the edge strengths decay with d .
Suppose that the model complexity parameters are constants and that n = ⌦ ( d 2 log p ) .
Then part (a) of Theorem 1 guarantees that our ML estimator does not falsely include entries (or edges in the underlying graph) that are not in the support of B ⇤ .
Part (b) establishes the element-wise ` 1 norm consistency of b B ; that is, k b B  B {+⇤ k 
+}
= O ( p (log p ) /n ) .
Finally, part (c) establishes sign consistency of b B , and hence, our estimator does not falsely exclude entries that are in the support of B ⇤ .
Crucial is the requirement of | B ⇤ min | = ⌦ ⇣p (log p ) /n ⌘ , which puts a limit on the minimum (in absolute) value of the entries in
B ⇤ .
This condition parallels the familiar beta-min condition in the LASSO literature (see [63, 59]).
We now present a corollary to Theorem 1 that gives consistency rates of convergence for b B in the
Frobenius and operator norms.
Let E ( B ⇤ )
= { ( i, j ) : B ⇤ ij 6 = 0 , for all i 6 = j } be the edge set of B ⇤ .
Corollary 1.
Let s = |E
( B ⇤ ) | be the cardinality of E ( B ⇤ ) .
Under the same hypotheses in Theorem 1, with probability greater than 1  1 p ⌧  2 , the estimator b
B satisﬁes k
[Equation]
Proof sketch .
Both the Frobenius and operator norm bounds follows by applying standard matrix norm inequalities to the ` 1 consistency bound in part (b) of Theorem 1.
Importantly, s + p is the bound on maximum number of non-zero entries in B ⇤ , where s , by deﬁnition, is the total number of off-diagonal non-zeros in B ⇤ .
Complete details are provided in Appendix A.3.
Thus far we have assumed that the nodal potentials Y i are sub-Gaussian random variables.
We now explore another broad class of random variables with bounded k th moments, which are known to have tails that decay according to some power law [44].
An important example of power law distributions are Pareto distributions which ﬁnds applications in a wide variety of areas [43, 41].
Motivated by such important practical considerations, we state our next result for random variables with bounded moments.
We begin with the following deﬁnition.
Deﬁnition 2.
(Bounded moments)
A random variable Z is said to have bounded 4 k th moment if there exists a constant M k 2 R such that E [Equation]
Results below parallel Theorem 1 and Corollary 1 for random variables with bounded moments.
Theorem 2.
(Support Recovery: Bounded Moments) Let X =
( X 1 , . . .
, X p ) be the vector of injected ﬂows such that for all i , the node potential Y i / p ⌃ ⇤ ii has bounded moment as in Deﬁnition 2 and assumptions [ A1-A2 ] hold.
Let the regularization parameter  n = Cp ⌧ (log 4 p ) /n , with C 0 deﬁned in Theorem 1.
Given n independent samples from Y , if the sample size n > C 4 d 2 p ⌧ /k , then with probability greater than 1  1 /p ⌧  2 , for some ⌧ > 2 , the following hold: (a) b B exactly recovers the sparsity structure of B ⇤ (that is b B [Equation]
); (b) the element-wise ` 1 bound k [Equation] and (c) b B satisﬁes sign consistency if | B ⇤ min |
[Equation]
The constants and their dependence on the model complexity parameters are given in Appendix A.3.
Corollary 2.
Suppose the hypotheses in Theorem 2 hold.
Then with probability greater than 1  1 p ⌧  2 : k b B  B ⇤ k F  Cq (
[Equation]
( p [Equation] and k b [Equation]
B [Equation] k
[Equation]
 C 5 min { d, p s + p } q p ⌧ /k n , where s = |E ( B ⇤ ) | .
Interpretations of Theorem 1 and Corollary 1 also hold for Theorem 2 and Corollary 2.
However, in this setting, we have different sample size n = ⌦ ( d 2 p ⌧ /k ) and | B ⇤ min | = ⌦ ( p p ⌧ /k /n ) , where k is given by Deﬁnition 2.
In contrast, for sub-Gaussian case we have logarithmic dependence in p (the number of vertices).
Finally, albeit fundamentally different from GLASSO estimator, we were able to obtain consistency rates for b B (3) that are similar to those in [48, 12].
Remark 3.
(Comparison with the GLASSO estimator).
For simplicity, suppose that ⌃ X is diagonal.
Then, it follows that B ⇤  0 is the unique square root of ⇥ ⇤ =
( B ⇤ ) 2 .
Thus, a naïve way to estimate
B ⇤ is by taking the square root of the GLASSO estimate b ⇥ .
Let us call this estimator b B SR and note that b B SR inherits its optimal properties from b ⇥ .
We show that b ⇥ has sub-optimal estimation rate than b B in (3) for estimating B ⇤ .
Let B ⇤ contains d non-zero elements in every row.
Then the underlying graph of ⇥ ⇤ is a two-hop network with degree d 2 .
Using sample complexity results from [48], it follows that b ⇥ requires n = ⌦ ( d 4 log p ) to estimate B ⇤ .
Instead, our ` 1 -regularized MLE requires n = ⌦ ( d 2 log p ) samples.
This reduction is more pronounced for networks with a large degree d .
3.2 Outline of Main Analysis
We provide an outline of our methods and main strategies to prove Theorem 1.
We employ the primaldual witness technique —a well-known method used derive to statistical guarantees for sparse convex estimators [63, 64, 34].
This technique involves constructing a primal-dual pair ( e B, e Z ) satisfying the zero-subgradient condition of the convex problem in (3), such that (the primal) e B has the correct (signed) support.
Suppose this construction succeeds, from the uniqueness result in Lemma 1, it follows that b
B = e B , and the dual e
Z is an optimal solution to the dual of (1).
Thus, at the heart of our analysis is in showing that the primal-dual construction succeeds with high-probability.
Similar technique is also used to prove Theorem 2 (i.e., the non-Gaussian case); see Appendix A.3.
While our proof methods are inspired from [48, 63], our analysis is more involved due to the presence of B 2 , as opposed to B , in the loss function of (3).
Consequently, we require more nuanced assumption (as in [A2] ) and dual feasiblity condition than the ones in [48] (see below).
3.3 Primal-dual pair and supporting lemmas
We brieﬂy introduce the primal-dual witness construction.
In Lemma 2, we provide sufﬁcient conditions under which this construction succeeds.
We construct the primal-dual pair ( e B, e Z ) as follows.
The primal solution e B is determined by solving
[Equation]
Here (6) is a restricted problem, in that, we impose B E c = 0 .
Also, we have e B  0 and e B E c = 0 . The dual e
Z [Equation] k e B k 1 , off is chosen such that it satisﬁes the zero-subgradient condition of (6).
This is obtained by setting 2  n e [Equation]
[Equation]
[ D [Equation] e BS ] ij , for all ( i, j ) 2 E c .
It can be veriﬁed that ( e B, e Z ) satisﬁes the zero-subgradient condition (see the statement of Lemma 1) of the original problem in (3).
Thus, it remains to establish the strict dual feasibility condition; that is | e Z ij
[Equation] < 1 , for any ( i, j ) 2 E c .
We introduce some notation.
Let W , S  ⇥ ⇤ 1 be a measure of noise in the data, where S is the sample covariance and ⇥ ⇤ 1 is the true covariance of Y . Let  , e
B  B ⇤ be a measure of distortion between the primal solution e B as deﬁned in equation (6) and the true matrix to be estimated B ⇤ .
We also need the the higher order terms (denoted by R (  ) ) of the Taylor expansion of the gradient r log det( e B ) centered around B ⇤ [10]:
[Equation]
Lemma 2.
(Sufﬁcient conditions for strict dual feasibility) Let the regularization parameter  n >and ↵ be deﬁned as in [A1] .
Suppose the following holds max
[Equation]
Then the dual vector e
Z
[Equation] c satisﬁes k e Z E c k 1 < 1 , and hence, [Equation]
Proof sketch :
The proof essentially involves expressing the sub-gradient condition in Lemma 1 as a vectorized form using R (  ) (in (7)) and W .
By manipulating the vectorized sub-gradient condition, we obtain an expression of e
Z E c that is a function of the quantities in (8).
We ﬁnish off the proof by repeated applications of triangle inequality of norms and invoking assumptions in Lemma (8).
The following results provides us with dimension and model complexity dependent bounds on the reminder term R (  ) in (7) and the distortion  .
Lemma 3.
(Control of reminder)
Suppose that the element-wise ` 1 -bound k  k 1  1 3 ⌫ B ⇤ 1 d holds, where ⌫ B ⇤ 1 =  B ⇤ 1 .
Then k R (  ) k 1  32 d k  k 2 1 ⌫ 3 B ⇤ 1 .
The proof, adapted from [48], is algebraic in nature and relies on certain matrix expansions.
The details are provided in Appendix.
In the following result, we provide a sufﬁcient condition under which the element-wise ` 1 -bound on  in Lemma 3 holds.
Lemma 4.
(Control of  ) Let [Equation] min
[Equation]
⇤ 1 ⌫ 3 B ⇤ 1 d o .
Then we have the element-wise ` 1 bound k  k 1 =
k e B  B ⇤ k 1  r .
Proof sketch :
By construction e B E c = B ⇤ E c = 0 .
Hence, k  k 1 = k
E k 1 , where  E = e B E  B ⇤ E .
We construct a continous vector valued function F :  E !  E that has a unique ﬁxed point.
Invoking assumptions [A1]-[A2] , we show that F ( · ) is a contractive map on the ` 1 ball deﬁned as B r = { A : k
A k 1  r } with r deﬁned in the statement of the lemma.
Speciﬁcally, we show that F (
B r ) ✓ B r .
Finally, we ﬁnish off the proof by an application of Brower’s ﬁxed point theorem [29] to show that the unique ﬁxed point is inside B r .
Consequently, k  k 1  r .
Finally, the result in Theorem 1 follows by putting these lemmas together for an appropriate choice of  n and the sample size requirement, and there upon, invoking some known concentration inequalities.
Refer Appendix A.3 for complete details.
Consequently, GLASSO+HR needs more than recover the support ( plots
Fig
Experiments
We validate the support recovery performance of our ` 1 -regularized MLE on synthetic and a benchmark power distribution network (see Fig. 2).
We choose  n proportional to p log p/n .
Our results are averaged over 100 trials of n independent samples of Y .
We compare ` 1 -regularized MLE performance with (i) the square-root estimator (hereafter, GLASSO+SR) that identiﬁes the support of
B ⇤ by determining ( i, j ) for which b ⇥ 12 i,j 6 = 0 ; and (ii) the GLASSO+2HR (Hop Reﬁnement) estimator [20] that identiﬁes the support of B ⇤ by determining ( i, j ) for which
[Equation]
Here b ⇥ is the GLASSO estimate of the inverse covariance matrix of Y [23].
These estimators are described in detail in Introduction.
To have a fair comparison with the GLASSO based estimators, we set ⌃ X ( X is the injected vector) to be diagonal.
However, as discussed earlier, our ` 1 regularized ML estimator works for any ⌃ X  0 .
We consider p to be as large as 64 nodes, Computational examples involving large data matrices for B ⇤ having a lower triangular matrix form has been reported in [28]. (i) Synthetic data : We consider two undirected graphs for B ⇤ , the chain graph and the grid graph for p = { 32 , 64 } nodes.
We set B ⇤ ij = 1 for ( i, j ) 2 E and B ⇤ ij = 0 for ( i, j ) 2 E c , where E can be the edge set of the chain or grid graph.
We then adjust the diagonal elements of B ⇤ to ensure B [-∗ ≻ 
-]⇤  0 .
ii)
Power network: We set B ⇤ to be the Laplacian of the IEEE 33 bus power distribution network [69].
For this data, we note that B ⇤ is non-invertible because of one zero eigenvalue.
We obtain the reduced B ⇤ by deleting the ﬁrst row and column of B ⇤ .
We also slightly modify the network by adding three loops of cycle length three, two of cycle length four, and one loop of cycle length ﬁve (see Fig 2).
We made these modiﬁcations to highlight that ` 1 -MLE imposes no connectivity assumptions on the graph underlying B ⇤ , except sparsity.
In contrast, GLASSO+2HR estimator [20] restricts the graph underlying B ⇤ from having cycles of length three (i.e., triangle-free).
In Fig.
3, we show empirical support recovery probabilities for all three estimators as a function of the number of samples n .
Both on synthetic and power network data, ` 1 -MLE achieved superior rates than the other competing estimators.
In fact, ` 1 -MLE exactly recovers the support of B ⇤ when the number of samples is in the order of d 2 log p , which is in excellent agreement with the proposed theory.
Instead, for a similar performance, GLASSO+SR needed d 4 log p samples (see Remark 3).
Discussions and Future Work
High dimensional networks obeying conservation laws of the form X =
B ⇤ Y are often used to model and study interactions among different conserved quantities in various engineering and scientiﬁc disciplines.
For such systems, with unknown structure of the network, or equivalently the sparsity pattern of B ⇤ , we estimate the the sparsity structure of B ⇤ using an ` 1 -regularized ML estimator.
Our estimator relies on n samples of the node potentials Y , with some knowledge of the statistics of the node injections X .
We showed that this estimator is the unique optimal solution to a variant of the convex log-determinant program.
Using novel mutual incoherence conditions, we have provided (theoretical) sparsity and support recovery consistency for our estimator.
Finally, using several numerical results we not only validated our theory but also showed that our ` 1 -regularized estimator outperforms methods that employ GLASSO based estimators.
In our setup, we neither require actual injected ﬂows ( X ), nor B ⇤ to be a Laplacian matrix, nor n > p , thereby allowing our learning problem to be general enough to be applicable for a variety of domains ranging from electrical networks to social networks.
Consequently, our framework and theoretical results admits many future extensions and reﬁnements. For instance, verifying the assumptions [ A1-A2 ] in practice is a fruitful future direction.
It is worth noting that, in order to run the proposed algorithm/estimator, one need not verify the assumptions.
Notice that from Lemma 1, the loss function/objective is convex and admits a unique solution if the regularization constant  n > 0 and B is positive deﬁnite, therefore the unknown structure of the network can be recovered by ﬁnding the unique minima.
However, in order to theoretically guarantee the properties of the estimator mentioned in Theorem 1 and Theorem 2, assumptions [ A1-A2 ] are needed.
A possible extension to the current problem is to recast the objective function in (2) as the minimization of the
Bregman divergence for more general loss functions. Another promising direction is to generalize (2) to include non-invertible network matrix B ⇤ .
We leave this as our future work.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] The limitations of this work will be discussed in detail in the appendix (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code and it’s related details will be made available in the appendix.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] These will be mentioned in the appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]