antGLasso: An Efficient Tensor Graphical Lasso Algorithm
Abstract
The class of bigraphical lasso algorithms (and, more broadly, ‘tensor’-graphical lasso algorithms) has been used to estimate dependency structures within matrix and tensor data.
However, all current methods to do so take prohibitively long on modestly sized datasets.
We present a novel tensor-graphical lasso algorithm that analytically estimates the dependency structure, unlike its iterative predecessors.
This provides a speedup of multiple orders of magnitude, allowing this class of algorithms to be used on large, real-world datasets.
Introduction
We often make independence assumptions to simplify the modelling of complex data.
The strength of an independence assumption exists on a spectrum.
One could assume complete independence, independence except on a sparse subset, or independence of samples but not features, among others.
For conditional dependencies, there is a natural way to model them in Gaussian data.
Two datapoints x, y are conditionally independent (with respect to a dataset D ) if knowing one provides no information about the other that is not already contained in the rest of the dataset: P ( x | y, D )
=
P ( x |D ) .
For normally distributed data, conditional dependencies are encoded in the inverse of the covariance matrix (the ‘precision’ matrix).
Two datapoints are conditionally dependent on each other if and only if their corresponding element in the precision matrix is not zero.
If our dataset d were in the form of a vector, we could then model it as d ∼ N ( 0 , Ψ − 1 ) for precision matrix Ψ .
However, datasets are often not simple vectors - for example, single-cell RNA sequencing data (scRNA-seq) comes in the form of a matrix of gene expression counts whose rows are cells and columns are genes.
Video data naturally requires a third-order tensor of pixels to represent it - rows, columns, and frames.
To address this, we ‘vectorize’ matrices by stacking the columns vertically.
Vectorization of tensors proceeds in much the same way.
Our matrix-variate dataset D can then be represented as vec[ D ] ∼ N ( 0 , Ψ − 1 ) .
Suppose our matrix were n × p dimensional - then vec[ D ] has np elements, and its precision matrix has n 2 p 2 elements.
This poses a problem: the estimation of our precision matrix requires substantially more parameters than we have in our dataset.
Since the columns of the matrix have interdependencies and the rows of the matrix have interdependencies, to reduce space we can compute the interdependencies of the elements of the matrix via some deterministic function of the row-wise and column-wise dependencies.
That is, for some function ζ , vec[ D ] ∼ N ( 0 , ζ (Ψ row , Ψ col ) − 1 ) .
Background
The Kronecker sum bigraphical lasso (BiGLasso) model was first considered by Kalaitzis et al.
BiGLasso is the multi-axis analog to graphical lasso methods [3, 1], which are used to estimate covariance matrices of data drawn from a multivariate Gaussian distribution.
The Kronecker sum of two matrices, A ⊕ B , can be expressed in terms of Kronecker products: A ⊗ I
+ I ⊗ B .
When the matrices A , B are adjacency matrices of graphs, the Kronecker sum has the interpretation as the Cartesian product of those graphs[16].
This sum is one choice of function ζ to combine the per-axis precision matrices into the precision matrix of the vectorized dataset, vec[ D ] ∼ N ( 0 , (Ψ row ⊕ Ψ col ) − 1 ) .
Other choices for ζ have been considered, such as using the Kronecker product[18, 23], or the square of the Kronecker sum[20].
Each method has its strengths; the benefits of a Kronecker sum structure are its interpretability as a graph product, stronger sparsity, and its allowance of inter-task transfer[7].
The original BiGLasso model was very slow to converge to a solution, in large part due to its nonoptimal space complexity of O ( n 2 p 2 ) .
This prohibited its use on large datasets (measuring in a couple hundred samples and/or features).
Numerous modifications have been made to the algorithm to improve its speed and achieve an optimal space complexity of O ( n 2 + p 2 ) , such as scBiGLasso[10],
TeraLasso[4], and EiGLasso[21].
Of these, TeraLasso is notable in that it generalizes to an arbitrary amount of axes, i.e. ζ ( Ψ 1 , ..., Ψ k )
=
Ψ 1 ⊕ ...
⊕ Ψ k .
However, all of these algorithms are still iterative.
The convergence time can vary greatly depending on the precise values of the data, and in general they struggle to handle datasets with a couple thousand samples and/or features.
Our proposed variant, Heuristic antGLasso ( an alytic t ensor- g raphical lasso ), does not suffer from these issues, preserves optimal memory complexity, and like TeraLasso can work on datasets with an arbitrary amount of axes.
All algorithms listed rely on a normality assumption.
One could manually transform non-normal data to a Gaussian, if they knew the right transform to use.
An alternative method is to use the
Nonparanormal Skeptic[11], an algorithm which directly estimates the Gram matrices 1 of data that has been transformed to a Gaussian by some unspecified function[10].
Since all previous BiGLasso algorithms mentioned can be framed in terms of accepting these Gram matrices as input, they can be made to circumvent the normality assumption.
Methodology and Results
In the following we introduce the main theorems and results of our study.
For our software implementation, we used Python Version 3.9.12, Numpy Version 1.22.2[5], Scipy Version 1.7.3[19], Sklearn Version 1.0.2[14], Matplotlib Version 3.5.1[6], and Pandas Version 1.4.2[17, 12].
All tests were run on a MacBook Pro (13-inch, M1, 2020) with 8GB of RAM, using Jupyter Notebooks[15, 8].
Code is fully available on GitHub; https://github.com/BaileyAndrew/antGLasso-Implementation.
Theorems
The general structure of the algorithm is as follows: first, we directly estimate the eigenvectors of the solution.
We then use the eigenvectors to diagonalize the covariance matrix of the tensor-variate distribution, such that we can directly estimate the eigenvalues from the variances of the data.
We follow the notation of Kolda and Bader [9] for tensor operations, and Greenewald, Zhou, and Hero [4] for BiGLasso variables.
K ab is shorthand for a matrix with a on the diagonals and b elsewhere.
The size of this matrix will be clear from context. Our proofs of the following theorems are given in the Supplementary Material.
Theorem (Eigenvectors of Precision Matrices) .
V ℓ (the eigenvectors of Ψ ℓ ) are also the eigenvectors of S ℓ ◦ K
2 m ℓ − 1 m ℓ , where m ℓ = (cid:81) Ki = 1 d i d ℓ and d ℓ is the size of the ℓ th axis. 1 Defined in the Supplementary Material.
Theorem (Eigenvalues of Precision Matrices) .
Let a be the vector of length (cid:81)
Kℓ d ℓ whose ( (cid:80) ℓ i ℓ (cid:81) ℓ − 1 j =1 d j ) th element is 1 var[[ Y× 1 V T 1 × 2 ...
× K V TK ] i 1 ...iK ] .
Then there is a system of linear equations relating a to the eigenvalues of ( Ψ 1 , ..., Ψ K ) .
Unfortunately, the calculation of the eigenvalues requires the whole dataset Y , whereas the use of the
Nonparanormal Skeptic requires that the algorithm be parametrizable in terms of the Gram matrices.
We can circumvent this by introducing a heuristic, var (cid:2)
[ Y × 1 V T 1 × 2 ... × K V TK ] i 1 ,...,i K (cid:3)
≈ ∆ i ℓ V Tℓ S ℓ V ℓ ∆ Ti ℓ , where ∆ i ℓ is a row vector of zeros except for a one at position i ℓ .
We call this variant of the algorithm ‘Heuristic antGLasso’, in contrast to the previous ‘Vanilla antGLasso’; further details are available in the Supplementary Material.  Worked Example for Vanilla antGLasso Suppose our input matrix were
D = (cid:20) 1 24 5(cid:21)
The Gram matrices are then
{+S 
 = m 1 = 2 D T D =             , S 
 = m 2 = 3 DD T = (cid:20)+}
(cid:21)
By Theorem 1 we can get the eigenvectors V 1 , V 2 of our precision matrices Ψ 1 , Ψ 2 using our Gram matrices:
S 1 ◦ K 2 · 2 − 1=            
S 2 ◦ K 2 · 3 − 1= (cid:20)   (cid:21)
V 1 = eigenvectors (cid:2) S
[Equation]
K
[Equation]
(cid:3) ≈ (cid:34) − 0 .
896 − 0 .
279 − 0 .0 .− 0 .
729 − 0 .0 .0 .−
0 .(cid:35)
V
[Equation]
= eigenvectors (cid:2)
[Equation]
◦ K 53 (cid:3) ≈ (cid:20) − 0 .
178 0 .0 .(cid:21)
To prepare to use Theorem 4, we calculate:
[Equation]
(cid:21) .
Calculating the reciprocal of the variance when you only have one sample is rather boring:
[Equation]
[Equation]
The vector a referenced in Theorem 4 is just the vectorization of this matrix of reciprocal variances.
As shown in Lemmas 1 & 2 (Supplementary Material), we have that diag( Λ 1 ⊕ Λ 2 )
= a , where Λ are diagonal matrices whose diagonal elements are the eigenvalues.
Specifically, we have:  
[Equation]
  3170 .
93269 .

The summation pattern follows from the definition of the Kronecker sum.
We can express this as a linear relationship, which can be solved using standard techniques such as the pseudo-inverse.
  1 0 0 11 0 0 00 1 0 10 1 0 00 0 1 10 0 1 0    λλλλλ  =   .
93269 .
   λλλλλ  ≈   −−− 90 . 
This gives us both the eigenvalues and the eigenvectors of our solution, which could be combined to produce our precision matrices.
Notably, many of these estimated eigenvalues are negative.
As our precision matrices should be positive definite, this is not ideal.
We could use alternative methods to solve the system with non-negative solutions, although we did not experiment with this as we found the solution was still sufficient.
The actual solution method we used is different and involves a considerably smaller matrix, detailed with an example in the Supplementary Material - however it can still produce negative eigenvalues.
Results on Simulated Data
We first tested our algorithm on simulated data.
The precision matrices were drawn from an inverse
Wishart distribution.
The iterative algorithms tended to converge slowly when the degrees of freedom parameter of the distribution was small, and quickly when it was large.
Additionally, they converged much slower on small sample data (Figure 1).
We can see that antGLasso is by far the fastest algorithm, giving up to two orders of magnitude improvement over the next fastest algorithm (EiGLasso).
Graphs for tensor data results are given in the Supplementary Material.
For three-axis tensors, antGLasso is still markedly faster than TeraLasso in the small sample case - we did not have enough RAM to test the large sample case.
For four-axis and larger, the difference between the algorithms becomes negligible - this is because the major runtime constraint becomes the calculation of the Gram matrices, whose complexity grows exponentially with the amount of axes and thus dominates.
Vanilla antGLasso does not perform as well as other algorithms (Figure 2), although this gap drops as the number of samples increase.
Interestingly, Heuristic antGLasso performs comparably to the others while preserving the speed of Vanilla antGLasso.
We believe this is because Vanilla antGLasso directly estimates the variances of the data, which will be inaccurate for small samples.
The heuristic also estimates variances, but by using info on the variance across other axes to in essence artificially increase the sample count.
Temporal Recovery
We have shown that on simulated data Heuristic antGLasso performs comparably but much quicker than existing algorithms.
This enables the application of BiGLasso methods to datasets in the size of thousands.
For real world experiments, we can consider the case of video data.
Given the graph of conditional dependencies, we might expect to be able to order the frames, as a frame should only be conditionally dependant on its nearest neighbors.
For example, consider the video of a rubber duck from the COIL-20 dataset[13].
If we scramble the rows, frames, and columns of the video, we can recover the original video from the outputs of antGLasso, as seen in Figure 3.
The reconstruction is perfect, except for the duck being cut in half.
This is due to the difficulty in figuring out which row to start on.
The strong results for video reconstruction are encouraging.
However, when we modify the video by duplicating frames and adding a bit of noise our reconstruction performs much worse.
Already by duplicating each frame 5 times we dip below 10% reconstruction accuracy.
This is arguably because duplicating frames interferes with the conditional dependency graph - each frame has more similar frames in the rest of the dataset, which then get conditioned out, reducing informativeness.
This suggests that temporal recovery can only be done in highly structured cases with few redundancies.
We verified that this behavior is seen in EiGLasso as well and hence is a systemic problem of
BiGLasso methods rather than a quirk of antGLasso.
In Figure we can see how choice of algorithm as well as duplicate frames and noise affect temporal recovery.
When frames are not duplicated, using the Nonparanormal Skeptic performs the best.
However, once frames are duplicated it seems that using Heurstic antGLasso is better.
Even though the heuristic results in a lower accuracy, visually we can see that when it is wrong we can still trace a sensible progression along the graph.
Once the Nonparanormal Skeptic starts going wrong, it starts connecting completely disparate parts of the graph rather than failing gracefully.
Clustering
To test antGLasso’s performance on clustering realistic data, we generated a dataset using Splatter[22].
Splatter generates scRNA-seq data, which does not follow a normal distribution - hence using the
Nonparanormal Skeptic is justified.
There are three different interpretations of the output graphs that we can use for clustering - we can use the outputs directly (the precision matrices), we can invert the outputs (covariance matrices), or we can consider only the negative values of our precision matrices.
Inversion is relatively cheap as our algorithm estimates the eigenvalues and eigenvectors separately.
Values in the precision matrix of a Gaussian Graphical Model can be interpreted as the ‘negative stiffness’ of a spring connecting two elements together, motivating the consideration of the negative values of our outputs.
In addition to the three interpretations of our outputs, we have three different models to use: Vanilla antGLasso, Heuristic antGLasso, and Heuristic antGLasso with the Nonparanormal Skeptic.
The results can be seen in Figure 5.
Heuristic antGLasso is the best to use, whereas the Nonparanormal
Skeptic does very poorly.
We also tested EiGLasso’s performance on the same dataset, and found similar results.
Regularization
The algorithm is based on an analytic solution to the maximum of the unregularized likelihood function.
We do not have an analytic solution for the L1-penalized likelihood function.
Instead of regularizing during the calculation of precision matrices, we first find the unregularized solution, and then apply a regularization step.
By regularizing the solution at the end, rather than simultaneously, it allows us to efficiently examine the solution across many different penalty strengths.
However, it does not necessarily converge to the solution to the L1-penalized likelihood function, unlike previous algorithms.
As seen in Figure 2, Heuristic antGLasso still achieves state-of-the-art recovery in spite of this.
If we were to apply Lasso to our unregularized solution, in a similar manner to how BiGLasso and scBiGLasso apply Lasso at the end of each iteration, it would be equivalent to just thresholding.
This is convenient: not only is thresholding a very fast operation, but it can also be framed in terms of the percentage of edges to keep.
As seen in Figure 6, if we simulate data to have a given sparsity level, and then use the known ground truth sparsity as the hyperparameter, we achieve a balance between precision and recall.
Thus, if we have an idea of how densely connected our data should be and the extent to which we wish to prioritize precision over recall, we will know in advance a good regularization strength.
Despite the benefits of this sparsity-based regularizer, we often chose to use problem-specific regularization regimes.
For temporal recovery, a simple technique is to keep only the two highest values in each row (for the immediate past and immediate future).
For clustering, it can make sense not to regularize, as the additional edges may contain useful information for the clusterer despite not actually being present in the real data.
Discussion and Conclusion
We have created a new BiGLasso algorithm, antGLasso, which is much faster than current competitors, while preserving (in the heuristic case) their accuracy.
It can be applied to tensors of arbitrary dimensions, like TeraLasso, and performs well on simulated data as well as highly-structured realworld data.
It falters when applied to more complicated data, which is a major limitation.
However, this is a systemic problem with BiGLasso algorithms rather than our implementation specifically.
For future work, we would like to understand better when the Nonparanormal Skeptic should be used.
We know that the Nonparanormal Skeptic can be useful for clustering, as Li et al.
[10] used it to find interpretable results on real scRNA-seq data.
It seems that the Nonparanormal Skeptic performs poorly when the data is multimodal - in the case of duplicating frames in Figure 4, the duplication has the effect of creating easily separable clusters of duplicates and we believe this is what caused the Nonparanormal Skeptic variant to perform so poorly just as it did in the clustering tests of Figure 5.
Further investigation is needed.
We would also like to investigate how to circumvent the known issues with applying this method to less structured datasets, such as scRNA data, for temporal recovery and other tasks.
This may involve changing aspects of the model to better fit the idiosyncrasies of specific types of datasets.
It would also be worthwhile to investigate solutions that guarantee positive eigenvalues
A Supplementary Material A.1 Hyperparameters
For K -axis tensor data, there are K + 1 hyperparameters.
The first K are regularization parameters for each axis - as we will see in Section A. they can be interpreted as a percent of values to keep and are thus naturally interpretable.
The remaining hyperparameter, b controls the accuracy of the Monte
Carlo speedup discussed in Section A.8.
In our implementation b is the amount of terms to average over, although one could replace it with a hyperparameter controlling a convergence threshold.
In Figure 7 we can see that when our value of the regularization hyperparameter matches the true value of the edges (on simulated data) we achieve an even balance of precision and recall.
A.2 Simulating Data and Metrics
All experiments on simulated data use data simulated from the same process.
We first simulate sparse precision matrices, whose Kronecker sum is used as the parameter for a zero-mean tensor-variate normal distribution using Corollary 1.
For calculating precision and recall, we consider when edges from the true conditional dependency graph match with the estimated dependency graph.
To construct our precision/recall plots we set the b hyperparameter to 1000 and varied our sparsity parameters from 0 to 1 to obtain somewhat smooth curves.
b = 1000 was chosen empirically because in tests values above it did not change the quality of the estimation.
In fact a lower value such as b =could have been chosen, but the runtime difference is negligible so we selected the higher value.
To simulate sparse precision matrices, we first draw precision matrices from an inverse Wishart distribution.
These are not sparse.
To sparsify them, we construct a mask in the following manner: construct a vector of i.i.d.
Bernoulli variables x , and construct the matrix M
= c xx T where c ∈ (0 , 1) (we chose 0.9).
Let N the diagonals of M to be 1 .
By changing the parameter of the Bernoulli distribution we can control the expected sparsity of N .
By construction we have guaranteed that it is positive definite, 2 and hence the Hadamard product of it with our precision matrix will also be positive definite.
A.3 Pseudocode
We give the pseudocode for antGLasso in Algorithm 1 (with the Monte Carlo speedup described in
Section A.8).
Input: {Y ( n ) d 1 × ...
× d K } , { β ℓ } , b
Output: { Ψ ℓ } for 1 ≤ ℓ ≤ n
S ℓ ← 1 nm ℓ (cid:80) ni Y
( i ) ( ℓ ) Y ( i ) T ( ℓ )
V ℓ ← eigenvectors[ S ℓ ◦ K 2 m ℓ − 1 m ℓ ] end for for 1 ≤ p ≤ m
X ( p ) ← Y ( p ) × 1 V 1 × 2 ...
× K V K end for for ⃗i ∈ [1 , ..., d 1 ] × ... × [1 , ..., d K ] a (cid:80) ℓ i ℓ d 1:( ℓ − 1) ← n (cid:80) np
( x ( p ) i 1 ,...,iK )end for Λ ←
Construct ˜ B − 1 as described in Section A.8.
for b iterations of different tuples ⃗i
Let P ⃗i 1 , P ⃗i 2 be the matrices representing the permutations that capture the process described in
Section A.8.
If first iteration, set each element in λ 1:( K − 1) ⃗i toΛ ← Λ + P ⃗i 1 ˜ B − 1 P ⃗i 2 (cid:20) a λ 1:( K − 1) ⃗i (cid:21) end for Λ ← Λ /b for 1 ≤ ℓ ≤ K Ψ ℓ ← V ℓ Λ ℓ U Tℓ for i th row r in Ψ ℓ r \ i ← sign( r \ i ) ◦ min(0 , | r \ i | − β ℓ 2 ) end for end for A.4 Proofs Let s be the number of samples of tensors ( Y 1 , ... Y s ),
each with shape ( d 1 , ..., d K ) .
Each of these tensors is assumed to be i.i.d from our tensor-variate Kronecker sum normal distribution for which we want to estimate the per-axis precision matrices. It is not uncommon to have n = 1 . It will be helpful to define m ℓ = (cid:81) Ki =
1 d i d ℓ .
Let Ψ ℓ be the precision matrix of the ℓ th axis, and S ℓ be the
Gram matrix for the ℓ th axis.
These can either be obtained via the Nonparanormal Skeptic, or by the formula 1 nm ℓ (cid:80) ni
=1 Y i, ( ℓ ) Y Ti, ( ℓ ) , where A ( ℓ ) is the ‘matricization’ of the ℓ th axis.
Another piece of tensor-specific notation we use is the ℓ -mode product, Y × ℓ M , which intuitively is multiplying a tensor by a matrix along its ℓ th dimension.
When the input is a two-dimensional tensor (a matrix), ×and × 2 correspond to left-multiplying by the transpose and right-multiplying, respectively.
We often encounter the case Y × 1 V 1 × 2 V 2 × 3 ... , which is abbreviated as (cid:74) Y ; { V ℓ } (cid:75) .
For an overview of tensor notation, we direct the reader to the comprehensive report by Kolda and Bader [9].
We use ◦ to represent the Hadamard product, and tr n [ M ] to be the blockwise trace obtained as follows: partition
M into n × n blocks, and then take the trace of each block.
The output is the matrix of these traces.
A.5 Analytic Solution
Theorem 1.
Let V ℓ Λ ℓ V
Tℓ be the eigendecomposition of Ψ ℓ .
V ℓ are the eigenvectors of S ℓ ◦ K
2 m ℓ − 1 m ℓ .
Proof.
Greenewald, Zhou, and Hero [4] present a maximum likelihood estimator for the tensor variate case: − log (cid:12)(cid:12)(cid:12)(cid:76) Kℓ =1 Ψ ℓ (cid:12)(cid:12)(cid:12) + (cid:80) Kℓ =1 m
k S
Tℓ Ψ ℓ .
Using the exact same argument as Kalaitzis et al.
[7], we can derive the fixed point S ℓ − 12 m ℓ S ℓ ◦ I =
1 m ℓ tr m ℓ [( (cid:76) i Ψ i ) − 1 ] − 12 m ℓ tr m ℓ [( (cid:76) i Ψ i ) − 1 ] ◦ I .
The order in which we compute (cid:76) i Ψ i matters - we’ll assume here that we’ve transposed the data such that (cid:76) i Ψ i =
Ψ ℓ ⊕ (cid:76) i ̸ = ℓ Ψ i .
It is only important that Ψ ℓ goes first, the rest of the order does not matter.
Let Ψ \ ℓ = (cid:76) i ̸ = ℓ Ψ
i
so that (cid:76) i Ψ i = Ψ ℓ [Equation]
Ψ \ ℓ
[Equation]
The penultimate step uses Proposition 3.1 of Li et al.
[10], and the last step requires Lemma 2 of Dahl et al.
As tr m ℓ [( Λ ℓ ⊕ Λ \ ℓ ) − 1 ] is diagonal, we can observe that V ℓ must be the eigenvectors of
S ℓ ◦ K 2 m ℓ − 1 m ℓ .
Lemma 1.
Suppose Y ∼ N (0 , (cid:76) { Ψ i } − 1 ) .
Then we can diagonalize the precision matrix as follows:
X =
Y × 1 V T 1 × 2 ...
× K V TK ∼ N ( 0 , (cid:76) { Λ i } − 1 ) .
Proof.
We will show that the probability density function of X is that of a Kronecker sum distribution with the desired parameters.
It will rely on the following useful property of ℓ -mode matrix multiplication:
[Equation]
The following corollary provides a convenient way to sample from tensor-variate Kronecker sum distributions without high dimensionality or expensive matrix inverses/decompositions.
Corollary 1.
Suppose you have (cid:81) i d i samples of N (0 , 1)
z , then we have that vec
[Equation]
[Equation]
i
[Equation]
[Equation]
×
[Equation]
[Equation]
×
[Equation]
KS
[Equation]
{ V i Λ
[Equation]
Ti }
[Equation]
Proof.
[Equation]
Lemma 2.
[Equation]
Proof.
Let d 1: n = (cid:81) nℓ =1 d ℓ .
Defining d 1:0 as 1, we can observe that:
[Equation]
Theorem 2.
We can obtain the eigenvalues from the variances via a linear system.
Proof.
It is easy to see that the approximation of Lemma 2 defines a linear system of the form a = BΛ , where a is a vector whose elements are
[Equation]
and Λ is a vector made from stacking the eigenvalues of each axis.
Since the RHS of Lemma 2 is a linear combination of eigenvalues, it is easy to construct a matrix B relating the two.
It is not invertible: the eigenvalues are underdetermined.
However, we can select the least squares solution by multiplyling both sides by the pseudo-inverse of B : B † a ≈ Λ .
The matrix B will be much larger than necessary - in Section A.8 we will see how to reduce its size.
The exact form of B is easy to understand: the first d 1 columns will represent the eigenvectors of axis 1, the next d 2 columns will represent the eigenvectors of axis 2, and so on.
Each row will contain exactly one 1 in the first d 1 columns, exactly one 1 in the next d 2 columns, and so on.
The rest is zeros.
The matrix contains all possible rows under that restriction.
The exact ordering of the rows of the matrix will depend on how you have vectorized your tensor.
A.6 Regularization
The original BiGLasso performs lasso independently on each row of the precision matrices at each iteration.
The natural analogy in this case would be to perform row-wise lasso at the end of antGLasso.
The function to minimize is, for a row ˆ r , f ( r ) =
( r − ˆ r ) T ( r − ˆ r )
+ β
|| r
|| 1 .
If r and ˆ r were restricted to be nonnegative, then this would be differentiable.
Suppose for now that that is the case.
[Equation]
Since the domain of ˆ r
i is nonnegative, and the function is monotonic, if ˆ r i − β 2 < 0 then the minimum on the domain occurs at r i = 0 .
We can easily enforce a nonnegative domain by performing our regularization after taking the absolute value of the row.
This gives us a regularizer shrink(ˆ r ) =
sign(ˆ r ) ◦ min(0 , | ˆ r
| − β 2 ) that is equivalent to performing row-wise Lasso on the output of our algorithm.
Note that this allows us to reframe the regularization as a thresholding.
In fact, the threshold does not depend on the row, but is rather a global constraint.
Thus, instead of using β as an argument, we could give the algorithm a percent of edge connections to keep.
When framed this way, our hyperparameters become easily interpretable!
We found that when setting the threshold percent to be the true percent of connections in simulated data, the result had roughly equal precision and recall.
A.7 Heuristic
The vanilla variant of antGLasso performs much faster, but noticeably worse than, current state-ofthe-art BiGLasso algorithms.
This gap closes as the number of samples increases.
The part of the algorithm that appears most responsible for this error is the estimation of the variance of X , which is used to calculate the eigenvalues.
In small sample cases, such as having only one sample, our variance estimate will necessarily be quite poor.
Thus it would be beneficial to devise a heuristic to estimate these variances in a different way.
As a side effect, our heuristic will happen to be framed in terms of the empirical covariance matrices, making antGLasso compatible with the Nonparanormal
Skeptic method[11] which would allow us to generalize to non-Gaussian data à la Li et al.
Our goal is to find the variance of each element of X , or
in other words 5 var X
[ X i 1 ...
i K ] .
The idea of the heuristic is to fix one index, i n , and consider the remaining indices as random variables ⃗i \ n .
That is, we wish to find var X ,⃗i \ n [ X i 1 ...
i K ] .
[Equation]
[Table]
[Equation]
Sum notation of eigendecomposition)
[Equation]
Using this, we define our heuristic as follows.
[Equation]
The heuristic only depends on the first dimension’s eigenvectors, which means that the eigenvalues for the other dimensions won’t typically be in the same order as their corresponding eigenvectors.
To get around this, we run the calculation multiple times, once for each dimension.
A.8 Monte Carlo Speedup
As the speedup is technical, it is helpful to see a worked example for the (2, 2, 3, 2) tensor case.
We have BΛ = a
and we want to find a smaller matrix ˜ B which preserves the system.
The idea is to guess the value of one eigenvalue for each axis, as doing so will fix a unique solution for the rest of them.
A series of algebraic manipulations will make finding such a unique solution easy.
In doing so we only look at a small subset of B ( ˜ B ) and hence need to make several guesses and average out the result so that our solution is not overly affected by any individual guess.
We’ll choose ( λ 12 = 1 , λ 21 = 1 , λ 32 = 1 , λ 42 = 1) as our initial guesses for this worked example.
In fact, we do not need to make a guess for one of the axes (the rank of the matrix is K − 1 less than maximal, so we only need K − 1 guesses).
However it’s simpler to describe if we ignore this for now.
Note that if we simultaneously swap the i, j th columns of B and the i, j th rows of Λ , the system is preserved.
Likewise for the i, j th rows of B and the i, j th rows of a .
In light of that, we will express the system as:
B a Λ T and perform a series of row and column swaps to simplify it.
We’ll say that two systems of equations are equal if they can be arranged into each other through a series of permutations and removal of redundant 7 rows/columns.
B a Λ T =         a 1
a 2        a 3        a
[Table]
a
[Table] a [Table] a 7        a 8        a 9        a 10        a 11        a 12        a 13        a 14        a 15        a 16        a 17        a 18        a 19        a 20        a 21        a 22        a 23        aλλλλλλλλλ(43)
We want to re-order this system so that it behaves as if we had chosen ( λ 11 , λ 21 , λ 31 , λ 41 ) as our initial guesses (i.e. guessing the first eigenvalue of each axis).
Note that if we swapped the first and second columns, and then swapped every odd column with every even row, the matrix would look exactly the same but now the λ 1 we guessed is the first column rather than the second.
A similar line of reasoning applies to the other guesses we made, although instead of swapping individual rows we need to swap chunks of rows.
The size of the chunks would be 2 for the second guess, 4 for the third guess, andfor the fourth guess.
This is because we want to preserve the structure from the previous guesses - the size of the tensor is (2, 2, 3, 2) and hence the size of the chunks are (1 , 2 , 2 × 2 , 2 × 2 × 3) .
In general, the size of the chunks will be (1 , d 1 , (cid:81) 2 ℓ =1 d ℓ , (cid:81) 3 ℓ =1 d ℓ , ..., (cid:81) K − 1 ℓ =1 d ℓ ) .
After doing this, you’ll get a system looking like this:
[Table]
a 3        a 10        a 9        a 12        aλλλλλλλλλ(44)
Note that the B component did not change, as expected, but the order of a and Λ did.
We can then shrink this matrix by grabbing the first row, and every row that differs from it by the position of exactly one of the 1s.
B a Λ T =         a 18
a 17        a 20        a 14        a 22        aλ 1λ 1λ 2λ 2λ 3λ 3λ 3λ 4λ 4(45)
The system of equations is no longer overdetermined but is still a subset of the old system.
Our goal is now to reshape this into an upper triangular matrix.
The columns corresponding to the values we guessed are mostly 1 s, except for d ℓ − 1 zeros.
The other columns form a zero-padded identity matrix when taken together.
Move all of our guess columns to the end, and then the top row to the bottom:
B a Λ T =         a 17
a 20        a 14        a 22        a 6        aλλλλλλλλλ(46)
We can treat our guesses as a linear equation: if we add these equations to our matrix then it will become square:
B a Λ T =         a 17
a 20        a 14        a 22        a         a         λ         λ         λλλλλλλλλλ(47)
Thus, we never actually need to create B .
We can directly create ˜ B , and perform the demonstrated permutations on a and Λ .
As mentioned earlier, we don’t actually need a guess for the first dimension ( λ 1 ).
To solve the system for Λ , we need to find the inverse of ˜ B .
We’ve partitioned the matrix into four blocks (indicated by the dashed lines) - we can then invert this matrix using the block matrix inversion formula.
Because of the simple forms of the submatrices involved, this is a cheap operation. ˜ B − 1 =   1 0 0 0 0 0 1 10 1 0 0 0 1 0 10 0 1 0 0 1 1 00 0 0 1 0 1 1 00 0 0 0 1 1 1 10 0 0 0 0 1 1 10 0 0 0 0 0 1 00 0 0 0 0 0 0 10 0 0 0 0 0
0  −(48)
= (cid:20)
I
M
N (cid:21) −(49)
=
(cid:20) I − MN −
N −(cid:21)
[Equation]
The last step follows from the application of the block matrix inverse formula.
Note that we have reduced the computation of ˜ B − 1 to the inverse of a K × K matrix ( N ) and one matrix multiplication.
We can see that N will always have the form it does (zeros except for ones at the diagonal and first row).
The ones on the diagonal are because we added the guesses at the end, and the ones on the first row are because we had a 1 at the top of every ‘guess’ column.
It is not hard to see that the inverse of such a matrix will have 1 s on the diagonal, − 1 on the off-diagonal first row, and 0 s elsewhere.
Hence, we can just construct its inverse directly.
We are interested in constructing − MN − 1 directly too.
For a small number of iterations b , this is not a bottleneck - however, by computing it directly we can leverage sparse matrix multiplication routines in a manner such that we can easily run this computation a thousand times 8 before the computation time becomes noticeable.
We can compute this product easily using block matrix multiplication.
− MN − 1 = −   0 1 11 0 11 1 01 1 01 1 1    1 − 1 − 1 −            
 (51) =
−  
+ [1 1 1] (cid:34) (cid:35)
[ − 1 − 1 − 1] + [1 1 1] I  
 + 
 [ − 1 − 1 − 1] +   0 11 01 01 1
   (52)
= −   [1 1 1]
[Equation]
 
[Equation]
− J +   0 11 01 01 1
   (53)
[Equation]

[Equation]

[Equation]
1 00 10 10 0    (54)
In terms of generalizing this beyond the example, it is easy to check that the upper block of M should have d 1 − 1 rows (but it will be the same row, repeated).
This translates into − MN − 1 also having the upper block repeated d 1 − 1 times.
The lower right-hand block is always the same as in M , but with the 1s and 0s swapped: this is due to the outer product of a vector of 1s and a vector of -1s being − J , in line 52, regardless of the size of the inputs.
We have now reduced a very large matrix inversion and multiplication problem into a series of small, sparse matrix multiplications with a pre-computed matrix.
This pushes the runtime bottleneck of our algorithm solely onto the eigendecomposition used in Theorem 1.
The most expensive part of this reduction is the permutations we perform on a .
It’s not noticeable for 2-axis tensor inputs, but can be noticeable for 3-axis inputs.
Since after permuting a we select a subset of it, we can achieve further speedups by directly working out the indices of the desired elements of a , rather than permuting them.
For details on this final speedup, we refer the reader to our Python implementation of antGLasso.
A.9 Asymptotic Memory Complexity
Given an input tensor with d ℓ elements along the ℓ th axis, the output graphs are of size d 2 ℓ .
Thus it is not possible to have a better memory complexity than O ( (cid:80) ℓ d 2 ℓ ) .
Note that S ℓ , V ℓ have size d 2 ℓ .
The matrix ˜ B − 1 from the Monte Carlo speedup has size K 2 (where K = (cid:80) ℓ d ℓ ), but is sparse.
As seen in Section A.8, ˜ B − 1 can be divided into four blocks, with the top left block being zero, the top right block having O ( d 1 K ) ones, the bottom left block having O ( K ) zeros, and the bottom right block having O ( K ) ones.
We can always reshape the input so that the first index is the smallest, giving that O ( d 1 K )
⊂ O ( (cid:80) ℓ max( d ℓ , d 1 ))
=
O ( (cid:80) ℓ d 2 ℓ ) .
The cause of non-optimal memory complexity is the step in which we estimate the variances, as there are (cid:81) ℓ d ℓ variances to estimate.
This is the same as the size of the input.
Thus the vanilla variant of antGLasso is not necessarily memory-optimal in the sense of the amount of new space it takes up ( (cid:81) ℓ d ℓ ), but it is still bounded by the size of the input.
Heuristic antGLasso only estimates d ℓ variances at a time, and hence maintains optimality.
In the many-axis case, concerns over memory usage are not so important, as the size of the input grows exponentially in the number of dimensions.
Figure 8 shows a runtime comparison for antGLasso and TeraLasso.
In the three-axis case, antGLasso is still substantially faster than alternatives.