A deep learning approach to recover conditional independence graphs
Abstract
Probabilistic Graphical Models are generative models of complex systems.
They rely on conditional independence assumptions between variables to learn sparse representations which can be visualized in a form of a graph.
Such models are used for domain exploration and structure discovery in poorly understood domains.
This work introduces a novel technique to perform sparse graph recovery by optimizing deep unrolled networks.
Assuming that the input data X ∈ R M × D comes from an underlying multivariate Gaussian distribution, we apply a deep model on X that outputs the precision matrix Θ .
Then, the partial correlation matrix P is calculated which can also be representated as the conditional independence graph.
Our model, uGLAD 1 , builds upon and extends the state-of-the-art model GLAD [43] to the unsupervised setting.
The key benefits of our model are (1) uGLAD automatically optimizes sparsity-related regularization parameters leading to better performance than existing algorithms.
(2) We introduce multi-task learning based ‘consensus’ strategy for robust handling of missing data in an unsupervised setting.
We evaluate performance on synthetic Gaussian data, non-Gaussian data generated from Gene
Regulatory Networks, and present a case study in anaerobic digestion.
Introduction
Probabilistic graphical models (PGMs) [30, 25] are generative models of complex systems, used to describe dependencies within a set of random variables and visualize the structure of a domain.
The models rely on conditional independence assumptions between variables, which result in sparse representation and enable efficient inference.
In the graphical representation of the models, conditional independence is indicated by an absence of an edge between two variables.
Such models can be learned from observational data [17, 12].
Structure discovery enabled by PGMs is important for new and poorly understood domains where relationships between variables are not known.
PGMs have been used in various domains, including medical diagnosis [19, 18], fault diagnosis, genomic data analysis via gene regulatory networks [27, 33, 1, 45], speech recognition, and in finance [16].
The problem of recovering the structure from observational data is particularly difficult in highdimensional settings, where the number of features may be larger than the number of observations.
We focus specifically on learning undirected models where the data is assumed to have been generated from a multivariate Gaussian distribution [12, 3, 52, 51, 43].
In such cases, the goal is to estimate a sparse inverse covariance matrix.
Sparsity is typically enforced by the use of ℓ 1 (lasso) regularization.
Assume we have a d -dimensional multivariate Gaussian random variable X =
[ X 1 , . .
, X d ] ⊤ with m observations.
The goal is to estimate its covariance matrix Σ ∗ and precision matrix Θ ∗ =
(Σ ∗ ) − 1 . Θ ∗ encodes conditional independence assumptions between variables: the ij -th component is zero if and only if X i and X j are conditionally independent given the other variables { X k } k ̸
= i,j .
This problem is known as the sparse graph recovery problem and usually formulated (following [12]), as the ℓ 1 -regularized maximum likelihood estimation
[Equation]
where (cid:98) Σ is the empirical covariance matrix based on m samples, S d ++ is the space of d × d symmetric positive definite matrices (SPD), and
∥ Θ ∥ 1 , off = (cid:80) i ̸
=
j | Θ ij | is the off-diagonal ℓ 1 regularizer with regularization parameter ρ .
The use of this estimator is justified even for non-Gaussian X , since it is minimizing an ℓ 1 -penalized log-determinant Bregman divergence [36].
The problem in Eq.
1 is a convex optimization problem.
It can be solved by many algorithms, see Section 2 for examples.
However, classic approaches have their limitations in both the statistical aspect and computational aspect.
Statistically, the classic formulation uses a single regularization parameter ρ for all entries in the precision matrix Θ , which may not be optimal.
A recent theoretical work [47] validates the use of adaptive parameters.
[43] has proposed a model with multiple regularization parameters called GLAD and has shown emprical evidence that such a model pushes the sample complexity limits.
Based on that evidence, we hypothesise that one may obtain better recovery results by allowing the regularization parameters to vary across the entries in the precision matrix.
However, it is hard for traditional approaches to search over a large number of hyperparameters.
Computationally, the complexity of solving the optimization depends on the convexity of the objective, the step sizes of the algorithm, the initialization, the design of the update steps, etc.
Different problems may require different designs of the algorithm to achieve a better efficiency.
A unified design for all problems may not be optimal.
In this work, we propose uGLAD (unsupervised- GLAD ), which is a deep model that can recover sparse graphs in an unsupervised manner.
As the name suggests, it builds upon and extends the GLAD model, which recovers sparse graphs under supervision.
uGLAD uses the same objective function as uGLAD .
Using an additional variable Z , the ℓ 1 -regularized maximum likelihood from Eq.
1 can be written as
[Equation]
Now, including the constraint as squared penalty term λ we obtain the reformulated objective as
[Equation]
Note that introducing the variable Z helps in splitting the objective into 2 parts and those can be optimized alternately using the Alternating Minimization algorithm.
A Conditional Independence (CI) graph can be obtained from this precision matrix by calculating the partial correlation matrix.
The CI graphs are very informative as they describe how the features in a domain interact with each other.
Furthermore, they can additionally capture negative partial correlations between features which are usually not modeled by traditional graphs.
this work:
• Extending GLAD to unsupervised setting : The uGLAD doesn’t rely on availability of ground truth to do graph recovery.
• Adaptive hyperparameters : The uGLAD architecture design enables the hyperparameters to optimally adapt at each step of the unrolled Alternating Minimization (AM) algorithm
[43] (also known as ‘deep unfolding’) that leads to its superior performance.
• Automatically decide optimum sparsity parameters : The sparsity of the recovered graph is highly sensitive to the choice of the regularization hyperparameters.
Instead, uGLAD models hyperparameters within the neural network framework and they are directly optimized for the uGLAD objective defined above.
So, there is no need to separately optimize the sparsity hyperparameters which is otherwise a computationally expensive process.
• Runtime efficiency : The uGLAD software can run on GPUs for higher time efficiency and scalability.
• Missing data handling : The uGLAD framework can also be used for multi-task learning.
We leverage this property further to develop a novel ‘consensus’ strategy to robustly handle missing data.
Related work
Traditional algorithms for graphical lasso : These are primarily iterative methods for optimizing the graphical lasso objective.
Main methods developed for this problem in the last two decades are detailed in the survey paper by Witten et al.
The variant of the Block Coordinate Descent (BCD) algorithm by [12] is widely used to recover graphs using the graphical lasso method.
This method is also implemented in the popular python ‘sklearn’ package.
The G-ISTA algorithm by [14] is based on the iterative shrinkage thresholding procedure; it is one of the prominent methods based on using the proximal gradient descent approach.
The Alternating Direction Method of Multipliers (ADMM) [9] has also been successfully used in various graphical lasso based applications.
Deep Learning approaches for graph recovery : DeepGraph (DG) [3], is a supervised deep learning method that takes in the input samples and outputs the corresponding adjacency matrix which shows the connections between input features.
DeepGraph architecture consists of many convolutional layers followed by multi-layer perceptrons that finally decides whether an edge is present between every combination of features.
Another relevant DL method roughly based on modeling the input data with a Variational Autoencoder for graph recovery is DAG-GNN [51].
These deep architectures have very high number of learnable parameters, which is a significant drawback.
Hence, we pursue a different line of research (using inductive biases) which gives similar performance with significantly reduced number of learnable parameters and brings more interpretability as shown in [43].
Deep learning models using inductive biases : Improved performance often results from including domain knowledge in the design/initialization of deep learning architectures.
For instance, [42] presents a generic technique to use a probabilistic graphical model as a prior to design a deep model.
The authors were able to show enhanced performance on the document classification task by leveraging the Latent Dirichlet Allocation prior.
Another way of including prior knowledge about the domain is using an optimization algorithm for a related objective function as a template to design the deep architecture.
Unrolling the optimization algorithms and parameterizing the step updates using neural networks have been fairly successful for many tasks [26, 8, 39, 34, 45].
This work focuses on recovering undirected graphs, specifically based on optimizing the graphical lasso objective function.
Hence, we skipped discussing the methods developed to recover Directed
Acyclic Graphs (DAGs).
The work most closely related to ours is the GLAD [43] model.
Since our algorithm builds upon GLAD ’s architecture we are going to describe it in detail in Section 3, while pointing out ways in which uGLAD differs from GLAD .
The uGLAD model
Given input data X ∈ R M × D , with M samples each having D features, the task is to recover a sparse graph showing correlations between the D features.
Recovering the sparse graph (the adjacency matrix) corresponds to obtaining the precision matrix Θ of the underlying multivariate Gaussian distribution.
3.1 Understanding the GLAD architecture
uGLAD uses the same architecture as GLAD [43] for the function Θ = f nn ( X ) .
The GLAD model uses the Alternating Minimization algorithm updates for the maximum likelihood objective, unrolled to some iterations and uses it as a template for its architecture.
Penalty constants ( ρ, λ ) are replaced by problem dependent neural networks, ρ nn and Λ nn .
These neural networks are minimalist in terms of the number of parameters as the input dimensions are mere { 3 , 2 } for { ρ nn , Λ nn } and outputs are a single value.
This unrolled algorithm with neural network augmentation can be viewed as a highly structured recurrent architecture as illustrated in Fig. Algorithm presented in Fig. 2 summarizes the update equations for the unrolled AM based model, GLAD .
The key features making the GLAD model a suitable choice are: (1) Minimalist learnable parameters (2) inherently maintain permutation invariance w.r.t.
the input (3) AM algorithm enforces positive definite constraint throughtout optimization (4) Interpretability and (5)
Linear convergence guarantee for faster runtime. For the sake of completeness of understanding the GLAD architecture, we graciously borrow the algorithm (see Alg.1) and neural network design details from [43] here.
3.2 The GLAD loss function
To learn the parameters in GLAD architecture, the authors used supervision in the form of true underlying graphs.
They leveraged the interpretable nature of the GLAD ’s deep architecture to define the loss for training.
Specifically, each iteration of the model will output a valid precision matrix estimation and this allowed them to add auxiliary losses to regularize the intermediate results of GLAD , guiding it to learn parameters which can generate a smooth solution trajectory.
Samples Λ  Norm Z
[Figure]
ˆ
[Figure]
Θ
[Figure]   ClosedForm Zk
[Figure] Θ  +
[Figure]
Thresholding Σ
ˆ Graphs
The authors used Frobenius norm in their loss function:
[Equation]
where (Θ ( i ) k , Z ( i ) k , λ ( i ) k ) =
GLADcell f
( (cid:98) Σ ( i ) , Θ ( i ) k − 1 , Z
( i ) k − 1 , λ ( i ) k − 1 ) is the output of the recurrent unit GLADcell at k -th iteration, K is number of unrolled iterations, γ ≤ 1 is a discounting factor and Θ ∗ is the ground truth precision matrix.
Then the stochastic gradient descent algorithm was used to train the parameters f in the GLADcell .
3.3 The uGLAD loss function
The GLAD model was developed as a supervised learning model, where the supervision is provided in terms of pairs of input data
X and the ground truth precision matrices Θ ∗ .
The parameters were trained by optimizing the mean square error between the predicted and the true precision matrices (see Sec.
3.2 above). In our model uGLAD (unsupervised GLAD ), we extend the GLAD model to the setting where the ground truth is unavailable. Note that this is a much more realistic scenario – in most real-world domains the data generating distribution is not known. In contrast to GLAD , uGLAD cannot take advantage of the underlying true generating distribution in the form of the true covariance matrix.
Thus, we need to replace GLAD ’s MSE loss from Equation 3 with a new loss.
Given a function Θ = f nn ( X ) , we optimize the log likelihood function given by
[Equation]
where S = cov( X ) is the covariance matrix.
We take the function f nn as the GLAD model and substitute it in the uGLAD loss function.
Algorithm 2 gives the pseudo code for learning the uGLAD model for doing sparse graph recovery.
3.4 Convergence properties of GLAD and uGLAD
The GLAD paper [43] evaluates convergence properties for the GLAD algorithm using normalized mean square error (NMSE) and probability of success (PS) to evaluate the algorithm performance.
NMSE is log 10 ( E ∥
Θ p − Θ ∗ ∥ 2 F / E ∥
Θ ∗ ∥ 2 F ) and PS is the probability of correct signed edge-set recovery, i.e., P (cid:2) sign(Θ pij ) =
sign(Θ ∗ ij ) , ∀ ( i, j
) ∈
E (Θ ∗ ) (cid:3) , where E (Θ ∗ ) is the true edge set.
Optimization objective always converges.
However, errors of recovering true precision matrices measured by
NMSE have very different behaviors given different regularity parameter ρ , which indicates the necessity of directly optimizing NMSE and hyperparameter tuning.
NMSE values are very sensitive to both ρ and the quadratic penalty λ of ADMM method.
In GLAD and uGLAD , ρ and λ are not fixed, but are optimized together with the rest of network parameters, leading to smooth convergence.
In experiments evaluating edge recovery success, GLAD consistently outperforms traditional methods in terms of sample complexity as it recovers the true edges with considerably fewer number of samples.
Since in uGLAD we are still using the AM minimization based GLAD architecture which is also based on optimizing the Eq. 1, we expect the linear convergence properties of the AM algorithm will hold for uGLAD as well.
The synthetic experiments in Sec.
6 show the results obtained from uGLAD are better than those for block coordinate descent based approach.
The CI graph shows the partial correlations between the input features.
A non-zero partial correlation betweenfeatures ( f A , f B ) indicates a direct dependence.
So, if all the other features are fixed, a positive partial correlation will indicate that increasing value of f A will increase the value of f B and vice-versa for the negative correlation.
To obtain this CI graph, we calculate the partial correlation matrix from the precision matrix.
Each entry of the partial correlation matrix P ij shows the correlation of the features x i , x j given the values of the other features are observed.
This helps us obtain the direct dependence of the features, P ij = − Θ ij √ Θ ii Θ jj .
We can then visualize the graphs and study the positive and negative correlations, with edge weights corresponding to correlation strengths.
Multi-task learning for precision matrix recovery
Most of the work in learning Gaussian graphical models has focused on estimating a single model.
In recent years, the framework was extended to jointly fitting a collection of such models, based on data that share the same variables, with dependency structure varying with some external category.
For example, in an NLP application, we can encounter different styles, which induce links between some concepts, even as the underlying grammar and semantics of the language stay the same. Algorithm
Input: Observations X ∈ R M × D
S = cov( X ) for e = 1 , · · · , E do ˆΘ e = GLAD ( S ) unrolled for L iterationsCompute loss L uGLAD ( S, ˆΘ e )
Backprop to update GLAD params end for return ˆΘ E
There have been extensive studies on the joint estimation of multiple undirected Gaussian graphical models [46, 20, 15, 6, 29, 9, 24, 28, 32, 50, 13, 48].
Most traditional algorithms construct a joint objective for multiple estimation tasks.
This objective typically incorporates similarities among various tasks by adding group norms or other regularizations.
However, in many practical problems, we only know that multiple tasks are related, without knowing how they are similar to each other quantitatively.
Manually constructing the joint objective may not best reflect the actual similarity.
In contrast to traditional algorithms, in uGLAD , we do not need to assume a specific similarity among different tasks.
Instead, we use a single network uGLAD to solve multiple tasks.
Since the parameters in uGLAD are shared across different tasks, the similarity among the tasks is automatically learned from data.
More specifically, given samples from K different models, X K
=
[ X 1 , X 2 , · · · , X K ] , we optimize the following objective
[Equation]
Alternatively, we can use the cross-validation split { X K train , X K val ← X K } for training.
Handling missing values
The missing data problem is ubiquitous in all data problems.
uGLAD can easily be extended to handle this problem.
If we observe that some specific feature columns have missing values and we have reasons to believe that these values are missing at random, we can run imputation algorithms for [Table] those columns to predict the missing entries.
Then, we will have the complete imputed input data
X imp ∈ R M × D over which we can run the uGLAD model and obtain the underlying precision matrix.
It can often happen that there are technical errors or human mistakes in collecting samples, which can often lead to missing values or noise seeping into the sample.
Also, we assume that the samples are independent and identically distributed (IID), so we cannot make use of the imputation techniques discussed in the case above.
For this case, we propose a novel multi-task learning technique based on utilizing the uGLAD ’s ability to optimize over a batch of input samples.
Consensus strategy: Multi-task learning over row-subsampled input
The key idea is to create a batch of row subsamples of the input data X ∈ R M × D .
Since all of these subsamples come from the same underlying distribution, we should ideally recover the same precision matrix for the entire batch.
Thus, if we have a model that can be jointly optimized over the entire batch for the uGLAD objective, resulting in the recovered precision matrix being robust against erroneous or noisy samples.
Steps for the multi-task learning approach to train the uGLAD model for a dataset with missing data:
Statistical imputation for the input : Replace all the missing entries of the input data X with their respective column mean (the mean is calculated ignoring the missing entries)
X [ i, c ] = mean
( X [: , c ]) .
Replacing by mean is usually a preferred approach as its contribution zeros out ( X − µ X ) while centering the data for the covariance matrix calculation.
Getting the batches : Perform stratified K-fold sampling to distribute the rows with missing values evenly among different batches.
Say, we have X K =
[ X 1 , X 2 , · · · , X K ] batches with each X k ∈ R M ∗ ( K − 1 K ) × D .
Thus, the batch input for the uGLAD model is X K ∈ R K × M ∗ ( K − 1 K ) × D . 3.
Optimizing uGLAD : It becomes a multi-task learning setting as we are jointly optimizing over a batch input X K .
The uGLAD model takes in the batch input X K and outputs the corresponding K precision matrices.
Since, the entire batch of data is coming from the same underlying distribution, we use the entire data X ∈ R M × D for the uGLAD loss to optimize the parameters of the uGLAD model.
Mathematically, we are minimizing the uGLAD loss over the batch as
[Equation]
Consensus among the batch to obtain the final precision matrix : After optimizing the uGLAD for the batch input, we will obtain K different precision matrices Θ K ∈ R K × D × D .
Ideally, all the precision matrices should be the same but there will be some discrepancies as we are working with missing values.
Our ‘consensus’ strategy to obtain the final precision matrix Θ f is to find the common edges with their correlation type (positive or negative) from the batch precision matrices.
Mathematically, we can obtain each entry [ i, j ] of the final precision matrix as
[Equation]
Here, the ‘max-count’ term determines whether the correlation among the batches for that entry is positive or negative.
The 2 nd term chooses the minimum absolute value for that entry among the batches as this facilitates sparsity and is conservative in terms of the strength of an edge.
[Table]
[Table]
The number of nodes D = 25 with varying samples M = [10 , 25 , 50] .
The BCD-avg considers each instance of the batch as a separate task and reports the average results over the batch. uGLAD -multi is used to recover jointly using a model
[Table] [Table]
Gene regulatory network data was used as described in Sec. 6.2. Increasing fraction of dropouts were introduced to the observed samples of the microarray expression data.
uGLAD -missing model is more robust for high dropout settings.
Such high dropout ratios are quite common in collecting samples for microarray gene expression data.
We believe that our model can be seamlessly integrated with existing pipelines for applications using other graphical lasso methods such as sklearn’s GraphicalLassoCV [31]. We are hopeful that it improves results over state-of-the-art methods.
Appendix C applies uGLAD to analyse real-world data collected for anaerobic digestion .
We use AUC (area under the ROC curve) and AUPR (area under the precision-recall curve) as primary evaluation metrics.
The sparsity of the graph leads to very few positive edges.
These two metrics account for such imbalance in the data and have the advantage of working without specifically setting a threshold for non-zero entries.
Their values reported in this work have the mean and the associated standard deviation values.
We primarily compare against the BCD algorithm.
It has been shown that the other traditional graphical lasso methods like ADMM and G-ISTA had performance similar to BCD [43].
We are not aware of any unsupervised deep learning approaches that optimize for the graphical lasso objective.
6.1 Performance on synthetically generated Gaussian samples
The synthetic data was generated based on the procedure similar to the one described in [14].
A d -dimensional precision matrix Θ was generated by initializing a d × d matrix with its off-diagonal entries sampled i.i.d.
from a uniform distribution Θ ij ∼ U ( − 1 , 1) .
These entries were then set to zero based on the sparsity pattern of the corresponding Erdos-Renyi random graph with a certain probability p .
Finally, an appropriate multiple of the identity matrix was added to the current matrix, so that the resulting matrix had the smallest eigenvalue of 1 .
In this way, Θ was ensured to be a well-conditioned, sparse and positive definite matrix and was used in the multivariate Gaussian distribution N (0 , Θ − 1 ) , to obtain M i.i.d samples.
Table 1 shows the results on this synthetic data.
6.2 Recovery of Gene Regulatory Networks
We conducted an exploratory study to gauge the generalization ability of uGLAD to non-Gaussian distributions.
We chose the GRN inference task for this purpose.
To this end, we use the SERGIO simulator [11] that provides a list of parameters to simulate cells from different types of biological processes and gene-expression levels with various amounts of intrinsic and technical noise.
We provide more details on data generation in Appendix B. Table 2 shows the comparison of the different graph recovery methods on the simulated data generated by SERGIO for cell types C = 5 and clean data setting.
6.3 Multi-task learning
This experiment verifies the ability of uGLAD to do multi-task learning.
We chose a collection of tasks as a set of data coming from graphs with varying sparsity.
For K different tasks, our input data is X ∈ R K × M × D .
We run the uGLAD model in multi-task learning mode as described in Sec.and recover K different precision matrices
Θ ∈ R K × D × D that are optimized for the loss function
L uGLAD -multitask ( X K ) given by equation 6.
Table 3 shows results from a single uGLAD model in multitask setting which is run on the synthetic data generated as described in Sec. 6.1.
uGLAD recovers multiple graphs with varying sparsity and shows promise for multitask learning problems.
6.4 Robustness testing for missing data
We artificially introduced missing values or ‘dropouts’ in the input data X ∈ R M × D to create noisy data.
Our aim is to study the effectiveness of the ‘consensus’ strategy discussed in Sec. 5.
We compare it with the baseline statistical imputation technique that does column-wise (or feature-wise) mean imputation as a preprocessing step.
BCD-mean and uGLAD -mean, report the results of running the corresponding methods on the column mean imputed data while the uGLAD -missing uses the ‘consensus’ strategy.
Tables 4 and 5 show the robustness of the ‘consensus’ strategy introduced in this work for synthetic Gaussian data as well as for the Gene regulatory networks recovery tasks.
Software details: Optimizing modes of uGLAD
Our software’s function signature is very much akin to the sklearn’s GraphicalLassoCV [31].
This was intended to make it easier for the users to try out our method with a minimal change to their existing code pipeline.
To optimize uGLAD , we have introduced 4 different modes of training.
In the software package, these modes can be switched from one to the other using an indicator flag.
Direct mode : The input to the uGLAD model is complete data X and the output precision matrix Θ = f nn ( X ) is optimized to reduce the uGLAD loss L uGLAD ( X ) , as defined in Eq. CV mode (recommended) : In the k-fold cross validation mode, we split the input samples X = ( X train , X valid ) .
We use the X train as input and optimize for the uGLAD loss L uGLAD ( X train ) .
Then, we select the best model that minimizes the L uGLAD ( X valid ) for the X valid samples.
Missing data mode : We give the entire data X as input with a ‘NaN’ indicator for the entries where the values are missing.
The software then follows the ‘consensus’ strategy for handling of missing data given in Sec.
5 and outputs the final precision matrix Θ f .
Multi-task mode :
Given a batch of input data X ∈ R K × M × D , we jointly optimize them for the uGLAD loss objective to obtain K different precision matrices Θ K ∈ R K × D × D , refer to Sec. 4.
Potential other applications of the uGLAD model
Listing some applications for which uGLAD can potentially improve the current state-of-the-art:
• Protein structure recovery: PSICOV [22] uses graphical lasso based approach to predict the contact matrix, which then eventually gives the 3D protein structure.
uGLAD can be substituted for predicting the contact matrix from the input correlation matrix between the amino acid sequences.
• Finance & Healthcare: Finding correlations between stocks to see how companies compare [16].
Systems for
studying important body vitals of ICU patients
, 5] and Infant Mortality analysis
• Class imbalance handling: We can potentially use uGLAD to find correlation between the features.
This correlation graph can be helpful in sampling down useful feature clusters.
This will help identify key features and in-turn improve performance in cases where there is less data or imbalanced data (
more
data points
for
one class over another).
Some of the
methods for class imbalance handling on which uGLAD model can act as a preprocessing steps are [35, 41, 4].
• Gaussian processes & time series problems: uGLAD can be extended to this interesting work by [7] on combining graphical lasso with Gaussian processes for learning gene regulatory networks.
Similarly, in a recent work on including negative data points for the Gaussian processes [38], uGLAD can be used for narrowing down the relevant features for doing the GP regression.
Our model can be used for time-series modeling, refer to [23] for an example.
• Video sequence predictions: uGLAD can be learned to narrow down potential future viable frames for generating unseen future video frames [10, 37].
Conclusions
We introduce a novel technique uGLAD to perform sparse graph recovery by optimizing a deep unrolled network for the graphical lasso objective.
This is an extension to the previous GLAD model that was designed to use supervision.
The key advantages of using our model over the state-of-the-art algorithms for the graphical lasso problems are (1) Sparsity related hyperparameters are modeled using neural networks which are automatically learned during the optimization.
We thus address the sensitivity issue of choosing the right sparsity parameters which is usually a tedious task and often manually set for the other algorithms.
(2) By design, neural networks of uGLAD enable the sparsity regularization to be adaptive over the iterations of the optimization leading to superior performance.
(3) Our software implementation supports GPU based acceleration and thus can be scaled efficiently to meet runtime requirements.
The uGLAD framework can do efficient multi-task learning.
The proposed ‘consensus’ strategy based on leveraging this property works well to robustly handle missing data.
As our experience with anaerobic digestion (see Appendix C) demonstrates, uGLAD can be successfully used as a tool for generating insight into growth dynamics of organisms in a digester and (hopefully) into domain structure in many other applications.
We hope that our model becomes one the widely used algorithms to solve the graphical lasso objective.
Appendix A Analysis of the uGLAD architecture A.1 uGLAD parameter details
We graciously borrow the algorithm (see Alg. 1) and neural network design details from the ‘ GLAD : Learning Sparse graph recovery’ paper.
GLAD parameter settings details are: ρ nn was a 4 layer neural network and Λ nn was a 2 layer neural network.
Both used 3 hidden units in each layer.
The nonlinearity used for hidden layers was tanh , while the final layer had sigmoid ( σ ) as the non-linearity for both, ρ nn and Λ nn (refer to Figure 2).
The learnable offset parameter of initial Θ 0 was set to t = 1 .
It was unrolled for L = 30 iterations.
The optimizer used was adam with the learning rates were chosen between [0 . 001 , 0 .
B Gene regulatory network recovery: additional experiment details
Additional details on the experiment done in Sec. 6.2.
For evaluation purposes in this work, we created random graphs (GRNs) that were used as input to SERGIO and will act as ground truth for evaluation.
First, we set the number of Transcription Factors (TFs) or master regulators.
Then, we randomly added edges between the TFs and the other genes based on sparsity requirements.
Also, we randomly added some edges between the TFs themselves but excluded any self-regulation edges and maintained connectivity of the graph.
When simulating data with no technical noise (clean data), we set the following parameters: samplingstate = 15 (determines the number of steps of simulations for each steady state); noise-param ∼ U [0 . 1 , 0 . 3] (controls the amount of intrinsic noise); noise-type = ‘dpd’ (the type of intrinsic noise is dual production decay noise, which is the most complex out of all types provided); we set genes decay parameter to 1 .
The parameters required to decide the master regulators’ basal production cell rate for all cell types: low expression range of production cell rate ∼ U [0 . 2 , 0 . 5] and high expression range of cell rate ∼ U [0 . 7 , 1] .
We chose K ∼ U [1 , 5] , where K denotes the maximum interaction strength between master regulators and target genes.
Positive strength values indicate activating interactions and negative strength values indicate repressive interactions and signs are randomly assigned.
C Case study: Anaerobic Digestion
Our algorithm development was inspired by a practical problem of domain exploration in anaerobic digestion.
Anaerobic digestion is a growing field addressing waste management with both environmental benefits (reduced odor and pathogens, improved soil health, reduction in methane emissions) and economic value from use of captured methane gas.
Despite numerous studies, the dynamics of organisms’ growth in digesters, their dependence on conditions (temperature, pH, feedstock mix, nitrogen to carbon ratio, etc.) and their impact on methane yield are not well understood.
We present findings based on a public dataset from a study of anaerobic digesters at Danish wastewater plants [21].
Data is available at NCBI under bioproject accession number PRJNA637463.
Data comes from a 6-year study of 46 digesters located at 22 Danish treatment plants.
We have three types of digesters, operating at different temperatures (mesophilic, mesophilic with thermal hydrolysis pretreatment, thermophilic).
Digesters operate continuously with sludge retention rate of 15.8 to 35.6 days.
Samples were taken at 3-month and 6-month intervals, so they can be treated as i.i.d.
We have a total of 1,010 sludge samples, 418 used to sequence archaea and 592 bacteria, performed using 16S rRNA gene amplicon sequencing.
Analysis resulted in identification of 33, bacterial and 878 archaeal unique amplicon sequence variants (ASVs).
70% of genera and 93% of the species were determined to be novel or previously unclassified.
This presents problems for all approaches attempting to utilize existing databases to determine organisms’ function for the purpose of grouping and feature selection.
In fact, one of the ways to determine an organism’s function is based on checking properties of organisms whose abundance numbers in the digester best correlate with the given organism’s numbers.
Our algorithm works with any input, including: ASVs filtered by frequency, ASVs rolled up to higher taxonomy levels (species, genus, family), ASVs abundance normalized in various ways [2].
We calculate the partial correlation matrix from the precision matrix.
Each entry of the partial correlation matrix P ij shows the correlation of the features x i , x j given the values of the other features are observed.
This helps us obtain the direct dependence of the features.
[Equation]
We use networkx package to visualize the graphs, presenting positive correlations in green and negative in red, with edge weights corresponding to the strength of the correlation.
Figures 3 and 4 present precision matrices recovered by our algorithm and BCD with empirical covariance shown for comparison.
Figures 5 and 6 show corresponding graphs for archaea at family and species level.
Figures 7 and 8 show a result of multitask learning based on digester type: thermophilic (operated at temperature 53.6 ◦ C) and mesophilic (operating at temperature 38 ◦ C).
The two graphs’ edges are filtered to show only edges common to both graphs, which is a small fraction of all edges.
Note that in some cases, the sign of the correlation (and the color of the edge) changes depending on digester’s type.
Our model is being used by domain experts to gain insight into the domain of anaerobic digestion.
One use case is to understand properties of newly discovered bacteria and archaea by analyzing which known organisms their abundance in digester samples correlates with (positively or negatively).
That can lead to focusing attention on a smaller organism set.
Another use case centers around understanding the role of digester conditions and feedstock mix on organisms’ growth and methane yield.
The results presented in recovered graphs lead to new hypotheses and new experiments being designed to test them.