V ARIATIONAL I NTERPRETABLE D EEP C ANONICAL C ORRELATION A NALYSIS
A BSTRACT
The main idea of canonical correlation analysis (CCA) is to map different views onto a common latent space with maximum correlation.
We propose a deep interpretable variational canonical correlation analysis (DICCA) for multi-view learning.
The developed model extends the existing latent variable model for linear
CCA to nonlinear models through the use of deep generative networks.
DICCA is designed to disentangle both the shared and view
-specific variations for multi view data.
To further make the model more interpretable
, we place a sparsity- inducing prior on the latent weight with a structured variational autoencoder that is comprised of view
-specific generators.
Empirical results on real-world datasets show that our method is competitive across domains
I NTRODUCTION
Canonical correlation analysis (CCA) (Hotelling, 1936) is a popular two-view data analysis technique which extracts the common information between two multivariate random variables by projecting them into a space in which they are maximally correlated.
CCA has been used as a standard unsupervised two-view learning (Andrew et al., 2013; Wang et al., 2015), a cross-view classification (Chang et al., 2018; Chandar et al., 2016; Kan et al., 2016), a representation learning on multiple views for prediction (Sargin et al., 2007; Dorfer et al., 2016), and for a classification from a single view when a second view is available (Arora and Livescu, 2012).
When the data comes from several different views or modalities of the same underlying source of variation, all views jointly characterize the same phenomenon.
We can consider that there is shared information amongst them (or amongst subsets of them) by which they are all related.
Further, we also expect that there might exist unique or private variations in each view, i.e., information specific to an individual view.
A variety of extensions of CCA have been developed to learn a shared low-dimensional feature space of multi-view data, such as kernel-based extensions (Lai and Fyfe, 2000; Akaho, 2001; Bach and
Jordan, 2002).
The limitation of kernel-based methods is the computational cost for large datasets since kernel-CCA requires an N × N eigenvalue decomposition, where N is the sample size.
To capture the nonlinearity presented in complex data, deep neural network CCA (DCCA) was proposed (Andrew et al., 2013).
DCCA maximizes the cross correlation between the nonlinear projections from the outputs of two deep nonlinear neural networks of both views.
DCCA is further extended to deep CCA autoencoder (DCCAE)
(Wang et al., 2015) to improve the representation learning over
DCCA by leveraging autoencoders to additionally reconstruct the inputs through reconstruction error terms for the objective function.
While DCCA learns embeddings that capture shared variation, it does not explicitly model view-specific noise as in probabilistic CCA (PCCA) (Bach and Jordan, 2005).
PCCA is a probabilistic generative interpretation to the classical CCA.
Latent variable modelling has been used widely for providing interpretable descriptions of data.
A generative model is a popular approach to achieve a compact representation through exploiting dependency structures in the observed data.
The latent probabilistic version of CCA (Ghahramani, 2015) is attractive in medical applications where the data are typically of small sample sizes but large feature spaces.
However, the generative interpretation of CCA will ignore nonlinear structure in complex data such as images.
Lasso and group lasso are commonly used for simple interpretable models (Tibshirani, 1994; Yuan and Lin, 2006).
They work by shrinking many model parameters towards zero and have shown great success in regression models, covariance selection
(Danaher et al., 2014), linear factor analysis (Hirose and Konishi, 2012), and group factor analysis (Klami et al., 2015).
Commonly, sparsity-inducing penalties are considered in the convex optimization literature due to their computational tractability using proximal gradient descent.
Recently, two deep variational
CCAs (VCCA) were proposed (Tang et al., 2017; Wang et al., 2016) which yield a generative model with shared and view-specific factors.
However, these two VCCA models are only applicable to two-view data.
Recent high-throughput techniques, such as next-generation sequencing, have generated a wide variety of multiomics datasets that enable the identification of biological functions and mechanisms via multiple facets.
However, integrating these large-scale multiomics data and discovering interpretable insights are, nevertheless, challenging tasks.
To address this, previous work embedded biological knowledge into the machine learning model for underlying mechanisms; e.g., interpretable deep neural network modeling (Wang et al., 2018; Ma et al., 2018).
The model architecture of those methods relies heavily on the prior biological knowledge.
We present a deep interpretable CCA generative model (DICCA), in which the linear probabilistic layers are extended to deep generative multi-view networks.
DICCA captures the variations of the views by a shared latent representation that describes most of the variability of multi-view data, and a set of view-specific factors.
Our main contributions can be summarized as follows:
• We propose a novel generative framework for CCA which can disentangle the shared and view-specific variations from multi-view data.
• We leverage the sparsity-inducing hierarchical priors on the latent weight to achieve an interpretable model understanding.
• We evaluate our approach on real datasets to demonstrate that our algorithm achieves better performance compared to the state-of-the-art methods and have wide applications in integrating multiomics data for biomarker discovery.
M ETHODS
We now describe the proposed DICCA for multi-view data.
We assume that the m th view X m ∈ R
d m × 1 is independent with N co-occurring observations.
Z m ∈ R K × 1 denote the K -dimensional latent representation specific to the m th view for m ∈ { 1 , ..., M } , where M is the total number of views.
Z ∈ R K × 1 denote the K -dimensional latent representation common to all views.
Z is the shared latent variable capturing the shared variation across M views, while the view-specific latent variables Z m accounts for the view-specific variation.
We then write the generative process of the latent variables as:
[Equation]
View Generator
The variational autoencoder (VAE) (Kingma and Welling, 2014) propose the idea of amortized inference to perform variational inference in probabilistic models that are parameterized by deep neural networks.
The limitation for deep generative models and VAE is that the learned representations are not easily interpretable due to complex interaction from latent dimensions to the observations.
We consider view-specific generators and a linear latent-to-generator mapping with weights from a single latent dimension to a specific view.
We model each view with separate generative networks f ( m ) θ m parameterized by θ m .
We write the generative process of the data as:
[Equation]
where Λ m , W m ∈ R
d m × K .
Ψ m is a diagonal matrix containing the marginal variances of each component of X m .
The latent representation Z is shared over all the view-specific generators, Z m is view-specific.
One of the main goals of this framework is to capture interpretable relationships between view-specific activation through the latent representation.
Interpretable Sparsity Prior We refer to Λ m , W m as the latent-to-view matrices .
When the j th column of the latent-to-view matrix for view m , i.e., Λ ( m ) : ,j , W ( m ) : ,j is all zeros, then the j th latent dimension, z j , will have no influence on view m .
We place a hierarchical prior on the columns Λ ( m ) : ,j and W ( m ) : ,j to induce the column-wise sparsity as follows (Kyung et al., 2010):
[Equation]
where Gamma( · , · ) is defined by shape and rate and d m is the number of rows in each Λ ( m ) and
W ( m ) .
The amount of sparsity is defined by the rate parameter λ , with larger λ implying more column-wise sparsity in Λ ( m ) and W ( m ) .
γ 2 mj controls the column specific variation for the m th view.
Marginalizing over γ 2 mj induces view sparsity over the columns of Λ ( m ) and W ( m ) ; the maximum a posterior estimator of the resulting posterior is equivalent to a group lasso penalized objective.
Different from linear factor models, the deep structure of this framework encourages the model to learn a set of Λ ( m ) and W ( m ) matrices with very small weights only to have the values revived to “appropriate” magnitudes in the following layers of f ( m ) θ m .
A standard normal prior on the parameters of each generative network was placed: θ m ∼ N ( 0 , I ) .
Figure 1 provides a graphical illustration of our DICCA model under m = 2.
2. V ARIATIONAL I NFERENCE
One unique feature of VAE (Kingma and Welling, 2014) is that it allows the conditional p ( x | z ) being a potentially highly nonlinear mapping from z to x .
The likelihood is then parameterized with a generative network (called decoder).
VAE uses q ( z | x ) with an inference network (called encoder) to approximate the posterior distribution of z .
For example, q ( z | x ) can be a Gaussian N ( µ, σ 2 I ) , where both µ and σ 2 are parameterized by a neural network: [ µ, log σ 2 ]
=
f ϕ ( x ) , where f ϕ is a neural network with parameters ϕ .
The parameters for both generative and inference networks are learned through variational inference.
Jensen’s inequality yields the evidence lower bound (ELBO) on the marginal likelihood of the data:
[Equation]
where KL( Q ∥ P ) is Kullback-Leibler (KL) divergence between two distributions Q and P .
q ( z ; ϕ ) is a tractable “variational” distribution meant to approximate the intractable posterior distribution p ( z | x ) ; it is controlled by some parameters ϕ .
We want to choose ϕ that makes the bound in
Eq.
equation 4 as tight as possible.
One can train a feedforward inference network to find good variational parameters ϕ ( x ) for a given x , where ϕ ( x ) is the output of a neural network with parameters ϕ that are trained to maximize L ( x ; θ, ϕ ( x ))
(Kingma and Welling, 2014).
The KL divergence between the approximate posterior and the prior distribution of the latent variables regularizes the prior knowledge about the latent variable for the learning algorithm.
Let z m and z represent the view-specific and shared latent variables, respectively.
We can write the approximate posterior of the set of latent variables as q ϕ ( z | x ) (cid:81)
Mm =
1 q ϕ ( z m | x m ) .
Therefore, the KL divergence term can be decomposed as
[Equation]
2. L EARNING
Traditionally, variational inference is learned by applying stochastic gradient methods directly to the ELBO in equation 4.
We borrow the idea from oi-VAE (Ainsworth et al., 2018) which extends the basic amortized inference procedure to incorporate the sparsity inducing prior over the columns of the latent-to-view matrices.
We consider a collapsed variational objective function.
Since our sparsity inducing prior over W ( m ) · ,j is marginally equivalent to the convex group lasso penalty, we can use proximal gradient descent on the collapsed objective and obtain the true group sparsity (Parikh and Boyd, 2014).
Following the standard VAE approach of Kingma and Welling (2014), we use simple point estimates for the variational distributions on the neural network parameters
W = (cid:0) W (1) , · · · , W ( m ) (cid:1) , Λ = (cid:0) Λ (1) , · · · , Λ ( m ) (cid:1) , θ = ( θ 1 , . . .
, θ m ) , and We take q ϕ ( z | x ) =
N ( µ ( x ) , σ 2 ( x ))) , where the mean and variances are parameterized by an inference network with parameters ϕ .
The collapsed objective Under m ∈ { 1 , ..., M } , the data likelihood is defined by
[Equation]
We construct a collapsed variational objective function by marginalizing the γ 2 mj to compute log p θ ( x ) as:
[Equation]
To maximize this collapsed ELBO over { ϕ, θ, W , Λ } , we use efficient proximal gradient descent updates on the latent-to-view matrices W and Λ .
Proximal algorithms achieve better rates of convergence than sub-gradient methods and have shown great success in solving convex objectives with group lasso penalties.
We use Adam for the remaining neural net
R ELATED WORK
Deep variational CCA (DVCCA) (Wang et al., 2016) is for two-view data representation learning.
DVCCA shows that by modeling the view-specific variables that are specific to each view, DVCCA can disentangle shared and private variables and provide higher-qualify features and reconstructions.
The key model architecture difference between DVCCA and our method is that DVCCA assumes the shared variation only comes from one set of data, we disentangle the shared information from all the datasets.
Two-view deep probabilistic CCA (DPCCA) was proposed (Gundersen et al., 2020) based on PCCA (Bach and Jordan, 2005) and convolutional neural networks for paired medical images and gene expression levels.
The estimation of DPCCA requires PCCA to be estimated first, then the projection obtained from PCCA is passed to the convolutional neural network.
The requirement for both PCCA and neural network to be available is a major limitation for reconstruction and that is why they perform worse than multimodal autoencoder (MAE)
(Ngiam et al., 2011) in their reconstruction experiment.
Another limitation is that DPCCA is designed specifically for paired image data.
Interpretable VAEs for nonlinear group factor analysis (oi-VAE)
(Ainsworth et al., 2018) is the first generative nonlinear group latent factor model.
It combines deep generative models with a hierarchical sparsity-inducing prior that leads to the ability to extract meaningful interpretations of latent-toobserved interactions when the observations are structured into groups.
Deep latent variable model for learning longitudinal multi-view data (DLGFA) (Qiu et al., 2020) is an advanced temporal extension of oi-VAE, which can learn the dynamic dependency among groups through the shared latent variable and disentangle the interpretable dynamics among groups.
Both oi-VAE and DLGFA use multiple decoders over the same latent variables, with the goal of having interpretable factors for the multiview data.
oi-VAE and DLGFA are more interpretable compared to DVCCA, however, they do not model the common and view-specific variations like DVCCA which leaves a challenge for modeling complex multi-view data.
Variational inference for deep probabilistic canonical correlation analysis (VPCCA) (Karami and Schuurmans, 2021) can disentangle the shared and view-specific variations, however, VPCCA lacks model interpretation.
Additionally, the existing models’ interpretation is based on latent factor level not feature level which also poses a big application challenge for dealing with high-dimensional problems, like multi-omics data, where researchers are more interested in some particular features.
Our model is built on the variational generative model framework for efficient approximation purpose and we jointly model share and view-specific variations for complex multi-view data.
Most importantly, we place sparsity-inducing prior on the latent weights to achieve feature-level interpretability.
E XPERIMENTS
We empirically evaluate the representation learning and model interpretability of the proposed method.
The performances of the proposed method are evaluated over the following two datasets.
4. N OISY MNIST DATASET
Two-view noisy MNIST datasets from (Wang et al., 2015; 2016) are widely used for testing multi-view models.
This dataset is generated from the MNIST dataset, the first view of the dataset is generated by rotating each image at angles randomly sampled from uniform distribution U ( − π/ 4 , π/ 4) , while the second view is from randomly sampled images with the same identity to the first view but not necessarily the same image.
Then, it is corrupted by random uniform noise.
Thus, both views share the same identity of the digit, but they are not of the same style of the handwriting in the same class.
The original training set is split into training/tuning sets of size 50 K /10 K .
The data generation process ensures that the digit identity is the only common variable underlying both views.
The performance is measured on the 10 K images in the test set.
We tune the latent dimension K over [10,20,30,40,50], and fix K 1 = K 2 = 30 , where K 1 and K 2 represent the latent dimensions for the two views of the dataset.
We choose λ = 1 based on the mean squared error on the test dataset.
Disentanglement learning To evaluate the learned representation, our model should be able to reconstruct both views using the shared and view-specific latent variables.
As baseline, we fit oi-VAE (Ainsworth et al., 2018), DPCCA (Gundersen et al., 2020) and VCCA (Wang et al., 2016) to both
[Table]
data views.
VCCA-p represents VCCA-private which has view-specific latent variables in the model.
We find that DICCA can reconstruct both views well relative to these baselines (Table 1).
oi-VAE performs the worst since it does not model the view-specific variations.
DPCCA also does not perform well because it requires optimizing PCCA in an inner loop so that the parameters are not optimized with other variables in the neural network.
VCCA and VCCA-p perform better than oi-VAE and
DPCCA, but they are worse than DICCA which indicates that modeling view-specific variations is more powerful to extract the hidden truth than modeling only the common variations.
DICCA performs better in reconstruction than VCCA and VCCA-p.
This confirms that the view-generator structure of DICCA is more suitable for learning multi-view data rather than using a single encoder of VCCA and VCCA-P. Figure 2 shows sample reconstruction of noisy MNIST dataset by DICCA for view 1 (left) and view 2 (right).
latent variable between views 1 and 2.
We can see that DICCA can capture the styles of each image very well and it can separate the background noise from the view 2 images.
In addition, Figure 3 provides 2D t -SNE embedding of the view-specific latent representations from view 2 images learned by VCCA, VCCA-p, oiVAE, DICCA, and DICCA-share (the shared latent projections).
All the methods show improved separation performance compared to the original input data.
VCCA and VCCA-p perform similarly, but, there are some digits not very well separated, e.g., digit 3 and digit 2, digit 6 and digit 5.
We also observe that oi-VAE has similar problem.
This indicates that oi-VAE cannot capture the view-specific variations.
DICCA and DICCA-share perform surprisingly well which quantitatively verify that the learned features of the images of different classes are well separated by view-specific latent variables and the shared latent projections.
4. G ENETIC DATA
We now analyze a data set of 200 patients with chronic lymphocytic leukaemia (CLL)
(Dietrich et al., 2018).
This data combines ex vivo drug response measurements ( D = 310 ) with somatic mutation status ( D = 69 ), transcriptome profiling ( D = 5000 ) and DNA methylation assays ( D = 4248 ).
Thus, there are four measurements on the same patients ( N = 200 ).
We apply DICCA on CLL dataset to show model interpretability by exploring group dependency relationship and latent dimensions’ interpretation and annotation.
After tuning, we choose K = 10 .
Group dependency relationship Each latent dimension of z influences only a sparse subset of the observational groups.
We can view the observational groups associated with a specific latent dimension.
The latent weight matrix can give us a bipartite graph in which we can quickly identify correlation and independence relationships among the groups themselves.
This group dependency correlation among multi-view data is attractive as an exploratory tool independent of building a generative model.
In our case, we explore the shared and view-specific dependency respectively.
We compare the group dependency extracted by oi-VAE (Ainsworth et al., 2018), DICCA-share and
DICCA-private in Figure 4 a, b, and c.
Both oi-VAE and DICCA show that methylation and mRNA data have the dominant variations across the 10 latent dimensions.
As expected, we observe the view-specific variations are captured by DICCA-private.
For example, under the latent dimension 6, the dominant variation is explained by mRNA in oi-VAE which is the same as in DICCA-share.
However, in DICCA-private, the variations are explained by methylation.
Another example is under latent dimension 2, oi-VAE shows the dominant variations are from the mRNA, however, in DICCAshare, we observe that the shared dominant variations are from methylation, but, the DICCA-private shows the view-specific variations are from mRNA.
Latent dimension interpretation and biomarker discovery In Figure 4 d, we plot the variance explained by each view, DICCA explains 90%, 86%, 75% variations in drug, methylation, mRNA, respectively, and only 22% in mutations.
This is much higher compared to MOFA (Argelaguet et al., 2018).
Based on the top weights in mRNA data, factor 2 is aligned with SEPT5 which is a member of the septin gene family of nucleotide binding proteins.
Disruption of septin function can disturb cytokinesis and result in large multinucleate or polyploid cells (Elzamly et al., 2018).
Cancer-associated chromosomal changes often involve regions containing fragile sites.
Factor 10 is aligned with TES which maps to a common fragile site on chromosome 7q31.2 designated FRA7G.
TES is a negative regulator of cell growth and may act as a tumour suppressor gene that is inactivated primarily by transcriptional silencing resulting from CpG island methylation (Tobias et al., 2001).
D ISCUSSION
In this work, we develop a deep interpretable variational canonical correlation analysis for multi-view learning.
It has been shown that following the view-generator and group sparsity formulation of the linear latent CCA model, we can obtain an interpretable learning algorithm for multi-view data.
Empirical results have shown that this can efficiently disentangle the relationship among multiple views to obtain a more powerful representation.
Besides the outperformed representation learning achieved by jointly modeling the share and view-specific variations, the proposed method can also have better interpretations for the latent dimensions and potentially wide applications in multiomics biomarker discovery.
R A. S UPPLEMENTARY M ETHODS A.1. Proof of additive property of KL in Equation 5.
The approximate posterior of the set of latent variables can be factorized as
[Equation]
We assume independent prior distribution on the latent variables which leads to
[Equation]
A.2. Proximal gradient descent
A proximal algorithm is an algorithm for solving a convex optimization problem which uses the proximal operators of the objective terms.
Consider the problem
[Equation]
where f : R n → R and g : R n → R ∪ { + ∞} are closed proper convex and f is differentiable
[Equation]
The proximal gradient method is
[Equation]
(11) where λ k > 0 is a step size, prox f ( x ) is the proximal operator for the function f .
Expanding the definition of prox λ k g , we can show that the proximal step corresponds to minimizing g ( x ) plus a quadratic approximation to g ( x ) centered on x k .
For g ( x ) =
η || x
|| 2 , the proximal operator is given by
[Equation]
According to Parikh and Boyd, we know ( v ) + ∆ = max(0 , v ) .
This operator can reduce x by λη , and x can be shrank to zero under || x || 2 ≤ λ k η .
A.3. Latent dimension interpretation
After the model has been trained, the first step is to disentangle the variations in each view.
We compute the fraction of the variance explained ( R 2 ) per view by
[Equation]
Subsequently, each dimension is characterized by two complementary analyses:
• Ordination of the samples in factor space: Visualize a low dimensional representation of the main drivers of sample heterogeneity.
• Inspection of top features with largest weight:
The loadings can give insights into the biological process underlying the heterogeneity captured by a latent dimension.
We scale each weight vector by its absolute value.
B. M ODEL ARCHITECTURE AND TRAINING PROCEDURE
Selection on λ and k
The parameter λ controls the model sparsity, larger λ implies more column-wise sparsity in W ( m ) : ,j . We propose to select λ based on the learned W ( m ) : ,j to check the sparsity and the MSE[test].
The latent dimension k is chosen based on interpretation purpose.
B.1. Two-view noisy MNIST experiments
We have view-specific encoder for each view: Encoder 1 , Encoder 2 and shared encoder Encoder share .
After tuning, we use d 1 = d 2 = d share = 30.
• Encoder 1 : - µ ( x 1 )
=
W 1 relu ( x 1 ) + b 1 . - σ ( x 1 ) =
W 2 softplus ( x 1 ) + b 2 .
• Encoder 2 : - µ ( x 2 )
=
W 3 relu ( x 2 ) + b 3 . - σ ( x 2 ) =
W 4 softplus ( x 2 ) + b 4 .
• Encoder share : - µ ( x 1 + x 2 ) =
W 5 ( x 1 + x 2 ) + b 5 . - σ ( x1 + x 2 )
=
exp ( W 6 ( x 1 + x 2 ) + b 6 ) .
• Decoder: - µ ( z ) =
W 7 tanh ( z ) + b 7 . - σ ( z ) =
exp ( b 8 ) .
The learning rate on W is 1e-4 for encoder and decoder, batch size is 128.
Optimization was run for 1,000 epochs.
B.2. CLL experiments
We have view-specific encoder for each view: Encoder drug , Encoder methylation , Encoder mRNA ,
Encoder mutation and shared encoder Encoder share .
After tuning, we use d drug =
d methylation = d mRNA
= d mutation =
d share = 10.
• Encoder drug : - µ ( x 1 ) =
W 1 relu ( x 1 ) + b 1 . - σ ( x 1 ) =
W 2 softplus ( x 1 ) + b 2 .
• Encoder methylation : - µ ( x 2 ) =
W 3 relu ( x 2 ) + b 3 . - σ ( x 2 ) =
W 4 softplus ( x 2 ) + b 4 .
• Encoder mRNA : - µ ( x 3 ) =
W 5 relu ( x 3 ) + b 5 . -
σ ( x 3 )
=
W 6 softplus ( x 3 ) + b 6 .
• Encoder mutation : - µ ( x 4 ) =
W 7 relu ( x 4 ) + b 7 . - σ ( x 4 ) =
W 8 softplus ( x 4 ) + b 8 .
• Encoder share : - µ ( x 1 + x 2 + x 3 + x 4 )
=
W 9 ( x 1 + x
+ x 3 + x 4 )
+ b 9 . -
σ ( x 1 + x 2 + x 3 + x 4 )
= exp ( W 10 ( x 1 + x 2 + x 3 + x 4 ) + b 10 ) .
• Decoder: - µ ( z )
=
W 11 tanh ( z ) + b 11 . -
σ ( z ) =
exp ( b 12 ) .
The learning rate on W is 1e-4 for encoder and decoder, batch size is 12.
Optimization was run for 2,000 epochs.