Representation Learning to Effectively Integrate and Interpret Omics Data
Abstract
The last decade has seen an increase in the amount of high throughput data available to researchers.
While this has allowed scientists to explore various hypotheses and research questions, it has also highlighted the importance of data integration to facilitate knowledge extraction and discovery.
Although many strategies have been developed over the last few years, integrating data whilst generating an interpretable embedding still remains challenging due to difﬁculty in regularisation, especially when using deep generative models.
Thus, we introduce a framework called Regularised Multi-View Variational Autoencoder (RMV-VAE) to integrate different omics data types whilst allowing researchers to obtain more biologically meaningful embeddings.
Introduction
In the last few decades, technological progress has yielded a large quantity of heterogeneous highthroughput data which has provided information about processes underpinning various diseases of interest.
This has allowed researchers to unveil molecular patterns in genomics, proteomics and lipidomics leading to a much more detailed understanding of the pathways and systems that regulate the immune response in humans.
The sudden increase in data highlighted, at the same time, the importance of integrating different types of molecular data together to enhance pattern discovery and improve patient stratiﬁcation.
Due to economic limitations, studies have historically focused on the use of uni-modal data, thus concentrating on a single aspect of the Dna-Rna-Protein paradigm.
While this approach has proven valuable to gain insights into various conditions, the lack of robust data integration has limited discoveries by failing to analyse biological systems as a whole.
This has been a limiting factor since, when looking at a particular data type, we are just analysing a snapshot of the condition of interest and, therefore, it can only provide us with limited insights on the complex processes underpinning a disease.
Being mostly unfeasible to extensively monitor omics changes over time, we can resort to the use of multiple data types to get a more complete picture of the pathways being modulated in the process [1].
In this paper, we consider different omics data types as different "views" of the same system [2, 3].
We aim to use a fusion approach to integrate these views together whilst obtaining an interpretable low dimensional data representation, or embedding, where points close in space share similar characteristics.
This is crucial when trying to use model’s embeddings for downstream tasks such as clustering or when we aim to obtain clinically meaningful groups. [Figure]
In the following sections we will explore related work and we will present a few experiments that support the use of our framework called Regularised Multi-View Variational Autoencoder (RMVVAE) on omics datasets.
Our approach is summarised in ﬁgure 1.
1.1 Related works
Data integration and dimensionality reduction are central aspects of many data analysis pipelines both in bioinformatics and other ﬁelds [4].
Among many models, Variational Autoencoders (VAEs) have been shown to be useful at solving a variety of problems such as data compression, de-noising, learning transferable representations of data and data fusion [5–7].
Nevertheless, a great limitation of using VAEs in biology has been the inability to obtain regularised, and hence interpretable, embeddings where points close in the original feature space are also close in the latent space, limiting their use in clinical practice [8, 9].
In the context of omics data integration, OmiEmbed and SubOmiEmbed have shown the potential of representation learning and self-representation learning at integrating data and performing downstream predictions in a uniﬁed framework [10, 11].
Other implementations, such as Multi-Encoder VAE applied to single cell data, have highlighted the importance of feeding different views of the same data to obtain more robust data representations and more separable classes [3].
In single cells RNA-Seq experiments, Graph Neural Networks (GNN) such as GLUE (graph-linked uniﬁed embedding) have also shown their potential at integrating datasets together by modeling regulatory interactions across omics layers explicitly [12].
Other models such as scVI have also been developed showing competitive results on single cell data [13].
While we acknowledge the performance and usefulness of GNNs in single cell RNA-Seq data, we recognise that adapting these models to bulk RNA-Seq data (used in these experiments) is a non-trivial task as it requires knowledge of the interaction between the different data-types to allow the construction of a meaningful graph.
Thus, our work takes inspiration from the aforementioned VAE research, expanding it on the generation of an interpretable, regularisable latent space.
In particular, we designed this model to allow researchers to not only perform downstream predictions, but also regularise the embeddings by a gene or protein known to be important in that condition, thus improving patient stratiﬁcation and treatment.
Proposed method
We propose a probabilistic framework that can be extended to N datasets allowing for dynamic integration of omics data.
The framework comprises two VAEs each composed of an encoder and a decoder.
Each VAE takes a dataset as input ( X ) and generates a low dimensional, or latent, representation of the data ( z ).
Once the data has been compressed, it is reconstructed by passing it through a decoder which is symmetrical to the encoder.
VAEs work by minimising the evidence lower bound (ELBO).
By using a reconstruction loss between the model’s input X and the output ˆ X and a KL divergence between the encoded data and a Normal distribution, the model is forced to learn the "real" signal present in the data, thus prioritising signal to noise 2.
In addition, the KL divergence forces the latent space to be more regularised than it would be just by using similar compression methods such as Autoencoders.
To encode an attribute a along one dimension r of the latent space ( z ) we used a regularisation loss introduced by Pati et al.
[14] where, as we traverse along r , the value of a increases.
The regularisation is achieved by adding an additional term to the loss of this model.
This allows users to generate embeddings that "order" patients by the expression of a speciﬁc variable of interest (eg.
BRCA1 gene for breast cancer) improving results’ interpretation.
The attribute loss is formulated as follows:
[Equation]
where MSE is the Mean Squared Error, D r is the distance matrix computed for the regularised dimension r , and D a is the distance matrix computed for the attribute we wish to encode along a dimension of z .
Tanh and sgn are the hyperbolic tangent function and sign function respectively.
This additional term is added to the standard Reconstruction and KL Loss 2 and forces a monotonic relationship between the encoded attribute and a dimension of z allowing users to obtain a more meaningful 2 and structured latent space.
[Equation]
2.1 Training overview
The models used in training were built using Tensorﬂow v. 2.9.2.
To ensure the generation of interpretable embeddings, we tested the size of the networks, as well as the hidden layers and the batch size manually, as in this particular setting optimising for performance would not necessarily imply an increase in the embedding interpretability, a problem that has been previously discussed in the literature [15].
We ﬁnally tested a range of nodes for the hidden layers [100, 80, 60, 40, 20, 10] to ﬁnd the most suitable architecture.
Model training was performed on CPUs. Average run time for the BRCA dataset (VAE/ RMV-VAE) was 30s and 28s respectively, while for the PAAD dataset was 11.8s (VAE) and 28.06s (RMV-VAE) respectively. You can ﬁnd our github repository here.  Experiments and results
We tested our framework on two diseases, breast cancer and pancreatic cancer.
Both experiments were performed using transcriptomics and methylomics datasets.
We assessed the ability of our framework to integrate different views of the same system 3 and then explored the interpretability of the latent space generated (see appendix for additional experiments). Transcriptomics Methylomics Data Integration + +
PC
PC
PC P C P C R M V - V
A E
V A E
ER + ER - ER + ER - ER + ERER + ERER + ERER + ER - 3.1 Experiment 1 - breast cancer datasets We retrieved transcriptomics and methylomics datasets from the UCSC Xena TCGA data repository containing 1212 and 885 patients respectively [16].
We preprocessed the data by only selecting the patients present in both datasets reaching a total number of 787 patients.
We preprocessed counts using EdgeR [17] and then scaled the datasets before model ﬁtting.
We started by looking at survival as outcome of interest.
To encode the transcriptomics, we used a twolayers Neural Network (NN) for both the encoder and the decoder while to encode the methylomics we used a three-layers NN.
Given the importance of the BRCA1 and TMEM101 genes in breast cancer survival, we chose them to regularise the latent space of each dataset respectively.
We then generated the ﬁnal embedding that we used for downstream predictions.
We evaluated the embedding on prediction tasks such as predicting survival and we compared the results to the ones obtained using individual datasets or a standard VAE.
As shown in table 1, the integration obtained by RMV-VAE outperformed the standard VAE as well as individual datasets in predicting survival using a Random
Forest Classiﬁer with ﬁve folds cross validation (mean accuracy 0.87 vs 0.85 e AUC of 0.62 vs 0.56).
We repeated the same experiment predicting Estrogen Receptor (ER) status.
In this case, we regularised the latent space by two genes known to play key roles in breast cancer’s ER status:
FOXA1 and AGR3.
We then generated the ﬁnal embedding that was used for predictions.
Results showed that our framework outperformed a standard VAE (using 5 folds CV, AUC of 0.91 vs 0.89 and mean accuracy of 0.88 vs 0.87) at generating a more organised, low dimensional data representation, as visible in table (see following section for more details) 1.
A comparison of the integration achieved by our framework and a standard VAE is shown in ﬁgure 2.
The ﬁgure shows that the data compression obtained by RMV-VAE allows to better separate
ER-positive from ER-negative patients in all datasets, therefore leading to an improvement in the ﬁnal embedding generated.
3.2 Experiment 2 - pancreatic cancer datasets
We applied RMV-VAE to the TCGA Pancreatic Adenocarcinoma datasets, consisting of 181 patients.
We used a three-layers NN for the transcriptomics and two-layers NN for the methylomics.
We preprocessed the datasets by removing methylation sites with low variance and scaled the data before [Table] [Table] proceeding at generating the individual embeddings.
The transcriptomics was preprocessed using standard bioinformatics pipelines (edgeR).
To obtain the individual embeddings, we normalised the latent space by KRAS and PTGDR, two driver genes that were shown to have diagnostic and prognostic value in pancreatic cancer [18, 19].
We then combined the embeddings and evaluated performance in predicting survival.
As shown in table 2, we obtained a better performance at predicting survival when using this framework compared to a standard VAE (AUC of 0.65 vs 0.57, mean accuracy of 0.62 vs 0.59).
Latent space interpretability
One of the key reasons to generate data embeddings using Variational Autoencoders is their ability to compress high dimensional data and obtain a low dimensional representation of it.
However, a problem that arises when using VAEs is the difﬁculty at generating ordered and interpretable latent spaces, as patients that are known to be clinically similar in the original input data are not necessarily close in the VAE embedding, slowing or limiting results’ interpretation.
As some genes might be known to play a crucial role in these conditions, we might want to not only compress the data, but also obtain a representation in which patients are ordered by a particular attribute to help relate possible embeddings’ clusters to already established clinical groups.
In the following section we will present some practical examples where this framework might be useful.
4.1 Clinical examples
Aggressive phenotypes of ER+ breast cancer are known to be driven by FOXA1 augmentation and expression which leads to activation of key mechanisms that promote metastatic programs [20].
Consequently, it might be useful to consider the expression of this gene when analysing breast cancer.
In ﬁgure 3, (A) , we show the embedding obtained by our model (RMV-VAE) and we compare it to the embedding obtained by a standard VAE and to the original expression data. For each embedding, we plotted the PCA to allow 2d comparisons between them. The panel shows the improvement in class separation (ER+/-) on the top row, and the improvement in regularisation on the bottom row. Results show that our framework allows us to obtain a more structured latent space where patients are ordered by the expression of the FOXA1 gene.
This leads to an improvement compared to the embedding of the original data and the standard VAE where patients with high and low expression were not easily separable.
Regularising the latent space by FOXA1 we achieve a much clearer separation between ER-negative patients, described in the literature as having low FOXA1 expression levels, and ER-positive patients that tend to display medium-to-high expression of this gene, improving overall data interpretation.
Similarly, the AGR3 gene is a gene known to regulate cell adhesion and migration in breast cancer, two key steps in tumour spreading and metastasis formation [21].
This gene is known to be highly methylated in ER-negative patients and it is therefore a good biomarker to identify these two clinically different groups.
Good identiﬁcation of ER-positive or negative patients would result in more adequate treatment choices and therefore higher chances of survival.
In ﬁgure 3 section (B) , we can compare the embedding obtained by our framework to the original expression data and to the embedding generated by a standard VAE.
The ﬁgure shows that using our framework we achieved a more organised latent space which results in more separable classes compared to the other embeddings.
The same patterns apply to genes such as BRCA1 and TMEM101 (not shown here).
To achieve similar results using a standard VAE, trial and error would be the only option and there would be no guarantee of achieving a good result.
Thus, we suggest the use of this framework to allow researchers to have more ﬂexibility in the regularisation of their embeddings.
Discussion
Data integration is a key aspect of biomedical research [1].
However, effective and meaningful data integration is still challenging to achieve due to technical limitations.
In this paper, we introduced a new framework, called RMV-VAE, to integrate different data types and obtain more meaningful, low dimensional representations of the data.
We tested our framework on two diseases, breast cancer and pancreatic cancer and we demonstrated the importance of integrating omics data to achieve higher performance at downstream tasks such as predicting ER status or survival.
We compared the results obtained using RMV-VAE to the results obtained using a standard VAE showing an increase in performance when using this framework.
In particular, we tested RMV-VAE at predicting survival and ER status for breast cancer, and survival for pancreatic cancer registering an increase in performance in all experiments.
The improvement in pancreatic cancer was striking given that this type of cancer is notoriously hard to classify due to the high heterogeneity of the human population.
In addition to integration, which is a key aspect of many data science pipelines, we tried to improve the latent space of VAEs to allow scientists and researchers to use them for downstream tasks.
To incorporate their expertise to the embeddings, we formulated an ah-hoc regularisation of the latent space to obtain embeddings where patients with similar expression of fundamental genes are found close together.
By doing so, we facilitated results interpretation allowing researchers to relate new results to already existent clinical groups.
Conclusion
Our framework provides a way to integrate and interpret omics data effectively allowing researchers to customise their analyses and explore genes or biomarkers that are known to play central roles in a given disease.
In this paper, we introduced this framework, we demonstrated its utility on two real life examples, and we discussed use cases in clinical practice. Acknowledgments and Disclosure of Funding
I would like to thank Lorenzo Sani for great discussions.
Sara Masarone is supported by The Alan
Turing Institute under the EPSRC grant
EP/N510129/
Benchmarking against MAUI
In these experiments, MAUI was set with a hidden layer/latent layer of 1300/600 (survival) and 1400/600 (ER) for the BRCA data and 1300/600 for the PAAD data.
The experiments were executed on Google Colab and the run time was on overage 16m.
[Table]
[Table]