Learning More Effective Cell Representations Efficiently
Abstract
Capturing similarity among cells is the core of many tasks in single-cell transcriptomics, such as the identification of cell types and cell states.
This problem can beformulated in a paradigm called metric learning.
Metric learning aims to learn dataembeddings (feature vectors) in a way that reduces the distance between featurevectors corresponding to cells belonging to the same cell type and increases thedistance between the feature vectors corresponding to different cell types.
Deepmetric learning on the other hand uses neural networks to automatically learn dis- criminative features from the cells and then compute the metric.
The (deep) metriclearning approaches have been successfully applied to computational biology taskslike similar cell identification and synthesis of heterogeneous single-cell modalities.
We identify two computational challenges: precise distance measurement betweencells and scalability over a large amount of data in the applications of (deep) metric learning.
And then we propose our solutions: optimal transport and coresetoptimization.
Optimal transport has the potential to measure cell similarity more effectively and coreset optimization is promising to train representation learning models more efficiently.
Empirical studies in image retrieval and clustering tasks show the promise of the proposed approaches.
We propose to further explore the applicability of our methods to cell representation learning.
Introduction
The success of machine learning algorithms largely depends on data representation.
Metric learninglearns data embeddings and feature vectors in a way that reduces the distance between feature vectorscorresponding to objects belonging to the same class and increases the distance between the featurevectors corresponding to different classes.
Deep metric learning, on the other hand, uses neuralnetworks to automatically learn discriminative features from the objects and then compute the metric.(Deep) metric learning falls into the broad umbrella of representation learning, whose quest forrepresentation learning is motivating the design of more powerful representation [3, 10].
These representation learning methods have achieved great successes in biology applications [31, 33 ?, 15, 17 ? , 27, 40, 17, 37, 1].
For example, Schema [31] uses a principled metric learning strategy to identify informative features in a modality to synthesize disparate modalities into a single coherentinterpretation.
It is used to infer cell types by integrating gene expression and chromatin accessibilitydata.
The deep metric learning framework SCimilarity (Invited Talk “Design for Inference in Drug
Discovery and Development” by Aviv Regev, ICML 2022), which employs a standard triplet lossdesign, has achieved impressive performance in identifying similar cells in a massive collection ofscRNA-Seq datasets.
The results can help answer questions like in which tissues and diseases wefind fibrotic macrophage-like cells.[11] proposes the Deep Wasserstein Metric Learning Framework, as shown in Figure 1, whichconducts multiple steps of adjustments over the original metric learning framework and achievesimproved performance in image retrieval and clustering tasks.
In the rest of the proposal, we introducethe adjustments accordingly and design empirical studies for single-cell applications.
Methods
2.1 Representation Learning: Metric Learning and Deep Metric Learning
Representation learning is a class of machine learning approaches that allow a system to discover therepresentations required for feature detection or classification from raw data [14, 18, 13, 19, 9, 39]
The requirement for manual feature engineering is reduced by allowing a machine to learn the featuresand apply them to a given activity.
Metric learning and deep metric learning, specifically, focuson similarity-based approaches to learning the representations.
Thus the similarity measurementbecomes very important.
Metric learning has only limited capability to capture non-linearity inthe data, while deep metric learning captures non-linear features better by learning the non-lineartransformation.
In the following, we present two adjustments that can improve metric learning’sperformances, one is optimal transport [8], and one is coreset optimization [23].
2.2 Optimal Transport
The most widely used loss functions for deep metric learning are the contrastive loss and the tripletloss, both use euclidean distance to measure the distance between objects.
A more comprehensiveillustration of so-called “ranking-based” loss functions are summarized in Figure 2.
Optimal transport(OT) is the general problem of moving one distribution of mass to another as efficiently as possible[32, 11, 26, 36].
Optimal transport has been used tremendously in computational biology [30, 29,34, 35].
For example, [30] uses scRNA-seq data collected across a time course to infer how theseprobability distributions evolve over time, by using the mathematical approach of optimal transport.
Wasserstein distance provides a mathematical tool to measure distances between functions, histograms,or more general objects in the optimal transport problem.
Wasserstein distance is also called Earth
Mover’s Distance, which is employed to develop PhEMD (Phenotypic Earth Mover’s Distance)
[6],which is used to embed the space of drug perturbations on the basis of the drugs’ effects on cellpopulations.
Wasserstein distance-based loss functions have shown superior performance in learningtasks [16].
Thus a new set of loss functions are proposed to replace the Euclidean distance with
Wasserstein distance in original contrastive loss and triplet loss by defining d w ( x, y )
=
W 1 ( x, y ) .
The new Wasserstein-contrastive (wcontrastive) loss and Wasserstein-triplet (wtriplet) loss can be formulated as the following [11]:
[Equation]
We propose to apply the two new loss functions to similar cell identification and synthesis ofheterogeneous modalities applications.
2.3 Coreset Optimization
Coreset optimization is about data-efficient methods to find subsets of massive data that can generalizeto the full data when trained on.
In other words, a coreset is a subset of the original training set thatis representative to train machine learning models [11, 38, 24, 21].
More specifically, Wassersteinmeasure coreset [7], is an extension of coresets that takes into account continuous data distributionand generalization.
Recently coreset has been successfully applied to the purification of singlecell transcriptomics data [25].
It focuses on alleviating potential replicate specific biases withinsingle-cell datasets.
The key is to select a “representative” subset (coreset) of cells from areas of thesingle-cell landscape where multiple replicates are represented.
[31] processes data from a Slide-seqreplicate (three modalities with 20823 transcriptomes * 17607 genes) in 34 mins.
[23] demonstratesa specific coreset optimization algorithm CRAIG can achieve the average speedup of 3x for similarloss residual and error rate.
So we expect for the single cell synthesis task in [31], we can reduce dataprocessing time from 34 mins to 11 mins.
For future work, We refer the interested reader to [1, 2] forfeature-efficient methods that remove irrelevant variables during the training.
Experiments Design
The experiments conducted on various datasets have demonstrated that optimal transport and coresetoptimization can achieve superior performance on image retrieval and clustering tasks [11, 12].
Weplan to follow the setup in [22] to apply the methods on multiple single-cell datasets includingsci-CAR cell line [4], SNARE-seq cell line [5], and 10X Multiome T-cell depleted bone marrow [28]to validate the methods’ effectiveness and develop new computational and biological insights fromthe downstream tasks.