Generating counterfactual explanations of tumor spatial proteomes to discover effective, combinatorial therapies that enhance cancer immunotherapy
Abstract
Recent advances in spatial omics methods enable the molecular composition of human tumors to be imaged at micron-scale resolution across hundreds of patients and ten to thousands of molecular imaging channels.
Large-scale molecular imaging datasets offer a new opportunity to understand how the spatial organization of proteins and cell types within a tumor modulate the response of a patient to different therapeutic strategies and offer potential insights into the design of novel therapies to increase patient response. However, spatial omics datasets require computational analysis methods that can scale to incorporate hundreds to thousands of imaging channels (ie colors) while enabling the extraction of molecular patterns that correlate with treatment responses across large number of patients with potentially heterogeneous tumors presentations.
Here, we have develop a machine learning strategy for the identification and design of signaling molecule combinations that predict the degree of immune system engagement with a specific patient tumors.
We specifically train a classifier to predict T cell distribution in patient tumors using the images from 30-40 molecular imaging channels.
Second, we apply a gradient descent based counterfactual reasoning strategy to the classifier and discover combinations of signaling molecules predicted to increase T cell infiltration. Applied to spatial proteomics data of melanoma tumor, our model predicts that increasing the level of CXCL9, CXCL10, CXCL12, CCL19 and decreasing the level of CCL8 in melanoma tumor will increase T cell infiltration by 10-fold across a cohort of 69 patients. The model predicts that the combination is many fold more effective than single target perturbations.
Our work provides a paradigm for machine learning based prediction and design of cancer therapeutics based on classification of immune system activity in spatial omics data.
Introduction
Recent advances in spatial proteomic and spatial genomics methods enable the molecular composition of a human tissues to be captured across hundreds to thousands of imaging channles at micron-scale resolution [1–4].
Spatial genomics and proteomic methods use a variety of strategies to bar-code molecules including protein and mRNA so that many different types of proteins and mRNA can be detected in a single sample at high-resolution.
Traditional tissue imaging methods like MRI, PET or H&E staining are either limited to gross tissue architecture, without providing molecular information, or are limited to a small number (1,2) of molecular channels. Therefore, conventional bioimaging data sets have one or two channels of information yielding one or two images per tissue region and are very similar to the conventional RGB images for which most ML methods from computer vision were developed [5].
Recent multiplexed imaging data contains orders-of-magnitude more channels per image due to new techniques such as imaging mass cytometry (IMC) [1] and sequential fluorescence in situ hybridization (seqFISH) [2], which enables simultaneous detection of proteins/mRNAs for 40/20,000+ genes in tissues at single micron resolution, respectively. These highly multiplexed data sets tends to yield thousands of images per tissue region where each image pixel provides information on the concentration of a specific molecular species (eg a protein abundance, an mRNA abundance) in a micron sized region of the tissue.
Spatial-omics and multiplexed imaging opens new opportunities for ML-based biomedical discovery.
Classically, very little is known about the molecular composition of human tissues in health and disease as classic pathology methods focus primarily on cellular morphology and gross markers like nucleus size and tissue boundary shape. Spatial-omics methods provide detailed molecular information on the signaling molecules and cell-types within a tumor, allowing molecular scale investigation of how different cell-types, for example tumor cells and immune cells, interact within a tumor and what specific signaling and regulatory processes might distinguish patients that respond to a specific type of therapy from non-responders [6].
Importantly, generating highly-multiplexed spatial data across hundreds of patients/conditions opens up the possibility of creating ML methods that may be powerful enough to discover new disease treatment strategies directly from tissue images, going beyond computer vision-based disease detection [7].
As an important specific example, cancer immunotherapy treats cancer by enabling the body’s own immune system to kill cancer cells, but many patients do not respond to immunotherapy and this is thought to be due to a lack of CD8+ T cell infiltration into the tumor core.
T cell infiltration is influenced by numerous factors including stromal structures and the spatial distribution of different cell populations and signaling molecules in the tumor microenvironment (TME). Existing strategies for improving CD8+ T cell infiltration focus on single-target perturbations and have shown limited efficacy.
In this work, we developed a machine learning strategy to discover molecular tumor perturbations predicted to drive T cell infiltration by identifying molecular determinants of T cell infiltration from large-scale spatial-omic datasets.
Specifically, our strategy involves training a classifier network to predict the T cell marker channels (representing T cell distribution) using other (non-marker) channels in IMC images. Next, we apply a gradient descent-based counterfactual reasoning procedure to the classifier network to find combinations of molecular perturbations that increase the predicted level of T cell infiltration.
We showed that convolutional neural networks (CNNs) can achieve over 70% precision and 60% recall in predicting T cell localization from IMC images of breast and melanoma tumors from human patients.
Furthermore, our model discovered a combinatorial therapy that is predicted to increase T cell infiltration in the majority of patients ( ∼ 30 ) with immune-excluded melanoma tumor from the dataset. This therapeutic strategy involves increasing the tumoral levels of chemokine CXCL9, CXCL10, CCL19, and CXCL12, and decreasing the level of CCL8.
The combinatorial strategy motivates the development of engineered proteins that achieve combinatorial function in a small number of molecular vehicles while being retained in the TME after administration.
Methods 2.1 Framework overview
We developed a machine learning-based optimization framework to discover molecular perturbations of tumors that can drive T cell infiltration by learning molecular features of IMC images of TMEs that are predictive of immune cell engagement.
Figure 2A shows an IMC image with d molecular channels, where we refer to a subset of the channels as marker channels and the rest as predictor channels.
The general logic of our framework is to train a classifier network to predict a specific aspect of the marker channels in the IMC images given the predictor channels.
In our example, we specifically use signaling molecules and chemokines (cell guidance molecules) to predict the abundance and state of T-cells within a local tumor region.
Then we apply a formal ‘counterfactual reasoning procedure’ to the input data where we perform gradient descent on the predictor channels for patients with low T-cell infiltration. The gradient descent procedure adjusts molecular channels pixel by pixel to discover changes to the tumor signaling environment that the classifier network predicts will increase T-cell infiltration, allowing us to discover perturbations to the predictor channels that alter the classifier’s prediction of the marker channels to a new, desired value.
Applying our framework to the problem of CD8+ T cell infiltration, we set marker channels to CD8+ T cell biomarkers.
We train a classifier to predict the presence of T cells within an IMC image patch given all other channels (predictor).
Then we use this classifier to search for changes to the predictor channels of tumor IMC images that can alter the classifier’s prediction from T cells being absent to T cells being present.
This altered image, known as a counterfactual instance in explainable AI [10], describes one perturbation of the TME that the model predicts will improve T cell infiltration.
2.2 Classifier training
We trained two classifiers, a multilayer perceptron (MLP) and a convolutional neural network (CNN), to predict the presence of non-exhausted CD8+ T cells from IMC images.
Figure 2B shows that we first divided IMC images into 40 × 40 µ m (40 × 40 pixels) patches.
To prepare each patch for network training, we removed four channels (CD3, CD8, Lag-3, PD-1) from which the label of each patch ("T cell" vs "no T cell") was determined, and we also removed signals from any channel that belong to the T cells whose distribution we are trying to predict (see [9] for detail).
This second step of removing signals belonging to T cells can be carried out without cell segmentation, through an approximate procedure where all signals within a 5 µ m radius of T cell marker signals are removed.
We then trained a CNN to compute the probability an IMC patch contains T cells given all remaining channels.
For the MLP model, instead of having individual pixel intensity as input features, we trained the model to perform the same prediction using only mean channel intensities x 0 ∈ R d averaged over the whole patch (Figure 2B).
Note the MLP is still taking advantage of spatial information in the data since each mean intensity vector characterizes a localized tissue region.
2.3 Perturbation optimization via counterfactual reasoning
Given an IMC patch i without T cells, and a classifier f as described above, our goal is to find a perturbation δ ( i ) for the patch such that f classifies the perturbed patch as having T cells (Figure 2C).
For CNN models, δ ( i ) ∈ R w × l × d is a 3D tensor that describes changes made for all d channel, at each pixel of the w × l pixel patch.
For MLP models, δ ( i ) ∈ R
d is a vector where each element represents changes to the average intensity of a channel in patch i .
This distinction between the two models highlights differences in the type of perturbations that each model can discover. The
MLP model can find tissue-wide perturbation such as changing the overall level of an extracellular molecule, whereas the CNN model allows for cell-type specific perturbations (i.e. cell type-specific viral vectors) since perturbations are now being computed at the level of micron-scale pixels.
For simplicity, we focus on the MLP model for optimization.
Given a MLP classifier f
and a
IMC patch i having mean channel intensity x
( i )
such that f ( x ( i ) 0 ) =
P (T cells present) < p , where p > 0 is the classification threshold below which the classifier predicts no T-cell, we aim to obtain a perturbation δ ( i ) such that f ( x ( i )
+ δ ( i ) ) > p , by solving the following optimization problem adopted from [11],
[Equation]
such that
[Equation]
where proto is an instance of the training set classified as having T cells, defined by first building a k-d tree of training instances classified as having T cells and setting the k -nearest item in the tree (in terms of euclidean distance to x ( i ) 0 ) as proto .
The first loss term in Equation 1 corresponds to an efficacy criteria whereas the second and third loss term represent key feasibility criteria that enables us to discover perturbations that are clinically feasible:
L pred encourages f ( x ( i )
+ δ ) to become greater than p , meaning we want a δ that makes the model predict that the perturbed patch has T cells.
L dist minimizes the distance between the original instance x ( i ) 0 and the counterfactual instance x ( i ) cf = x ( i )
+ δ with elastic net regularization to generate sparse perturbations, in other words we favor perturbations that require making as few changes as possible since perturbations are often difficult to make in a real-world clinical setting.
L proto explicitly guides the perturbation towards a counterfactual instance x cf which falls in the distribution of the class of IMC patches containing T cells.
This loss term helps to generate counterfactual instances that are similar to what has already been observed in IMC image of immune-inflamed tumors, making the predicted improvement in T cell infiltration more likely to hold in practice.
Figure 2D illustrates how the optimization algorithm selects the “best” perturbation through a simple schematic.
Given the original instance x 0 which lacks T cells, the algorithm obtains x cf by moving for the shortest distance possible, in the direction of the nearest training instance that contains T cells.
Our strategy above may find different perturbations for different patches of a tumor, while in practice it is more feasible to apply the same perturbations to the whole tumor.
We obtain a tumor-wide perturbation by taking the median across the set of patch-wise perturbations { δ ( i ) } i .
Results 3.1 Datasets
We applied our optimization framework to two published IMC datasets of tumors from melanoma and breast cancer patients [9, 12].
The melanoma dataset contains IMC images of 159 tumor cores from 69 patients.
Each core was approximately 1 mm in diameter, imaged at 1 µ m resolution across 41 channels, corresponding to different proteins and mRNA measured [9].
The breast cancer dataset contains IMC images of 749 breast tumor cores from 693 patients [12].
Most tissue images (93%) were 0 .
6 mm in diameter, has 1um resolution across 37 channels.
3.2 Classifier performance
Across IMC images of both melanoma and breast tumor, our simple MLP model was able to effectively predicted T cell distribution.
Figure 3A shows some example predictions made by the trained MLP model, which was applied to every patch of different tumor cores.
The model was able to capture the general distribution of T cells quite well.
In addition, Figure 3B shows that our MLP model is able to accurately predict the proportion of patches in a tumor image that contains T cells, which is especially important given this metric is directly associated with the infiltration status of a tumor.
Images with low proportion of tumor patches containing T cell correspond to immune-excluded tumors and images with high proportion of tumor patches containing T cells correspond to immune-inflamed tumors.
Across all classification metrics, Table 1 shows CNN models outperform MLP models, suggesting that even at ∼ 40um scale (size of patch), there may be meaningful spatial structures that influence T cell localization.
This performance difference between models is especially evident for breast tumors in comparison to melanoma tumors.
Lastly, the precision of a classifier is especially important for our application because we want to be confident that when a perturbed patch is predicted to have T cells, that it is indeed true.
By setting the classification threshold p = 0 .
8 , we obtain high precision across all four models, so we set p = 0 .
8 for Equation 2.
Confidence intervals in Table 1 obtained by retraining each model 8 times with different, randomized train-validation-test split and network initialization.
[Table]
3.3 Perturbation performance
Applying our counterfactual reasoning procedure using the MLP classifier trained on melanoma IMC images [Table] we discovered a combinatorial therapy predicted to be highly effective in improving
T cell infiltration in melanoma patients. For simplicity, we restricted the optimization procedure to only perturb the level of chemokines, which is a family of secreted proteins that are known for their ability to stimulate cell migration [13].
Figure 4A shows perturbations computed on IMC patches of melanoma tumors.
Across all patches, the levels of CXCL9, CXCL10, CXCL12, and CCL are consistently raised and CCL8 levels are consistently reduced.
Indeed, these five perturbations are all that remains after taking the median across all perturbations (bar graph).
Figure 4B shows that after applying these five perturbations to the IMC image of a tumor, T cells infiltration level (roughly corresponds to the percent of tumor cells with T cells nearby) is predicted to increase byfold.
Furthermore, Figure 4C shows that this predicted improvement in T cell infiltration holds across nearly all 34 patients with immune-excluded tumors.
The combinatorial nature of our perturbation strategy is crucial to its effectiveness.
Figure 4D shows that when each of the five perturbations are applied in isolation, the median T cell infiltration level across all tumors does not undergo significant increase.
Furthermore we systematically explore the importance of combinatorial perturbation by changing parameters β of Equation 3 which adjusts the sparsity of the solution, where sparser solution means less molecules are perturbed.
Figure 4E shows that although the model predicts the optimal single-target perturbation is not effective, perturbing four targets drastically increases T cell infiltration.
Furthermore, perturbing more than five targets leads to diminishing returns in terms of the improvement in T cell infiltration.
In conclusion, combinatorial perturbation of the TME appears necessary for improving T cell infiltration, at least within the scope of the chemokine targets considered
Discussion
Our optimization framework combines deep neural networks with optimization methods from explainable AI to directly predict therapeutic interventions.
Our predictions can be tested and the results fed back to update the model.
One of the major strength of our model is that it can readily scale to deal with larger datasets and neural network models, which will be very important as more spatial transcriptomics and proteomics datasets are quickly becoming available [14].
For future work, on the computational side, we plan to incorporate cell-type specific perturbations into our framework, which can be done by performing optimization on neural network architectures such as CNNs and vision transformers.
On the experimental side, we plan to test our therapeutic predictions experimentally in murine models of melanoma.
A Appendix A.1 Classifier training information
Our work is implemented using Keras with Tensorflow v2.10 [15] as backend.
Classifier training was conducted on NVIDIA GeForce RTX 3090 Ti GPU with 24GB RAM workbench.
The metric MCC (matthew correlation coefficient) was used to decide when to stop training.
The same MLP architecture was trained on both the breast and melanoma dataset, consisting of two hidden layers of 30 and 10 nodes, respectively.
The same CNN architecture was trained on both the breast and melanoma dataset, consisting of a convolutional layer, followed by max pooling, and another convolutional layer, a dropout layer, and finally two dense layers.
The output of both the MLP and
CNN represent the probability of T cells present.
A.2 Hyperparameters
For perturbation optimization [Table] we used the following set of values for hyperparameters from Equation[-
-]
[Table]
A.3 Classifier performance
[Table]