Generating counterfactual explanations of tumor spatial proteomes to discover effective, combinatorial therapies that enhance cancer immunotherapy
Abstract
Recent advances in tissue imaging technologies have led to the generation of massive datasets of spatial profiles of human tissues, taken at micron scale resolution,spanning hundreds of patients, and across tens to thousands of molecular biomarkers.
These high-dimensional imaging data necessitate the development of new,
AI-based tools to uncover new biology and to support therapeutic developmentsagainst diseases.
Currently, a major need in the treatment of solid tumor cancers arestrategies that can drive the infiltration of T cells into the tumor.
In this study, wedeveloped an optimization strategy combining supervised ML and counterfactualexplanations to discover clinically feasible tumor perturbations that drive T cellinfiltration, and applied our framework to spatial proteomes of breast cancer andmelanoma tissue.
Our model predicts that altering the levels of four molecules(CCL4, CXCL12, CXCL13, CCL8) in immune-excluded melanoma tissues canincrease T cell infiltration by 10 fold, across an entire cohort of 69 patients.
Ourwork provides a paradigm for machine learning based prediction and design ofcancer therapeutics based on classification of immune system activity in spatialproteomics data.
Introduction
Recent advances in bioimaging technology allow us to capture the spatial organization of cellularactivity in tissue with unprecedented molecular detail [1–4].
Traditional bioimaging data weremostly in grayscale or contained only 1-2 color channels, corresponding to 1-2 different molecularbiomarkers, similar to conventional RGB images for which most ML methods from computervision were developed.
With limited signals, bioimaging applications of ML focused mainly onobject detection such as cell segmentation [5].
Recent multiplexed imaging data contains orders-ofmagnitude more channels per image due to new techniques such as imaging mass cytometry (IMC)[1] and sequential fluorescence in situ hybridization (seqFISH) [2], which enables simultaneousdetection of proteins/mRNAs for 40/20,000+ genes in tissues at single micron resolution, respectively.
Spatial omics and multiplexed imaging opens new opportunities for ML-based biomedical discovery.
The spatial organization of cell populations within tissues fundamentally shapes the health of ourbodies by determining how cells interact [6], akin to how physical proximity between people shapesour society by structuring how we interact with one another.
Although the cell interaction networksembedded in spatial omics data is likely a key determinant of health, we need new computationalmethods to interpret these high-dimensional spatial patterns in terms of health and disease [7].
Impor- tantly, generating highly-multiplexed spatial data across hundreds of patients/conditions opens up thepossibility of creating ML methods that may be powerful enough to discover new disease treatmentstrategies directly from tissue images, going beyond computer vision-based disease detection [8].
As large amounts of spatials proteomics data on human tumors are becoming widely available [9, 10],
ML-based methods may be applied to improve cancer immunotherapy .
Immunotherapy treats cancerby enabling the body’s own immune system to kill cancer cells, but many patients do not respondto immunotherapy and this is thought to be due to a lack of T cell infiltration into the tumor
core(Figure 1) [11].
Although many factors in the tumor microenvironment (TME) affect T cell infiltration,such as vasculature structure, anti-inflammatory cytokines and chemokines, existing strategies focuson single-target perturbations and have shown limited efficacy [12].
Perturbing multiple targets islikely more effective, but testing all possible combinations of molecules to perturb is experimentallyinfeasible.
Existing IMC data contains spatial profiles (Figure 1) of both immune-inflamed (infiltratedby T cells) and immune-excluded tumors (T cells excluded from tumor core)[9, 10], which enablesthe training of ML models to predict T cell infiltration from IMC images and using the trained modelsas a substrate to search for perturbations that turn immune-excluded tumors immune-inflamed.
In this work, we apply supervised ML and counterfactual explanations to IMC images of breast andmelanoma tumors to predict tumor perturbations that support T cell infiltration.
Our strategy involvestraining a classifier to predict the presence of T cell markers using other (non-marker) channelsin IMC images, then using this classifier to find tissue perturbations that increase the predictedprobability of T cells.
We showed that convolutional neural networks (CNNs) can achieve over 80%precision and 70% recall in predicting T cell localization from IMC images of breast and melanomatumors.
Furthermore, our model predicts altering the level of CCL4, CXCL12, CXCL13 and CCL8in melanoma tumors can convert immune-excluded tumors in a cohort of 69 melanoma patientsto be immune-inflamed.
No single target perturbations were predicted to be effective.
Our workprovides a paradigm for machine learning based prediction and design of cancer therapeutics basedon classification of immune system activity in spatial proteomics data.
[Figure]
We propose an optimization framework that uses IMC images of TMEs to generate tumor perturbationstrategies that drive T cell infiltration.
Figure 2A shows an IMC image with d molecular channels,where we refer to a subset of the channels as marker channels and the rest as predictor channels.
Thegeneral logic of our framework is to train a classifier to predict aspects of the marker channels in the
IMC images given the predictor channels.
Then we apply counterfactual reasoning to the data byperforming gradient descent on the predictor channels, allowing us to discover perturbations to thepredictor channels that alter the classifier’s prediction of the marker channels to a new, desired value.
Applying our framework to the problem of T cell infiltration, we set marker channels to T cellbiomarkers.
We train a classifier to predict the presence of T cells within an IMC image patch givenall other channels (predictor).
Then we use this classifier to search for changes to the predictorchannels of tumor IMC images that can alter the classifier’s prediction from T cells being absent to Tcells being present.
This altered image, known as a counterfactual instance in explainable AI
[13],describes one perturbation of the TME that the model predicts will improve T cell infiltration.
2.2 Classifier training
We trained two classifiers, a MLP and a CNN, to predict the presence of T cells (non-exhausted,cytotoxic) from IMC images.
Figure 2B shows that we first divided IMC images into 64 × 64 µ m (64× 64 pixels) patches.
For each patch, we then removed four channels (CD3, CD8, Lag-3, PD-1) fromwhich the label of each patch ("T cell" vs "no T cell") was determined (see [9] for detail).
We traineda CNN to compute the probability an IMC patch contains T cells given all remaining channels.
Forthe MLP model, instead of having individual pixel intensity as input features, we trained the model toperform the same prediction using only mean channel intensities x 0 ∈ R d averaged over the wholepatch (Figure 2B).
Note the MLP is still taking advantage of spatial information in the data sinceeach mean intensity vector characterizes a localized tissue region.
2.3 Perturbation optimization
Given an IMC patch i without T cells, and a classifier f as described above, our goal is to find aperturbation δ ( i ) for the patch such that f classifies the perturbed patch as having T cells (Figure 2C).
For CNN models, δ ( i ) ∈ R w × l × d is a 3D tensor that describes changes made for every channel,at each pixel of the patch.
For MLP models, δ ( i ) ∈ R
d is a vector where each element representschanges to the average intensity of a channel in patch i .
This distinction between the two modelshighlights differences in the type of perturbations that each model can discover.
The MLP modelcan find region-wide perturbation such as changing the overall level of an extracellular molecule,whereas the CNN model allows for cell-type specific perturbations since perturbations are now beingcomputed at the level of micron-scale pixels.
For simplicity, we focus on the MLP model for optimization.
Given a MLP classifier f
and a
IMC patch i having mean channel intensity x
( i )
such that f ( x ( i ) 0 ) =
P (T cells present) <
p , wherep > 0 is the classification threshold below which the classifier predicts no T-cell, we aim to obtain aperturbation
δ ( i ) such that f ( x ( i )
+ δ ( i ) ) > p , by solving the following optimization problem adoptedfrom [14],
[Equation]
such that
[Equation]
where proto is an instance of the training set classified as having T cells, defined by first building ak-d tree of training instances classified as having T cells and setting the k -nearest item in the tree (interms of euclidean distance to x ( i ) 0 ) as proto .
Each of the loss term in Equation 1 corresponds to adesirable property of the perturbation δ :
L pred encourages f ( x ( i )
+ δ ) to become greater than p , meaning we want a δ that makes
the model predict that the perturbed patch has T cells.
L dist minimizes the distance between the original instance x ( i ) 0 and the counterfactual
instance x ( i ) cf = x ( i )
+ δ with elastic net regularization to generate sparse perturbations
L proto explicitly guides the perturbation towards a counterfactual instance x cf which fallsin the distribution of the class of IMC patches containing T cells.
This loss term helps to
generate counterfactual instances that are similar to what has already been observed in IMC
image of immune-inflamed tumors.
Figure 2D illustrates how the optimization algorithm selects the “best” perturbation through a simpleschematic.
Given the original instance x 0 which lacks T cells, the algorithm obtains x cf by movingfor the shortest distance possible, in the direction of the nearest training instance that is predicted tohave T cells.
Our strategy above may find different perturbations for different patches of a tumor,while in practice it is more feasible to apply the same perturbations to the whole tumor.
We obtain atumor-wide perturbation by taking the median across the set of patch-wise perturbations { δ ( i ) } i .
Results
3.1 Datasets
We applied our optimization framework to two publicly available IMC datasets of tumors frommelanoma and breast cancer patients [9, 10].
The melanoma dataset was obtained by IMC imagingof 159 tumor cores from 69 patients.
Each tissue was approximately 1 mm in diameter, imaged at1 µ m resolution across 41 channels, corresponding to different proteins measured [9].
The breastcancer dataset was obtained by IMC imaging of 749 breast tumor cores from 693 patients [10].
Mosttissues (93%) were 0 .
6 mm in diameter, image at 1um resolution across 37 channels.
3.2 Classifier performance
Across all classification metrics, Table 1 shows CNN models consistently outperform MLP models,suggesting that even at ∼ 60um scale (size of patch), there may be meaningful spatial structures thatinfluence T cell localization.
This performance difference between models is especially evident forbreast tumors in comparison to melanoma tumors.
Lastly, the precision of a classifier is especiallyimportant for our application because we want to be confident that when a perturbed patch is predictedto have T cells, that it is indeed true.
By setting the classification threshold p = 0 .
8 , we obtainhigh precision across all four models, so we set p = 0 .
8 for Equation 2.
Err in Table 1 obtainedby retraining each model 8 times with different, randomized train-validation-test split and networkinitialization.
the presence of T cells, 95% confidence interval shown for different train-validation-test split and initialization, (see Table 3 for p = 0 . 5 )
[Table]
3.3 Perturbation performance
Applying our optimization procedure using the MLP classifier trained on melanoma IMC images, wediscovered a combinatorial therapy predicted to be highly effective in improving T cell infiltration inmelanoma patients.
For simplicity, we restricted the optimization algorithm to only perturb the levelof chemokines, which is a family of secreted proteins that are known for their ability to stimulatecell migration [15].
Figure 3A shows perturbations computed on IMC patches of melanoma tumors.
Across all patches, average levels of CCL4, CXCL12, and CXCL13 are consistently raised and CCL8levels are consistently reduced.
Indeed, these four perturbations are all that remains after takingthe median across all perturbations (bar graph).
Interestingly, one of these perturbations, injectionof CCL4, have been tested in mouse melanoma [16].
Figure 3B shows that after applying thesefour perturbations to the IMC image of a tumor, T cells infiltration level (% of tumor cells with T cells nearby) is predicted to increase by 10 fold.
Furthermore, Figure 3C shows that this predictedimprovement in T cell infiltration holds across nearly all 34 patients with immune-excluded tumors.
The combinatorial nature of our perturbation strategy is crucial to its effectiveness.
Figure 3D showsthat when each of the four perturbations are applied in isolation, the T cell infiltration level averagedacross all tumors does not undergo significant increase.
Furthermore, we systematically explore theimportance of combinatorial perturbation by changing parameters β of Equation 3 which adjusts thesparsity of the solution, where sparser solution means less molecules are perturbed.
Figure 3E showsthat although the model predicts the optimal single-target perturbation is not effective, perturbing twotargets drastically increases T cell infiltration.
Furthermore, perturbing more than four targets leadsto diminishing returns in terms of the improvement in T cell infiltration.
In conclusion, combinatorialperturbation of the TME appears necessary for improving T cell infiltration, at least within the scopeof the chemokine targets considered.
Discussion
Our optimization framework combines deep neural networks with optimization methods from explainable AI to directly predict therapeutic interventions.
Our predictions can be tested and the results fed back to update the model.
One of the major strength of our model is that it can readilyscale to deal with larger datasets and neural network models, which will be very important as morespatial transcriptomics and proteomics datasets are quickly becoming available [17].
For future work,we plan to incorporate cell-type specific perturbations into our framework, which can be done byperforming optimization on neural network architectures such as CNNs and vision transformers
A Appendix
A.1 Classifier training information
Our work is implemented using Keras [18] with Tensorflow v2.10 [19] as backend.
Classifiertraining was conducted on NVIDIA GeForce RTX 3090 Ti GPU with 24GB RAM workbench.
Themetric MCC (matthew correlation coefficient) was used to decide when to stop training.
The same
MLP architecture was trained on both the breast and melanoma dataset, consisting of two hiddenlayers of 30 and 10 nodes, respectively.
The same CNN architecture was trained on both the breastand melanoma dataset, consisting of a convolutional layer, followed by max pooling, and anotherconvolutional layer, a dropout layer, and finally two dense layers.
The output of both the MLP and
CNN represent the probability of T cells present.
A.2 Hyperparameters
For perturbation optimization, we used the following set of values for hyperparameters from Equation
[Table]
[Table]