Self-supervised learning improves dMMR/MSI detection from histology slides across multiple cancers
Editor: TBA
Abstract
Microsatellite instability (MSI) is a tumor phenotype whose diagnosis largely impacts patient care in colorectal cancers (CRC), and is associated with response to immunotherapy in all solid tumors.
Deep learning models detecting MSI tumors directly from H&E stained slides have shown promise in improving diagnosis of MSI patients.
Prior deep learning models for MSI detection have relied on neural networks pretrained on ImageNet dataset, which does not contain any medical image.
In this study, we leverage recent advances in self-supervised learning by training neural networks on histology images from the TCGA dataset using MoCo V2.
We show that these networks consistently outperform their counterparts pretrained using ImageNet and obtain state-of-the-art results for MSI detection with AUCs of 0.92 and 0.83 for CRC and gastric tumors, respectively.
These models generalize well on an external CRC cohort (0.97 AUC on PAIP) and improve transfer from one organ to another.
Finally we show that predictive image regions exhibit meaningful histological patterns, and that the use of MoCo features highlighted more relevant patterns according to an expert pathologist.
1. Introduction
Microsatellite Instability (MSI) is a frequent tumor phenotype characterized by an abnormal repetition of short DNA motifs caused by a deﬁciency of the DNA mismatch repair system (MMR).
MMR deﬁcient tumors (dMMR) result from defects in the major MMR genes, namely MLH1, MSH2, MSH6, PMS2 .
These defects arise either sporadically or as a hereditary condition named Lynch syndrome (LS), predisposing patients to develop cancers in several organs.
Recent studies have shown that immune checkpoint blockade therapy has a promising response in dMMR/MSI cancers regardless of the tissue of origin
[Le et al.
In 2017, this genomic instability phenotype became the ﬁrst pan-cancer biomarker approved by the
US FDA, allowing the use of pembrolizumab (Keytruda) for patients with dMMR/MSI solid tumors at the metastatic stage [Prasad et al.
As of today, systematic MSI screening is only recommended for colorectal cancer (CRC) and endometrial cancer [Svrcek et al.
(2019)] where the prevalence is relatively high (10% to 20%), principally to detect LS patients and provide them with adequate follow-up.
In early stages of CRC, MSI tumors are associated with good prognosis and resistance to chemotherapy [Sargent et al.
(2010)], making the diagnosis of this phenotype all the more essential for patient care and therapeutic decision.
dMMR/
MSI diagnosis is traditionally done using immunohistochemistry (IHC), polymerase chain reaction (PCR) assays, or next generation sequencing.
Those methods can be time-consuming, expensive, and rely on speciﬁc expertise which may not be available in every center.
Deep learning based MSI classiﬁers using H&E stained digital images oﬀer a new alternative for a broader and more eﬃcient screening [Echle et al.
In CRC, previous work suggests that the use of such models as pre-screening tools could eventually replace
IHC and PCR for a subset of tumors classiﬁed as microsatellite stable (MSS) or unstable (MSI) with a high probability [Kacew et al.
In other locations where MSI prevalence is lower and screening not done as routine practice, predictive models of MSI status from WSI could be used as eﬃcient pre-screening tools.
In this work, we leverage recent advances in self-supervised learning (SSL) on images.
We show that SSL permits to reach state-of-the-art results on colorectal and gastric cancer cohorts from The Cancer Genome Atlas (TCGA), generalizes well on an unseen colorectal cohort (PAIP), and could pave the way for classiﬁers on locations with low MSI prevalence.
2. Related Work Expert models
Several histology patterns on H&E images have been reported to correlate with MSI, such as tumor-inﬁltrating lymphocytes, lack of dirty necrosis or poor diﬀerentiation [(Greenson et al.
A series of models based on clinico-pathological features have been developed [Greenson et al.
; Jenkins et al.
; Hyde et al.
; Fujiyoshi et al.
(2017); Rom´an et al.
and reported ROC-AUC performances ranging from 0.85 to 0.92 in various cohorts of patients with CRC.
These methods however require time-consuming annotations from expert pathologists and are prone to inter-rater variability.
Deep learning models
In a seminal publication, Kather et al.
(2019) proved the feasibility to determine the dMMR/MSI status from H&E stained whole slide images (WSI) using deep learning.
They trained a ﬁrst ResNet [He et al.
(2016)] to segment tumor regions on WSI, and a second one to predict MSI/MSS status in each tumor tile.
Each ResNet was pretrained on ImageNet and the last 10 layers were ﬁne-tuned.
Models were trained and validated on diﬀerent TCGA cohorts and obtained respectively AUCs of 0.77, 0.81 and 0.
on Colorectal, Gastric and Endometrial formalin-ﬁxed paraﬃn-embedded (FFPE) datasets.
In a larger scale study focusing on CRC only [Echle et al.
(2020)], the same team later trained a model on n = 6406 patients, reaching 0.96 AUC (95% CI 0.93–0.98) on an external dataset of n = 771 patients.
Tumor tissues were manually outlined by pathologists.
Since then, diﬀerent works based on deep learning methods have been published [Zhang et al.
; Cao et al.
(2020); Hong et al.
Yamashita et al.
; Bilal et al.
(2021); Lee et al.
and are reviewed in Hildebrand et al.
The vast majority rely on networks pretrained on the ImageNet dataset [Deng et al.
and only the last layers are re-trained or ﬁne-tuned.
Most of these papers also rely on tumor segmentation as a ﬁrst step in their models (either by a pathologist or by a deep learning model).
Self-Supervised learning
Over the past few years, rapid progress has been made in the ﬁeld of SSL using contrastive learning or self-distillation strategies:
simCLR [Chen et al.
(2020a,b)] , MoCo [He et al.
; Chen et al.
(2020c, 2021)], BYOL [Grill et al.
(2020)], achieving impressive performances on ImageNet without using any labels.
Such models have also been shown to outperform supervised models in transfer learning tasks [Chen et al.
(2020a)
; Caron et al.
; Li et al.
(2021b)].
These advances are of particular interest in medical imaging applications where labeled datasets are hard to collect, and especially in histology where each WSI contains thousands of unlabeled images.
There is growing evidence that SSL is a powerful method to obtain relevant features for various prediction tasks from histology images [Dehaene et al.
Lu et al.
; Li et al.
(2021a)
; Koohbanani et al.
; Gildenblat and Klaiman (2019);
Abbet et al.
; Srinidhi et al.
In this work, we show that self-supervision can be eﬃciently used to detect dMMR/
MSI tumors from histology slides, and outperform
ImageNet pretrained models across and between several tumors.
3. Methods 3.1 Proposed pipeline
First, a U-Net neural network [Ronneberger et al.
(2015)] is used to segment tissue on the input WSI and discard the background, as well as artifacts.
Second, segmented tissue is divided into N (typically between 10,000 and 30,000) smaller images called tiles.
Each tile has a ﬁxed shape of 224 × 224 pixels (resolution of 0.5 micron per pixel).
Third, the N tiles are embedded into feature vectors of shape D using a pretrained convolutional neural network.
Fourth, the N × D features are aggregated using a multiple instance learning model.
This ﬁnal model is the only one trained using MSI/MSS labels.
In this study, we benchmarked 2 diﬀerent feature extractors (ResNet-50 pretrained with supervised learning on ImageNet [Deng et al. (2009)] or with SSL on TCGA) and 3 multiple instance learning models (MeanPool, Chowder and DeepMIL).
[Table] (cid:148)(cid:138)(cid:155)(cid:134)(cid:3)(cid:340)(cid:340)(cid:342)(cid:153)(cid:340)(cid:340)(cid:342)(cid:279)(cid:3)(cid:9)(cid:130)(cid:132)(cid:137)(cid:3)(cid:149)(cid:138)(cid:141)(cid:134)(cid:3)(cid:138)(cid:148)(cid:3)(cid:134)(cid:143)(cid:132)(cid:144)(cid:133)(cid:134)(cid:133)(cid:3)(cid:150)(cid:148)(cid:138)(cid:143)(cid:136)(cid:3)(cid:130)(cid:3)(cid:145)(cid:147)(cid:134)(cid:149)(cid:147)(cid:130)(cid:138)(cid:143)(cid:134)(cid:133)(cid:3)(cid:22)(cid:134)(cid:148)(cid:18)(cid:134)(cid:149)(cid:3)(cid:312)(cid:138)(cid:143)(cid:3)(cid:141)(cid:138)(cid:136)(cid:137)(cid:149)(cid:3)(cid:131)(cid:141)(cid:150)(cid:134)(cid:313)(cid:280)(cid:3)(cid:3)(cid:130)(cid:143)(cid:133)(cid:3)(cid:149)(cid:137)(cid:134)(cid:3)(cid:148)(cid:134)(cid:149)(cid:3)(cid:144)(cid:135)(cid:3)(cid:135)(cid:134)(cid:130)(cid:149)(cid:150)(cid:147)(cid:134)(cid:148)(cid:3)(cid:138)(cid:148)(cid:3)(cid:149)(cid:137)(cid:134)(cid:143)(cid:3)(cid:130)(cid:136)(cid:136)(cid:147)(cid:134)(cid:136)(cid:130)(cid:149)(cid:134)(cid:133)(cid:3)(cid:150)(cid:148)(cid:138)(cid:143)(cid:136)(cid:3)(cid:130)(cid:3)(cid:142)(cid:150)(cid:141)(cid:149)(cid:138)(cid:145)(cid:141)(cid:134)(cid:3)(cid:138)(cid:143)(cid:148)(cid:149)(cid:130)(cid:143)(cid:132)(cid:134)(cid:3)(cid:141)(cid:134)(cid:130)(cid:147)(cid:143)(cid:138)(cid:143)(cid:136)(cid:3)(cid:142)(cid:144)(cid:133)(cid:134)(cid:141)(cid:3)(cid:312)(cid:133)(cid:130)(cid:147)(cid:140)(cid:3)(cid:131)(cid:141)(cid:150)(cid:134)(cid:313)(cid:3)(cid:149)(cid:144)(cid:3)(cid:145)(cid:147)(cid:134)(cid:133)(cid:138)(cid:132)(cid:149)(cid:3)(cid:17)(cid:23)(cid:13)(cid:3)(cid:145)(cid:147)(cid:144)(cid:131)(cid:130)(cid:131)(cid:138)(cid:141)(cid:138)(cid:149)(cid:154)(cid:279) (cid:18)(cid:3)(cid:149)(cid:138)(cid:141)(cid:134)(cid:148)(cid:3)(cid:312)(cid:340)(cid:340)(cid:342)(cid:280)(cid:340)(cid:340)(cid:342)(cid:313) (cid:27)(cid:137)(cid:144)(cid:141)(cid:134)(cid:3)(cid:23)(cid:141)(cid:138)(cid:133)(cid:134)(cid:3)(cid:13)(cid:142)(cid:130)(cid:136)(cid:134) (cid:18)(cid:153)(cid:8)(cid:3)(cid:10)(cid:134)(cid:130)(cid:149)(cid:150)(cid:147)(cid:134)(cid:148) [Figure]
We ﬁrst extracted features using the last layer of a
ResNet-50 pretrained using supervised learning on the ImageNet-1k dataset.
We used an auto-encoder to reduce dimension to D = 256 because we observed empirically that it significantly improves the performances of Chowder while yielding similar results for MeanPool and DeepMIL. We did not observe similar improvements with MoCo features.
MoCo feature extraction
Following Dehaene et al.
, we trained several ResNet50 models using Momentum Contrast v2 (MoCo v2[Chen et al.
(2020c)]).
We used the exact same parameters and data augmentation scheme but a bigger ResNet backbone was used (the bottleneck number of channels is twice larger in every block).
Three diﬀerent feature extractors were trained: MoCo-CRC using 4.7M tiles from TCGA-CRC, MoCo-Gastric using 4.7M tiles from TCGA-Gastric, and MoCo-CRC-Gastric using the concatenation of the two previous datasets.
Each model was trained for 200 epochs (approximately 30 hours) on 16 NVIDIA Tesla V100.
The obtained MoCo features have a dimension D = 2048.
MeanPool
As a baseline multiple instance learning method, we used a simple average pooling of the tile features followed by a logistic regression either with or without L 2 penalization ( C = 0, 0 .
5 or 1).
Chowder
We implemented a variant of Chowder [Courtiol et al.
A multilayer perceptron (MLP) with 128 hidden neurons and sigmoid activation is applied to each tile’s features to output one score.
The R ( R = 10, 25 or 100) top and bottom scores are then concatenated and fed into a MLP with 128 and 64 hidden neurons and sigmoid activations.
DeepMIL
We reimplemented the attention based model proposed by Ilse et al.
A linear layer with N neurons ( N = 64, 128 or 256 here) is applied to the embedding followed by a Gated Attention layer with N hidden neurons.
A linear layer followed by a sigmoid activation is then applied to the output.
For both Chowder and DeepMIL, a random subset of n = 8000 tiles per WSI was used to accelerate training.
All hyperparameters were tuned on the diﬀerent training sets (see
Results).
Chowder and DeepMIL were trained with a learning rate of 0.001 using crossentropy loss and the Adam optimizer [Kingma and Ba (2014)].
When multiple WSI were available for a given patient, we averaged the predictions of the models at test time. [Table] [Table]
3.2 Datasets
Three diﬀerent cohorts were used in this study and are summarized in Table 1.
For all cohorts, only FFPE images were used.
TCGA-CRC is a dataset of n = 555 patients from 36 centers in the US with colorectal tumors.
It is a combination of two cohorts from TCGA: TCGA-COAD (colon adenocarcinoma) and TCGA-READ (rectum adenocarcinoma).
TCGA-STAD (stomach adenocarcinoma), later referred as TCGA-Gastric, is a dataset of n = 375 patients from 22 centers in the US with gastric cancer.
For both datasets, MSS/MSI-H labels deﬁned by PCR assays were retrieved using TCGA-biolinks [Colaprico et al.
As recommended by ESMO guidelines [Luchini et al.
(2019)], MSI-L patients were classiﬁed as MSS.
TCGA-CRC-Kather and TCGA-Gastric-Kather are variants of respectively TCGA-CRC and TCGA-Gastric datasets published by Kather et al.
They were used here for comparison purposes.
These datasets consist of a lower number of cases because MSI-L patients were excluded.
The exact same MSI labels were used.
PAIP (cohort from the Pathology AI Platform, http://www.wisepaip.org) is a dataset of n = 47 patients from 3 centers in Korea with colorectal tumors.
MSS/MSI-H labels were determined using PCR assays.
[Table]
4. Results 4.1 Cross-validations on TCGA-CRC and TCGA-Gastric
We ﬁrst compared three multiple instance learning models, MeanPool, Chowder and DeepMIL, using the MoCo-CRC features on the TCGA-CRC-Kather cohort, and the MoCoGastric features on the TCGA-Gastric-Kather cohort.
For the sake of comparison, we used the exact same train / test split as in [Kather et al.
and compared our results with the ones published in the literature on this split.
We tuned the hyperparameters of Chowder ( R = 10, 25 or 100), DeepMIL (size of attention layer = 64, 128 or 256), MeanPool (l2 penalization = 0, 0.5 or 1) and the number of epochs (ranging from 5 to 120) using a grid search on the training sets.
For both TCGACRC-Kather and TCGA-Gastric-Kather, Chowder model performed best and obtained respectively AUCs of 0.92 and 0.83, achieving state-of-the-art results on these datasets (see Table 2).
To analyze the gain of MoCo features over ImageNet ones, we ran larger cross-validation experiments using 15 distinct splits on the full TCGA cohorts (5 fold cross-validation, repeated 3 times) and reported results in Table 3.
For a fair comparison, we tuned the hyperparameters of all models on the training set of the Kather split.
In all our experiments,
MoCo substantially outperformed ImageNet in both TCGA-CRC and TCGA-Gastric cohorts.
Results obtained with MoCo features were also better than the ones previously reported with cross-validation.
We also report cross-validation results using center split in
Supplementary Table S1.
4.2 External validation on CRC dataset PAIP
A limitation of the previous experiments is that ResNet using SSL were pretrained on the full TCGA cohorts, slightly breaking the train/test split independence assumption even if no MSI/MSS labels were used.
Thus, we further evaluated MSI detection on an independent cohort of n = 47 colon cases from PAIP organisation.
We used the median prediction of the [Table] [Table] ensemble of models trained during cross-validation in the previous experiment.
Chowder with MoCo-CRC features yielded an AUC of 0.97 on par with the 0.98 reported in Bilal et al.
(2021), and signiﬁcantly higher than Chowder with ImageNet features (AUC of 0.82, p = 0 .
03 with DeLong’s test
[DeLong et al.
The superiority of MoCo-CRC features was observed for all models (Table 4), conﬁrming that the MoCo-CRC pretrained backbone is a more robust feature extractor than the ImageNet for histology images.
4.3 Transfer CRC to Gastric
We assessed the performances for MSI detection when transferring models trained on
TCGA-CRC to TCGA-Gastric with three diﬀerent feature extractors: ImageNet, MoCoCRC, MoCo-CRC-Gastric.
For fair comparison, all hyperparameters were tuned on the training split of TCGA-CRC-Kather (Supplementary Table S2).
MoCo-CRC-Gastric models consistently yielded the best results on TCGA-Gastric with
AUCs up to 0.80 (Table 5) while also demonstrating high performances on TCGA-CRC (Supplementary Table S3).
Surprisingly, the ImageNet MeanPool model, which performed poorly on PAIP, also obtained a high AUC of 0.76, however signiﬁcantly lower compared to 0.80 ( p = 0 .
05 with DeLong’s test).
To our knowledge, this is the best performance reported in a transfer setting from one organ to another.
5. Interpretability
Within Chowder, each tile is associated with a single score and a MLP is applied to the top and lowest scores of the WSI (Section 3.1).
To explore potential diﬀerences in interpretability when using ImageNet or MoCo features, a pathologist expert in dMMR tumors reviewed the subset of tiles associated with extreme scores within TCGA-CRC and TCGA-Gastric cohorts, for both feature extractors.
[Figure] Among the top scored tiles (Figure 2 A, B), the pathologist recognized histological patterns which have been previously described as associated with MSI tumors [Greenson et al. (2009)], including tumor lymphocyte inﬁltration, mucinous diﬀerentiation and presence of dirty necrosis - the latter being more frequently detected in gastric tumors.
While ImageNetbased models outputs were based on patterns within and outside tumor regions (such as lymphocyte inﬁltration in the tumor microenvironment), MoCo-based models clearly focused on tumor epithelium, highlighting both poor diﬀerentiation of epithelial cells and tumor inﬁltrating lymphocytes (TILs) as patterns associated with MSI tumors.
Speciﬁcally, MoCo models better segmented tumoral regions, as shown by the tile scores heatmap at the WSI level (Figure 2 C).
Overall, lowest scored tiles were unrelated to known MSI patterns, displaying non tumoral tissues such as muscle cells.
In the TCGA-CRC cohort, the MoCo-CRC model additionally highlighted the presence of intratumoral epithelial cells without MSI tumor characteristics, showing once again a higher focus on tumor regions, in line with the regions actually inspected by a pathologist when searching for MSI related patterns.
6. Discussion
In this work, we showed that feature extractors pretrained using SSL (MoCo V2) on TCGA reach state-of-the-art results for MSI prediction both in colorectal and gastric cancers (Table 2).
Extensive cross-validations showed the clear superiority of these models over their counterparts pretrained using ImageNet (Table 3).
In addition, the former generalize better on an external CRC cohort (Table 4).
Finally, we observed that using a feature extractor pretrained on several organs using SSL (Table 5) opens the way to both state-of-the-art performances in cross-validation and robust generalization from one organ to another.
There are several limitations to our current study.
First, our models could beneﬁt from being trained on more data as shown in the training curve from Echle et al.
(2020) (Figure 1.c).
Second, our models should be validated on larger cohorts, encompassing diﬀerent patient populations, treatments (neoadjuvant chemotherapy can impact cell morphology), scanner manufacturer and sample types (resections, biopsies).
Third, SSL techniques are evolving rapidly using new architectures such as Vision Transformers [Dosovitskiy et al.
(2020)] and more experiments are required to ﬁnd how to apply them in histology, including taking into account the spatial arrangement of the tiles.
However, such experiments require access to extensive computational resources, which limits reproducibility.
Finally, in contrast to several concurrent works [Echle et al.
Bilal et al.
; Kather et al.
that ﬁne-tuned the backbones, while our study kept them entirely frozen.
A recent study, Kacew et al.
(2021) showed that performances of deep learning models may impact the diagnosis of MSI for patients with CRC.
Notably, MSI diagnosis is not routinely done for patients with other solid tumors, missing the identiﬁcation of candidate patients for immunotherapy.
Our results indicate that SSL is a promising solution to develop accurate models for frequent tumor localization such as oesophagus, pancreas, small intestine or even brain, where images are available but MSI prevalence is too low for systematic IHC or molecular testing.
Acknowledgments
We thank Florence Renaud, pathologist at CHU Lille, for her insightful analysis of the most predictive regions.
We thank Patrick Sin-Chan for his corrections of the manuscript.
This work was granted access to the HPC resources of IDRIS under the allocation 2020AD011011731 made by GENCI.
The results published here are part based upon data generated by the TCGA
Research
Network: https://www.cancer.gov/tcga.
Regarding the PAIP dataset: De-identiﬁed pathology images and annotations used in this research were prepared and provided by the Seoul National University Hospital by a grant of the Korea Health Technology R&D Project through the Korea Health Industry
Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of
Korea (grant number: HI18C0316).
Supplementary Tables and Figures
[Table]
Feature extractor
Train dataset
MeanPool
Chowder
DeepMIL
ImageNet CRC C = R =
N =100 epochs 30 epochs
ImageNet
Gastric
C = 0.
R =
N =30 epochs 20 epochs
MoCo
[Table]
CRC
C = 0.
R =
N =10 epochs 10 epochs
MoCo
[Table]
Gastric
Gastric
C = 1.
R =
N =30 epochs 10 epochs
[Table]
CRC
C = 0.
R =
N =30 epochs 10 epochs
[Table]
Epochs
V a li d a t i o n
A U
C