Towards computationally efficient prediction of molecular signatures from routine histology images
Authors: Maxime W Lafarge, Viktor H Koelzer
Journal: The Lancet Digital Health (2021)

Identification of actionable genomic alterations in diagnostic tissue samples provides key information for personalised cancer treatment. However, current diagnostic tests used to predict the status of molecular pathways from standard histopathology material are commonly tissue destructive, generate relevant costs, and can take hours or even days to return conclusive results. Relating molecular alterations to digital image information derived from standard haematoxylin and eosin-stained histopathology slides has become a task that machine learning models are able to solve across multiple cancer types (appendix).1Coudray N Ocampo PS Sakellaropoulos T et al.Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning.Nat Med. 2018; 24: 1559-1567Crossref PubMed Scopus (922) Google Scholar, 2Echle A Grabsch HI Quirke P et al.Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning.Gastroenterology. 2020; 159: 1406-1416.e11Summary Full Text Full Text PDF PubMed Scopus (83) Google Scholar, 3Kather JN Heij LR Grabsch HI et al.Pan-cancer image-based detection of clinically actionable genetic alterations.Nat Can. 2020; 1: 789-799Crossref PubMed Scopus (127) Google Scholar, 4Popovici V Budinská E Dušek L Kozubek M Bosman F Image-based surrogate biomarkers for molecular subtypes of colorectal cancer.Bioinformatics. 2017; 33: 2002-2009Crossref PubMed Scopus (10) Google Scholar, 5Sirinukunwattana K Domingo E Richman SD et al.Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning.Gut. 2021; 70: 544-554Crossref PubMed Scopus (46) Google Scholar, 6Veta M Heng YJ Stathonikos N et al.Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge.Med Image Anal. 2019; 54: 111-121Crossref PubMed Scopus (98) Google Scholar, 7Weitz P Wang Y Kartasalo K et al.Transcriptome-wide prediction of prostate cancer gene expression from histopathology images using co-expression based convolutional neural networks.arXiv. 2021; (https://arxiv.org/abs/2104.09310 (preprint) published online April 19)Google Scholar For digitised pathology labs, this technology promises the introduction of support tools that will be able to flag the presence of molecular alterations from histology slides in the intermediate future. Such image-based molecular classification algorithms have the potential to facilitate the cost-effective preselection of patients for molecular testing and to screen for hereditary cancer syndromes. In future development stages, image-based classification tools trained on outcome data could also directly aid clinical stratification and help to achieve optimal treatment decisions in difficult clinical scenarios.5Sirinukunwattana K Domingo E Richman SD et al.Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning.Gut. 2021; 70: 544-554Crossref PubMed Scopus (46) Google Scholar Slide-level representation learning is an important area of research in computational pathology because discriminative morphological features need to be automatically discovered and located to make the desired prediction. However, due to their large size, processing whole-slide images comes with technical challenges that so far limit the efficacy of conventional computational approaches and impair the progress towards clinical deployment. As existing computational resources do not enable the direct processing of whole-slide images with standard deep learning architectures, current designs typically entail a tradeoff between the quality of the learned representation and how much computational complexity to use. To answer this technical challenge, the original task of classifying gigapixel images can be simplified into a weakly supervised learning problem by splitting whole-slide images into tiles, each of which is then processed individually. This approach provides class predictions at the tile level, which are statistically aggregated to determine the overall slide classification. Although this popular design assumes an equivalence with the hypothetical end-to-end approach, it remains computationally demanding on large datasets. In their study published in The Lancet Digital Health, Mohsin Bilal and colleagues8Bilal M Raza SEA Azam A et al.Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; (published online Oct 19)https://doi.org/10.1016/S2589-7500(21)00180-1Summary Full Text Full Text PDF PubMed Scopus (19) Google Scholar present a weakly supervised, deep learning framework composed of three separate convolutional neural network models, trained on 502 haematoxylin and eosin-stained whole-slide images from patients with colorectal cancer. Bilal and colleagues8Bilal M Raza SEA Azam A et al.Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; (published online Oct 19)https://doi.org/10.1016/S2589-7500(21)00180-1Summary Full Text Full Text PDF PubMed Scopus (19) Google Scholar investigated the ability of a novel iterative draw and rank sampling algorithm to directly train a model to calculate a prediction score for each tile in a whole-slide image, which represents the likelihood of a tile belonging to the positive class within the binary classification scheme (ie, the presence or absence of a defined molecular signature of interest); model performance was evaluated through cross-validation, comparison with published studies, and external validation on 47 slides from the Pathology Artificial Intelligence Platform challenge cohort. They report that their iterative draw and rank sampling-based model outperformed previously published methods for most class predictions. Iterative draw and rank sampling provides a new solution to tackle the computational complexity inherent to the analysis of histopathology whole-slide images by focusing the learning procedure on so-called informative tiles to achieve state-of-the-art performances. The authors' results support the benefit of efficient sampling strategies to improve learning procedures. This notion is in line with the commonly made assumption that most information in a whole-slide image is irrelevant to solve a given task, and underlies the relevance of sampling methodologies, as established in the computational pathology literature.6Veta M Heng YJ Stathonikos N et al.Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge.Med Image Anal. 2019; 54: 111-121Crossref PubMed Scopus (98) Google Scholar To help to explain the prediction results, many tile-based approaches are designed to locate the regions in the whole-slide images that were the most informative in making the final prediction and to provide a visual mark-up of these regions to the domain expert. Besides visual inspection, Bilal and colleagues8Bilal M Raza SEA Azam A et al.Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; (published online Oct 19)https://doi.org/10.1016/S2589-7500(21)00180-1Summary Full Text Full Text PDF PubMed Scopus (19) Google Scholar also interpreted the meaning of these regions via an automated cellular composition analysis based on HoVer-Net.9Graham S Vu QD Raza SEA et al.Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images.Med Image Anal. 2019; 58: 101563Summary Full Text Full Text PDF PubMed Scopus (184) Google Scholar This analysis provides interesting insights at the intersection of computational pathology and tumour biology. Cell-level analysis of the tumour microenvironment is not restricted to the iterative draw and rank sampling framework, as this analysis could be applied to other frameworks that enable the localisation of informative regions. This analysis opens broad possibilities to expand our understanding of the perturbations in the cellular contexture of cancer introduced by the presence of molecular alterations in the neoplastic cell population. The study by Bilal and colleagues8Bilal M Raza SEA Azam A et al.Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; (published online Oct 19)https://doi.org/10.1016/S2589-7500(21)00180-1Summary Full Text Full Text PDF PubMed Scopus (19) Google Scholar underlines that computational models can accurately predict the status of molecular pathways and key mutations in colorectal cancer from standard histology sections. However, further questions need to be answered before purely image-based methods can be confidently used in clinical practice. Perhaps the highest priority is the need for independent benchmark datasets for morpho-molecular classification to enable a direct comparison of methodologies. Only a minority of related studies validate their model on independent test sets or only for a subset of the investigated molecular markers (appendix). To advance clinical translation, image-based molecular classifiers will need to be independently validated in multicentre studies with robust statistical interpretation following artificial intelligence-specific reporting guidelines for diagnostic studies, which include a systematic, prospective comparison of diagnostic accuracy to state-of-the-art molecular assays.10Sounderajah V Ashrafian H Aggarwal R et al.Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: the STARD-AI Steering Group.Nat Med. 2020; 26: 807-808Crossref PubMed Scopus (74) Google Scholar From a technological perspective, the need for further advances in the field of robust representation learning with efficient training procedures and further improvements in clinical-grade slide imaging technologies remain key bottlenecks to be addressed in future studies. We declare no competing interests. Download .pdf (.08 MB) Help with pdf files Supplementary appendix Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective studyAfter large-scale validation, our proposed algorithm for predicting clinically important mutations and molecular pathways, such as microsatellite instability, in colorectal cancer could be used to stratify patients for targeted therapies with potentially lower costs and quicker turnaround times than sequencing-based or immunohistochemistry-based approaches. Full-Text PDF Open Access