Opportunistic deep learning of retinal photographs: the window to the body revisited
Authors: Sebastian M Waldstein
Journal: The Lancet Digital Health (2020)

Could a doctor ascertain a patient's age, gender, smoking status, risk of chronic kidney disease, and cardiovascular mortality just by examining the ocular fundus? What sounds quite impossible to clinicians might already be achieved by new artificial intelligence tools. To understand the concept of a current movement within the field (opportunistic deep learning), it is worth looking into the underlying methodological principles. Artificial intelligence models automatically learn the connection between input data and so-called target variables.1Schmidt-Erfurth U Sadeghipour A Gerendas BS Waldstein SM Bogunovic H Artificial intelligence in retina.Prog Retin Eye Res. 2018; 67: 1-29Crossref PubMed Scopus (340) Google Scholar These target variables can be chosen freely, as long as there are sufficient data (typically several thousand instances) to enable the system to abstract the link, and assuming a correlation actually exists. The list of potential target variables for opportunistic deep learning in medicine is virtually endless. Sabanayagam and colleagues2Sabanayagam C Xu D Ting DSW et al.A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations.Lancet Digital Health. 2020; (published online May 12.)https://doi.org/10.1016/S2589-7500(20)30063-7Summary Full Text Full Text PDF Scopus (47) Google Scholar introduced a new target, chronic kidney disease, for an existing deep learning network architecture receiving colour fundus photographs. The authors report that it was possible to diagnose the presence of chronic kidney disease with reasonable accuracy (sensitivity and specificity between 75% and 85% in the external test and development datasets respectively). Similar studies have used opportunistic targets such as demographics3Poplin R Varadarajan AV Blumer K et al.Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning.Nat Biomed Eng. 2018; 2: 158-164Crossref PubMed Scopus (742) Google Scholar or atherosclerosis,4Chang J Ko A Park SM et al.Association of cardiovascular mortality and deep learning-funduscopic atherosclerosis score derived from retinal fundus images.Am J Ophthalmol. 2020; (published online March 25)DOI:10.1016/j.ajo.2020.03.027Summary Full Text Full Text PDF Scopus (25) Google Scholar coupled with retinal photographs as input. For some of these targets, accuracy well beyond our expectation as clinicians could be achieved. Many aspects of Sabanayagam and colleagues' study2Sabanayagam C Xu D Ting DSW et al.A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations.Lancet Digital Health. 2020; (published online May 12.)https://doi.org/10.1016/S2589-7500(20)30063-7Summary Full Text Full Text PDF Scopus (47) Google Scholar were conducted in an exemplary manner. The authors had access to a large dataset on which to train and validate their algorithm, as well as two external test sets. Owing to the general lack of data sharing in ophthalmic research, most work in this field cannot offer such extensive external validation, and open access to validation databases remains one of the major unmet needs in artificial intelligence in retinal imaging.5Berens P Waldstein SM Ayhan MS et al.[Potential of methods of artificial intelligence for quality assurance].Ophthalmologe. 2020; 117: 320-325Crossref PubMed Scopus (2) Google Scholar Chronic kidney disease is a condition with a low prevalence,6Reutens AT Epidemiology of diabetic kidney disease.Med Clin North Am. 2013; 97: 1-18Summary Full Text Full Text PDF PubMed Scopus (181) Google Scholar which is reflected in some of the datasets used by the authors. In a low prevalence setting, the often-used outcome measure of the area under the receiver operating characteristics curve can overestimate the performance of the model, and the area under the precision-recall curve could be a more accurate outcome measure,7Yu M Tham YC Rim TH Ting DS Wong TY Cheng CY Reporting on deep learning algorithms in health care.Lancet Digit Health. 2019; 1: e328-e329Summary Full Text Full Text PDF PubMed Scopus (13) Google Scholar as the authors acknowledge. In this study,2Sabanayagam C Xu D Ting DSW et al.A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations.Lancet Digital Health. 2020; (published online May 12.)https://doi.org/10.1016/S2589-7500(20)30063-7Summary Full Text Full Text PDF Scopus (47) Google Scholar this more conservative measure dropped steeply along with the positive predictive value when the algorithm was deployed on test data with a lower prevalence. Consensus has not been reached on what are clinically acceptable levels of sensitivity and specificity in settings of low, medium, or high prevalence. Autonomous screening for diabetic retinopathy has taken this first step, and has been approved as first-in-class by the United States Food and Drug Administration.8Abramoff MD Tobey D Char DS Lessons learnt about Autonomous AI: finding a safe, efficacious and ethical path through the development process.Am J Ophthalmol. 2020; (published online March 11.)DOI:10.1016/j.ajo.2020.02.022Summary Full Text Full Text PDF PubMed Scopus (39) Google Scholar Use of non-invasive photography of the back of the eye to screen for systemic medical conditions is appealing, particularly as the availability of autonomous image analysis software will boost the widespread application of retinal photography in primary care and optometry settings. For this concept to be applied, no logical, or even biologically plausible link need exist between a candidate opportunistic target and the input. A deep learning algorithm could learn the link between retinal photographs and peoples' taste in cars, if such a link exists. Because of the inherent black box characteristic of deep learning, establishing exactly how an artificial intelligence algorithm came to its conclusions is still a challenge.9Castelvecchi D Can we open the black box of AI?.Nature. 2016; 538: 20-23Crossref PubMed Scopus (606) Google Scholar This phenomenon comes to mind when analysing the results reported by Sabanayagam and colleagues.2Sabanayagam C Xu D Ting DSW et al.A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations.Lancet Digital Health. 2020; (published online May 12.)https://doi.org/10.1016/S2589-7500(20)30063-7Summary Full Text Full Text PDF Scopus (47) Google Scholar Their image-based deep learning model did not outperform an alternative model based on five simple clinical risk factors (age, sex, ethnicity, diabetes, and hypertension); and the addition of these clinical risk factors to the image-based model did not meaningfully improve overall performance. This calls into question whether the retinal images contain any additional information relevant to chronic kidney disease other than the clinical risk factors themselves. Obviously, diabetic and hypertensive retinopathy leave their trace in the fundus, as do sex and age.3Poplin R Varadarajan AV Blumer K et al.Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning.Nat Biomed Eng. 2018; 2: 158-164Crossref PubMed Scopus (742) Google Scholar An open question in Sabanayangam and colleagues' Article2Sabanayagam C Xu D Ting DSW et al.A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations.Lancet Digital Health. 2020; (published online May 12.)https://doi.org/10.1016/S2589-7500(20)30063-7Summary Full Text Full Text PDF Scopus (47) Google Scholar remains the degree of correlation between the risk factors and the fundus image. Are deep learning algorithms demonstrating superhuman performance because they can achieve these tasks? Very few ophthalmology studies include a direct comparison with a human benchmark,10Bogunovic H Waldstein SM Schlegl T et al.Prediction of anti-VEGF treatment requirements in neovascular AMD using a machine learning approach.Invest Ophthalmol Vis Sci. 2017; 58: 3240-3248Crossref PubMed Scopus (95) Google Scholar because doctors usually do not try to predict targets such as age or sex from retinal photographs (and it is not established whether predicting these targets would be possible after appropriate training). The introduction of proper benchmarks for opportunistic deep learning is therefore another open question for future research. I report personal fees from Novartis, grants from Bayer, and grants from Genentech, outside the submitted work. In addition, I have patents US 15571480 and US 15554414 pending. A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populationsA retinal image DLA shows good performance for estimating chronic kidney disease, underlying the feasibility of using retinal photography as an adjunctive or opportunistic screening tool for chronic kidney disease in community populations. Full-Text PDF Open Access