Evaluating AI in breast cancer screening: a complex task
Authors: Magnus Dustler
Journal: The Lancet Digital Health (2020)

Today, artificial intelligence (AI) seems to be making an impact in almost every field of research. Breast cancer screening is one of them. Digital mammography is the current standard modality for early breast cancer detection, but as with every screening tool, it has its limitations. Sensitivity in a given screening round is estimated at 75–85%,1Domingo L Hofvind S Hubbard RA et al.Cross-national comparison of screening mammography accuracy measures in US, Norway, and Spain.Eur Radiol. 2016; 26: 2520-2528Crossref PubMed Scopus (34) Google Scholar meaning that a substantial fraction of cancers go undetected. In recent screening studies, digital breast tomosynthesis has found up to 40% more cancers than digital mammography,2Zackrisson S Lang K Rosso A et al.One-view breast tomosynthesis versus two-view mammography in the Malmo Breast Tomosynthesis Screening Trial (MBTST): a prospective, population-based, diagnostic accuracy study.Lancet Oncol. 2018; 19: 1493-1503Summary Full Text Full Text PDF PubMed Scopus (87) Google Scholar, 3Skaane P Bandos AI Gullien R et al.Comparison of digital mammography alone and digital mammography plus tomosynthesis in a population-based screening program.Radiology. 2013; 267: 47-56Crossref PubMed Scopus (700) Google Scholar, 4Ciatto S Houssami N Bernardi D et al.Integration of 3D digital mammography with tomosynthesis for population breast-cancer screening (STORM): a prospective comparison study.Lancet Oncol. 2013; 14: 583-589Summary Full Text Full Text PDF PubMed Scopus (579) Google Scholar suggesting that previously reported sensitivities for digital mammography are too high. Additionally, the specificity of screening mammography is estimated at 90–95%, which in a screening modality where more than 99% of examined women are healthy means that the vast majority of detections are false positives. Every false positive means that a woman needs to be recalled, re-examined (often with additional tests such as ultrasound imaging), and often biopsied, which, apart from the additional health-care expenditure, can lead to anxiety and other negative psychosocial consequences.5Bolejko A Hagell P Wann-Hansson C Zackrisson S Prevalence, long-term development, and predictors of psychosocial consequences of false-positive mammography among women attending population-based screening.Cancer Epidemiol Biomarkers Prev. 2015; 24: 1388-1397Crossref PubMed Scopus (20) Google Scholar AI, either as a detection aid together with a human reader (an advanced form of computer-aided detection) or as an independent second reader, could, in different ways, be used to bypass the limitations of screening mammography. Studies using various algorithms have shown human-like or superhuman performance under conditions that are sometimes fairly realistic.6Rodriguez-Ruiz A Lang K Gubern-Merida A et al.Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists.J Natl Cancer Inst. 2019; 111: 916-922Crossref PubMed Scopus (115) Google Scholar, 7Becker AS Marcon M Ghafoor S Wurnig MC Frauenfelder T Boss A Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer.Invest Radiol. 2017; 52: 434-440Crossref PubMed Scopus (150) Google Scholar, 8Kooi T Litjens G van Ginneken B et al.Large scale deep learning for computer aided detection of mammographic lesions.Med Image Anal. 2017; 35: 303-312Crossref PubMed Scopus (460) Google Scholar, 9McKinney SM Sieniek M Godbole V et al.International evaluation of an AI system for breast cancer screening.Nature. 2020; 577: 89-94Crossref PubMed Scopus (400) Google Scholar In The Lancet Digital Health, Hyo-Eun Kim and colleagues' multireader study provides new insights in the field.10Kim H-E Kim HH Han BK et al.Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study.Lancet Digital Health. 2020; (published online Feb 6. https://doi.org/10.1016/S2589-7500(20)30003-0.)Scopus (53) Google Scholar In short, the South Korean academic–industrial partnership behind the study trained, validated, and tested a promising AI algorithm to aid detection of breast cancer in mammography. It was trained on a large set of images (n=170 230 mammograms) from three different countries and at least three different manufacturers. This is an important feature, as diverse training material should help to make the algorithm more generally applicable. With the assistance of AI, radiologists markedly improved both their sensitivity and specificity in a retrospective reader study on a cancer-enriched dataset of 320 mammograms drawn from the South Korean breast cancer screening programme. An interesting feature of the reader study dataset, which is not stressed in the Article, is that because of the way in which breast cancer screening is done in South Korea, the dataset includes a substantial but unspecified proportion of ultrasound-detected cancers that can be expected to not be visible on mammography. This could be a major strength of the study. AI validation sets in other studies have often included only screen-detected cancers, but this does not allow AI to find cancers that radiologists have missed, which might be a substantial portion according to the numbers cited above. The ground truth is unknown. Adding information about interval cancers (ie, cancers detected in the interval between screening rounds) gives AI the opportunity to at least find interval cancers, but this introduces a new problem of whether interval cancers were actually present in the breast at the screening occasion or surfaced clinically as fast-growing tumours after screening. Kim and colleagues' study has a more accurate ground truth, as the ultrasound-detected cancers were present in the breast, even if they might have been non-detectable (ie, present only as minimal signs or not possible to see) to radiologists on digital mammography. This gave the AI algorithm an opportunity to truly show superhuman performance by helping to find cancers that would not have been detectable by an unaided radiologist. It would have been very interesting to see sensitivity separated by detection modality; this angle should be further explored, as an algorithm that can, on one modality, reliably detect cancers that a human reader would need a second modality to find could conceivably remove the need for the second modality. The standalone performance of the AI algorithm was evaluated on three validation sets from three different countries (South Korea, the UK, and the USA), with higher reported sensitivity and specificity than from the reader study dataset. The overall sensitivity of the algorithm in the combined validation dataset seems high, at 0·914 (95% CI 0·897–0·930), but cannot be directly compared with that of human readers in a screening situation. The reported specificity of 0·860 (0·845–0·875) is low if compared with that shown generally by screening radiologists. It should be stressed that all results presented in the study are derived from retrospective evaluations on image sets selected from screening, not on an actual screening set or in an actual screening situation. Non-enriched screening material has 0·5–1·0% cancer prevalence, whereas the enriched material used by Kim and colleagues had 50% prevalence.10Kim H-E Kim HH Han BK et al.Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study.Lancet Digital Health. 2020; (published online Feb 6. https://doi.org/10.1016/S2589-7500(20)30003-0.)Scopus (53) Google Scholar The performance of radiologists in a real-world screening situation is not necessarily, or even likely to be, the same as it would be on the material evaluated in Kim and colleagues' reader study. For example, the specificity scores shown by radiologists, AI-aided radiologists, and standalone AI are all much lower than what is normally seen in screening.10Kim H-E Kim HH Han BK et al.Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study.Lancet Digital Health. 2020; (published online Feb 6. https://doi.org/10.1016/S2589-7500(20)30003-0.)Scopus (53) Google Scholar This is the main limitation of the study, which is a limitation shared with most other studies on AI in breast cancer screening. Thus, while promising algorithms, such as the one evaluated by Kim and colleagues, could potentially improve breast screening, to prove this it is imperative to conduct retrospective studies on unaltered screening materials and, even more importantly, randomised prospective trials. I declare no competing interests. Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader studyThe AI algorithm developed with large-scale mammography data showed better diagnostic performance in breast cancer detection compared with radiologists. The significant improvement in radiologists' performance when aided by AI supports application of AI to mammograms as a diagnostic support tool. Full-Text PDF Open Access