Minimum thresholds deemed acceptable by patients and physicians for sensitivity and specificity of mobile skin cancer screening algorithms
Authors: Noor Anvery, Bianca Kang, Rachel E. Christensen, McKenzie A. Dirr, Umer Nadir, Joaquin C. Brieva, M. Laurin Council, Jeffrey S. Dover, Timothy M. Kuzel, Kira Minkis, Bharat B. Mittal, Jeffrey D. Wayne, Simon S. Yoo, Murad Alam
Journal: Journal of the American Academy of Dermatology (2023)

To the Editor: Mobile applications for detection of suspicious skin lesions have the potential to transform skin cancer prevention.1Sar-Graycar L. Rotemberg V.M. Matsoukas K. Halpern A.C. Marchetti M.A. Hay J.L. Interactive skin self-examination digital platforms for the prevention of skin cancer: a narrative literature review.J Am Acad Dermatol. 2021; 84: 1459-1468https://doi.org/10.1016/j.jaad.2020.07.014Abstract Full Text Full Text PDF PubMed Scopus (7) Google Scholar,2John A.M. Ragi S.D. Goldberg D.J. Mobile applications in skin cancer detection: a descriptive analysis.Dermatol Surg. 2021; 47: 1285-1286https://doi.org/10.1097/DSS.0000000000003134Crossref PubMed Scopus (1) Google Scholar Sensitivities and specificities of mobile applications †Sensitivity is the ability of a positive test to correctly identify a subject with disease, while specificity is the ability of a negative test to correctly identify a subject without disease.3Ngoo A. Finnane A. McMeniman E. Tan J.M. Janda M. Soyer H.P. Efficacy of smartphone applications in high-risk pigmented lesions.Australas J Dermatol. 2018; 59: e175-e182https://doi.org/10.1111/ajd.12599Crossref PubMed Scopus (33) Google Scholar are generally inferior to those of an in-person dermatologist visit (Supplementary Tables I and II, available via Mendeley at https://doi.org/10.17632/5mgxkdrf2h.1).3Ngoo A. Finnane A. McMeniman E. Tan J.M. Janda M. Soyer H.P. Efficacy of smartphone applications in high-risk pigmented lesions.Australas J Dermatol. 2018; 59: e175-e182https://doi.org/10.1111/ajd.12599Crossref PubMed Scopus (33) Google Scholar This may be the first cross-sectional study to determine minimum levels of sensitivity and specificity that patients and physicians expect from mobile applications for skin cancer screening. Seventy-five physicians (Table I) completed a 2-round modified Delphi process, and 60 patients each underwent a one-on-one interview (Supplementary Table I and Fig 1, available via Mendeley at https://doi.org/10.17632/5mgxkdrf2h.1). Participants were provided the purpose of mobile applications; meaning of "screening test"; US Food and Drug Administration definitions for "sensitivity" and "specificity", and lay definitions; and several clinical vignettes. Patients were given additional education (Supplementary Fig 2, available via Mendeley at https://doi.org/10.17632/5mgxkdrf2h.1). Participants were asked to specify the minimum acceptable sensitivities and specificities for skin cancer screening applications, and how likely they would be to use or recommend an application based on sensitivity.Table IFor each type of skin cancer, differences in minimum acceptable sensitivities and specificities between (a) physicians and patients, and (b) within physician specialtiesAll physicians (n = 75)Patients (n = 60)P-valueSensitivity Melanoma92% [95% CI: 91%-93%]92% [95% CI: 91%-94%].295 Basal Cell Carcinoma88% [95% CI: 87%-90%]92% [95% CI: 90%-93%].002 Squamous Cell Carcinoma89% [95% CI: 88%-91%]92% [95% CI: 90%-93%].010Specificity Melanoma79% [95% CI: 74%-83%]89% [95% CI: 86%-92%]<.001 Basal Cell Carcinoma77% [95% CI: 74%-80%]89% [95% CI: 87%-92%]<.001 Squamous Cell Carcinoma77% [95% CI: 74%-81%]89% [95% CI: 87%-92%]<.001Physicians who did not answer round 2 of the Delphi survey asked to have their responses carried forward. Open table in a new tab Physicians who did not answer round 2 of the Delphi survey asked to have their responses carried forward. Compared to physicians, patients expected higher minimum sensitivities for BCC (92% vs 88%, P = .002) and SCC (92% vs 89%, P = .01), and higher specificities for melanoma (89% vs 79%, P < .001), BCC (89% vs 77%, P < .001), and SCC (89% vs 77%, P < .001). Both patients and physicians expected a minimum sensitivity of 92% for melanoma. Among physicians, PCPs expected lower specificities for melanoma versus dermatologists (69% vs 84%, P = .002) (Table II). There were no differences in sensitivities or specificities by age, sex, race, or ethnicity. If mobile applications were as sensitive as dermatologists, 59% of physicians and 57% of patients would be extremely likely to recommend or use them, compared to only 4% of physicians and 18% of patients if they were 75% as sensitive.Table IIFor each type of skin cancer, differences in minimum acceptable sensitivities and specificities between physician specialtiesPrimary care Providers (n = 15)Oncologists (n = 30)Dermatologists (n = 30)P-valueSensitivity Melanoma93% [95% CI: 91%-95%]91% [95% CI: 89%-93%]92% [95% CI: 90%-94%].381 Basal Cell Carcinoma90% [95% CI: 87%-93%]88% [95% CI: 86%-91%]88% [95% CI: 85%-90%].597 Squamous Cell Carcinoma91% [95% CI: 88%-93%]88% [95% CI: 85%-91%]89% [95% CI: 87%-92%].485Specificity Melanoma69% [95% CI: 56%-82%]77% [95% CI: 70%-85%]84% [95% CI: 80%-89%].022 Basal Cell Carcinoma74% [95% CI: 66%-81%]77% [95% CI: 72%-83%]79% [95% CI: 74%-84%].485 Squamous Cell Carcinoma72% [95% CI: 62%-82%]77% [95% CI: 70%-83%]80% [95% CI: 75%-84%].290 Open table in a new tab Given public service messaging about skin cancer, it is understandable that patients wanted mobile applications that were highly accurate. Notably, since mobile applications for lay use will likely be used to screen low-risk lesions, unlike more equivocal lesions typically evaluated by dermatologists, to provide comparable levels of absolute performance, sensitivities for mobile applications may be expected to be higher than the 90% to 95% range reported for dermatologists.4Seyed Ahadi M. Firooz A. Rahimi H. Jafari M. Tehranchinia Z. Clinical diagnosis has a high negative predictive value in evaluation of malignant skin lesions.Dermatol Res Pract. 2021; 20216618990https://doi.org/10.1155/2021/6618990Crossref PubMed Scopus (1) Google Scholar That PCPs expected significantly lower specificities for melanoma than dermatologists may be explained by the low threshold PCPs have for referring patients to a dermatologist.5Lowell B.A. Froelich C.W. Federman D.G. Kirsner R.S. Dermatology in primary care: prevalence and patient disposition.J Am Acad Dermatol. 2001; 45: 250-255https://doi.org/10.1067/mjd.2001.114598Abstract Full Text Full Text PDF PubMed Scopus (138) Google Scholar In summary, mobile applications designed to facilitate skin cancer detection do not currently meet physician and patient preferences for minimum thresholds of test specificity and sensitivity. In addition to evidence-based methods of assessing health benefits, patient expectations are an important benchmark for evaluating applications, with skin cancer detection applications ideally judged in the context of both these criteria. Clinical studies may also be undertaken to demonstrate any incremental screening benefit of mobile applications. Since even good mobile applications can merely highlight suspicious lesions, not make diagnoses, it may be appropriate for users with abnormal results to follow up with board-certified dermatologists. None disclosed.