Reply to the letter to the editor ‘What type of man against machine?’ by H. Smith
Authors: H.A. Haenssle, C. Fink
Journal: Annals of Oncology (2018)

Dear Dr Smith, Dear Editor, We appreciate the request [1.Smith H.R. What type of man against machine?.Ann Oncol. 2018; 29: 2023-2024Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar] for a further characterisation of the group of study participants in the challenge ‘Man against machine: diagnostic performance of a deep learning convolutional neural network (CNN) for dermoscopic melanoma recognition in comparison to 58 dermatologists’ [2.Haenssle H.A. Fink C. Schneiderbauer R. et al.Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.Ann Oncol. 2018; 29: 1836-1842Abstract Full Text Full Text PDF PubMed Scopus (661) Google Scholar]. Participants were personally invited by our group via 172 selected email addresses including some members of the International Dermoscopy Society (IDS) which currently looks at approximately 14 000 members from 168 countries. We designed an overview of the available and most important characteristics of the 58 dermatologists who send in their completed voting sheets (Table 1).Table 1Characteristics of study participantsAll participants (n = 58)CNN outperformersaOutperformers were defined as individual dermatologists showing sensitivity/specificity coordinates in study level-I lying above the CNN ROC curve (for reference please view Figure 2A of the original publication [1]). (n = 13)n(%)n(%)Sex (P = 1.0) Male27(46.6%)7(53.8%) Female28(48.3%)6(46.2%) Anonymous3(5.2%)–– Sum58(100%)13(100%)Dermatologists (P = 0.31) Board certified dermatologists34(58.6%)11(84.6%) Dermatologists in residency train21(36.2%)2(15.4%) Anonymous3(5.2%)–– Sum58(100%)13(100%)Specialisation in skin cancer screening (P < 0.01) Yes25(43.1%)12(92.3%) No30(51.7%)1(7.7%) Anonymous3(5.2%)–– Sum58(100%)13(100%)Experience dermoscopy (P = 0.03) <2 years (beginner)17(29.3%)–– 2–5 years (skilled)11(19.0%)2(15.4%) >5 years (expert)30(51.7%)11(84.6%) Sum58(100%)13(100%)Country (P = 0.24) Germany33(56.9%)6(46.2%) USA4(6.9%)2(15.4%) Austria2(3.4%)1(7.7%) Italy2(3.4%)1(7.7%) Sweden2(3.4%)– Argentina1(1.7%)– Brazil1(1.7%)– Czech Republic1(1.7%)– Croatia1(1.7%)– Denmark1(1.7%)– Egypt1(1.7%)– Georgia1(1.7%)1(7.7%) Great Britain1(1.7%)– Greece1(1.7%)1(7.7%) Japan1(1.7%)– Latvia1(1.7%)– Switzerland1(1.7%)1(7.7%) Anonymous3(5.2%)– Sum58(100%)13(100%)All P values by two-tailed Fisher–Freeman–Halton exact test.a Outperformers were defined as individual dermatologists showing sensitivity/specificity coordinates in study level-I lying above the CNN ROC curve (for reference please view Figure 2A of the original publication [1]). Open table in a new tab All P values by two-tailed Fisher–Freeman–Halton exact test. All study participants whose identity is known to us were dermatologists (34 board certified dermatologists and 21 dermatologists in residency training) while 3 participants send in their voting sheets anonymously. Forty-three per cent of participating dermatologists have further specialized in skin cancer screening. The indicated level of experience in dermoscopy was based on the participants’ self-assessment (Table 1). Most participants came from Germany (57%) or the USA (7%) while the remaining 18 dermatologists originated from 15 different countries. Thirteen (22.4%) of the 58 participants showed an outperformance of the CNN in study level-I (diagnostic classification of solely dermoscopic images) as defined by their individual sensitivity/specificity coordinates lying outside of the area under the CNN ROC curve (i.e. individual coordinates lying above of the CNN ROC curve, please also refer to Figure 2A of the original publication [1.Smith H.R. What type of man against machine?.Ann Oncol. 2018; 29: 2023-2024Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar]). Table 1 presents the outperformers’ characteristics in comparison with the overall group of participants. In comparison to the overall group of participants, outperformers were more frequent among (i) board certified dermatologists (P= 0.31), (ii) dermatologists with specialization in skin cancer screening (P< 0.01), and (iii) self-reported experts in dermoscopy (P= 0.03, all P by two-tailed Fisher–Freeman–Halton exact test). As approximately 60% of participants and also of outperformers originate from countries with a high melanoma incidence rate, no conclusion may be drawn. It seems well conceivable that ‘outperformance of the CNN’ goes hand-in-hand with ‘experience and training’ in both skin cancer screening and dermoscopy. None declared.