Discrimination between invasive and in situ melanomas using a convolutional neural network
Authors: Martin Gillstedt, Ella Hedlund, John Paoli, Sam Polesie
Journal: Journal of the American Academy of Dermatology (2021)

To the Editor: Melanomas are more frequently detected by patients and not dermatologists, suggesting that the clinical diagnosis most often is uncomplicated.1McGuire S.T. Secrest A.M. Andrulonis R. Ferris L.K. Surveillance of patients for early detection of melanoma: patterns in dermatologist vs patient discovery.Arch Dermatol. 2011; 147: 673-678Google Scholar However, deciding whether a melanoma is in situ (MIS) or invasive may prove more difficult, even for expert dermatologists. A machine learning tool could potentially prove helpful for dermatologists facing this binary classification problem. Convolutional neural networks (CNNs) have recently contributed to significant advances in image classification,2Krizhevsky A. Sutskever I. Hinton G.E. Imagenet classification with deep convolutional neural networks.Commun ACM. 2017; 60: 84-90Google Scholar and several models have been designed to classify different types of skin lesions with outstanding performance in a research setting.3Esteva A. Kuprel B. Novoa R.A. et al.Dermatologist-level classification of skin cancer with deep neural networks.Nature. 2017; 542: 115-118Google Scholar,4Haenssle H.A. Fink C. Schneiderbauer R. et al.Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.Ann Oncol. 2018; 29: 1836-1842Google Scholar In this retrospective investigation, we trained a CNN using a relatively small number of dermatoscopic images. The model did not have any pretrained parameters (Supplemental material available via Mendeley at https://data.mendeley.com/datasets/y883xdgw86/1). A total of 1137 histopathologically verified lesions were included: 515 invasive (45.3%) and 622 MIS (54.7%). All images were obtained from our department from January 2016 to February 2020. The image set was randomized into 3 groups: training set (n = 749), validation set (n = 188), and test set (n = 200). The randomization preserved the proportion of invasive to MIS in each group. The model's performance was compared with that of 7 dermatologists (1 resident physician and 6 board-certified dermatologists) who independently reviewed the same test set of lesions. The model was evaluated using the test set, yielding a receiver operating characteristic (ROC) curve with an area under the curve of 0.76 (95% confidence interval [CI], 0.69-0.83). A combined assessment by the dermatologists on each case was defined by taking the mean of the individual assessments (1 = invasive, 0 = MIS), yielding a score ranging from 0 to 1. Using this score, an ROC curve for the dermatologists' combined assessment was defined, producing an area under the curve of 0.81 (95% CI, 0.75-0.87; P = .080) (Fig 1). Selecting the point where sensitivity and specificity were nearest, the CNN model correctly classified 139 out of 200 images in the test set, yielding an overall accuracy rate of 69.5% (95% CI, 62.6%-75.8%). The corresponding point on the dermatologists' ROC curve had an accuracy of 75.0% (95% CI, 68.4%-80.8%) (Fig 2). When comparing the 2 above-mentioned points for the 200 cases, the dermatologists' combined answer was accurate in 23 cases in which the CNN was wrong. The CNN's answer was accurate in 12 cases in which the dermatologists were wrong (P = .090). A limitation was that only dermatoscopic images were used. In real life, physicians integrate other metadata, including patient history and the clinical image, when determining whether a melanoma is invasive or MIS. Importantly, dysplastic nevi were not included, and the same weight was applied to misclassification of invasive melanomas and MIS. Although the training set used was relatively small, a previous investigation showed that CNN models trained on small datasets can perform well.5Fujisawa Y. Otomo Y. Ogata Y. et al.Deep-learning-based, computer-aided classifier developed with a small dataset of clinical images surpasses board-certified dermatologists in skin tumour diagnosis.Br J Dermatol. 2019; 180: 373-381Google Scholar Before a CNN can be implemented in clinical practice to aid a dermatologist in this classification problem, it needs further refinement and evaluation in a prospective setting. However, it is interesting that the performance level of the CNN model presented here was not much lower than that of the dermatologists, considering that it was trained de novo on merely 749 images. None disclosed.