  We consider machine-learning-based malignancy prediction and lesion
identification from clinical dermatological images, which can be indistinctly
acquired via smartphone or dermoscopy capture. Additionally, we do not assume
that images contain single lesions, thus the framework supports both focal or
wide-field images. Specifically, we propose a two-stage approach in which we
first identify all lesions present in the image regardless of sub-type or
likelihood of malignancy, then it estimates their likelihood of malignancy, and
through aggregation, it also generates an image-level likelihood of malignancy
that can be used for high-level screening processes. Further, we consider
augmenting the proposed approach with clinical covariates (from electronic
health records) and publicly available data (the ISIC dataset). Comprehensive
experiments validated on an independent test dataset demonstrate that i) the
proposed approach outperforms alternative model architectures; ii) the model
based on images outperforms a pure clinical model by a large margin, and the
combination of images and clinical data does not significantly improves over
the image-only model; and iii) the proposed framework offers comparable
performance in terms of malignancy classification relative to three board
certified dermatologists with different levels of experience.
