  How does the accuracy of deep neural network models trained to classify
clinical images of skin conditions vary across skin color? While recent studies
demonstrate computer vision models can serve as a useful decision support tool
in healthcare and provide dermatologist-level classification on a number of
specific tasks, darker skin is underrepresented in the data. Most publicly
available data sets do not include Fitzpatrick skin type labels. We annotate
16,577 clinical images sourced from two dermatology atlases with Fitzpatrick
skin type labels and open-source these annotations. Based on these labels, we
find that there are significantly more images of light skin types than dark
skin types in this dataset. We train a deep neural network model to classify
114 skin conditions and find that the model is most accurate on skin types
similar to those it was trained on. In addition, we evaluate how an algorithmic
approach to identifying skin tones, individual typology angle, compares with
Fitzpatrick skin type labels annotated by a team of human labelers.
