  Existing skin attributes detection methods usually initialize with a
pre-trained Imagenet network and then fine-tune the medical target task.
However, we argue that such approaches are suboptimal because medical datasets
are largely different from ImageNet and often contain limited training samples.
In this work, we propose Task Agnostic Transfer Learning (TATL), a novel
framework motivated by dermatologists' behaviors in the skincare context. TATL
learns an attribute-agnostic segmenter that detects lesion skin regions and
then transfers this knowledge to a set of attribute-specific classifiers to
detect each particular region's attributes. Since TATL's attribute-agnostic
segmenter only detects abnormal skin regions, it enjoys ample data from all
attributes, allows transferring knowledge among features, and compensates for
the lack of training data from rare attributes. We extensively evaluate TATL on
two popular skin attributes detection benchmarks and show that TATL outperforms
state-of-the-art methods while enjoying minimal model and computational
complexity. We also provide theoretical insights and explanations for why TATL
works well in practice.
