  Deep learning approaches often require huge datasets to achieve good
generalization. This complicates its use in tasks like image-based medical
diagnosis, where the small training datasets are usually insufficient to learn
appropriate data representations. For such sensitive tasks it is also important
to provide the confidence in the predictions. Here, we propose a way to learn
and use probabilistic labels to train accurate and calibrated deep networks
from relatively small datasets. We observe gains of up to 22% in the accuracy
of models trained with these labels, as compared with traditional approaches,
in three classification tasks: diagnosis of hip dysplasia, fatty liver, and
glaucoma. The outputs of models trained with probabilistic labels are
calibrated, allowing the interpretation of its predictions as proper
probabilities. We anticipate this approach will apply to other tasks where few
training instances are available and expert knowledge can be encoded as
probabilities.
