We investigate omni-supervised learning, a special regime of semi-supervised learning in which the learner exploits all available labeled data plus internet-scale sources of unlabeled data. Omni-supervised learning is lowerbounded by performance on existing labeled datasets, offering the potential to surpass state-of-the-art fully supervised methods. To exploit the omni-supervised setting, we propose data distillation, a method that ensembles predictions from multiple transformations of unlabeled data, using a single model, to automatically generate new training annotations. We argue that visual recognition models have recently become accurate enough that it is now possible to apply classic ideas about self-training to challenging realworld data. Our experimental results show that in the cases of human keypoint detection and general object detection, state-of-the-art models trained with data distillation surpass the performance of using labeled data from the COCO dataset alone.
Data distillation by Radosavovic et al. REF aggregates the inferences of multiple transformations of a data point for keypoint detection.
Data Distillation: Towards Omni-Supervised Learning
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Computer Science