We present two techniques to improve landmark localization in images from partially annotated datasets. Our primary goal is to leverage the common situation where precise landmark locations are only provided for a small data subset, but where class labels for classification or regression tasks related to the landmarks are more abundantly available. First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data. A key aspect of our approach is that errors can be backpropagated through a complete landmark localization model. Second, we propose and explore an unsupervised learning technique for landmark localization based on having a model predict equivariant landmarks with respect to transformations applied to the image. We show that these techniques, improve landmark prediction considerably and can learn effective detectors even when only a small fraction of the dataset has landmark labels. We present results on two toy datasets and four real datasets, with hands and faces, and report new state-of-the-art on two datasets in the wild, e.g. with only 5% of labeled images we outperform previous state-of-the-art trained on the AFLW dataset.
Multi-task CNNs with auxiliary losses were applied by Sina et al. REF for training a localization scheme using partially annotated datasets where accurate landmark locations are only provided for a small data subset, but where class labels for additional related cues are available.
Improving Landmark Localization with Semi-Supervised Learning
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Computer Science