  General purpose semantic segmentation relies on a backbone CNN network to
extract discriminative features that help classify each image pixel into a
'seen' object class (ie., the object classes available during training) or a
background class. Zero-shot semantic segmentation is a challenging task that
requires a computer vision model to identify image pixels belonging to an
object class which it has never seen before. Equipping a general purpose
semantic segmentation model to separate image pixels of 'unseen' classes from
the background remains an open challenge. Some recent models have approached
this problem by fine-tuning the final pixel classification layer of a semantic
segmentation model for a Zero-Shot setting, but struggle to learn
discriminative features due to the lack of supervision. We propose a recursive
training scheme to supervise the retraining of a semantic segmentation model
for a zero-shot setting using a pseudo-feature representation. To this end, we
propose a Zero-Shot Maximum Mean Discrepancy (ZS-MMD) loss that weighs high
confidence outputs of the pixel classification layer as a pseudo-feature
representation, and feeds it back to the generator. By closing-the-loop on the
generator end, we provide supervision during retraining that in turn helps the
model learn a more discriminative feature representation for 'unseen' classes.
We show that using our recursive training and ZS-MMD loss, our proposed model
achieves state-of-the-art performance on the Pascal-VOC 2012 dataset and
Pascal-Context dataset.
