  Representational learning hinges on the task of unraveling the set of
underlying explanatory factors of variation in data. In this work, we operate
in the setting where limited information is known about the data in the form of
groupings, or set membership, where the underlying factors of variation is
restricted to a subset. Our goal is to learn representations which isolate the
factors of variation that are common across the groupings. Our key insight is
the use of cycle consistency across sets(CCS) between the learned embeddings of
images belonging to different sets. In contrast to other methods utilizing set
supervision, CCS can be applied with significantly fewer constraints on the
factors of variation, across a remarkably broad range of settings, and only
utilizing set membership for some fraction of the training data. By curating
datasets from Shapes3D, we quantify the effectiveness of CCS through mutual
information between the learned representations and the known generative
factors. In addition, we demonstrate the applicability of CCS to the tasks of
digit style isolation and synthetic-to-real object pose transfer and compare to
generative approaches utilizing the same supervision.
