  The effective application of representation learning to real-world problems
requires both techniques for learning useful representations, and also robust
ways to evaluate properties of representations. Recent work in disentangled
representation learning has shown that unsupervised representation learning
approaches rely on fully supervised disentanglement metrics, which assume
access to labels for ground-truth factors of variation. In many real-world
cases ground-truth factors are expensive to collect, or difficult to model,
such as for perception. Here we empirically show that a weakly-supervised
downstream task based on odd-one-out observations is suitable for model
selection by observing high correlation on a difficult downstream abstract
visual reasoning task. We also show that a bespoke metric-learning VAE model
which performs highly on this task also out-performs other standard
unsupervised and a weakly-supervised disentanglement model across several
metrics.
