  For machine learning models to be most useful in numerous sociotechnical
systems, many have argued that they must be human-interpretable. However,
despite increasing interest in interpretability, there remains no firm
consensus on how to measure it. This is especially true in representation
learning, where interpretability research has focused on "disentanglement"
measures only applicable to synthetic datasets and not grounded in human
factors. We introduce a task to quantify the human-interpretability of
generative model representations, where users interactively modify
representations to reconstruct target instances. On synthetic datasets, we find
performance on this task much more reliably differentiates entangled and
disentangled models than baseline approaches. On a real dataset, we find it
differentiates between representation learning methods widely believed but
never shown to produce more or less interpretable models. In both cases, we ran
small-scale think-aloud studies and large-scale experiments on Amazon
Mechanical Turk to confirm that our qualitative and quantitative results
agreed.
