  We design the first multi-layer disentanglement metric operating at all
hierarchy levels of a structured latent representation, and derive its
theoretical properties. Applied to object-centric representations, our metric
unifies the evaluation of both object separation between latent slots and
internal slot disentanglement into a common mathematical framework. It also
addresses the problematic dependence on segmentation mask sharpness of previous
pixel-level segmentation metrics such as ARI. Perhaps surprisingly, our
experimental results show that good ARI values do not guarantee a disentangled
representation, and that the exclusive focus on this metric has led to
counterproductive choices in some previous evaluations. As an additional
technical contribution, we present a new algorithm for obtaining feature
importances that handles slot permutation invariance in the representation.
