Paper:
SCAN: LEARNING HIERARCHICAL COMPOSITIONAL VISUAL CONCEPTS
The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry. We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts. If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain. SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner. Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts.

Review:
Summary
---
This paper proposes a new model called SCAN (Symbol-Concept Association Network) for hierarchical concept learning. It trains one VAE on images then another one on symbols and aligns their latent spaces. This allows for symbol2image and image2symbol inference. But it also allows for generalization to new concepts composed from existing concepts using logical operators. Experiments show that SCAN generates images which correspond to provided concept labels and span the space of concepts which match these labels.
The model starts with a beta-VAE trained on images (x) from the relevant domain (in this case, simple scenes generated from DeepMind Lab which vary across a few known dimensions). This is complemented by the SCAN model, which is a beta-VAE trained to reconstruct symbols (y; k-hot encoded concepts like {red, suitcase}) with a slightly modified objective. SCAN optimizes the ELBO plus a KL term which pushes the latent distribution of the y VAE toward the latent distribution of the x (image) VAE. This aligns the latent representations so now a symbol can be encoded into a latent distribution z and decoded as an image.
One nice property of the learned latent representation is that more specific concepts have more specific latent representations. Consider latent distributions z1 and z2 for a more general symbol {red} and a more specific symbol {red, suitcase}. Fewer dimensions of z2 have high variance than dimensions of z1. For example, the latent space could encode red and suitcase in two dimensions (as binary attributes). z1 would have high variance on all dimensions but the one which encodes red and z2 would have high variance on all dimensions but red and suitcase. In the reported experiments some of the dimensions do seem to be interpretable attributes (figure 5 right).
SCAN also pays particular attention to hierarchical concepts. Another very simple model (1d convolution layer) is learned to mimic logical operators. Normally a SCAN encoder takes {red} as input and the decoder reconstructs {red}. Now another model is trained that takes "{red} AND {suitcase}" as input and reconstructs {red, suitcase}. The two input concepts {red} and {suitcase} are each encoded by a pre-trained SCAN encoder and then those two distributions are combined into one by a simple 1d convolution module trained to implement the AND operator (or IGNORE/IN COMMON). This allows images of concepts like {small, red, suitcase} to be generated even if small red suitcases are not in the training data.
Experiments provide some basic verification and analysis of the method:
1) Qualitatively, concept samples are correct and diverse, generating images with all configurations of attributes not specified by the input concept.
2) As SCAN sees more diverse examples of a concept (e.g. suitcases of all colors instead of just red ones) it starts to generate more diverse image samples of that concept.
3) SCAN samples/representations are more accurate (generate images of the right concept) and more diverse (far from a uniform prior in a KL sense) than JMVAE and TELBO baselines.
4) SCAN is also compared to SCAN_U, which uses an image beta-VAE that learned an entangled (Unstructured) representation. SCAN_U performed worse than SCAN
and baselines.
5) Concepts expressed as logical combinations of other concepts generalize well for both the SCAN representation and the baseline representations.
Strengths
---
The idea of concept learning considered here is novel and satisfying. It imposing logical, hierarchical structure on latent representations in a general way. This suggests opportunities for inserting prior information and adds interpretability to the latent space.
Weaknesses
---
I think this paper is missing some important evaluation.
Role/Nature of Disentangled Features not Clear (major):
* Disentangled features seem to be very important for SCAN to work well (SCAN vs SCAN_U). It seems that the only difference between the unstructured (entangled) and the structured (disentangled) visual VAE is the color space of the input (RGB vs HSV). If so, this should be stated more clearly in the main paper. What role did beta-VAE (tuning beta) as opposed to plain VAE play in learning disentangled features?
* What color space was used for the JMVAE and TELBO baselines? Training these with HSV seems especially important for establishing a good comparison, but it would be good to report results for HSV and RGB for all models.
* How specific is the HSV trick to this domain? Would it matter for natural images?
* How would a latent representation learned via supervision perform? (Maybe explicitly align dimensions of z to red/suitcase/small with supervision through some mechanism. c.f. "Discovering Hidden Factors of Variation in Deep Networks" by Cheung et al.)
Evaluation of sample complexity (major):
* One of the main benefits of SCAN is that it works with less training data. There should be a more systematic evaluation of this claim. In particular, I would like to see a Number of Examples vs Performance (Accuracy/Diversity) plot for both SCAN and the baselines.
Minor questions/comments/concerns:
* What do the logical operators learn that the hand-specified versions do not?
* Does training SCAN with the structure provided by the logical operators lead to improved performance?
* There seems to be a mistake in figure 5 unless I interpreted it incorrectly. The right side doesn't match the left side. During the middle stage of training object hues vary on the left, but floor color becomes less specific on the right. Shouldn't object color become less specific?
Prelimary Evaluation
---
This clear and well written paper describes an interesting and novel way of learning a model of hierarchical concepts. It's missing some evaluation that would help establish the sample complexity benefit more precisely (a claimed contribution) and add important details about unsupervised disentangled representations. I would happy to increase my rating if these are addressed.