Paper:
SYSTEMATIC GENERALIZATION: WHAT IS REQUIRED AND CAN IT BE LEARNED?
Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated. We compare both types of models in how much they lend themselves to a particular form of systematic generalization. Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them. Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected. We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization. We find that endto-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization. Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

Review:
This paper presents a targeted empirical evaluation of generalization in models
for visual reasoning. The paper focuses on the specific problem of recognizing
(object, relation, object) triples in synthetic scenes featuring letters and
numbers, and evaluates models' ability to generalize to the full distribution of
such triples after observing a subset that is sparse in the third argument. It
is found that (1) NMNs with full layout supervision generalize better than other
state-of-the art visual reasoning models (FiLM, MAC, RelNet), but (2) without
supervised layouts, NMNs perform little better than chance, and without
supervised question attentions, NMNs perform better than the other models but
fail to achieve perfect generalization.
STRENGTHS
- thorough analysis with a good set of questions
WEAKNESSES
- some peculiar evaluation and presentation decisions
- introduces *yet another* synthetic visual reasoning dataset rather than
  reusing existing ones
I think this paper would have been stronger if it investigated a slightly
broader notion of generalization and had some additional modeling comparisons.
However, I found it interesting and think it successfully addresses the set of
questions it sets out to answer. If it is accepted, there are a few things that
can be done to improve the experiments.
MODELING AND EVALUATION
- Regarding the dataset: the proliferation of synthetic reasoning datasets is
  annoying because it makes it difficult to compare results without downloading
  and re-running a huge amount of code. (The authors have, to their credit, done
  so for this paper.) I think all the experiments here could have been performed
  successfully using either the CLEVR or ShapeWorld rendering engines: while the
  authors note that they require a "large number of different objects", this
  could have been handled by treating e.g. "red circle" and "red square" as
  distinct atomic primitives in questions---the fact that redness is a useful
  feature in both cases is no different from the fact that a horizontal stroke
  detector is useful for lots of letters.
- I don't understand the motivation behind holding out everything on the
  right-hand side. For models that can't tell that the two are symmetric, why
  not introduce sparsity everwhere---hold out some LHSs and relations?
  
- Table 1 test accuracies: arbitrarily reporting "best of 3" for some model /
  dataset pairs and "confidence interval of 5" for others is extremely
  unhelpful: it would be best to report (mean / max / stderr) for 5. Also, it's
  never stated which convidence interval is reported.
- Table 1 baselines: why not run Conv+LSTM and RelNet with easier #rhs/lhs data?
- How many MAC cells are used? This can have significant performance
  implications. I think if you used their code out of the box you'll wind up
  with way bigger structures than you need for this task.
- I'm not sure how faithful the `find` module used here is to the one in the
  literature, and one of the interesting claims in this work is that module
  implementation details matter! The various Hu papers use an attentional
  parameterization; the use of a ReLU and full convolution in Eq. 14 suggest
  that that one here can pass around more general feature maps. This is fine but
  the distinction should be made explicit, and it would be interesting to see
  additional comparisons to an NMN with purely attentional bottlenecks.
- Why do all the experiments after 4.3 use #rhs/lhs of 18? If it was 8 it would
  be possible to make more direct comparisons to the other baseline models.
- The comparison to MAC in 4.2 is unfair in the following sense: the NMN
  effectively gets supervised textual attentions if the right parameters are
  always plugged into the right models, while the MAC model has to figure out
  attentions from scratch. A different way of structuring things would be to
  give the MAC model supervised parameterizations in 4.2, and then move the
  current MAC experiment to 4.3 (since it's doing something analogous to
  "parameterization induction".
  
- The top-right number in Table 4---particularly the fact that it beats MAC and
  sequential NMNs under the same supervision condition---is one of the most
  interesting results in this paper. Most of the work on relaxing supervision
  for NMNs has focused on (1) inducing new question-specific discrete structures
  from scratch (N2NMN) or (2) finding fixed sequential structures that work well
  in general (SNMN and perhaps MAC). The result this paper suggests an
  alternative, which is finding good fixed tree-shaped structures but continuing
  to do soft parameterization like N2NMN.
- The "sharpness ratio" is not super easy to interpret---can't you just report
  something standard like entropy? Fig 4 is unnecessary---just report the means.
- One direction that isn't explored here is the use of Johnson- or Hu-style
  offline learning of a model to map from "sentences" to "logical forms". To the
  extent that NMNs with ground-truth logical forms get 100% accuracy, this turns
  the generalization problem studied here into a purely symbolic one of the kind
  studied in Lake & Baroni 18. Would be interesting to know whether this makes
  things harder (b/c no grounding signal) or easier (b/c seq2seq learning is
  easier.)
PRESENTATION
- Basically all of the tables in this paper are in the wrong place. Move them
  closer to the first metnion---otherwise they're confusing.
- It's conventional in this conference format to put all figure captions below
  the figures they describe. The mix of above and below here makes it hard to
  attach captions to figures.
- Some of the language about how novel the idea of studying generalization in
  these models is a bit strong. The CoGenT split of the CLEVR dataset is aimed
  at answering similar questions. The original Andreas et al CVPR paper (which btw
  appears to have 2 bib entries) also studied generalization to structurally
  novel inputs, and Hu et al. 17 notes that the latent-variable version of this
  model with no supervision is hard to train.
MISCELLANEOUS
- Last sentence before 4.4: "NMN-Chain" should be "NMN-Tree"?
- Recent paper with a better structure-induction technique:
  https://arxiv.org/abs/1808.09942. Worth citing (or comparing if you have
  time!)