Paper:
Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.

Review:
This paper focuses on the zero-shot learning compositional capabilities of modern sequence-to-sequence RNNs.  Through a series of experiments and a newly defined dataset, it exposes the short-comings of current seq2seq RNN architectures.  The proposed dataset, called the SCAN dataset, is a selected subset of the CommonAI navigation tasks data set.  This subset is chosen such that each command sequence corresponds to exactly one target action sequence, making it possible to apply standard seq2seq methods.  Existing methods are then compared based on how accurately they can produce the target action sequence based on the command input sequence.
The introduction covers relevant literature and nicely describes the motivation for later experiments. Description of the model architecture is largely done in the appendix, this puts the focus of the paper on the experimental section. This choice seems to be appropriate, since standard methods are used. Figure 2 is sufficient to illustrate the model to readers familiar with the literature.
The experimental part establishes a baseline using standard seq2seq models on the new dataset, by exploring large variations of model architectures and a large part of the hyper-parameter space.   This papers experimentation sections sets a positive example by exploring a comparatively large space of standard model architectures on the problem it proposes. This search enables the authors to come to convincing conclusions regarding the shortcomings of current models. The paper explores in particular:
1.) Model generalization to unknown data similar to the training set.
2.) Model generalization to data-sequences longer than the training set.
3.) Generalization to composite commands, where  a part of the command is never observed in sequence in the training set.
4.) A recreation of a similar problem in the machine translation context.
These experiments show that modern sequence to sequence models do not solve the systematicity problem, while making clear by application to machine translation, why such a solution would be desirable. The SCAN data-set has the potential to become an interesting test-case for future research in this direction.
The experimental results shown in this paper are clearly compelling in exposing the weaknesses of current seq2seq RNN models.  However, where the paper falls a bit short is in the discussion / outlook in terms of suggestions of how one can go about tackling these shortcomings.