Original: S YSTEMATIC G ENERALIZATION : W HAT I S R EQUIRED AND C AN I T B E L EARNED ?
Revised: S YSTEMATIC G ENERALIZATION : W HAT I S R EQUIRED AND C AN I T B E L EARNED ?

Original: Anonymous authors
Revised: AdeptMind Scholar

Original: Paper under double-blind review
Revised: Element AI CIFAR Fellow

Original: A BSTRACT
Revised: A BSTRACT

Original: Numerous models for grounded language understanding have been recently proposed, including (i) generic modules that can beused easily adapted to any given taskwith little adaptation and (ii) intuitively appealing modular models that require background knowledge to be instantiated.
Revised: Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated.

Original: We compare generic and modular models in how much they lend themselves to a particular form of systematic generalization.
Revised: We compare both types of models in how much they lend themselves to a particular form of systematic generalization.

Original: Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them.
Revised: Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them.

Original: Our ﬁndings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected.
Revised: Our ﬁndings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected.

Original: We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization.
Revised: We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization.

Original: We show how end-to-end methods from prior work often learn a wrong layout and a spurious parametrization that do not facilitate systematic generalization.
Revised: We ﬁnd that endto-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization.

Original: Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.
Revised: Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, neural network based models have become the workhorse of natural language understanding and generation.
Revised: In recent years, neural network based models have become the workhorse of natural language understanding and generation.

Original: They empower industrialsystems in machine translation (Wu et al., 2016) and text generation (Kannan et al., 2016)  and show state-of-the-art performance on numerous benchmarks, including Recognizing Textual Entailment(RTE)
Revised: They empower industrial machine translation (Wu et al., 2016) and text generation (Kannan et al., 2016) systems and show state-of-the-art performance on numerous benchmarks including Recognizing Textual Entailment

Original: (Gong et al., 2017), Visual Question Answering(VQA) (Jiang et al., 2018), and Reading Comprehension (Wang et al., 2018).
Revised: (Gong et al., 2017), Visual Question Answering (Jiang et al., 2018), and Reading Comprehension (Wang et al., 2018).

Original: Despite these successes, a growing body of literature suggests that these approaches do not generalize outside of the speciﬁc distributions  they are trained on, something that is necessary for a language understanding system to be widely deployed in the real world.
Revised: Despite these successes, a growing body of literature suggests that these approaches do not generalize outside of the speciﬁc distributions on which they are trained, something that is necessary for a language understanding system to be widely deployed in the real world.

Original: Investigations on the three aforementioned tasks have shown that neural models easily latch onto statistical regularities which are omnipresent in existing datasets (Agrawal et al., 2016; Gururangan et al., 2018; Jia & Liang, 2017) and extremely hard to avoid in large scale data collection.
Revised: Investigations on the three aforementioned tasks have shown that neural models easily latch onto statistical regularities which are omnipresent in existing datasets (Agrawal et al., 2016; Gururangan et al., 2018; Jia & Liang, 2017) and extremely hard to avoid in large scale data collection.

Original: Having learned such dataset-speciﬁc solutions, neural networks fail to deal with examples that are even slightly out of domain, yet are trivial for humans.
Revised: Having learned such dataset-speciﬁc solutions, neural networks fail to make correct predictions for examples that are even slightly out of domain, yet are trivial for humans.

Original: These ﬁndings have been corroborated by a recent investigation on a synthetic instruction-following task (Lake & Baroni, 2018), in which seq2seq models (Sutskever et al., 2014; Bahdanau et al., 2015) have shown little systematicity (Fodor & Pylyshyn, 1988) in how they generalize, i.e. they would not learna general rule on how to compose words and fail spectacularly when  asked to interpret “jump twice” after training on “jump”, “run twice” and “walk twice”.
Revised: These ﬁndings have been corroborated by a recent investigation on a synthetic instruction-following task (Lake & Baroni, 2018), in which seq2seq models (Sutskever et al., 2014; Bahdanau et al., 2015) have shown little systematicity (Fodor & Pylyshyn, 1988) in how they generalize, that is they do not learn general rules on how to compose words and fail spectacularly when for example asked to interpret “jump twice” after training on “jump”, “run twice” and “walk twice”.

Original: An appealing direction to improve the generalization capabilities of neural models is to add modularity and structure to their design to make them structurally resemble the kind of rules they are supposed to learn (Andreas et al., 2016; Gaunt et al., 2017).
Revised: An appealing direction to improve the generalization capabilities of neural models is to add modularity and structure to their design to make them structurally resemble the kind of rules they are supposed to learn (Andreas et al., 2016; Gaunt et al., 2016).

Original: For example, in the Neural Module Network paradigm (NMN, Andreas et al.
Revised: For example, in the Neural Module Network paradigm (NMN, Andreas et al.

Original: , a neural network is assembled from several neural modules , where each module is meant to perform a particular subtask of the input processing, much like a computer program composed of functions.
Revised: , a neural network is assembled from several neural modules , where each module is meant to perform a particular subtask of the input processing, much like a computer program composed of functions.

Original: The NMN approach is intuitively appealing but its widespread adoption has been hindered by the large amount of domain knowledge it requires to decide (Andreas et al., 2016) or predict (Johnson et al., 2017;
Revised: The NMN approach is intuitively appealing but its widespread adoption has been hindered by the large amount of domain knowledge that is required to decide (Andreas et al., 2016) or predict (Johnson et al., 2017;

Original: Hu et al., 2017) how the modules should be created (parametrization) and how they should be connected (layout) based on a natural language utterance.
Revised: Hu et al., 2017) how the modules should be created ( parametrization ) and how they should be connected ( layout ) based on a natural language utterance.

Original: Besides, their performance has often been matched by more traditional neural models, such as FiLM (Perez et al., 2017), Relations Networks (Santoro et al., 2017), and MAC networks (Hudson & Manning, 2018).
Revised: Besides, their performance has often been matched by more traditional neural models, such as FiLM (Perez et al., 2017), Relations Networks (Santoro et al., 2017), and MAC networks (Hudson & Manning, 2018).

Original: Lastly, generalization properties of NMNs, to the best of our knowledge, have not been rigorously studied prior of this work.
Revised: Lastly, generalization properties of NMNs, to the best of our knowledge, have not been rigorously studied prior to this work.

Original: In this work we investigate the impact of explicit modularity and structure on systematic generalizationby studying the generalization of NMNs and contrasting it to those of generic models.
Revised: Here, we investigate the impact of explicit modularity and structure on systematic generalization of NMNs and contrast their generalization abilities to those of generic models.

Original: For this case study we focus on the task of visual question answering (VQA), in particular its simplest binary form when the answer is either “yes” or “no”.
Revised: For this case study, we focus on the task of visual question answering (VQA), in particular its simplest binary form, when the answer is either “yes” or “no”.

Original: Such a binary VQA taskthat we focus on here can be seen as a fundamental task of language understanding, as it requires one to evaluate the truth value of the utterance in a context.
Revised: Such a binary VQA task can be seen as a fundamental task of language understanding, as it requires one to evaluate the truth value of the utterance with respect to the state of the world.

Original: There are manyways to formalize the systematic generalization requirements that are desirable for a VQA model,so we chooseto focus on the following basic requirement: a good model should be able to reason about all possible object combinations despite being trained on a very small subset of them .
Revised: Among many systematic generalization requirements that are desirable for a VQA model, we choose the following basic one: a good model should be able to reason about all possible object combinations despite being trained on a very small subset of them .

Original: We believe that this is a key prerequisite to using VQA models in the real world, because they should be robust at handling unlikely combinations of objects.
Revised: We believe that this is a key prerequisite to using VQA models in the real world, because they should be robust at handling unlikely combinations of objects.

Original: We instantiate our generalization demands in the form of a new synthetic dataset called S patial Q ueries O n O bject P airs ( SQOOP ), in which a model has to performbasic spatial relational reasoning about pairs of randomly scattered letters and digits in the image (e.g. answering the question “ Is there a letter A left of a letter B? ”).
Revised: We implement our generalization demands in the form of a new synthetic dataset, called S patial Q ueries O n O bject P airs (SQOOP), in which a model has to perform spatial relational reasoning about pairs of randomly scattered letters and digits in the image (e.g. answering the question “ Is there a letter A left of a letter B?” ).

Original: The main challenge in SQOOP is that models are evaluated on all possible object pairs, but trained on only a subset of them.
Revised: The main challenge in SQOOP is that models are evaluated on all possible object pairs, but trained on only a subset of them.

Original: Our ﬁrst ﬁnding is that NMNs do generalize better than other neural models whenan appropriate choice of layout and parametrization is made.
Revised: Our ﬁrst ﬁnding is that NMNs do generalize better than other neural models when layout and parametrization are chosen appropriately.

Original: We then investigate which factors contribute to improved generalization performance and ﬁnd that using a  tree layout, as opposed to a chain layout, is crucial for solving the hardest version of our dataset.
Revised: We then investigate which factors contribute to improved generalization performance and ﬁnd that using a layout that matches the task (i.e. a tree layout, as opposed to a chain layout), is crucial for solving the hardest version of our dataset.

Original: Lastly, and perhaps most importantly, we experiment with existing methods for making NMNs more end-to-end by inducing the module layout (Johnson et al., 2017) or learning module parametrization through soft-attention over the question (Hu et al., 2017).
Revised: Lastly, and perhaps most importantly, we experiment with existing methods for making NMNs more end-to-end by inducing the module layout (Johnson et al., 2017) or learning module parametrization through soft-attention over the question (Hu et al., 2017).

Original: Our experiments show that such end-to-end approaches often fail by not converging to tree layouts or by learning a blurred parameterization for modules, which results in poor generalization on the hardest version of our dataset.
Revised: Our experiments show that such end-to-end approaches often fail by not converging to tree layouts or by learning a blurred parameterization for modules, which results in poor generalization on the hardest version of our dataset.

Original: We believe that our ﬁndings challenge the intuition of researchers in the ﬁeld and provide a foundation for improving systematic generalization of neural approaches to language understanding.
Revised: We believe that our ﬁndings challenge the intuition of researchers in the ﬁeld and provide a foundation for improving systematic generalization of neural approaches to language understanding.

Original:  T HE SQOOP D ATASET F OR T ESTING S YSTEMATIC G ENERALIZATION
Revised:  T HE SQOOP D ATASET F OR T ESTING S YSTEMATIC G ENERALIZATION

Original: We perform all experiments of this study on the SQOOP dataset.
Revised: We perform all experiments of this study on the SQOOP dataset.

Original: SQOOP is a minimalistic VQA task that is designed to testone particular type of generalization: the  ability to disentangle the meaning of  relationwords and object words and then compose these meanings in novel contexts to perform basic relational reasoning in a consistent way.
Revised: SQOOP is a minimalistic VQA task that is designed to test the model’s ability to interpret unseen combinations of known relation and object words.

Original: Clearly, given known objects X , Y and a known relation R , a human can easily verify whether or not the objects X
Revised: Clearly, given known objects X , Y and a known relation R , a human can easily verify whether or not the objects X

Original: and Y are in relation R .
Revised: and Y are in relation R .

Original: Some instances of such queries are common in daily life ( is there a cup on the table ), some are extremely rare ( is there a violin under the car ), and some are unlikely but have similar, morecommon counter-parts that are likely  ( is there grass on the frisbee vs is there a frisbee on the grass ).
Revised: Some instances of such queries are common in daily life ( is there a cup on the table ), some are extremely rare ( is there a violin under the car ), and some are unlikely but have similar, more likely counter-parts ( is there grass on the frisbee vs is there a frisbee on the grass ).

Original: Still, a person can easily answer these questions by understanding them as just the composition of the three separate concepts.
Revised: Still, a person can easily answer these questions by understanding them as just the composition of the three separate concepts.

Original: Such compositional reasoning is extremely desirable for our models, andwe have created SQOOP  to test for it.
Revised: Such compositional reasoning skills are clearly required for language understanding models, and SQOOP is explicitly designed to test for them.

Original: Concretely speaking, SQOOP requires observing a 64 × 64 RGB image x and answering a yesno question q = X R Y about whether objects X and Y are in a spatial relation R .
Revised: Concretely speaking, SQOOP requires observing a 64 × 64 RGB image x and answering a yes-no question q = X R Y about whether objects X and Y are in a spatial relation R .

Original: The questions are represented in a redundancy-free X R Y form; we did not aim to make the questions look like natural language.
Revised: The questions are represented in a redundancy-free X R Y form; we did not aim to make the questions look like natural language.

Original: Each image contains 5 randomly chosen and randomly positioned objects.
Revised: Each image contains 5 randomly chosen and randomly positioned objects.

Original: There are 36 objects: the latin letters A-Z and digits 0-9, and there are 4 relations: LEFT OF , RIGHT OF , ABOVE , and BELOW .
Revised: There areobjects: the latin letters A-Z and digits 0-9, and there are 4 relations: LEFT OF , RIGHT OF , ABOVE , and BELOW .

Original: This results in 36 · 35 · 4 = 5040 possible unique questions (we do not allow questions about identical objects).
Revised: This results in 36 · 35 · 4 = 5040 possible unique questions (we do not allow questions about identical objects).

Original: To make negative examples challenging, we ensure that both X and Y of a question are always present in the associated image and that there arealways distractor objects Y (cid:48) (cid:54) =Y and X (cid:48) (cid:54) = X such that X R Y (cid:48) and X (cid:48)
Revised: To make negative examples challenging, we ensure that both X and Y of a question are always present in the associated image and that there are distractor objects Y (cid:48) (cid:54) =

Original: R Y are both true for the image. These extra precautions guarantee that answering a question requires the model to locate all possible X and Y then check if any pair of them are in the relation R. Two SQOOP examples are shown in Figure 2. Our goal is to discover which models can correctly answer questions about all 36 · 35 possible object pairs in SQOOP after having been trained on only a subset. For this purpose we build training sets containing 36 · 4 · k unique questions by sampling k different right-hand-side (RHS) objects Y1 ,
Revised: Y

Original: R x yY
Revised: R x yY

Original: X
Revised: X

Original: R x yY
Revised: R x yY

Original: X
Revised: X

Original: R
Revised: R

Original: X
Revised: X

Original: Y 
Revised: Y and X (cid:48) (cid:54) =

Original: Refer to Section-3.2 for details.
Revised: X such that X R Y (cid:48) and X (cid:48) R Y are both true for the image. These extra precautions guarantee that answering a question requires the model to locate all possible X and Y then check if any pair of them are in the relation R. Two SQOOP examples are shown in Figure 2.

Original: [Figure]
Revised: Our goal is to discover which models can correctly answer questions about all 36 · 35 possible object pairs in SQOOP after having been trained on only a subset. For this purpose we build training sets containing 36 · 4 · k unique questions by sampling k different right-hand-side (RHS) objects Y 1 ,

Original: Y 2 , ..., Y k for each left-hand-side (LHS) object X .
Revised: Y 2 , ..., Y k for each left-hand-side (LHS) object X .

Original: We use this procedure instead of just uniformly sampling object pairs in order to ensure that each object appears in at least one training question, thereby keeping the all versions of the dataset solvable.
Revised: We use this procedure instead of just uniformly sampling object pairs in order to ensure that each object appears in at least one training question, thereby keeping the all versions of the dataset solvable.

Original: We will refer to k as the #rhs/lhs parameter of the dataset.
Revised: We will refer to k as the #rhs/lhs parameter of the dataset.

Original: Our test set is composed from the remaining 36 · 4 · (35 − k ) questions.
Revised: Our test set is composed from the remaining 36 · 4 · (35 − k ) questions.

Original: We generate training and test sets for rhs/lhs values of 1,2,4,8 and 18, as well as a dataset in which both the training and the test set contain all the questions (with different images).
Revised: We generate training and test sets for rhs/lhs values of 1,2,4,8 and 18, as well as a control version of the dataset, #rhs/lhs=35, in which both the training and the test set contain all the questions (with different images).

Original: We will refer to this control version of SQOOP as #rhs/lhs=35. Note that lower #rhs/lhs versions are harder for generalization due to the presence of spurious correlations betweenlhs and rhs objects that models may adapt toin the extreme case of #rhs/lhs=1, a model may learn to predict the rhs object from the lhs.
Revised: Note that lower #rhs/lhs versions are harder for generalization due to the presence of spurious dependencies between the words X and Y to which the models may adapt.

Original: In order to exclude a possible compounding factor of overﬁtting on the training images, all our training sets contain 1 million examples, so for a dataset with  krhs/lhs we generate approximately 10 6 36 · 4 · k  different images per  question. 
Revised: In order to exclude a possible compounding factor of overﬁtting on the training images, all our training sets contain 1 million examples, so for a dataset with #rhs/lhs = k we generate approximately 10 6 / (36 · 4 · k ) different images per unique question. Appendix D contains pseudocode for SQOOP generation.

Original:  M ODELS
Revised:  M ODELS

Original: A great variety of VQA models have been recently proposed in the literature, among which we can distinguish two trends.
Revised: A great variety of VQA models have been recently proposed in the literature, among which we can distinguish two trends.

Original: Some of the recently proposed models, such as FiLM (Perez et al., 2017) and
Revised: Some of the recently proposed models, such as FiLM (Perez et al., 2017) and

Original: Relation Networks (RelNet, Santoro et al. (2017)) are highly generic and do not require any taskspeciﬁc knowledge to be applied on a new dataset.
Revised: Relation Networks (RelNet, Santoro et al. (2017)) are highly generic and do not require any taskspeciﬁc knowledge to be applied on a new dataset.

Original: On the opposite end of the spectrum are modular and structured models, typically ﬂavours of Neural Module Networks (Andreas et al., 2016), that  require some knowledge about the task at hand to be instantiated.
Revised: On the opposite end of the spectrum are modular and structured models, typically ﬂavours of Neural Module Networks (Andreas et al., 2016), that do require some knowledge about the task at hand to be instantiated.

Original: We evaluate  several state-of-theart models in both families.
Revised: Here, we evaluate systematic generalization of several state-of-the-art models in both families.

Original: In all models, the image x is ﬁrst fed through a CNN based network, that we refer to as the stem , to produce a feature-level 3D tensor h x .
Revised: In all models, the image x is ﬁrst fed through a CNN based network, that we refer to as the stem , to produce a feature-level 3D tensor h x .

Original: This is passed through a modelspeciﬁc computation conditioned on the question q , to produce a joint representation
Revised: This is passed through a model-speciﬁc computation conditioned on the question q , to produce a joint representation

Original: h q x .
Revised: h q x .

Original: Lastly, this representation is fed into a fully-connected classiﬁer network to produce logits for prediction.
Revised: Lastly, this representation is fed into a fully-connected classiﬁer network to produce logits for prediction.

Original: Therefore, the main difference between the models we consider is how the computation h q x = model
Revised: Therefore, the main difference between the models we consider is how the computation h q x = model

Original: ( h x , q ) is performed.
Revised: ( h x , q ) is performed.

Original: 3. G ENERIC M ODELS
Revised: 3. G ENERIC M ODELS

Original: We consider four generic models in this paper: CNN+LSTM, FiLM, Relation Network (RelNet), and Memory-Attention-Control (MAC) network.
Revised: We consider four generic models in this paper: CNN+LSTM, FiLM, Relation Network (RelNet), and Memory-Attention-Control (MAC) network.

Original: For CNN+LSTM, FiLM, and RelNet models, the question q is ﬁrst encoded into a ﬁxed-size representation h q using a unidirectional LSTM network.
Revised: For CNN+LSTM, FiLM, and RelNet models, the question q is ﬁrst encoded into a ﬁxed-size representation h q using a unidirectional LSTM network.

Original: CNN+LSTM ﬂattens the 3D tensor h x to a vector and concatenates it with h q to produce h q x  .[Equation]
Revised: CNN+LSTM ﬂattens the 3D tensor h x to a vector and concatenates it with h q to produce h q x : h q x = [ flatten ( h x ); h q ] .

Original: RelNet  uses a network g which is applied to all pairs of feature columns of h x concatenated with the question representation h q , all of which is then pooled to obtain h q x :
Revised: RelNet (Santoro et al., 2017) uses a network g which is applied to all pairs of feature columns of h x concatenated with the question representation h q , all of which is then pooled to obtain h q x :

Original: [Equation]
Revised: [Equation]

Original: where h x ( i ) is the i -th feature column of h x .
Revised: where h x ( i ) is the i -th feature column of h x .

Original: FiLM networks  use N convolutional FiLM blocks applied to h x .
Revised: FiLM networks (Perez et al., 2017) use N convolutional FiLM blocks applied to h x .

Original: A FiLM block is a residual block (He et al., 2016) in which a feature-wise afﬁne transformation (FiLM layer) is inserted after the 2 nd convolutional layer.
Revised: A FiLM block is a residual block (He et al., 2016) in which a feature-wise afﬁne transformation (FiLM layer) is inserted after the 2 nd convolutional layer.

Original: The FiLM layer is conditioned on the question at hand via prediction of the scaling and shifting parameters γ n and β n :
Revised: The FiLM layer is conditioned on the question at hand via prediction of the scaling and shifting parameters γ n and β n :

Original: [Equation]
Revised: [Equation]

Original: where BN stands for batch normalization, ∗ stands for convolution and (cid:12) stands for element-wise multiplications.
Revised: where BN stands for batch normalization (Ioffe & Szegedy, 2015), ∗ stands for convolution and (cid:12) stands for element-wise multiplications.

Original: h nq x is the output of the n -th FiLM block and h
Revised: h nq x is the output of the n -th FiLM block and h

Original: q x = h x .
Revised: q x = h x .

Original: The output of the last FiLM block h
Revised: The output of the last FiLM block h

Original: Nq x undergoes an extra 1 × 1 convolution and max-pooling to produce h q x .
Revised: Nq x undergoes an extra 1 × 1 convolution and max-pooling to produce h q x .

Original: MAC network of Hudson & Manning (2018) produces h q x by repeatedly applying a Memory-AttentionComposition (MAC) cell that is conditioned on the question through an attention mechanism.
Revised: MAC network of Hudson & Manning (2018) produces h q x by repeatedly applying a Memory-Attention-Composition (MAC) cell that is conditioned on the question through an attention mechanism.

Original: The MAC model is quite complex  and we refer the reader to the original paper for details.
Revised: The MAC model is too complex to be fully described here and we refer the reader to the original paper for details.

Original: 3. N EURAL M ODULE N ETWORKS
Revised: 3. N EURAL M ODULE N ETWORKS

Original: Neural Module Networks  (Andreas et al., 2016) are an elegant solution that constructs a questionspeciﬁc network by composing together trainable neural modules, drawing inspiration from symbolic approaches to question answering (Malinowski & Fritz, 2014) .
Revised: Neural Module Networks (NMN) (Andreas et al., 2016) are an elegant approach to question answering that constructs a question-speciﬁc network by composing together trainable neural modules, drawing inspiration from symbolic approaches to question answering (Malinowski & Fritz, 2014).

Original: To answer a question with an NMN, one ﬁrst constructs the computation graph by making the following decisions: (a) how many modules and of which types will be used, (b) how  the moduleswill be connected to each other, and (c) how  these modulesare parametrized based on the question.
Revised: To answer a question with an NMN, one ﬁrst constructs the computation graph by making the following decisions: (a) how many modules and of which types will be used, (b) how will the modules be connected to each other, and (c) how are these modules parametrized based on the question.

Original: We refer to the aspects (a) and (b) of the computation graph as the layout and the aspect (c) as the parametrization .
Revised: We refer to the aspects (a) and (b) of the computation graph as the layout and the aspect (c) as the parametrization .

Original: In the original NMN and in many follow-up works, different module types are used to perform very different computations, e.g. the Find module from Hu et al.
Revised: In the original NMN and in many follow-up works, different module types are used to perform very different computations, e.g. the Find module from Hu et al.

Original: (2017) performs trainable convolutions on the input attention map, whereas the And module from the same paper computes an element-wise maximum for two input attention maps.
Revised: (2017) performs trainable convolutions on the input attention map, whereas the And module from the same paper computes an element-wise maximum for two input attention maps.

Original: In this work we follow the trend of using more homogeneous modules started by Johnson et al.
Revised: In this work, we follow the trend of using more homogeneous modules started by Johnson et al.

Original: (2017), who use only two types of modules: unary and binary, both performing similar computations.
Revised: (2017), who use only two types of modules: unary and binary, both performing similar computations.

Revised: We restrict our study to NMNs with homogeneous modules because they require less prior knowledge to be instantiated and because they performed well in our preliminary experiments despite their relative simplicity.

Original: We go one step further  and retain a single binary module type, using a zero tensor for the second input when only one input is available.
Revised: We go one step further than Johnson et al. and retain a single binary module type, using a zero tensor for the second input when only one input is available.

Original: For simplicity, wealso choose to use exactly three modules and therefore simplify deﬁning the layout of the NMNs to just deﬁning how the modules are connected.
Revised: Additionally, we choose to use exactly three modules, which simpliﬁes the layout decision to just determining how the modules are connected.

Original: Our preliminary experiments have shown that, even after this simpliﬁcation, NMNs are far ahead of other models in terms of generalization.
Revised: Our preliminary experiments have shown that, even after these simpliﬁcations, NMNs are far ahead of other models in terms of generalization.

Original: In the original NMN, the layout and parametrization were set in an ad-hoc manner for each question by analyzing a dependency parse.
Revised: In the original NMN, the layout and parametrization were set in an ad-hoc manner for each question by analyzing a dependency parse.

Original: In the follow-up works (Johnson et al., 2017; Hu et al., 2017), these aspects of the computation are predicted by learnable mechanisms with the goal of reducing the amount of background knowledge required to apply the NMN approach to a new task.
Revised: In the follow-up works (Johnson et al., 2017; Hu et al., 2017), these aspects of the computation are predicted by learnable mechanisms with the goal of reducing the amount of background knowledge required to apply the NMN approach to a new task.

Original: We experiment with the End-to-end NMN (N2NMN) (Hu et al., 2017) paradigm from this family, which predicts the layout with a seq2seq model (Sutskever et al., 2014) and computes the parametrization of the modules using a soft attention mechanism.
Revised: We experiment with the End-to-End NMN (N2NMN) (Hu et al., 2017) paradigm from this family, which predicts the layout with a seq2seq model (Sutskever et al., 2014) and computes the parametrization of the modules using a soft attention mechanism.

Original: Since all the questions in SQOOP have the same structure, we can get away with not employing any seq2seq models and have a trainable layout variable and trainable attention variables for each module.
Revised: Since all the questions in SQOOP have the same structure, we do not employ a seq2seq model but instead have a trainable layout variable and trainable attention variables for each module.

Original: Formally, our NMN is constructed by repeatedly applying a generic neural module f
Revised: Formally, our NMN is constructed by repeatedly applying a generic neural module f

Original: ( θ, γ, h l , h r ) , which takes as inputs the shared parameters θ , the question-speciﬁc parametrization γ and the lefthand side and right-hand side inputs h l and h r .
Revised: ( θ, γ, s 0 , s 1 ) , which takes as inputs the shared parameters θ , the question-speciﬁc parametrization γ and the lefthand side and right-hand side inputs s 0 and s 1 .

Original: M such modules are connected and conditioned on a question q = ( q 1 , q 2 , q 3 ) as follows:
Revised: Three such modules are connected and conditioned on a question q = ( q 1 , q 2 , q 3 ) as follows:

Original: [Equation]
Revised: [Equation]

Original: In the equations above, h − 1 = 0 is the zero tensor input, h 

Revised: In the equations above, s − 1 = 0 is the zero tensor input, s

Original: = h x are the image features outputted by the stem, and e is the embedding table for the questions words, and we refer to A

Original: =
Revised: =

Original:  ( α k,i ) and 
Revised: h x are the image features outputted by the stem, e is the embedding table for question words. k ∈ { 1 , 2 , 3 } is the module number, s k is the output of the k -th module and s mk are its left ( m = 0 ) and right ( m = 1 ) inputs.

Original: T =
Revised: We refer to A =

Original: ( τ k,i 0 , τ k,i 1 ) as the parametrization attention matrix and the layout tensor respectively.
Revised: ( α k,i ) and T = ( τ k,jm ) as the parametrization attention matrix and the layout tensor respectively.

Original: We perform our experiments with  the Find module from Hu et al.
Revised: We experiment with two choices for the NMN’s generic neural module: the Find module from Hu et al.

Original: (2017) and the Residual module from Johnson et al.
Revised: (2017) and the Residual module from Johnson et al.

Original: (2017) with very minor modiﬁcations - we use 64 dimensional CNNs in our

Original: Residual blocks since our dataset consists of 64 × 64 images.

Original: The equations for the Residual module are as follows:
Revised: The equations for the Residual module are as follows:

Original: [Equation]
Revised: [Equation]

Original: and for Find module as follows:
Revised: and for Find module as follows:

Original: [Equation]
Revised: [Equation]

Original: In  formulas above  W 1 , W 2 , W 3 are convolution weights, and  b 1 , b 2 , b 3 are biases.
Revised: In the formulas above all W ’s stand for convolution weights, and all b ’s are biases.

Revised: Equations 10 and 13 should be understood as taking vectors γ k and θ respectively and chunking them into weights and biases.

Original: The main difference between Residual and Find is that in Residual all parameters depend on the questions words, where as in Find convolutional weights are the same for all questions, and only the element-wise multipliers γ  vary based on the question.
Revised: The main difference between Residual and Find is that in Residual all parameters depend on the questions words (hence θ is omitted from the signature of f Residual ), where as in Find convolutional weights are the same for all questions, and only the element-wise multipliers γ k vary based on the question.

Original: We note that the speciﬁc Find module we use in this work is slightly different from the one used in (Hu et al., 2017) in that it outputs a feature tensor, not just an attention map.
Revised: We note that the speciﬁc Find module we use in this work is slightly different from the one used in (Hu et al., 2017) in that it outputs a feature tensor, not just an attention map.

Original: This change was required in order to connect multiple Find modules in the same way as we connect multiple residual ones.
Revised: This change was required in order to connect multiple Find modules in the same way as we connect multiple residual ones.

Original: Based on the generic NMN model described above, we experiment with several speciﬁc architectures as shown inFigure 1. Each of the models uses M = 3 modules, which are connected and parametrized differently.
Revised: Based on the generic NMN model described above, we experiment with several speciﬁc architectures that differ in the way the modules are connected and parametrized (see Figure 1).

Original: In NMN-Chain  modules form a sequential chain.
Revised: In NMN-Chain the modules form a sequential chain.

Original: Modules 1, 2 andare parametrized based on the ﬁrst object word, second object word and the relation word respectively, which is achieved by setting the attention  α 1 , α 2 , α 3 to the corresponding one-hot vectors.
Revised: Modules 1, 2 and 3 are parametrized based on the ﬁrst object word, second object word and the relation word respectively, which is achieved by setting the attention maps α 1 , α 2 , α 3 to the corresponding one-hot vectors.

Original: We also experiment with giving the image features h x as the right-hand side input to all 3 modules and call the resulting model NMN-Chain-Shortcut.
Revised: We also experiment with giving the image features h x as the right-hand side input to all 3 modules and call the resulting model NMN-ChainShortcut .

Original: NMN-Tree is similar to NMN-Chain in that the attention vectors are similarly hard-coded, but we change the connectivity between the modules to be tree-like.
Revised: NMN-Tree is similar to NMN-Chain in that the attention vectors are similarly hardcoded, but we change the connectivity between the modules to be tree-like.

Original: Stochastic N2NMN follows the N2NMN approach by Hu et al.
Revised: Stochastic N2NMN follows the N2NMN approach by Hu et al.

Original: (2017) for inducting layout.
Revised: (2017) for inducing layout.

Original: We treat the layout T as a stochastic latent variable.
Revised: We treat the layout T as a stochastic latent variable.

Original: T is allowed to take two values: T tree as in NMN-Tree, and T chain as in NMN-Chain.
Revised: T is allowed to take two values: T tree as in NMN-Tree, and T chain as in NMN-Chain.

Original: We calculate the output probabilities by marginalizing out the layout i.e. probability of answer being “yes” is computed as p ( yes | x, q ) = (cid:80) T ∈{ T tree ,T chain } p ( yes | T, x, q ) p ( T ) .
Revised: We calculate the output probabilities by marginalizing out the layout i.e. probability of answer being “yes” is computed as p ( yes | x, q ) = (cid:80) T ∈{ T tree ,T chain } p ( yes | T, x, q ) p ( T ) .

Original: Lastly, Attention N2NMN uses the N2NMN method for learning parametrization (Hu et al., 2017).
Revised: Lastly, Attention N2NMN uses the N2NMN method for learning parametrization (Hu et al., 2017).

Original: Is structured just like NMN-Tree but has α k computed as softmax(˜ α k ) , where ˜ α k is a trainable vector.
Revised: It is structured just like NMN-Tree but has α k computed as softmax(˜ α k ) , where ˜ α k is a trainable vector.

Original: We use Attention N2NMN only with the Find module because using it with the Find would involve a highly non-standard interpolation between convolutional weights.
Revised: We use Attention N2NMN only with the Find module because using it with the Residual module would involve a highly non-standard interpolation between convolutional weights.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we aimed to: (a) understand which models are capable of exhibiting systematic generalization as required by SQOOP , and (b) understand whether it is possible to induce, in an end-to-end way, the successful architectural decisions that distinguish the best models.
Revised: In our experiments we aimed to: (a) understand which models are capable of exhibiting systematic generalization as required by SQOOP, and (b) understand whether it is possible to induce, in an end-to-end way, the successful architectural decisions that lead to systematic generalization.

Original: All models share the same stem architecture which consists of 6(8 for Relations Networks) layers of convolution, batch normalization and max pooling.
Revised: All models share the same stem architecture which consists of 6 layers of convolution (8 for Relation Networks), batch normalization and max pooling.

Original: The input to the stem is a 64 × 64 × 3 image, and the feature dimension used throughout the stem is 64.
Revised: The input to the stem is a 64 × 64 × 3 image, and the feature dimension used throughout the stem is 64.

Original: Further details can be found in Appendix
Revised: Further details can be found in Appendix

Original: A.
Revised: A.

Original: The code for all experiments will be released in the near future.
Revised: The code for all experiments will be released in the near future.

Original: 4. W HICH M ODELS G ENERALIZE B ETTER ?
Revised: 4. W HICH M ODELS G ENERALIZE B ETTER ?

Original: We report the performance for all models on datasets of varying difﬁculty in Figure 3.
Revised: We report the performance for all models on datasets of varying difﬁculty in Figure 3.

Revised: Our ﬁrst observation is that the modular and tree-structured NMN-Tree model exhibits strong systematic generalization.

Revised: Both versions of this model, with Residual and Find modules, robustly solve all versions of our dataset, including the most challenging #rhs/lhs=1 split.

Revised: The results of NMN-Tree should be contrasted with those of generic models.

Original: 2 out ofmodels (Conv+LSTM and RelNet) were not able to learn to answer all SQOOP questions, no matter howhigh easy the split was (for high #rhs/lhs Conv+LSTM overﬁtted and RelNet did not train).
Revised: 2 out of 4 models (Conv+LSTM and RelNet) are not able to learn to answer all SQOOP questions, no matter how easy the split was (for high #rhs/lhs Conv+LSTM overﬁtted and RelNet did not train).

Original: The results of other two models, MAC and FiLM, are similar.
Revised: The results of other two models, MAC and FiLM, are similar.

Original: Both models are clearly able to solve the SQOOP task, as suggested by their almost perfect < 1% error rate on the control #rhs/lhs=35 split, yet they struggle to generalize on splits with lower #rhs/lhs.
Revised: Both models are clearly able to solve the SQOOP task, as suggested by their almost perfect < 1% error rate on the control #rhs/lhs=35 split, yet they struggle to generalize on splits with lower #rhs/lhs.

Original: In particular, we observe 13 .
Revised: In particular, we observe 13 .

Original: 67 ± 9 .
Revised: 67 ± 9 .

Original: 97% errors for MAC and a 34 .
Revised: 97% errors for MAC and a 34 .

Original: 73 ± 4 .
Revised: 73 ± 4 .

Original: 61% errors for FiLM on the hardest #rhs/lhs=1 split.
Revised: 61% errors for FiLM on the hardest #rhs/lhs=1 split.

Original: For the splits of intermediate difﬁculty we saw the error rates of both models decreasing as we increased the #rhs/lhs ratio from 2 to 18.
Revised: For the splits of intermediate difﬁculty we saw the error rates of both models decreasing as we increased the #rhs/lhs ratio from 2 to 18.

Original: Interestingly, even with 18 #rhs/lhs some MAC and FiLM runs result in a test error rate of ∼ 2% .
Revised: Interestingly, even with 18 #rhs/lhs some MAC and FiLM runs result in a test error rate of ∼ 2% .

Original: Given the simplicity and minimalism of SQOOP questions, we believe that these results should be considered a failure to pass the SQOOP test for both MAC and FiLM.
Revised: Given the simplicity and minimalism of SQOOP questions, we believe that these results should be considered a failure to pass the SQOOP test for both MAC and FiLM.

Original: That said, we note a difference in how exactly FiLM and MAC fail on #rhs/lhs=1: in several runs ( out of 15) MAC exhibited a strong generalization performance ( ∼ 0 .
Revised: That said, we note a difference in how exactly FiLM and MAC fail on #rhs/lhs=1: in several runs (3 out of 15) MAC exhibits a strong generalization performance ( ∼ 0 .

Original: 5% error rate) , whereas in all runs of FiLM the error rate was about 30% .
Revised: 5% error rate), whereas in all runs of FiLM the error rate is about 30% .

Original: We examined the successful MAC models and found that those that perform well converged to a successful setting of the control attention weights, that is the weights with which MAC units  attend to  questions words.
Revised: We examine the successful MAC models and ﬁnd that they converge to a successful setting of the control attention weights, where speciﬁc MAC units consistently attend to the right questions words.

Original: In particular, MAC models that generalize strongly for each question would have a unit focusing strongly on X and a unit focusing strongly on Y . The reader is referred to Appendix B for more details on MAC results.
Revised: In particular, MAC models that generalize strongly for each question seem to have a unit focusing strongly on X and a unit focusing strongly on Y (see Appendix B for more details).

Original: The results of the generic models that we described above should be contrasted with the performance of the modular and tree-structured NMN-Tree model. Both versions of this model, with Residual and Find modules, robustly solve all versions of our dataset, including the most challenging #rhs/lhs=1 split. As MAC was the strongest competitor of NMN-Tree across generic models, we have performed an ablation study for this model, in which we varied the number of modules and hidden units, as well as experimented with weight decay.
Revised: As MAC was the strongest competitor of NMN-Tree across generic models, we perform an ablation study for this model, in which we vary the number of modules and hidden units, as well as experiment with weight decay.

Original: These modiﬁcations have not resulted in any signiﬁcant reduction of the gap between MAC and NMN-Tree.
Revised: These modiﬁcations do not result in any signiﬁcant reduction of the gap between MAC and NMN-Tree.

Original: Interestingly, we found that using the default high number of MAC units, namely 12, had been helpful, possibly  it made it more likely that some units are initialized to focus on X and Y words (see Appendix B for details).
Revised: Interestingly, we ﬁnd that using the default high number of MAC units, namely 12, is helpful, possibly because it increases the likelihood that at least one unit converges to focus on X and Y words (see Appendix B for details).

Original: 4. W HAT IS E SSENTIAL TO S TRONG G ENERALIZATION OF NMN?
Revised: 4. W HAT IS E SSENTIAL TO S TRONG G ENERALIZATION OF NMN?

Revised: The superior generalization of NMN-Tree raises the following question: what is the key architectural difference between NMN-Tree and generic models that explains the performance gap between them?

Revised: We consider two candidate explanations.

Original: The NMN-Tree model differs from the generic models in that it  is  built from modules that are parametrized by question words directly, side-stepping a language encoder.
Revised: First, the NMN-Tree model differs from the generic models in that it does not use a language encoder and is instead built from modules that are parametrized by question words directly.

Original: Besides that, NMNTree is structured in a particular way, with the idea that modules 1 and 2 may learn to locate objects and module 3 can learn to reason about object locations independently of their identities.
Revised: Second, NMN-Tree is structured in a particular way, with the idea that modules 1 and 2 may learn to locate objects and module 3 can learn to reason about object locations independently of their identities.

Original: To understand which of the two differences is responsible for the superior generalization, we compare the performance of the NMN-Tree, NMN-Chain and NMN-Chain-Shortcut models (see Figure 1).
Revised: To understand which of the two differences is responsible for the superior generalization, we compare the performance of the NMN-Tree, NMN-Chain and NMNChain-Shortcut models (see Figure 1).

Revised: These 3 versions of NMN are similar in that none of them are using a language encoder, but they differ in how the modules are connected.

Original: The results in Figure 3 show that for both Find and Residual module architectures that we consider in this paper, using a tree layout is absolutely crucial (and sufﬁcient) for generalization.
Revised: The results in Figureshow that for both Find and Residual module architectures, using a tree layout is absolutely crucial (and sufﬁcient) for generalization, meaning that the generalization gap between NMN-Tree and generic models can not be explained merely by the language encoding step in the latter.

Original: In particular,
Revised: In particular,

Original: NMN-Chain models perform barely above random chance, doing even worse than generic models on the #rhs/lhs=1 version of the dataset and dramatically failing even on the easiest #rhs/lhs=18 split. 
Revised: NMN-Chain models perform barely above random chance, doing even worse than generic models on the #rhs/lhs=1 version of the dataset and dramatically failing even on the easiest #rhs/lhs=18 split. [Figure] generalization of NMN-Chain-Shortcut.

Revised: Means and standard deviations after at leastruns are reported.

Original: This is in stark contrast with NMN-Tree models that exhibits nearly perfect performance on the hardest #rhs/lhs=1 split.
Revised: This is in stark contrast with NMN-Tree models that exhibits nearly perfect performance on the hardest #rhs/lhs=1 split.

Original: As a sanity check we trained NMN-Chain models on the vanilla #rhs/lhs= split.
Revised: As a sanity check we train NMN-Chain models on the vanilla #rhs/lhs= split.

Original: We found that NMN-Chainmodel has little difﬁculty to learn to answer SQOOP questions when it sees all of them atthe training time, even though it  showsvery poor generalization in our other experiments.
Revised: We ﬁnd that NMN-Chain has little difﬁculty learning to answer SQOOP questions when it sees all of them at training time, even though it previously shows poor generalization when testing on unseen examples.

Original: Interestingly, NMN-Chain-Shortcut performed much better than NMN-Chain and quite similarly to generic models.
Revised: Interestingly, NMN-Chain-Shortcut performs much better than NMN-Chain and quite similarly to generic models.

Original: We ﬁnd it remarkable that such a slight change in the model  as addingthe shortcut connections from image features h x to thechain modules results in a drastic change in generalization performance.
Revised: We ﬁnd it remarkable that such a slight change in the model layout as adding shortcut connections from image features h x to the modules results in a drastic change in generalization performance.

Original: In an attempt to understand why NMN-Chain generalizes so poorly we compared the test set responses of the 5 NMN-Chain models trained on #rhs/lhs= split.
Revised: In an attempt to understand why NMN-Chain generalizes so poorly we compare the test set responses of the 5 NMN-Chain models trained on #rhs/lhs=1 split.

Original: Notably, there was very little agreement between predictions of these 5 runs (Fleiss κ = 0 . 05 ), suggesting that NMN-Chain performs rather randomly outside of the training set.[Figure] of chains with shortcuts. Means and standard deviations after at least 5 runs are reported.
Revised: Notably, there was very little agreement between predictions of these 5 runs (Fleiss κ = 0 . 05 ), suggesting that NMN-Chain performs rather randomly outside of the training set.

Original: 4. C AN THE R IGHT K IND OF NMN B E I NDUCED ?
Revised: 4. C AN THE R IGHT K IND OF NMN B E I NDUCED ?

Original: The strong generalization of the NMN-Treemodel is impressive, but a signiﬁcant amount of prior knowledge about the task was required to come up with the successful layout and parametrization used in this model.
Revised: The strong generalization of the NMN-Tree is impressive, but a signiﬁcant amount of prior knowledge about the task was required to come up with the successful layout and parametrization used in this model.

Original: We therefore investigate whether the amount of such prior knowledge can be reduced by ﬁxing one of these structural aspects and inducing another.
Revised: We therefore investigate whether the amount of such prior knowledge can be reduced by ﬁxing one of these structural aspects and inducing the other.

Original: 4.3. L AYOUT I NDUCTION
Revised: 4.3. L AYOUT I NDUCTION

Original: In our layout induction experiments, we use the Stochastic N2NMN model which treats the layout as a stochastic latent variable with two values ( T tree and T chain , see Section 3.2 for details).
Revised: In our layout induction experiments, we use the Stochastic N2NMN model which treats the layout as a stochastic latent variable with two values ( T tree and T chain , see Section 3.2 for details).

Original: We experiment with N2NMNs using both Find and Residual modules and report results with different initial conditions, p 0 ( tree ) ∈ 0 . 1 , 0 . 5 , 0 . 9 .
Revised: We experiment with N2NMNs using both Find and Residual modules and report results with different initial conditions, p 0 ( tree ) ∈ 0 . 1 , 0 . 5 , 0 . 9 .

Original: We believe that the initial probability p 0 ( tree ) = [-
-]
Revised: We believe that the initial probability p 0 ( tree ) = [-
-]0 . [Figure] [Figure] [Figure]

Original: .should not be considered small, as in more challenging datasets the space of layouts would be exponentially large, and sampling the right layout in 10% of all cases could be considered a very lucky initialization.
Revised: [Table] should not be considered small, since in more challenging datasets the space of layouts would be exponentially large [Table] and sampling the right layout in 10% of all cases should be considered a very lucky initialization.

Original: We repeat all experiments on #rhs/lhs=1 and on #rhs/lhs=18 splits, the former to study generalization, and the latter to control whether the failures on #rhs/lhs=1 are caused speciﬁcally by the difﬁculty of this split.
Revised: We repeat all experiments on #rhs/lhs=1 and on #rhs/lhs=18 splits, the former to study generalization, and the latter to control whether the failures on #rhs/lhs=1 are caused speciﬁcally by the difﬁculty of this split.

Original: The results (see Table 1) show that the success of layout induction (i.e. converging to a p ( tree ) close to 0 .
Revised: The results (see Table 1) show that the success of layout induction (i.e. converging to a p ( tree ) close to 0 .

Original: 9 ) depends in a complex way on all the factors that we considered in our experiments.
Revised: 9 ) depends in a complex way on all the factors that we considered in our experiments.

Original: The initialization has the most inﬂuence: models initialized with p 0 ( tree ) = 0 .
Revised: The initialization has the most inﬂuence: models initialized with p 0 ( tree ) = 0 .

Original: 1 typically do not converge to a tree (exception being experiments with Residual module on #rhs/lhs=18, in which 3 out of 5 runs converged to a solution with a high p ( tree ) ).
Revised: 1 typically do not converge to a tree (exception being experiments with Residual module on #rhs/lhs=18, in which 3 out of 5 runs converged to a solution with a high p ( tree ) ).

Original: Likewise, models initialized with p 0 ( tree ) = 0 .
Revised: Likewise, models initialized with p 0 ( tree ) = 0 .

Original: 9 always stay in a regime with a high p ( tree ) .
Revised: 9 always stay in a regime with a high p ( tree ) .

Original: In the intermediate setting of p 0 ( tree ) = 0 .
Revised: In the intermediate setting of p 0 ( tree ) = 0 .

Original: 5 we observe differences in behaviors for Residual and Find modules.
Revised: 5 we observe differences in behaviors for Residual and Find modules.

Original: In particular, N2NMN based on Residual modules stays spurious with p ( tree ) = 0 .
Revised: In particular, N2NMN based on Residual modules stays spurious with p ( tree ) = 0 .

Original: 5 ± 0 .
Revised: 5 ± 0 .

Original: when #rhs/lhs=1, whereas N2NMN based on Find modules always converges to a tree.[Figure] [Figure] [Figure]
Revised: when #rhs/lhs=1, whereas N2NMN based on Find modules always converges to a tree.

Original: [Table]
Revised: [Table]

Original: One counterintuitive result in Table 1 is that  Stochastic N2NMNs with Residual modules that were trained with p 0 ( tree ) = 0 .
Revised: One counterintuitive result in Table 1 is that for the Stochastic N2NMNs with Residual modules, trained with p 0 ( tree ) = 0 .

Original: 5 and #rhs/lhs=1 make just 1 .
Revised: 5 and #rhs/lhs=1, make just 1 .

Original: 64 ± 1 .
Revised: 64 ± 1 .

Original: 79% errors on the generalization set despite being spurious mixtures between a tree and a chain.
Revised: 79% test error despite never resolving the layout uncertainty through training ( p 200 K ( tree ) = 0 .

Original: Our explanation for this phenomenon is as follows: when connected in a tree, modules of such spurious models generalize well, and when connected as a chain they generalize poorly.

Original: The output distribution of the whole model is thus a mixture of the mostly correct p ( y | T = T tree , x, q ) and mostly random p ( y | T = T chain , x, q ) . We veriﬁed our reasoning by explicitly evaluating test accuracies for p ( y | T = T tree , x, q ) and p ( y | T = T chain , x, q ) , and we found them to be around 99% and 60% respectively, conﬁrming our hypothesis.
Revised: 56 ± 0 . 06 ).

Original: As a result the predictions of the spurious models with p ( tree )

Original: ≈ 0 .

Original: 5 have lower conﬁdence than those of sharp tree models, as indicated by the high log loss of 0 . 27 ± 0 . We visualize the progress of structure induction for the Residual module with p 0 ( tree ) = 0 . 5 in Figure 4 which shows how p ( tree ) saturates to 1.0 for #rhs/lhs=18 and remains around 0.5 when #rhs/lhs=1.
Revised: We offer an investigation of this result in Appendix C

Original: [Table]
Revised: [Table]

Original:  P ARAMETRIZATION I NDUCTION
Revised: 4.3. P ARAMETRIZATION I NDUCTION

Original: Next
Revised: Next

Original: [Table]
Revised: [Table]

Original: we experiment with the Attention N2NMN model (see Section 3.2) in which the parametrization is learned for each module as an attention-weighted average of word embeddings.
Revised: we experiment with the Attention N2NMN model (see Section 3.2) in which the parametrization is learned for each module as an attention-weighted average of word embeddings.

Original: In these experiments [Table] we ﬁx the layout to be tree-like and sample the pre-softmax attention weights ˜
Revised: In these experiments, we ﬁx the layout to be tree-like and sample the pre-softmax attention weights ˜

Original: α
Revised: α

Original: from a uniform distribution U [0; 1] .
Revised: from a uniform distribution U [0; 1] .

Original: As in the layout induction investigations
Revised: As in the layout induction investigations, we experiment with several

Original: [Table] we experiment with several SQOOP splits [Table] namely we try #rhs/lhs ∈ { 1 , 2 , 18 } .
Revised: SQOOP splits, namely we try #rhs/lhs ∈ { 1 , 2 , 18 } .

Original: The results (reported in Table 2) show that Attention N2NMN fails dramatically on #rhs/lhs=1 but quickly catches up as soon as #rhs/lhs is increased to 2.
Revised: The results (reported in Table 2) show that Attention N2NMN fails dramatically on #rhs/lhs=1 but quickly catches up as soon as #rhs/lhs is increased to 2.

Original: Notably, 9 out of 10 runs on #rhs/lhs=2 resulted in almost perfect performance, and 1 run completely failed to generalize (26% error rate), resulting in a high 8 .
Revised: Notably, 9 out of 10 runs on #rhs/lhs=2 result in almost perfect performance, andrun completely fails to generalize (26% error rate), resulting in a high 8 .

Original: 18% variance of the mean error rate.
Revised: 18% variance of the mean error rate.

Original: All 10 runs on the split with 18 rhs/lhs generalized ﬂawlessly.
Revised: All 10 runs on the split with 18 rhs/lhs generalize ﬂawlessly.

Original: We furthermore inspected the learned attention weights and found that for typical successful runs, module 3 focuses on the relation word, whereas modules 1 and 2 focus on different object words (see Figure 6) while still focusing on the relation word.
Revised: Furthermore, we inspect the learned attention weights and ﬁnd that for typical successful runs, module 3 focuses on the relation word, whereas modules 1 and 2 focus on different object words (see Figure 6) while still focusing on the relation word.

Original: To better understand the relationship between successful layout induction and generalization, we deﬁne an attention quality metric κ = min w ∈{ X,Y } max k ∈ 1 , 2 α k,w / (1 − α k,R ) .
Revised: To better understand the relationship between successful layout induction and generalization, we deﬁne an attention quality metric [Equation]

Original: Intuitively, κ is large when for each word w ∈ X, Y there is a module i that focuses mostly on this word.
Revised: Intuitively, κ is large when for each word w ∈ X, Y there is a module i that focuses mostly on this word.

Original: The renormalization by 1 / (1 − α i,R ) is necessary to factor out the amount of attention that modules 1 and 2 assign to the relation word.
Revised: The renormalization by 1 / (1 − α k,R ) is necessary to factor out the amount of attention that modules 1 and 2 assign to the relation word.

Original: For the ground-truth parametrization that we use for NMN-Tree κ takes a value of 1, and if both modulesand 2 focus on X, completely ignoring Y, κ equals 0.
Revised: For the ground-truth parametrization that we use for NMN-Tree κ takes a value of 1, and if both modules 1 and 2 focus on X, completely ignoring Y, κ equals 0.

Original: The scatterplot of the test error rate versus κ (Figure 5) shows that for #rhs/lhs=1 high generalization is strongly associated with higher κ , meaning that it is indeed necessary to have different modules strongly focusing on different object words in order to generalize in this most challenging setting.
Revised: The scatterplot of the test error rate versus κ (Figure 5) shows that for #rhs/lhs=1 high generalization is strongly associated with higher κ , meaning that it is indeed necessary to have different modules strongly focusing on different object words in order to generalize in this most challenging setting.

Original: Interestingly, for #rhs/lhs=2 we see a lot of cases where N2NMN generalizes well despite attention being rather spurious ( κ ≈ 0 . 6 ).
Revised: Interestingly, for #rhs/lhs=2 we see a lot of cases where N2NMN generalizes well despite attention being rather spurious ( κ ≈ 0 . 6 ).

Original: In order to put Attention N2NMN results in context were compare them to those of MAC (see Table 2).
Revised: In order to put Attention N2NMN results in context we compare them to those of MAC (see Table 2).

Original: Such a comparison can be of interest because both models perform attention over the question.
Revised: Such a comparison can be of interest because both models perform attention over the question.

Original: We note that Attention N2NMN succeeds in 9 out of 10 cases on the #rhs/lhs=2 split, much more often than 1 success out of 10 observed for MAC 1 .
Revised: For 1 rhs/lhs MAC seems to be better on average, but as we increase #rhs/lhs to 2 we note that Attention N2NMN succeeds in 9 out of 10 cases on the #rhs/lhs=2 split, much more often than 1 success out of 10 observed for MAC 1 .

Original: This result suggests that Attention N2NMNs retains some of the strong generalization potential of NMNs with hard-coded parametrization.
Revised: This result suggests that Attention N2NMNs retains some of the strong generalization potential of NMNs with hard-coded parametrization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Revised: The notion of systematicity was originally introduced by (Fodor & Pylyshyn, 1988) as the property of human cognition whereby “the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents”.

Revised: They illustrate this with an example that no English speaker can understand the phrase “John loves the girl” without being also able to understand the phrase “the girl loves John”.

Revised: The question of whether or not connectionist models of cognition can account for the systematicity phenomenon has been a subject of a long debate in cognitive science (Fodor & Pylyshyn, 1988; Smolensky, 1987; Marcus, 1998; 2003; Calvo & Colunga, 2003).

Revised: Most recently Lake & Baroni (2018) and Loula et al.

Original:  Our  study is inspired by  priorsubstantial evidence that neural networks do not generalize on  language understanding by performing systematic reasoning, but instead latch on to dataset-speciﬁc regularities.
Revised: (2018) have shown that lack of systematicity in the generalization is still a concern for the modern seq2seq models. Our ﬁndings about the weak systematic generalization of generic VQA models corroborate the aforementioned seq2seq results. We also go beyond merely stating negative generalization results and showcase the high systematicity potential of adding explicit modularity and structure to modern deep learning models. Besides the theoretical appeal of systematicity, our study is inspired by highly related prior evidence that when trained on downstream language understanding tasks, neural networks often generalize poorly and latch on to dataset-speciﬁc regularities.

Original: Agrawal et al.
Revised: Agrawal et al.

Original: (2016) report how neural models exploit biases in a VQA dataset, e.g. responding “snow” to the question “what covers the ground” regardless of the image because “snow” is the most common answer to this question.
Revised: (2016) report how neural models exploit biases in a VQA dataset, e.g. responding “snow” to the question “what covers the ground” regardless of the image because “snow” is the most common answer to this question.

Original: Gururangan et al.
Revised: Gururangan et al.

Original: (2018) report that many successes in natural language entailment are actually due to exploiting statistical biases as opposed to solving entailment, and that state-of-the-art systems are much less performant when tested on unbiased data.
Revised: (2018) report that many successes in natural language entailment are actually due to exploiting statistical biases as opposed to solving entailment, and that state-of-the-art systems are much less performant when tested on unbiased data.

Original: Jia & Liang (2017) demonstrate that seemingly state-of-the-art reading comprehension system can be misled by simply appending an unrelated sentence that resembles the question to the document.
Revised: Jia & Liang (2017) demonstrate that seemingly state-ofthe-art reading comprehension system can be misled by simply appending an unrelated sentence that resembles the question to the document.

Original: Using synthetic VQA datasets to study grounded language understanding is a recent trend started by the CLEVR dataset (Johnson et al., 2016).
Revised: Using synthetic VQA datasets to study grounded language understanding is a recent trend started by the CLEVR dataset (Johnson et al., 2016).

Original: CLEVR images are 3D-rendered and CLEVR ques- tions are longer and more complex than ours, yet the color-shape generalization split that CLEVR includes arguably lacks a clear motivation.
Revised: CLEVR images are 3D-rendered and CLEVR questions are longer and more complex than ours, but in the associated generalization split CLEVR-CoGenT the training and test distributions of images are different.

Revised: In our design of SQOOP we aimed instead to minimize the difference between training and test images to make sure that we test a model’s ability to interpret unknown combinations of known words.

Original: More closely related to our work is the ShapeWorld family of datasets by Kuhnle & Copestake (2017), that involves a number ofVQA generalization tests. ShapeWorld only contains 10 different objects, making it insufﬁcient for our study.
Revised: The ShapeWorld family of datasets by Kuhnle & Copestake (2017) is another synthetic VQA platform with a number of generalization tests, but none of them tests SQOOP-style generalization of relational reasoning to unseen object pairs.

Original: Most closely related to our work is the recent study of generalization to long-tail questions about rare objects done by Bingham et al.
Revised: Most closely related to our work is the recent study of generalization to long-tail questions about rare objects done by Bingham et al.

Original: They do not, however, consider as many models as we do and do not study the question of whether the best-performing models can be made end-to-end.
Revised: They do not, however, consider as many models as we do and do not study the question of whether the best-performing models can be made end-to-end.

Original: The key paradigm that we test in our experiments is Neural Module Networks (NMN).
Revised: The key paradigm that we test in our experiments is Neural Module Networks (NMN).

Original: Andreas et al.
Revised: Andreas et al.

Original: (2016) introduced NMNs as a modular, structured VQA model where a ﬁxed number of handcrafted neural modules (such as Find , or Compare ) are chosen and composed together in a layout determined by the dependency parse of  question.
Revised: (2016) introduced NMNs as a modular, structured VQA model where a ﬁxed number of handcrafted neural modules (such as Find , or Compare ) are chosen and composed together in a layout determined by the dependency parse of the question.

Revised: Andreas et al.

Revised: (2016) show that the modular structure allows answering questions that are longer than the training ones, a kind of generalization that is complementary to the one we study here.

Original: Hu et al.
Revised: Hu et al.

Original: (2017) and Johnson et al.
Revised: (2017) and Johnson et al.

Original: (2017) followed up by making NMNs end-to-end, removing the non-differentiable parser.
Revised: (2017) followed up by making NMNs end-to-end, removing the non-differentiable parser.

Original: The former chose to keep the handcrafted modules and uses reinforcement learning to learn the layout and modules end-toend.

Original: The latter used a ground truth module layout learned separately, and changes the hand-crafted modules for a generic ResNet block structure (He et al., 2016) for every module.

Original: Both Hu et al.
Revised: Both Hu et al.

Original: (2017) and Johnson et al.
Revised: (2017) and Johnson et al.

Original: (2017) reported that several thousands of ground-truth layouts are required to pretrain the layout predictor in order for their approaches to work.
Revised: (2017) reported that several thousands of ground-truth layouts are required to pretrain the layout predictor in order for their approaches to work.

Original: In a recent work Hu et al.
Revised: In a recent work, Hu et al.

Original: (2018) attempt to remove theneed for hard stochastic layout decisions, but training their models end-to-end from scratch performed substantially lower than best models on the CLEVR task.
Revised: (2018) attempt to soften the layout decisions, but training their models end-to-end from scratch performed substantially lower than best models on the CLEVR task.

Revised: Gupta & Lewis (2018) report successful layout induction on CLEVR for a carefully engineered heterogeneous NMN that takes a scene graph as opposed to a raw image as the input.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: We have conducted a rigorous investigation of an important form of systematic generalization required for grounded language understanding: the ability to reason about all possible pairs of objects despite being trained on a small subset.
Revised: We have conducted a rigorous investigation of an important form of systematic generalization required for grounded language understanding: the ability to reason about all possible pairs of objects despite being trained on a small subset of such pairs.

Original: Our results allow one to draw two important conclusions.
Revised: Our results allow one to draw two important conclusions.

Original: For one, the intuitive appeal of modularity and structure in designing neural architectures for language understanding is now supported by our results, which show how a modular model consisting of general purpose residual blocks generalizes much better than a number of baselines, including architectures such as MAC, FiLM and RelNet that were designed speciﬁcally for visual reasoning.
Revised: For one, the intuitive appeal of modularity and structure in designing neural architectures for language understanding is now supported by our results, which show how a modular model consisting of general purpose residual blocks generalizes much better than a number of baselines, including architectures such as MAC, FiLM and RelNet that were designed speciﬁcally for visual reasoning.

Original: While this may seem unsurprising, to the best of our knowledge, the literature has lacked such a clear empirical evidence in favor of modular and structured networks before this work.
Revised: While this may seem unsurprising, to the best of our knowledge, the literature has lacked such a clear empirical evidence in favor of modular and structured networks before this work.

Original: Importantly, we have also shown how sensitive the high performance of the modular models is to the layout of modules, and how a tree-like structure generalizes much stronger than a typical chain of layers.
Revised: Importantly, we have also shown how sensitive the high performance of the modular models is to the layout of modules, and how a tree-like structure generalizes much stronger than a typical chain of layers.

Original: Our second key conclusion is that coming up with an end-to-end and/or soft version of modular models may be not sufﬁcient for strong generalization.
Revised: Our second key conclusion is that coming up with an end-to-end and/or soft version of modular models may be not sufﬁcient for strong generalization.

Original: In the very setting where strong generalization is required, end-to-end methods may converge to a different, less compositional solution (e.g. a chain layout or blurred attention).
Revised: In the very setting where strong generalization is required, end-to-end methods often converge to a different, less compositional solution (e.g. a chain layout or blurred attention).

Original: This can be observed especially clearly in our NMN layout and parametrization induction experiments on the #rhs/lhs=1 version of SQOOP , but notably, strong initialization sensitivity of layout induction remains an issue even on the #rhs/lhs=18 split.
Revised: This can be observed especially clearly in our NMN layout and parametrization induction experiments on the #rhs/lhs=1 version of SQOOP, but notably, strong initialization sensitivity of layout induction remains an issue even on the #rhs/lhs=18 split.

Original: This conclusion is relevant in the view of recent work in the direction of making NMNs more end-toend (Suarez et al., 2018;
Revised: This conclusion is relevant in the view of recent work in the direction of making NMNs more end-toend (Suarez et al., 2018;

Original: Hu et al., 2018; Hudson & Manning, 2018; Gupta & Lewis, 2018).
Revised: Hu et al., 2018; Hudson & Manning, 2018; Gupta & Lewis, 2018).

Original: Our ﬁndings suggest that merely replacing hard-coded components with learnable counterparts may be insufﬁcient, and that research on regularizers or priors that steer the learning towards more systematic solutions may be required.
Revised: Our ﬁndings suggest that merely replacing hard-coded components with learnable counterparts can be insufﬁcient, and that research on regularizers or priors that steer the learning towards more systematic solutions can be required.

Original: That said, our parametrization induction results on the #rhs/lhs= split are encouraging, as they show that compared to generic models, a weaker nudge (in the form of a richer training signal or a prior) towards systematicity may sufﬁce for end-to-end NMNs.
Revised: That said, our parametrization induction results on the #rhs/lhs= split are encouraging, as they show that compared to generic models, a weaker nudge (in the form of a richer training signal or a prior) towards systematicity may sufﬁce for end-to-end NMNs.

Original: While our investigation has been performed on a synthetic dataset, we believe that it is the realworld language understanding where our ﬁndings may be most relevant.
Revised: While our investigation has been performed on a synthetic dataset, we believe that it is the realworld language understanding where our ﬁndings may be most relevant.

Original: It is possible to construct a synthetic dataset that is bias-free and that can only be solved if the model has understood the entirety of the dataset’s language.
Revised: It is possible to construct a synthetic dataset that is bias-free and that can only be solved if the model has understood the entirety of the dataset’s language.

Original: It is, on the contrary, much harder to collect real-world datasets that do not permit highly dataset-speciﬁc solutions, as numerous dataset analysis papers of recent years have shown (see Section 5 for a review).
Revised: It is, on the contrary, much harder to collect real-world datasets that do not permit highly dataset-speciﬁc solutions, as numerous dataset analysis papers of recent years have shown (see Section 5 for a review).

Original: We believe that approaches that can generalize strongly from imperfect and biased data will likely be required, and our experiments can be seen as a simulation of such a scenario.
Revised: We believe that approaches that can generalize strongly from imperfect and biased data will likely be required, and our experiments can be seen as a simulation of such a scenario.

Original: We hope, therefore, that our ﬁndings will inform researchers working on language understanding and provide them with a useful intuition about what facilitates strong generalization and what is likely to inhibit it. 
Revised: We hope, therefore, that our ﬁndings will inform researchers working on language understanding and provide them with a useful intuition about what facilitates strong generalization and what is likely to inhibit it. A CKNOWLEDGEMENTS

Revised: We thank Maxime Chevalier-Boisvert, Yoshua Bengio and Jacob Andreas for useful discussions.

Revised: This research was enabled in part by support provided by Compute Canada (www.computecanada.ca), NSERC, Canada Research Chairs and Microsoft Research.

Revised: We also thank Nvidia for donating NVIDIA DGX-1 used for this research

Original: A E XPERIMENT D ETAILS
Revised: A E XPERIMENT D ETAILS

Original: We trained all models by minimizing the cross entropy loss log p ( y | x, q ) on the training set, where y ∈ { yes , no } is the correct answer, x is the image, q is the question.
Revised: We trained all models by minimizing the cross entropy loss log p ( y | x, q ) on the training set, where y ∈ { yes , no } is the correct answer, x is the image, q is the question.

Original: In all our experiments we used the Adam optimizer (Kingma & Ba, 2014) with hyperparameters α = 0 . 0001 , β 1 = 0 .
Revised: In all our experiments we used the Adam optimizer (Kingma & Ba, 2015) with hyperparameters α = 0 . 0001 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 10 .
Revised: 10 − 10 .

Original: We continuously monitored validation set performance of all models during training, selected the best one and reported its performance on the test set.
Revised: We continuously monitored validation set performance of all models during training, selected the best one and reported its performance on the test set.

Original: The number of training iterations for each model was selected in preliminary investigations based on our observations of how long it takes for different models to converge.
Revised: The number of training iterations for each model was selected in preliminary investigations based on our observations of how long it takes for different models to converge.

Original: This information, as well as other training details, can be found in Table 
Revised: This information, as well as other training details, can be found in Table 3.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL R ESULTS FOR MAC M ODEL
Revised: B A DDITIONAL R ESULTS FOR MAC M ODEL

Original: We performed an ablation study in which we varied the number of MAC units, the model dimensionality and the level of weight decay for the MAC model.
Revised: We performed an ablation study in which we varied the number of MAC units, the model dimensionality and the level of weight decay for the MAC model.

Original: The results can be found in Table 4.
Revised: The results can be found in Table 4.

Original: weight decay.
Revised: decay.

Original: For each experiment we report means and standard deviations based on 5 repetitions.
Revised: For each experiment we report means and standard deviations based on 5 repetitions.

Original: [Table]
Revised: [Table]

Original: We also perform qualitative investigations to understand the high variance in MAC’s performance.
Revised: We also perform qualitative investigations to understand the high variance in MAC’s performance.

Original: In particular
Revised: In particular

Original: [Table]
Revised: [Table]

Original: we focus on control attention weights ( c ) for each run and aim to understand if runs that generalize have clear differences when compared to runs that failed.
Revised: we focus on control attention weights ( c ) for each run and aim to understand if runs that generalize have clear differences when compared to runs that failed.

Original: Interestingly, we observe that in successful runs each word w ∈ X , Y has a unit that is strongly focused on it.
Revised: Interestingly, we observe that in successful runs each word w ∈ X , Y has a unit that is strongly focused on it.

Original: To present our observations in quantitative terms, we plot attention quality κ = min w ∈{ X,Y } max k ∈ [1;12] α k,w / (1 − α k,R ) , where α are control scores vs accuracy in Figure 7 for each run (see Section 4.3.2 for an explanation of κ ).
Revised: To present our observations in quantitative terms, we plot attention quality κ = min w ∈{ X,Y } max k ∈ [1;12] α k,w / (1 − α k,R ) , where α are control scores vs accuracy in Figure 7 for each run (see Section 4.3.2 for an explanation of κ ).

Original: We can clearly see astrong positive correlation between κ and error rate.
Revised: We can clearly see a positive correlation between κ and error rate, especially for low #rhs/lhs.

Original: [Figure]
Revised: [Figure]

Original: Next, we experiment with a hard-coded variation of MAC.
Revised: Next, we experiment with a hard-coded variation of MAC.

Original: In this model, we use hard-coded control scores such that given a SQOOP question X R Y , the ﬁrst half of all modules focuses on X while the second half focuses on Y .
Revised: In this model, we use hard-coded control scores such that given a SQOOP question X R Y , the ﬁrst half of all modules focuses on X while the second half focuses on Y .

Original: The relationship between MAC and hardcoded MAC is similar to that between NMN-Tree and end-to-end NMN with parameterization induction.
Revised: The relationship between MAC and hardcoded MAC is similar to that between NMN-Tree and end-to-end NMN with parameterization induction.

Original: However, this model has not performed as well as the successful runs of MAC.
Revised: However, this model has not performed as well as the successful runs of MAC.

Original: We hypothesize that this could be due to the interactions between the control scores and the visual attention part of the model.
Revised: We hypothesize that this could be due to the interactions between the control scores and the visual attention part of the model.

Revised: C I NVESTIGATION OF C ORRECT P REDICTIONS WITH S PURIOUS L AYOUTS

Revised: In Section 4.3.1 we observed that an NMN with the Residual module can answer test questions with a relative low error rate of 1 . 64 ± 1 .

Revised: 79% , despite being a mixture of a tree and a chain (see results in Table 1, p 0 ( tree ) =

Revised: Our explanation for this phenomenon is as follows: when connected in a tree, modules of such spurious models generalize well, and when connected as a chain they generalize poorly.

Revised: The output distribution of the whole model is thus a mixture of the mostly correct p ( y | T = T tree , x, q ) and mostly random p ( y | T =

Revised: T chain , x, q ) .

Revised: We verify our reasoning by explicitly evaluating test accuracies for p ( y | T = T tree , x, q ) and p ( y | T = T chain , x, q ) , and ﬁnd them to be around 99% and 60% respectively, conﬁrming our hypothesis.

Revised: As a result the predictions of the spurious models with p ( tree )

Revised: ≈ 0 .

Revised: 5 have lower conﬁdence than those of sharp tree models, as indicated by the high log loss of 0 .

Revised: 27 ± 0 .

Revised: We visualize the progress of structure induction for the Residual module with p 0 ( tree ) = 0 .

Revised: 5 in Figure 4 which shows how p ( tree ) saturates to 1.0 for #rhs/lhs=18 and remains around 0.5 when #rhs/lhs=1.

Revised: D SQOOP P SEUDOCODE

Revised: 1: S ← { A,B,C, ..., Z, 0,1,2,3, ..., 9 } 2: Rel ← { LEFT - OF , RIGHT - OF , ABOVE , BELOW } (cid:46) relations 3: function C REATE SQOOP(k) 4:

Revised: TrainQuestions ← [] 5:

Revised: AllQuestions ← [] 6: for all X in S do 7:

Revised: AllRhs ← RandomSample( S \ { X } , k) (cid:46) sample without replacement from S \ { X } 8:

Revised: AllQuestions ← { X } ×

Revised: Rel × ( S \ { X } ) ∪ AllQuestions 9: for all R, Y in AllRhs ×

Revised: Rel do 10:

Revised: TrainQuestions ← ( X, R, Y ) ∪ TrainQuestions 11: end for 12: end for 13:

Revised: TestQuestions ← AllQuestions \ TrainQuestions 14: function G ENERATE E XAMPLE ( X, R, Y ) 15: a ∼ { Yes, No } 16:

Revised: if a =

Revised: Yes then 17:

Revised: I ← place X and Y objects so that R holds (cid:46) create the image 18:

Revised: I ← sample 3 objects from S and add to I 19: else 20: repeat 21:

Revised: X (cid:48)

Revised: ← Sample X (cid:48) from S \ { X } 22:

Revised: Y (cid:48)

Revised: ← Sample Y (cid:48) from S \ { Y } 23:

Revised: I ← place X (cid:48) and Y objects so that R holds (cid:46) create the image 24:

Revised: I ← add X and Y (cid:48) objects to I so that R holds 25:

Revised: I ← sample 1 more object from S and add to I 26: until X and Y are not in relation R in I 27: end if 28: return I , X, R, Y , a 29: end function 30:

Revised: Train ← sample 10 6 | TrainQuestions | examples for each (X,R,Y

Revised: ) ∈

Revised: TrainQuestions from

Revised: G ENERATE E XAMPLE ( X, R, Y ) 31:

Revised: Test ← sample 10 examples for each (X,R,Y) ∈ TestQuestions from G ENERATE E XAMPLE

Revised: ( X, R, Y ) 32: end function