I TERATED LEARNING FOR EMERGENT SYSTEMATICITY IN VQA
Anonymous authors
Paper under double-blind review
A BSTRACT
Although neural module networks have an architectural bias towards compositionality, they require gold standard layouts to generalize systematically in practice.
When instead learning layouts and modules jointly, compositionality does not arise automatically, and an explicit pressure for systematic generalization is necessary for the emergence of layouts exhibiting the right structure.
We propose to address this problem using iterated learning, a cognitive science theory of the emergence of compositional languages in nature that has primarily been applied to simple referential games in machine learning.
Considering the layouts of module networks as samples from an emergent language, we use iterated learning to encourage the development of structure within this language.
We show that the resulting layouts support systematic generalization in neural agents solving the more complex task of visual question-answering.
Our regularized iterated learning method can outperform baselines without iterated learning on SHAPES-SyGeT ( SHAPES Sy stematic Ge neralization T est), a new split of the SHAPES dataset we introduce to evaluate systematic generalization, and on CLOSURE, an extension of CLEVR also designed to test systematic generalization.
We demonstrate superior performance in recovering ground-truth compositional program structure with limited supervision on both SHAPES-SyGeT and CLEVR.
I NTRODUCTION
Although great progress has been made in visual question answering (VQA), recent methods still struggle to generalize systematically to inputs coming from a distribution different from that seen during training (Bahdanau et al., 2019b;a).
Neural module networks (NMNs) present a natural solution to improve generalization in VQA, using a symbolic layout or program to arrange neural computational modules into computation graphs.
If these modules are learned to be specialized, they can be composed in arbitrary legal layouts to produce different processing ﬂows.
However, for modules to learn specialized roles, programs must support this type of compositionality; if programs reuse modules in non-compositional ways, modules are unlikely to become layout-invariant.
This poses a substantial challenge for the training of NMNs.
Although Bahdanau et al.
(2019b)
and
Bahdanau et al.
(2019a)
both observe that NMNs can systematically generalize if given humandesigned ground-truth programs, creating these programs imposes substantial practical costs.
It becomes natural to jointly learn a program generator alongside the modules (Johnson et al., 2017b;
Hu et al., 2017; Vedantam et al., 2019), but the generated programs often fail to generalize systematically and lead to worse performance (Bahdanau et al., 2019b).
Iterated learning (IL) offers one way to address this problem.
Originating in cognitive science,
IL explains how language evolves to become more compositional and easier-to-acquire in a repeated transmission process, where each new generation acquires the previous generation’s language through a limited number of samples (Kirby et al., 2014).
Early works on human experiments (Kirby et al., 2008) as well as agent-based simulations (Zuidema, 2003) support this hypothesis.
The machine learning community has also recently shown an increasing interest in applying IL towards emergent communication (Guo et al., 2019; Li & Bowling, 2019; Cogswell et al., 2019; Dagan et al., 2020;
Ren et al., 2020).
Different from previous works, we believe that IL is an algorithmic principle that is equally applicable to recovering compositional structure in more general tasks.
We thus propose treating NMN programs as a “layout language” and applying IL to the challenging executionengine program generator is a green shape left of a square?
and( color[green]( scene ), transform[left_of]( shape[square]( scene ) ) )
yes
Bidirectional RNN encoder
RNN decoder with attention transform[left_of] shape[square] color[green] and scene
Classifier
Feature extractor problem of VQA.
Our efforts highlight the potential of IL for broader machine learning applications beyond the previously-explored scope of language emergence and preservation (Lu et al., 2020).
To demonstrate our method, we introduce a lightweight benchmark for systematic generalization research based on the popular SHAPES dataset (Andreas et al., 2016) called SHAPES-SyGeT ( SHAPES Sy stematic Ge neralization T est).
Our experiments on SHAPES-SyGeT, CLEVR (Johnson et al., 2017a), and CLOSURE (Bahdanau et al., 2019a) show that the IL algorithm accelerates the learning of compositional program structure, leading to better generalization to both unseen questions from the training question templates and unseen question templates.
With only 100 groundtruth programs for supervision, our method achieves CLEVR performance comparable to Johnson et al.
(2017b)
and Vedantam et al.
(2019), which use 18000 and 1000 programs for supervision respectively.
R ELATED WORK
Systematic generalization.
Systematicity was ﬁrst proposed as a topic of research in neural networks by Fodor & Pylyshyn (1988), who argue that cognitive capabilities exhibit certain symmetries, and that representations of mental states have combinatorial syntactic and semantic structure.
Whether or not neural networks can exhibit systematic compositionality has been a subject of much debate in the research community (Fodor & Pylyshyn, 1988; Christiansen & Chater, 1994; Marcus, 1998; Phillips, 1998; Chang, 2002; Marcus, 2018; van der Velde et al., 2004; Botvinick & Plaut, 2009; Bowers et al., 2009; Brakel & Frank, 2009; Fodor & Lepore, 2002; Marcus, 2018; Calvo &
Symons, 2014).
Bahdanau et al.
(2019b) investigate various VQA architectures such as neural module networks (NMNs) (Andreas et al., 2016), MAC (Hudson & Manning, 2018), FiLM (Perez et al., 2018), and relation networks (Santoro et al., 2017) on their ability to systematically generalize on a new synthetic dataset called SQOOP.
They show that only NMNs are able to robustly solve test problems, but succeed only when a ﬁxed tree-structured layout is provided.
When learning to infer the module network layout, robust tree-structured layouts only emerged if given a strong prior to do so.
The authors conclude that explicit regularization and stronger priors are required.
CLEVR (Johnson et al., 2017a) is a popular VQA dataset, and various benchmarks achieve almostperfect validation scores (Hu et al., 2017;
Hudson & Manning, 2018; Perez et al., 2018;
Santoro et al., 2017).
Bahdanau et al.
(2019a) proposed an extension of CLEVR with a new evaluation dataset called CLOSURE, containing novel combinations of linguistic concepts found in CLEVR.
The authors found that many of the existing models in the literature fail to systematically generalize to CLOSURE.
Moreover, there is a signiﬁcant gap between the performance achieved with groundtruth layouts and learned layouts on CLOSURE.
Language emergence and compositionality.
Agents interacting in a cooperative environment can learn a language to communicate to solve a particular task.
The emergence of such a communication protocol has been studied extensively in multi-agent referential games.
In these games, one agent must describe what it saw to another agent, which is tasked with ﬁguring out what the ﬁrst agent saw (Lewis, 2008; Skyrms, 2010; Steels & Loetzsch, 2012).
To encourage a dialogue between agents, several multi-stage variants of such a game have also been proposed (Kottur et al., 2017; Evtimova et al., 2018).
Most approaches to learning a discrete communication protocol between agents use reinforcement learning (Foerster et al., 2016;
Lazaridou et al., 2016; Kottur et al., 2017; Jorge et al., 2016; Havrylov & Titov, 2017).
However, the Gumbel straight-through estimator (Jang et al., 2017) can also be used (Havrylov & Titov, 2017), as can backpropagation when the language in question is continuous (Foerster et al., 2016; Sukhbaatar & Fergus, 2016; Singh et al., 2019).
Several works in the literature have found that compositionality only arises in emergent languages if appropriate environmental pressures are present (Kottur et al., 2017; Choi et al., 2018;
Lazaridou et al., 2018; Chaabouni et al., 2020).
While generalization pressure is not sufﬁcient to guarantee compositionality, compositional languages tend to exhibit better systematic generalization (Bahdanau et al., 2019b; Chaabouni et al., 2020).
The community still lacks strong research indicating what general conditions are necessary or sufﬁcient for compositional language emergence.
Iterated learning.
The origins of the compositionality of human language, which leads to an astounding open-ended expressive power, have attracted much interest over the years.
Kirby (2001) suggests that this phenomenon is a result of a learning bottleneck arising from the need to learn a highly expressive language with only a limited set of supervised learning data.
The iterated application of this bottleneck, as instantiated by IL, has been demonstrated to cause artiﬁcial languages to develop structure in experiments with human participants (Kirby et al., 2008; Silvey et al., 2015).
Ren et al.
(2020) present neural IL following the principles of Kirby (2001), where neural agents play a referential game and evolve a communication protocol through IL.
They use topographic similarity (Brighton & Kirby, 2006) to quantify compositionality, and ﬁnd that high topographic similarity improves the learning speed of neural agents, allows the listener to recognize more objects using less data, and increases validation performance.
However, these experiments are limited to domains with extremely simple object and message structure.
Several ablation studies (Li & Bowling, 2019; Ren et al., 2020) have found that re-initializing the speaker and the listener between generations is necessary to reap the beneﬁts of compositionality from IL.
However, seeded IL (Lu et al., 2020) proposes to seed a new agent with the previous generation’s parameters at the end of the learning phase.
Since an interacting phase has not yet ﬁnetuned this initialization, it has not had the opportunity to develop a non-compositional language to ﬁt the training data.
The authors ﬁnd that seeded IL helps counter language drift in a translation game.
They suggest that this might be because IL maintains the compositionality of the natural language.
M ETHOD
We are interested in solving the task of visual question-answering (VQA).
Let X be the space of images about which our model will be required to answer questions.
Next, let Q be the space of natural-language questions and Y the space of all possible answers to the questions.
Additionally, we consider a space Z of programs, which represent computation graphs of operations that can be performed on an image in X to produce an output in Y .
We consider a question template T to be a set of tuples ( q , z ) , where q ∈ Q and z ∈ Z .
Each question template contains questions with the same structure but varying primitive values.
For example, the questions “Is a triangle blue” and “Is a square red” belong to a template “Is a SHAPE COLOR .”
The program z corresponding to the question q in a template deﬁnes a computation graph of operations that would produce the correct answer in Y to q for any input image x ∈ X .
Finally, let T be a ﬁnite set of question templates.
The dataset for training our model and evaluating VQA performance constitutes tuples of the form ( q , z , x , y ) .
First, a template T ∈ T is sampled and a tuple ( q , z ) is sampled from T .
Then, an
[Figure]
image x ∈ X is sampled and the answer y is produced by passing x through the program z .
These collected variables ( q , z , x , y ) form a single example in the task dataset.
To evaluate our model’s performance on unseen question templates, we deﬁne T train ⊂ T to be a subset of training templates and T test =
T − T train to be the subset of test templates.
The training dataset D is prepared from templates in T train and the out-of-distribution test dataset D test from templates in T test .
We allow a program z to be absent in D , in which case it is not used for auxiliary supervision during training.
Our goal of systematic generalization is to learn a model p ( Y | X , Q ) that performs well on the dataset D test created using unseen question templates, where Y , X , and Q are random variables taking values in Y , X , and Q .
We deﬁne our model to be a composition of a program generator
PG θ ( Z | Q ) and execution engine EE φ ( Y | X , Z ) , parameterized by θ and φ respectively.
3. M ODEL ARCHITECTURE
Our overall model architecture is illustrated in Figure 1.
We use the attention-based sequence-tosequence model from Bahdanau et al. (2019a) as the program generator, which translates an input question q into a sampled program ˆ z .
The execution engine assembles modules into a layout dictated by ˆ z to instantiate an NMN that takes the input image x to predict an answer ˆ y .
In this work, we explore three execution engine architecture choices, including Tensor-NMN , which is the module architecture from Johnson et al.
(2017b), and Vector-NMN from Bahdanau et al. (2019a).
We also experiment with a novel hybrid of these architectures that performs better on SHAPES-SyGeT, which we dub Tensor-FiLM-NMN .
Appendix B elaborates the details of our model architecture.
The goal of the Tensor-FiLM-NMN architecture is to combine the minimalist, FiLM-based deﬁnition of modules in Vector-NMN with the spatial representational power of Tensor-NMN.
As such, the modules in Tensor-FiLM-NMN use FiLM to condition global operations on module-speciﬁc embeddings, as in Vector-NMN, but module inputs and outputs are tensor-valued feature maps like in Tensor-NMN.
The following equations illustrate the working of a Tensor-FiLM-NMN module:
[Equation]
Here, x is the input image, h 1 and h 2 are the module inputs, and y is the module output.
γ · and β · are FiLM parameters computed using different 2 -layer MLPs per layer on the module-speciﬁc embeddings.
All weights W · and biases b · are shared across all modules.
We further strengthen our model’s ability for spatial reasoning by using cumulative sums of an intermediate representation across locations in left-to-right and top-to-bottom directions.
This allows our model, for example, to select regions to the ‘left of’ or ‘below’ objects through appropriate scaling and thresholding.
3. I TERATED LEARNING FOR NMN S
We can view the program generator and the execution engine as communicating agents in a cooperative VQA game where the programs passed between agents are messages drawn from an emergent language.
Introducing more compositional structure to this language, such as reusing low-level concepts using the same tokens and high-level concepts using the same sequence of tokens, helps address the combinatorial complexity of the question space, allowing agents to perform better on new question templates containing previously-unseen combinations of known concepts.
We use IL to encourage the emergence of structure in the generated programs.
We iteratively spawn new program generator and execution engine agents, train them on the VQA task, and transfer their knowledge to the next generation of agents.
Limiting this transmission of knowledge between generations imposes a learning bottleneck , where only the easy-to-learn linguistic concepts survive.
Since the central hypothesis of neural IL that structure is easier for neural networks to learn than idiomatic non-compositional rules (Li & Bowling, 2019; Ren et al., 2020), our IL algorithm pushes the language of the programs to be more compositional.
The combination of learning a compositional structure and performing well on the VQA training task engenders systematic generalization.
We follow Ren et al.
(2020) in dividing our method into three stages, an interacting phase, a transmitting phase, and a learning phase, which are cycled through iteratively until the end of training.
Figure 2 illustrates the phases of our method for training module networks with compositional emergent layouts and Appendix C presents a formal algorithm for the same.
Interacting phase.
The program generator and the execution engine must work together.
Without programs that consistently use the same tokens to mean the same operations, the execution engine cannot assign the correct semantics to modules.
Simultaneously, the program generated depends on the language grounding provided by the execution engine to provide reinforcement signal for beneﬁcial layouts.
To make the problem more tractable, it is common to provide the model with a small set of ground-truth programs Johnson et al.
(2017b)
; Vedantam et al.
We ﬁnd that providing program supervision with a small number of ground-truth programs throughout the interacting phase helps in generalization.
This can be seen as a form of supervised self-play (Lowe* et al., 2020), and has the effect of simulating an inductive bias towards the desired program language.
The agents are jointly trained for T i steps to minimize answer prediction error on data sampled from the training dataset D , with predictions given by ˆ z ∼ PG θ ( q ); ˆ y = EE φ ( x , ˆ z ) .
We train EE φ by minimizing the cross-entropy loss L between the true answer label y and the predicted distribution ˆ y .
Additionally, we estimate the gradients for the parameters θ using REINFORCE:
[Equation]
Here, p θ is the distribution over programs returned by PG θ ( q ) .
We use the negative cross-entropy loss − L clipped between − 5 and 5 as the reward for the program generator.
When a ground-truth program z is available, we additionally minimize a weighted cross-entropy loss between z and the generated program ˆ z using teacher forcing.
In practice, the model operates on minibatches of data, and a subset of every minibatch is subsampled from data examples with ground-truth programs.
Transmitting phase.
During the transmitting phase, a new dataset D with T t samples is prepared for use during the learning phase.
Questions q , as well as ground-truth programs
z if available, are sampled from D .
For data examples without ground-truth programs, a program ˆ z is sampled from the execution engine using q .
Finally, question-program pairs ( q , z ) , if z is available, or ( q , ˆ z ) are added to D .
By transmitting the ground-truth programs when available, we continue simulating the inductive bias towards desirable programs during the learning phase.
Learning phase.
Program generators and execution engines are initialized in the learning phase, forming a new generation of agents to play the VQA game.
The new program generator then acquires the previous agents’ language from the transmitted data D .
However, it does so imperfectly due to a learning bottleneck that exerts pressure towards compositionality (Ren et al., 2020).
We implement this learning bottleneck primarily by limiting the number of gradient updates in the learning phase, effectively performing early stopping .
A small program generator learning phase length T p leads to an underﬁt program generator that wastes computation during the interacting phase to re-learn global linguistic rules.
On the other hand, setting T p to a high value leads to overﬁtting and learning of the previous generation’s idiosyncrasies, losing the beneﬁts of the learning bottleneck.
In addition to limiting the number of training iterations, we optionally further regularize the model by applying spectral normalization (Miyato et al., 2018) on the program generator’s decoder LSTM parameters.
We train the new program generator PG ˜ θ by minimizing the cross-entropy loss between model samples ˜
z ∼ PG ˜
θ ( q ) and transmitted programs ˆ z corresponding to q in D for T p gradient steps.
Then, the new execution engine EE ˜ φ is trained with programs from the new program generator
PG ˜
θ , using the entire dataset D for T e steps.
The forward pass in the execution engine learning phase is similar to that in the interacting phase, but the backward pass only updates the execution engine parameters ˜ φ .
Adapting the new execution engine to the new program generator ensures stable training in the upcoming interacting phase.
S YSTEMATIC GENERALIZATION TEST FOR SHAPES
The SHAPES dataset (Andreas et al., 2016) is a popular yet simple VQA dataset.
The lower image and question complexity of SHAPES relative to CLEVR make it an attractive choice for experimentation since the training time can be signiﬁcantly lower and it requires fewer resources.
Each of the 15616 data points in SHAPES contains a unique image.
However, there are only 244 unique questions.
Although the validation and test splits of SHAPES contain unique questions not present in any of the other splits, the questions are of the same form as the ones in the training split.
To the best of our knowledge, there is no published systematic generalization evaluation setup for
SHAPES.
Thus, we present SHAPES-SyGeT , a SHAPES Sy stematic Ge neralization T est.
To understand the distribution of question types in SHAPES, we categorized questions into 12 standard templates, out of which 7 are used for training, and 5 to test systematic generalization.
To evaluate the in-distribution and out-of-distribution generalization performance of our models, we prepare the SHAPES-SyGeT training set with only a subset of the questions under each train template and use the rest as an in-distribution validation set ( Val-IID ).
Questions belonging to the evaluation templates are used as an out-of-distribution validation set ( Val-OOD ).
Please see Appendix D for further details about the question templates and the dataset splits.
E XPERIMENTS
In this section, we present results on SHAPES-SyGeT, CLEVR, and CLOSURE.
The SHAPESSyGeT experiments are run with 3 different seeds and the CLEVR experiments use 8 seeds.
We report the mean and standard deviation of metrics over these trials.
CLEVR experiments use representations of images from a pre-trained ResNet-101 (He et al., 2016) as input, whereas the SHAPESSyGeT runs use standardized raw images.
Please see Appendix A for the hyperparameters we use.
For our NMN baselines without IL, we still utilize the multi-task objective of using REINFORCE to backpropagate gradients to the program generator and using a cross-entropy loss to provide program supervision when available.
We ﬁnd this method of training NMNs to generalize better than pretraining on the available programs and then interacting without supervision as is done in Johnson et al.
(2017b).
One can also view our baselines as executing one long interacting phase without any other phases or resetting of the model.
5. SHAPES-S Y G E T
Due to its lightweight nature, we use SHAPES-SyGet for a series of experiments designed to illuminate the essential components of our method.
For all SHAPES-SyGeT experiments, we record the in-distribution Val-IID and the out-of-distribution Val-OOD accuracies.
However, model selection and early stopping are done based on Val-IID.
For our IL experiments, we use spectral normalization of the program generator during the learning phase, and we study its utility in Appendix E.
We do not report results on Vector-NMN, which performs very poorly.
We believe that this is because SHAPES-SyGeT requires intermediate module outputs to contain detailed spatial information, which Vector-NMN has an inductive bias against thanks to its module outputs being vectors.
We note that the reported results on SHAPES in the literature make use of NMN architectures with specially designed modules for each operation (Andreas et al., 2016; Hu et al., 2017; Vedantam et al., 2019).
In contrast, we use generic modules in order to study compositional layout emergence with minimal semantic priors, making our results hard to compare directly to prior work using SHAPES. [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] In-distribution and out-of-distribution accuracies.
Table 1 illustrates the difference in the performance of various conﬁgurations of our models.
All the reported models achieve perfect accuracy on the training data but generalize differently to Val-IID and Val-OOD.
We ﬁnd that Tensor-FiLMNMN generalizes better than Tensor-NMN trained with supervision of both 20 and 135 ground-truth programs.
Moreover, we ﬁnd that IL improves generalization across all conﬁgurations.
We further evaluate the relationship between program supervision and performance in Figure 3 and ﬁnd that IL improves performance at all levels of supervision, effectively improving data efﬁciency.
Learning bottleneck.
Although all models achieve perfect training accuracy in under 5000 steps, we notice that IL leads to gradually increasing systematic generalization as training progresses.
Figure 4 shows the effect of the learning bottleneck of IL, comparing our two architectures with and without IL.
Here, we ﬁnd that program accuracy 1 increases steadily as training progresses, indicating increasing structure in the language of the programs.
Although generated programs that do not match the ground-truth programs exactly could conceivably be compositional, it is unlikely that the 20 ground-truth supervision programs can successfully be incorporated in a compositional structure other than the ground-truth.
Importantly, this process is due entirely to the inﬂuence of the
IL learning bottleneck, as program accuracy without IL actually declines later in training.
Ablation tests.
We use SHAPES-SyGeT to perform a series of ablations tests.
We report full results for these experiments in Appendix E. Instead of re-initializing agent parameters from scratch at each generation, we consider seeded IL (Lu et al., 2020), in which agents are re-initialized with parameters from the end of the learning phase of the previous generation, as well as simply not reinitializing agents.
We explore these strategies for the execution engine, observing large declines in generalization compared to the standard setting.
We also study the effect of spectral normalization, ﬁnding it highly beneﬁcial when applied to the program generator decoder during the learning phase of IL, but of only marginal beneﬁt otherwise.
Finally, we examine a setting where the program generator is reinitialized but not trained to acquire the previous generation’s language, which we also ﬁnd to greatly harm performance, indicating that the learning phase is key. [Figure] [Figure] [Figure]
[Table]
5. CLEVR AND CLOSURE
CLEVR is signiﬁcantly larger and more complex than SHAPES-SyGeT.
It takes an execution engine over a day to reach 95% task accuracy on the CLEVR validation set on a Nvidia RTX-8000 GPU, even when trained with ground-truth programs without a program generator.
Re-training the execution engine from scratch for every generation of IL is thus computationally infeasible.
Between using seeded IL and not resetting the execution engine at all, we ﬁnd not resetting the execution engine generalizes better.
CLEVR contains 699989 training questions, and we provide 100 groundtruth programs for program supervision, one-tenth of that used by Vedantam et al.
For
CLEVR, we found that Tensor-FiLM-NMN did not improve over Vector-NMN, so for conciseness we report results only on the two standard architectures.
Figure 5 illustrates training dynamics with and without IL on CLEVR, and Table 2 reports model performance on the CLEVR validation set and CLOSURE categories.
The CLEVR validation curves through training are presented in Appendix F. Similar to Bahdanau et al. (2019a), we ﬁnd that
Vector-NMN generalizes better than Tensor-NMN.
However, Tensor-NMN systematically generalizes substantially better with IL.
Across both models, using IL improves generalization on CLOSURE, greatly increases program accuracy, and achieves validation performance on CLEVR close to ProbNMN (Vedantam et al., 2019) despite using far fewer ground-truth programs.
C ONCLUSION
We establish IL as a practical tool for the emergence of structure in machine learning by demonstrating its utility in challenging VQA tasks using NMNs.
Our work shows that IL leads to improved performance with less supervision and facilitates systematic generalization.
As the study of IL in
[Figure]
neural agents is relatively young and has largely been explored in the narrow domain of language emergence in referential games, our setting presents several important new challenges for IL methods. Unlike in simple games (Ren et al., 2020), the emergence of compositional language without any supervision is thus far infeasible in VQA due to the difﬁcult joint optimization of the program generator and the execution engine.
However, by exploring learning phase regularization, suitable model architectures, and the learning dynamics of IL, we are able to dramatically reduce the amount of ground-truth supervision necessary; the surprising success of spectral normalization in particular should motivate further research into the role of regularization during the learning phase.
We hope that this progress will spur future work to improve IL methods and their theoretical understanding, as well as extend them to other challenging tasks in machine learning.
A H YPERPARAMETERS
Table 3 details the hyperparameters used in our experiments in Section 5 for both SHAPES-SyGeT and CLEVR.
B M ODEL ARCHITECTURE DETAILS B. P ROGRAM GENERATOR
The program generator is a seq2seq model that translates input questions to programs.
We use the
Seq2SeqAtt model of Bahdanau et al. (2019a), where unlike Johnson et al.
(2017b)
, the decoder uses an attention mechanism (Bahdanau et al., 2014) over the encoder hidden states.
We found this model to generalize better on held-out in-distribution questions in preliminary experiments.
Ideally, the program generator’s target must be tree-structured since the output program represents a computation graph.
Since a seq2seq model cannot directly model graphs, we instead model a serialized representation of the programs.
Like Johnson et al.
(2017b), we chose to use the Polish (or preﬁx) notation to serialize the syntax tree.
Considering the program generator parameterized by θ
[Table]
the probability distribution over layouts ˆ z is given by
[Equation] [Equation]
During training, we sample programs from this distribution but take the arg max at each timestep when evaluating our model.
We use an LSTM (Hochreiter & Schmidhuber, 1997) trained using teacher forcing to represent the distribution p θ (ˆ z i | ˆ z 1: i − 1 , q ) at every timestep i .
To condition on q for each decoder step, we perform attention on the hidden states of a bidirectional LSTM encoder over the question q .
B. E XECUTION ENGINE
To run the program ˆ z produced by the program generator, the execution engine assembles neural modules according to ˆ z into a neural network that takes x as input to produce an answer ˆ y .
We consider this execution engine EE φ ( x , ˆ z ) to be parameterized by φ .
To illustrate a forward pass through the module network, consider the program ‘ and color[green] scene transform[left of] shape[square] scene ’ from Figure 1.
‘ scene ’ is a special module which represents the input image features from a CNN feature extractor.
According to the annotators, the ‘ color[green] ’ module should ﬁlter green shapes from the scene, and the
‘ shape[square] ’ module should ﬁlter squares.
‘ transform[left of] ’ should select regions to the left of the ﬁltered squares.
In Figure 1, the ﬁnal ‘ and ’ module should then compute the intersection of green shapes and the region to the left of squares.
A classiﬁer takes this output from the top-level module to produce a distribution ˆ y over answers.
We consider the ﬁnal answer ˆ y to be the arg max of ˆ y .
Each module corresponds to one token of the program and can be reused in different conﬁgurations.
Like Johnson et al.
(2017b), our modules are not hand-designed to represent certain functions but have a similar architecture for all operators.
Other than the ‘ scene ’ module which has arity 0 , all other modules are implemented as a CNN on the module inputs to produce a module output.
In this work, we explore three module architecture choices to make up the execution engine: Tensor-NMN ,
Vector-NMN , and Tensor-FiLM-NMN .
Here, the Tensor-NMN module architecture is the module network proposed by Johnson et al.
(2017b), and Vector-NMN is the architecture from Bahdanau et al. (2019a).
C I TERATED LEARNING ALGORITHM
We present the full IL algorithm described in Section 3.2 in Algorithm 1.
Algorithm 1: Iterated learning for compositional NMN layout emergence.
[Equation] [Equation]
do
end / * do
Sample
˜
L ←
Update θ minimize L spectral normalization; end / do
←
end 31 end
[Table]
[Table]
D SHAPES-S Y G E T TEMPLATES AND SPLITS
The images in SHAPES are arranged in a 3 × 3 grid, where each grid cell can contain a triangle, square, or circle that is either red, green, or blue, or the cell can be empty.
Table 4a shows the standard splits for training and evaluating SHAPES used in the literature (Andreas et al., 2016;
Hu et al., 2017; Vedantam et al., 2019).
We categorize SHAPES questions into 12 standard templates, listed in Table 5.
We then derive new splits for the SHAPES data, such that each split has the same primitives, but different ways in which the primitives are combined.
The ﬁnal split of the
SHAPES-SyGeT dataset is presented in Table 4b.
E A BLATION EXPERIMENTS
We do ablation experiments on SHAPES-SyGeT rather than CLEVR or CLOSURE, as the computational requirements of experiments on this dataset are nearly an order of magnitude lighter.
Results are presented in Table 6.
We focus primarily on the Tensor-FiLM-NMN, which we ﬁnd beneﬁts far more from IL than the default Tensor-NMN.
In all cases we use 20 ground-truth supervision programs, a setting which we ﬁnd to most clearly reﬂect differences between methods.
To conﬁrm that our results are not solely due to our use of spectral normalization, we examine the use of SN throughout training in non-IL runs (”FullSN”); we ﬁnd that it has at best marginal impacts on performance, indicating that it is not responsible for the success of our method.
[Table]
+FullSN -
[Table]
+NoSN -
[Table]
+FullSN -
[Table]
+NoSN -
[Table]
+IL +IL
[Table]
+
[Table]
+
[Table]
+
[Table]
+NoRetrain
[Table]
+
[Table]
+NoSN
[Table]
+IL+NoRetrain +
[Table]
+
[Table]
+
[Table]
+
[Table]
+IL+NoSN
[Table]
NoReset
[Table]
+IL+NoRetrain +IL+NoReset
[Table]
generation.
We also consider various schemes for the reinitialization of the execution engine parameters ˜ φ , including the standard method (reinitialization from the same random distribution initially used),
Seeded reinitialization, where parameters from the end of the previous generation’s learning phase (but before its interaction phase) are used, and not resetting the network at all (’NoReset’).
We ﬁnd a clear beneﬁt to reinitializing the execution engine from scratch, although seeded IL outperforms not resetting the execution engine at all.
Finally, we also consider the effects of simply not retraining the program generator at all during the learning phase, entirely ablating the aspect of generational transmission.
We ﬁnd that this setting performs roughly comparably to the no-IL setting; this is expected, as in this case little generational transmission occurs, meaning that the IL bottleneck that encourages gradual drift towards compositionality cannot take effect.
In summary, these results indicate that the generational transmission aspect of our algorithm is a key component of its success, and bolster our contention that regularization during the student’s acquisition of its predecessors’ language has a uniquely positive effect, far outweighing its impact during regular training.
F CLEVR VALIDATION CURVES
The validation curves for CLEVR, illustrated in Figure 6, are similar to the training curves in Figure 5.
Although CLEVR training exhibits a lower amount of overﬁtting compared to SHAPESSyGeT, we observe higher systematic generalization performance on CLOSURE with models trained using IL.
[Figure]