T RANSFORMER M ODULE N ETWORKS FOR S YSTEMATIC G ENERALIZATION IN V ISUAL Q UESTION A NSWERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Transformers achieve great performance on Visual Question Answering (VQA).
However, their systematic generalization capabilities, i.e., handling novel combinations of known concepts, is unclear.
We reveal that Neural Module Networks (NMNs), i.e., question-specific compositions of modules that tackle a sub-task, achieve better or similar systematic generalization performance than the conventional Transformers, even though NMNs’ modules are CNN-based.
In order to address this shortcoming of Transformers with respect to NMNs, in this paper we investigate whether and how modularity can bring benefits to Transformers.
Namely, we introduce Transformer Module Network (TMN), a novel NMN based on compositions of Transformer modules.
TMNs achieve state-of-the-art systematic generalization performance in three VQA datasets, improving more than 30% over standard Transformers for novel compositions of sub-tasks.
We show that not only the module composition but also the module specialization for each sub-task are the key of such performance gain.
I NTRODUCTION
Visual Question Answering (VQA) (Antol et al., 2015) is a fundamental testbed to assess the capability of learning machines to perform complex visual reasoning.
The compositional structure inherent to visual reasoning is at the core of VQA: Visual reasoning is a composition of visual subtasks, and also, visual scenes are compositions of objects, which are composed of attributes such as textures, shapes and colors.
This compositional structure yields a distribution of image-question pairs of combinatorial size, which cannot be fully reflected in an unbiased way by training distributions.
Systematic generalization is the ability to generalize to novel compositions of known concepts beyond the training distribution (Lake & Baroni, 2018; Bahdanau et al., 2019;
Ruis et al., 2020).
A learning machine capable of systematic generalization is still a distant goal, which contrasts with the exquisite ability of current learning machines to generalize in-distribution.
In fact, the most successful learning machines, i.e., Transformer-based models, have been tremendously effective for
VQA when evaluated in-distribution (Tan & Bansal, 2019; Chen et al., 2020; Zhang et al., 2021).
Yet, recent studies stressed the need to evaluate systematic generalization instead of in-distribution generalization (Gontier et al., 2020; Tsarkov et al., 2021; Bergen et al., 2021), as the systematic generalization capabilities of Transformers for VQA are largely unknown.
A recent strand of research for systematic generalization in VQA investigates Neural Module Networks (NMNs) (Bahdanau et al., 2019; 2020; D’Amario et al., 2021).
NMNs decompose a question in VQA into sub-tasks, and each sub-task is tackled with a shallow neural network called module .
Thus, NMNs use a question-specific composition of modules to answer novel questions.
NMNs alleviate the gap between in-distribution generalization and systematic generalization due to its inherent compositional structure.
In our experiments, we found that CNN-based NMNs outperform
Transformers on systematic generalization to novel compositions of sub-tasks.
This begs the question of whether we can combine the strengths of Transformers and NMNs in order to improve the systematic generalization capabilities of learning machines. [Figure] Module
[Figure]
Module C …
Module architecture
Transformer Encoder Transformer Encoder
Module D
Module E 𝐾
Image
Sub-task_C(Arg1)Sub-task_A(Arg2)
Sub-task_E(Arg3)
Module E
Module C
Module A
Answer
Sub-task_A(Arg1)Sub-task_A(Arg2)Sub-task_D(Arg3)Sub-task_B(Arg4)
Module B
Module A
Answer
Module A
Module D
Image 𝐿
Example Example
Program 𝑃
Program 𝑃 1 𝐼𝐼𝐴𝐴
[Figure]
X, 13, 90, 79 X, 13, 126,
Question 𝑄 Question 𝑄
[Figure]
Module C …
Module architecture
Transformer Encoder Transformer Encoder
Module D
Module E 𝐾
Image
Sub-task_C(Arg1)Sub-task_A(Arg2)
Sub-task_E(Arg3)
Module E
Module C
Module A
Answer
Sub-task_A(Arg1)Sub-task_A(Arg2)Sub-task_D(Arg3)Sub-task_B(Arg4)
Module B
Module A
Answer
Module A
Module D
Image 𝐿
Example Example
Program 𝑃
Program 𝑃 1 𝐼
[Figure]
𝐼𝐴
[Figure]
𝐴…
X, 13, 90, 79 X, 13, 126,
[Figure]
Transformer layers is the number of Transformer layers of each module ( K ) × the program length ( L ).
In this paper, we introduce Transformer Module Network (TMN) , a novel NMN for VQA based on compositions of Transformer modules.
In this way, we take the best of both worlds: the capabilities of Transformers given by attention mechanisms, and the flexibility of NMNs to adjust to questions based on novel compositions of modules.
TMN allows us to investigate whether and how modularity brings benefits to Transformers in VQA.
An overview of TMNs is depicted in Fig.
To foreshadow the results, we find that TMNs achieve state-of-the-art systematic generalization accuracy in the following three VQA datasets:
CLEVR-CoGenT (Johnson et al., 2017), CLOSURE (Bahdanau et al., 2020), and a novel test set based on GQA (Hudson & Manning, 2019) that we introduce for evaluating systematic generalization performance with natural images, which we call GQA-SGL (Systematic Generalization to Linguistic combinations).
Remarkably, TMNs improve systematic generalization accuracy over standard Transformers more than 30% in the CLOSURE dataset, i.e., systematic generalization to novel combinations of known linguistic constructs (equivalently, sub-tasks).
Our results also show that both module composition and module specialization to a sub-task are key to TMN’s performance gain.
R ELATED WORK
We review previous works on systematic generalization in VQA.
We first revisit the available benchmarks and then introduce existing approaches.
Benchmarking VQA.
Even though systematic generalization capabilities are the crux of VQA, attempts to benchmark these capabilities are only recent.
The first VQA datasets evaluated indistribution generalization, and later ones evaluated generalization under distribution shifts that do not require systematicity.
In the following, we review progress made towards benchmarking systematic generalization in VQA:
– In-distribution generalization: There is a plethora of datasets to evaluate in-distribution generalization, e.g., VQA-v2 (Goyal et al., 2019) and GQA (Hudson & Manning, 2019).
It has been reported that these datasets are biased and models achieve high accuracy by relying on spurious correlations instead of performing visual reasoning (Agrawal et al., 2018; Kervadec et al., 2021).
– Out-of-distribution generalization: VQA-CP (Agrawal et al., 2018) and GQA-OOD (Kervadec et al., 2021) were proposed to evaluate generalization under shifted distribution of question-answer pairs.
While this requires a stronger form of generalization than in-distribution, it does not require tackling the combinatorial nature of visual reasoning, and models can leverage biases in the images and questions.
– Systematic generalization: CLEVR-CoGenT (Johnson et al., 2017) and CLOSURE (Bahdanau et al., 2020) are datasets that require systematic generalization as models
need
to
tackle novel combinations of visual attributes and sub-tasks Since these datasets include only synthetic
images,
we introduce GQA-SGL, a novel test set based on GQA to evaluate systematic generalization with natural images.
Approaches for Systematic Generalization.
We now revisit Transformer-based models and NMNs for systematic generalization in VQA as they are the basis of TMNs:
– Transformer-Based Models: Currently, most approaches to VQA are based on Transformers (Vaswani et al., 2017), e.g., Tan & Bansal (2019); Chen et al.
; Zhang et al.
(2021), and pre-training is at the core of these approaches.
These works focus on in-distribution generalization, and it was not until recently that Transformers for systematic generalization have been investigated.
To the best of our knowledge, in VQA the only related work is MDETR (Kamath et al., 2021), which uses a novel training approach that captures the long tail of visual concepts and achieves state-of-theart performance on many vision-and-language datasets, including CLEVR-CoGenT. As we show in the sequel, our approach shows better systematic generalization capabilities in CLEVR-CoGenT and CLOSURE without requiring pre-trained Transformer encoders.
– Neural Module Networks (NMNs): They represent a question in the form of a program in which each sub-task is implemented with a neural module.
Thus, modules are composed into a network specific for the question (Andreas et al., 2016).
Some of the most successful modular approaches include NS-VQA (Yi et al., 2018), which uses a symbolic execution engine instead of modules based on neural networks, and the Meta-Module Network (MMN)
(Chen et al., 2021), which introduces a module that can adjust to novel sub-tasks.
While these approaches are effective for in-distribution generalization, they fall short in terms of systematic generalization
Vector-NMN (Bahdanau et al., 2020) is the state-of-the-art NMN on systematic generalization and outperforms all previous modular approaches.
We present evidence that the systematic generalization capabilities of modular approaches can be improved by using the attention mechanisms of Transformer architectures.
T RANSFORMER M ODULE N ETWORKS (TMN S )
In this section we introduce TMNs, i.e., our novel Transformer-based model for VQA that composes neural modules into a question-specific Transformer network.
Unlike previous NMNs, TMNs regard the tokens that represent the input image as a workspace shared among the modules, which transform the token representations.
This architectural change is not straightforward and thus we need to architect NMNs taking into account the philosophy of tokens to leverage the attention mechanism.
Question-specific Composition of Modules.
Figure 1 provides an overview.
TMNs use a library of modules, and each module tackles a different sub-task we defined (e.g., FILTER , COUNT ).
TMNs represent the question in VQA with a program, which is a sequence of sub-tasks each of which is implementable with a module from the library.
For instance, a question “How many red squares are there?” can be converted to a program { FILTER(square) , FILTER(red) , COUNT } , and then the modules corresponding to the sub-tasks are composed to form a network to answer the question.
In TMN, each module is a stack of Transformer encoders.
Unlike the standard Transformers, which are a stack of a fixed number of Transformer encoders (typically 12) regardless of the input question,
TMN composes the Transformer modules into a question-specific network, and thus, the total number of encoder layers in TMN varies according to the question.
Let K be the number of Transformer encoder layers in each module, and let L be the number of sub-tasks in the program.
Then, TMN has a total of K × L layers to implement a program of length L .
We follow the same approach in previous works that directly use the program rather than the raw question (Bahdanau et al., 2020; 2019; D’Amario et al., 2021).
This allows to analyse systematic generalization in isolation by focusing only on visual reasoning aspects, and omits the performance of the language parser.
Transformer-based Architecture.
The aforementioned question-specific composition of modules is preceded by a feature extractor and followed by an output classifier, as shown in Fig. 2(a).
We use a feature extractor to obtain the visual features of the input image.
We follow the common procedure in Transformers: The visual features are transformed into the visual feature embeddings
Answer :
FILTER(square)FILTER(red)
COUNT
FILTER Module
Feature extractor
COUNT Module square null
Headtoken
Image
Program
FILTER Module 1 red
Embedding
Embedding 0 1 2 3 …
Library of modules …
Classifier
ICLR
[Figure]
Program
Answer
Module (6)
Module (1)
Module (2)
Module (3)
Module (4)
AND / OR (5)
Image
Program
Answer
Module (6)
Module (1)
Module (2)
Module (3)
Module (4)
AND / OR (5)
Image
Program Image 45 x23,72 x(Stack) (Tree)
[Figure]
Then the first Transformer module in the question network of tokens of the representations of the visual features the head token, and the module’s ar gument coming from the program. The Transformer module the transformed of the head token and visual features. This output is fed into the next module which also receives its corresponding argument coming from the program. Unlike Vector-NMN, whose modules always receive the initial visual representations, in TMN only the first module receives the initial visual rep resentations.
Also, due to the versatility of Transformer architectures, in TMN we can easily change the number of Transformer encoders in each module according to the complexity of the sub-task.
Finally, a classifier outputs an answer from the final representation of the head token.
Stack and Tree Modular Structures.
We further analyze the effect of modular structures.
Some modules with two inputs may be useful for constructing a network corresponding to a complex program.
For instance, a logical sub-task ( AND or OR ) or sub-tasks that compare two objects.
For programs containing such sub-tasks, we investigate two modular structures exemplified in Fig. 2(b).
Stack and tree structures are widely studied in previous works (Andreas et al., 2016;
Hu et al., 2017).
The stack structure simply stacks all modules and executes the modules in sequence.
The tree structure runs two threads in parallel and then merges their outputs with a module ( AND / OR in this example), which takes two sets of representations of visual features and a head token as an input and outputs the transformed representations of the first set in our case.
We compare these two structures in the experiments.
E XPERIMENTAL S ETUP
We use three datasets for VQA: two exising datasets and a novel test set we built.
We compare our
TMN with two Transformer-based baselines and several state-of-the-art models on those datasets.
Datasets.
To evaluate the models for systematic generalization to novel combinations of known visual attributes or linguistic constructs, we use the following three datasets.
Under review as a conference paper at ICLR 2023 Image
What color is the circle?
Question
Extractor
What color …
Stack of Transformer
Encoders
Answer
Image
Program
Extractor
FILTER
Stack of Transformer
Encoders
Answer
FILTER(circle)COLOR circle COLOR
Image
[Figure]
Question
Extractor
What
[Figure]
Stack of Transformer
Encoders
Answer Image
Program
Extractor
[Figure]
Stack of Transformer
Encoders
Answer
[Figure] [Figure]
Emb Emb
Emb
[Figure]
Image
What color is the circle?
Question
Extractor
What color …
Stack of Transformer
Encoders
Answer
Image
Program
Extractor
FILTER
Stack of Transformer
Encoders
Answer
FILTER(circle)COLOR circle COLOR
[Figure]
Question
[Figure]
– CLEVR-CoGenT: CLEVR is a diagnostic dataset for VQA models (Johnson et al., 2017) that consists of synthetic 3D scenes with multiple objects and automatically generated questions, associated with a ground-truth program formed by multiple sub-tasks.
This dataset comes with additional splits to test systematic generalization, namely, the Compositional Generalization Test (CLEVRCoGenT).
CLEVR-CoGenT is divided in two conditions where objects appear with limited color and shape combinations, that are inverted between training and testing.
Using this dataset, we can test systematic generalization to novel combinations of visual attributes.
– CLOSURE: This is a test set for models trained with CLEVR, and provides 7 benchmarks (Bahdanau et al., 2020).
The benchmarks test the systematic generalization to novel combinations of known linguistic constructs, which can be considered as sub-tasks, e.g., “is the same size as” can be a sub-task SAME SIZE .
CLOSURE uses the same synthetic images in CLEVR but contains questions which require the models to recombine the known sub-tasks in a novel way.
– GQA-SGL: GQA is a VQA dataset that consists of complex natural images and questions, associated with ground-truth programs (Hudson & Manning, 2019).
GQA-SGL (Systematic Generalization to Linguistic combinations) is a novel test set we built based on GQA to test systematic generalization to novel combinations of known linguistic constructs with natural images.
We generate new questions with ground-truth programs by combining the existing programs as shown in Fig.
4 in the Appendix A. To build the test set, we use the test-dev set from the balanced split.
For the train set, in all experiments we always use the balanced split that controls bias.
See Appendix A for details.
Methods and Settings.
We compare TMNs with two Transformer-based baseline methods shown in Fig.
In (a), standard Transformer (Transformer) takes an image and a question as the inputs.
In (b), standard Transformer with programs (Transformer w/ PR) takes a program as an input instead of the question and treat it as a sequence of tokens.
TMNs also take the program as an
input
and executes each module with corresponding arguments as shown in
Fig.
Thus, comparing Transformer w/ PR with TMNs allow us to assess the effectiveness of our Transformer modules and question-specific compositions of the modules.
We treat VQA as a classification task and train the models with crossentropy loss across the TMNs and our baselines.
– Architecture: We use vanilla Transformer encoder for both standard Transformers and our TMNs for fair comparison.
Standard Transformers consist of 12 Transformer layers.
In TMNs, the number of Transformer layers in each module is 1 for CLEVR-CoGenT and CLOSURE, and 2 for GQASGL because of the difference in the complexity of their sub-tasks.
Transformer encoders in all models have a hidden state size of 768 and 12 attention heads.
– Sub-tasks and programs: For CLEVR-CoGenT and CLOSURE, we use the sub-tasks and programs defined in the dataset.
For GQA-SGL, we follow the definitions proposed in Chen et al.
– Visual features: For CLEVR-CoGenT and CLOSURE, we use grid features extracted by a ResNet-101 pre-trained with ImageNet to obtain the visual feature of H × W × 2048 dimensions, where H and W are the height and the width of the grid feature map.
We add a positional encoding to the grid features and flatten them with a linear projection.
We treat the flattened features as a sequence of tokens and thus the number of the tokens is H × W ( 150 on these datasets).
The
input images in GQA contain various objects and are much more complex than those in CLOSURE.
Therefore, for GQA-SGL, we use the popular regional features (Anderson et al., 2018) extracted by a Faster R-CNN (Ren et al., 2015) (with a ResNet-101 backbone) pre-trained on Visual Genome dataset (Krishna et al., 2016).
We consistently keep 36 regions for each image.
Each region has 2048 -dimensional visual feature and 5 -dimensional spatial location feature.
– Embeddings: The input representation of the program is a sum of word embeddings, segment embeddings and position embeddings.
They represent a sub-task or an argument (e.g., FILTER or circle ), a thread index, and word position, respectively.
For the standard Transformer, we adopt BERT tokenizer and the standard sentence embeddings described in (Li et al., 2020).
– Hyperparameters: We use the Adam optimizer for all cases.
Learning rates for standard Transformers and TMNs are 2e-5 and 1e-5 for CLEVR-CoGenT, 2e-5 for CLEVR, and 1e-5 and 4e-5 for GQA.
We search best learning rates for each model.
We train all the models with batch size ofon 4 Tesla V100 GPUs for 20 epochs except standard Transformer on CLEVR, where we useepochs to achieve convergence.
Training of standard Transformers and TMNs finished in 3 anddays, respectively.
We use this experimental setup and hyperparameter tuning for all experiments.
– Other methods: We also experiment with other state-of-the-art methods, including VectorNMN (Bahdanau et al., 2020), MDETR (Kamath et al., 2021) and MMN (Chen et al., 2021) (LXMERT (Tan & Bansal, 2019) in Appendix G).
We use the hyperparamters described in the original papers.
For a fair comparison, we train Vector-NMN and MMN using their publicly available code on the same training set as the rest of the methods.
We also use
the ground-truth programs as in TMN.
MDETR was already trained on the same training set as the rest of the methods, and hence, we use the publicly available trained model.
Also, we report the performance of NS-VQA (Yi et al., 2018) from its original paper and Akula et al.
R ESULTS AND A NALYSIS
In this section, we first report the systematic generalization performance to novel combinations of visual attributes or linguistic constructs with synthetic images (CLEVR-CoGenT and CLOSURE) and complex natural images (GQA-SGL).
Then, we analyze the effect of module composition, module specialization, and pre-training.
5. S YSTEMATIC G ENERALIZATION P ERFORMANCE
Tables 1 and 2 show the mean and standard deviation accuracy of TMNs and the rest of approaches for both in-distribution and systematic generalization (see Appendix B, C, D and J for further details).
In the following, we first introduce the results in the three datasets, and then, we discuss them:
CLEVR-CoGenT.
It evaluates the systematic generalization performance to novel combinations of known visual attributes.
All models are trained on CLEVR-CoGenT condition A.
Our results show that both standard Transformers and TMNs largely outperform the state-of-the-art modular approaches (Vector-NMN and NS-VQA).
TMNs’ systematic generalization performance also surpasses MDETR, which is the state-of-the-art Transformer-based model proposed to capture longtail visual concepts.
TMNs achieve superior performance over the standard Transformer but their performances are slightly lower than the Transformer with programs (Transformer w/ PR).
As expected, Transformer w/ PR outperforms standard Transformer as the use of program facilitates visual grounding.
CLOSURE.
It evaluates the systematic generalization performance to novel combinations of known linguistic constructs.
All models are trained on CLEVR.
Our results show that standard
Transformers struggle with novel compositions of known linguistic constructs even when the program is provided (Transformer w/ PR).
TMNs achieve much better performance than standard Transformers and MDETR, and also outperform the other modular approaches (Vector-NMN and NSVQA).
Remarkably, TMN-Tree improves systematic generalization accuracy over standard Transformers more than 30% in this dataset.
Tree structure seems effective for CLOSURE because the questions often have tree structure. [Table] [Table]
GQA-SGL.
We evaluate the systematic generalization performance to novel combinations of known linguistic constructs with natural images.
All models are trained on GQA.
In Table
[Table]
we evaluate the exact matching between the predicted and ground-truth answers, on both GQA and
GQA-SGL.
TMN-Tree achieves superior performance on systematic generalization with a smaller drop of in-distribution accuracy compared to standard Transformers.
Although MMN is the recently proposed NMN for GQA and achieves great in-distribution performance, TMN-Tree outperforms
MMN on systematic generalization.
Discussion of the Results.
Our results in CLOSURE and GQA-SGL show
that TMNs (i
[Table]
e., introducing modularity to Transformers) improve the systematic generalization performance to novel combinations of known linguistic constructs (i
[Table]
e., novel compositions of sub-tasks) of Transformers and NMNs.
The performance gain in CLOSURE is relatively larger than that in GQA-SGL tests.
One reason to explain this is the complexity of the questions: the maximum program length isin CLOSURE, while it is 9 in GQA.
CLOSURE provides more complex questions which require stronger systematic generalization capabilities.
Results in CLEVR-CoGenT show that TMNs do not improve systematic generalization of Transformers for novel compositions of visual attributes.
This may be expected as D’Amario et al. (2021) demonstrated that the visual feature extractor is the key component for generalization to novel combinations of visual attributes, and Transformer and TMNs have the same visual feature extractor.
Finally, we observe that TMNs achieve the best accuracy across all tested methods in CLOSURE and GQA-SGL, and competitive accuracy with the best performing method in CLEVR-CoGenT.
[Table] [Table]
Methods
[Table]
VL
[Table]
5. A NALYSIS Behavioral Analysis.
We analyze the behavior of our trained modules, i
[Table]
e., how Transformer modules work, by visualizing attention maps in Appendix H. The qualitative results indicate that the modules learn their functions properly.
We also analyze the Transformer w/ PR in Appendix I.
Module Specialization.
While the question-specific composition of modules is a key property of
NMNs, a recent work reports that a degree of modularity of the NMN also has large influence on systematic generalization (D’Amario et al., 2021).
Here we also investigate this aspect for TMNs.
Table 3 shows the mean accuracy of TMN-Stack with four different libraries of modules, each with a different degree of specialization.
“Individual” denotes that each module is assigned to a single sub-task ( 26 sub-tasks defined in CLEVR in total).
This is the library used in the previous sections.
“Semantic group” denotes that each module tackles a group of sub-tasks that are semantically similar.
“Random group” denotes that each module tackles a group of sub-tasks randomly selected.
“Order” denotes that each module is always placed in the same position of the question-specific network, independently of the sub-task performed in that position.
If there are more sub-tasks in the program than modules in the library, the modules are repeated following the same order.
In this way, we compare the importance of modules tackling sub-tasks with the importance of having a question-specific network.
See Appendix E for further details about each library.
Results shows that the systematic generalization accuracy largely drops when the modules are not specialized, even if the number of modules is the same (compare “Order” with “Semantic group” and “Random group”).
The degree of modularity (Individual vs Semantic group) and how sub-tasks are grouped (Semantic vs Random) also affects the systematic generalization performance.
These results suggest that question-specific composition of modules is not enough and specialization of modules is crucially important.
Finally, note that the results of Semantic group and Random group also show that TMN performs better than the standard Transformer ( 93 .
7 vs 64 . 5 )
even if the total number of parameters that all modules have is the same as the number of parameters of the standard
Transformer (i.e., TMN has 12 modules of one layer each, while Transformers have 12 layers).
Ablation Analysis.
We investigate the contribution of TMN’s specialization of modules (SoM) to
TMNs’ superior performance over standard Transformer.
Besides SoM, TMNs also have a variable
[Table]
number of layers, while standard Transformer have a fixed number of layers.
We analyze if it is this difference in number of layers or the SoM that is causing TMNs’ higher accuracy.
To do so, we add a property named Variable number of layers (VL) to the Transformer w/ PR.
With the VL, the Transformer w/ PR uses as many encoder layers as the program length (e.g., 18 layers are used to output an answer for an input program with a length of 18).
VL can be regarded as an imitation of the question-specific compositions of the modules in TMN, but each layer is not specialized for specific sub-tasks.
Another key difference between TMNs and Transformers besides SoM is that
TMN modules takes as argument tokens related to the sub-task as indicated by the program, while every layer of standard Transformer takes all input tokens in the program, because Transformers’ layers are not specialized to sub-tasks.
Splitting the program tokens may help the model to steer the computations that should be executed at each layer.
To investigate this, we add this property, named as Split Tokens (ST), to the Transformer w/ PR.
4 shows that any of the aforementioned properties do not improve the systematic generalization performance without the specialization of modules to sub-tasks, i.e., SoM. Even with Split tokens, the performance does not improve because the layers can not be specialized and splitting the token can spoil the flexibly of the standard Transformer.
Pre-training.
We analyze how the additional image data from other domains affects systematic generalization to novel combinations of visual attributes or linguistic constructs, i.e., CLEVRCoGenT and CLOSURE, respectively.
Table 5 shows the mean accuracy with two different visual features, namely, regional features extracted by an object detector pre-trained on Visual Genome (VG) and grid features extracted by a ResNet-101 without pre-training on VG.
Across all the tested models, we observe, as expected, that the performance gains from pre-training the visual feature extractor are larger for generalization to novel visual combinations and smaller for novel linguistic combinations.
This highlights the need to control pre-training in order to ensure a fair comparison of different methods for systematic generalization.
See Appendix F and G for further experiments.
C ONCLUSIONS
We have compared the systematic generalization capabilities of TMNs with the most promising approaches in the literature, i.e., Transformers and NMNs, on three VQA datasets.
TMNs achieve state-of-the-art systematic generalization performance across all tested benchmarks.
We have shown that the question-specific composition of Transformer modules and the specialization of the modules to each sub-task are the key of the performance gains.
Also, our results highlight that a fair comparison for systematic generalization requires controlling that all compared methods are pre-trained in the same way.
We hope that our investigation unleash the potential of modular approaches for systematic generalization as they motivate promising future work towards creating large-scale modular architectures.
Limitations.
TMNs have similar limitations as NMNs, i.e., it is challenging to decompose a question into sub-tasks.
We have used the sub-tasks and the programs provided in the datasets generated by humans.
This makes it difficult to apply NMNs and TMNs to other datasets that do not contain the programs.
Another limitation is that we only investigated two types of systematic generalization (novel compositions of visual attributes and linguistic constructs).
These limitations are key open questions that we will tackle in future work.
Reproducibility Statement.
The source code and new dataset used in this study are included in the supplementary materials for review and will be made publicly available on GitHub.com upon acceptance.
URLs of other existing assets (datasets, source code and trained models) used in this study are provided in the reference section.
Experimental setup is described in Section 4.
B S UPPLEMENTAL R ESULTS ON CLEVR-C O G EN T
We report the systematic generalization performance for each question type on CLEVR-CoGenT in
Table 6.
All methods are trained on CLEVR-CoGenT condition A and tested on CLEVR-CoGenT validation condition B.
C S UPPLEMENTAL R ESULTS ON CLOSURE
We report the systematic generalization performance for each test in CLOSURE in Table 7.
All methods are trained on CLEVR and tested on CLOSURE tests.
D S UPPLEMENTAL R ESULTS ON GQA-SGL
We report the systematic generalization performance for each question type on GQA-SGL in Table 8.
All methods are trained on GQA balanced split.
E L IBRARIES OF M ODULES
We design three libraries of modules for TMN, “Individual”, “Semantic group”, and “Random group”, as shown in Table 9.
Unlike these libraries, a module in “Order” library is selected by the order of a sub-task in a given program and thus in this library the modules are not specialized for specific sub-tasks.
F E FFECT OF R EGIONAL F EATURES
We compare the systematic generalization performances on CLEVR-CoGenT with grid features and regional features, as shown in Table 10.
We use ResNet-101 and region of interest pooling to extract regional features, while we use only the same ResNet-101 to extract grid features.
In case of the standard Transformer and our TMNs, the performance gaps are less than 1 . 5% .
In case of the standard Transformer with ground-truth programs, the performance gap is slightly larger than it ( 3 . 6% ).
G E FFECT OF P RE -T RAINING WITH I MAGE –T EXT PAIRS
Table 11 reports the results of TMN-Tree and state-of-the-art Transformer-based models (MDETR (Kamath et al., 2021) and LXMERT (Tan & Bansal, 2019)) on GQA and GQA-SGL.
The table also indicates the pre-training datasets with image-text pairs.
The methods use different pre-training datasets, and hence, they cannot be compared in a fair manner.
Yet, we observe that
TMN-Tree strike a balance between amount of pre-training data for the Transformer encoders and systematic generalization performance. [Table] scene query color [Table]
count count query shape
[Table]
exist exist union [Table] intersect intersect same shape relate relate equal shape union union greater than unique unique equal material greater than greater than filter shape less than intersect equal integer unique less than equal color same material equal material filter material equal shape count equal size same size equal color filter color equal size filter material query material filter shape scene filter size equal color equal integer query color query size query material less than query shape exist query size relate equal material same color filter size same material equal integer same shape filter color same size same color equal shape equal size filter color filter material filter shape filter size query color query material query shape query size same color same material same shape same size [Table]
[Table] [Figure] H B EHAVIOURAL A NALYSIS OF M ODULES Figure 6 shows an visualization of how Transformer modules work, obtained by using examples in CLEVR, i e., in-distribution case.
In the left example, we give TMN-Stack an input image with a program { scene , filter shape(sphere) , filter color(yellow) , count } .
Three modules corresponding to the program operate their sub-tasks sequentially and then a classifier outputs an answer.
We visualize the attention map in each module, where darker grey represents higher attention scores.
We average attention scores across all attention heads.
We can see that filter shape(sphere) and filter color(yellow) modules highly attend to regions with sphere and yellow objects, respectively.
More precisely, the queries of the regions highly attend to the keys of sphere and yellow which are the input for each module, and the representations of the regions change by adding the values of sphere and yellow .
Although filter color(yellow) module executes its sub-task after filter shape(sphere) , filter color(yellow) also attends to a yellow cube as well as yellow spheres.
In the right example, we give an another image with a program { scene , filter material(metal) , exist } .
These results indicate that the modules learn their
[Table]
functions properly. Figure 7 shows a behavioural analysis on an example in CLOSURE, i.e., novel combinations of known linguistic constructs. In this example, we give TMN-Tree an input image with a program { scene , filter shape(cube) , scene , filter size(large) , filter shape(cube) , unique , same material , filter shape(sphere) , union , count } . scene represents the beginning of a thread and this program contains two threads.
TMN-Tree operates the threads individually and merges them with union ( OR ). For simplicity, we overlay attention maps on the input image. The result show that the each module works properly as we expect even though this example is much more complex than the ones shown in Fig. 6 and also the sequence of the sub-tasks are unseen in the training.
Figure 8 and 9 show how Transformer w/ PR works on the same example in CLOSURE shown in Fig. 7 and an example in CLEVR similar to the CLOSURE example.
Both examples contain union ( OR ), however the CLEVR example is in-distribution while the CLOSURE example requires the model to handle novel combinations of known linguistic constructs, i.e., systematic generalization.
We visualize the attention map of the head token for the visual regions and the input sequence of the tokens.
The results show that the Transformer w/ PR first attends to the input program then changes where to attend in the visual regions.
In Fig.
9, i.e., in-distribution case, the model attends to the token almost from left to right (i.e., order of sub-tasks) and after reaching the last token it stop to attend the program.
Specifically, the model first attends to the regions with metal cube from layer 2 to 3 and the regions of sphere at layer 5 , then attends to the both regions at layer 6 after attending the union at the previous layer.
Finally, the model attends to the count .
This result indicates that the Transformer w/ PR learns how to operate the input programs to output the correct answers.
In Fig. 8, i.e., systematic generalization case, the input program contains same material in addition to union .
Both sub-tasks appear in the training while the combination is novel.
Fig.shows that the model tried to operate the input program in the same manner as the CLEVR example shown in Fig.
9 but failed to lead a correct answer.
Specifically, at the same layer as the CLEVR example, the model attends to the union and count , however the attended regions are not correct.
After that the model seems to operate same material that should be operated before union . These results indicate that Transformer w/ PR struggles to operate the programs that contain the novel combinations of the sub-tasks. J A BLATION S TUDIES FOR T RANSFORMERS We test Transformers and Transformers w/ PR with the different number of encoder layers to investigate the effect of model size.
We train all models in the same way we described in the main paper. Table 12 shows that the performance of Transformers with fewer layers slightly drops except Transformer with 6 layers.
These results show that we cannot improve the systematic performance by simply changing the number of layers. filter_shape(cube) filter_size(large) filter_shape(cube)
same_material filter_shape(sphere) count union filter_color(green) filter_shape(cube) same_material ‘no’ filter_color(green) filter_shape(cube) same_material ‘0’ Correct : ‘yes’ Correct : ‘1’
[Figure]
Prediction: 1 (correct: 3 - fail)
[Figure] [Figure] [Figure]