V ISUAL R EASONING BY P ROGRESSIVE M ODULE N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans learn to solve tasks of increasing complexity by building on top of previously acquired knowledge.
Typically, there exists a natural progression in the tasks that we learn – most do not require completely independent solutions, but can be broken down into simpler subtasks.
We propose to represent a solver for each task as a neural module that calls existing modules (solvers for simpler tasks) in a functional program-like manner.
Lower modules are a black box to the calling module, and communicate only via a query and an output.
Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output.
Our model effectively combines previous skill-sets, does not suffer from forgetting, and is fully differentiable.
We test our model in learning a set of visual reasoning tasks, and demonstrate improved performances in all tasks by learning progressively.
By evaluating the reasoning process using human judges, we show that our model is more interpretable than an attention-based baseline.
I NTRODUCTION
Humans acquire skills and knowledge in a curriculum by building on top of previously acquired knowledge.
For example, in school we ﬁrst learn simple mathematical operations such as addition and multiplication before moving on to solving equations.
Similarly, the ability to answer complex visual questions often requires the skills to understand attributes such as color, recognize a variety of objects, and be able to spatially relate them.
Just like humans, machines may also beneﬁt by sequentially learning tasks in progressive complexity and composing knowledge along the way.
The process of training a machine learning model to be able to solve multiple tasks, or multi-task learning (MTL), has been widely studied (Long et al., 2017; Ruder, 2017; Ruder et al., 2017; Rusu et al., 2016).
The dominant approach is to have a model that shares parameters ( e.g ., bottom layers of a CNN) with individualized prediction heads (Caruana, 1993;
Long et al., 2017).
By sharing parameters, models are able to learn better task-agnostic data representations.
However, the tasks are disconnected as their outputs are not combined to solve tasks of increasing complexity.
It is desirable if one task can learn to process the predictions from other tasks thereby reaping the beneﬁts of MTL.
In this paper, we address the problem of MTL where tasks exhibit a natural progression in complexity.
We propose Progressive Module Networks (PMN), a framework for multi-task learning by progressively designing modules on top of existing modules.
Each module is a neural network that can query modules for lower-level tasks, which in turn may query modules for even simpler tasks.
The modules communicate by learning to query other modules and process their outputs, while the internal module processes are a blackbox.
This is similar to a computer program that uses available libraries without having to know their internal operations.
Parent modules can choose which lower-level modules they want to query via a soft gating mechanism.
Examining the queries, replies, and choices a parent module makes, we can understand the reasoning behind the module’s output.
PMN is related but different from Andreas et al.
(2016) and Hu et al.
PMN’s modules are task-level modules, and they are compositional,
i.e . modules build on modules which build on modules.
It allows efﬁcient use of data by not needing to re-learn previously acquired knowledge.
By learning selective information ﬂow between modules, interpretability arises naturally.
We demonstrate PMN in learning a set of visual reasoning tasks such as counting, captioning, and
Visual Question Answering (VQA).
PMN outperform baselines without module composition on all tasks.
We further analyze the interpretability of PMN’s reasoning process with human judges.
Minput
[Figure]
M
M M
M M
M
M1    78 11 =
[ , , ] L 3 M 0 M 1 M 2 = [ , ] L 2 M 0 M= [ ] L 1 M= L n ordered list of modules calls M n
R ELATED W ORK
Multi-task learning.
The dominant approach to multi-task learning is to have a model that shares parameters in a soft (Duong et al., 2015; Yang & Hospedales, 2017) or hard way (Caruana, 1993).
Soft sharing refers to each task having independent weights that are constrained to be similar ( e.g . via (cid:96) 2 regularization (Duong et al., 2015), trace norm (Yang & Hospedales, 2017)) while hard sharing typically means that all tasks share the base network but have independent layers on the top (Kokkinos, 2017; Misra et al., 2016).
While sharing parameters helps to compute a task-agnostic representation that is not overﬁt to a speciﬁc task, tasks do not directly share information or help each other.
Bilen & Vedaldi (2016) propose the Multinet architecture where tasks can interact with each other in addition to shared image features.
Multinet solves one task at each time step and appends the encoded output of each task to existing data representation.
A similar idea, Progressive Neural
Networks (PNNs) (Rusu et al., 2016) use a new neural network for each task, but are designed to prevent catastrophic forgetting as they transfer knowledge from previous tasks by making lateral connections to representations of previously learned tasks.
Recently, Wang et al.
(2017) propose the VQA-Machine which exploits a set of existing algorithms to solve the VQA problem.
Zamir et al.
(2018) learn a computational taxonomy for task transfer learning on several vision problems.
However, the major differences to this work are PMN’s compositional modular structure, ability to directly query other modules, and the overall process of learning increasingly complex tasks.
Module networks.
Pioneering work in modular structure, NMN (Andreas et al., 2016;
Hu et al., 2017) addresses VQA where questions have a compositional structure.
Given an inventory of small networks, or modules, NMN produces a layout for assembling the modules for any question.
PMN is different from NMN as it can be easily extended to new tasks using its compositional structure.
It also treats each task as modules which opens up promising ways to train models that can compartmentalize and perform multiple tasks as they progressively improve their abilities.
Visual question answering.
VQA has seen great progress in recent years: improved multimodal pooling functions (Fukui et al., 2016; Kim et al., 2018), multi-hop attention (Yang et al., 2016), driving attention through both bottom-up and top-down schemes (Anderson et al., 2018), and modeling attention between words and image regions recurrently (Hudson & Manning, 2018) are some of the important advances.
There are also attempts to generate programs or sequence of modules automatically that yield a list of interpretable steps (Hu et al., 2017; Johnson et al., 2017b) using policy gradient optimization.
Our approach treats visual reasoning as a compositional multi-task problem, and shows that using sub-tasks compositionally can help improve performance and interpretability.
P ROGRESSIVE M ODULE N ETWORKS
Most complex reasoning tasks can be broken down into a series of sequential reasoning steps.
We hypothesize that there exists a hierarchy with regards to complexity and order of execution: high level tasks ( e.g . counting) are more complex and beneﬁt from leveraging outputs from lower level tasks ( e.g . classiﬁcation).
For any task, Progressive Module Networks (PMN) learn a module that requests and uses outputs from lower modules to aid in solving the given task.
This process is compositional, i.e ., lower-level modules may call modules at an even lower level.
Solving a task corresponds to executing a directed acyclic computation graph where each node represents a module (see Fig. 1).
PMN has a plug-and-play architecture where modules can be replaced by their improved versions.
This opens up promising ways to train intelligent robots that can compartmentalize and perform multiple tasks while progressively learning from experience and improving abilities.
PMN also chooses which lower level modules to use through a soft-gating mechanism.
A natural consequence of PMN’s modularity and gating mechanism is interpretability.
While we do not need to know the internal workings of modules, we can examine the queries and replies along with the information about which modules were used to reason about why the parent module produced a certain output.
Formally, given a task n at level i , the task module M n can query other modules M k at level j such that j < i .
Each module is designed to solve a particular task (output its best prediction) given an input and environment E .
Note that E is accessible to every module and represents a broader set of “sensory” information available to the model.
For example, E may contain visual information such as an image, and text in the form of words ( i.e ., question).
PMN has two types of modules: (i) terminal modules execute the simplest tasks that do not require information from other modules (Sec. 3.1); and (ii) compositional modules that learn to efﬁciently communicate and exploit lower-level modules to solve a task (Sec. 3.2).
We describe the tasks studied in this paper in Sec.
3.3 and provide a detailed example of how PMN is implemented and executed for VQA (Sec. 3.4).
3. T ERMINAL M ODULES
Terminal modules are by deﬁnition at the lowest level 0 .
They are analogous to base cases in a recursive function.
Given an input query q , a terminal module M (cid:96) generates an output o = M (cid:96) ( q ) , where M (cid:96) is implemented with a neural network.
A typical example of a terminal module is an object classiﬁer that takes as input a visual descriptor q , and predicts the object label o .
3. C OMPOSITIONAL M ODULES
A compositional module M n makes a sequence of calls to lower level modules which in turn make calls to their children, in a manner similar to depth-ﬁrst search (see Fig. 1).
We denote the list of modules that M n is allowed to call by L n =
[ M m , . . .
, M l ] .
Every module in L n has level lower than
M n .
Since lower modules need not be sufﬁcient in fully solving the new task, we optionally include a terminal module ∆ n that performs “residual” reasoning.
Also, many tasks require an attention mechanism to focus on certain parts of data.
We denote Ω n as a terminal module that performs such soft-attention.
∆ n and Ω n are optionally inserted to the list L n and treated as any other module.
The compositional aspect of PMN means that modules in L n can have their own hierarchy of calls.
We make L n an ordered list, where calls are being made in a sequential order, starting with the ﬁrst in the list.
This way, information produced by earlier modules can be used when generating the query for the next.
For example, if one module is performing object detection, we may want to use its output (bounding box proposals), for querying other modules such as an attribute classiﬁer.
For this work, the list L n , and thus the levels of tasks, are determined by hand.
Relaxing this and letting the model learn the task hierarchy itself is a challenging direction that we leave for future work.
Also, notice that the number of back-and-forth communications increases exponentially if each module makes use of every lower-level module.
Thus, in practice we restrict the list L n to those lower-level modules that may intuitively be needed by the task.
We emphasize that M n still (softly) chooses between them, and thus the expert intervention only removes the lower-level modules that are uninformative to the task.
Our compositional module M n runs (pre-determined) T n passes over the list L n .
It keeps track of a state variable s t at time step t ≤ T n .
This contains useful information obtained by querying other modules.
For example, s t can be the hidden state of a Recurrent Neural Network.
Each time step corresponds to executing every module in L n and updating the state variable.
We describe the module components below, and Algorithm 1 shows how the computation is performed.
An example implementation of the components and demonstration of how they are used is detailed in Sec.
State initializer.
Given a query (input) q n , the initial state s 1 is produced using a state initializer I n .
Importance function.
For each module M k (and ∆ n , Ω n ) in L n , we compute an importance score g kn with G n ( s t ) .
The purpose of g kn is to enable M n to (softly) choose which modules to use.
This also enables training all module components with backpropagation.
Notice that g kn is input dependent, and thus the module M n can effectively control which lower-level module outputs to use in state s t .
Here, G n can be implemented as an MLP followed by either a softmax over submodules, or a sigmoid that outputs a score for each submodule.
However, note that the proposed setup can be modiﬁed to adopt hard-gating mechanism using a threshold or sampling with reinforcement learning.
Query transmitter and receiver.
A query for module M k in L n is produced using a query transmitter , as q k = Q n → k ( s t , V, G n ( s t )) .
The output o k = M k ( q k ) received from M k is modiﬁed using a receiver function , as v k
= R k
→ n ( s t , o k ) .
One can think of these functions as translators of the inputs and outputs into the module’s own “language".
Note that each module has a scratch pad V to store outputs it receives from a list of lower modules L n , i.e ., v k is stored to V .
Q n → k and R k → n stand for the query transmitter from task n to task k and receiver from task k to task n , respectively.
1: function M n ( q n ) (cid:46)
The environment E variables 2: s
1 = I n ( q n ) (cid:46) state variable 3: for t ← 1 to T n do (cid:46) is maximum time step 4:
V =
[] (cid:46) scratch pad V 5: g 1 n , . .
g |L n | n = G n ( s t ) (cid:46) scores 6: for k ← 1 to |L n | do (cid:46)
[ M
M l ] 7: q k = Q n → k s t
V
G n s t ))
8: o k =
L n [ k ]( q k ) (cid:46) call
9: v k
= R k
→ n ( s t , o k ) (cid:46)
V. append( v k ) (cid:46)
11: s t +
=
U n ( s t , V, E , G n ( s t ))
(cid:46) state 12: o n = Ψ n ( s 1 , . .
, s T n , q n , E ) (cid:46)
13: return o n
State update function.
After every module in L n is executed, module M n updates its internal state using a state update function U n as s t +
=
U n ( s t , V, E , G n ( s t )) .
This completes one time step of the module’s computation.
Once the state is updated, the scratch pad V is wiped clean and is ready for new outputs.
An example can be a simple gated sum of all outputs, i.e ., s t
+1 =
(cid:80)
k g kn · v k .
Prediction function.
After T n steps, the ﬁnal module output is produced using a prediction function Ψ n as o n = Ψ n ( s 1 , . . .
, s T n , q n , E ) .
Recall that E is the environment.
All module functions: state initializer I , importance function G , query transmitter Q , receiver R , state update function U , residual module ∆ , attention module Ω , and prediction function Ψ are implemented as neural networks or simple assignment functions ( e.g . set q k = v l ).
Note that all variables ( e.g . o k , q k , v k , s t ) are continuous vectors to allow learning with standard backpropagation.
For example, the output of the relationship detection module that predicts an object bounding box is a N dimensional soft-maxed vector (assuming there are total of N boxes or image regions in E ).
Training.
We train our modules sequentially, from low level to high level tasks, one at a time.
The internal weights of the lower level modules are not updated, thus preserving their performance on the original task.
The new module only learns to communicate with them via the query transmitter Q and receiver R .
We do train the weights of ∆ and Ω .
We train I , G , Q , R , U , and Ψ , by allowing gradients to pass through the lower level modules.
The loss function depends on the task n .
3. P ROGRESSIVE M ODULE N ETWORKS FOR V ISUAL R EASONING
We present an example of how PMN can be adopted for several tasks related to visual reasoning.
In particular, we consider six tasks: object classiﬁcation, attribute classiﬁcation, relationship detection, object counting, image captioning, and visual question answering.
Our environment E consists of: (i) image regions : N image features X =
[ X 1 , . . .
, X N ] , each X i ∈ R
d with corresponding bounding box coordinates b =
[ b 1 , . . .
, b N ] extracted from Faster R-CNN (Ren et al., 2015); and (ii) language : vector representation of a sentence S (in our example, a question).
S is computed through a Gated
Recurrent Unit (Cho et al., 2014) by feeding in word embeddings [ w 1 , . . . , w T ] at each time step.
Below, we discuss each task and a module designed to solve it.
We provide detailed implementation and execution process of the VQA module in Sec. 3.4.
For other modules, we present a brief overview of what each module does in this section.
Further implementation details of all module architectures are in Appendix A.
Object and Attribute Classiﬁcation (level 0).
Object classiﬁcation is concerned with naming the object that appears in the image region, while attribute classiﬁcation predicts the object’s attributes ( e.g . color).
As these two tasks are fairly simple (not necessarily easy), we place M obj and M att as terminal modules at level 0 .
M obj consists of an MLP that takes as input a visual descriptor for a bounding box b i , i.e ., q obj =
X i , and produces o obj =
M obj ( q obj ) , the penultimate vector prior to classiﬁcation.
Attribute module M att has a similar structure.
These are the only modules for which we do not use actual output labels, as we obtained better results for higher level tasks empirically.
Image Captioning (level 1).
In image captioning, one needs to produce a natural language description of the image.
We design our module M cap as a compositional module that uses information
[Figure]
Environment
[Figure] [Figure]
from L cap =
[Ω cap , M obj , M att , ∆ cap ] .
We implement the state update function as a two-layer GRU network with s t corresponding to the hidden states.
Similar to Anderson et al.
(2018), at each time step, the attention module Ω cap attends over image regions X using the hidden state of the ﬁrst layer.
The attention map m is added to the scratch pad V .
The query transmitters produce a query (image vector at the attended location) using m to obtain nouns M obj and adjectives M att .
The residual module ∆ cap processes other image-related semantic information.
The outputs from modules in
L cap are projected to a common vector space (same dimensions) by the receivers and stored in the scratch pad.
Based on their importance score, the gated sum of the outputs is used to update the state.
The natural language sentence o cap is obtained by predicting a word at each time step using a fully connected layer on the hidden state of the second GRU layer.
Relationship Detection (level 1).
In this task the model is expected to produce triplets in the form of “subject - relationship - object” (Lu et al., 2016).
We re-purpose this task as one that involves ﬁnding the relevant item (region) in an image that is related to a given input through a given relationship.
The input to the module is q rel =
[ b i , r ] where b i is a one-hot encoding of the input box and r is a one-hot encoding of the relationship category ( e.g . above, behind).
The module produces o rel =
b out corresponding to the box for the subject/object related to the input b i through r .
We place M rel on the ﬁrst level as it may use object and attribute information that can be useful to infer relationships, i.e ., L rel =
[ M obj , M att , ∆ rel ] .
We train the module using the cross-entropy loss.
Object Counting (level 2).
Our next task is counting the number of objects in the image.
Given a vector representation of a natural language question ( e.g . how many cats are in this image?), the goal of this module is to produce a numerical count.
The counting task is at a higher-level since it may also require us to understand relationships between objects.
For example, “how many cats are on the blue chair?”, requires counting cats on top of the blue chair.
We thus place M cnt on the second level and provide it access to L cnt =
[Ω cnt , M rel ] .
The attention module Ω cnt ﬁnds relevant objects by using the input question vector.
M cnt may also query M rel if the question requires relational reasoning.
To answer “how many cats are on the blue chair”, we can expect the query transmitter
Q cnt → rel to produce a query q rel =
[ b i , r ] for the relationship module M rel that includes the chair bounding box b i and relationship “on top of” r so that M rel outputs boxes that contain cats on the chair.
Note that both Ω cnt and M rel produce attention maps on the boxes.
The state update function softly chooses a useful attention map by calculating softmax on the importance scores of Ω cnt and
M rel .
For prediction function Ψ cnt , we adopt the counting algorithm by Zhang et al.
(2018), which builds a graph representation from attention maps to count objects.
M cnt returns o cnt which is the count vector corresponding to softmaxed one-hot encoding of the count (with maximum count ∈ Z ).
Visual Question Answering (level 3).
VQA is our ﬁnal and most complex task.
Given a vector representation of a natural language question, q vqa , the VQA module M vqa uses L vqa =
[Ω vqa , M rel , M obj , M att , ∆ vqa , M cnt , M cap ] .
Similar to M cnt , M vqa makes use of Ω vqa and
M rel to get an attention map.
The produced attention map is fed to the downstream modules [ M obj , M att , ∆ vqa ] using the query transmitters.
M vqa also queries M cnt which produces a count vector.
For the last entry M cap in L vqa , the receiver attends over the words of the entire caption produced by M cap to ﬁnd relevant answers.
The received outputs are used depending on the importance scores.
Finally, Ψ vqa produces an output vector based on q vqa and all states s t .
3. E XAMPLE : M vqa FOR V ISUAL Q UESTION A NSWERING
We give a detailed example of how PMN is implemented for the VQA task.
The entire execution process is depicted in Fig. 2, and the general algorithm is tabulated in Alg. 1.
The input q vqa is a vector representing a natural language question ( i.e . the sentence vector S ∈ E ).
The state variable s t is represented by a tuple ( q t vqa , k t − 1 ) where q t vqa represents query to ask at time t and k t − 1 represents knowledge gathered at time t − 1 .
The state initializer I vqa is composed of a
GRU with hidden state dimension 512.
The ﬁrst input to GRU is q vqa , and I vqa sets s 1 = ( q 1vqa , 0 ) where q 1 vqa is the ﬁrst hidden state of the GRU and 0 is a zero vector (no knowledge at ﬁrst).
For t in T vqa = 2 , M vqa does the following seven operations:
(1) The importance function G vqa is executed.
It is implemented as a linear layer R
→ R 7 (for the seven modules in L vqa ) that takes s t , speciﬁcally q t vqa ∈ s t as input.
(2) Q vqa → Ω passes q t vqa to the attention module Ω vqa which attends over the image regions X with q t vqa as the key vector.
Ω vqa is implemented as an MLP that computes a dot-product soft-attention similar to Yang et al.
The returned attention map v Ω is added to the scratch pad V .
(3) Q vqa → rel produces an input tuple [ b, r ] for M rel .
The input object box b is produced by a MLP that does soft attention on image boxes, and the relationship category r is produced through a MLP with q t vqa as input.
M rel is called with [ b, r ] and the returned map v rel is added to V .
(4) Q vqa → obj , Q vqa → att , and Q vqa → ∆ ﬁrst compute a joint attention map m as summation of ( v Ω , v rel ) weighted by the softmaxed importance scores of (Ω vqa , M rel ) , and they pass the sum of visual features X weighted by m to the corresponding modules.
∆ vqa is implemented as an MLP.
The receivers project the outputs into 512 dimensional vectors v obj , v att , and v ∆ through a sequence of linear layers, batch norm, and tanh() nonlinearities.
They are added to V .
(5) Q vqa → cnt passes q t vqa to M cnt which returns o cnt .
R cnt → vqa projects the count vector o cnt into a 512 dimensional vector v cnt through the same sequence of layers as above.
v cnt is added to V .
(6) M vqa calls M cap and R cap → vqa receives natural language caption of the image.
It converts words in the caption into vectors [ w 1 , . . .
, w T ] through an embedding layer.
The embedding layer is initialized with 300 dimensional GloVe vectors (Pennington et al., 2014) and ﬁne-tuned.
It does softmax attention operation over [ w 1 , . . .
, w T ] through a MLP with q t vqa ∈ s t as the key vector, resulting in word probabilities p 1 , . . .
, p T .
The sentence representation (cid:80)
Ti p i · w i is projected into a 512 dimensional vector v cap using the same sequence as v cnt .
v cap is added to V .
The state update function U vqa ﬁrst does softmax operation over the importance scores of ( M obj , M att , ∆ vqa , M cnt , M cap ) .
We deﬁne an intermediate knowledge vector k t as the summation of ( v obj , v att , δ vqa , v cnt , v cap ) weighted by the softmaxed importance scores.
U vqa passes k t as input to the GRU initialized by I vqa , and we get q t
+1vqa the new hidden state of the GRU.
The new state s t +1 is set to ( q t +1vqa , k t ) .
This process allows the GRU to compute new question and state vectors based on what has been asked and seen .
After T vqa steps, the prediction function Ψ vqa computes the ﬁnal output based on the initial question vector q vqa and all knowledge vectors k t ∈ s t .
Here, q vqa and k t are fused with gated-tanh layers and fed through a ﬁnal classiﬁcation layer similar to Anderson et al.
(2018), and the logits for all time steps are added.
The resulting logit is the ﬁnal output o vqa that corresponds to an answer in the vocabulary of the VQA task.
Note that the exact form of each module can be different.
While we leave a more general architecture across tasks as future work, we stress that one of PMN’s strengths is that once a module is trained, it can be used as a blackbox by the higher-level modules.
Details of other modules’ architectures are provided in Appendix A.
E XPERIMENTS
We present experiments demonstrating the impact of progressive learning of modules.
We also analyze and evaluate the reasoning process of PMN as it is naturally interpretable.
We conduct experiments on three datasets (see Appendix B.1 for details): Visual Genome (VG) (Krishna et al., 2016), VQA 2.0 (Goyal et al., 2017), MS-COCO (Lin et al., 2014).
These datasets contain natural images and are thus more complex in visual appearance and language diversity than CLEVR (Johnson et al., 2017a) that contains synthetic scenes.
Neural module networks (Andreas et al., 2016; Hu et al., 2017) show excellent performance on CLEVR but their performance on natural images is quite below the state-of-the-art.
For all datasets, we extract bounding boxes and their feature representations using a pretrained model from Anderson et al.
4. P ROGRESSIVE L EARNING OF T ASKS AND M ODULES
Object and Attribute Classiﬁcation.
We train these modules with annotated bounding boxes from the VG dataset.
We follow Anderson et al.
and use 1,600 and 400 most commonly occurring object and attribute classes, respectively.
Each extracted box is associated with the ground truth label of the object with greatest overlap.
It is ignored if there are no ground truth boxes with IoU > 0.5.
This way, each box is annotated with one object label and zero or more attribute labels.
M obj achieves 54.9% top-1 accuracy and 86.1% top-5 accuracy.
We report mean average precision (mAP) for attribute classiﬁcation which is a multi-label classiﬁcation problem.
M att achieves 0.14 mAP and 0.49 weighted mAP.
mAP is deﬁned as the mean over all classes, and weighted mAP is weighted by the number of instances for each class.
As there are a lot of redundant classes ( e.g . car, cars, vehicle) and boxes have sparse attribute annotations, the accuracy may seem artiﬁcially low.
Image Captioning.
We report results on MS-COCO for image captioning.
We use the standard split from the 2014 captioning challenge to avoid data contamination with VQA 2.0 or VG.
This split contains 30% less data compared to that proposed in Karpathy & Fei-Fei (2015) that most current works adopt.
We report performance using the CIDEr (Vedantam et al., 2015) metric.
A baseline (non-compositional module) achieves a strong CIDEr score of 108.
Using the object and attribute modules we are able to obtain 109 CIDEr.
While this is not a large improvement, we suspect a reason for this is the limited vocabulary.
The MS-COCO dataset has a ﬁxed set of 80 object categories and does not beneﬁt by using knowledge from modules that are trained on more diverse data.
We believe the beneﬁts of PMN would be clearer on a diverse captioning dataset with many more object classes.
Also, including high-level modules such as M vqa would be an interesting direction for future work.
[Table] [Table]
Relationship Detection.
We use top 20 commonly occurring relationship categories, which are deﬁned by a set of words with similar meaning ( e.g . in, inside, standing in).
Relationship tuples in the form of “subject - relationship - object” are extracted from Visual Genome (Krishna et al., 2016; Lu et al., 2016).
We train and validate the relationship detection module using 200K/38K train/val tuples that have both subject and object boxes overlapping with the ground truth boxes (IoU > 0.7).
Table 1 shows improvement in performance when using modules.
Even though accuracy is relatively low, model errors are reasonable, qualitatively.
This is partially attributed to multiple correct answers although there is only one ground truth answer.
Object Counting.
We extract questions starting with ‘how many’ from VQA 2.0 which results in a training set of ∼ 50K questions.
We additionally create ∼ 89K synthetic questions based on the VG dataset by counting the object boxes and forming ‘how many’ questions.
This synthetic data helps to increase the accuracy by ∼ 1% for all module variants.
Since the number of questions that have relational reasoning and counting ( e.g .
how many people are sitting on the sofa?
how many plates on table?) is limited, we also sample relational synthetic questions from VG.
These questions are used only to improve the parameters of query transmitter Q cnt → rel for the relationship module.
Table 2 shows a large improvement (4.6%) of the compositional module over the non-modular baseline.
When training for the next task (VQA), unlike other modules whose parameters are ﬁxed, we ﬁne-tune the counting module because counting module expects the same form of input - embedding of natural language question.
The performance of counting module depends crucially on the quality of attention map over bounding boxes.
By employing more questions
[Table] [Table]
[Table]
from the whole VQA dataset, we obtain a better attention map, and the performance of counting module increases from 50.0% ( c.f . Table 2) to 55.8% with ﬁnetuning (see Appx B.2 for more details).
Visual Question Answering.
We present ablation studies on the val set of VQA 2.0 in Table 3.
As seen, PMN strongly beneﬁts from utilizing different modules achieving a performance improvement of 2.6% over the baseline.
Note that all results here are without additional questions from the VG data.
We also compare performance of PMN for the VQA task with state-of-the-art models in Table 4.
Models are trained on the train split for results on VQA val, while for test-dev and test-std, models are trained on both the train and val splits.
Although we start with a much lower baseline performance of 62.05% on the val set (vs. 65.42% (Zhang et al., 2018), 63.15% (Teney et al., 2018), 66.04% (Kim et al., 2018)), PMN’s performance is on par with these models.
Note that entries with * are parallel works to ours.
Also, as Jiang et al.
(2018) showed, the performance depends strongly on engineering choices such as learning rate scheduling and ensembling models with different architectures. Plug-and-play architecture.
The query-answer communication within PMN results in a plugand-play architecture where modules can be replaced by their improved versions.
Instead of using generated captions from M cap , when we feed in the ground-truth captions to trained M vqa 5 in Table(with 64.68% acc.), it achieves 65.43%.
We also tried training and validating M vqa 5 with ground-truth captions, and this achieved 67.84%.
These results shows how PMN can be continually improved.
Three additional experiments on VQA.
(1) To verify that the gain is not from the increased model capacity, we trained a baseline with the number of parameters approximately matching that of the full PMN model.
This baseline with more capacity also achieves 62.0%, thus conﬁrming our claim.
We evaluated the impact of the additional data available.
We convert the subj-obj-rel triplets used for the relationship detection task to additional QAs (e.g. Q: what is on top of the desk?
, A: laptop) and train the M vqa 1 model (Table 3).
This results in an accuracy of 63.05%, not only lower than M vqa(63.64%) that uses the relationship module via PMN, but also lower than M vqa 1 at 63.38%.
This suggests that while additional data may change the question distribution and reduce performance, PMN is robust and beneﬁts from a separate relationship module.
(3) Lastly, we conducted another experiment to show that PMN does make efﬁcient use of the lower level modules.
We give equal importance scores to all modules in M vqa 5 model (Table 3) (thus, ﬁxed computation path), achieving 63.65% accuracy.
While this is higher than the baseline at 62.05%, it is lower than M vqa 5 at 64.68% which softly chooses which sub-modules to use.
4. I NTERPRETABILITY A NALYSIS
Visualizing the model’s reasoning process.
We present a qualitative analysis of the answering process.
In Fig.
2, M vqa makes query q rel =
[ b i , r ] to M rel where b i corresponds to the blue box ‘bird’ and r corresponds to ‘on top of’ relationship.
M vqa correctly chooses ( i.e . higher importance score) to use M rel rather than its own output produced by Ω vqa since the question requires relational reasoning.
With the attended green box obtained from M rel , M vqa mostly uses the object and captioning modules to produce the ﬁnal answer.
More examples are presented in Appendix C.
[Table]
Judging Answering Quality.
The modular structure and gating mechanism of PMN makes it easy to interpret the reasoning behind the outputs.
We conduct a human evaluation with Amazon Mechanical Turk on 1,600 randomly chosen questions.
Each worker is asked to rate the explanation generated by the baseline model and the PMN like a teacher grades student exams in the scale of 0 (very bad), 1 (bad), 2 (satisfactory), 3 (good), 4 (very good).
The baseline explanation is composed of the bounding box it attends to and the ﬁnal answer.
For PMN, we form a rule-based natural language explanation based on the prominent modules used.
An example is shown in Fig.
Each question is assessed by three human workers.
Incorrect reasoning steps are penalized, so if PMN produces wrong reasoning steps, it could get a low score.
On the other hand, the baseline model often scores well on simple questions that do not need complex reasoning ( e.g .
what color is the cat?).
[Table]
[Figure]
We report results in Table 5, and show more examples in
Appendix D. Human evaluators tend to give low scores to wrong answers and high scores to correct answers regardless of explanations, but PMN always scores higher if both PMN and baseline gets a question correct or wrong.
Interestingly, a correct answer from PMN gets 1.38 points higher than wrong baseline, but a correct baseline scores only 0.74 higher than a wrong PMN answer.
This shows that PMN gets partial marks even when it gets an answer wrong since the reasoning steps are partially correct.
Low Data Regime.
PMN beneﬁts from re-using modules and only needs to learn the communication between them.
This allows us to achieve good performance even when using a fraction of the training data.
Table 6 presents the absolute gain in accuracy
PMN achieves.
For this experiment, we use L vqa =
[Ω vqa , M rel , M obj , M att , ∆ vqa , M cap ] (because of overlapping questions from M cnt ).
When the amount of data is really small (1%), PMN does not help because there is not enough data to learn to communicate with lower modules.
The maximum gain is obtained when using 10% of data.
It shows that PMN can help in situations where there is not a huge amount of training data since it can exploit previously learned knowledge.
The gain remains constant at about 2% from then on.
[Table]
[Figure]
[Table]
C ONCLUSION AND D ISCUSSION
In this work, we proposed Progressive Module Networks (PMN) that train task modules in a compositional manner, by exploiting previously learned lower-level task modules.
PMN can produce queries to call other modules and make use of the returned information to solve the current task.
Given experts in speciﬁc tasks, the parent module only needs to learn how to effectively communicate with them.
It can also choose which lower level modules to use.
Thus, PMN is data efﬁcient and provides a more interpretable reasoning processes.
Also, since there is no need to know about the inner workings of the children modules, it opens up promising ways to train intelligent robots that can compartmentalize and perform multiple tasks as they progressively improve their abilities.
Moreover, one task can beneﬁt from unrelated tasks unlike conventional multi-task learning algorithms.
PMN as it stands has few limitations with respect to hand-designed structures and the need for additional supervision.
Nevertheless, PMN is an important step towards more interpretable, compositional multi-task models.
Some of the questions to be solved in the future include: 1) learning module lists automatically; 2) choosing few modules (hard attention) to reduce overhead; 3) more generic structure of module components across tasks; and 4) joint training of all modules.
A PPENDICES A M ODULE A RCHITECTURES
We discuss the detailed architecure of each module.
We ﬁrst describe the shared environment and soft attention mechanism architecture.
Environment.
The sensory input that form our environment E consists of: (i) image regions : N image regions X
=
[ X 1 , . .
, X N ] , each X i ∈ R
d with corresponding bounding box coordinates b =
[ b 1 , . . .
, b N ] extracted from Faster R-CNN (Ren et al., 2015); and (ii) language : vector representation of a sentence S (in our example, a question).
S is computed through a one layer GRU by feeding in the embedding of each word [ w 1 , . . .
, w T ] at each time step.
For (i), we use a pretrained model from Anderson et al.
(2018) to extract features and bounding boxes.
Soft attention.
For all parts that use soft-attention mechanism, an MLP is emloyed.
Given some key vector k and a set of data to be attended { d 1 , . . .
, d N } , we compute
[Equation]
where f and g are a sequence of linear layer followed by ReLU activation function that project k and d i into the same dimension, and z is a linear layer that projects the joint representation into a single number.
Note that we do not specify softmax function here because sigmoid is used for some cases.
A. O BJECT AND A TTRIBUTE C LASSIFICATION (L EVEL 0)
The input to both modules M obj , M att is a visual descriptor for a bounding box b i in the image, i.e ., q obj =
X i .
M obj and M att projects the visual feature
X
i to a 300-dimensional vector through a single layer neural network followed by tanh() non-linearity.
We expect this vector to represent the name and attributes of the box b i .
A.
I MAGE C APTIONING (L
EVEL 1)
M cap takes zero vector as the model input and produces natural language sentence as the output based on the environment E (detected image regions in an image).
It has L cap =
[Ω cap , M obj , M att , ∆ cap ] and goes through maximum of T cap = 16 time steps or until it reaches the end of sentence token.
M cap is implemented similarly to the captioning model in Anderson et al.
We employ two layered GRU (Cho et al., 2014) as the recurrent state update function U cap where s t =
( h t 1 , h t 2 ) with hidden states of the ﬁrst and second layers of U cap .
Each layer has 1000-d hidden states.
The state initializer I cap sets the initial hidden state of U cap , or the model state s t , as a zero vector.
For t in T cap = 16 , M cap does the following four operations:
(1) The importance function G cap is executed.
It is implemented as a linear layer R 1000 → R 4 (for the four modules in L cap ) that takes s t , speciﬁcally h t 1 ∈ s t as input.
(2) Q cap → Ω passes h t 1 to the attention module Ω cap which attends over the image regions X with h tas the key vector.
Ω cap is implemented as a soft-attention mechanism so that it produces attention probabilities p
i (via softmax) for each image feature X i ∈ E .
The returned attention map v Ω is added to the scratch pad V .
(3) Q cap → obj and Q cap → att pass the sum of visual features X weighted by v Ω ∈ V to the corresponding modules.
∆ cap is implemented as an MLP.
The receivers project the outputs intodimensional vectors v obj , v att , and v ∆ through a sequence of linear layers, batch norm
and tanh() nonlinearities.
They are added to V .
(4) As stated above, U cap is a two-layered GRU.
At time t , the ﬁrst layer takes input the average visual features from the environment E , 1 N (cid:80)
i X i , embedding vector of previous word w t − 1 , and h t 2 .
For time t =
, beginning-of-sentence embedding and zero vector are inputs for w 1 and h 11 , respectively.
The second layer is fed h t 1 as well as the information from other modules,
[Equation]
which is a gated summation of outputs in V with softmaxed importance scores.
We now have a new state s t +
=
( h t +11 , h t +12 ) .
The output of M cap , o cap , is a sequence of words produced through Ψ cap which is a linear layer projecting each h t 2 in s t to the output word vocabulary.
A. R ELATIONSHIP D ETECTION (L EVEL 1)
Relationship detection task requires one to produce triplets in the form of “subject - relationship - object” (Lu et al., 2016).
We re-purpose this task as one that involves ﬁnding the relevant item (region) in an image that is related to a given input through a given relationship.
The input to the module is q rel =
[ b i , r ] where b i is a one-hot encoded input bounding box (whose i -th entry is 1 and others 0) and r is a one-hot encoded relationship category ( e.g . above, behind).
M rel has
L rel =
[ M obj , M att , ∆ rel ] and goes through T rel =
N steps where N is the number of bounding boxes (image regions in the environment).
So at time step t , the module looks at the t -th box.
M rel uses M obj and M att just as feature extractors for each bounding box.
Therefore, it does not have a complex structure.
The state initializer I rel projects r to a 512 dimensional vector with an embedding layer, and the resulting vector is set as the ﬁrst state s 1 .
For t in T rel =
N , M rel does the following three operations:
(1) Q rel → obj and Q rel → att pass the image vector corresponding to the bounding box b t to M obj and M att .
R obj
→ rel and R att → rel are identity functions,
i
e ., we do not modify the object and attribute vectors.
The outputs v obj and v att are added to V .
(2) ∆ rel projects the coordinate of the current box b t to a 512 dimensional vector.
This resulting v ∆ is added to V .
(3) U rel concatenates the visual feature X t with v obj , v att , v ∆ from V .
The concatenated vector is fed through a MLP
resulting in 512 dimensional vector.
This corresponds to the new state s t +1 .
After N steps, the prediction function Ψ rel does the following operations:
The ﬁrst state s 1 which contains relationship information is multiplied element-wise with s i +
(Note: s i +1 corresponds to the input box b i ).
Let such a vector be l .
It produces an attention map b out over all bounding boxes in b .
The inputs to the attention function are s 2 , . . .
, s T rel (i
e. all image regions) and the key vector l .
o rel =
b out is the output of M rel which represents an attention map indicating the bounding box that contains the related object.
A. C OUNTING (L EVEL 2)
Given a vector representation of a natural language question ( e.g . how many cats are in this image?), the goal of this module is to produce a count.
The input q cnt =
S ∈ E is a vector representing a natural language question.
When training M cnt , q cnt is computed through a one layer GRU with hidden size of 512 dimensions.
The input to the GRU at each time step is the embedding of each word from the question.
Word embeddings are initialized with 300 dimensional GloVe word vectors (Pennington et al., 2014) and ﬁne-tuned thereafter.
Similar to visual features obtained through CNN, the question vector is treated as an environment variable.
M cnt has L cnt =
[Ω cnt , M rel ] and goes through only one time step.
The state initializer I cnt is a simple function that just sets s 1 = q cnt .
For t in T cnt = 1 , M cnt does the following four operations:
(1) The importance function G cnt is executed.
It is implemented as a linear layer R
→ R 2 (for the two modules in L cnt ) that takes s t as input.
(2) Q cnt → Ω passes s t to the attention module Ω cnt which attends over the image regions X with s t as the key vector.
Ω cnt is implemented as an MLP that computes a dot-product soft-attention similar to Yang et al.
The returned attention map v Ω is added to the scratch pad V .
(3) Q cnt → rel produces an input tuple [ b, r ] for M rel .
The input object box b is produced by a MLP that does soft attention on image boxes, and the relationship category r is produced through a MLP with s t as input.
M rel is called with [ b, r ] and the returned map v rel is added to V .
(4) U cnt ﬁrst computes probabilities of using v Ω or v rel by doing a softmax over the importance scores
v Ω and v rel are weighted and summed with the softmax probabilities resulting in the new state s 2 containing the attention map.
Thus, the state update function chooses the map from M rel if the given question involves in relational reasoning.
The prediction function Ψ cnt returns a count vector.
The count vector is computed through the counting algorithm by Zhang et al.
(2018), which builds a graph representation from attention maps to count objects.
The method uses s 2 through a sigmoid and bounding box coordinates b as inputs.
Zhang et al.
(2018) is a fully differentiable algorithm and the resulting count vector corresponds to one-hot encoding of a number.
We let the range of count be 0 to 12 ∈ Z .
Please refer to Zhang et al.
(2018) for details of the counting algorithm.
A. V ISUAL Q UESTION A NSWERING (L EVEL 3)
The description for the VQA task (Sec. 3.4) is included here again for completeness.
The input q vqa is a vector representing a natural language question ( i.e . the sentence vector S ∈ E ).
The state variable s t is represented by a tuple ( q t vqa , k t − 1 ) where q t vqa represents query to ask at time t and k t − 1 represents knowledge gathered at time t − 1 .
The state initializer I vqa is composed of a GRU with hidden state dimension 512.
The ﬁrst input to GRU is q vqa , and I vqa sets s 1 = ( q 1vqa , 0 ) where q 1vqa is the ﬁrst hidden state of the GRU and 0 is a zero vector (no knowledge at ﬁrst).
For t in T vqa = 2 , M vqa does the following seven operations:
(1) The importance function G vqa is executed.
It is implemented as a linear layer R
→ R 7 (for the seven modules in L vqa ) that takes s t , speciﬁcally q t vqa ∈ s t as input.
(2) Q vqa → Ω passes q t vqa to the attention module Ω vqa which attends over the image regions X with q t vqa as the key vector.
Ω vqa is implemented as an MLP that computes a dot-product soft-attention similar to Yang et al.
The returned attention map v Ω is added to the scratch pad V .
(3) Q vqa → rel produces an input tuple [ b, r ] for M rel .
The input object box b is produced by a MLP that does soft attention on image boxes, and the relationship category r is produced through a MLP with q t vqa as input.
M rel is called with [ b, r ] and the returned map v rel is added to V .
(4) Q vqa → obj , Q vqa → att , and Q vqa → ∆ ﬁrst compute a joint attention map m as summation of ( v Ω , v rel ) weighted by the softmaxed importance scores of (Ω vqa , M rel ) , and they pass the sum of visual features X weighted by m to the corresponding modules.
∆ vqa is implemented as an MLP.
The receivers project the outputs into 512 dimensional vectors v obj , v att , and v ∆ through a sequence of linear layers, batch norm, and tanh() nonlinearities.
They are added to V .
(5) Q vqa → cnt passes q t vqa to M cnt which returns o cnt .
R cnt → vqa projects the count vector o cnt into a 512 dimensional vector v cnt through the same sequence of layers as above.
v cnt is added to V .
(6) M vqa calls M cap and R cap → vqa receives natural language caption of the image.
It converts words in the caption into vectors [ w 1 , . . .
, w T ] through an embedding layer.
The embedding layer is initialized with 300 dimensional GloVe vectors (Pennington et al., 2014) and ﬁne-tuned.
It does softmax attention operation over [ w 1 , . . .
, w T ] through a MLP with q t vqa ∈ s t as the key vector, resulting in word probabilities p 1 , . . .
, p T .
The sentence representation (cid:80)
Ti p i · w i is projected into a 512 dimensional vector v cap using the same sequence as v cnt .
v cap is added to V .
The state update function U vqa ﬁrst does softmax operation over the importance scores of ( M obj , M att , ∆ vqa , M cnt , M cap ) .
We deﬁne an intermediate knowledge vector k t as the summation of ( v obj , v att , δ vqa , v cnt , v cap ) weighted by the softmaxed importance scores.
U vqa passes k t as input to the GRU initialized by I vqa , and we get q t
+1vqa the new hidden state of the GRU.
The new state s t +1 is set to ( q t +1vqa , k t ) .
This process allows the GRU to compute new question and state vectors based on what has been asked and seen
After T vqa steps, the prediction function Ψ vqa computes the ﬁnal output based on the initial question vector q vqa and all knowledge vectors k t ∈ s t .
Here, q vqa and k t are fused with gated-tanh layers and fed through a ﬁnal classiﬁcation layer similar to Anderson et al.
(2018), and the logits for all time steps are added.
The resulting logit is the ﬁnal output o vqa that corresponds to an answer in the vocabulary of the VQA task.
B A DDITIONAL E XPERIMENTAL D ETAILS
In this section, we provide more details about datasets and module training.
B. D ATASETS
We extract bounding boxes and their visual representations using a pretrained model from Anderson et al.
(2018)which is a Faster-RCNN (Ren et al., 2015) based on ResNet-101 (He et al., 2016).
It produces 10 to 100 boxes with 2048-d feature vectors for each region.
To accelerate training, we remove overlapping bounding boxes that are most likely duplicates (area overlap IoU > 0.7) and keep only the 36 most conﬁdent boxes (when available).
MS-COCO contains ∼ 100K images with annotated bounding boxes and captions.
It is a widely used dataset used for benchmarking several vision tasks such as captioning and object detection.
Visual Genome is collected to relate image concepts to image regions.
It has over 108K images with annotated bounding boxes containing 1.7M visual question answering pairs, 3.8M object instances, 2.8M attributes and 1.5M relationships between two boxes.
Since the dataset contains MS-COCO images, we ensure that we do not train on any MS-COCO validation or test images.
VQA 2.0 is the most popular visual question-answering dataset, with 1M questions on 200K natural images.
Questions in this dataset require reasoning about objects, actions, attributes, spatial relations, counting, and other inferred properties; making it an ideal dataset for our visual-reasoning PMN.
B. T RAINING
Here, we give training details of each module.
We train our modules sequentially, from low level to high level tasks, one at a time.
When training a higher level module, internal weights of the lower level modules are not updated, thus preserving their performance on the original task.
We do train the weights of the residual module ∆ and the attention module Ω .
We train I , G , Q , R , U , and Ψ , by allowing gradients to pass through the lower level modules.
Thus, while the existing lower modules are held ﬁxed, the new module learns to communicate with them via the query transmitter Q and receiver R .
Object and attribute classiﬁcation.
M obj is trained to minimize the cross-entropy loss for predicting object class by including an additional linear layer on top of the module output.
M att also include an additional linear layer on top of the module output, and is trained to minimize the binary cross-entropy loss for predicting attribute classes since one detected image region can contain zero or more attribute classes.
We make use of 780K/195K train/val object instances paired with attributes from the Visual
Genome dataset.
They are trained with the Adam optimizer at learning rate of 0.0005 with batch size 32 for 20 epochs.
Image captioning.
M cap is trained using cross-entropy loss at each time step (maximum likelihood).
Parameters are updated using the Adam optimizer at learning rate of 0.0005 with batch size 64 forepochs.
We use the standard split of MS-COCO captioning dataset.
Relationship detection.
M rel is trained using cross-entropy loss on “subject - relationship - object” pairs with Adam optimizer with learning rate of 0.0005 with batch size 128 for 20 epochs.
The pairs are extracted from the Visual Genome dataset that have both subject and object boxes overlapping with the ground truth boxes (IoU > 0.7), resulting in 200K/38K train/val tuples.
Counting.
M cnt is trained using cross-entropy loss on questions starting with ‘how many’ from the VQA 2.0 dataset.
We use Adam optimizer with learning rate of 0.0001 with batch size 128 for 20 epochs.
As stated in the experiments section, we additionally create ∼ 89K synthetic questions to increase our training set by counting the object boxes and forming ‘how many’ questions from the VG dataset ( e.g .
(Q: how many dogs are in this picture?, A:3) from an image containing three bounding boxes of dog).
We also sample relational synthetic questions from each training image from
VG that are used to train only the module communication parameters when the relationship module is included.
We use the same 200K/38K split from the relationship detection task by concatenating ‘how many’+subject+relationship’ or ‘how many’+relationship+object ( e.g .
how many plates on table which compute a relationship category and the input image region to be passed to M rel .
To be clear, we supervise q rel =
[ b i , r ] to be sent to M rel by reducing cross entropy loss on b i and r .
Visual Question Answering.
M vqa is trained using binary cross-entropy loss on o vqa with Adam optimizer with learning rate of 0.0005 with batch size 128 for 7 epochs.
We empirically found binary cross-entropy loss to work better than cross-entropy which was also reported by Anderson et al.
Unlike other modules whose parameters are ﬁxed, we ﬁne-tune only the counting module because counting module expects the same form of input - embedding of natural language question.
The performance of counting module depends crucially on the quality of attention map over bounding boxes.
By employing more questions from the whole VQA dataset, we obtain a better attention map, and the performance of counting module increases from 50.0% to 55.8% with ﬁnetuning.
Since
M vqa and M cnt expect the same form of input, the weights of attention modules Ω { vqa , cnt } and query transmitters for the relationship module Q { vqa , cnt }→ rel are shared.
C PMN E XECUTION I LLUSTRATED
We provide more examples of the execution traces of PMN on the visual question answering task in Figure 4.
Each row in the ﬁgure corresponds to different examples.
For each row in the ﬁgure, the left column shows the environment E , the middle column shows the ﬁnal answer & visualizes step 3 in the execution process, and the right column shows computed importance scores along with populated scratch pad.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
D E XAMPLES OF PMN’ S R EASONING
We provide more examples of the human evaluation experiment on interpretability of PMN compared with the baseline model in Figure
[Table]
[Equation]
[Table]
[Figure]
 