We describe a question answering model that applies to both images and structured knowledge bases . The model uses natural language strings to automatically assemble neural networks from a collection of composable modules . Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning , with only ( world , question , answer ) triples as supervision . Our approach , which we term a dynamic neural module network , achieves state-of-theart results on benchmark datasets in both visual and structured domains . This paper presents a compositional , attentional model for answering questions about a variety of world representations , including images and structured knowledge bases . The model translates from questions to dynamically assembled neural networks , then applies these networks to world representations ( images or knowledge bases ) to produce answers . We take advantage of two largely independent lines of work : on one hand , an extensive literature on answering questions by mapping from strings to logical representations of meaning ; on the other , a series of recent successes in deep neural models for image recognition and captioning . By constructing neural networks instead of logical forms , our model leverages the best aspects of both linguistic compositionality and continuous representations . Our model has two components , trained jointly : first , a collection of neural " modules " that can be freely composed ( Figure 1b ) ; second , a network layout predictor that assembles modules into complete deep networks tailored to each question ( Figure 1a ) . Previous work has used manually-specified modular structures for visual learning ( Andreas et al . , 2016 ) . Here we : • learn a network structure predictor jointly with module parameters themselves • extend visual primitives from previous work to reason over structured world representations Training data consists of ( world , question , answer ) triples : our approach requires no supervision of network layouts . We achieve state-of-the-art performance on two markedly different question answering tasks : one with questions about natural images , and another with more compositional questions about United States geography.1 We have introduced a new model , the dynamic neural module network , for answering queries about both structured and unstructured sources of information . Given only ( question , world , answer ) triples as training data , the model learns to assemble neural networks on the fly from an inventory of neural models , and simultaneously learns weights for these modules so that they can be composed into novel structures . Our approach achieves state-of-the-art results on two tasks . We believe that the success of this work derives from two factors : Continuous representations improve the expressiveness and learnability of semantic parsers : by replacing discrete predicates with differentiable neural network fragments , we bypass the challenging combinatorial optimization problem associated with induction of a semantic lexicon . In structured world representations , neural predicate representations allow the model to invent reusable attributes and relations not expressed in the schema . Perhaps more importantly , we can extend compositional questionanswering machinery to complex , continuous world representations like images . Semantic structure prediction improves generalization in deep networks : by replacing a fixed network topology with a dynamic one , we can tailor the computation performed to each problem instance , using deeper networks for more complex questions and representing combinatorially many queries with comparatively few parameters . In practice , this results in considerable gains in speed and sample efficiency , even with very little training data . These observations are not limited to the question answering domain , and we expect that they can be applied similarly to tasks like instruction following , game playing , and language generation .

Challenge: Existing works on visual learning use manually-specified modular structures.
Approach: They propose a question-answering model trained jointly to translate from questions to dynamically assembled neural networks then produce answers with using images or knowledge bases.
Outcome: The proposed model achieves state-of-the-arts on visual and structured domain datasets showing that coutinous representations improve the expressiveness and learnability of semantic parsers.