Paper:
VISUAL REASONING BY PROGRESSIVE MODULE NETWORKS
Humans learn to solve tasks of increasing complexity by building on top of previously acquired knowledge. Typically, there exists a natural progression in the tasks that we learn -most do not require completely independent solutions, but can be broken down into simpler subtasks. We propose to represent a solver for each task as a neural module that calls existing modules (solvers for simpler tasks) in a functional program-like manner. Lower modules are a black box to the calling module, and communicate only via a query and an output. Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output. Our model effectively combines previous skill-sets, does not suffer from forgetting, and is fully differentiable. We test our model in learning a set of visual reasoning tasks, and demonstrate improved performances in all tasks by learning progressively. By evaluating the reasoning process using human judges, we show that our model is more interpretable than an attention-based baseline.

Review:
[Summary]
This paper presents a multi-task learning approach for VQA that represent a solver for each task as a neural module that calls existing modules in a program manner. The authors manually design the task hierarchy and propose a progressive module network to recursive calls the lower modules and gather the information by soft-attention. The final prediction uses all the states and question to infer the final answers. The authors verify the effectiveness of the proposed method on the performance of different tasks and modules. Experiment on VQA shows the proposed model benefits from utilizing different modules. The authors also qualitatively show the model's reasoning process and human study on judging answering quality.
[Strength]
1. The proposed method is novel and explores to use the existing modules as a black box for visual question answering.  This is different from most existing work. 
2: By examing different modules, the proposed method is more interpretable compare to canonical methods. 
3: The experiment results are good, especially for the counting problem. 
[Weakness] 
1. The title of the paper is "visual reasoning by progressive module networks." The title may be a little overstated since the major task is focused on visual question answering (VQA).  
2. Annotation is not clear in this paper. For example, on page 3, Query transmitter and receiver, "the output o_k = M_k(q_k) received from M_k is modified using receiver function as v_k = R_{k->n}(s^t, o_k). " There are multiple new variables in this paragraph, without specifying the dimension and meaning for each attribute, it's really hard to understand. On page 4, State update function, what is the meaning of variable "Epsilon" in the equation? From the supplementary, it seems Epsilon means the environment? 
3. On the object counting task, the query transmitter needs to produce a query for a relationship module. The authors mentioned that this is softly calculated by softmax on the importance score. Since q_rel require one hot vector as input, how to sample the q_rel given the importance score and how backprob the gradient in this case? 
4. The cider score of image captioning is 109 compared to the baseline 108. The explanation is the COCO dataset has a fixed set of 80 object categories and does not benefit from training the diverse data. Since the input visual feature is the same, the only difference is the proposed model has additional label embedding as input. My assumption is the visual feature already contains the label information for image captioning. 
5. On relational detection task, is there a way to compare with the STOA method on some specific data split? This will leads to much more convincing results. 
6. Similar as above question, on the object counting task, is there a way to compare with previous counting methods? 
7. In Table 4, the accuracy of number on Zhang et.al is 49.39, which is higher than other methods, while on test-dev, the accuracy is 51.62, which is lower than others. Is the number right?