Paper:
A general problem that received considerable recent attention is how to perform multiple tasks in the same network, maximizing both efficiency and prediction accuracy. A popular approach consists of a multi-branch architecture on top of a shared backbone, jointly trained on a weighted sum of losses. However, in many cases, the shared representation results in non-optimal performance, mainly due to an interference between conflicting gradients of uncorrelated tasks. Recent approaches address this problem by a channel-wise modulation of the feature-maps along the shared backbone, with task specific vectors, manually or dynamically tuned. Taking this approach a step further, we propose a novel architecture which modulate the recognition network channel-wise, as well as spatial-wise, with an efficient top-down image-dependent computation scheme. Our architecture uses no task-specific branches, nor task specific modules. Instead, it uses a top-down modulation network that is shared between all of the tasks. We show the effectiveness of our scheme by achieving on par or better results than alternative approaches on both correlated and uncorrelated sets of tasks. We also demonstrate our advantages in terms of model size, the addition of novel tasks and interpretability. Code will be released.

Review:
> What is the specific question/problem tackled by the paper?
This paper tackles a restricted multi-task setting where the task is known. The main contribution is a new architecture for training a task conditional model. The new architecture is reminiscent of an encoder-decoder-classifier with ladder (latent) connections, the decoder is conditioned on task ID. The claim is this is a type of modulation, it is unclear. Results on three multi-task datasets show that the proposed method is slightly better than compared methods and single task learning. There is no theory or loss function to analyze.
> Is the approach well motivated, including being well-placed in the literature?
In my opinion, this is lacking. The assumption that task ID is known is fairly severe. Unfortunately the prior works cited also have this restriction, whereas few papers under the topic of continual learning have removed this limitation. This assumption/drawback needs to be clearly mentioned in the paper and discussed if it is realistic? A related shortcoming is that the training data simultaneously comes from all the tasks, whereas prior work has looked at the more interesting setup where tasks arrive sequentially and incrementally. 
- Reference [1] seems relevant and should be cited as it shows context dependent gating of tasks / modulation as well. Other missing references e.g. learning without forgetting (LwF) [2] and [3]. 
- There is not a clear explanation to think that this is modulation since the result is only passed through a residual connection. More importantly there is no discussion on these important issues. I found the writing to be brief and sketched. 
- in the introduction, it would be good to define multi-task learning with the assumption made clear. It would be good to introduce what you mean by TD and BU clearly
- Another drawback is assuming the tasks being encoded as integers, whereas there might be a continuous task space with interpolation, or hierarchical task structure.
- "However, all of these works modulate the recognition network
channel-wise, using the same modulation vector for all the spatial dimension of the feature-maps." - why is this not enough? A nontrivial explanation or discussion is needed. Simply extending to W(t, y, x, ch) would increase performance by a little.
- how is the proposed model different from a conditional model like a task conditional classifier? Also in experiments.
- How is the proposed model different from an encoder-decoder? The impact of "modulation" is not clear.
- "We can scale the number of tasks with no additional layers." - task conditional classifier can also scale in this way to the number of tasks. This claim is not valid.
- Page 3: "uncorrelated gradients from the different tasks" - need not be uncorrelated, but still can be interfering
- next about Kendall (2018) and Sener (2018) - need to compare and contrast to them.
- Last para on page 3 seems not relevant.
- Modulation equations: this seems specific to CNNs. How would you extend this technique to beyond CNNs to recurrent units or even simpler MLPs? Modulation as a technique has been successfully applied in these architectures as well.
- "added to the input tensor X through a residual connection" - this is not clear at all. Are the residual connections not shown in Fig 1(d)? 
- "it to be unfeasible due to their large dimensions" - can you explain please? later you say "To avoid the unfeasible computation burden of directly optimizing W"
- Fig 1d, would be good to mark the modulation arrows in a different color
> Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
Having said that, I like the experimental section even in the restricted setup. But a lot of details are missing. It is not surprising that there is slight increase in performance over channel modulation due to the increase expressivity. 
- table 1: why is there degradation in the LL task across all methods? the introduction of an additional task seems to bring the performance back up. It seems to be a weakness of your method. Please improve the discussion. I'm inclined to think that the tasks are not uncorrelated, as claimed by the authors. 
- table 1: how did you arrive at the number of parameters like 1.12x? Doesn't the separate BU and TD nets mean you have at least 2x parameters compared to single task? It seems the larger number of params in single task is mainly coming from the hidden layers?
- table 1: it would be fair for the comparison methods to have equal number of parameters as the proposed method.
- Missing experimental comparison to Kendall et. al. 2018
- Missing details about reproduction of results from Sener (2018)
- An important baseline would be to show image sensitive full tensor modulation without the new architecture. Similar to XdG.
- Another baseline should be a task-conditional classifier that takes task as input along with the image. 
- ablation study: what are the auxiliary losses? I could not find any details.
- The third experiment with CUB seems to use a different loss function that the other methods. This is somewhat hard to evaluate.
- number of parameters are not reported for the CUB experiment
- "where only a single pixel is labeled as foreground, blurred by a Gaussian kernel" needs more details about the smoothing
- In the CUB experiment, due to lateral connections, the top-down result 224x224 image is not the only input to the BU2 classifier, the interpretability argument is weak.
- Why did you choose these 4 questions from CLEVR? There are many interesting types of questions that can be handled.