Paper:
Achieving machine intelligence requires a smooth integration of perception and reasoning, yet models developed to date tend to specialize in one or the other; sophisticated manipulation of symbols acquired from rich perceptual spaces has so far proved elusive. Consider a visual arithmetic task, where the goal is to carry out simple arithmetical algorithms on digits presented under natural conditions (e.g. hand-written, placed randomly). We propose a two-tiered architecture for tackling this problem. The lower tier consists of a heterogeneous collection of information processing modules, which can include pre-trained deep neural networks for locating and extracting characters from the image, as well as modules performing symbolic transformations on the representations extracted by perception. The higher tier consists of a controller, trained using reinforcement learning, which coordinates the modules in order to solve the high-level task. For instance, the controller may learn in what contexts to execute the perceptual networks and what symbolic transformations to apply to their outputs. The resulting model is able to solve a variety of tasks in the visual arithmetic domain, and has several advantages over standard, architecturally homogeneous feedforward networks including improved sample efficiency.

Review:
Summary: This work is a variant of previous work (Zaremba et al. 2016) that enables the use of (noisy) operators that invoke pre-trained neural networks and is trained with Actor-Critic. In this regard it lacks a bit of originality. The quality of the experimental evaluation is not great. The clarity of the paper could be improved upon but is otherwise fine. The existence of previous work (Zaremba et al. 2016) renders this work (including its contributions) not very significant. Relations to prior work are missing. But let's wait for the rebuttal phase. 
Pros 
-It is confirmed that noisy operators (in the form of neural networks) can be used on the visual arithmetic task
Cons
-Not very novel
-Experimental evaluation is wanting
The focus of this paper is on integrating perception and reasoning in a single system. This is done by specifying an interface that consists of a set of discrete operations (some of which involve perception) and memory slots. A parameterized policy that can make use of these these operations is trained via Actor-Critic to solve some reasoning tasks (arithmetics in this case). 
The proposed system is a variant of previous work (Zaremba et al. 2016) on the concept of interfaces, and similarly learns a policy that utilizes such an interface to perform reasoning tasks, such as arithmetics. In fact, the only innovation proposed in this paper is to incorporate some actions that invoke a pre-trained neural network to “read” the symbol from an image, as opposed to parsing the symbol directly. However, there is no reason to expect that this would not function in previous work (Zaremba et al. 2016), even when the network is suboptimal (in which case the operator becomes noisy and the policy should adapt accordingly). Another notable difference is that the proposed system is trained with Actor-Critic as opposed to Q-learning, but this is not further elaborated on by the authors. 
The proposed system is evaluated on a visual arithmetics task. The input consists of a 2x2 grid of extended MNIST characters. Each location in the grid then corresponds to the 28 x 28 pixel representation of the digit. Actions include shifting the “fovea” to a different entry of the grid, invoking the digit NN or the operator NN which parse the current grid entry, and some symbolic operations that operate on the memory. The fact that the input is divided into a 2x2 grid severely limits the novelty of this approach compared to previous work (Zaremba et al. 2016). Instead it would have been interesting to randomly spawn digits and operators in a 56 x 56 image and maintain 4 coordinates that specify a variable-sized grid that glimpses a part of the image. This would make the task severely more difficult, given fixed pre-trained networks. The addition of the salience network is unclear to me in the context of MNIST digits, since any pixel that is greater than 0 is salient? I presume that the LSTM uses this operator to evaluate whether the current entry contains a digit or an operator. If so, wouldn’t simply returning the glimpse be enough?
In the experiments the proposed system is compared to three CNNs on two different visual arithmetic tasks, one that includes operators as part of the input and one that incorporates operators only in the tasks description. In all cases the proposed method requires fewer samples to achieve the final performance, although given enough samples all of the CNNs will solve the tasks. This is not surprising as this comparison is rather unfair. The proposed system incorporates pre-trained modules, whose training samples are not taken into account. On the other hand the CNNs are trained from scratch and do not start with the capability to recognize digits or operators. Combined with the observation that all CNNs are able to solve the task eventually, there is little insight in the method's performance that can be gained from this comparison. 
Although the visual arithmetics on a 2x2 grid is a toy task it would at least be nice to evaluate some of the policies that are learned by the LSTM (as done by Zaremba) to see if some intuition can be recovered from there. Proper evaluation on a more complex environment (or at least on that does not assume discrete grids) is much desired. When increasing the complexity (even if by just increasing the grid size) it would be good to compare to a recurrent method (Pyramid-LSTM, Pixel-RNN) as opposed to a standard CNN as it lacks memory capabilities and is clearly at a disadvantage compared to the LSTM.
Some detailed comments are:
The introduction invokes evidence from neuroscience to argue that the brain is composed of (discrete) modules, without reviewing any of the counter evidence (there may be a lot, given how bold this claim is).
From the introduction it is unclear why the visual arithmetic task is important.
Several statements including the first sentence lack citations.
The contribution section is not giving any credit to Zaremba et al. (2016) whereas this work is at best a variant of that approach.
In the experiment section the role of the saliency detector is unclear.
Experiment details are lacking and should be included.
The related work section could be more focused on the actual contribution being made.
It strikes me as odd that in the discussion the authors propose to make the entire system differentiable, since this goes against the motivation for this work.
Relation to prior work:
p 1: The authors write: "We also borrow the notion of an interface as proposed in Zaremba et al. (2016). An interface is a designed, task-specific machine that mediates the learning agent’s interaction with the external world, providing the agent with a representation (observation and action spaces) which is intended to be more conducive to learning than the raw representations. In this work we formalize an interface as a separate POMDP I with its own state, observation and action spaces." 
This interface terminology for POMDPs was actually introduced in:
J.  Schmidhuber. Reinforcement learning in Markovian and non-Markovian environments. In D. S. Lippman, J. E. Moody, and D. S. Touretzky, editors, Advances in Neural Information Processing Systems 3, NIPS'3, pages 500-506. San Mateo, CA: Morgan Kaufmann, 1991.
p 4: authors write: "For the policy πθ, we employ a Long Short-Term Memory (LSTM)" 
Do the authors use the (cited) original LSTM of 1997, or do they also use the forget gates (recurrent units with gates) that most people are using now, often called the vanilla LSTM, by Gers et al (2000)?
p 4: authors write: "One obvious point of comparison to the current work is recent research on deep neural networks designed to learn to carry out algorithms on sequences of discrete symbols. Some of these frameworks, including the Differen-tiable Forth Interpreter (Riedel and Rocktäschel, 2016) and TerpreT (Gaunt et al., 2016b), achieve this by explicitly generating code, while others, including the Neural Turing Machine (NTM; Graves et al., 2014), Neural Random-Access Machine (NRAM; Kurach et al., 2015), Neural Programmer (NP; Neelakan- tan et al., 2015), Neural Programmer-Interpreter (NPI; Reed and De Freitas, 2015) and work in Zaremba et al. (2016) on learning algorithms using reinforcement learning, avoid gen- erating code and generally consist of a controller network that learns to perform actions in a (sometimes differentiable) external computational medium in order to carry out an algorithm."
Here the original work should be mentioned, on differentiable neural stack machines: 
G.Z. Sun and H.H. Chen and  C.L. Giles and Y.C. Lee and D. Chen. Connectionist Pushdown Automata that Learn Context-Free Grammars. IJCNN-90, Lawrence Erlbaum, Hillsdale, N.J., p 577, 1990.
Mozer, Michael C and Das, Sreerupa. A connectionist symbol manipulator that discovers the structure of context-free languages. Advances in Neural Information Processing Systems (NIPS), p 863-863, 1993.