Paper:
Saliency-based Sequential Image Attention with Multiset Prediction
Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

Review:
This submission proposes a recurrent attention architecture (composed of multiple steps) that operates on a saliency map that is computed with a feed forward network. The proposed architecture is based on the biological model of covert attention and overt attention. This is translated into a meta-controller (covert attention) that decides which regions need overt attention (which are then treated by lower-level controllers that glimpse in the chosen regions). This is implemented as a form of hierarchical reinforcement learning, where the reward function for the meta-controller is explicitly based on the idea of  multiset labels. The (lower-level) controllers receive reward for directing attention to the area with high priority, as output by a previously computed priority map. The whole architecture is trained using REINFORCE, a standard RL algorithm. The proposed architecture is used for multi-label, specifically multi-set setting where each image can have multiple labels, and labels may appear multiple times. The experiments are conducted on MNIST and MNIST multiset datasets, where images contain one single digit or multiple digits. 
	The paper is well written and the model is well explained. The detailed visualizations in the supplementary material are required to understand the architecture with many components. The details of the architecture when considered with the supplementary material, are well motivated and explained; the paper is comprehensive enough to allow the reproduction of results. The topic is interesting and relevant for the community.
	Some questions are as follows:
	- The “meta controller” unit takes a saliency map and the class label prediction from the previous step as inputs and learns a covert attention mask. How is the saliency map initialized in the first step? Which saliency map does the “interface” unit use, the saliency map learned in the previous step or the initial one?
	- The controller takes the priority map and the glimpse vector as inputs, runs them through a recurrent net (GRU) and each GRU unit outputs a label. How are the number of GRU units determined here? Using the number of gaussians learned by the gaussian attention mechanism? Is this the same as the number of peaks in the priority map?
	A missing related work would be Hendricks et.al., Generating Visual Explanations, ECCV 2016 that also trains a recurrent net (LSTM) using REINFORCE algorithm to generate textual explanations of the visual classification decision. However it does not contain an integrated visual attention mechanism. 
	The main weakness of the paper is the experimental evaluation. The only experimental results are reported on synthetic datasets, i.e. MNIST and MNIST multi-set. As the objects are quite salient on the black background, it is difficult to judge the advantage of the proposed attention mechanism. In natural images, as discussed in the limitations, detecting saliency with high confidence is an issue. However, this work having been motivated partially as a framework for an improvement to existing saliency models should have been evaluated in more realistic scenarios. Furthermore, the proposed attention model is not compared with any existing attention models. Also, a comparison with human gaze based as attention (as discussed in the introduction) would be interesting. A candidate dataset is CUB annotated with human gaze data in Karessli et.al, Gaze Embeddings for Zero-Shot Image Classification, CVPR17 (another un-cited related work) which showed that human gaze based visual attention is class-specific. 
	Minor comment:
	- The references list contains duplicates and the publication venues and/or the publication years of many of the papers are missing.