Paper:
Goal-oriented learning has become a core concept in reinforcement learning (RL), extending the reward signal as a sole way to define tasks. However, as parameterizing value functions with goals increases the learning complexity, efficiently reusing past experience to update estimates towards several goals at once becomes desirable but usually requires independent updates per goal. Considering that a significant number of RL environments can support spatial coordinates as goals, such as on-screen location of the character in ATARI or SNES games, we propose a novel goal-oriented agent called Q-map that utilizes an autoencoder-like neural network to predict the minimum number of steps towards each coordinate in a single forward pass. This architecture is similar to Horde with parameter sharing and allows the agent to discover correlations between visual patterns and navigation. For example learning how to use a ladder in a game could be transferred to other ladders later. We show how this network can be efficiently trained with a 3D variant of Q-learning to update the estimates towards all goals at once. While the Q-map agent could be used for a wide range of applications, we propose a novel exploration mechanism in place of ε-greedy that relies on goal selection at a desired distance followed by several steps taken towards it, allowing long and coherent exploratory steps in the environment. We demonstrate the accuracy and generalization qualities of the Q-map agent on a grid-world environment and then demonstrate the efficiency of the proposed exploration mechanism on the notoriously difficult Montezuma's Revenge and Super Mario All-Stars games.

Review:
Authors propose to overcome the sparse reward problem using an exploration strategy that incentivizes the agent to visit different parts of the game screen. This is done by building Q-maps, a 3D tensor that measures the value of the agent's current state (defined as the position of the agent) and action in reaching other (x, y) locations in the map. Each 2D slice of the Q-map measures the value at different (x, y) locations for one action. Such 2D slices (i.e. channels) are stacked together to form the Q-map. Taking the max across the channels, thus, provides the Q-value for the optimal action. 
A policy for maximizing the rewards is trained using DQN. The Q-map based exploration is used as a replacement for \epsilon-greedy exploration. 
The Q-map is used for exploration in the following way:
(a) Chose a random action with probability \epsilon_r. 
(b) If neither a random action nor a "goal" is chosen, a new goal is chosen with probability \epislon_g. The goal is a (x, y) location, chosen so that is not too hard or too easy to reach it (i.e. Q-map values are neither too high or low; intuitively [1 - Q-map(x, y, a)] (for normalized/clipped Q) is a measure of distance of the goal).  
     -- If a "goal" is chosen, the greedy action to go towards the goal is chosen. 
(c) If neither a goal or random action is chosen, DQN is used to chose the greedy exploration. 
Authors also bias the goal selection to match DQN's greedy action. This is done as following -- from a set of goals that satisfy (b) above; chose the goal for which Q-map selected action matches the DQN's greedy action. 
Results are presented on simple 2D maze environments, Mario and Montezuma's revenge. 
I have multiple concerns with the papers:
(i) The writing is informal and the ideas are not well explained. It would really benefit -- if authors introduce an algorithm box or talk about the method as a sequence of points. Right now, the ideas are scattered throughout the paper. I am still confused by figure 3 -- when are random goals chosen? Do random goals correspond to (b) above? Also, when the Horde architecture, GVF and UVF are mentioned, the references are missing -- I would love for the authors to include the corresponding  references.  
(ii) The idea of reaching as many states as possible has been explored in count based visitation (Bellemare et al, Tang et al) — but no comparisons have been made to any previous work. Its always good to put a new work in the perspective of old work with similar ideas. 
(iii) The authors propose biased and random goal sampling — I would love to see how much improvement does biased goal sampling offer over random goal sampling. 
(iv) “…compare the performance of our proposed agent and a baseline DQN with a similar proportion of exploratory actions” .. I don’t agree with this a metric — I think the total number of steps is a good metric. Exploration is part of the agent’s algorithm to find the goal, we shouldn’t compare against DQN by matching the number of exploratory actions. 
(v) “The Q-map is trained with transitions generated by randomly starting from any free locations in the environment and taking a random action.” Does this mean that when the agent is trained with Mario — the game is reset after every episode and the agent is placed a random starting location? If yes, then this is not a realistic assumption. 
(vi) I would like to see — how do Q-maps generalize across levels of Mario or Montezuma’s revenge? Does Q-map trained on level-1 help in good exploration on future levels without any further fine-tuning? 
Overall, I like the idea of incentivizing exploration without changing the reward function as is done in multiple prior works. However, I think more thorough quantitative evaluation is required and it will be interesting to see transfer of Q-maps outside the 2D-domains. I am happy to increase my score if such evidence is provided. 
Other references worth including:
(a) Strategies for goal generation: Automatic Goal Generation for Reinforcement Learning Agents (https://arxiv.org/abs/1705.06366)