Paper:
CONTINGENCY-AWARE EXPLORATION IN REINFORCEMENT LEARNING
This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards.
1 For example, we report a state-of-the-art score of >11,000 points on MONTEZUMA'S REVENGE without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

Review:
This paper investigates the problem of extracting a meaningful state representation to help with exploration in RL, when confronted to a sparse reward task. The core idea consists in identifying controllable (learned) features of the state, which in an Atari game for instance typically corresponds to the position of the player-controlled character / vehicle on the screen. Once this position is known (as x, y coordinates on a custom low-resolution grid), one can use existing count-based exploration mechanisms to encourage the agent to visit new positions (NB: in addition to the x, y coordinates, extra information is also used to disambiguate the states for counting purpose, namely the current score and the state’s cluster index obtained with a basic clustering scheme). To find the position, the algorithm trains one inverse dynamics model per x, y cell on the grid: each model tries to predict the action taken by the agent given two consecutive states, both represented by their feature map (at coordinate x, y) learned by a convolutional network applied to the pixel representation. The outputs of these inverse dynamics models are combined through an attention mechanism to output the final prediction for the action: the intuition is that the attention model will learn to focus on the grid cell with best predictive power (for a given state), which should correspond to where the controllable parts of the state are. Experiments on several Atari games (including Montezuma’s Revenge) indeed show that this mechanism is able to track the true agent’s coordinates (obtained from the RAM state) reasonably well. Using these coordinates for count-based exploration (in A2C) also yields significantly better results compared to vanilla A2C, and beats several previously proposed related techniques for exploration in sparse reward settings.
The topic being investigated here (hard-exploration tasks) is definitely very relevant to current RL research, and the proposed technique introduces some novel ideas to address it, notably the usage of an attention model combined with multiple inverse dynamics models so as to identify controllable features in the environment. The approach seems sound to me and is clearly explained. Combined with pretty good results on well known hard Atari games, I am leaning toward recommending acceptance at ICLR.
I have a few significant concerns though, the first one being that the end result seems quite tailored to the specific Atari games of interest: trying to apply it to other tasks (or even just Atari games with different characteristics) may require significant changes (ex: the assumption that a single region of the screen is being controlled by the agent, the clustering to identify the various “rooms” of a game, and using the total score as a proxy to important state information). I do believe that some components are more general though (in particular the main new ideas in the paper), so this is not necessarily a major issue, but another example of application of these ideas to a different domain could have strengthened the submission.
In addition, even if experiments definitely investigate relevant aspects of the algorithm, I wish there had been an ablation study on the three components of the state representation used for counting (coordinates, cluster and reward). In particular it would be disappointing if similar results could be obtained with just the cluster and reward... even if I do not expect it to be the case, an empirical validation would have been welcome to be 100% sure.
The good results obtained here from exploration alone also beg the question whether this state representation could be useful to train the agent, by plugging it directly as input to the policy network (which by the way may not be trivial due to the co-training, but you get the idea). I realize that the focus of the paper is on exploration, and this is fine, but it seems to me a bit of a waste to build such a powerful state abstraction mechanism and not give the agent access to it. I was surprised that it was not at least mentioned in the discussion or conclusion. Note by the way that the conclusion says the agent “benefits from a compact, informative representation of the world”, which can be misinterpreted as using it in its policy.
Regarding the algorithm itself, one potential limitation is the fact that the inverse dynamics models rely on a single time step to identify the action that was taken. This means that they can only identify controllable state features that change immediately after taking a given action. But if an action has “cascading” effects (the immediate state change causing further changes down the road), there may be other important state features that could be controlled (across longer timesteps), but the algorithm will ignore them (also, in a POMDP one may need to wait for more than one timestep to even observe a single change in the state). I suspect that a more generic variant of this idea, better accounting for long term effects of actions, may thus be needed in order to work optimally in more varied settings.
Finally, I believe more papers deserve to be cited in the “Related Work” section. In particular, the idea of controlling features of the environment, (even if not specifically for exploration), has also been explored in (at least) the following papers:
- “Reinforcement Learning with Unsupervised Auxiliary tasks” (Jaderberg et al, 2017)
- “Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning” (Dilokthanakul et al, 2017)
- “Independently Controllable Factors” (Thomas et al, 2017)
- “Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World” (Sawada, 2018)
Relying on the position of the agent on the screen to drive exploration in Atari games has also been used in: “Deep Curiosity Search: Intra-Life Exploration Improves Performance on Challenging Deep Reinforcement Learning Problems” (Stanton & Clune, 2018)
Other remarks:
- Please share the code if possible
- In the Introduction, the sentence “it is still an open question on how to construct an optimal representation for exploration” seems to repeat “there is an ongoing open question about the most effective way of using neural network representations for exploration” => I wonder if one was supposed to replace the other?
- On p.2, last line containing citations: Pathak et al should be in the parentheses
- Please explicitly refer to Fig. 1 (Right) in 3.1
- On p.4, three lines above eq. 5, there is a hat{alpha} that should probably be hat{a}
- Is the left hand side L in eq. 5 the same as L^inv in Alg. 1? If so please use the same notations
- “privious” work in 3.2
- In 3.2 please briefly explain what psi is going to be. It is a bit confusing to have it appear “out of nowhere“, with no details on how it is constructed.
- Please explain what the different shades mean in Fig. 2-3
- In Table 2’s caption please add a reference for DQN-PixelCNN. Also what do the star and cross symbols mean next to the algorithms’ names?
- “coule” at end of 4.6
- The “Watson” citation is duplicated in references
- Why are there games with no tau in Table 4? Is it because there was no such clustering on these games? (if yes, that was not clear in the paper). And how was tau chosen for other games? (in particular I want to make sure the RAM state was not used to optimize it)
Update 2018-11-23: I am reducing my rating to 5 (from 6) due to the absence of author response regarding a potential revision addressing my comments/questions as well as those from other reviewers
Update 2018-11-27: I am increasing my rating to 7 (from 5) after the authors responded to reviewers' comments and uploaded a revised version of the paper