Paper:
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various highdimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domaindependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

Review:
This paper is already available on arxiv and cited 10 times. It is a very good paper introducing a new approach to count-based exploration in deep reinforcement learning based on using binary hashcodes. The approach is interesting, the presentation is didactical, the results are good and the related literature is well covered. I learned a lot from reading this paper, so my only criticisms are on secondary aspects.
I have few global points:
- I think it would be interesting to first train the convolutional autoencoder, and only then use your exploration method to compare its performance to state-of-the-art methods. This would allow for a more detailed analysis by disentangling the factors of potential inefficiency.
- I would put the "related work" section more forward in the paper, as the presentation of the methods would benefit from the points made in the first paragraph of related work and in the comparison with [4]. As a reader, I found that these points were important and arriving too late.
- Maybe section 5.4 of supplementary material should move into the main paper, and I don't share your main conclusion about it: If you look closely at Table 6, using n(s,a) is most often better than using n(s). To me, prefering n(s) to n(s,a) based just on the max in your table is inadequate, because maybe with an "optimized" (k,\beta) pair, you will find that using n(s,a) is better...
More local points:
line 41: "However, we have not seen a very simple and fast method that can work across different
"domains." How do you disqualify VIME here?
line 113: "An alternative method must be used to ensure that distinct states are mapped to distinct binary codes." Maybe the writing could be more direc there, presenting directly the appropriate approach.
Fig5: despite magnifying the picture several times, I cannot see the baseline. Didn't you forget it?
Footnote 4: the statement is weak. You should give reference to the state-of-the-art performance papers.
In the supplementary part:
Section 1 would benefit from a big table giving all params and their values. Beyond you, it would be nice if the community could move towards such better practices. As is, Section 1 is hardly readable and it is very difficult to check if everything is given.
Section 3: why do you show these images? Do you have a message about them? Either say something of interest about them, or remove them!
line 136: can (be) obtained (this was the only typo I could find, congratulations! ;))