Paper:
Efficient exploration plays a key role in reinforcement learning tasks. Commonly used dithering strategies, such as -greedy, try to explore the action-state space randomly; this can lead to large demand for samples. In this paper, We propose an exploration method based on the uncertainty in reward space. There are two policies in this approach, the exploration policy is used for exploratory sampling in the environment, then the benchmark policy try to update by the data proven by the exploration policy. Benchmark policy is used to provide the uncertainty in reward space, e.g. td-error, which guides the exploration policy updating. We apply our method on two grid-world environments and four Atari games. Experiment results show that our method improves learning speed and have a better performance than baseline policies.

Review:
This paper suggests an exploration driven by uncertainty in the reward space.
In this way, the agent receives a bonus based on its squared error in reward estimation.
The resultant algorithm is then employed with DQN, where it outperforms an e-greedy baseline.
There are several things to like about this paper:
- The idea of exploration wrt uncertainty in the reward is good (well... actually it feels like it's uncertainty in *value* that is important).
- The resultant algorithm, which gives bonus to poorly-estimated rewards is a good idea.
- The algorithm does appear to outperform the basic DQN baseline on their experiments.
Unfortunately, there are several places where the paper falls down:
- The authors wrongly present prior work on efficient exploration as "exploration in state space" ... rather than "reward space"... in fact prior work on provably-efficient exploration is dominated by "exploration in value space"... and actually I think this is the one that makes sense. When you look at the analysis for something like UCRL2 this is clear, the reason we give bonus on rewards/transitions is to provide optimistic bounds on the *value function*... now for tabular methods this often degenerates to "counts" but for analysis with generalization this is not the case: https://arxiv.org/abs/1403.3741, https://arxiv.org/abs/1406.1853
- So this algorithm falls into a pretty common trope of algorithms of "exploration bonus" / UCB, except this time it is on the squared error of rewards (why not take the square root of this, so at least the algorithm is scale-invariant??)
- Once you do start looking at taking a square root as suggested (and incorporating a notion of transition uncertainty) I think this algorithm starts to fall back on something a lot more like lin-UCB *or* the sort of bonus that is naturally introduced by Thompson (posterior) sampling... for an extension of this type of idea to value-based learning maybe look at the line of work around "randomized value functions"
- I don't think the experimental results are particularly illuminating when comparing this method to other alternatives for exploration. It might be helpful to distill the concept to simpler settings where the superiority of this method can be clearly demonstrated.
Overall, I do like the idea behind this paper... I just think that it's not fully thought through... and that actually there is better prior work in this area.
It could be that I am wrong, but in this case I think the authors need to include a comparison to existing work in the area that suggests "exploration by uncertainty in value space"... e.g. "deep exploration via randomized value functions"