Paper:
Exploration bonuses derived from the novelty of observations in an environment have become a popular approach to motivate exploration for reinforcement learning (RL) agents in the past few years. Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models. In this paper, we introduce the concept of optical flow estimation from the field of computer vision to the RL domain and utilize the errors from optical flow estimation to evaluate the novelty of new observations. We introduce a flow-based intrinsic curiosity module (FICM) capable of learning the motion features and understanding the observations in a more comprehensive and efficient fashion. We evaluate our method and compare it with a number of baselines on several benchmark environments, including Atari games, Super Mario Bros., and ViZDoom. Our results show that the proposed method is superior to the baselines in certain environments, especially for those featuring sophisticated moving patterns or with high-dimensional observation spaces.

Review:
The paper proposes a novel way to formulate intrinsic reward based on optical flow prediction error. The prediction is done with Flownet-v2 architecture and the training is formulated as self-supervision (instead of the ground-truth-based supervised learning in the original Flownet-v2 paper). The flow predictor takes two frames, predicts forward and backward flows, then warps the first/second frame respectively and compares the warped result with real frame. The comparison error serves as the intrinsic reward signal. The results are demonstrated on 7 environments: SuperMario + 5 Atari games + ViZDoom. On those environments, the proposed method performs better or on-par with ICM and RND baselines. 
I am leaning towards rejecting this paper. Two key factors motivate this decision. 
First, the motivation for this work is not fully clear: why would the error in flow prediction be a good driving force for curiosity? Optical flow has certain weaknesses, e.g. might not work well for textureless regions because it's hard to find a match. Why would those weaknesses drive the agent to new locations? 
Second, the choice of tasks where the largest improvement is shown (i.e. 5 Atari games) seems not well-motivated and rather crafted for the proposed method. Those 5 Atari games are not established hard exploration games.
Detailed arguments for the decision above:
[major concerns]
* Analysis is need on how the method deals with known optical flow problems: occlusion, large displacements, matching ambiguities. Those problems don't fully go away with learning and it is unclear how correlated corresponding errors would be with state novelty.
* "Please note that ri is independent of the action taken by the agent, which distinguishes FICM from the intrinsic curiosity module (ICM) proposed in Pathak et al. (2017)" - but would it then be susceptible to spurious curiosity effects when the agent is drawn to motion of unrelated things? Like leaves trembling in the wind. ICM was proposed to eliminate those effects in the first place, but what is this paper's solution to that problem? Furthermore, the experiments on BeamRider show that this concern is not a theoretical one but quite practical.
* "CrazyClimber, Enduro, KungFuMaster, Seaquest, and Skiing" - none of those Atari environments are known to be hard exploration games (which are normally Gravitar, Montezuma Revenge, Pitfall!, PrivateEye, Solaris, Venture according to Bellemare et al "Unifying count-based exploration and intrinsic motivation"). I understand that every game becomes hard-exploration if the rewards are omitted but then there is a question why those particular games. Moreover, if you omit the rewards the question remains how to select hyperparameters of your method. Was the game reward used for selecting hyperparameters? If not, what is the protocol for their selection? This is a very important question and I hope the authors will address this.
* "These games are characterized by moving objects that require the agents to concentrate on and interact with." - this looks like tailoring the task to suit the method.
* Figure 6 - those results are not great compared to the results of Episodic Curiosity: https://arxiv.org/abs/1810.02274 . Maybe this is because of the basic RL solver (A3C vs PPO) but that brings up another question: why are different solvers used for different tasks in this paper? PPO is normally significantly better than A3C, why not use throughout the whole paper?
[minor concerns]
* Figures are very small and the font in them is not readable. Figure 2 is especially difficult to read because the axes titles are tiny.
* "complex or spare reward" -> sparse
* "However, RND does not consider motion features, which are essential in motivating an agent for exploration." - this is unclear, why are those features essential?
* "We demonstrated the proposed methodology and compared it against a number of baselines on Atari games, Super Mario Bros., and ViZDoom." - please state more clearly that only 5 out of 57 Atari games are considered, here and in the abstract.
* "Best extrinsic returns on eight Atari games and Super Mario Bros." - but only 5 games are shown, where are the other 3?
Suggestions on improving the paper:
1) Better motivating the approach in the paper would help. Why using the flow prediction error as a curiosity signal?
2) Better motivating the choice of the environments and conducting experiments on more environments would be important for evaluating the impact of the paper.