Paper:
This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on all of the replay data of an online DQN agent, can outperform online DQN on Atari 2600 games. We also present Random Ensemble Mixture (REM), a simple Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Using the DQN replay dataset, offline REM surpasses the gains from online C51 and outperforms offline QR-DQN. Furthermore, REM performs comparably to QR-DQN in the online RL setting on Atari 2600 games. The DQN replay dataset can serve as an offline RL benchmark and will be released.

Review:
*Summary*
Paper tackles an important issue, which is benchmarking off-policy deep RL algorithms. The motivation is important, because 1) assessing the performance of off-policy RL algorithm is hard due to the mixed effect of exploration and exploitation while collecting data; hence the paper suggest to fix the data collection process by a standard algorithm (DQN) and then compare the off-policy methods. 2) A fixed dataset will help the reproducibility of the results. 
Additionally the paper assess the performance of some off-policy RL methods (mainly DQN and QR-DQN) and showed that given offline data, these algorithms can obtain a superhuman performance. 
Besides the mentioned contribution, authors strived for a simple algorithm (easy to implement), and suggests a new simple algorithm, Random Ensemble Mixture (REM) by training multiple Q-functions and randomly sampling a subset for ensemble estimate. The mina motivation come from supervised learning literature. 
*Decision*
I enjoyed reading the paper and I think authors tackled a very important problem. Although a very important step, I vote for a weak reject for this paper as I believe the contribution of the current paper is limited. I made my decision mainly based on the following points:
[Limited Task Diversity]: I agree with the authors that having a fix offline data for testing different algorithms is important; however the suggestion (DQN logged data on Atari2600) is a very limited dataset, in terms of 1) task diversity 2) Data collection strategy. 
For a benchmark, I would like the dataset to have different strategies for data collection, so that the effect of data collection can be stripped off. In the current assessment, we compare the performance of algorithms condition on the DQN e-greedy data collection procedure. (For example, the right claim to make is: QR-DQN showed a better performance condition on the current data generation process). And it is not clear to me the same observation will hold given a different data generation process (for example a random policy, or a human playing data, or …)
Also, I think only focusing on Atari tasks is limited in nature, as many of these games share a similar underlying mechanism. 
[Overfitting] I believe having a fixed dataset for testing off-policy RL is great, however the current suggestion is very prone to overfitting. I can foresee that if the community start to use this benchmark as a test case, soon we will overfit to this task, in a sense that I try algorithm X on offline data, test on Atari see the results, tune algorithm X test on Atari again, … This way I finally have an algorithm that only learned from offline data, but is performing great on online Atari. But I am overfitting!
[Simplicity Trade-Off] Authors focused on getting a simple algorithm, but to me it is unclear why should we optimize for simple algorithm, and my impression of simple as authors have in mind is a “easy to implement” algorithm. 
Simplicity is great, but usually I believe we seek a simple algorithm to be able to get more insight into the theory and build more intuition. However, in this work there is a lot emphasis on simple algorithm, but no extra significant insight into off-policy learning has been gained. Additionally, I found the suggested algorithm ad-hoc and of limited contribution. 
Minor Comments/ Question that did not significantly contribute to my decision: 
[Valid Q-function]: authors mentioned a convex combination of Q-values are a valid Q-value, but never defined a “valid” Q-value. Do authors mean, if Q1 is generated by \pi_1 and Q2 by \pi_2 then convex combination of Q1 and Q2, Q^* can be also generated by some policy \pi^*? 
If so, this needs clarification. 
And if so, I’m not sure why this is important, since in Q-learning the Q-values that we obtain before convergence might be “invalid” (by the definition above, so that no real policy can generate those values exactly), so what is the point of paying attention to a valid Q values in the first place?
*Improvement*
I think some clear improvement avenues for the future are
1. More diverse set of tasks: I would like to see some non-Atari tasks included in the dataset.
2. More diverse data collection strategy: Example can be different exploration strategies than e-greedy. Or even random exploration, or maybe data of human players. Since we don’t want to have an algorithm that is good only for state distribution induced by e-greedy exploration.
3. More diverse task will help with overfitting by itself. But a good benchmark needs a test case that cannot be exploited. 
4. More insight into why REM is outperforming QR-DQN or other algorithms. Currently I am not convinced that we understand why REM is outperforming, or if it’s only because we designed it to be good for this specific task and dataset? 
Generally, I believe authors made a very important step, and I really enjoyed reading the paper; however, I think the current contribution is not sufficient to merit a publication. This makes a good workshop paper at this point.