Paper:
Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. Finally, we apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.

Review:
*Summary*
Authors proposed a variety of metrics to measure the reliability of an RL algorithm. Mainly looking at Dispersion and Risk across time and runs while learning, and also in the evaluation phase. 
Authors have further proposed ranking and also confidence intervals based on bootstrapped samples. They also compared the famous continuous control and discrete actions algorithms on Atari and OpenAI Gym on the metrics they defined.
*Decision*
I believe the paper is discussing a very important issue, and some possible solutions to it, even if not perfect it's an important step toward paying more attention to maybe similar metrics. I am in favor of the paper in general,  but I have some concerns.
1. My main concern is that why authors think that community will adopt these metrics and report them? I like how authors have proposed different metrics, but having one or two easy to compute metric is much more likely to be adopted, than 6 different metrics, which I’m not sure how easy it is to use the python package? It’s of main importance, because if community don’t use these metrics in the future, the contribution of the paper is minimal. 
2. There is no question of the importance of reliability of RL algorithms, but we need to be careful that RL algorithms are not optimizing for metics like CVaR, so maybe a better learning algorithm (in the sense of expectation learning) might not have better reliability metrics because it is not the main objective.  
So following this, how would authors think their metrics can be used to design a more reliable algorithms? For example there is good literature on CVaR learning for safe policies. Do you think there exists a proxy for metrics you introduced that can be used to for the objective of the optimization?
3. Another main concern is the effect of exploration strategy: All these metrics can be highly affected  by different exploration strategy in different environments. For example if an environment has a chain like structure, then given the exploration strategy you may have an extremely high CVaR or IQR. How do authors think they can strip off this effect? (Running all algorithm with the same exploration strategy is not sufficient, since the interplay of learning algorithm and exploration may be important)
4. Generalizability: How do authors think these metrics are generalizable. For example if algorithm A has better metrics than algorithm B on open AI Gym task for continuous control, how much we expect the same ranking applies while learning on a new environment. I am asking this, because to me, some of these metrics are very environment dependent, and being reliable in some environments may not imply reliability in other environments.
*Note*:
Code and modularity of it: The main contribution of this paper will be shown when other researchers start using it and report the metric, if the code is hard to use, the contribution of the paper is hard to be significant. 
==== Post Rebuttal ====
Thanks for the responses authors posted, I think there is a good chance that the community will benefit from this experimental metrics in the future, so I increase my rating to accept.