Paper:
Recent work has shown that quantization can help reduce the memory, compute, and energy demands of deep neural networks without significantly harming their quality. However, whether these prior techniques, applied traditionally to imagebased models, work with the same efficacy to the sequential decision making process in reinforcement learning remains an unanswered question. To address this void, we conduct the first comprehensive empirical study that quantifies the effects of quantization on various deep reinforcement learning policies with the intent to reduce their computational resource demands. We apply techniques such as post-training quantization and quantization aware training to a spectrum of reinforcement learning tasks (such as Pong, Breakout, BeamRider and more) and training algorithms (such as PPO, A2C, DDPG, and DQN). Across this spectrum of tasks and learning algorithms, we show that policies can be quantized to 6-8 bits of precision without loss of accuracy. We also show that certain tasks and reinforcement learning algorithms yield policies that are more difficult to quantize due to their effect of widening the models' distribution of weights and that quantization aware training consistently improves results over post-training quantization and oftentimes even over the full precision baseline. Additionally, we show that quantization aware training, like traditional regularizers, regularize models by increasing exploration during the training process. Finally, we demonstrate usefulness of quantization for reinforcement learning. We use half-precision training to train a Pong model 50% faster, and we deploy a quantized reinforcement learning based navigation policy to an embedded system, achieving an 18× speedup and a 4× reduction in memory usage over an unquantized policy.

Review:
This paper investigates the impact of using a reduced precision (i.e., quantization) in different deep reinforcement learning (DRL) algorithms. It shows that overall, reducing the precision of the neural network in DRL algorithms from 32 bits to 16 or 8 bits doesn't have much effect on the quality of the learned policy. It also shows how this quantization leads to a reduced memory cost and faster training and inference times.
I don't think this paper contributes with many novel results in the field, with most results being known or expected. The result that is interesting, in my opinion, is not properly explored.  The paper is well-written but it is a bit repetitive. It seems to me that the first 3 pages could be compressed in 1, as the same information is introduced over and over again. 
With respect to the results being known, quantization is known to succeed in supervised learning tasks. In a deep reinforcement learning algorithm, when you apply post-training quantization in a deep reinforcement learning algorithm, mainly when that algorithm uses a value function (e.g., A2C or DQN), the problem is reduced to a regression problem. It is no different than a supervised learning problem. One has the original network’s prediction and they need to match that prediction. The complexities introduced in the reinforcement learning problem (bootstrapping, exploration, stability) don’t exist anymore as they arise during training. Thus, it doesn’t seem to me that these results are novel or surprising. In a sense it is neat to see that eventual errors do not compound, but that’s it. If I were to write this paper I would make this set of experiments much shorter just as a sanity check. One thing that I feel is missing is a notion of the impact of the quantization not in the rewards accumulated but in the policy/value function. How often does the quantized agent take a different action than the original agent, for example? Does it happen often but only when it doesn’t matter, or is it rare?
The quantization during training is potentially interesting. It was not properly explored though. I wonder if the quantization during training has a regularization effect, which is known to improve agent’s performance in reinforcement learning (e.g., Cobbe et al., 2018, Farebrother et al., 2018). Does the agent generalize better when using a network with fewer bits of precision? How does this change impact training? These are all questions that could potentially make the results in this paper novel (i.e., quantization as a form of regularization), but as it is now, the results are not that surprising.
Importantly, there are important details missing in the paper that make it hard for me to evaluate the validity of the results presented. Are the results reported over multiple runs? What is the version of the Atari games used, is it the one with stochasticity? How much variance do we have if we replicate this process over different networks that perform well? These are questions I would like to see answered because they also inform us about the impact of the proposed idea. For example, if by repeating this experiment multiple times one observe a high variance, it might mean that different models might be impacted in different ways.
The results in the “real-world” (Pong is not real-world) are not that surprising as well. Basically they show that if one uses a network with lower precision training and inference are faster, which, again, is not surprising. 
There’s also an important distinction in the results that is not discussed in the paper: DQN estimates a value function while methods such as PPO directly estimate a policy. The reason DQN might have a wider distribution is exactly because it is estimating a different objective. These are important details that should be acknowledged and discussed in the paper. In my opinion, for this paper be relevant, it should have a very thorough evaluation of these different dimensions of reinforcement learning algorithms, with explicit discussions about it. Variance, the impact of quantization during learning, the distinction between parametrizing policies versus value functions, etc.
Finally, there are some aspects of the presentation of this paper that could also be improved. Aside from typos, below are some other comments on the presentation.
- There’s no such thing as Atari environment, it is either Arcade Learning Environment (Bellemare et al., 2013) or Atari games.
- I’d introduce/explain quantization in the beginning of the second paragraph of the Introduction for those not familiar with the term.
- No references are provided for the environments used. You should refer to Bellemare et al.’s (2013) work as well as Brockman et al.’s (2016).
- Is it really necessary to explain Fp16 quantization as it is done now, with even a picture of two bytes? I’d expect most readers are familiar with how numbers are represented in a computer.
- The equation for Uniform Affine Quantization is pretty much the same as the one in the Section Quantization Aware Training. All these “repetitions”, or discussions that are common-knowledge give the impression that the paper is trying to fill all the pages without necessarily having enough content.
- The references are not standardized (e.g., sometimes names are shortened, sometimes they are not) and the paper “Efficient inference engine on compressed deep neural network” is cited twice.
References:
Marc G. Bellemare, Yavar Naddaf, Joel Veness, Michael Bowling: The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res. 47: 253-279 (2013)
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba: OpenAI Gym. CoRR abs/1606.01540 (2016)
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, John Schulman: Quantifying Generalization in Reinforcement Learning. CoRR abs/1812.02341 (2018)
Jesse Farebrother, Marlos C. Machado, Michael Bowling: Generalization and Regularization in DQN. CoRR abs/1810.00123 (2018)
------
>>> Update after rebuttal: I stand by my score after the rebuttal. 
The rebuttal did acknowledge some points I made to me the paper took a gradient update towards the right direction. I don't think the paper is quite there yet though. It is repetitive, spending too much time with basic concepts, and it still ignores small details that matter (e.g., calling it Atari Arcade Learning). I strongly recommend the authors to follow my recommendations closely and then submit the paper again to a next conference. The discussion about generalization is potentially interesting, going beyond the regularization for exploration aspect. A better discussion about quantization during learning is also essential. The first three pages could probably be compressed by half.