Paper:
We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communication bandwidth. CuLE generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlighting the differences between CPU and GPU emulators in the context of reinforcement learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C+V-trace. CuLE is available at [URL revealed upon acceptance].

Review:
The work contributes a library emulating Atari games in GPU in parallel and allowing to speed-up the execution of reinforcement learning algorithms.
I see that the paper qualifies to the conference; in particular there is listed the topic:
- “implementation issues, parallelization, software platforms, hardware”
However, this is not a research paper, and I do not really see how I should asses it. What I can say about it is that it is considerable amount of work, not only implementing the simulator but also looking at what RL methods need, and how to optimize the allocation and exchange of the data so that everything would work on GPU more efficiently.
From the practical perspective, I am somewhat confused. The speed-up factors in the experiments are rather modest: about 4x for simulating and rendering frames, 2.5x for full RL, on a single GPU. Better with scaling to multi-GPU systems. In Table 1 the total training time per resources used differs dramatically. However if I look at the lines with A2C it is about the same time with 100-200 CPU cores + 1 GPU versus 12 cores + 1 GPU. So this is about factor 10 in the resources, versus CPU parallelization probably suffering overheads.
It appears that the maximum steed-ups are achieved for a particular type of the reinforcement learning algorithms, and using it in a general case would give a modest improvement.
The paper itself consists of introduction, related work, 1 page overview of what it means to simulate the Atari games, and experiments. So it is mostly about measuring the speedups, with several implementations / platforms.
I tend to think that this work will not very much boost the research for new RL methods. It is limited to Atari games, mostly helps to sample-inefficient RL methods and if it helps, the speed-up factors are not of the order that would make experiments by the researchers otherwise impossible. 
I would also give priority to theoretical contributions at ICLR. In the end, we all are using CUDA and cnDNN, but presentations about how they implement things are rather given at GPU computing conferences.