Paper:
Policy gradients methods often achieve better performance when the change in policy is limited to a small Kullback-Leibler divergence. We derive policy gradients where the change in policy is limited to a small Wasserstein distance (or trust region). This is done in the discrete and continuous multi-armed bandit settings with entropy regularisation. We show that in the small steps limit with respect to the Wasserstein distance W 2 , policy dynamics are governed by the heat equation, following the Jordan-Kinderlehrer-Otto result. This means that policies undergo diffusion and advection, concentrating near actions with high reward. This helps elucidate the nature of convergence in the probability matching setup, and provides justification for empirical practices such as Gaussian policy priors and additive gradient noise.

Review:
The paper ‘Diffusing policies: Towards Wasserstein policy gradient flows’ explores 
the connections between reinforcement learning and the theory of quadratic optimal transport (i.e.
using the Wasserstein_2 as a regularizer of an iterative problem that converges toward
an optimal policy). Following a classical result from Jordan-Kinderlehrer-Otto, they show that 
the policy dynamics are governed by the heat equation, that translates in an advection-diffusion 
scheme. This allows to draw insights on the convergence of empirical practices in the field.
The paper is clear and well-written, and provides a comprehensive survey of known results in the 
field of Optimal Transport. The insights on why empirical strategies such as additive gradient noise
are very interesting and helps in understanding why they work in practical settings. That being said, 
most of the results presented in the paper are already known (e.g. from the book of Samtambrogio or the work 
of G. Peyré on entropic Wasserstein gradient flows) and it is not exactly clear what are the original
contributions of the paper. The fact that the objective is to learn policies
has little to no impact on the derivations of calculus. It clearly suggests that the entropy 
regularized Wasserstein_2 distance should be used in numerical experiments but this point is not 
supported by experimental results. Their direct applications is rapidly ruled out by highlighting the 
computational complexity of solving such gradient flows but in the light of recent papers (see 
the work of Genevay https://arxiv.org/abs/1706.00292 or another paper submitted to ICLR on large scale optimal transport 
https://openreview.net/forum?id=B1zlp1bRW) numerical applications should be tractable. For these reasons 
I feel that the paper would clearly be more interesting for the practitioners (and maybe to some extent 
for the audience of ICLR) if numerical applications of the presented theory were discussed or sketched 
in classical reinforcement learning settings.  
Minor comments:
 - in Equation (10) why is there a ‘d’ in front of the coupling \gamma ? 
 - in Section 4.5, please provide references for why numerical estimators of gradient of Wasserstein distances
are biased.