Paper:
Although Reinforcement Learning (RL) algorithms have found tremendous success in simulated domains, they often cannot directly be applied to physical systems, especially in cases where there are hard constraints to satisfy (e.g. on safety or resources). In standard RL, the agent is incentivized to explore any behavior as long as it maximizes rewards, but in the real world undesired behavior can damage either the system or the agent in a way that breaks the learning process itself. In this work, we model the problem of learning with constraints as a Constrained Markov Decision Process, and provide a new on-policy formulation for solving it. A key contribution of our approach is to translate cumulative cost constraints into state-based constraints. Through this, we define a safe policy improvement method which maximizes returns while ensuring that the constraints are satisfied at every step. We provide theoretical guarantees under which the agent converges while ensuring safety over the course of training. We also highlight computational advantages of this approach. The effectiveness of our approach is demonstrated on safe navigation tasks and in safety-constrained versions of MuJoCo environments, with deep neural networks.

Review:
In this work, the authors studied solving the CMDP problem, in particular to model the constraint propagation using the idea of the backward value functions. Using the time-reversed probability property, they first established Bellman optimality conditions for the backward value function, which accounts for the tail expected return, conditioned on the final state. 
Utilizing  the notion of backward value function, they further use that to model the constraint in the CMDPs, and proposed several safe policy iteration algorithms for both cases with discrete actions and with continuous actions. Experiments show that this method has better constraint guarantees than the state-of-the-art algorithms, such as the Lyapunov approach.
The idea of using backward value functions to model constraints in CMDP is interesting and so far I have not seen it in other places. The algorithms developed with this approach also appear to work reasonably well (especially in constraint guarantees) on benchmark domains such as gridworld and mujoco control tasks. The authors also provide several properties such as consistent feasibility and policy improvement, similar to the Lyapunov method,  and derive several versions of safe policy optimization algorithms. However, I found the intuition of using backward value function rather unclear. Unlike the Lyapunov function, which attempts to estimate a near-optimal "remaining constraint budget" w.r.t. CMDP constraint, what is the motivation behind using the backward value. Does the backward probability have any connections to occupation measures? Without similar motivations it is unclear how close the performance of the policy computed from this backward value approach with that of an optimal CMDP policy. Also unlike the Lyapunov approach in Chow'18, the consistent feasibility  property in this work appears to be more restricted as it is only limited to a slow policy update. Finally the experimental results look promising, but it would be great to also compare with other state-of-the-art results such as Lagrangian method, and conservative policy improvement (such as CPO).