Paper:
Planning at roundabout is crucial for autonomous driving in urban and rural environments. Reinforcement learning is promising not only in dealing with complicated environment but also taking safety constraints into account as a as a constrained Markov Decision Process. However, the safety constraints should be explicitly mathematically formulated while this is challenging for planning at roundabout due to unpredicted dynamic behaviour of the obstacles. Therefore, to discriminate the obstacles' states as either safe or unsafe is desired which is known as situation awareness modeling. In this paper, we combine variational learning and constrained reinforcement learning to simultaneously learn a Conditional Representation Model (CRM) to encode the states into safe and unsafe distributions respectively as well as to learn the corresponding safe policy. Our approach is evaluated in using Simulation of Urban Mobility (SUMO) traffic simulator and it can generalize to various traffic flows. (Anonymous code is available to reproduce the experimental results and additional videos are also available 1 .)

Review:
This paper tackles the challenge of control with safety constraints using a learned soft constraint formulation. It models data as coming from a hierarchical generative model p(s) p(z|s) p(x|z,s) where s is a safety indicator variable, z is a latent variable, and x is an observation. Then it learns a variational approximation q(z|x) to the true posterior p(z|x). By then measuring a divergence between q(z|x) and p(z|s), the authors aim to evaluate whether a state is safe or unsafe. They then combine this learned safety constraint with a constrained RL method, SSAC, to learn safe policies.
While this setting is interesting and worthy of further exploration, there are sufficient issues with this work that it should not be accepted at this time. 
For one thing, the motivation behind the constrained MDP formulation of reinforcement learning, such as the original SSAC paper, is to provide safety throughout training. However, this work learns a soft constraint over the course of training by using examples where the policy leads to catastrophic failures. This means that, unlike SSAC, this work does not encourage safety at all early in training. Since it collects a large number of catastrophic experiences, this method is not categorically different from e.g. training a policy with large negative rewards for crashing.
Furthermore, since there is no constraint preventing crashing, a poorly-performing learned constraint might lead to better performance as measured by reward alone. Since safety during training comes with a tradeoff against rewards, as shown by SSAC, a poorly-functioning learned constraint might lead to improved reward. This work lacks an evaluation of the number of crashes to complement the rewards depicted in Figure 6.
The particulars of the method used for computing whether the safety constraint is violated are somewhat surprising. The authors use a Wasserstein distance to compute a cost as a function of the q(z|x) and p(z|s=1). They motivate this choice by the fact that the KL divergence would go to infinity for non-overlapping distributions; however, in equation (12) that would not significantly affect the computed cost. Since this divergence is calculated in the low-dimensional latent space and one of the distributions is fixed, it is also unclear that this would ever arise. It also seems that a likelihood ratio test between p(x|s=1) and p(x|s=0) would provide a more meaningful signal than simply classifying whether a point is near the "dangerous" prior p(z|s=1).
Overall I think there are some interesting ideas inside this work, but it needs some improvements:
1. Reframing to make the setting make more sense. If you want to compare against SSAC, you need to be minimizing the total number of catastrophic events during training. It might make sense to assume a pre-existing set of "dangerous" examples, e.g. labeled ahead of time by a human.
2. Textual editing and work to make the notation more consistent, e.g. the top of page 3 uses s for states as well as the "safe" indicator. 
3. Significantly improved evaluation. The results here lack crucial information about the number of catastrophic events during training. I would also like to see ablations or probe experiments showing what is learned by the encoder. Furthermore, this one environment is extremely simple (one degree of freedom) and to have confidence that the method works more generally, I would like to see the method applied to richer tasks.