Paper:
Naively applying Reinforcement Learning algorithms to continuous control problems -such as locomotion and robot control -to maximize task reward often results in policies which rely on high-amplitude, high-frequency control signals, known colloquially as bang-bang control. While such policies can implement the optimal solution, particularly in simulated systems, they are often not desirable for real world systems since bang-bang control can lead to increased wear and tear and energy consumption and tends to excite undesired second-order dynamics. To counteract this issue, multi-objective optimization can be used to simultaneously optimize both the reward and some auxiliary cost that discourages undesired (e.g. high-amplitude) control. In principle, such an approach can yield the sought after, smooth, control policies. It can, however, be hard to find the correct trade-off between cost and return that results in the desired behavior. In this paper we propose a new constraint-based approach which defines a lower bound on the return while minimizing one or more costs (such as control effort). We employ Lagrangian relaxation to learn both (a) the parameters of a control policy that satisfies the desired constraints and (b) the Lagrangian multipliers for the optimization. Moreover, we demonstrate policy optimization which satisfies constraints either in expectation or in a per-step fashion, and we learn a single conditional policy that is able to dynamically change the trade-off between return and cost. We demonstrate the efficiency of our approach using a number of continuous control benchmark tasks as well as a realistic, energy-optimized quadruped locomotion task.

Review:
This paper proposes a model free reinforcement learning algorithm with constraint on reward, with demonstration on cartpole and quadruped locomotion. 
strength: (1) challenging examples like the quadruped.
                (2) result seems to indicate the method is effective
There are several things I would like the authors to clarify:
(1) In section 3.2, why is solving (4) would give a "exactly the desired trade-off between reward and cost"? First of all, how is the desired trade-off defined? And how is (4) solved exactly? If it is solved iteratively, i.e, alternating between the inner min and outer max, then during the inner loop, wouldn't the optimal value for \lambda be infinity when constrained is violated (which will be the case at the beginning)? And when the constrained is satisfied, wouldn't \lambda = 0? How do you make sure the constrained will still be satisfied during the outer loop since it will not incurred penalty(\lambda=0). Even if you have a lower bound on \lambda, this is introducing additional hyperparameter, while the purpose of the paper is to eliminate hyperparamter?
(2) In section 3.2, equation 6. This is clearly not a convex combination of Qr-Vr and Qc, since convex combination requires nonnegative coefficients. The subtitle is scale invariance, and I cannot find what is the invariance here (in fact, the word invariance\invariant only appears once in the paper). By changing the parametrization, you are no longer solving the original problem (equation 4), since in equation (4), the only thing that is related to \lambda is (Qr-Vr), and in (6), you introduce \lambda to Qc as well. How is this change justified?
(3)If I am not mistaken, the constrained can still be violated with your method. While from the result it seems your method outperforms manually selecting weights to do trade off,  I don't get an insight on why this automatic way to do tradeoff is better. And this goes back to "exactly the desired trade-off between reward and cost" in point(1), how is this defined?
(3) The comparison in the cartpole experiment doesn't seem fair at all, since the baseline controller is not optimized for energy, there is no reason why it would be comparable to one that is optimized for energy. And why would a controller " switch between maximum and minimum actuation is indeed the optimal solution" after swingup? Maybe it is "a" optimal solution, but wouldn't a controller that does nothing is more optimal(assuming there is no disturbance)?
(4)For Table I, the error column is misleading. If I understand correctly, exceeding the lower bound is not an error (If I am wrong, please clarify it in the paper). And it is interesting that for target=0.3, the energy consumption is actually the lowest.
(5)Another simple way to impose constrained would be to terminate the episode and give large penalty, it will be interesting to see such comparison.
minor points: 
* is usually used for optimal value, but is used in the paper as a bound.