Paper: Handling Cost and Constraints with Off-Policy Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
Although the studied problem is interesting, the manuscript is not written and hard to follow.

Author response:
We agree with the criticism that the initial version of the paper was hard to follow, and have made numerous changes to correct this. In particular, we edited Section 3, as it was not clearly written and seemed to be a general source of confusion.

Reviewer comment (Writing):
Some notations in Algorithm 1 are not mentioned in Section 3 and Section 4; it is suggested that the authors explain some important notations in Section 3 and 4.

Author response:
We have revised Sections 2-4 as well as Algorithm 1 in an effort to make them clearer and to ensure that all notation is defined.

Reviewer comment (Writing):
For mixed-signed reward, it is suggested to give more practical examples; otherwise the statements seem confusing. The authors first claim the mixed-signed reward, later in Equation 8, the $c(t)$ is cost. Then in Section 4.2.1, the $C(\tau)$ is constraint. So the constraint could be part of negative reward?

Author response:
We have added further examples of mixed-sign rewards to the text. These range from robotics (navigation in the presence of obstacles, robotic surgery) to resource allocation (when both performance and efficiency must be considered), to financial decision-making (where the risk of losses exists in the pursuit of gains).

We have tried to clarify what we mean by cost, and in fact removed its definition in Section 3 to help with this. Cost, $c(t)$, is provided separately from reward at each time step of a constrained Markov decision process (CMDP). The constraint $C(\tau)$ is a function of the costs $c(1)… c(T)$ over the trajectory $\tau \equiv \mathbf{s}_1, \mathbf{a}_1, \ldots,  \mathbf{s}_T, \mathbf{a}_T $. 

In our case,  $C(\tau)$ was simply the total cost accumulated by the agent over the trajectory: $C(\tau) = \sum_{t=1}^T c(t)$.

Reviewer comment (Presentation):
In the abstract, the authors mention removing the maximization term and resetting the networks; however, this does not seem to be the crucial part of the manuscript.

Author response:
We have updated the abstract in an effort to make the contribution more clear. In particular, we never meant to imply that the use of resetting in conjunction with SAC and TD3 is novel. Rather, it was included as a way to study and address the issue we observed with mixed-sign rewards, as well as a comparison point for our approach (OPAC$^2$).

Reviewer comment (Novelty):
The authors consider the constrained RL problem, and it is suggested to consider and compare some related reference [1].

Author response:
We did in fact compare our approach with existing constrained RL methods, including Reward Constrained Policy Optimization (RCPO). We apologize that this was not made clear. More specifically, we compared our constrained results with those reported in [1], which leveraged RCPO in conjunction with PPO, TRPO, and a risk-sensitive approach to produce state-of-the-art on-policy constrained learning in Safety Gym. FOCOPS [2] was also included in that work. We used the same environments and cost targets as [1], achieving performance very similar to the best results there with something like 50 times less data.

Reviewer comment (Theory):
In Section 3, the author mentions “When Q is underestimated, |$Q_c$| will tend to be overestimated and |$Q_r$| underestimated.” Why will |$Q_c$| tend to be overestimated and |$Q_r$| underestimated? Could the author give more insights or theoretical explanations? Why not both Q networks underestimated?

Author response:
We agree that our explanation in Section 3 was inadequate, and in fact significantly edited it to better explain why bias in $Q$ estimation is likely to impact competing objectives asymmetrically. The key insight is that, as previously reported [3], agents trained off-policy and with objectives that maximize $Q$ will tend to overemphasize experiences from early in training. When multiple terms are present, this amounts to fixation on the reward terms that are most easily accessed by an agent of low capability. The issue becomes acute when terms explicitly compete, as in environments with mixed-sign rewards.

Reviewer comment (Experiments):
There are concerns about why two Q networks are used for $Q_r$ and $Q_c$. Even if the rewards are assigned at the same time, we can still use one network to estimate. The reviewer will consider increasing the rating when these concerns are fully resolved.

Author response:
We see how the way we wrote Section 3 could lead to misunderstandings of how $Q$ estimation of rewards and costs was handled. We have edited the paper to resolve this issue; here is a summary of how it is handled in our algorithms. For unconstrained learning, OPAC$^2$ requires one $Q$ function, one $V$ function, and the policy $\pi$. In removing a $Q$ function and adding $V$, our approach maintains the same number of networks and has slightly fewer parameters than SAC and TD3 (since the input to $V$ is $\mathbf{s}$ rather than $(\mathbf{s},\mathbf{a})$). For constrained learning, we have two $Q$ functions, two $V$ functions, and a policy $\pi$. The two $Q$ and two $V$ are necessitated by separate handling of positive rewards and costs in a CMDP and the changing scaling factor between them ($\beta$). Again, we have the same number of networks and fewer parameters than the constrained versions of SAC and TD3.

Reviewer comment (Evaluation):
The authors analyze why mixed-sign rewards are problematic in Sec.3, which is not so obvious to me. Could you please provide some more pieces of evidence to help me understand this?

Author response:
We agree that Section 3 was not written clearly, and have edited it to better explain why bias in $Q$ estimation is likely to impact competing objectives asymmetrically.  The key insight is that, as previously reported [1], agents trained off-policy and with objectives that maximize $Q$ will tend to overemphasize experiences from early in training. When multiple terms are present, this amounts to fixation on the reward terms that are most easily accessed by an agent of low capability. The issue becomes acute when terms explicitly compete, as in environments with mixed-sign rewards.

Reviewer comment (Theory):
I'm not quite sure why resetting can improve this issue. Do you have any intuition on this?

Author response:
Regarding why resetting helps with this problem: as noted in [2], resets combat learning failure due to inaccurate value estimates. Intuitively, they combat overfitting to experiences collected early in training. Indeed, we found (Figure 2 and Appendix D) that resetting significantly decreased both TD error and $Q$ estimation error in validation, leading to better agent performance. OPAC$^2$ achieves superior agent performance without resetting because it is not as inherently prone to inaccurate value estimates.

Reviewer comment (Experiments):
Similarly, I don't quite understand the intuition behind OPAC. Are there any ablation studies available?

Author response:
Regarding the intuition behind OPAC$^2$: there are a few elements that come together, which can be thought of in terms of bias and variance tradeoffs. In replacing the policy update of SAC (which maximizes Q via the reparameterization trick) with the "gentler" OPAC update, we largely remove the systematic bias on $Q$ estimates that is problematic in environments with competing objectives. The policy update in SAC is provably lower variance than the basic off-policy actor-critic, though this gap is closed through the use of a state-dependent baseline (i.e., value function). Our approach also leverages the ``squashing” trick of SAC to remove bias due to action clipping associated with control bounds, and may employ different forms of entropy regularization to enhance exploration.  Our experiments demonstrate that we are in fact able to build a practical algorithm around the policy update of the off-policy actor-critic; that is, in environments with mixed-sign rewards, the benefits of the superior value estimation of OPAC$^2$ more than outweigh the downsides of the potentially higher variance of its policy updates.

Reviewer comment (Experiments):
Do you compare OPAC with other Constrained RL algorithms?

Author response:
Regarding comparison of OPAC$^2$ with other constrained RL algorithms: yes, we do. Given that multiple reviewers missed this, we must apologize for not making this clear. See Section 5.2 of the revised manuscript. In addition to outperforming constrained SAC and TD3 in every environment tested, our approach essentially matches the performance of state-of-the-art on-policy approaches \emph{using roughly 50 times less data}. This latter statement is based on [3], where the same Car and Point environments were tested with the same cost thresholds.

Reviewer comment (Theory):
I assume the main selling point of the work is Section 3, where the authors discuss the limitations of learning two Q-functions. This section, however, is not clear and not convincing. First, the authors state that under mixed-sign rewards, the Q function can be decomposed into two Q functions, one for the rewards and one for the costs. Why should this decomposition be considered and analyzed? Why shouldn't we keep the overall Q function and learn it based on the total reward $r_{total}$? I can see that the decomposition would lead to over or underestimations for both the rewards and the costs, as the summation would fail to manage how the costs and rewards contribute to the overall rewards. So, it may not be a suitable approach to handle mixed-sign reward situations.

Author response:
First, we want to provide a clarification about an item in your summary: “The authors then argue that learning based on maximizing the summation of two $Q$-function approximations would lead to overestimates.” In the unconstrained setting, we don’t actually learn two $Q$-functions and sum them; we only learn one $Q$-function. We do learn different $Q$ functions for incentives and costs in the constrained setting, as well as a weighting factor for the cost term relative to the incentive term. The confusion here likely originates from our initial version of Section 3, which we agree was not written clearly. This section has been revised in the new draft (see next point).

We see the key contributions of this work being (1) the identification of the issue in Section 3 and (2) the proposal of OPAC$^2$ to address it. We rewrote Section 3 so as to better explain why bias in $Q$ estimation is likely to impact competing objectives asymmetrically. The key insight is that, as previously reported [1], agents trained off-policy and with objectives that maximize $Q$ will tend to overemphasize experiences from early in training. When multiple terms are present, this amounts to fixation on the reward terms that are most easily accessed by an agent of low capability. The issue becomes acute when terms explicitly compete, as in environments with mixed-sign rewards. The decomposition originally included in Section 3 was meant simply to illustrate the implication of independent terms in the reward function; in unconstrained learning we only ever learn one $Q$ function.

Reviewer comment (Theory):
Later on, Algorithm 1 is also based on two Q and two V functions. This raises the question of how this way of learning compares to learning one Q function based on $r_{total}$. In other words, can we say anything about the equivalence or convergence of this decomposition approach compared to single-Q learning?

Author response:
For unconstrained learning, OPAC$^2$ requires one $Q$ function, one $V$ function, and the policy $\pi$. In removing a $Q$ function and adding $V$, our approach maintains the same number of networks and has slightly fewer parameters than SAC and TD3 (since the input to $V$ is $\mathbf{s}$ rather than $(\mathbf{s},\mathbf{a})$). For constrained learning, we have two $Q$ functions, two $V$ functions, and a policy $\pi$. The two $Q$ and two $V$ are necessitated by separate handling of positive rewards and costs in a CMDP and the changing scaling factor between them ($\beta$).  Again, we have the same number of networks and fewer parameters than the constrained versions of SAC and TD3. We do see how these points were unclear in the initial draft, and have tried to improve the clarity in our revision.

Reviewer comment (Experiments):
The constrained approach requires introducing the threshold $d$. I am not sure how this can be done systematically as all we know are only the costs and rewards and an upper bound on the accumulated costs is essentially not available. In the experiments, the authors say that "we chose a target cost level equal to half the cost accumulated by a fully-trained TRPO agent unaware of cost." It is very unclear why it should be chosen this way. This needs more justifications.

Author response:
Regarding your comment requesting justification on the choice of cost threshold $d$: we should have included the rationale initially but have now added it to the draft. The intention was to provide a level that required the agent to strongly consider safety, but not be fully deterred from pursuing goals.  This choice also had the benefit of matching the thresholds used in [2], allowing us to directly compare the performance of our approach with that of state-of-the-art on-policy methods.

Reviewer comment (Theory):
The constrained RL approach based on Lagrange multipliers is simply to convert the overall rewards from $r+c$ to $r+\beta c$. This does not seem to be a good approach.

Author response:
Regarding your comment on the merits of Lagrangian approaches: we do agree that they are suboptimal in that they do not enforce safety during training. However, we find them to be effective in allowing the final agent to maximize positive rewards subject to an acceptable level of average cost.   This is still a valuable capability, and removes the need for reward shaping. Our formulation ties directly to the cost accumulated by the current agent (calculating the constraint over the most recent epoch in the replay buffer), allowing for an interpretable notion of safety.

Reviewer comment (Novelty):
The Resetting approach is clearly not new. The authors simply apply this to their problem context and find improvements. So it should not be considered a major contribution.

Author response:
We want to be clear that we do not consider our use of resetting to be a major contribution, and have tried to make this point clearer in the revised draft. We consider the observation that resetting helps SAC and TD3 significantly in environments with mixed-sign rewards to be interesting; however, the results with resetting are more intended to provide a comparison point for our OPAC$^2$ approach.

Reviewer comment (Experiments):
This is an empirical paper as the proposed method is nothing but the combination and examination of existing ideas without introducing new ones. Being an empirical paper is not a negative point, however, it requires comprehensive and thorough results. Unfortunately, it is not the case in this paper. In particular, the paper uses OpenAI Safety Gym benchmark to evaluate their method but failed to include more Safe-RL methods, like [1], etc. It was shown in previous papers that CMDP methods work the best in this benchmark and using standard RL methods (MDP-based not CMDP) don't result in good performance.

Author response:
Regarding the point that CMDP methods work best on Safety Gym:  We propose and analyze two variants of our OPAC$^2$ algorithm: one in the unconstrained setting and one in the constrained setting, comparing each to standard baselines in their respective settings. We compare the unconstrained version of our algorithm with SAC and TD3, and compare the constrained version of our algorithm with constrained versions of SAC and TD3; in all cases configurations among the three were as similar as possible. We agree that previous unconstrained and constrained approaches either do not perform well or are extremely sample inefficient (for instance, [1]); hence our OPAC$^2$ results represent a major improvement in both the unconstrained and constrained settings.

Reviewer comment (Experiments):
The authors' main contribution appears to be Algorithm 1, which is briefly described in Section 4.2.1. There are many issues here. First of all, this algorithm has many moving parts and is a rather very complicated method. For instance, it requires assigning 5 different learning rates (i.e. $\lambda_\beta, \lambda_\phi, \lambda_\psi, \lambda_\theta, \lambda_\alpha$) which clearly shows level of complexity in this method. In addition, while this method sometimes shows some improvement in some of the benchmarks, it remains unclear what drives these enhancements. It's crucial to note that the proposed algorithm is evaluated against not right baselines which are not designed for this specific problem setting. The results are also mixed and this method doesn't show a consistent trend in the experiments. For instance, compare results of DoggoGoal and CarPush in Figure 2. Finally, writing of section 4.2.1 and experiments section need major work as it's either too shallow (e.g. 4.2.1) or excessively and unnecessarily detailed, making it challenging to follow (especially in the experiment section).

Author response:
We agree that our pseudocode for Algorithm 1 appeared more complicated than needed, and have simplified our presentation accordingly. We also include the simpler unconstrained algorithm in the Appendix. We’d like to note a couple of things about complexity. First, we initially included all of the different learning rates in the paper in order to emphasize the flexibility available in tuning the method. However, our experiments did not require us to do excessive tuning. We simply chose $\lambda_\theta = \lambda_\psi = \lambda_\phi = 0.0001$, as in the SAC implementation that accompanied Safety Gym.  The default value of $\lambda_\alpha$ for SAC was used, and $\lambda_\beta$ was chosen to be significantly smaller than $\lambda_\theta$, as recommended by RCPO. Second, we note that our method does not require additional networks compared to SAC or TD3. OPAC$^2$ does add the value network but does not require a second $Q$ network; hence the total number of networks is the same and OPAC$^2$ actually uses slightly fewer parameters (because the input to $V$ is $\mathbf{s}$ rather than $(\mathbf{s},\mathbf{a})$).

Regarding your comment “In addition, while this method sometimes shows some improvement in some of the benchmarks, it remains unclear what drives these enhancements”: we agree that our original explanation (Section 3) could have been written better, and have revised it. In terms of what drives the improved performance, we provide strong empirical evidence that this is due to removing the explicit maximization of the Q function present in SAC and TD3, which is subject to much stronger systematic estimation bias than our approach.

We respectfully disagree that the algorithm is not evaluated against the right baselines, or that the results are inconsistent. In all 8 Safety Gym environments tested, OPAC$^2$ outperforms SAC and TD3 (both with and without resetting). This is true with both small and large static penalties and when the problem is formulated in the constrained setting (24 experiments total).

We agree with your assessment on the length/depth of section 4.2.1 and the experiments section.  We have revised section 4.2.1 to include more detail (though we are still limited by space), and have edited the experiments section to be clearer and more concise.