Paper: Preventing Reward Hacking with Occupancy Measure Regularization (ICLR-2024)

Reviewer comment (Theory):
The authors claimed that there are no direct relationships hold for the returns of two policies under any reward and their action distribution divergence. However, Theorem 1 of [1] shows that we can indeed bound the returns of two policies by their action distribution, which from my perspective, just verifies the effectiveness of KL divergence minimization. And I highly recommend authors to thoroughly read [1], which already theoretically and empirically illustrates OM regularization is better than action distribution regularization.

Author response:
* **Novelty of our theoretical results:** We would like to emphasize that our theoretical findings are unique in their application to the difficult *reward hacking* setup. We acknowledge that there has been some previous work related to the result we show in Proposition 3.2, which bounds the difference between returns of policies in terms of their distance from each other in occupancy measure space [1, 2]. However, we do believe that the result proved in Proposition 3.1 is novel and significant. This result in particular highlights the challenges of using action distribution KL for regularizing goal-oriented agent activity to avoid reward hacking, and to the best of our knowledge, it has not appeared in the literature before. As the reviewer notes, Theorem 1 of Xu et al. does show the result that you can bound difference in returns by action distribution divergences, but this has an additional $\frac{1}{1-\gamma}$ term compared to our bound. Thus, as $\gamma$ grows closer to 1, the previously proved bound becomes less tight. In contrast, our result in Proposition 3.1 utilizes an arbitrarily large $\gamma$ to show that it is impossible to bound the difference in returns using action distribution divergences in a manner that does not depend on the effective horizon $\frac{1}{1-\gamma}$. This is particularly important in long-horizon environments, such as the ones we explore in our experiments. We will be more explicit about which of our theoretical results are similar to those that have appeared in the literature previously and which are novel in the final version of the paper.

Reviewer comment (Novelty):
Regularizing the learned policy by OM divergence is not a novel idea, and has already been explored in RL, such as [2].

Author response:
* **Novelty of ORPO:** We acknowledge that occupancy measure matching has been utilized previously in RL for different applications, and as the reviewer notes, our method of keeping the learned policy within a certain distance in occupancy measure space of the safe policy is similar to the approach proposed by Kang et al. However, we are applying our algorithm to quite a different setting. In particular, in addition to empowering the agent to explore more like in previous works, ORPO is also able to prevent potentially unsafe reward hacking. Additionally, we also consider a suite of more realistic settings when evaluating our method and find that it can perform well in complex environments with both continuous and discrete action spaces.

Reviewer comment (Evaluation):
Why would the occupancy measure KL suddenly increases when $\lambda$ increases (Figure 3)? If I am not wrong, as $\lambda$ increases, we focus more on minimizing the OM divergence of the learned policy and the safe policy, so the occupancy measure KL should decrease?

Author response:
* **OM KL increasing as $\lambda$ increases:** We sometimes see instability at high values of $\lambda$ because the added rewards from the discriminator are oftentimes very large compared to the underlying rewards from the environment, and they are changing over time. As we show in Figure 3, ORPO works best with more moderate values of $\lambda$ since moderate regularization allows for both reward hacking prevention and improvement upon the provided safe policy. So, we do not believe that instability with high values of $\lambda$ presents a challenge to the practical adoption of our regularization method.

Reviewer comment (Writing):
What do you mean by "safe" policy? I can not find a formal definition of it. Is it the policy that never visits some unsafe state-action pairs? Then some safe-RL or constrained RL methods should also be compared.

Author response:
* **Definition of a safe policy:** A safe policy is any policy which has reasonable, potentially quite suboptimal, performance and does not reward hack. The objective of ORPO is to stay close enough to the safe policy such that unsafe, reward hacking behaviors can appropriately be avoided, while also allowing for enough exploration to actually perform better than the safe policy. Safe policies can be hardcoded or learned from human data. For instance, as noted in section 4, we use the Intelligent Driver Model, which is a widely accepted approximation of human drivers, as the safe policy for the traffic environment and for the glucose environment, we use glucose values that were selected through rigorous scientific testing. Section 2 of our paper presents several comparisons to previous safe and constrained RL methods.

Reviewer comment (Experiments):
Is proposition 3.1 just to conculde that the parameter $\lambda$ is hard to tune if we use action distribution regularization? But in your proposed OM regularization, we need to adversarially train a discriminator, which has been claimed (e.g., [1]) to be unstable and sensitive to hyper-parameters, especially in high-dimensional domains.

Author response:
* **Clarifying Proposition 3.1:** Proposition 3.1 actually does not claim that $\lambda$ is hard to tune—it instead shows that *no value of $\lambda$ will work for regularization.* Proposition 3.1 shows that arbitrarily small changes in action distributions between policies can cause arbitrarily large differences in return, and vice versa: arbitrarily large changes in action distributions between policies can cause arbitrarily small differences in return. Due to this result, there is a contradiction that exists in selecting the $\lambda$ value: if it is high, then it will prevent any improvement on the safe policy, but if it is low, then it will allow reward hacking to occur, rendering it impossible to regularize effectively using this method.

Reviewer comment (Experiments):
During training, do we need to roll out the safe policy? If so, OM regularization may need more environment interactions than action distribution regularization.

Author response:
* **Generating rollouts for the safe policy:** ORPO does require more interaction with the environment because it needs to sample data from both the safe policy and the policy currently being learned in order to train the discriminator. However, because the safe policy is not being updated when ORPO is trained, we can collect rollouts from it at the beginning of training and store it in a replay buffer. This way, we will incur a single initial sampling cost that is amortized over the course of the training process, as these pre-collected samples are utilized throughout the algorithm's iterations.

Reviewer comment (Novelty):
While the paper discusses with a new problem formulation on reward hacking, the problem definition itself does not seem to be very different from what offline RL algorithms do, which assume a transition/reward function mismatch and learns a robust policy for underlying true transition/reward function. The potential differences could be, e.g. surrogate reward function can be largely different, or transition function stays exact, etc. However, the authors do not make much difference in designing their algorithm, and I could not find much difference in proposed OM algorithm and other offline RL algorithms out there.

Author response:
We thank the reviewer for their interest in our problem definition and their appreciation of our theoretical and empirical results. The reviewer primarily challenged our algorithm's novelty by making comparisons to offline RL methods; however, we argue below that the motivation behind and algorithms for offline RL are fundamentally different from ORPO:

* **Difference between our motivation and that of offline RL:** Offline RL aims to avoid issues from having a state distribution mismatch between the training data available and the states that will be seen during deployment. In contrast, we are trying to solve a problem of when there is a *reward* function mismatch. Specifically, offline RL doesn’t necessarily consider any reward function, and even with knowledge of the environment’s transition dynamics or infinite amounts of data, offline RL algorithms can still perform horribly without any ground truth reward signal, resulting in catastrophic outcomes [1]. Several previous offline RL theoretical results have only provided performance guarantees in the case of when the dataset actually reflects the true reward function [2]. The limitations that are addressed by offline RL methods are also separate from the problem of reward hacking that ORPO addresses. In particular, in the offline RL setting, we will practically have limited amounts of data available, whereas in our setting, we are challenged by a misspecified or “hackable” reward that can motivate unsafe behavior from the agent. 

* **Difference between ORPO and offline RL algorithms:**  Offline RL algorithms typically optimize over the empirical transition or reward function found within the provided dataset, which is subject to estimation errors due to the limited amount of data practically available. So far, the approach to account for this estimation error has been to act pessimistically, applying different kinds of reward penalties based on the error [3]; however, this pessimism can result in suboptimal policies that do not explore enough [4].  We acknowledge the fact the occupancy measures have also been used previously in the literature; however, again, ORPO is unique in its emphasis on preventing *reward hacking*. The reviewer in particular mentions algaeDICE, OptiDICE, and other methods from the DICE family [5], which have a dual objective of estimating the ratio between the occupancy measures of both the expected optimal policy and the policy under which the dataset was collected and optimizing the learned policy so that the ratio previously calculated is minimized. These methods do not actually calculate occupancy measures of a particular policy; instead, they use a duality trick to approximate the corrections that will be needed to turn the offline dataset distribution into the expected optimal policy’s distribution. On the other hand, we actually calculate the occupancy measures using a discriminator and incorporate them into our algorithm as a crucial value for regularizing the learned policy to a provided safe policy, rather than trying to approximate some desired policy and adjusting the safe policy’s distribution to match that. We will include a thorough comparison to these and other related methods in the final paper.

Reviewer comment (Novelty):
In that sense, there are a number of occupancy matching algorithms (algaeDICE, OptiDICE, ...) for offline RL, and I believe that it should have been shown how the proposed OM algorithm is different to those and why it is better suited in this case.

Author response:
We thank the reviewer for their interest in our problem definition and their appreciation of our theoretical and empirical results. The reviewer primarily challenged our algorithm's novelty by making comparisons to offline RL methods; however, we argue below that the motivation behind and algorithms for offline RL are fundamentally different from ORPO:

* **Difference between ORPO and offline RL algorithms:**  Offline RL algorithms typically optimize over the empirical transition or reward function found within the provided dataset, which is subject to estimation errors due to the limited amount of data practically available. So far, the approach to account for this estimation error has been to act pessimistically, applying different kinds of reward penalties based on the error [3]; however, this pessimism can result in suboptimal policies that do not explore enough [4].  We acknowledge the fact the occupancy measures have also been used previously in the literature; however, again, ORPO is unique in its emphasis on preventing *reward hacking*. The reviewer in particular mentions algaeDICE, OptiDICE, and other methods from the DICE family [5], which have a dual objective of estimating the ratio between the occupancy measures of both the expected optimal policy and the policy under which the dataset was collected and optimizing the learned policy so that the ratio previously calculated is minimized. These methods do not actually calculate occupancy measures of a particular policy; instead, they use a duality trick to approximate the corrections that will be needed to turn the offline dataset distribution into the expected optimal policy’s distribution. On the other hand, we actually calculate the occupancy measures using a discriminator and incorporate them into our algorithm as a crucial value for regularizing the learned policy to a provided safe policy, rather than trying to approximate some desired policy and adjusting the safe policy’s distribution to match that. We will include a thorough comparison to these and other related methods in the final paper.

Reviewer comment (Theory):
In offline RL, having KL-divergence regularization usually results in an over-regularization, since we could have more flexible learned policy by having other divergences that matches support only. Is there any specific reason why KL divergence is chosen in this paper? The similar mathematical results should be able to be derived using many other divergences.

Author response:
* **Using other divergences for ORPO:** After  experimenting with regularization based on the TV and Wasserstein distances between policy occupancy measures, we found them to be much more unstable compared to KL-based regularization. KL divergences are generally favorable since they tend to be more stable to calculate and are easier to approximate using various tricks, such as discriminators. Current state-of-the-art methods also tend to rely on KL divergences for regularizing to safe policies [6, 7, 8, 9], which also contributed to us choosing it for ORPO.

Reviewer comment (Evaluation):
The authors don't show the training curves either (performance over time) so I'm unable to assess convergence.

Author response:
* **Stability of ORPO:** The reviewer asked if occupancy measure regularization is stable and will converge. Deep RL is poorly understood theoretically [1, 2], and furthermore, it has been generally difficult to make theoretical progress on other algorithms that use a discriminator, like GANs or GAIL. Thus, it is very difficult to theoretically analyze the stability and convergence of ORPO. However, in practice, we find that training is stable. Please reference Appendix B, Figure 4 to see learning curves that show the true reward, proxy reward, and estimated occupancy measure KL over time for ORPO in each environment. In this graph, the training is relatively stable and appears to converge. We do sometimes see instability at higher values of $\lambda$ because the added rewards from the discriminator are very large compared to the underlying rewards and are changing over time. This explains why the occupancy measure divergence for glucose appears to be increasing for large values of $\lambda$. As we show in Figure 3, ORPO works best with moderate values of $\lambda$ since moderate regularization both prevents reward hacking and allows for improvement upon the provided safe policy. Thus, we do not believe that instability with high values of $\lambda$ presents a challenge to the practical adoption of our regularization method.  

* **Comments about specific results:** As we noted above, OM KL calculations can be more unstable at higher values of $\lambda$, which is why the OM divergence increases in glucose for larger $\lambda$ values. We find that large values of $\lambda$ over-regularize and are too unstable, and smaller $\lambda$ values do not regularize enough to prevent reward hacking, resulting in poor performance in all of the environments. Please reference the traffic environment plots in Appendix B, Figure 4 for a clear example of learning curves that show an initial spike in proxy reward and a dip in true reward, followed by an increase in true reward as the policy is increasingly regularized.

Reviewer comment (Evaluation):
I'd also like to see some discussion of what sort of environments the authors expect the method to do well in, extrapolating from the experiments. I.e. is this *the* solution to reward hacking or should it be used in conjunction with other methods, and is it effective anywhere or only in environments with particular properties?

Author response:
* **Applicability to a diverse set of environments:** The reviewer also asked about which environments we expect ORPO to be most useful in. We anticipate ORPO to be broadly applicable across a wide range of realistic environments since our theoretical results suggest that occupancy measure KL is an effective regularizer for **any** environment and **any** reward function. While our approach is a crucial step towards preventing reward hacking, we do not believe that it, as a regularization method, can single-handedly address all challenges posed by reward misspecification. Ideally, our algorithm should be combined with methods to improve the reward function and/or to monitor anomalous behaviors. However, we believe that ORPO can be a key part of preventing reward hacking and should replace action distribution regularization as the standard method for regularizing to a safe policy.

Reviewer comment (Novelty):
The paper claims Proposition 3.2 (page 5) as a primary theoretical contribution, yet its proof in the appendix is not novel: the supporting Prop. A.1 is standard in MDP theory, as the RHS is the objective of the dual LP formulation for solving an MDP; the remaining bound is trivial (it is also previously known -- see its use in, e.g., the proof of Theorem 1 in [Qu et al., *Scalable multi-agent reinforcement learning for networked systems with average reward*, NeurIPS 2020]). This undermines the theoretical contribution of the paper.

Author response:
* **Significance of theoretical results:** We would like to emphasize that our theoretical findings are unique in their application to the *reward hacking* setup. We do acknowledge that there has been some previous work related to the result we show in Proposition 3.2, which bounds the difference between returns of policies in terms of their distance from each other in occupancy measure space [1, 2]; however, we believe that the result we prove in Proposition 3.1 is novel and significant. Proposition 3.1 shows that arbitrarily small changes in action distributions between policies can cause arbitrarily large differences in return, and vice versa: arbitrarily large changes in action distributions between policies can cause arbitrarily small differences in return. This result in particular highlights the challenges of using action distribution KL for regularizing goal-oriented agent activity to avoid reward hacking, and to the best of our knowledge, it has not appeared in the literature before. We will be more clear about which of our theoretical results are similar to those that have appeared in the literature previously and which are novel in the final version of the paper.

Reviewer comment (Experiments):
The experiments are not as extensive as one could hope. It is noted in the appendix that three replications are performed for each method, but only the median rewards are reported -- without error bars (and ideally more replications), it is tough to assess the statistical significance of the results reported in Table 1 and Fig. 3. In addition, though hyperparameters are reported, it is unclear how they were selected to ensure a fair comparison between all methods. In light of issue 1, it would be helpful to have more thorough experimentation to address 3.

Author response:
* **More Robust Experimentation:** We have done some more experimentation, increasing the coverage of $\lambda$ values. Figure 3 and Table 1 in the paper have been updated with the new results, along with error bars and metric standard deviations. In some cases, the variance across seeds is quite high, oftentimes due to the general instability associated with RL training, but in the best-performing cases, the variance is relatively lower and shows that ORPO outperforms action distribution regularization. For instance, in the case of the glucose and traffic environments, even the lowest performing seed of the best coefficient for occupancy measure regularization outperforms the best performing seed of the best coefficient for action distribution regularization. 

* **Hyperparameter Selection:** As noted in Appendix D.1, we largely used the hyperparameters for all environments from Pan et al. In some cases, we adjusted hyperparameter values to ensure that reward hacking consistently occurred when an unregularized RL agent (i.e., PPO without regularization) was taking actions in the environment, so that we could clearly compare methods for preventing this reward hacking. Between action distribution regularization and ORPO we used the same hyperparameters, so we do not believe that our hyperparameter selection advantaged one method over the other.

Reviewer comment (Novelty):
How significant are the examples constructed in Prop. 3.1? These seem to require some work in the appendix -- do you view Prop. 3.1 as a significant contribution on its own?

Author response:
* **Significance of theoretical results:** We would like to emphasize that our theoretical findings are unique in their application to the *reward hacking* setup. We do acknowledge that there has been some previous work related to the result we show in Proposition 3.2, which bounds the difference between returns of policies in terms of their distance from each other in occupancy measure space [1, 2]; however, we believe that the result we prove in Proposition 3.1 is novel and significant. Proposition 3.1 shows that arbitrarily small changes in action distributions between policies can cause arbitrarily large differences in return, and vice versa: arbitrarily large changes in action distributions between policies can cause arbitrarily small differences in return. This result in particular highlights the challenges of using action distribution KL for regularizing goal-oriented agent activity to avoid reward hacking, and to the best of our knowledge, it has not appeared in the literature before. We will be more clear about which of our theoretical results are similar to those that have appeared in the literature previously and which are novel in the final version of the paper.

Reviewer comment (Evaluation):
What are the variance/confidence intervals for the values reported in Table 1? What about Fig. 3?

Author response:
* **More Robust Experimentation:** We have done some more experimentation, increasing the coverage of $\lambda$ values. Figure 3 and Table 1 in the paper have been updated with the new results, along with error bars and metric standard deviations. In some cases, the variance across seeds is quite high, oftentimes due to the general instability associated with RL training, but in the best-performing cases, the variance is relatively lower and shows that ORPO outperforms action distribution regularization. For instance, in the case of the glucose and traffic environments, even the lowest performing seed of the best coefficient for occupancy measure regularization outperforms the best performing seed of the best coefficient for action distribution regularization.

Reviewer comment (Novelty):
While the idea of the paper is neat, it is not super novel as similar ideas have been explored in previous works such as GAIL and [1]. It is unclear if the theoretical finding in the paper is fully new as it has been known in the literature that KL divergence as the action distribution regularization may not be ideal and occupancy measure matching can capture the total return variation more accurately [2].

Author response:
* **Novelty of using occupancy measures:** We acknowledge the fact that occupancy measure (OM) matching has been studied and utilized in the literature previously; however, our usage of OM divergence for policy regularization to resolve the relatively unexplored, yet critical problem of *reward hacking* is what makes our method novel. The reviewer in particular mentioned GAIL [1]. While GAIL does also rely on state-action occupancy measures, similar to our method, it is primarily focused on finding a policy that replicates as expert’s demonstrations. In contrast, our method ORPO is focused on *improving* beyond a safe policy (e.g., one trained via imitation learning). Because ORPO combines occupancy measure divergence and reward objectives, it can both outperform the safe policy and avoid reward hacking. We will include a detailed comparison to GAIL and other prior works that use occupancy measure divergences in the final paper if accepted.
 * **Novelty of theoretical analysis:** We also acknowledge that there has been previous work that has focused on bounding, e.g., the difference in value functions between policies whose occupancy measure divergence is bounded [2]—this is quite similar to our Proposition 3.2. However, we believe our Proposition 3.1 is novel. It shows that arbitrarily small changes in action distributions between policies can cause arbitrarily large differences in return, and vice versa: arbitrarily large changes in action distributions between policies can cause arbitrarily small differences in return. This result in particular highlights the challenges of using action distribution KL for regularizing RL to avoid reward hacking, and to the best of our knowledge has not appeared in the literature before. We will be more clear about which of our theoretical results are similar to those which have appeared in the literature before and which are novel in the final version of the paper.

Reviewer comment (Experiments):
Moreover, the empirical results in the paper are less realistic. The experiments are all in simulated environment with not very complex control space. It would add more value if the authors can perform experiments in more complicated settings such as continuous-control settings like robotic manipulation/locomotion or large language models alignment setting that require RLHF.

Author response:
* **How realistic are our experiments:** The reviewer mentioned that they feel our experiments are less realistic and suggested evaluating on robotics domains. However, we argue that the diverse environments used in our paper for experimentation are actually highly realistic, with the exception of the toy tomato-watering gridworld environment. In general, we believe that these settings are better for robustly evaluating ORPO than robotics tasks. The traffic simulation and glucose monitoring environments have been crafted by domain experts and are backed by extensive scientific studies. As we note in Section 4, the Flow traffic simulator is based on SUMO [3], which was developed by members of the Institute of Transportation Systems. It has been used to study various complex real-world scenarios related to mixed-autonomy traffic and the practical deployment of autonomous vehicles [4,5]. In addition, the safe policy for the traffic environment is the Intelligent Driver Model that is widely accepted by professionals in the field [6]. The SimGlucose Type 1 Diabetes monitoring was built off of the FDA-approved UVa/Padova Simulator [7], and the blood glucose and other patient state values used in the simulated scenarios have been calibrated by leading diabetes researchers from the Harvard Medical School and Boston Children's Hospital [8]. Unlike many common robotics benchmarks, our environments require balancing objectives that are difficult to specify, such as “good traffic” and “patient health/financial wellness”; this leads to naive reward functions being easily hackable, as we find in our experiments. In contrast, most common robotics benchmarks, like DM Control, have relatively simple reward functions that cannot be hacked, so it is more difficult to evaluate ORPO in those.

Reviewer comment (Novelty):
Please clarify the novelty of the paper.

Author response:
* **Novelty of using occupancy measures:** We acknowledge the fact that occupancy measure (OM) matching has been studied and utilized in the literature previously; however, our usage of OM divergence for policy regularization to resolve the relatively unexplored, yet critical problem of *reward hacking* is what makes our method novel. The reviewer in particular mentioned GAIL [1]. While GAIL does also rely on state-action occupancy measures, similar to our method, it is primarily focused on finding a policy that replicates as expert’s demonstrations. In contrast, our method ORPO is focused on *improving* beyond a safe policy (e.g., one trained via imitation learning). Because ORPO combines occupancy measure divergence and reward objectives, it can both outperform the safe policy and avoid reward hacking. We will include a detailed comparison to GAIL and other prior works that use occupancy measure divergences in the final paper if accepted.

Reviewer comment (Theory):
Please justify the new findings in the theoretical analysis that differs from previous works.

Author response:
* **Novelty of theoretical analysis:** We also acknowledge that there has been previous work that has focused on bounding, e.g., the difference in value functions between policies whose occupancy measure divergence is bounded [2]—this is quite similar to our Proposition 3.2. However, we believe our Proposition 3.1 is novel. It shows that arbitrarily small changes in action distributions between policies can cause arbitrarily large differences in return, and vice versa: arbitrarily large changes in action distributions between policies can cause arbitrarily small differences in return. This result in particular highlights the challenges of using action distribution KL for regularizing RL to avoid reward hacking, and to the best of our knowledge has not appeared in the literature before. We will be more clear about which of our theoretical results are similar to those which have appeared in the literature before and which are novel in the final version of the paper.

Reviewer comment (Experiments):
Perform more realistic tasks such as LLM RLHF tasks and complex robotic manipulation/locomotion.

Author response:
**Note about Experiments on LLM RLHF:** The reviewer also suggested we perform experiments on regularizing RL as part of RLHF for large language models (LLMs). In this case, action distribution-based policy regularization is widely used [9]. It turns out that for LLMs, action distribution regularization is actually *equivalent* to occupancy measure regularization. The occupancy measure $\mu$ of an LLM's state $s$, the response generated until timestep $t$ using words $w_1, w_2,...w_t$, can be written in terms of the autoregressive model $\pi$'s predictions:
   $$\log \mu(s) = \log \pi(w_t|w_{t-1},...w_1) + \log \pi(w_{t-1}|w_{t-2},...w_1) + \log \pi(w_1)$$

   Here, the next word given the previous sequence is the action $a$ to be taken. Thus, the KL divergence $D_\text{KL}$ between the SFT model $\pi_{safe}$ and the RLHF model $\pi$ and the KL divergence between their corresponding occupancy-measures, $\mu_\text{safe}$ and $\mu$ are roughly equivalent:
   $$D_\text{KL}(\mu \| \mu_\text{safe}) = \sum_{s,a}\mu(s,a) * \sum_t [\log\mu(s,a) - \log\mu_\text{safe}(s,a)]$$
   Combining the information from the two equations above, we get the desired result:
   $$D_\text{KL}(\mu \| \mu_\text{safe}) = D_\text{KL}(\pi \mid \pi_\text{safe})$$
   Therefore, while we show that our method is very valuable for regularizing the behavior of agents trained with reinforcement learning in general, for LLMs in particular it is equivalent to the already widespread practice of action distribution regularization. This is why we do not evaluate ORPO for RLHF of LLMs. We will include these results showing why ORPO is not necessarily applicable to LLM training in the final version of the paper.