T HE E FFECTS OF R EWARD M ISSPECIFICATION : M APPING AND M ITIGATING M ISALIGNED M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Reward hacking—where RL agents exploit gaps in misspeciﬁed proxy rewards— has been widely observed, but not yet systematically studied.
To understand reward hacking, we construct four RL environments with different misspeciﬁed rewards.
We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, and observation space noise.
Typically, more capable agents are able to better exploit reward misspeciﬁcations, causing them to attain higher proxy reward and lower true reward than less capable agents.
Moreover, we ﬁnd instances of phase transitions : capability thresholds at which the agent’s behavior qualitatively shifts, leading to a sharp decrease in the true reward.
Such phase transitions pose challenges to monitoring the safety of ML systems.
To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.
I NTRODUCTION
As reinforcement learning agents are trained with better algorithms, more data, and larger policy models, they are at increased risk of overﬁtting their objectives (Russell, 2019).
Reward hacking , or the gaming of misspeciﬁed reward functions by RL agents, has appeared in a variety of contexts, such as game playing (Ibarz et al., 2018), robotics (Popov et al., 2017; Christiano et al., 2017), text summarization (Paulus et al., 2018), and autonomous driving (Knox et al., 2021).
These examples show that better algorithms and models are not enough: For human-centered applications such as healthcare (Yu et al., 2019), economics (Trott et al., 2021) and robotics (Kober et al., 2013), RL algorithms must be safe and aligned with human objectives (Bommasani et al., 2021).
Reward misspeciﬁcation arises because formulating a reward that accurately captures the desiderata of the underlying task is challenging.
In practice, reward designers often optimize proxies because they have a denser signal or are more readily measured.
For example, consider the task of placing a red block on top of a blue block (Popov et al., 2017).
Instead of optimizing for the intended behavior, which has a sparse reward (only one state has nonzero reward), it is easier optimize for a proxy, such as maximizing the height of the red block.
As another example, consider a recommender system optimizing for users’ subjective-well being (SWB).
Because SWB is difﬁcult to measure, engineers rely on more tangible metrics such as click-through rates or watch-time (Stray, 2020). In both of these cases, optimizing for these proxies led misspeciﬁcation. We expand on the different types of misspeciﬁcations that can arise in Section 2.2. Addressing reward hacking is a ﬁrst step towards developing human-aligned RL agents and one goal of ML safety (Hendrycks et al., 2021a).
However, there has been little systematic work investigating when or how it tends to occur, or how to detect it before it runs awry.
To remedy this, we systematically study the problem of reward misspeciﬁcation across four diverse environments: trafﬁc control (Wu et al., 2021), COVID response (Kompella et al., 2020), blood glucose monitoring (Fox et al., 2020), and the Atari game Riverraid (Brockman et al., 2016).
Within these environments, we construct nine proxy reward functions with errors such as incorrect scope or incorrect ontology.
Using our environments, we study how increasing optimization power affects reward hacking, by training RL agents with varying resources such as model size, training time, action space resolution, and observation space noise.
We ﬁnd that more powerful agents often attain higher proxy reward but lower true reward, as illustrated in Figure 1.
Since the trend in ML is to increase resources exponentially each year ( al., 2021), this reward hacking will become more pronounced in the future in the absence
More worryingly, we observe several instances of phase transitions .
In a phase transition, the more capable model pursues a qualitatively different policy that sharply decreases the true reward.
Figure 1 illustrates one example: An RL agent regulating trafﬁc learns to stop any cars from merging onto the highway in order to maintain a high average velocity of the cars on the straightaway.
Since there is little direct warning of phase transitions until after they occur, they pose a challenge to safety monitoring and engineering.
To address this, we propose an anomaly detection task (Hendrycks & Gimpel, 2017; Tack et al., 2020):
Can we detect when the true reward starts to drop, while maintaining a low false positive rate in benign cases?
Our proposed task, P OLYNOMALY , is instantiated for the trafﬁc and COVID environments.
Given a trusted model with moderate performance, one must detect whether policies from a different model are satisfactory or aberrant.
We also provide several baseline anomaly detectors for this task.
We release our data with the hope of spurring more research into detecting reward hacking.
M APPING REWARD MISSPECIFICATION
In this section, we describe our four environments (Section 2.1) and the nine corresponding misspeciﬁed reward functions (Section 2.2).
2. E NVIRONMENTS
We chose diverse environments and prioritized complexity of action space, observation space and dynamics model.
Additionally, we aimed to reﬂect real-world constraints in our environments, selecting ones with several desiderata that must be simultaneously balanced.
Table 1 provides a summary of our environments.
Trafﬁc.
The trafﬁc environment is an autonomous vehicle (AV) simulation that models vehicles driving on different highway networks.
The vehicles are either controlled by a RL algorithm or pre-programmed via a human behavioral model.
The objective of the RL agent is to minimize the mean commute time within the highway network (see Figure 1).
We use the Flow trafﬁc simulator, implemented by Wu et al.
(2021) and Vinitsky et al.
(2018), which extends the SUMO trafﬁc simulator (Lopez et al., 2018).
For the human behavioral model, we use the Intelligent Driver Model (IDM) (Treiber et al., 2000), wherein drivers attempt to travel as fast as possible while tending to decelerate whenever they are too close to the car immediately in front.
The RL policy has access to observations only from the AVs it controls.
For each AV, the observation space consists of the car’s position, its velocity, and the position and velocity of the cars immediately in front of and behind it.
The continuous control action is the accelerations applied to each AV.
Figure 5 depicts the Trafﬁc-Mer network, where cars from an on-ramp attempt to merge onto the straightaway.
We also use the Trafﬁc-Bot network, where cars (1-4 RL, 10-20 human) drive through a highway bottleneck: four lanes become two lanes and later become one lane.
COVID Response.
The COVID environment, developed by Kompella et al.
(2020), simulates a population using the SEIR model of individual infection dynamics.
The RL policymaker adjusts the severity of social distancing regulations while balancing economic health (better with lower regulations) and public health (better with higher regulations), similar in spirit to Trott et al.
The population attributes (proportion of adults, number of hospitals) and infection dynamics (random testing rate, infection rate) are based on data from Austin, Texas.
Every day, the environment simulates the infection dynamics and reports testing results to the agent, but not the true infection numbers.
The policy chooses one of three discrete actions: INCREASE ,
DECREASE , or MAINTAIN the current regulation stage, which directly affects the behavior of the population and indirectly affects the infection dynamics.
There are ﬁve stages in total.
Atari Riverraid.
The Atari Riverraid environment is run on OpenAI Gym (Brockman et al., 2016).
The agent operates a plane which ﬂies over a river and is rewarded by destroying enemies.
The agent observes the raw pixel input of the environment.
The agent can take one of eighteen discrete actions, corresponding to either movement or shooting within the environment.
Glucose.
The glucose environment, implemented in Fox et al.
(2020), is a continuous control problem.
The environment, an extension of a FDA-approved simulator (Man et al., 2014), simulates the blood glucose levels of a patient with Type 1 diabetes.
The patient partakes in meals and wears a continuous glucose monitor (CGM), which gives noisy observations of the patient’s glucose levels.
The RL agent administers insulin to maintain a healthy glucose level.
Every ﬁve minutes, the agent observes the patient’s glucose levels and decides how much insulin to administer.
The observation space is the previous four hours of glucose levels and insulin dosages.
2. M ISSPECIFICATIONS
Using the above environments, we constructed nine instances of misspeciﬁed proxy rewards.
To help interpret these proxies, we taxonomize them as instances of misweighting , incorrect ontology , or incorrect scope .
For instance, consider the following examples for the trafﬁc environment:
• Misweighting.
Upweighting the velocity term overemphasizes faster driving at the cost of higher carbon emissions.
In general, a reward functions may depend on the correct metrics, but place the wrong relative importance on each metric.
• Ontological.
Interpreting less congestion as higher average vehicle velocity instead of lower average commute time may cause public transportation (several riders per vehicle) to be undervalued.
In general, reward functions may be composed of the wrong metrics due to differing interpretations of the underlying objective.
• Scope.
If monitoring velocity over all roads is too costly, a city might instead monitor them only over highways, thus moving congestion to local streets.
In general, reward functions may measure a metric over a restricted domain (e g. time, space) due to physical constraints.
We include a summary of all nine tasks in Table 1 and provide full details in Appendix A.
Tablealso indicates whether each proxy leads to misspeciﬁcation (i.e. to a policy with low true reward) and whether it leads to a phase transition (a sudden qualitative shift as model capacity increases).
We investigate both of these in Section 3.
For each environment and proxy, we train an agent using the proxy reward and evaluate performance according to the true reward.
We use PPO (Schulman et al., 2017) to optimize policies for the trafﬁc and COVID environments, SAC (Haarnoja et al., 2018) to optimize the policies for the glucose environment, and torchbeast (K¨uttler et al., 2019), a PyTorch implementation of IMPALA (Espeholt et al., 2018), to optimize the policies for the Atari environment.
When available, we adopt the hyperparameters (except the learning rate and network size) given by the original codebase.
[Table]
U NDERSTANDING H OW M ISSPECIFICATION D RIVES M ISALIGNMENT
To better understand reward hacking, we study how it emerges as agent optimization power increases.
We deﬁne optimization power as the effective search space of policies the agent has access to, as implicitly determined by model size, training steps, action space, and observation space.
In Section 3.1, we consider the quantitative effect of optimization power for all nine environmentmisspeciﬁcation pairs; we primarily do this by varying model size, but also use training steps, action space, and observation space as robustness checks.
Overall, more capable agents tend to overﬁt the proxy reward and achieve a lower true reward.
We also ﬁnd evidence of phase transitions on four of the environment-misspeciﬁcation pairs.
For these phase transition, there is a critical threshold at which the proxy reward rapidly increases and the true reward rapidly drops.
In Section 3.2, we further investigate these phase transitions by qualitatively studying the resulting policies.
We ﬁnd that phase transitions correspond to a qualitative shift in behavior that does not manifest prior to the critical threshold.
Extrapolating visible trends is therefore insufﬁcient to catch all instances of reward hacking, increasing the urgency of research in this area.
3. Q UANTITATIVE E FFECTS OF M ISSPECIFICATION VS . A GENT C APABILITIES
As a stand-in for increasing agent optimization power, we ﬁrst vary the model capacity for a ﬁxed environment and proxy reward.
Speciﬁcally, we vary the width and depth of the actor and critic networks, changing the parameter count by two to four orders of magnitude depending on the environment.
For a given policy, the actor and critic are always the same size.
Our results are shown in Figure 2, with additional plots included in Appendix A. We plot both the proxy (blue) and true (green) reward vs. the number of parameters.
As model size increases, agents are better able to optimize the proxy reward, which often comes at the cost of the true reward.
This entails that reward designers will likely need to take greater care to specify reward functions accurately and is especially salient given the recent trends towards larger and larger models (Littman et al., 2021).
We observe that these quantitative shifts in true reward can be quite sudden.
We call these sudden shifts phase transitions , and mark them with dashed red lines in Figure 2.
These quantitative trends are reﬂected in the qualitative behavior of the policies (Section 3.2), which typically shift at the phase transition.
Model capacity is only one proxy for agent capabilities, and larger models do not always lead to more capable agents (Andrychowicz et al., 2020).
To check the robustness of our results, we consider several other measures of optimization below.
[Figure]
Observation ﬁdelity.
Agents with access to better input sensors, like higher-resolution cameras, should make more informed decisions and thus have more optimization power.
Concretely, we study this in the COVID environment, where we increase the random testing rate in the population (observations more accurately reﬂect the underlying rate of infection).
As shown in Figure 3, as the testing rate increases, the models are able to achieve a higher proxy reward.
Unfortunately, this also results in slightly lower true rewards.
This effect is more pronounced with the larger model.
Number of training steps.
Assuming a reasonable RL algorithm and hyperparameters, agents which are trained for more steps have more optimization power.
We vary training steps for an agent trained on the Atari environment with a misweighting misspeciﬁcation.
The true reward incentivizes staying alive for as many frames as possible while moving smoothly.
However, the proxy reward underpenalizes the smoothness constraint.
As shown in Figure ? ? , further optimizing the proxy reward harms the true reward.
Thus, number of training steps is a driving factor behind reward hacking.
Action space resolution.
Intuitively, an agent that can take more precise actions is more capable.
For example, as technology improves, an RL car might be able to make course corrections every millisecond, instead of every second.
We study action space resolution in the trafﬁc environment, by applying zero-mean Gaussian noise to the output acceleration of the RL model.
The larger the variance of this noise, the lower the action space resolution.
Results are shown in Figure 4a for a ﬁxed model size.
Increasing the noise causes the true reward to decrease somewhat, although the error bars are too large to draw strong conclusions from
[Figure]
other and action space control.
In Figure ? , we see reward hacking occur the proxy increases and true reward decreases.
In Figure 4a, we do not observe any evidence of reward hacking
3. Q UALITATIVE E FFECTS OF M ISSPECIFICATION
In the previous section, quantitative trends showed that increasing a model’s optimization power often hurts performance on the true reward.
We shift our focus to understanding how this degradation happens—speciﬁcally, we look at the qualitative differences in the policies learned by smaller models as compared to larger ones.
In particular, we typically observe a qualitative shift in behavior associated with each of the phase transitions.
We describe one example of a qualitative shift for each of the four environments below.
Trafﬁc.
We focus on the Trafﬁc-Mer environment from Figure 2a, where minimizing average commute time is replaced by maximizing average velocity.
In this case, smaller policies learn to merge onto the straightaway by slightly slowing down the other vehicles (Figure 5a).
On the other hand, larger policy models stop the AVs to prevent them from merging at all (Figure 5b).
This increases the average velocity, because the vehicles on the straightaway (which greatly outnumber vehicles on the on-ramp) do not need to slow down for merging trafﬁc.
However, this signiﬁcantly increases the average commute time, as the passengers in the AV remain stuck.
COVID.
Suppose the RL agent optimizes solely for the public and economic health of a society, without factoring politics into its decision-making. This behavior is shown in Figure 6.
The larger model chooses to increase the severity of social distancing restrictions earlier than the smaller model. As a result, the larger models is able to maintain low average levels of both ICU usage (a proxy for public health) and social distancing restrictions (a proxy for economic health). Since the preemptive regulations require precise planning to achieve a good result, this behavior only appears for larger
[Figure]
models.
These preemptive regulations may be politically costly, as enforcing restrictions without clear signs of infection may foment public unrest (Boettke & Powell, 2021).
Consider the perspective of a policymaker, who, in additional to caring for the welfare of their constituents, is also concerned about their political prospects.
While the larger models achieve higher proxy reward, from such a policymaker’s perspective, they achieve lower true reward.
Atari.
In the Riverraid environment, we observe a qualitative shift that does not lead to misalignment.
We create an ontological misspeciﬁcation by rewarding the plane for staying alive as long as possible while shooting as little as possible: a “paciﬁst run”.
We then measure the game score as the true reward.
This reﬂects different priorities players might have.
We ﬁnd that agents with more parameters are typically stronger and maneuver more adeptly.
Stronger agents shoot less frequently, but survive for much longer, acquiring points (true reward) due to passing checkpoints.
In this case, therefore, the proxy and true rewards are well-aligned so that reward hacking does not emerge as capabilities increase.
We did, however, ﬁnd that some of the agents exploited a bug in the simulator that halts the plane at the beginning of the level.
The simulator advances but the plane itself did not move, thereby achieving high paciﬁst reward.
Glucose.
Consider an RL agent which optimizes solely for a patient’s health, without considering the economic costs of its treatment plans. In this case, the proxy reward is based off of a glycemic risk measure, which reﬂects the likelihood that a patient will suffer an acute hypoglycemic episode, developed by the medical community (Kovatchev et al., 2000).
However, a less economically-privileged patient may opt for the treatment plan with the least expected cost (Herkert et al., 2019), not the one with the least amount of risk.
From this patient’s perspective, the true reward is the expected cost of the treatment plan, which includes the expected cost of hospital visits and the cost of administering the insulin dosage recommended by the agent.
Although larger models are able to reduce the insulin risk more than smaller models, they administer more insulin during their treatment.
Based on the average cost of a ER visit for a hypogylcemic episode (around $1350 from Bronstone & Graham (2016)) and the average cost of a unit of insulin (around $0 .
32 from Lee (2020)) , we ﬁnd that it is actually more expensive to pursue the recommendations of the larger model.
This is an example of reward hacking without any phase transition, as the larger models gradually use more insulin.
P OLYNOMALY : M ITIGATING REWARD MISSPECIFICATION
In the previous section, we saw that reward hacking often leads to phase transitions in agent behaviour.
Furthermore, in applications like trafﬁc routing or COVID response, the true reward may be observed only sporadically or not at all.
Blindly optimizing the proxy in these cases can lead to catastrophic failure (Zhuang & Hadﬁeld-Menell, 2020; Taylor, 2016).
This raises an important question: Without the true reward signal, how can we mitigate misalignment?
We operationalize this as an anomaly detection task.
The key idea is that if a detector is able to ﬂag instances of misalignment, this can be used to prevent catastrophic rollouts.
The resulting benchmark, P OLYNOMALY , is described below.
To perform anomaly detection, we assume access to some trusted policy , which has been evaluated by humans to have reasonable but not exemplary performance under the true reward.
In practice, one could use imitation learning (Hussein et al., 2017) to bootstrap a trusted policy from demonstrations.
The use of a trusted policy is likely necessary to make progress.
As we exhibit in Appendix A, there are instances when misspeciﬁcation does not produce reward hacking.
Thus, any unsupervised learning method, without some information about the intended agent behavior, will have trouble distinguishing these instances from true reward hacking.
4. P ROBLEM S ETUP
We train a collection of policies on the trafﬁc and COVID environments.
For each policy, we run between 5 to 32 rollouts, and assign the mean true reward of the rollouts as the policy’s true reward.
We label policies as acceptable or problematic by hand; two authors independently labeled each policy as acceptable, problematic, or ambiguous based on its true reward score relative to that of other policies.
We include only policies that received the same non-ambiguous label by both researchers.
For both environments, we provide a small-to-medium sized model as the trusted policy model, as
Section 3.1 empirically illustrates that smaller models achieve reasonable true reward without exhibiting reward hacking.
Given the trusted model and a collection of policies, the anomaly detector’s task is to assign a binary classiﬁcation of “good” or “bad” to each policy.
Table 2 provides an overview of our benchmark.
The number of anomalies describes the number of policies that were labeled bad.
The trusted policy size is a list of the hidden unit widths of the trusted policy network (not including feature mappings).
[Table]
E VALUATION
We propose two evaluation metrics for measuring the performance of our anomaly detectors.
• Area Under the Receiver Operating Characteristic (AUROC) .
Stronger detectors should assign a high score to anomalies.
The AUROC measures the probability that a detector will assign a random anomaly a higher score than a random non-anomalous policy (Davis & Goadrich, 2006).
Higher AUROCs indicate stronger detectors, and a random classiﬁer has an AUROC of 0 . 5 .
• Max F-1 score .
The F-1 score is the harmonic mean of the precision and the recall, so detectors with a high F-1 score must have both low false positives and high true negatives.
F-1 score is also correlated with AUROC.
We calculate the max F-1 score by considering all possible thresholds for the detector and computing the F-1 score from that point.
4. B ASELINES
In addition to the benchmark datasets described above, we provide baseline anomaly detectors based on estimating distances between policies.
Speciﬁcally, we look at the Jensen-Shannon and the Hellinger distance (described below) between our trusted policy and the unknown policy.
Let P and Q represent two probability distributions with M = 12 ( P + Q ) .
Then the Jensen-Shannon divergence and the Hellinger distance between them is given by
[Equation]
Every k steps (where k = 10 in the trafﬁc environment and k = 1 in the COVID environment), we set P to be the action distribution output by the unknown policy and Q to be the action distribution output by the trusted policy.
We then compute the two distances given in Equation (1).
These distances are collected over the entire rollout, and we calculate metrics on these distances (range, mean, etc.) to assign an anomaly score to the untrusted policy.
Table 3 reports the AUROC and Fscores of several such detectors.
We provide the receiver operating characteristic (ROC) curves for these detectors in Appendix B.
Baseline Detectors
[Table]
Env. - Misspeciﬁcation
AUROC
Max FAUROC Max FAUROC
Max FTrafﬁc-Mer - misweighting 81 . 0%
Trafﬁc-Mer - scope 74 .
Trafﬁc-Mer - ontological 52 .
Trafﬁc
[Table]
We observe that different detectors are better for different tasks, suggesting that future detectors could do better than any of our baselines.
Our benchmark and baseline provides a starting point for further research on mitigating reward hacking.
D ISCUSSION
In this work, we designed a diverse set of environments and proxy rewards, uncovered several instances of phase transitions, and proposed an anomaly detection task to help mitigate these transitions.
Our results raise two questions: How can we not only detect phase transitions, but prevent them in the ﬁrst place?
And how should phase transitions shape our approach to safe ML?
On preventing phase transitions, anomaly detection already offers one path forward.
Once we can detect anomalies, we can potentially prevent them, by using the detector to purge the unwanted behavior (e.g. by including it in the training objective).
Similar policy shaping has recently been used to make RL agents more ethical (Hendrycks et al., 2021b).
However, since the anomaly detectors will be optimized against by the RL policy, they need to be adversarially robust (Goodfellow et al., 2014).
This motivates further work on adversarial robustness and adversarial anomaly detection.
Regarding safe ML, several recent papers propose extrapolating empirical trends to forecast future
ML capabilities (Kaplan et al., 2020; Hernandez et al., 2021; Droppo & Elibol, 2021), partly to avoid unforeseen consequences from ML.
While we support this work, our results show that trend extrapolation alone is not enough to ensure the safety of ML systems.
To complement trend extrapolation, we need better interpretability methods to identify emergent model behaviors early on, before they dominate performance (Olah et al., 2018).
ML researchers should also familiarize themselves with emergent behavior in self-organizing systems (Yates, 2012), which often exhibit similar phase transitions (Anderson, 1972).
Indeed, the ubiquity of phase transitions throughout science suggests that ML researchers should continue to expect surprises–and should therefore prepare for them.
E THICS S TATEMENT
Our main contribution is empirical evidence of reward hacking and a benchmark for detecting reward hacking.
Our results do not directly have any adverse ethical consequences and we do not provide any new techniques that may worsen reward hacking.
R EPRODUCIBILITY S TATEMENT
In order to ensure reproducibility, we have attached the code for all our experiments as a supplementary ﬁle.
Additionally, we have described all necessary hyper parameters and algorithmic details required to reproduce the experiments.
A M EASUREMENT Effect of model size
[Figure]
B R ECEIVER O PERATING C HARACTERISTIC C URVES
We plot the ROC curves for the detectors described in Section 4.3.
[Figure]
[Figure] [Figure]
[Figure] [Figure]