Original: A Study of Causal Confusion in Preference-Based Reward Learning
Revised: A Study of Causal Confusion in Preference-Based Reward Learning

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: While there is much empirical and theoretical analysis of causal confusion and reward gaming behaviors in reinforcement learning and behavioral cloning approaches, we provide the first systematic study of causal confusion in the context of learning reward functions from preferences.
Revised: While there is much empirical and theoretical analysis of causal confusion and reward gaming behaviors in reinforcement learning and behavioral cloning approaches, we provide the first systematic study of causal confusion in the context of learning reward functions from preferences.

Original: We identify a set of three benchmark domains where we observe causal confusion when learning reward functions from offline datasets of pairwise trajectory preferences: a simple reacher domain, an assistive feeding domain, and an itchscratching domain.
Revised: We identify a set of three benchmark domains where we observe causal confusion when learning reward functions from offline datasets of pairwise trajectory preferences: a simple reacher domain, an assistive feeding domain, and an itch-scratching domain.

Original: To gain insight into this observed causal confusion, we perform a sensitivity analysis on the effect of different factors—the reward model capacity and feature dimensionality— on the robustness of rewards learned from preferences.
Revised: To gain insight into this observed causal confusion, we perform a sensitivity analysis on the effect of different factors—the reward model capacity and feature dimensionality—on the robustness of rewards learned from preferences.

Original: We find evidence that learning rewards from preferences is highly sensitive and nonrobust to spurious features and increasing model capacity.
Revised: We find evidence that learning rewards from preferences is highly sensitive and non-robust to spurious features and increasing model capacity.

Revised: Videos, code, and supplemental results are available at https://sites.google.com/ view/causal-reward-confusion .

Original: 1. Introduction
Revised: 1. Introduction

Original: Preference-based reward learning (Wirth et al., 2017; Sadigh et al., 2017; Christiano et al., 2017;
Revised: Preference-based reward learning (Wirth et al., 2017; Sadigh et al., 2017; Christiano et al., 2017;

Original: Brown et al., 2020a) is a well-studied technique for learning from pairwise preferences or rankings.
Revised: Brown et al., 2020a) is a well-studied technique for learning from pairwise preferences or rankings.

Original: and holds the potential of allowing AI systems to learn specifications for tasks without requiring a human to write down an explicit reward function and to adapt the AI system’s behavior to individual preferences and needs However, recent anecdotal evidence suggests that these methods are prone to learning rewards that overfit to spurious correlations in the data, especially when learning from limited numbers of preferences (Christiano et al., 2017; Ibarz et al., 2018; Javed et al., 2021). While the effects of reward misspecification have recently been studied in the context of reinforcement learning agents that optimize a 1 Anonymous Institution, Anonymous City, Anonymous Region,{+ICML 
+}
Revised: and holds the potential of allowing AI systems to learn specifications for tasks without requiring a human to write down an explicit reward function and to adapt the AI system’s behavior to individual preferences at the {+ICML 
+}

Original: Anonymous Country.
Revised: Workshop USA.

Original: Correspondence to: Anonymous Author < anon.email@domain.com > . Preliminary work. Under review by the SCIS workshop.
Revised: Copyright 2022 by the author(s).

Original: [Figure]
Revised: [Figure]

Original:  proxy reward function (Pan et al., 2022) and behavioral cloning approaches that directly mimic an expert (De Haan et al., 2019; Zhang et al., 2020; Swamy et al., 2022), there has not been a systematic study of causal confusion when learning reward functions.
Revised: environments in which performing reward function learning from pairwise trajectory preferences leads to causal confusion over the true reward. and needs However, recent anecdotal evidence suggests that these methods are prone to learning rewards that overfit to spurious correlations in the data, especially when learning from limited numbers of preferences (Christiano et al., 2017; Ibarz et al., 2018; Javed et al., 2021). While the effects of reward misspecification have recently been studied in the context of reinforcement learning agents that optimize a proxy reward function (Pan et al., 2022) and behavioral cloning approaches that directly mimic an expert (De Haan et al., 2019; Zhang et al., 2020; Swamy et al., 2022), there has not been a systematic study of causal confusion when learning reward functions.

Original: As an example of the type of causal confusion we study in this paper, consider the assistive feeding task in Figure 1b.
Revised: As an example of the type of causal confusion we study in this paper, consider the assistive feeding task in Figure 1b.

Original: Note that successful robot executions will cause the spoon to make contact with the patient’s mouth, applying small amounts of force, whereas failed executions may not make any contact.
Revised: Note that successful robot executions will cause the spoon to make contact with the patient’s mouth, applying small amounts of force, whereas failed executions may not make any contact.

Original: We find that a preference-based learning approach will often miss the correct causal relationship between “contact” and “feeding”, leading the robot to optimize a policy that seeks any kind of contact with the patient, including contact with the patient’s torso or head.
Revised: We find that a preference-based learning approach will often miss the correct causal relationship between “contact” and “feeding”, leading the robot to optimize a policy that seeks any kind of contact with the patient, including contact with the patient’s torso or head.

Original: We provide the following contributions: (1) We identify a set of 3 robot preference learning benchmarks that exhibit causal confusion when learning reward functions from preferences.
Revised: We provide the following contributions: (1) We identify a set of 3 robot preference learning benchmarks that exhibit causal confusion when learning reward functions from preferences.

Original: (2) We perform the first systematic study of causal confusion in preference-based reward learning by varying the feature-space dimensionality and the capacity of the reward function model.
Revised: (2) We perform the first systematic study of causal confusion in preference-based reward learning by varying the feature-space dimensionality and the capacity of the reward function model.

Original: (3) We identify spurious features and model capacity as the main sources of causal confusion when learning rewards from pairwise trajectory preferences.
Revised: (3) We identify spurious features and model capacity as the main sources of causal confusion when learning rewards from pairwise trajectory preferences.

Original: 2. Reward Learning from Preferences
Revised: 2. Reward Learning from Preferences

Original: We model the environment as a finite horizon MDP (Puterman, 2014), with state space S , action space A , horizon
Revised: We model the environment as a finite horizon MDP (Puterman, 2014), with state space S , action space A , horizon

Original: T , and reward function r : S ×
Revised: T , and reward function r : S ×

Original: A → R .
Revised: A → R .

Original: We assume that the reward function is unobserved and must be learned.
Revised: We assume that the reward function is unobserved and must be learned.

Original: The reward function is learned from preferences over trajectories where, using the popular Bradley-Terry model (Bradley &
Revised: The reward function is learned from preferences over trajectories where, using the popular Bradley-Terry model (Bradley &

Original: Terry, 1952), the probability a trajectory, τ B , is preferred over another trajectory, τ A , is given by
Revised: Terry, 1952), the probability a trajectory, τ B , is preferred over another trajectory, τ A , is given by

Original: [Equation]
Revised: [Equation]

Original: where r ( τ ) =
Revised: where r ( τ ) =

Original: (cid:80) ( s,a ) ∈
Revised: (cid:80) ( s,a ) ∈

Original: τ r ( s, a ) and where we define a trajectory, τ , by a sequence of state-action pairs: τ = ( s 0 , a 0 , . . .
Revised: τ r ( s, a ) and where we define a trajectory, τ , by a sequence of state-action pairs: τ = ( s 0 , a 0 , . . .

Original: , s T , a T ) .
Revised: , s T , a T ) .

Original: To learn a reward function from preferences, we assume access to a set of pair-wise preference labels, P , over trajectories, τ 1 , . .
Revised: To learn a reward function from preferences, we assume access to a set of pair-wise preference labels, P , over trajectories, τ 1 , . .

Original: . , τ N , where ( i, j ) ∈ P implies that τ i ≺ τ j .
Revised: . , τ N , where ( i, j ) ∈ P implies that τ i ≺ τ j .

Original: We then optimize a reward function r θ : S ×
Revised: We then optimize a reward function r θ : S ×

Original: A → R , parameterized by θ that maximizes the likelihood:
Revised: A → R , parameterized by θ that maximizes the likelihood:

Original: [Equation]
Revised: [Equation]

Original: 3. Environments for Preference Learning
Revised: 3. Environments for Preference Learning

Original: We identify a set of robot learning benchmarks that exhibit causal confusion when learning reward functions from preferences.
Revised: We identify a set of robot learning benchmarks that exhibit causal confusion when learning reward functions from preferences.

Original: In Reacher (Brockman et al., 2016)
Revised: In Reacher (Brockman et al., 2016)

Original: (Figure 1a) the goal is to move an end effector to a desired goal location.
Revised: (Figure 1a) the goal is to move an end effector to a desired goal location.

Original: In Feeding (Erickson et al., 2020) (Figure 1b)
Revised: In Feeding (Erickson et al., 2020) (Figure 1b)

Original: the goal is to feed the human using a spoon carrying pieces of food.
Revised: the goal is to feed the human using a spoon carrying pieces of food.

Original: Finally, in Itch Scratching (Erickson et al., 2020) (Figure 1c) the goal is to repeatedly scratch a desired itch location on the human’s arm.
Revised: Finally, in Itch Scratching (Erickson et al., 2020) (Figure 1c) the goal is to repeatedly scratch a desired itch location on the human’s arm.

Original: Each domain has a predefined “true” reward function r .
Revised: Each domain has a predefined “true” reward function r .

Original: This enables us to create synthetic demonstrations and preference labels.
Revised: This enables us to create synthetic demonstrations and preference labels.

Original: Note that while we use the ground-truth reward function for obtaining preference labels ( r ( τ 1 ) <
Revised: Note that while we use the ground-truth reward function for obtaining preference labels ( r ( τ 1 ) <

Original: r ( τ 2 ) =
Revised: r ( τ 2 ) =

Original: ⇒ τ 1 ≺ τ 2 ), we assume no access to this reward function during policy learning, but rather seek to learn a policy that obeys a users preferences by first learning a model r θ of the true reward function from preference labels P and then running RL on the learned reward function.
Revised: ⇒ τ 1 ≺ τ 2 ), we assume no access to this reward function during policy learning, but rather seek to learn a policy that obeys a users preferences by first learning a model r θ of the true reward function from preference labels P and then running RL on the learned reward function.

Original: We can then evaluate the learned policy on the true reward function r .
Revised: We can then evaluate the learned policy on the true reward function r .

Original: To facilitate reproducibility and encourage future research on causal reward confusion, we will open-source our code and training datasets.
Revised: To facilitate reproducibility and encourage future research on causal reward confusion, we have open-sourced our code and training datasets.

Original: This combination of domains and training data forms the first set of benchmarks for studying causal confusion when learning reward functions.
Revised: This combination of domains and training data forms the first set of benchmarks for studying causal confusion when learning reward functions.

Revised: Links to download and install source code for the domains and links to download the preference training data used in our experiments are available at https://sites.google.

Revised: com/view/causal-reward-confusion .

Original: 4. Evidence of Causal Confusion
Revised: 4. Evidence of Causal Confusion

Original: To demonstrate that each of these domains exhibits causal confusion, we show that learning a reward function from preferences followed by policy optimization using the learned reward leads to behavior that performs poorly under P REF . A CC . [Table]
Revised: To demonstrate that each of these domains exhibits causal confusion, we show that learning a reward function from preferences followed by policy optimization using the learned reward leads to behavior that performs poorly under the true reward function (unobserved, but used to provide synthetic preferences).

Original: T RAIN

Original: V AL [Table] the true reward function (unobserved, but used to provide synthetic preferences). Synthetic Preference Generation To enhance scalability and reproducibility, we generate a large number of trajectory preferences by using a pretrained RL policy (trained using the true reward provided with each environment) and then injecting noise into the policy.
Revised: Synthetic Preference Generation To enhance scalability and reproducibility, we generate a large number of trajectory preferences by using a pretrained RL policy (trained using the true reward provided with each environment) and then injecting noise into the policy.

Original: See Appendix C for details.
Revised: See Appendix C for details.

Original: Causal Confusion Results In Table 1, we show both the  pairwise classification accuracy on the train and validation sets as well as the subsequent performance of the resulting learned policies after training on 2,000 pairwise preference labels.
Revised: Causal Confusion Results In Table 1, we show both the learned reward’s pairwise classification accuracy on the train, validation, and test sets as well as the subsequent performance of the resulting learned policies.

Original: Surprisingly, we find that despite having high validation accuracy on distinguishing between better or worse trajectories, the learned reward functions do not lead to correct behavior in the learned policy (as indicated by the low cumulative reward values), even when given large amounts of pairwise preferences and when the preferences can be perfectly inferred from the input features (ie., they are fullyobservable ).
Revised: Surprisingly, we find that despite having high test accuracy on distinguishing between better or worse trajectories, the learned reward functions do not lead to correct behavior in the learned policy (as indicated by the low cumulative reward values), even when given large amounts of pairwise preferences and when the preferences can be perfectly inferred from the input features (ie., they are fully-observable ).

Original: In addition to this, we observe that the learned reward function consistently assigns higher rewards to the policy trained from preference learning rather than the policy trained on the ground truth reward; these results provide strong empirical evidence of causal confusion.
Revised: In addition to this, we observe that the learned reward function consistently assigns higher rewards to the policy trained from preference learning rather than the policy trained on the ground truth reward; these results provide strong empirical evidence of causal confusion.

Original: Figuredisplays examples of the learned policies.
Revised: Figure 5 displays examples of the learned policies.

Revised: See the supplementary website for videos of these and other examples.

Original: 5. Factors that May Lead to Causal Confusion
Revised: 5. Factors that May Lead to Causal Confusion

Original: Our results in the previous section demonstrate strong evidence of causal confusion during reward learning.In partic- [Table]
Revised: Our results in the previous section demonstrate strong evidence of causal confusion during reward learning.

Original: [Table] [Table] [Table] ular, we found that preference-based reward learning fails even when using an expressive neural network to learn the reward and even when given large numbers of pairwise preferences—for comparison, prior work showed successful reward learning for Atari games and MuJoCo locomotion tasks using less than 300 pairwise preferences (Brown et al., 2019).
Revised: In particular, we found that preference-based reward learning fails even when using an expressive neural network to learn the reward and even when given large numbers of pairwise preferences—for comparison, prior work showed successful reward learning for Atari games and MuJoCo locomotion tasks using less than 300 pairwise preferences (Brown et al., 2019).

Original: In the remainder of this paper, we systematically vary different aspects of preference-based reward learning to gain insight into the following questions: How does (1) the choice of observation feature space and (2) the model capacity of the reward-learning network affect causal confusion?
Revised: In the remainder of this paper, we systematically vary different aspects of preference-based reward learning to gain insight into the following questions: How does (1) the choice of observation feature space and (2) the model capacity of the reward-learning network affect causal confusion?

Original: See Appendix F for an analysis of training data’s effects on causal confusion. 
Revised: See Appendix F for an analysis of training data’s effects on causal confusion. [Table] [Figure]

Original: 5.1. The Choice of Observation Feature Space
Revised: 5.1. The Choice of Observation Feature Space

Original: We study reward learning when hand-crafted (causal) features are combined with raw observation features.
Revised: We study reward learning when hand-crafted (causal) features are combined with raw observation features.

Original: Surprisingly, the concatenation of the two—despite having strictly more information—leads to worse reward learning results than using hand-crafted features alone.
Revised: Surprisingly, the concatenation of the two—despite having strictly more information—leads to worse reward learning results than using hand-crafted features alone.

Original: This type of causal misidentification mirrors previous results shown in the context of behavioral cloning (De Haan et al., 2019).
Revised: This type of causal misidentification mirrors previous results shown in the context of behavioral cloning (De Haan et al., 2019).

Original: Hand-crafted Privileged Features:
Revised: Hand-crafted Privileged Features:

Original: Based on the results in Section 4, preference-learning does not work well on the given observation spaces of our three benchmark domains.
Revised: Based on the results in Section 4, preference-learning does not work well on the given observation spaces of our three benchmark domains.

Original: Thus, we use a set of hand-designed “privileged” features, as defined in Table 2, that directly correspond to the ground truth reward.
Revised: Thus, we use a set of hand-designed “privileged” features, as defined in Table 2, that directly correspond to the ground truth reward.

Original: We then train a linear reward model on these features.
Revised: We then train a linear reward model on these features.

Original: We find that removing the bias incentivizes more desirable behavior.
Revised: We find that removing the bias incentivizes more desirable behavior.

Original: For more details, see Appendix D.
Revised: For more details, see Appendix D.

Original: Given only causal features for the ground truth reward, preference learning is generally able to successfully learn weights over these features, leading to a good RL policy.
Revised: Given only causal features for the ground truth reward, preference learning is generally able to successfully learn weights over these features, leading to a good RL policy.

Original: We found that Feeding achieves a mean cumulative reward of  132.18 and task success rate of 99.7% , and Itch Scratching achieves a mean cumulative reward of 205.12 and task success rate of 90.7% .
Revised: We found that Feeding achieves a mean cumulative reward of [Table] 132.18 and task success rate of 99.7% , and Itch Scratching achieves a mean cumulative reward of 205.12 and task success rate of 90.7% .

Original: Reacher achieves a cumulative reward of − 18 .
Revised: Reacher achieves a cumulative reward of − 18 .

Original: 53 , which increases to -12.13 ( − 3 .
Revised: 53 , which increases to -12.13 ( − 3 .

Original: 75 being considered good) when preferences are made fully-observable.
Revised: 75 being considered good) when preferences are made fully-observable.

Original: Adding Spurious Features: We next concatenate these privileged causal features with the features from the raw observation space and perform reward learning on this augmented feature space.
Revised: Adding Spurious Features: We next concatenate these privileged causal features with the features from the raw observation space and perform reward learning on this augmented feature space.

Original: Figure 2 displays a sensitivity analysis on the number of raw features.
Revised: Figure 2 displays a sensitivity anal- [Figure] ysis on the number of raw features.

Original: Interestingly, we find that the ability of the agent to successfully complete its task drops off as more raw features are introduced, suggesting the existence of spurious correlations.
Revised: Interestingly, we find that the ability of the agent to successfully complete its task drops off as more raw features are introduced, suggesting the existence of spurious correlations.

Original: Note that the reward function still has access to the privileged features as we progressively add raw features; however, it appears that simply ignoring the concatenated raw features when learning the reward is difficult, even for a linear network, leading to causal confusion and poor policy performance.
Revised: Note that the reward function still has access to the privileged features as we progressively add raw features; however, it appears that simply ignoring the concatenated raw features when learning the reward is difficult, even for a linear network, leading to causal confusion and poor policy performance.

Original: We next inspected the weights of the learned linear reward models to understand the relative contributions of various features.
Revised: We next inspected the weights of the learned linear reward models to understand the relative contributions of various features.

Original: Interestingly, in both the Feeding and Itch Scratching environments, the feature that corresponds to the total amount of force applied by the spoon or scratching tool is consistently given a large weighting factor.
Revised: Interestingly, in both the Feeding and Itch Scratching environments, the feature that corresponds to the total amount of force applied by the spoon or scratching tool is consistently given a large weighting factor.

Original: This phenomenon is likely a result of the fact that all successful Feeding and Itch Scratching demonstrations applied some force on the human (touching the mouth or the arm), whereas the majority of failing demos simply dropped the food or swung the end effector without applying any force on the human.
Revised: This phenomenon is likely a result of the fact that all successful Feeding and Itch Scratching demonstrations applied some force on the human (touching the mouth or the arm), whereas the majority of failing demos simply dropped the food or swung the end effector without applying any force on the human.

Original: This proves to be a somewhat problematic proxy; as seen in Figure 3, rather than actually feed or scratch, the robot learns the undesirable behavior of applying a large amount of force to the human’s head to maximize the spoon force on human (Figure 3a) or applying a large amount of force on its own base to maximize total tool force (Figure 3b).
Revised: This proves to be a somewhat problematic proxy; as seen in Figure 3, rather than actually feed or scratch, the robot learns the undesirable behavior of applying a large amount of force to the human’s head to maximize the spoon force on human (Figure 3a) or applying a large amount of force on its own base to maximize total tool force (Figure 3b)

Original: Discussion: Our findings in this section demonstrate the need for caution when using raw observational feature spaces originally designed for RL tasks to perform reward learning.
Revised: Discussion: Our findings in this section demonstrate the need for caution when using raw observational feature spaces originally designed for RL tasks to perform reward learning.

Original: In the three tasks we consider, the raw observation spaces introduce spurious correlations that lead to a causally-confused learned reward and poor policy performance.
Revised: In the three tasks we consider, the raw observation spaces introduce spurious correlations that lead to a causally-confused learned reward and poor policy performance.

Original: Using hand-selected features works well, but designing the right set of features is difficult.
Revised: Using hand-selected features works well, but designing the right set of features is difficult.

Original: On top of that, we find evidence that increasing the number of features available to the reward network can exacerbate causal confusion, likely because it increases the probability that the  learned reward network focuses on spurious correlations.
Revised: On top of that, we find evidence that increasing the number of features available to the reward network can exacerbate causal confusion, likely because it increases the probability that the [Figure] learned reward network focuses on spurious correlations.

Original: 5.2. The Effect of Reward Function Model Capacity
Revised: 5.2. The Effect of Reward Function Model Capacity

Original: We next study the effect of the model capacity of the reward function, r θ .
Revised: We next study the effect of the model capacity of the reward function, r θ .

Original: Analyzing the effect of model capacity on causal confusion is inspired by recent results in the reinforcement learning setting, where Pan et al.
Revised: Analyzing the effect of model capacity on causal confusion is inspired by recent results in the reinforcement learning setting, where Pan et al.

Original: (Pan et al., 2022) find that increased model capacity for the policy network often increases the likelihood of reward hacking behavior.
Revised: (Pan et al., 2022) find that increased model capacity for the policy network often increases the likelihood of reward hacking behavior.

Original: In contrast, we study the model capacity of the reward-learning network and its effect on performance.
Revised: In contrast, we study the model capacity of the reward-learning network and its effect on performance.

Original: [Figure]

Original: [Figure]

Original: Increasing Model Capacity: See Appendix E for training details.
Revised: Increasing Model Capacity: See Appendix E for training details.

Original: We find that as the model capacity of the reward-learning network increases, the mean task success rate and cumulative ground truth reward the RL agent receives roughly decreases, all else being equal.
Revised: We find that as the model capacity of the reward-learning network increases, the mean task success rate and cumulative ground truth reward the RL agent receives roughly decreases, all else being equal.

Original: Figureshows agent performance in each environment as a function of reward-learning model capacity, trained with observations consisting of either purely privileged features or a mixture of raw and privileged features.
Revised: Figureshows agent performance in each environment as a function of reward-learning model capacity, trained with observations consisting of either purely privileged features or a mixture of raw and privileged features.

Original: Discussion: As expected, the models trained on just the privileged features perform better than those trained on the augmented features, but, notably, the larger models generally perform worse than the smaller models, especially in the augmented feature space, where raw features are present.
Revised: Discussion: As expected, the models trained on just the privileged features perform better than those trained on the augmented features, but, notably, the larger models generally perform worse than the smaller models, especially in the augmented feature space, where raw features are present.

Original: These findings go against the common belief of increasing model size to enable better generalization that is widely held in other machine learning domains (He et al., 2016;
Revised: These findings go against the common belief of increasing model size to enable better generalization that is widely held in other machine learning domains (He et al., 2016;

Original: Devlin et al., 2018; Brown et al., 2020b).
Revised: Devlin et al., 2018; Brown et al., 2020b).

Original: Our results instead suggest that the increased expressivity appears to increase the potential for inferring the wrong reward function, which the RL agent is then able to hack to a greater degree.[Figure] e a n c [Figure]
Revised: Our results instead suggest that the increased expressivity appears to increase the potential for inferring the wrong reward function, which the RL agent is then able to hack to a greater degree.

Original: u l a t i v e r e w a

Original: [Figure]

Original: 6. Conclusion
Revised: 6. Conclusion

Original: We provide the first systematic study of factors that may lead to causal confusion for reward learning.
Revised: We provide the first systematic study of factors that may lead to causal confusion for reward learning.

Original: In particular, we find that spurious features and increased model capacity lead to causal confusion over the true reward function, even when learning from thousands of pairwise preferences.
Revised: In particular, we find that spurious features and increased model capacity lead to causal confusion over the true reward function, even when learning from thousands of pairwise preferences.

Original: We hope that our empirical study will inspire and facilitate future work on learning reward functions that are robust to causal confusion, even when using high-capacity neural network reward function approximators.
Revised: We hope that our empirical study will inspire and facilitate future work on learning reward functions that are robust to causal confusion, even when using high-capacity neural network reward function approximators.

Revised: Finally, we note that our work has only studied causal confusion when learning reward functions from offline preference data.

Revised: Investigating how quickly and successfully active preference learning can ameliorate the types of causal confusion discussed in this paper is an important and interesting area of future work.

Original: A. Why study offline preference-based reward learning?
Revised: A. Why study offline preference-based reward learning?

Original: We note that, for the sake of systematically studying causal confusion, our focus in this paper is on learning reward functions from offline sets of trajectory preferences.
Revised: We note that, for the sake of systematically studying causal confusion, our focus in this paper is on learning reward functions from offline sets of trajectory preferences.

Original: We do this for three reasons.
Revised: We do this for three reasons.

Original: First, using offline trajectory preference data facilitates reproducibility by allowing us to create benchmarks (with fixed training datasets) for studying causal confusion in reward learning, similar to existing standard supervised learning benchmarks.
Revised: First, using offline trajectory preference data facilitates reproducibility by allowing us to create benchmarks (with fixed training datasets) for studying causal confusion in reward learning, similar to existing standard supervised learning benchmarks.

Original: Second, learning from offline data is known to often lead to overfitting in batch or offline reinforcement learning (Levine et al., 2020), and prior work has shown anecdotal evidence that learning a reward function from a fixed set of preferences can lead to causal confusion (Christiano et al., 2017; Ibarz et al., 2018; Javed et al., 2021)—hence, learning from offline data naturally lends itself to a study of causal confusion.
Revised: Second, learning from offline data is known to often lead to overfitting in batch or offline reinforcement learning (Levine et al., 2020), and prior work has shown anecdotal evidence that learning a reward function from a fixed set of preferences can lead to causal confusion (Christiano et al., 2017; Ibarz et al., 2018; Javed et al., 2021)—hence, learning from offline data naturally lends itself to a study of causal confusion.

Original: Third, learning reward functions from offline data, as described above, removes the need for running reinforcement learning (Christiano et al., 2017) or trajectory optimization (Sadigh et al., 2017) in the inner loop, as is required for many active preference learning methods.
Revised: Third, learning reward functions from offline data, as described above, removes the need for running reinforcement learning (Christiano et al., 2017) or trajectory optimization (Sadigh et al., 2017) in the inner loop, as is required for many active preference learning methods.

Original: This enables us to learn a reward function by optimizing the likelihood in
Revised: This enables us to learn a reward function by optimizing the likelihood in

Original: Equation 2 (which amounts to optimizing a standard cross-entropy loss with the predicted cumulative rewards as logits) and then learn a corresponding policy using any off-the-shelf reinforcement learning algorithm to optimize r θ .
Revised: Equation 2 (which amounts to optimizing a standard cross-entropy loss with the predicted cumulative rewards as logits) and then learn a corresponding policy using any off-the-shelf reinforcement learning algorithm to optimize r θ .

Original: This sequential approach—learning a reward function followed by learning a policy—significantly improves the scalability and simplicity of experiments, facilitating the rapid testing and evaluation needed here for a systematic study of causal confusion.
Revised: This sequential approach—learning a reward function followed by learning a policy—significantly improves the scalability and simplicity of experiments, facilitating the rapid testing and evaluation needed here for a systematic study of causal confusion.

Original: B. Implementation and Optimization Details
Revised: B. Implementation and Optimization Details

Original: In practice, we use the Adam optimizer in PyTorch to learn the reward function, r θ , and then use PPO (Schulman et al., 2017) or SAC (Haarnoja et al., 2018) for policy optimization given r θ .
Revised: In practice, we use the Adam optimizer in PyTorch to learn the reward function, r θ , and then use PPO (Schulman et al., 2017) or SAC (Haarnoja et al., 2018) for policy optimization given r θ .

Original: C. Synthetic Preference Generation
Revised: C. Synthetic Preference Generation

Original: To enhance scalability and reproducibility, we automatically generate a large amount of synthetic trajectory preferences.
Revised: To enhance scalability and reproducibility, we automatically generate a large amount of synthetic trajectory preferences.

Original: This was done using a pretrained RL policy for each of these domains that was trained using the ground-truth reward function provided with each of these environments.
Revised: This was done using a pretrained RL policy for each of these domains that was trained using the ground-truth reward function provided with each of these environments.

Original: We then generate a large number of diverse trajectories by adding ϵ -greedy noise during policy rollouts, where ϵ is the probability that the policy takes an action uniformly at random from its action space.
Revised: We then generate a large number of diverse trajectories by adding ϵ -greedy noise during policy rollouts, where ϵ is the probability that the policy takes an action uniformly at random from its action space.

Original: Thus, ϵ = 0 corresponds to the fully trained RL policy and ϵ = 1 corresponds to a uniformly random policy.
Revised: Thus, ϵ = 0 corresponds to the fully trained RL policy and ϵ = 1 corresponds to a uniformly random policy.

Original: As noted by
Revised: As noted by

Original: Brown et al.
Revised: Brown et al.

Original: (Brown et al., 2020a), adding this type of disturbance will result in monotonically decreasing performance in expectation.
Revised: (Brown et al., 2020a), adding this type of disturbance will result in monotonically decreasing performance in expectation.

Original: To generate pairwise preferences over trajectories, we randomly sample pairs of trajectorieswith replacement (while ensuring that the two trajectories within a pair are nonidentical) from a set of 120 total trajectories  generated with20 ϵ -greedy rollouts for ϵ ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } .
Revised: To generate pairwise preferences over trajectories, we select all pairs of trajectories from a set of 40, 120, andtotal trajectories (for the ’small’, ’medium’, and ’large’ dataset sizes, respectively) generated with ϵ -greedy rollouts for ϵ ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } .

Original: From these pairwise preferences, we randomly set aside 10% as a validationset, and use the rest to train.
Revised: We use held-out sets of trajectories for validation and testing.

Original: We then use the ground-truth reward functions provided by each environment to provide ground-truth preference labels.
Revised: We then use the ground-truth reward functions provided by each environment to provide ground-truth preference labels.

Original: Using the dataset of preferences, we train a neural network reward function approximator with two hidden layers (128 units and 64 units, respectively) and Leaky ReLU activations, after which we perform 1,000,000 timesteps of reinforcement learning with PPO (Schulman et al., 2017) (for Feeding and Itch Scratching) and SAC (Haarnoja et al., 2018) (for Reacher) using the learned reward function in place of the ground-truth reward function.
Revised: Using the dataset of preferences, we train a neural network reward function approximator with two hidden layers (128 units and 64 units, respectively) and Leaky ReLU activations, after which we perform 1,000,000 timesteps of reinforcement learning with PPO (Schulman et al., 2017) (for Feeding and Itch Scratching) and SAC (Haarnoja et al., 2018) (for Reacher) using the learned reward function in place of the ground-truth reward function.

Revised: Hyperparameters–weight decay, learning rate–are tuned separately for each environment using the validation set.

Original: We optimize the reward function approximator using stochastic gradient descent with weight decayof 0 . 01 and early-stopping on the validation loss (with a patience of 10 epochs).
Revised: We optimize the reward function approximator using stochastic gradient descent with weight decay and early-stopping on the validation loss (with a patience of 10 epochs).

Original: D. The Choice of Observation Feature Space, Experimental Details
Revised: D. The Choice of Observation Feature Space, Experimental Details

Original: When training, we empirically found that the model tends to learn a large bias, and that removing the bias helps incentivize
Revised: When training, we empirically found that the model tends to learn a large bias, and that removing the bias helps incentivize

Original: RL to learn more desirable behavior.
Revised: RL to learn more desirable behavior.

Original: Thus, we remove biases from the final layers of subsequent models.
Revised: Thus, we remove biases from the final layers of subsequent models.

Original: We train the reward-learning model on 2000 pairwise preferences of randomly-selected whole trajectories from the ϵ -greedy rollouts that are at least 60 ‘ranks apart’ when ranked using the ground truth reward (defined as ∆ pair = 60 ).
Revised: We train the reward-learning model on 2000 pairwise preferences of randomly-selected whole trajectories from the ϵ -greedy rollouts that are at least 60 ‘ranks apart’ when ranked using the ground truth reward (defined as ∆ pair = 60 ).

Original: We employ early-stopping with a patience of 10 epochs and maximum of 100 epochs, and apply l 1 -regularization with λ = 0 . 1 .
Revised: We employ early-stopping with a patience of 10 epochs and maximum of 100 epochs, and apply l 1 -regularization with λ = 0 . 1 .

Original: PPO is then used to learn a policy from the learned reward for Feeding and Itch Scratching, whereas SAC is used for Reacher.
Revised: PPO is then used to learn a policy from the learned reward for Feeding and Itch Scratching, whereas SAC is used for Reacher.

Original: Raw features in Feeding consist of spoon and head position, robot joint angles, and amount of force applied on human, for a total raw observation dimensionality of 25.
Revised: Raw features in Feeding consist of spoon and head position, robot joint angles, and amount of force applied on human, for a total raw observation dimensionality of 25.

Original: Raw features in Reacher are end effector and target position, arm angles, and angular velocities, for a total dimensionality of 11.
Revised: Raw features in Reacher are end effector and target position, arm angles, and angular velocities, for a total dimensionality of 11.

Original: Itch Scratching’s raw features consist of tool and target position, robot
Revised: Itch Scratching’s raw features consist of tool and target position, robot

Original: [Figure]
Revised: [Figure]

Original: E. The Effect of Reward Function Model Capacity, Experimental Details
Revised: E. The Effect of Reward Function Model Capacity, Experimental Details

Original: Similar to the above feature-space sensitivity analysis, we learn a reward function from 2000 ground-truth pairwise preference labels over randomly selected trajectory pairs with a ∆ pair of 60.
Revised: Similar to the above feature-space sensitivity analysis, we learn a reward function from 2000 ground-truth pairwise preference labels over randomly selected trajectory pairs with a ∆ pair of 60.

Original: The model is optimized using stochastic gradient descent with weight decay ( λ = 0 . 01 ) and regularized with l 1 -regularization ( λ 1 = 0 . 01 ).
Revised: The model is optimized using stochastic gradient descent with weight decay ( λ = 0 . 01 ) and regularized with l 1 -regularization ( λ 1 = 0 . 01 ).

Original: We use early-stopping with a patience of 10 epochs and an upper limit of 100 epochs.
Revised: We use early-stopping with a patience of 10 epochs and an upper limit of 100 epochs.

Original: In order to measure the effect of reward network model capacity, we train the following models with increasing order of complexity: linear model, one-layer fully-connected neural network with a hidden dimension of 64, 128; two-layer fully-connected networks with hidden dimensions of (64, 64), (128, 64), (128, 128), (256, 128) and (256, 256).
Revised: In order to measure the effect of reward network model capacity, we train the following models with increasing order of complexity: linear model, one-layer fully-connected neural network with a hidden dimension of 64, 128; two-layer fully-connected networks with hidden dimensions of (64, 64), (128, 64), (128, 128), (256, 128) and (256, 256).

Original: For each model architecture, we train the reward network on pairwise preferences, run the RL optimization for 3 seeds, and evaluate the cumulative ground truth reward and task success on 100 rollouts from a validation seed.
Revised: For each model architecture, we train the reward network on pairwise preferences, run the RL optimization for 3 seeds, and evaluate the cumulative ground truth reward and task success on 100 rollouts from a validation seed.

Original: F. The Effect of Data Collection
Revised: F. The Effect of Data Collection

Original: Further, we explore the process through which we generate training data and the differences in agent performance that can result.
Revised: Further, we explore the process through which we generate training data and the differences in agent performance that can result.

Original: Specifically, we study (1) how to generate trajectories for demonstrations, and (2) how to select or sample pairwise preferences from these generated trajectories.
Revised: Specifically, we study (1) how to generate trajectories for demonstrations, and (2) how to select or sample pairwise preferences from these generated trajectories.

Original: Trajectory Generation:
Revised: Trajectory Generation:

Original: To study the effect of training data on causal confusion for preference-based reward learning, we examine two trajectory generation methods:
Revised: To study the effect of training data on causal confusion for preference-based reward learning, we examine two trajectory generation methods:

Original: RL+noise.
Revised: RL+noise.

Original: We first train an RL policy on a given ground truth reward, then generate demonstrations by adding ϵ -greedy noise to rollouts from the trained policy.
Revised: We first train an RL policy on a given ground truth reward, then generate demonstrations by adding ϵ -greedy noise to rollouts from the trained policy.

Original: Demonstrations are labeled using the ground truth reward.
Revised: Demonstrations are labeled using the ground truth reward.

Original: T-REX.
Revised: T-REX.

Original: Following Brown et al.
Revised: Following Brown et al.

Original: (Brown et al., 2019), we also create a diverse set of training trajectories by (1) periodically checkpointing an RL policy trained on the ground-truth reward to generate synthetic demonstrations with varying levels of performance and (2) using human-teleoperation to pedagogically provide a set of trajectories with a wide range of demonstration qualities.
Revised: (Brown et al., 2019), we also create a diverse set of training trajectories by (1) periodically checkpointing an RL policy trained on the ground-truth reward to generate synthetic demonstrations with varying levels of performance and (2) using human-teleoperation to pedagogically provide a set of trajectories with a wide range of demonstration qualities.

Original: Pairwise Preference Selection:
Revised: Pairwise Preference Selection:

Original: Given a set of ranked trajectories, we study two ways of creating pairwise preferences: 
Revised: Given a set of ranked trajectories, we study two ways of creating pairwise preferences: 1.

Original: Random ∆ pair -sampling.
Revised: Random ∆ pair -sampling.

Original: We randomly sample pairs of trajectories with replacement and discard pairs that are within ∆ pair ranks of each other.
Revised: We randomly sample pairs of trajectories with replacement and discard pairs that are within ∆ pair ranks of each other.

Original: We do this until we have the desired number of pairwise preferences.
Revised: We do this until we have the desired number of pairwise preferences.

Original: We hypothesize that enforcing a ∆ pair difference within pairs ensures that the differences between the two trajectories are more salient and thus easier for the reward network to learn
Revised: We hypothesize that enforcing a ∆ pair difference within pairs ensures that the differences between the two trajectories are more salient and thus easier for the reward network to learn.

Original: Systematic all-pairs selection.
Revised: Systematic all-pairs selection.

Original: We select M evenly-spaced demonstrations from the set of provided demonstrations and generate all possible (cid:0) M 2 (cid:1) pairwise preferences.
Revised: We select M evenly-spaced demonstrations from the set of provided demonstrations and generate all possible (cid:0) M 2 (cid:1) pairwise preferences.

Original: This method was originally proposed by Brown et al.
Revised: This method was originally proposed by Brown et al.

Original: (Brown et al., 2019) and requires no additional hyperparameters.[Figure] a t i v [Figure]
Revised: (Brown et al., 2019) and requires no additional hyperparameters.

Original: r

Original: [Figure]

Original: a r d

Original: [Figure]

Original: Synthetic T-REX vs. RL+noise

Original: [Figure]
Revised: [Figure]

Original: Experimental Setup:
Revised: Experimental Setup:

Original: In each of the four configurations (2 generation methods × 2 preference selection methods), we train a linear model, using an augmented feature space consisting of half of the raw observation features (10 in Feeding, 5 in Reacher) and all of the privileged features (3 in Feeding, 1 in Reacher) from each environment.
Revised: In each of the four configurations (2 generation methods × 2 preference selection methods), we train a linear model, using an augmented feature space consisting of half of the raw observation features (10 in Feeding, 5 in Reacher) and all of the privileged features (3 in Feeding, 1 in Reacher) from each environment.

Original: We train the reward learning network for 100 epochs with early stopping (patience of 10), and add weight decay with λ = 0 .
Revised: We train the reward learning network for 100 epochs with early stopping (patience of 10), and add weight decay with λ = 0 .

Original: 01 and l 1regularization with λ 1 = 0 .
Revised: 01 and l 1regularization with λ 1 = 0 .

Original: In the RL+noise trajectory generation method, we generate 20 rollouts at each level of ϵ =
Revised: In the RL+noise trajectory generation method, we generate 20 rollouts at each level of ϵ =

Original: For the human-teleoperated T-REX trajectories, we had the human demonstrator perform 8 demonstrations that ‘successfully complete the task’, 7 demonstrations that ‘fail at the task’, and 5 demonstrations that ‘half-succeed, half-fail’ (eg., demonstration would spill some food, then feed the rest in the case of the Feeding task), for a total of 20 teleoperated demonstrations.
Revised: For the human-teleoperated T-REX trajectories, we had the human demonstrator perform 8 demonstrations that ‘successfully complete the task’, 7 demonstrations that ‘fail at the task’, and 5 demonstrations that ‘half-succeed, half-fail’ (eg., demonstration would spill some food, then feed the rest in the case of the Feeding task), for a total of 20 teleoperated demonstrations.

Original: To generate the T-REX trajectories taken from a checkpointed policy, we take 20 rollouts at each of the following checkpoints (expressed as a proportion of the total number of training iterations): [0 . 01 , 0 .
Revised: To generate the T-REX trajectories taken from a checkpointed policy, we take 20 rollouts at each of the following checkpoints (expressed as a proportion of the total number of training iterations): [0 . 01 , 0 .

Original: We use a fixed ∆ pair = 60 for our random ∆ pair -sampling configurations.
Revised: We use a fixed ∆ pair = 60 for our random ∆ pair -sampling configurations.

Original: Discussion:
Revised: Discussion:

Original: Figure 6 displays our analysis of the aforementioned trajectory generation and preference selection methods when used in conjunction with one another.
Revised: Figure 6 displays our analysis of the aforementioned trajectory generation and preference selection methods when used in conjunction with one another.

Original: Firstly, we observe in Figure 6a that performing preference learning on using pairwise preferences over synthetic demonstrations generated by adding ϵ -greedy noise to a pretrained policy (RL+noise) performs on par with, if not better than, the approach suggested by Brown et al.
Revised: Firstly, we observe in Figure 6a that performing preference learning on using pairwise preferences over synthetic demonstrations generated by adding ϵ -greedy noise to a pretrained policy (RL+noise) performs on par with, if not better than, the approach suggested by Brown et al.

Original: (Brown et al., 2019)
Revised: (Brown et al., 2019)

Original: (T-REX) which takes rollouts from different partially trained policy checkpoints and then providing pairwise preferences over these rollouts.
Revised: (T-REX) which takes rollouts from different partially trained policy checkpoints and then providing pairwise preferences over these rollouts.

Original: The implication is that we do not find any evidence that the causal confusion for reward inference is due to the trajectory generation being too simplistic or biased.
Revised: The implication is that we do not find any evidence that the causal confusion for reward inference is due to the trajectory generation being too simplistic or biased.

Original: Future work should examine whether active methods that obtain pairwise preferences by synthesizing informative trajectory pairs from scratch (Sadigh et al., 2017) lead to better performance.
Revised: Future work should examine whether active methods that obtain pairwise preferences by synthesizing informative trajectory pairs from scratch (Sadigh et al., 2017) lead to better performance.

Original: Next, we note that human-teleoperated demonstrations that attempt to give a diverse set of good, mediocre, and poor demonstrations enable high preference learning performance in fewer demonstrations, as shown in Figure 6b.
Revised: Next, we note that human-teleoperated demonstrations that attempt to give a diverse set of good, mediocre, and poor demonstrations enable high preference learning performance in fewer demonstrations, as shown in Figure 6b.

Original: However, when given more demonstrations and sampled with ∆ pair = 60 , using synthetically-generated RL+noise trajectories approaches the performance of using human-teleoperated demonstrations.
Revised: However, when given more demonstrations and sampled with ∆ pair = 60 , using synthetically-generated RL+noise trajectories approaches the performance of using human-teleoperated demonstrations.

Original: This reinforces the previous finding—at least for large data regimes, the trajectory generation method is not the culprit of the poor reward inference performance demonstrated in the main results.
Revised: This reinforces the previous finding—at least for large data regimes, the trajectory generation method is not the culprit of the poor reward inference performance demonstrated in the main results.

Original: Lastly, we observe that human demonstrations, though preferred over synthetic demonstrations, can also lead to causal confusion over the true reward function.
Revised: Lastly, we observe that human demonstrations, though preferred over synthetic demonstrations, can also lead to causal confusion over the true reward function.

Original: In Figure 6c, the mean cumulative reward increases with the addition of privileged features and the removal of raw features from the input observation of the reward network.
Revised: In Figure 6c, the mean cumulative reward increases with the addition of privileged features and the removal of raw features from the input observation of the reward network.