Original: R EWARD

Original: U NCERTAINTY

Original:  FOR E XPLORATION IN P REFERENCE - BASED R EINFORCEMENT L EARNINGAnonymous authors
Revised: R EWARD U NCERTAINTY FOR E XPLORATION IN P REFERENCE - BASED R EINFORCEMENT L EARNING

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering.
Revised: Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering.

Original: Preference-based RL methods are able to learn a more ﬂexible reward model based on human preferences by actively incorporating human feedback, i e. teacher’s preferences between two clips of behaviors.
Revised: Preference-based RL methods are able to learn a more ﬂexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher’s preferences between two clips of behaviors.

Original: However, poor feedback-efﬁciency still remainsas a problem in current preference-based RL algorithms, as tailored human feedback is very expensive.
Revised: However, poor feedback-efﬁciency still remains a problem in current preference-based RL algorithms, as tailored human feedback is very expensive.

Original: To handle this issue, previous methods have mainly focused on improving query selection and policy initialization.
Revised: To handle this issue, previous methods have mainly focused on improving query selection and policy initialization.

Original: At the same time, recent exploration methods have proven to be a recipe for improving sample-efﬁciency in RL.
Revised: At the same time, recent exploration methods have proven to be a recipe for improving sample-efﬁciency in RL.

Original: We present an exploration method speciﬁcally for preference-based RL algorithms.
Revised: We present an exploration method speciﬁcally for preference-based RL algorithms.

Original: Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward.
Revised: Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward.

Original: Speciﬁcally, we utilize disagreement across ensemble of learned reward models.
Revised: Speciﬁcally, we utilize disagreement across ensemble of learned reward models.

Original: Our intuition is that disagreement in learned reward model reﬂects uncertainty in tailored human feedback and could be useful for exploration.
Revised: Our intuition is that disagreement in learned reward model reﬂects uncertainty in tailored human feedback and could be useful for exploration.

Original: Our experiments show that reward uncertainty exploration improves both feedbackand sample-efﬁciency of preference-based RL algorithms on complex robot manipulation tasks from Meta-World benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.
Revised: Our experiments show that reward uncertainty exploration improves both feedbackand sample-efﬁciency of preference-based RL algorithms on complex robot manipulation tasks from Meta-World benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In reinforcement learning (RL), reward function speciﬁes correct objectives to RL agents.
Revised: In reinforcement learning (RL) reward function speciﬁes correct objectives to RL agents.

Original: However, it is difﬁcult and time-consuming to carefully design suitable reward functions for a variety of complex behaviors (e.g., cooking or book summarization (Wu et al., 2021)).
Revised: However, it is difﬁcult and time-consuming to carefully design suitable reward functions for a variety of complex behaviors (e.g., cooking or book summarization (Wu et al., 2021)).

Original: Furthermore, if there are complicated social norms we want RL agents to understand and follow, conveying a reliable reward function to include such information may remain to be an open problem (Amodei et al., 2016;
Revised: Furthermore, if there are complicated social norms we want RL agents to understand and follow, conveying a reliable reward function to include such information may remain to be an open problem (Amodei et al., 2016;

Original: Hadﬁeld-Menell et al., 2017).
Revised: Hadﬁeld-Menell et al., 2017).

Original: Overall, engineering reward functions purely by human efforts for all tasks remains to be a signiﬁcant challenge.
Revised: Overall, engineering reward functions purely by human efforts for all tasks remains to be a signiﬁcant challenge.

Original: An alternative to resolve the challenge of reward engineering is preference-based RL (Christiano et al., 2017; Ibarz et al., 2018;
Revised: An alternative to resolve the challenge of reward engineering is preference-based RL (Christiano et al., 2017; Ibarz et al., 2018;

Original: Lee et al., 2021b).
Revised: Lee et al., 2021b).

Original: Compared to traditional RL setup, preferencebased RL algorithms are able to teach RL agents without the necessity of designing reward functions.
Revised: Compared to traditional RL setup, preferencebased RL algorithms are able to teach RL agents without the necessity of designing reward functions.

Original: Instead, the agent uses feedback, usually in the form of (human) teacher preferences between two behaviors, to learn desired behaviors indicated by teacher.
Revised: Instead, the agent uses feedback, usually in the form of (human) teacher preferences between two behaviors, to learn desired behaviors indicated by teacher.

Original: Therefore, instead of using carefullydesigned rewards from the environment, the agent is able to learn a more ﬂexible reward function suitably aligned to teacher feedback.
Revised: Therefore, instead of using carefullydesigned rewards from the environment, the agent is able to learn a more ﬂexible reward function suitably aligned to teacher feedback.

Original: However, preference-based RL usually requires a large amount of teacher feedback, which may be timely or sometimes infeasible to collect.
Revised: However, preference-based RL usually requires a large amount of teacher feedback, which may be timely or sometimes infeasible to collect.

Original: To improve feedback-efﬁciency, prior works have investigated several sampling strategies (Biyik & Sadigh, 2018;
Revised: To improve feedback-efﬁciency, prior works have investigated several sampling strategies (Biyik & Sadigh, 2018;

Original: Sadigh et al., 2017; Biyik et al., 2020; Lee et al., 2021c).
Revised: Sadigh et al., 2017; Biyik et al., 2020; Lee et al., 2021c).

Original: These methods aim to select more informative queries to improve the quality of the learned reward function while asking for fewer feedback from teacher.
Revised: These methods aim to select more informative queries to improve the quality of the learned reward function while asking for fewer feedback from teacher.

Original: Another line of works focus on policy initialization.
Revised: Another line of works focus on policy initialization.

Original: Ibarz et al.
Revised: Ibarz et al.

Original: (2018) initialized the agent’s policy with imitation learning from the expert demonstrations, and Lee et al.
Revised: (2018) initialized the agent’s policy with imitation learning from the expert demonstrations, and Lee et al.

Original: (2021b) utilized unsupervised pre-training  of RL agents before collecting for teacher preferences in the hope of learning diverse behaviors in a self-supervised way to reduce total amount of human feedback.
Revised: (2021b) utilized unsupervised pre-training ∗ Equal advising. of RL agents before collecting for teacher preferences in the hope of learning diverse behaviors in a self-supervised way to reduce total amount of human feedback.

Original: Exploration, in the context of standard RL, has addressed the problems of sample-efﬁciency (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; 2019; Liu & Abbeel, 2021; Seo et al., 2021b).
Revised: Exploration, in the context of standard RL, has addressed the problems of sample-efﬁciency (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; 2019; Liu & Abbeel, 2021; Seo et al., 2021b).

Original: When extrinsic rewards from the environment is limited, exploration has been demonstrated to allow
Revised: When extrinsic rewards from the environment is limited, exploration has been demonstrated to allow

Original: RL agents to learn diverse behaviors.
Revised: RL agents to learn diverse behaviors.

Original: However, limited previous works have studied the effects of exploration in preference-based RL.
Revised: However, limited previous works have studied the effects of exploration in preference-based RL.

Original: Inspired by the impact of exploration, we present RUNE: R eward UN certainty for E xploration, a simple and efﬁcient exploration method speciﬁcally for preference-based RL algorithms.
Revised: Inspired by the impact of exploration, we present RUNE: R eward UN certainty for E xploration, a simple and efﬁcient exploration method speciﬁcally for preference-based RL algorithms.

Original: Our main idea is to incorporate uncertainty from learned reward function as an exploration bonus.
Revised: Our main idea is to incorporate uncertainty from learned reward function as an exploration bonus.

Original: Speciﬁcally, we capture the novelty of human feedback by measuring the reward uncertainty (e.g., variance in predictions of ensemble of reward functions).
Revised: Speciﬁcally, we capture the novelty of human feedback by measuring the reward uncertainty (e.g., variance in predictions of ensemble of reward functions).

Original: Since reward functions is optimized and learned to align to human feedback, exploration based on reward uncertainty may also reﬂect high uncertainty in information from teacher feedback.
Revised: Since reward functions is optimized and learned to align to human feedback, exploration based on reward uncertainty may also reﬂect high uncertainty in information from teacher feedback.

Original: We hope that the proposed intrinsic reward contains information from teacher feedback and can guide exploration that better align to human preferences.
Revised: We hope that the proposed intrinsic reward contains information from teacher feedback and can guide exploration that better align to human preferences.

Original: Our experiment results show that RUNE can improve both sample- and feedback-efﬁciency of preference-based RL algorithms (Lee et al., 2021b).
Revised: Our experiment results show that RUNE can improve both sample- and feedback-efﬁciency of preference-based RL algorithms (Lee et al., 2021b).

Original: We highlight the main contributions of our paper below:
Revised: We highlight the main contributions of our paper below:

Original: • For preference-based RL, we propose a new exploration method based on uncertainty in learned reward functions.
Revised: • For preference-based RL, we propose a new exploration method based on uncertainty in learned reward functions.

Original: • For the ﬁrst time, we show that exploration can improve the sample- and feedback-efﬁciency of preference-based RL algorithms.
Revised: • For the ﬁrst time, we show that exploration can improve the sample- and feedback-efﬁciency of preference-based RL algorithms.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Human-in-the-loop reinforcement learning .
Revised: Human-in-the-loop reinforcement learning .

Original: We mainly focus on one promising direction that utilizes the human preferences (Akrour et al., 2011; Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b; Leike et al., 2018; Pilarski et al., 2011; Wilson et al., 2012) to train RL agents.
Revised: We mainly focus on one promising direction that utilizes the human preferences (Akrour et al., 2011; Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b; Leike et al., 2018; Pilarski et al., 2011; Wilson et al., 2012) to train RL agents.

Original: Christiano et al.
Revised: Christiano et al.

Original: (2017) scaled preference-based learning to utilize modern deep learning techniques, and Ibarz et al.
Revised: (2017) scaled preference-based learning to utilize modern deep learning techniques, and Ibarz et al.

Original: (2018) improved the efﬁciency of this method by introducing additional forms of feedback such as demonstrations.
Revised: (2018) improved the efﬁciency of this method by introducing additional forms of feedback such as demonstrations.

Original: Recently, Lee et al.
Revised: Recently, Lee et al.

Original: (2021b) proposed a feedback-efﬁcient RL algorithm by utilizing off-policy learning and pre-training.
Revised: (2021b) proposed a feedback-efﬁcient RL algorithm by utilizing off-policy learning and pre-training.

Original: To improve sample- and feedback-efﬁciency of human-in-the-loop RL, previous works (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b; Leike et al., 2018) mainly focus on methods such as selecting more informative queries Christiano et al.
Revised: To improve sample- and feedback-efﬁciency of human-in-the-loop RL, previous works (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b; Leike et al., 2018) mainly focus on methods such as selecting more informative queries Christiano et al.

Original: and pre-training of RL agents Ibarz et al.
Revised: and pre-training of RL agents Ibarz et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: (2021b).
Revised: (2021b).

Original: We further investigate effects of different exploration methods in preference-based RL algorithm.
Revised: We further investigate effects of different exploration methods in preference-based RL algorithm.

Original: We follow a common approach of exploration methods in RL: generating intrinsic rewards as exploration bonus Pathak et al.
Revised: We follow a common approach of exploration methods in RL: generating intrinsic rewards as exploration bonus Pathak et al.

Original: Instead of only using learned reward function from human feedback as RL training objective, we alter the reward function to include a combination of the extrinsic reward (the learned rewards) and an intrinsic reward (explo- ration bonus).
Revised: Instead of only using learned reward function from human feedback as RL training objective, we alter the reward function to include a combination of the extrinsic reward (the learned rewards) and an intrinsic reward (exploration bonus).

Original: In particular, we present an exploration method with intrinsic reward that measures the disagreement from learned reward models.
Revised: In particular, we present an exploration method with intrinsic reward that measures the disagreement from learned reward models.

Original: Exploration in reinforcement learning.
Revised: Exploration in reinforcement learning.

Original: The trade off between exploitation and exploration is a critical topic in RL.
Revised: The trade off between exploitation and exploration is a critical topic in RL.

Original: If agents don’t explore enough, then they may learn sub optimal actions.
Revised: If agents don’t explore enough, then they may learn sub optimal actions.

Original: Exploration algorithms aim to encourage the RL agent to visit a wide range of states in the environment.
Revised: Exploration algorithms aim to encourage the RL agent to visit a wide range of states in the environment.

Original: Thrun (1992) showed that exploration methods that utilize the agent’s history has been shown to perform much better than random exploration.
Revised: Thrun (1992) showed that exploration methods that utilize the agent’s history has been shown to perform much better than random exploration.

Original: Hence, a common setup is to include an intrinsic reward as an exploration bonus.
Revised: Hence, a common setup is to include an intrinsic reward as an exploration bonus.

Original: The intrinsic reward can be deﬁned by Count-Based methods which keep count of previously visited states and rewards the agents for visiting new states Bellemare et al.
Revised: The intrinsic reward can be deﬁned by Count-Based methods which keep count of previously visited states and rewards the agents for visiting new states Bellemare et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: (2017); Ostrovski et al.
Revised: (2017); Ostrovski et al.

Original: Another option is to use a curiosity bonus for the intrinsic reward Houthooft et al.
Revised: Another option is to use a curiosity bonus for the intrinsic reward Houthooft et al.

Original: ; Pathak et al.
Revised: ; Pathak et al.

Original: (2017); Sekar et al.
Revised: (2017); Sekar et al.

Original: Curiosity represents how expected and unfamiliar the state is.
Revised: Curiosity represents how expected and unfamiliar the state is.

Original: One way to quantify curiosity is to predict the next state from current state and action Pathak et al.
Revised: One way to quantify curiosity is to predict the next state from current state and action Pathak et al.

Original: (2017), then use prediction error as an estimate of curiosity.
Revised: (2017), then use prediction error as an estimate of curiosity.

Original: If the error is high, that means the next state is unfamiliar and should be explored more.
Revised: If the error is high, that means the next state is unfamiliar and should be explored more.

Original: Similarly, instead of predicting the next state, prediction errors from training a neural network to approximate a random function Burda et al.
Revised: Similarly, instead of predicting the next state, prediction errors from training a neural network to approximate a random function Burda et al.

Original: (2018) can serve as a valid estimate of curiosity.
Revised: (2018) can serve as a valid estimate of curiosity.

Original: If there are multiple models, then curiosity can also be described as the disagreement between the models Pathak et al.
Revised: If there are multiple models, then curiosity can also be described as the disagreement between the models Pathak et al.

Original: A high disagreement means that the models are unsure about the prediction and need to explore in that direction more.
Revised: A high disagreement means that the models are unsure about the prediction and need to explore in that direction more.

Original: A different approach maximizes the entropy of visited states by incorporating state entropy into the intrinsic reward.
Revised: A different approach maximizes the entropy of visited states by incorporating state entropy into the intrinsic reward.

Original: State entropy can be estimated by approximating the state density distribution
Revised: State entropy can be estimated by approximating the state density distribution

Original: Hazan et al.
Revised: Hazan et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: (2019), approximating the k-nearest neighbor entropy of a randomly initialized encoder Seo et al.
Revised: (2019), approximating the k-nearest neighbor entropy of a randomly initialized encoder Seo et al.

Original: (2021a), or using off-policy RL algorithms to maximize the k -nearest neighbor state entropy estimate in contrastive representation space for unsupervised pre-training
Revised: (2021a), or using off-policy RL algorithms to maximize the k -nearest neighbor state entropy estimate in contrastive representation space for unsupervised pre-training

Original: Srinivas et al.
Revised: Srinivas et al.

Original: ; Liu & Abbeel (2021).
Revised: ; Liu & Abbeel (2021).

Original: These methods all encourage agents to explore diverse states.
Revised: These methods all encourage agents to explore diverse states.

Original: Our approach adds an intrinsic reward that drives exploration to preference-based RL algorithms.
Revised: Our approach adds an intrinsic reward that drives exploration to preference-based RL algorithms.

Original: We take advantage of an ensemble of reward models in preference-based RL algorithms, which is not available in other traditional RL settings.
Revised: We take advantage of an ensemble of reward models in preference-based RL algorithms, which is not available in other traditional RL settings.

Original: To estimate novelty of states and actions, we utilize the disagreement between reward models for our intrinsic reward, in hope of encouraging exploration aligned to directions of human preferences.
Revised: To estimate novelty of states and actions, we utilize the disagreement between reward models for our intrinsic reward, in hope of encouraging exploration aligned to directions of human preferences.

Revised: Trajectory generation in preference-based reinforcement learning.

Revised: Previous works in preference-based reinforcement learning have investigated several methods to better explore diverse trajectories but close to current optimal policy Wirth et al.

Revised: One line of works computes agent’s stochastic policies that are slightly deviated from optimal policies.

Original: P RELIMINARIES
Revised: Christiano et al. (2017) uses Trust Region Policy Optimization (TRPO) Schulman et al. and synchronized A3C Mnih et al. These RL algorithms deﬁne stochastic policies to ensure exploration of action space and deviations from optimal policies.

Revised: However, these exploration methods based on stochastic RL algorithms does not include information from human preferences to drive exploration.

Revised: Another line of works designs one or multiple criterion to select from multiple possible stochastic policy candidates.

Revised: Wilson et al.

Revised: (2012) proposes to sample several policies from posterior distribution of policy space after updating human preferences.

Revised: However, such methods come a the cost of requiring many samples collected beforehand.

Revised: While these methods similarly aims to reduce uncertainty in human preferences, RUNE uses a different metric to estimate such uncertainty through reward functions ensemble.

Revised: This is different from previous works and is simple, scalable, and easy to implement.

Revised: A different approach allows human to guide exploration by directly providing additional trajectories.

Revised: Zucker et al.

Revised: (2010) proposes a user-guided exploration method that shows samples of trajectories to human.

Revised: Human can provide additional feedback to guide exploration.

Revised: While this method receives exact information from human, it requires additional human labels, which are usually expensive and time-consuming to collect.

Revised: RUNE however tries to extract information from human feedback revealed in learned reward functions, which doesn’t require additional human input.

Revised:  P RELIMINARIES

Original: Preference-based reinforcement learning .
Revised: Preference-based reinforcement learning .

Original: We consider an agent interacting with an environment in discrete time Sutton & Barto (2018).
Revised: We consider an agent interacting with an environment in discrete time Sutton & Barto (2018).

Original: At each timestep t , the agent receives a state s t from the environment and chooses an action a t based on its policy π .
Revised: At each timestep t , the agent receives a state s t from the environment and chooses an action a t based on its policy π .

Original: In traditional reinforcement learning, the environment also returns a reward r ( s t , a t ) that evaluates the quality of agent’s behavior at timestep t .
Revised: In traditional reinforcement learning, the environment also returns a reward r ( s t , a t ) that evaluates the quality of agent’s behavior at timestep t .

Original: The goal of agent is to maximize the discounted sum of rewards.
Revised: The goal of agent is to maximize the discounted sum of rewards.

Original: However, in the preference-based RL framework, we don’t have such a reward function returned from the environment.
Revised: However, in the preference-based RL framework, we don’t have such a reward function returned from the environment.

Original: Instead, a (human) teacher provides preferences between the agent’s behaviors and the agent learns its policy from feedbacks (Christiano et al., 2017; Ibarz et al., 2018;
Revised: Instead, a (human) teacher provides preferences between the agent’s behaviors and the agent learns its policy from feedbacks (Christiano et al., 2017; Ibarz et al., 2018;

Original: Lee et al., 2021b; Leike et al., 2018).
Revised: Lee et al., 2021b; Leike et al., 2018).

Original: Formally, a segment σ is a sequence of time-indexed observations and actions { ( s 1 , a 1 ) , ..., ( s H , a H ) } .
Revised: Formally, a segment σ is a sequence of time-indexed observations and actions { ( s 1 , a 1 ) , ..., ( s H , a H ) } .

Original: Given a pair of segments ( σ 0 , σ 1 ) that describe two behaviors, a teacher indicates which segment is preferred, i.e., y = (0 , 1) or (1 , 0) , that the two segments are equally preferred y =
Revised: Given a pair of segments ( σ 0 , σ 1 ) that describe two behaviors, a teacher indicates which segment is preferred, i.e., y = ( σ 0 (cid:31) σ 1 ) or ( σ 1 (cid:31) σ 0 ) , that the two segments are equally preferred y =

Original: (0 . 5 , 0 . 5) , or that two segments are incomparable, i.e., discarding the query.
Revised: ( σ 1 = σ 0 ) , or that two segments are incomparable, i.e., discarding the query.

Original: The goal of preference-based RL is to train an agent to perform behaviors desirable to a human teacher using as few feedback as possible.
Revised: The goal of preference-based RL is to train an agent to perform behaviors desirable to a human teacher using as few feedback as possible.

Original: In preference-based RL algorithms, a policy π φ and reward function (cid:98) r ψ are updated as follows:
Revised: In preference-based RL algorithms, a policy π φ and reward function (cid:98) r ψ are updated as follows:

Original: • Step 1 (agent learning) :
Revised: • Step 1 (agent learning) :

Original: The policy π φ interacts with environment to collect experiences and we update it using existing RL algorithms to maximize the sum of the learned rewards (cid:98) r ψ .
Revised: The policy π φ interacts with environment to collect experiences and we update it using existing RL algorithms to maximize the sum of the learned rewards (cid:98) r ψ .

Original: • Step 2 (reward learning) :
Revised: • Step 2 (reward learning) :

Original: The reward function (cid:98) r ψ is optimized based on the feedback received from a teacher.
Revised: The reward function (cid:98) r ψ is optimized based on the feedback received from a teacher.

Original: • Repeat Step 1 and Step 2 .
Revised: • Repeat Step 1 and Step 2 .

Original: To incorporate human preferences into reward learning , we optimize reward function (cid:98) r ψ as follows.
Revised: To incorporate human preferences into reward learning , we optimize reward function (cid:98) r ψ as follows.

Original: Following the Bradley-Terry model (Bradley & Terry, 1952), we ﬁrst model preference predictor of a pair of segments based on reward function (cid:98) r ψ as follows:
Revised: Following the Bradley-Terry model (Bradley & Terry, 1952), we ﬁrst model preference predictor of a pair of segments based on reward function (cid:98) r ψ as follows:

Original: [Equation]
Revised: [Equation]

Original: where σ i (cid:31) σ j denotes the event that segment i is preferable to segment j .
Revised: where σ i (cid:31) σ j denotes the event that segment i is preferable to segment j .

Original: Here, the intuition is that segments with more desirable behaviors should have higher predicted reward values from (cid:98) r ψ .
Revised: Here, the intuition is that segments with more desirable behaviors should have higher predicted reward values from (cid:98) r ψ .

Original: To align preference predictors of (cid:98) r ψ with labels received from human preferences, preference-based
Revised: To align preference predictors of (cid:98) r ψ with labels received from human preferences, preference-based

Original: RL algorithms translate updating reward function to a binary classiﬁcation problem.
Revised: RL algorithms translate updating reward function to a binary classiﬁcation problem.

Original: Speciﬁcally, the reward function (cid:98) r ψ parametrized by ψ is updated to minimize the following cross-entropy loss: 
Revised: Speciﬁcally, the reward function (cid:98) r ψ parametrized by ψ is updated to minimize the following cross-entropy loss: L Reward

Original: [Equation]
Revised: [Equation]

Original: where we are given a set D of segment pairs and corresponding human preferences.
Revised: where we are given a set D of segment pairs and corresponding human preferences.

Original: Once reward function (cid:98) r ψ has been optimized from human preferences, preference-based RL algorithms train RL agents with any existing RL algorithms, treating predicted rewards from (cid:98) r ψ as reward function returned from the environment.
Revised: Once reward function (cid:98) r ψ has been optimized from human preferences, preference-based RL algorithms train RL agents with any existing RL algorithms, treating predicted rewards from (cid:98) r ψ as reward function returned from the environment.

Original:  RUNE
Revised:  RUNE

Original: In this section, we present RUNE: R eward UN certainty for E xploration (Figure 1), which encourages human-guided exploration for preference-based RL.
Revised: In this section, we present RUNE: R eward UN certainty for E xploration (Figure 1), which encourages human-guided exploration for preference-based RL.

Original: The key idea of RUNE is to incentivize exploration by providing an intrinsic reward based on reward uncertainty.
Revised: The key idea of RUNE is to incentivize exploration by providing an intrinsic reward based on reward uncertainty.

Original: Our main hypothesis is that the reward uncertainty captures the novelty of human feedback, which can lead to useful behaviors for preference-based RL. 
Revised: Our main hypothesis is that the reward uncertainty captures the novelty of human feedback, which can lead to useful behaviors for preference-based RL. [Figure]

Original: 4. R EWARD UNCERTAINTY FOR EXPLORATION
Revised: 4. R EWARD UNCERTAINTY FOR EXPLORATION

Original: In preference-based RL, capturing the novelty of human feedback can be crucial for efﬁcient reward learning.
Revised: In preference-based RL, capturing the novelty of human feedback can be crucial for efﬁcient reward learning.

Original: To this end, we propose to utilize an intrinsic reward based on ensemble of reward functions.
Revised: To this end, we propose to utilize an intrinsic reward based on ensemble of reward functions.

Original: Speciﬁcally, for each timestep, the intrinsic reward is deﬁned as follows:
Revised: Speciﬁcally, for each timestep, the intrinsic reward is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) r std is the empirical standard deviation of all reward functions { (cid:98) r ψ i } Ni =1 .
Revised: where (cid:98) r std is the empirical standard deviation of all reward functions { (cid:98) r ψ i } Ni =1 .

Original: We initialize the model parameters of all reward functions with random parameter values for inducing an initial diversity.
Revised: We initialize the model parameters of all reward functions with random parameter values for inducing an initial diversity.

Original: Here, our intuition is high variance of reward functions indicates high uncertainty from human preferences, which means we collected less teacher preferences on those states and actions.
Revised: Here, our intuition is high variance of reward functions indicates high uncertainty from human preferences, which means we collected less teacher preferences on those states and actions.

Original: Therefore, in order to generate more informative queries, we encourage our agent to visit more unseen state-action pairs with respect to the learned reward functions.
Revised: Therefore, in order to generate more informative queries and improve conﬁdence in learned reward functions, we encourage our agent to visit more uncertain state-action pairs with respect to the learned reward functions.

Original: We remark that exploration based on ensembles has been studied in the literature (Osband et al., 2016; Chen et al., 2017; Pathak et al., 2019; Lee et al., 2021a).
Revised: We remark that exploration based on ensembles has been studied in the literature (Osband et al., 2016; Chen et al., 2017; Pathak et al., 2019; Lee et al., 2021a).

Original: For example, Chen et al.
Revised: For example, Chen et al.

Original: (2017) proposed an exploration strategy that considers both best estimates (i.e., mean) and uncertainty (i.e., variance) of Q-functions and Pathak et al.
Revised: (2017) proposed an exploration strategy that considers both best estimates (i.e., mean) and uncertainty (i.e., variance) of Q-functions and Pathak et al.

Original: (2019) utilized the disagreement between forward dynamics models.
Revised: (2019) utilized the disagreement between forward dynamics models.

Original: However, our method is different in that we propose an alternative intrinsic reward based on reward ensembles, which can capture the uncertainty from human preferences.
Revised: However, our method is different in that we propose an alternative intrinsic reward based on reward ensembles, which can capture the uncertainty from human preferences.

Original: 4. T RAINING OBJECTIVE BASED ON INTRINSIC REWARDS
Revised: 4. T RAINING OBJECTIVE BASED ON INTRINSIC REWARDS

Original: Once we learn reward functions { (cid:98) r ψ i } Ni =1 from human preferences, agent is usually trained with RL algorithm guided by extrinsic reward:[Equation] t ,
Revised: Once we learn reward functions { (cid:98) r ψ i } Ni =1 from human preferences, agent is usually trained with RL algorithm guided by extrinsic reward:

Original: [Equation]
Revised: [Equation]

Original: t ) , (4) [Figure] where (cid:98) r mean is the empirical mean of all reward functions { (cid:98) r ψ i } Ni =1 .
Revised: where (cid:98) r mean is the empirical mean of all reward functions { (cid:98) r ψ i } Ni =1 .

Original: To encourage exploration, we train a policy to maximize the sum of both extrinsic reward and intrinsic reward in equation 3:r total t := [Equation] [Equation] +
Revised: To encourage exploration, we train a policy to maximize the sum of both extrinsic reward and intrinsic reward in equation 3:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) , (5) where β t ≥ 0 is a hyperparameter that determines the trade off between exploration and exploitation at training time step t .
Revised: where β t ≥ 0 is a hyperparameter that determines the trade off between exploration and exploitation at training time step t .

Original: Similar to Seo et al.
Revised: Similar to Seo et al.

Original: (2021b), we use an exponential decay schedule for β t throughout training to encourage the agent to focus more on extrinsic reward from learned reward function predictions as training proceeds, i.e., β t = β 0 (1 − ρ ) t , where ρ is a decay rate.
Revised: (2021b), we use an exponential decay schedule for β t throughout training to encourage the agent to focus more on extrinsic reward from learned reward function predictions as training proceeds, i.e., β t = β 0 (1 − ρ ) t , where ρ is a decay rate.

Original: While the proposed intrinsic reward would converge to 0 as more feedback queries are collected during training, we believe our learned reward functions improve over training as we collect more feedback queries from teacher preferences.
Revised: While the proposed intrinsic reward would converge to 0 as more feedback queries are collected during training, we believe our learned reward functions improve over training as we collect more feedback queries from teacher preferences.

Original: The full procedure of RUNE is summarized in Algorithm 1.
Revised: The full procedure of RUNE is summarized in Algorithm 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We designed our experiments to answer the following questions:
Revised: We designed our experiments to answer the following questions:

Original:  • Can exploration methods improve the sample- and feedback-efﬁciency of preference-based RL algorithms?
Revised: [Figure] • Can exploration methods improve the sample- and feedback-efﬁciency of preference-based RL algorithms?

Original: • How does RUNE compare to other exploration schemes in preference-based RL setting?
Revised: • How does RUNE compare to other exploration schemes in preference-based RL setting?

Original: • How does RUNE inﬂuence query selection and reward learning?
Revised: • How does RUNE inﬂuence query selection and reward learning?

Original: 5. S ETUP
Revised: 5. S ETUP

Original: In order to verify the efﬁcacy of exploration in preference-based RL, we focus on having an agent solve a range of complex robotic manipulation skills from Meta-World (Yu et al., 2020)
Revised: In order to verify the efﬁcacy of exploration in preference-based RL [Figure] we focus on having an agent solve a range of complex robotic manipulation skills from Meta-World (Yu et al., 2020)

Original: (see Figure 2).
Revised: (see Figure 2).

Original: Similar to prior works (Christiano et al., 2017; Lee et al., 2021b;c), the agent learns to perform a task only by getting feedback from a scripted teacher that provides preferences between trajectory segments according to the ground truth reward function.
Revised: Similar to prior works (Christiano et al., 2017; Lee et al., 2021b;c), the agent learns to perform a task only by getting feedback from a scripted teacher that provides preferences between trajectory segments according to the ground truth reward function.

Original: Because the simulated human teacher’s preferences are generated by a ground truth reward, we measure the true average return of trained agents as evaluation metric.
Revised: Because the simulated human teacher’s preferences are generated by a ground truth reward, we measure the true average return of trained agents as evaluation metric.

Original: For our method, we consider a combination of RUNE and PEBBLE (Lee et al., 2021b), a preference-based RL algorithm that utilizes unsupervised pre-training and soft actor-critic (SAC) method (Haarnoja et al., 2018)
Revised: For our method, we consider a combination of RUNE and PEBBLE (Lee et al., 2021b), a preference-based RL algorithm that utilizes unsupervised pre-training and soft actor-critic (SAC) method (Haarnoja et al., 2018)

Original: (see Appendix 1 for more details about algorithm procedure and
Revised: (see Appendix A for more details about algorithm procedure and

Original: Appendix B for more experimental details).
Revised: Appendix B for more experimental details).

Original: For all experiments, we do not alter any hyperparameters ofthe original preference-based RL algorithm and train three ensemble  reward functions. More experimental details are in Appendix B.
Revised: For all experiments, we use original hyperparameters of preference-based RL algorithm; we train an ensemble of 3 reward functions according to Equation 2; we report mean and standard deviations across 5 runs respectively.

Original: 5. I MPROVING SAMPLE - EFFICIENCY
Revised: 5. I MPROVING SAMPLE - EFFICIENCY

Original: To evaluate sample-efﬁciency of our method, we compare to the following exploration methods:
Revised: To evaluate sample-efﬁciency of our method, we compare to the following exploration methods:

Original: [Figure]

Original: • State entropy maximization (StateEnt; Mutti et al. 2021; Liu & Abbeel 2021):
Revised: • State entropy maximization (StateEnt; Mutti et al. 2021; Liu & Abbeel 2021):

Original: Maximizing the entropy of state visitation distribution H ( s ) .
Revised: Maximizing the entropy of state visitation distribution H ( s ) .

Original: We utilize a particle estimator (Singh et al., 2003), which approximates the entropy by measuring the distance between k-nearest neighbors ( k -NN) for each state.
Revised: We utilize a particle estimator (Singh et al., 2003), which approximates the entropy by measuring the distance between k-nearest neighbors ( k -NN) for each state.

Original: • Disagreement (Pathak et al.
Revised: • Disagreement (Pathak et al.

Original: ): Maximizing disagreement proportional to variance in predictions from ensembles V ar { g i ( s t
Revised: ): Maximizing disagreement proportional to variance in predictions from ensembles V ar { g i ( s t

Original: +1 | s t , a t ) } Ni =1 .
Revised: +1 | s t , a t ) } Ni =1 .

Original: We train an ensemble of forward dynamics models to predict ground truth next state from current state and action g i ( s t +1 | s t , a t ) by minimizing sum of prediction errors, i[Figure] e. [Figure]
Revised: We train an ensemble of forward dynamics models to predict ground truth next state from current state and action g i ( s t +1 | s t , a t ) by minimizing sum of prediction errors, i

Original: Ni [Equation]{+e. (cid:80) N i =
+}
Revised: {+e. (cid:80) N i =
+}

Original: 1 || g
Revised: || g

Original: i ( s t +1 | s t , a t ) − s t +

Revised: i ( s t +1 | s t , a t ) − s t +

Original: || 2 .
Revised: || 2 .

Original: • ICM
Revised: • ICM

Original: (Pathak et al.
Revised: (Pathak et al.

Original: Maximizing intrinsic reward proportional to prediction error
Revised: Maximizing intrinsic reward proportional to prediction error

Original: ||
Revised: ||

Original: g
Revised: g

Original: ( s t +1 | s t , a t ) − s t +

Revised: ( s t +1 | s t , a t ) − s t +

Original: || 2 .
Revised: || 2 .

Original: We train a single dynamics model to predict ground truth next state from current state and action g ( s t +1 | s t , a t ) via regression.
Revised: We train a single dynamics model to predict ground truth next state from current state and action g ( s t +1 | s t , a t ) via regression.

Original: For all methods we consider, we carefully tune a range of hyperparameters and report the best results.
Revised: For all methods we consider, we carefully tune a range of hyperparameters and report the best results.

Original: In particular, we consider β 0 = 0 .
Revised: In particular, we consider β 0 = 0 .

Original: 05 and ρ ∈ { 0 .
Revised: 05 and ρ ∈ { 0 .

Original: 001 , 0 . 0001 , 0 . 00001 } for all exploration methods, and k ∈ { 5 , 10 } for state entropy based exploration.
Revised: 001 , 0 . 0001 , 0 . 00001 } for all exploration methods, Task Queries Method

Revised: Final Convergence Success Rate

Revised: Door Close

Original: We provide more details about training and implementation in Appendix B.
Revised: PEBBLE 1 ( ± 0) PEBBLE + RUNE 1 ( ± 0) PEBBLE 0.8 ( ± 0.4) PEBBLE + RUNE 1 ( ± 0)

Revised: Door Open

Revised: PEBBLE 1 ( ± 0)

Revised: PEBBLE + RUNE 1 ( ± 0)

Revised: PEBBLE 0.9 ( ± 0.2)

Revised: PEBBLE + RUNE 1 ( ± 0)

Revised: Drawer Open

Revised: PEBBLE 0.98 ( ± 0.08)

Revised: PEBBLE + RUNE 1 ( ± 0)

Revised: PEBBLE 0.94 ( ± 0.08)

Revised: PEBBLE + RUNE 0.99 ( ± 0.02)

Revised: Sweep Into

Revised: PEBBLE 0.8 ( ± 0.4)

Revised: PEBBLE + RUNE 1 ( ± 0)

Revised: PEBBLE 0.8 ( ± 0.08)

Revised: PEBBLE + RUNE 0.9 ( ± 0.14)

Revised: Window Close

Revised: PEBBLE 0.94 ( ± 0.08)

Revised: PEBBLE + RUNE 1 ( ± 0)

Revised: PEBBLE 0.86 ( ± 0.28)

Revised: PEBBLE + RUNE 0.99 ( ± 0.02)

Revised: Door Unlock

Revised: PEBBLE 0.66 ( ± 0.42)

Revised: PEBBLE + RUNE 0.8 ( ± 0.4)

Revised: PEBBLE 0.64 ( ± 0.45)

Revised: PEBBLE + RUNE 0.8 ( ± 0.4) and k ∈ { 5 , 10 } for state entropy based exploration.

Revised: For all experiements, we train an ensemble ofreward functions according to Equation 2 respectively.

Revised: We provide more details about training and implementation in Appendix B.

Original: Figure 3 shows the learning curves of PEBBLE with various exploration methods.
Revised: Figure 3 shows the learning curves of PEBBLE with various exploration methods.

Original: First, we remark that previous exploration methods (i.e., StateEnt, Disagree and ICM), which encourage agents to visit novel states, consistently improve sample-efﬁciency of PEBBLE on various tasks.
Revised: First, we remark that previous exploration methods (i.e., StateEnt, Disagree and ICM), which encourage agents to visit novel states, consistently improve sample-efﬁciency of PEBBLE on various tasks.

Original: In particular, the proposed exploration method (i.e., RUNE) consistently exhibits superior sample efﬁciency in most tasks.
Revised: In particular, the proposed exploration method (i.e., RUNE) consistently exhibits superior sample efﬁciency in most tasks.

Original: This further demonstrates that exploration based on reward uncertainty is suitable for preference-based RL algorithms, as such it can capture the novelty of human feedback.
Revised: This further demonstrates that exploration based on reward uncertainty is suitable for preference-based RL algorithms, as such it can capture the novelty of human feedback.

Original: We also emphasize the simplicity and efﬁciency of RUNE compared to other existing schemes (such as ICM and Disagree) because our method does not introduce additional model architectures (e.g., ensemble of forward dynamics models).
Revised: We also emphasize the simplicity and efﬁciency of RUNE compared to other existing schemes (such as ICM and Disagree) because our method does not introduce additional model architectures (e.g., ensemble of forward dynamics models).

Original: 5. I MPROVING FEEDBACK - EFFICIENCY
Revised: 5. I MPROVING FEEDBACK - EFFICIENCY

Original: In this section, we also verify whether the proposed exploration method can improve the feedbackefﬁciency of preference-based RL.
Revised: In this section, we also verify whether our proposed exploration method can improve the feedbackefﬁciency of preference-based RL.

Original: As shown in Figure 4, RUNE with fewer total queries can achieve competitive or superior performance to PEBBLE with more queries.
Revised: As shown in Table 1, we compare performance PEBBLE and RUNE using different budgets of feedback queries during training. With fewer total queries, we stop asking for human preferences earlier in the middle of training.

Original: Figure 3 suggests that utilizing the same number of feedback queries, RUNE can learn to successfully solve tasks within shorter training time.
Revised: We use asymptotic success rate evaluated at the end of training as evaluation metric. Table 1 suggests that RUNE achieves consistently better asymptotic performance using fewer number of human feedback; additionally

Original: Once agents learn the policy to successfully solve the task, we no longer need to collect human preferences for reward learning. Therefore, by stopping to ask for teacher feedback in the middle of training, RUNE is able to improve feedback-efﬁciency of  preference-based RL algorithms.
Revised: RUNE has more robust performance with respect to different budgets of available human feedbacks. This shows potential of exploration in scaling preference-based RL to real world scenarios where human feedback are usually expensive and time-consuming to obtain. We report corresponding

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: learning curves to show better sample efﬁciency of RUNE compared to PEBBLE in Figure 7 of Appendix C.

Original: 5. A BLATION STUDY
Revised: 5. A BLATION STUDY

Original: Ensemble size.
Revised: Ensemble size.

Original: As intrinsic rewards of RUNE only depend on learned reward functions, we investigate the effects of different number of reward function ensembles.
Revised: As intrinsic rewards of RUNE only depend on learned reward functions, we investigate the effects of different number of reward function ensembles.

Original: In particular, we consider { 3 , 5 , 7 } number of reward function ensembles.
Revised: In particular, we consider { 3 , 5 , 7 } number of reward function ensembles.

Original: In Figure 5(a), RUNE can achieve better sample-efﬁciency in preference-based RL while learning fewer number of reward functions ensembles.
Revised: In Figure 4(a), RUNE can achieve better sample-efﬁciency in preference-based RL while learning fewer number of reward functions ensembles.

Original: This shows potentials of RUNE speciﬁcally suitable for preference-based RL in improving compute efﬁciency because it reduces the necessity of training additional reward functions architectures. 
Revised: This shows potentials of RUNE speciﬁcally suitable for preference-based RL in improving compute efﬁciency because it reduces the necessity of training additional reward functions architectures. Number of queries per feedback session.

Revised: Human preferences are usually expensive and timeconsuming to collect.

Revised: As shown in Table 1, RUNE is able to achieve better asymptotic performance under different budgets of total human preferences.

Revised: We further investigate the effects of different queries in each feedback session on performance of PEBBLE and RUNE.

Original:  Disagreement level of selected feedback queries.
Revised: In particular, we consider { 10 , 50 } number of queries per feedback session equally spread out throughout training. In Figure 4(b), as 80% of feedbacks are eliminated, asymptotic success rate of PEBBLE baseline largely drops, while in contrast RUNE remains stable asymptotic performance. This shows RUNE is more robust to different number of available feedback queries and thus a suitable exploration method speciﬁcally for preference-based RL setting. Disagreement level of selected feedback queries.

Original: To improve the feedback-efﬁciency, many preference-based RL algorithms (Christiano et al., 2017; Lee et al., 2021c) utilize an uncertainty sampling scheme, which selects queries with high uncertainty.
Revised: To improve the feedback-efﬁciency, many preference-based RL algorithms (Christiano et al., 2017; Lee et al., 2021c) utilize an uncertainty sampling scheme, which selects queries with high uncertainty.

Original: To understand whether RUNE can change query selection, we measure average uncertainty value of 50 selected queries per feedback session in Door Open.
Revised: To understand whether RUNE can change query selection, we measure average uncertainty value of 50 selected queries per feedback session in Door Open.

Original: Speciﬁcally, we measure the disagreement value of a segment pair ( σ 0 , σ 1 ) is introduced in Christiano et al.
Revised: Speciﬁcally, we measure the disagreement value of a segment pair ( σ 0 , σ 1 ) is introduced in Christiano et al.

Original: For each learned reward function (cid:98) r ψ i , we model a preference predictor  level of ( σ 0 , σ 1 ) is deﬁned as variance of preference predictions from learned reward functions, i.e. V ar { P ψ 1 [ σ 1 (cid:31) σ 0 ] , ..., P ψ N [ σ 1 (cid:31) σ 0 ] } .
Revised: For each learned reward function (cid:98) r ψ i , we model a preference predictor P ψ i [ σ 1 (cid:31) σ 0 ] between ( σ 0 , σ 1 ) from (cid:98) r ψ i based on Equation 1. An ensemble of reward functions { (cid:98) r ψ i } Ni =1 predicts preferences { P ψ 1 [ σ 1 (cid:31) σ 0 ] , ..., P ψ N [ σ 1 (cid:31) σ 0 ] } from ( σ 0 , σ 1 ) . Disagreement level of ( σ 0 , σ 1 ) is deﬁned as variance of preference predictions from learned reward functions, i.e. V ar { P ψ 1 [ σ 1 (cid:31) σ 0 ] , ..., P ψ N [ σ 1 (cid:31) σ 0 ] } .

Original: Higher disagreement level means that learned reward functions are uncertain to recognize which behavior is preferred in ( σ 0 , σ 1 ) .
Revised: Higher disagreement level means that learned reward functions are uncertain to recognize which behavior is preferred in ( σ 0 , σ 1 ) .

Original: As in Figure 5(b), we found that average disagreement of selected queries generated by both PEBBLE and RUNE increase in early training and decrease at the end.
Revised: As in Figure 4(c), we found that average disagreement of selected queries generated by both PEBBLE and RUNE increase in early training and decrease at the end.

Original: While with RUNE, the curve decreases slightly faster in the middle of training.
Revised: While with RUNE, the curve decreases slightly faster in the middle of training.

Original: This observation shows the possibility of faster reward learning with RUNE.
Revised: This observation shows the possibility of faster reward learning with RUNE.

Original: Since learned reward functions should be conﬁdent in predicting preference between pair of segments at the end of training, faster decrease with RUNE should indicate better reward learning in the middle of the training.Quality of learned reward functions.
Revised: Since learned reward functions should be conﬁdent in predicting pref- erence between pair of segments at the end of training, faster decrease with RUNE should indicate better reward learning in the middle of the training.

Original: A signiﬁcant challenge in preference-based RL is to learn reliable reward functions from limited human feedback.

Original: Figure 5(c) shows the learned reward functions optimized by PEBBLE with RUNE.

Original: Because we bound the output of the reward function using Tanh function, the scale is different with the ground truth reward.

Original: We observe well-alignment between learned reward functions and true reward values.

Original: Particularly for ﬂuctuations of ups and downs in true rewards, RUNE can learn such patterns and predict with reliable reward values.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we present RUNE, a simple and efﬁcient exploration method for preference-based

Original: RL algorithms.

Original: To improve sample- and feedback-efﬁciency of preference-based RL, different from previous works, we investigate the beneﬁts of exploration methods in preference-based RL.

Original: We show the signiﬁcant potential of incorporating intrinsic rewards to drive exploration because it improves sample-efﬁciency of preference-based RL.

Original: For our proposed exploration scheme RUNE, we show that it is useful for preference-based RL because it showcases consistently superior performance in both sample- and feedback-efﬁciency compared to other existing exploration methods.

Original: Here we emphasize that RUNE takes advantage of information in reward functions learned from human feedback, to measure the novelty of states and actions for exploration.

Original: This is different from existing estimates of uncertainty for exploration, as our method in particular encourages exploration aligned to teacher preferences.

Original: In conclusion, we hope that our work could demonstrate the potential of exploration to improve sample- and feedback-efﬁciency of preference-based RL, and to encourage future works to develop novel exploration methods guided by human feedback.

Original: A A LGORITHM D ETAILS

Original: RUNE.

Original: We provide the full procedure for RUNE with PEBBLE Lee et al.

Original: (2021b) in Algorithm 1.

Original: 1: Initialize

Original: B 3:

Original: 4: // P OLICY LEARNING 5: for each training step t do 6: //

Original: [-R EWARD LEARNING 7: if iteration % K == 
-]
Revised: [-R EWARD LEARNING 7: if iteration % K == 
-]

Original: then 8:

Original: Generate queries from replay buffer { ( σ 0 , σ 1 ) } N query i = ∼ B and corresponding human feedback ( y i ) N query i = 9:

Original: Update reward model (cid:98) r ψ according to equation10: // R ELABEL BOTH EXTRINSIC

Original: AND

Original: INTRINSIC REWARD 11:

Original: Relabel entire replay buffer B using (cid:98) r ψ 12:

Original: Relabel extrinsic reward r ext ψ ( s j , a j ) =

Original: ˆ r mean ( s j , a j ) 13:

Original: Relabel intrinsic reward r int ψ ( s j , a j ) =

Original: ˆ r std ( s j , a j ) 14: end if 15: // U PDATE POLICY 16: for each gradient step do 17:

Original: Sample minibatch from replay buffer { ( s j , a j , s j +1 , r ext ψ ( s j , a j ) , r int ψ ( s j , a j ) }

Original: Bj =1 ∼ B 18:

Original: Update β t ← β 0 (1 − ρ ) t 19:

Original: Let r total j ← r ext j + β t · r int j 20:

Original: Update φ with transitions { ( s j , a j , s j +1 , r total ψ ( s j , a j ) } Bj = 21: end for 22: // I NTERACTION WITH ENVIRONMENT 23: for each timestep t do 24:

Original: Collect s t +1 by taking a t ∼ π φ ( a t

Original: | s t ) 25:

Original: Store transitions B ← B ∪ { ( s t , a t , s t

Original: [Table]

Original: 1 , (cid:98) r ψ ( s t , a t )) } 26: end for 27: end for

Original: [Table]

Original: [Table]

Original: B E XPERIMENTAL D ETAILS B. T RAINING D ETAILS
Revised: B E XPERIMENTAL D ETAILS B. T RAINING D ETAILS

Original: For our method, we use the publicly released implementation repository of the PEBBLE algorithm ( https://github.com/pokaxpoka/B_Pref ) with the full list of hyperparameters speciﬁed in Table 1 and the SAC algorithm ( https://github.com/denisyarats/pytorch_sac ) with the full list of hyperparameters speciﬁed in Table 2.
Revised: For our method, we use the publicly released implementation repository of the PEBBLE algorithm ( https://github.com/pokaxpoka/B_Pref ) with the full list of hyperparameters speciﬁed in Table 2 and the SAC algorithm ( https://github.com/denisyarats/pytorch_sac ) with the full list of hyperparameters speciﬁed in Table 3.

Original: B. I MPLEMENTATION D ETAILS OF E XPLORATION M ETHODS
Revised: B. I MPLEMENTATION D ETAILS OF E XPLORATION M ETHODS

Original: For all exploration methods, we follow implementation of the publicly release repository of RE ( https://github.com/younggyoseo/RE3 ) to update policy with r total =
Revised: For all exploration methods, we follow implementation of the publicly release repository of RE ( https://github.com/younggyoseo/RE3 ) to update policy with r total =

Original: r ext + β t · r int .
Revised: r ext + β t · r int .

Original: RUNE.
Revised: RUNE.

Original: We emphasize one simplicity in the implementation of RUNE.
Revised: We emphasize one simplicity in the implementation of RUNE.

Original: Following relabeling technique in Lee et al.
Revised: Following relabeling technique in Lee et al.

Original: (2021b), after each update in reward learning step, we relabel the entire replay buffer B to store both predicted reward functions learned from human feedback (extrinsic rewards r ext ), and uncertainty in updated reward functions (intrinsic rewards r int in RUNE).
Revised: (2021b), after each update in reward learning step, we relabel the entire replay buffer B to store both predicted reward functions learned from human feedback (extrinsic rewards r ext ), and uncertainty in updated reward functions (intrinsic rewards r int in RUNE).

Original: This is because our learned reward functions parameters remain unchanged during RL training or agent interaction with the environment between subsequent feedback sessions.
Revised: This is because our learned reward functions parameters remain unchanged during RL training or agent interaction with the environment between subsequent feedback sessions.

Original: Thus, in RL training, r total = r ext + β t
Revised: Thus, in RL training, r total = r ext + β t

Original: r int can be directly taken from replay buffer.
Revised: r int can be directly taken from replay buffer.

Original: As for hyperparameter related to exploration, we consider β 0 = 0 .
Revised: As for hyperparameter related to exploration, we consider β 0 = 0 .

Original: 05 , ρ ∈ { 0 .
Revised: 05 , ρ ∈ { 0 .

Original: State Entropy Maximization.
Revised: State Entropy Maximization.

Original: Seo et al.
Revised: Seo et al.

Original: (2021b)
Revised: (2021b)

Original: We use intrinsic reward r int ( s t )
Revised: We use intrinsic reward r int ( s t )

Original: =
Revised: =

Original: || s
Revised: || s

Original: t − s k -NN t ||followed Seo et al.
Revised: t − s k -NN t ||followed Seo et al.

Original: (2021b)
Revised: (2021b)

Original: and Liu & Abbeel (2021).
Revised: and Liu & Abbeel (2021).

Original: We use raw state space to compute entropy values as intrinsic rewards.
Revised: We use raw state space to compute entropy values as intrinsic rewards.

Original: We use code for unsupervised pre-training from the publicly release implementation repository of PEBBLE algorithm ( https://github.com/pokaxpoka/B_
Revised: We use code for unsupervised pre-training from the publicly release implementation repository of PEBBLE algorithm ( https://github.com/pokaxpoka/B_

Original: Pref ) to compute k -NN and estimate state entropy.
Revised: Pref ) to compute k -NN and estimate state entropy.

Original: As the size of replay buffer grows significantly as training time increases, we sample a random mini-batch B (cid:48) from replay buffer B and compute k -NN with respect to random mini-batch.
Revised: As the size of replay buffer grows significantly as training time increases, we sample a random mini-batch B (cid:48) from replay buffer B and compute k -NN with respect to random mini-batch.

Original: We use minibatch size 512 , which is reported as a default hyperparameter in Seo et al.
Revised: We use minibatch size 512 , which is reported as a default hyperparameter in Seo et al.

Original: (2021b).
Revised: (2021b).

Original: As for hyperparameters related to exploration, we consider β 0 = 0 .
Revised: As for hyperparameters related to exploration, we consider β 0 = 0 .

Original: 05 , ρ ∈ { 0 .
Revised: 05 , ρ ∈ { 0 .

Original: 001 , 0 . 0001 , 0 . 00001 } , and k ∈ { 5 , 10 } .
Revised: 001 , 0 . 0001 , 0 . 00001 } , and k ∈ { 5 , 10 } .

Original: Disagreement.
Revised: Disagreement.

Original: Pathak et al.
Revised: Pathak et al.

Original: We train an ensemble of 5 forward dynamics model g i , following original results in Pathak et al.
Revised: We train an ensemble of 5 forward dynamics model g i , following original results in Pathak et al.

Original: These dynamics models are trained to predict next state s t +1 based on current state and action ( s t , a t ) , i.e. minimize (cid:80) 5 i =

Revised: These dynamics models are trained to predict next state s t +1 based on current state and action ( s t , a t ) , i.e. minimize (cid:80) 5 i =

Original: ||
Revised: ||

Original: g
Revised: g

Original: i
Revised: i

Original: ( s t +1 | s t , a t ) − s t +

Revised: ( s t +1 | s t , a t ) − s t +

Original: || 2 .
Revised: || 2 .

Original: We use intrinsic reward proportional to variance in ensemble of predictions r int ( s t , a t ) =
Revised: We use intrinsic reward proportional to variance in ensemble of predictions r int ( s t , a t ) =

Original: V ar { g i ( s t +1 | s t , a t ) } Ni =1 .
Revised: V ar { g i ( s t +1 | s t , a t ) } Ni =1 .

Original: For consistency we use same neural network architecture and optimization hyperparameters as reward functions speciﬁed in Table 1.
Revised: For consistency we use same neural network architecture and optimization hyperparameters as reward functions speciﬁed in Table 2.

Original: We follow implementation of Disagreement agent in publicly released repository URLB ( https://anonymous.4open.
Revised: We follow implementation of Disagreement agent in publicly released repository URLB ( https://anonymous.4open.

Original: science/r/urlb ).
Revised: science/r/urlb ).

Original: As for hyperparameter related to exploration, we consider β 0 = 0 .
Revised: As for hyperparameter related to exploration, we consider β 0 = 0 .

Original: 05 , ρ ∈ { 0 .
Revised: 05 , ρ ∈ { 0 .

Original: ICM.
Revised: ICM.

Original: Pathak et al.
Revised: Pathak et al.

Original: We train a single forward dynamics model g to predict next state s t +1 based on current state and action ( s t , a t ) , i.e. minimize || g
Revised: We train a single forward dynamics model g to predict next state s t +1 based on current state and action ( s t , a t ) , i.e. minimize || g

Original: ( s t +1 | s t , a t ) − s t +

Revised: ( s t +1 | s t , a t ) − s t +

Original: || 2 .
Revised: || 2 .

Original: We use intrinsic reward proportional to prediction error of trained dynamics model, i.e. r int ( s t , a t ) =
Revised: We use intrinsic reward proportional to prediction error of trained dynamics model, i.e. r int ( s t , a t ) =

Original: || g ( s t +1 | s t , a t ) − s t +

Revised: || g ( s t +1 | s t , a t ) − s t +

Original: || 2 .
Revised: || 2 .

Original: For consistency we use same neural network architecture and optimization hyperparameters as reward functions speciﬁed in Table 1.
Revised: For consistency we use same neural network architecture and optimization hyperparameters as reward functions speciﬁed in Table 2.

Original: We follow implementation of ICM agent
Revised: We follow implementation of ICM agent

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in publicly released repository URLB ( https://anonymous.4open.science/r/urlb ).
Revised: in publicly released repository URLB ( https://anonymous.4open.science/r/urlb ).

Original: As for hyperparameter related to exploration, we consider β 0 = 0 .
Revised: As for hyperparameter related to exploration, we consider β 0 = 0 .

Original: 05 , ρ ∈ { 0 .
Revised: 05 , ρ ∈ { 0 .

Original: C A DDITIONAL E XPERIMENT R ESULTS
Revised: C A DDITIONAL E XPERIMENT R ESULTS

Original: Improving Sample-Efﬁciency.
Revised: Improving Feedback-Efﬁciency.

Original: We provide additional experiment result to Figure 3 for more tasks in Meta-World.
Revised: We provide additional experimental results on various tasks from Meta-World benchmark Yu et al.

Revised: (2020) using different budgets of feedback queries during training.

Revised: We report asymptotic evaluation success rate at the end of training in Table 1.

Revised: Here we report corresponding learning curves and additional results of wider choices of total feedback queries.

Revised: We observe RUNE performs consistently at least same or better than PEBBLE baseline in different tasks with different budgets of feedback queries.

Original:  As shown in Figure 6, RUNE outperforms other existing exploration baselines measured on the episode return of Pick Place.
Revised: Improving Sample-Efﬁciency. As shown in Figure 5, RUNE outperforms other existing exploration baselines measured on the episode return of Pick Place.

Original: This shows applicability of our method on harder tasks.
Revised: This shows applicability of our method on harder tasks.

Original: Disagreement Level of Selected Queries.
Revised: Disagreement Level of Selected Queries.

Original: We provide additional analysis of selected queries by
Revised: We provide additional analysis of selected queries by

Original: PEBBLE and RUNE in more environments in Figure 6.
Revised: PEBBLE and RUNE in more environments in Figure 5.

Original: There is no signiﬁcant difference between two curves, so RUNE exploration and PEBBLE generate feedback queries with similar average disagreement values throughout training.
Revised: There is no signiﬁcant difference between two curves, so RUNE exploration and PEBBLE generate feedback queries with similar average disagreement values throughout training.

Original: Reward Functions Learned by RUNE.
Revised: Reward Functions Learned by RUNE.

Original: We provide additional analysis of reward functions learned by RUNE, compared to true reward function deﬁned by Meta-World environments.
Revised: We provide additional analysis of reward functions learned by RUNE, compared to true reward function deﬁned by Meta-World environments.

Original: Figure 7 demonstrates in all tasks we consider, learned reward functions optimized by RUNE is well-aligned to true reward from teacher feedback, because it can capture diverse patterns of true reward values from the environment. 
Revised: Figure 6 demonstrates in all tasks we consider, learned reward functions optimized by RUNE is well-aligned to true reward from teacher feedback, because it can capture diverse patterns of true reward values from the environment. [Figure] [Figure] [Figure]

Revised: a t e (

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]