Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning
Abstract
Setting up a well-designed reward function has been challenging for many reinforcement learning applications.
Preference-based reinforcement learning (PbRL) provides a new framework that avoids reward engineering by leveraging human preferences (i.e., preferring apples over oranges) as the reward signal.
Therefore, improving the efﬁcacy of data usage for preference data becomes critical.
In this work, we propose Meta-Reward-Net (MRN), a data-efﬁcient PbRL framework that incorporates bi-level optimization for both reward and policy learning.
The key idea of MRN is to adopt the performance of the Q-function as the learning target.
Based on this, MRN learns the Q-function and the policy in the inner level while updating the reward function adaptively according to the performance of the
Q-function on the preference data in the outer level.
Our experiments on robotic simulated manipulation tasks and locomotion tasks demonstrate that MRN outperforms prior methods in the case of few preference labels and signiﬁcantly improves data efﬁciency, achieving state-of-the-art in preference-based RL.
Ablation studies further demonstrate that MRN learns a more accurate Q-function compared to prior work and shows obvious advantages when only a small amount of human feedback is available.
The source code and videos of this project are released at https://sites.google.com/view/meta-reward-net 1 .
Introduction
In recent years, reinforcement learning has achieved great success in solving complex sequential decision-making tasks, such as gaming AI [1, 2, 3, 4, 5], autonomous driving [6, 7], robotic manipulation [8, 9], order dispatching [10], population biology [11], quantitative ﬁnance [12], automation system [13, 14, 15], etc.
For common decision making tasks, the goal of the agent is to maximize the cumulative reward.
However, one central challenge to reinforcement learning is how to design reward functions.
On the one hand, the quality of the designed reward function largely depends on the problem solver’s understanding of task objective, operation logic, and related background knowledge.
Even excellent engineers still need plenty of time to try different methods in complex RL tasks.
On the other hand, there is a problem that the agent might hack the reward function.
In policy learning, the agent utilizes the defect of the reward function to maximize the cumulative reward instead of solving the expected task.
Besides, in human-involved scenarios, the objective of the agent is to maximize happiness of humans, making it hard to specify a reward function.
implicit derivative !, #, ! !
[Figure]
human preferences 2" ∙ $ℒ meta ('(()))
$ '( % ⋅ $, & (-,/) $( ⋅ $ ̂1 ' (-,/) $)
[Figure]
③ Previous work has provided some ideas to avoid directly constructing reward functions, such as imitation learning [16]. Although imitation learning has an excellent performance in some tasks, its performance is difﬁcult to surpass human level. Preference-based reinforcement learning is a more ﬂexible and convenient alternative method.
A human expert can easily give a preference to a trajectory pair of the agent, which implies the desired behaviors, that is to say, the goal of the task. In preference-based RL, the reward function is learned through preferences given by the human teacher on trajectory pairs. The feedback from humans guides the agent to achieve speciﬁed goals or learn desired behaviors. Recent work [17, 18, 19] shows that providing sufﬁcient feedback achieves better performance under this setting.
However, preference data queried from human experts is expensive.
In reality, there is likely to be only a small amount of data available.
With the limitation of the amount of feedback, previous methods perform poorly or even do not work.
Meanwhile, recent work on semi-supervised learning takes success in computer vision [20], pseudo labels can be adjusted according to the performance of the student and further improve student’s performance.
Inspired by this, we consider utilizing the performance from the Q-function on the preference labels for reward learning via bi-level optimization.
By considering the implicit differentiation in reward learning, the reward function is aware of the accuracy of the Q-function, which is beneﬁcial for the learning of the Q-function and the policy.
In this work, we focus on the efﬁciency of feedback in the learning of preference-based RL.
We use bilevel optimization method for reward learning which contains two loops.
In the inner loop, we update the Q-function and the policy via the reward function, while optimizing the reward function according to the performance of the Q-function on the preference data in the outer loop.
Our experiments demonstrate that our method considerably improves feedback efﬁciency.
Besides, the results of further evaluation show that MRN exceeds other methods by a large margin when few preference labels are available and learns a more accurate Q-function.
In summary, the main contributions of our work are three-fold.
First, we propose a new preferencebased RL algorithm Meta-Reward-Net, which utilizes bi-level optimization methods in reward learning.
Second, we show that MRN substantially improves the feedback efﬁciency and outperforms preference-based RL baselines on a variety of robotic simulated manipulation tasks from Metaworld [21] and locomotion tasks from DeepMind Control Suite (DMControl)
Last, we demonstrate that beneﬁting from bi-level optimization, the advantage becomes obvious compared to PEBBLE when only few preference labels are provided.
We also show that considering the performance of the Q-function in reward learning is beneﬁcial for agent learning, leading to a more accurate Q-function and a better policy.
Related Work
Reinforcement Learning.
Reinforcement learning has gradually become an effective and powerful method to solve complex sequential decision-making problems.
In recent years, much prior work has proven the capacity of reinforcement learning.
Reinforcement learning applications include video games [1, 2, 3], robot control [24], manipulation [14, 15], board games [4], autonomous driving [6] and so on.
In addition, there are many applications of reinforcement learning in the ﬁelds of computer vision [25], natural language processing [26], recommendation system [27, 28] and game theory [29, 30].
In the framework of reinforcement learning, an agent obtains data from interaction with the environment to optimize policy to maximize the expected return.
In this process, the reward function plays a crucial role.
Our method differs in that we do not assume that there is a reward function from engineering.
Instead, we use the preferences of humans to guide the agent towards desired behaviors.
Preference-based Reinforcement Learning.
Prior work has successfully trained the agent to complete speciﬁc tasks or achieve goals through the feedback from the teacher.
[31] provides a general learning framework of preference-based reinforcement learning.
[32] utilizes expert demonstrations and human preference feedback, initializes with imitation learning policy, and further improves the performance of the policy.
In this work, we mainly focus on feedback efﬁciency in the learning of preference-based RL.
Much previous work [33, 34, 35, 36, 37] considered learning reward function from the most informative data to be consistent with the human preferences.
Recently, several feedback-efﬁcient preference-based RL algorithms have been proposed.
PEBBLE [17] combines unsupervised pre-training and the technique of relabeling experience to improve feedback efﬁciency.
SURF [18] learns the reward function by semi-supervised learning and data augmentation.
RUNE [19] facilitates exploration via reward uncertainty to reduce the number of preference labels.
However, these methods only focus on guiding reward learning through supervised loss between human preferences and estimated preferences.
We have a different approach in that in addition to utilizing the supervised loss, we also consider the performance of the Q-function on the labeled preference dataset to assist reward learning, thus beneﬁcial for agent learning.
By introducing this extra signal,
MRN gets more task-related information from human preferences to improve feedback efﬁciency remarkably.
Bi-level Optimization.
In computer vision, there are several bi-level optimization algorithms that achieve great success.
Meta-Weight-Net [38] provides weights for samples from an unbalanced dataset in a bi-level manner, while Meta Label Correction [39] view this problem as a label correction problem.
Meta Pseudo Labels [20] combines Pseudo Labels methods and bi-level optimization to generate high quality pseudo labels.
In reinforcement learning, LIRPG [40] learns parametric intrinsic rewards and combines them with extrinsic rewards to improve the performance of policy-gradientbased learning methods.
PR2 [41] and GR2 [42] propose to conduct multi-level recursive reasoning for multi-agent interactions.
Bi-level method has also been studied for multi-agent coordination [43].
LIIR [44] learns an intrinsic reward function for each agent to achieve better cooperation in multiagent reinforcement learning.
CAIL [45] proposes to re-weight demonstrations of different optimality in imitation learning.
[46] proposes a scalable meta-gradient framework to learn useful intrinsic reward functions to capture a rich form of knowledge across over multiple episodes.
Neural autocurricula [47] learns to create an autonomous and diverse agenda [48, 49] for solving two-player zero-sum games.
Our method uses a similar bi-level method for reward learning.
To the best of our knowledge, we are the ﬁrst to introduce bi-level optimization into preference-based RL.
Meta-learning from Demonstrations.
Previous research has many effects on meta-learning from demonstrations.
[50] presents a new framework, PEMIRL, which enables meta-learning of rewards by leveraging unstructured demonstrations from multi-task.
[51] presents a meta-learning method for one-shot imitation, which combine meta-learning with imitation learning.
Speciﬁcally, this approach enables the agent to learn how to learn more efﬁciently and makes it acquire new skills from a single demonstration.
[52] proposes a meta-learning algorithm that can utilize a small amount of trial experience to learn new behaviors illustrated by a single demonstration.
[53] introduces an inverse reinforcement learning algorithm, which optimizes cost functions by differentiating at the inner optimization step.
Unlike prior work, our method focuses on preference data rather than demonstrations, which is usually more expensive.
Preliminaries
Preference-based Reinforcement Learning.
In standard reinforcement learning framework, a ﬁnite
Markov decision process (MDP) can be presented as a tuple of h S , A , R, P,  i , which consists of state space S , action space A , transition function, reward function, and discount factor.
P ( s 0 | s, a ) represents stochastic dynamics of the environment, which is the probability of selecting action a to transit to s 0 in a given state s .
R ( s, a ) represents the reward obtained by selecting an action a in a given state s .
The policy ⇡ ( a | s ) is a mapping from state space to action space.
The objective of the agent is to collect trajectories from interaction with the environment to maximize the expected return.
In the general preference-based RL from [31], there is no reward function from reward engineering and a reward function estimator b r should be learned to be consistent with preferences from the human expert.
Speciﬁcally, a segment  is a sequence of states and actions which is ( s t +1 , a t +1 , · · · , s t + k , a t + k ) .
Human expert provides a preference y on given two segments (  0 ,  1 ) and y is the distribution over { 0 , 1 } , y 2 { (1 , 0) , (0 , 1) , (0 . 5 , 0 . 5) } .
Following the Bradley-Terry model [54], a preference predictor constructed by the reward function estimate b r is formulated as:
[Equation]
where  0   1 denotes  0 is more consistent with the expectations of human experts compared with  1 .
The reward function learning can be solved by minimizing the cross-entropy loss between predictions from preference predictors and human preferences.
[Equation]
We refer to this objective as supervised loss in the following sections.
By optimizing the reward function using this loss, segments that are more in line with human preferences obtain a higher cumulative reward.
Soft Actor-Critic.
SAC [55] is an off-policy algorithm based on the maximum entropy RL, which encourages the agent to explore the environment by acting as randomly as possible.
SAC consists of soft Q-function Q ✓ ( s, a ) with parameters ✓ and policy ⇡  ( a | s ) with parameters  .
Q-function with parameters ✓ is deﬁned as the expectation of return:
[Equation]
where  2 [0 , 1] is a discount factor.
The parameters ✓ of soft Q-function are trained by minimizing the soft Bellman residual:
[Equation]
where ¯ V ( s t ) =
E a t ⇠ ⇡  ⇥
[Equation]
⇤ , ⌧ t = ( s t , a t , s t +1 , r t ) is the transition at time step t , ↵ is a learnable temperature parameter that controls the item of entropy, ¯ ✓ are parameters of the target soft Q-function, and B is replay buffer.
After the updating of the Q-function, policy ⇡  is updated by minimizing the loss:
[Equation]
By performing policy evaluation and policy improvement alternately, SAC trains an agent with excellent and stable performance.
In this work, we consider using SAC as our backbone reinforcement learning algorithm.
Meta-Reward-Net
In this section, we formally present Meta-Reward-Net, which includes two key components, optimizing the reward function based on the performance of the Q-function in outer loop and learning the agent in the inner loop.
In the following, we ﬁrst provide a new perspective that the Q-function can be used to compute preference labels, then deﬁne the objective of MRN and formulate a bi-level optimization problem.
4.1 The Objective
The probability that segment  0 is preferred is proportional to the exponential return of it.
Motivated by this, we use Q ✓ ( s 00 , a 00 ) and Q ✓ ( s 10 , a 10 ) to respectively measure of the return of  0 and  1 since the Q-value equal to the expectation of segment return.
Therefore, the probability that  0 is preferred to  1 is computed through the Q-function:
[Equation]
Given human preference labels y , the performance of the Q-function is measured by the cross-entropy loss between preference predictions computed by (6) and ground-truth labels:
[Equation]
where ✓ ( ) denotes the updating of ✓ depends on the reward provided by
[Equation]
The core of MRN is making reward learning be aware of the Q-function, which means that the optimization of the reward function takes the performance of current Q-function into consideration.
Based on this idea, we formulate the objective using (7) as a measurement of the Q-function.
The objective of MRN is to minimize the loss of Q ✓ on a labeled preference dataset and the Q-function is trained by minimizing the Bellman residual.
The overall objective is formulated as:
[Equation]
By formulating MRN as a bi-level optimization algorithm, this allows the reward function to provide rewards that are beneﬁcial for improving the performance of the Q-function, which further leads to a better policy ⇡  .
4.2 Bi-level Optimization
The objective in (8) is solved by the following bi-level optimization algorithm: ✓ is optimized by the reward estimation from b r in inner loop while is updated according to the performance of the
Q-function on the labeled data in outer loop.
Pseudo Updating: Building Connection between ✓ and .
To improve the performance of Qfunction, we formulate the outer loop updating as optimizing the loss of ✓ on the labeled preference data with respect to .
However, we can not directly optimize this objective since the objective is independent of .
So the ﬁrst step is to build a connection between ✓ and .
Sample a minibatch state-action pairs from the replay buffer and use them to query current reward function with parameters ( k ) to obtain reward estimation
[Equation]
where k denotes the current iteration step.
Then (4) becomes:
[Equation]
At each bi-level optimization step, we ﬁrst pseudo update the parameters of the Q-function.
Pseudo updating means that we do not directly perform the updating on the Q-function, but update the parameters of a copy of the current Q-function by minimizing (9) with learning rate ↵ :
[Equation]
where ˆ ✓ ( k ) denotes the updated copy of ✓ ( k ) .
By performing (10), the connection between ˆ ✓ ( k ) and ( k ) is built.
Outer Loop: Optimizing
to Improve the Performance of Q ✓ on Labeled Data.
After building connections through pseudo updating, the copy of Q-function with parameters ˆ ✓ ( k ) is tested on labeled preference data.
The predicted preference label P ✓ ( x ) is computed by the Q-function using (6), where x denotes a segment pair (  0 ,  1 ) .
We use implicit differentiation in our method.
The objective of outer loop is formulated in (7), and the implicit derivative of the outer loss with respect to is calculated using the chain rule:
[Equation]
where h = 2 ↵ · ⇣ r ˆ ✓ L
[Equation]
(ˆ
[Equation]
⌘ > · r ✓ Q ( s t , a t ; ✓ ( k ) ) and full derivation can be found in Appendix B.
(11) formulates the implicit derivative from the Q-function and this can be done easily using automatic differentiation in Pytorch [56].
Since pseudo updating is performed and the connection between ✓ and is built, is updated to improve the performance of the Q-function by minimizing the cross-entropy between preference labels from Q ✓ and ground-truth labels:
[Equation]
where  is the learning rate of the outer loop.
Inner Loop: Optimizing ✓ and  .
In the inner loop, the objective is the same as (4) and (5) in the training of SAC.
To calculate the new reward b r ( s t , a t ) , use the same batch of state-action pairs in pseudo updating to query the updated reward function.
However, we do not need the connection in the inner loop since the connection is used for outer level optimization.
With newly obtained reward estimation, we update Q-function Q ✓ by minimizing (4) with learning rate ↵ :
[Equation]
) and update policy ⇡  by minimizing (5) with learning rate ↵ :
[Equation]
Auxiliary Loss.
In addition to optimizing the loss in the outer loop, the reward function is augmented with a supervised loss generally used in preference-based RL, which is formulated in (2).
Our intuition is that the optimization of the reward function needs both supervised learning and awareness of the performance of the Q-function, and neither of them can be removed.
On the one hand, supervised loss is necessary for it ensures the reward estimation is consistent with human preferences.
On the other hand, the outer loss is beneﬁcial because it improves the performance of the Q-function, leading to a more accurate Q-function and ﬁnally a better policy.
Additional experiments are conducted to discuss the relation between the two losses in Appendix F.
The full procedure of our method is detailed in Appendix A. Before reward learning, we ﬁrst initialize the policy and replay buffer with unsupervised exploration, which is proposed in PEBBLE [17] and can be found in Appendix D.
We use off-policy RL algorithm SAC to collect transitions and save them in the replay buffer.
The Q-function and the policy is optimized in each training step.
4.3 Algorithm Convergence
For theoretical analysis, we provide convergence guarantee of Meta-Reward-Net.
Theorem 1 demonstrates the convergence rate of the outer loss, while Theorem 2 shows the convergence of the inner loss.
The detailed proofs of two theorems are presented in Appendix C.
Theorem 1.
Assume the outer loss L meta is Lipschitz smooth with constant L , and the gradient of L meta and J Q is bounded by ⇢ .
Let b r be twice differential, with its gradient and Hessian respectively bounded by  and B .
For some c 1 > 0 , suppose the learning rate of the inner updating ↵ k = min { 1 , c 1 T } , where c 1 < T .
For some c 2 > 0 , suppose the learning rate of the outer updating  k = min { 1 L , c 2 p T } , where p Tc 2  L ,
[Equation]
and
P
[Equation]
Meta-Reward-Net can achieve:
[Equation]
Theorem 2.
Under the conditions in Theorem 1, Meta-Reward-Net can achieve:
[Equation]
Experiments
In this section, our method is evaluated on a variety of robotic simulated manipulation tasks from
Meta-world [21] and locomotion tasks from DeepMind Control Suite (DMControl) [22, 23].
The tasks used in our experiments are shown in Appendix E, which are the same as the tasks used in
SURF.
5.1 Setup
Baselines.
Reward-based SAC and three state-of-the-art preference-based RL algorithms are used for comparison:
• SAC [55]: SAC is considered as the upper bound algorithm since the agent is provided with ground-truth reward function, which is not the case in preference-based RL.
SAC is evaluated in our experiments because it is the backbone RL algorithm of PEBBLE.
• Preference PPO [31]: the method is a re-implementation using PPO [57].
It uses an ensemble of reward functions and disagreement sampling for querying.
• PEBBLE [17]: the method is a preference-based RL method with unsupervised exploration and reward relabeling.
• SURF [18]: the method combines temporal data augmentation and pseudo labels in semisupervised learning, which is the state-of-the-art algorithm in preference-based RL.
• Meta-Reward-Net
(MRN): the proposed method, which is aware of the performance of the Q-function in reward learning through bi-level optimization.
Implementation Details.
For all methods, we use unsupervised exploration proposed in PEBBLE [17].
For the sampling of queries, disagreement-based sampling is used for all preference-based
RL methods, following the setting in [31].
An ensemble of three reward functions is used and the reward output is computed by averaging output of three reward functions.
To systematically evaluate the performance and speed up the training process, following the setting in PEBBLE [17] and SURF [18], we consider a script teacher that can always provide the ground-truth preference label of a segment pair.
Concretely, this is implemented by comparing the ground-truth return of each segment, but the reward is not accessible to the agent under the setting of preference-based RL.
For the implementation of SAC, Preference PPO and PEBBLE, we use the publicly released repository of B-Pref [58] in our experiments.
In their implementation, Preference PPO is re-implemented using on-policy algorithm PPO.
SURF is also implemented using their released code.
3 For SAC,
PEBBLE and SURF, the hyperparameters and network architectures we use are the same as them (e.g., number of network layers, learning rate, frequency of supervised reward learning).
For the amount of human’s preference feedback, we set 100 for Walker, Cheetah, Button Press and Window
Open, 700 for Quadruped, 1000 for Door Open and Drawer Open, 4000 for Sweep Into, andfor Hammer.
Our method is implemented by using PEBBLE as the backbone.
We use bi-level updating frequency
N = 5000 for Cheetah, Hammer, Door Open, Button Press, Drawer Open and Window Open,
N = 1000 for Walker, N = 3000 for Quadruped, and N = 10000 for Sweep Into.
For each task, we run all algorithms independently for ten times and report the average with a standard deviation.
Tasks of Meta-world are measured on success rate while the tasks of DMControl are measured on ground-truth episode return.
The experiments are run on a single machine with one
NVIDIA RTX 2080 Ti GPU.
Details on hyperparameters, network architectures can be found in
Appendix E.
5.2 Results
Meta-world Tasks.
Examples of the six continuous control tasks from Meta-world are shown in
Appendix E. These tasks are selected for our experiments, including robotic simulated manipulation skills of various difﬁculty.
The details of the tasks can be found in Appendix E.
Figure 2 shows the training curves of MRN and the baselines on the Meta-world tasks.
In this ﬁgure,
SAC achieves the best performance in each task by using the ground-truth reward function.
Since little human feedback is provided, we observe that there is a gap between all preference-based RL methods and the best performance, but MRN still exceeds the preference-based RL baselines by a large margin.
These results demonstrate that MRN considerably improves the performance when only few preference labels are available.
We also notice that Preference PPO does not work in most tasks.
The reason is that Preference PPO works well when the amount of preference labels is large.
Once the number of labels reduces, the performance will have a signiﬁcant drop.
DMControl Tasks.
For DMControl, three locomotion tasks in Appendix E are used for evaluation, including Walker, Cheetah and Quadruped.
These tasks encourage the agent to move forward by providing the agent with the reward that is positively correlated with agent’s velocity.
However, the agent is not accessible to the ground-truth reward function, and all preference-based RL methods are provided with human preference labels.
Similar to tasks of Meta-world, we only provide the baselines and MRN with few but the same amount of preference labels.
Figure 3 shows the results of ﬁve methods on DMControl tasks.
SURF achieves almost the same return as PEBBLE, while MRN shows obvious advantages compared with baseline methods.
The results show that MRN performs well with few preference labels and improves feedback efﬁciency.
We remark that MRN can be regarded as being aware of the performance of the Q-function through bi-level optimization based on PEBBLE.
Comparing the results of MRN and PEBBLE in Figureand Figure 3, we ﬁnd that using bi-level optimization can signiﬁcantly improve performance when few preference labels are provided.
[Figure]
5.3 Ablation Study
Number of Human’s Feedback.
Figure 2 and 3 show that our method outperforms Preference PPO,
PEBBLE and SURF under relatively small amounts of human’s feedback.
To further analyze the performance of different algorithms with different number of preference labels, Extensive experiments are conducted on Walker and Door Open with varying amounts of feedback: { 100 , 400 , 1000 , 2000 } .
The results in Figure 4 suggest that our method performs better when the number of human’s feedback is small.
As the number increases, the performance of three algorithms becomes closer.
Intuitively, this phenomenon could be considered that as feedback information becomes more sufﬁcient, the performance gap caused by insufﬁcient preference labels disappears.
Accuracy of Q-function.
To compare the quality of Q-function trained by MRN and baselines, we use the mean squared error between ground-truth Q-values and the output of Q-function.
Each method is evaluated on the same ten trajectories by calculating the MSE of the output of the Q-function
[Figure] [Figure]
a l k e r
[Figure]
D oo r O p e n
[Figure]
and the ground-truth Q-values.
We report mean and standard deviation across ten runs in Table 1.
SAC is presented as the upper bound of the quality of Q-function and MRN achieves the lowest
MSE among three methods.
In MRN, reward learning additionally takes the performance of the
Q-function into consideration.
So the reward provided by the reward function is not only consistent with the ground-truth preference labels but also suitable for the performance of the current Q-function.
Therefore, the Q-function learned by MRN is more accurate compared to the baselines.
[Table]
Conclusion
In this work, we propose Meta-Reward-Net, a novel feedback-efﬁcient preference-based RL method.
By incorporating bi-level optimization for reward and policy learning, we demonstrate MRN outperforms prior methods when a small amount of human feedback is available and considerably improves the feedback efﬁciency on a variety of robotic simulated tasks.
In particular, our method exceeds the baselines by a large margin when there is few preference labels.
From empirical results and analysis, we conclude that the efﬁciency of feedback in our method mainly beneﬁts from: Firstly, our method learns a more accurate Q-function by aligning it with human preferences.
Secondly, our method learns the Q-function and policy in the inner loop and optimizes reward function according to the performance of Q-function on the preference data in the outer loop.
By this way, MRN successfully establishes an efﬁcient mode of information transmission, which can extract more information.
The potential negative social impact includes the carbon footprint of the experiments and future work based on MRN.
We hope our method can provide inspiration for future work and encourage preference-based reinforcement learning to be better extended to practical applications.
Limitations.
There are several limitations of MRN as follows.
First, MRN relies on qualities of human feedback.
Also, MRN cannot discriminate between good and bad actions within one trajectory.
Last, representation learning on visual input requires additional parameters and thus requires more data.
While this is not the focus of this work, we consider this as future works.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See abstract, Section 1 and Section 6.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the ethics and make sure our work conforms to them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
and Appendix C. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See abstract and implementation details in Section 5.1.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See implementation details in Section 5.1 and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 2, Figure 3, Figure 4 and Table 1 in Section 5.2 and Section 5.3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Our code refers to the source code published by B-Pref and SURF, and their papers are cited in our paper.
See Section 5.1.
(b) Did you mention the license of the assets?
[Yes] All referenced code is open source and we notice that B-Pref provides MIT license and mujoco provides Apache License.
As for DMControl and Meta-world, you can see references for license details.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We release the link of our source code in abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] MIT license permits us to copy and modify the source code of B-Pref.
We also get the permission of the author of SURF.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]