SURF: S EMI - SUPERVISED R EWARD L EARNING WITH D ATA A UGMENTATION FOR F EEDBACK - EFFICIENT P REFERENCE - BASED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-deﬁned reward function by learning the reward with a supervisor’s preference between the two agent behaviors.
However, preference-based learning often requires a large amount of human feedback, making it difﬁcult to apply this approach to various applications.
This data-efﬁciency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning.
Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation.
In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the conﬁdence of the preference predictor.
To further improve the label-efﬁciency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors.
Our experiments demonstrate that our approach signiﬁcantly improves the feedback-efﬁciency of the state-ofthe-art preference-based method on a variety of locomotion and robotic manipulation tasks.
I NTRODUCTION
Reward function plays a crucial role in reinforcement learning (RL) to convey complex objectives to agents.
For various applications, where we can design an informative reward function, RL with deep neural networks has been used to solve a variety of sequential decision-making problems, including board games (Silver et al., 2017; 2018), video games (Mnih et al., 2015; Berner et al., 2019;
Vinyals et al., 2019), autonomous control (Schulman et al., 2015; Bellemare et al., 2020), and robotic manipulation (Kober & Peters, 2011; Kober et al., 2013; Kalashnikov et al., 2018; Andrychowicz et al., 2020).
However, there are several issues in reward engineering.
First, designing a suitable reward function requires more human effort as the tasks become more complex.
For example, deﬁning a reward function for book summarization (Wu et al., 2021) is non-trivial because it is hard to quantify the quality of summarization in a scale value.
Also, it has been observed that RL agents could achieve high returns by discovering undesirable shortcuts if the hand-engineered reward function does fully specify the desired task (Amodei et al., 2016; Hadﬁeld-Menell et al., 2017; Lee et al., 2021a).
Furthermore, there are various domains, where a single ground-truth function does not exist, and thus personalization is required by modeling different reward functions based on the user’s preference.
Preference-based RL (Akrour et al., 2011; Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021a) provides an attractive alternative to avoid reward engineering.
Instead of assuming a handengineered reward function, a (human) teacher provides preferences between the two agent behaviors, and an agent learns how to show the desired behavior by learning a reward function, which is consistent with the teacher’s preferences.
Recent progress of preference-based RL has shown that the teacher can guide the agent to perform novel behaviors (Christiano et al., 2017; Stiennon et al., 2020; Wu et al., 2021), and mitigate the effects of reward exploitation (Lee et al., 2021a).
However, existing preference-based approaches often suffer from expensive labeling costs, and this makes it hard to apply preference-based RL to various applications.
Meanwhile, recent state-of-the-art system in computer vision, the label-efﬁciency problem has been successfully addressed through semi-supervised learning (SSL) approaches (Berthelot et al., 2019; 2020; Sohn et al., 2020; Chen et al., 2020b).
By leveraging unlabeled dataset, SSL methods have improved the performance with low cost.
Data augmentation also plays a signiﬁcant role in improving the performance of supervised learning methods (Cubuk et al., 2018; 2019).
By using multiple augmented views of the same data as input, the performance has been improved by learning augmentation-invariant representations.
Inspired by the impact of semi-supervised learning and data augmentation, we present SURF: a
S emi-s U pervised R eward learning with data augmentation for F eedback-efﬁcient preference-based
RL.
To be speciﬁc, SURF consists of the following key ingredients:
(a) Pseudo-labeling (Lee, 2013; Sohn et al., 2020): We leverage unlabeled data by utilizing the artiﬁcial labels generated by learned preference predictor, which makes the reward function produce a conﬁdent prediction (see Figure 1a).
We remark that such a SSL approach is particularly attractive in our setup as an unlimited number of unlabeled data can be obtained with no additional cost,
i
e., from past experiences stored in the buffer.
(b) Temporal cropping augmentation: We generate slightly shifted or resized behaviors, which are expected to have the same preferences from a teacher, and utilize them for reward learning (see Figure 1b).
Our data augmentation technique enhances the feedback-efﬁciency by enforcing consistencies (Xie et al., 2019; Berthelot et al., 2020; Sohn et al., 2020) to the reward function.
We remark that SURF is not a naïve application of these two techniques, but a novel combination of semi-supervised learning and the proposed data augmentation, which has not been considered or evaluated in the context of the preference-based RL.
Our experiments demonstrate that SURF signiﬁcantly improves the preference-based RL method (Lee et al., 2021a) on complex locomotion and robotic manipulation tasks from DeepMind
Control Suite (Tassa et al., 2018; 2020) and Meta-world (Yu et al., 2020), in terms of feedbackefﬁciency.
In particular, our framework could make RL agents achieve ∼ 100% of success rate on complex robotic manipulation task using only a few hundred preference queries, while its baseline method only achieves ∼ 50% of the success rate under the same condition (see Figure 3).
R ELATED WORK
Preference-based RL .
In the preference-based RL framework, a (human) supervisor provides preferences between the two agent behaviors and the agent uses this feedback to perform the task (Christiano et al., 2017; Ibarz et al., 2018; Leike et al., 2018; Stiennon et al., 2020; Wu et al., 2021; Lee et al., 2021a;b).
Since this approach is only feasible if the feedback is practical for a human to provide, several strategies have been studied in the literature.
Ibarz et al.
(2018) initialized the agent’s policy with imitation learning from the expert demonstrations, while Lee et al.
(2021a) utilized unsupervised pre-training for policy initialization.
Several sampling schemes (Sadigh et al., 2017;
Biyik & Sadigh, 2018; Biyik et al., 2020) to select informative queries also have been adopted for improving the feedback-efﬁciency.
Our approach differs in that we utilize unlabeled samples for reward learning, and also provide a novel data augmentation technique for the agent behaviors.
Data augmentation for RL .
In the context of RL, data augmentation has been widely investigated for improving data-efﬁciency (Srinivas et al., 2020; Yarats et al., 2021), or RL generalization (Cobbe et al., 2019; Lee et al., 2019).
For example, RAD (Laskin et al., 2020) demonstrated that data augmentation, such as random crop, can improve both data-efﬁciency and generalization of RL algorithms.
While these methods are known to be beneﬁcial to learn policy in the standard RL setup, they have not been tested for learning rewards .
To the best of our knowledge, we present the ﬁrst data augmentation method specially designed for learning reward function.
Semi-supervised learning .
The goal of semi-supervised learning (SSL) is to leveraging unlabeled samples to improve a model’s performance when the amount of labeled samples are limited.
In an attempt to leverage the information in the unlabeled dataset, a number of techniques have been proposed, e.g., entropy minimization (Grandvalet & Bengio, 2004; Lee, 2013) and consistency regularization (Sajjadi et al., 2016; Miyato et al., 2018; Xie et al., 2019; Sohn et al., 2020).
Recently, the combination of these two approaches have shown state-of-the-art performance in benchmarks, e.g., MixMatch (Berthelot et al., 2019), and ReMixMatch (Berthelot et al., 2020), when used with
[Figure]
advanced data augmentation techniques (Zhang et al., 2018; Cubuk et al., 2019).
Speciﬁcally, FixMatch (Sohn et al., 2020)
revisits pseudo-labeling technique and demonstrates that joint usage of pseudo-labels and consistency regularization achieves remarkable performance due to its simplicity.
P RELIMINARIES
Reinforcement learning (RL) is a framework where an agent interacts with an environment in discrete time (Sutton & Barto, 2018).
At each timestep t , the agent receives a state s t from the environment and chooses an action a t based on its policy π ( a t
| s t ) .
In conventional RL framework, the environment gives a reward r ( s t , a t ) and the agent transitions to the next state s t +1 .
The return
R t = (cid:80) ∞ k =0 γ k r ( s t + k , a t + k ) is deﬁned as discounted cumulative sum of the reward with discount factor γ ∈ [0 , 1) .
The goal of the agent is to learn a policy that maximizes the expected return.
Preference-based reinforcement learning .
In this paper, we consider a preference-based RL framework, which does not assume the existence of hand-engineered reward.
Instead, a (human) teacher provides preferences between the agent’s behaviors and the agent uses this feedback to perform the task (Christiano et al., 2017; Ibarz et al., 2018; Leike et al., 2018; Stiennon et al., 2020; Lee et al., 2021a;b; Wu et al., 2021) by learning a reward function, which is consistent with the observed preferences.
We formulate a reward learning problem as a supervised learning problem (Wilson et al., 2012;
Christiano et al., 2017).
Formally, a segment σ is a sequence of observations and actions { ( s k , a k ) , ..., ( s k + H − 1 , a k + H − 1 ) } .
Given a pair of segments ( σ 0 , σ 1 ) , a teacher gives a feedback indicating which segment is preferred, i.e., y ∈ { 0 , 1 , 0 . 5 } , where 1 indicates σ 1 (cid:31) σ 0 , 0 indicates σ 0 (cid:31) σ 1 , and 0.5 implies an equally preferable case.
Each feedback is stored in a dataset D as a triple ( σ 0 , σ 1 , y ) .
Then, we model a preference predictor using the reward function (cid:98) r ψ following the
Bradley-Terry model (Bradley & Terry, 1952):
[Equation]
(1) where σ i (cid:31) σ j denotes the event that segment i is preferable to segment j .
The underlying assumption of this model is that the teacher’s probability of preferring a segment depends exponentially on the accumulated sum of the reward over the segment.
The reward model is trained through supervised learning with teacher’s preferences.
Speciﬁcally, given a dataset of preferences D , the reward function is updated by minimizing the binary cross-entropy loss:
[Equation]
Require:
Hyperparameters: unlabeled batch ratio µ , threshold parameter τ , and loss weight λ
Require: Set of collected labeled data D l , and unlabeled data D u 1: for each gradient step do 2:
Sample labeled batch { ( σ 0 l , σ 1 l , y ) ( i ) }
Bi =1 ∼ D l 3:
Sample unlabeled batch { ( σ 0 u , σ 1 u ) ( j ) } µBj =1 ∼ D u 4: //
D ATA AUGMENTATION FOR LABELED DATA 5: for i in 1 . . .
B do 6: ( (cid:98) σ 0 l , (cid:98) σ 1 l ) ( i ) ← TDA (( σ 0 l , σ 1 l ) ( i ) ) in Algorithm7: end for 8: //
P SEUDO - LABELING AND DATA AUGMENTATION FOR UNLABELED DATA 9: for j in 1 . . .
µB do 10:
Predict pseudo-labels (cid:98) y
(( σ 0 u , σ 1 u ) ( j ) ) 11: ( (cid:98) σ 0 u , (cid:98) σ 1 u ) ( j ) ← TDA (( σ 0 u , σ 1 u ) ( j ) ) in Algorithm12: end for 13:
Optimize L SSL (3) with respect to ψ 14: end for
The reward function (cid:98) r ψ is usually optimized only using labels from real human, which are expensive to obtain in practice.
Instead, we propose a simple yet effective method based on semi-supervised learning and data augmentation to improve the feedback-efﬁciency of preference-based learning.
SURF
In this section, we present SURF: a S emi-s U pervised R eward learning with data augmentation for F eedback-efﬁcient preference-based RL, that can be used in conjunction with any existing preference-based RL methods.
Our main idea is to leverage a large number of unlabeled samples collected from environments for reward learning, by inferring pseudo-labels.
To further increase the effective number of training samples, we propose a new data augmentation that temporally crops the subsequence of the agent behaviors.
The full procedure of our uniﬁed framework in Algorithm(See Figure 1 for the overview of our method).
4. S EMI - SUPERVISED REWARD LEARNING
To improve the feedback efﬁciency, we propose a semi-supervised learning (SSL) method for leveraging unlabeled experiences in the buffer for reward learning.
In addition to a labeled dataset
D l = { ( σ 0 l , σ 1 l , y ) ( i ) } N l i =1 , we utilize an unlabeled dataset D u
= { ( σ 0 u , σ 1 u ) ( i ) }
N u i =1 to optimize the reward model r ψ .
Speciﬁcally, we generate the artiﬁcial labels (cid:98) y by pseudo-labeling (Lee, 2013;
Sohn et al., 2020) for the unlabeled dataset D u .
We infer a preference (cid:98) y for an unlabeled segment pair ( σ 0 u , σ 1 u ) as a class with higher probability as follows:
[Equation]
By generating labels from the prediction model, we can obtain free supervision for optimizing our reward model.
However, pseudo-labels from low-conﬁdence predictions can be inaccurate, and such noisy feedback can signiﬁcantly degrade the peformance of preference-based learning (Lee et al., 2021b).
To ﬁlter out inaccurate pseudo-labels, we only use unlabeled samples for training when the conﬁdence of the predictor is higher than a pre-deﬁned threshold (Rosenberg et al., 2005).
Then the reward model r ψ is optimized by minimizing the following objective:
[Equation]
[Table]
where k ∗ = arg max j ∈{ 0 , 1 } (cid:98) y
( j ) is an index of the preferred segment from the pseudo-label, λ is a hyperparameter that balances the losses, and τ is a conﬁdence threshold.
Training with the pseudolabels encourages the model to output more conﬁdent predictions on unlabeled samples.
This can be seen as a form of entropy minimization (Grandvalet & Bengio, 2004), which is essential to the success of recent SSL methods (Berthelot et al., 2019; 2020).
The entropy minimization can improve the reward learning by forcing the preference predictor to be low-entropy (i.e., high-conﬁdence) on unlabeled samples.
During training, we sample a larger minibatch of unlabeled samples than labeled ones by a factor of µ following (Sohn et al., 2020), since unlabeled samples with low conﬁdence are dropped within minibatch.
4. T EMPORAL DATA AUGMENTATION FOR REWARD LEARNING
To further improve the feedback-efﬁciency in preference-based RL, we propose a new data augmentation technique specially designed for reward learning.
Speciﬁcally, for a given two segments and preference ( σ 0 , σ 1 , y ) , we generate augmented segments ( (cid:98) σ 0 , (cid:98) σ 1 , y ) by cropping the subsequence from each segment (see Algorithm 2 for more details).
Then, we utilize augmented samples ( (cid:98) σ 0 , (cid:98) σ 1 ) to optimize the cross-entropy loss in (3).
The intuition behind the augmentation is that for a given pair of behavior clips, the human teacher may keep their relative preferences for slightly shifted or resized versions of them.
In the context of SSL, data augmentation is also related to consistency regularization (Xie et al., 2019; Berthelot et al., 2020; Sohn et al., 2020) approaches that train the model to output similar predictions on augmented versions of the same sample.
Namely, this temporal cropping method enables our framework can also enjoy the beneﬁts of consistency regularization.
E XPERIMENTS
We design our experiments to investigate the following:
◦ How does SURF improve the existing preference-based RL method in terms of feedback efﬁciency?
◦
What is the contribution of each of the proposed components in SURF?
◦
How does the number of queries affect the performance of SURF?
◦ Is temporal cropping better than existing state-based data augmentation methods in terms of feedback efﬁciency?
5. S ETUPS
We evaluate SURF on several complex robotic manipulation and locomotion tasks from Metaworld (Yu et al., 2020) and DeepMind Control Suite (DMControl; Tassa et al. 2018; 2020), respectively.
Similar to prior works (Christiano et al., 2017; Lee et al., 2021a;b), in order to systemically
[Figure] [Figure]
evaluate the performance, we consider a scripted teacher that provides preferences between two trajectory segments to the agent according to the underlying reward function.
3 Since preferences of the scripted teacher exactly reﬂects ground truth reward of the environment, one can evaluate the algorithms quantitatively by measuring the true return.
We remark that SURF can be combined with any preference-based RL algorithms by replacing the reward learning procedure of its backbone method.
In our experiments, we choose state-of-theart approach, PEBBLE (Lee et al., 2021a), as our backbone algorithm.
Since PEBBLE utilizes
SAC (Haarnoja et al., 2018) algorithm to learn the policy, we also compare to SAC using the ground truth reward directly, as an upper bound of PEBBLE and our method.
We note that our goal is not to outperform SAC, but rather to perform closely using as few preference queries as possible.
Implementation details of SURF .
For all experiments, we use the same hyperparameters used by the original SAC and PEBBLE algorithms, such as learning rate of neural networks and frequency of the feedback session.
For query selection strategy, we use the disagreement-based sampling scheme,
[Figure]
which selects queries with high uncertainty, i.e., ensemble disagreement (see Appendix B for more details).
At each feedback session, we sample unlabeled samples as 10 times of labeled ones by uniform sampling scheme, unless otherwise noted.
Although we only use such amount of unlabeled samples for time-efﬁcient training, we note that one can utilize much more unlabeled samples as needed.
For the hyperparameters of SURF, we ﬁx the loss weight λ
= 1 , and unlabeled batch ratio µ = 4 for all experiments, and use threshold parameter τ = 0 .
999 for Window Open, Sweep Into,
Cheetah tasks, and τ = 0 .
99 for the others.
We provide more experimental details in Appendix B.
5. B ENCHMARK TASKS WITH SCRIPTED TEACHERS
Meta-world experiments .
Meta-world consists of 50 robotic manipulation tasks, which are designed for learning diverse manipulation skills.
We consider six tasks from Meta-world, to investigate how SURF improves a preference-based learning method on a range of complex robotic manipulation tasks (see Figure 2).
Figure 3 shows the learning curves of SAC, PEBBLE and SURF (which combined with PEBBLE) on the manipulation tasks.
In each task, PEBBLE and SURF utilize the same number of preference queries for fair comparison.
As shown in the ﬁgure, SURF signiﬁcantly improves the performance of PEBBLE given the same number of feedback on all tasks we considered, and matches the performance of SAC using the ground truth reward on four tasks.
For example, we ﬁnd that when using 400 preference queries, SURF (red) reaches the same performance as SAC (green) while PEBBLE (blue) is far behind to SAC on Window Open task.
We also observe that SURF achieves similar performance to PEBBLE with much less labels.
For example, to achieve comparable performance to SAC on Window Open task, PEBBLE needs 2,500 queries (reported in (Lee et al., 2021a)), requiring about 6 times more queries than SURF.
These results demonstrate that SURF signiﬁcantly reduces the feedback requirement to solve the complex tasks.
DMControl experiments .
For locomotion tasks, we choose three complex environments from
DMControl: Walker-walk, Cheetah-run, and Quadruped-walk.
Figure 4 shows the learning curves of the algorithms with same number of queries.
We ﬁnd that using a budget of 100 or 1,000 queries (which takes only few human minutes), SURF (red) could signiﬁcantly improve the performance of
PEBBLE (blue).
These results again demonstrate that that SURF improves the feedback-efﬁciency of preference-based RL methods on a variety of complex tasks.
5. A BLATION STUDY
Component analysis .
To evaluate the effect of each technique in SURF individually, we incrementally apply semi-supervised learning (SSL) and temporal cropping (TC) to our backbone algorithm,
PEBBLE.
Figure 5a shows the learning curves of SURF on Walker-walk task with 100 queries.
We observe that leveraging unlabeled samples via pseudo-labeling (green) signiﬁcantly improves
PEBBLE, in terms of both sample-efﬁciency and asymptotic performance, while standard PEBBLE (blue) suffers from lack of supervision.
In addition, both supervised (blue) and semi-supervised
[Figure]
Effects of query size .
To investigate how the number of queries affects the performance of SURF, we evaluate the performance of SURF with a varying number of queries N ∈ { 50 , 100 , 200 , 400 } .
As shown in Figure 5b, SURF (solid lines) consistently improves the performance of PEBBLE (dotted lines) across a wide range of query sizes.
The gain from SURF becomes even more signiﬁcant in the extreme label-scarce scenarios, i.e., N ∈ { 50 , 100 } .
Comparison to other augmentation for state-based inputs .
To demonstrate that temporal cropping can enduce signiﬁcant improvements for reward learning, we compare our method to other augmentation methods for state-based inputs.
We consider random amplitude scaling (RAS) and adding Gaussian noise (GN) proposed in Laskin et al.
(2020) as our baselines.
RAS multiplies an uniform random variable z to the state, i.e., (cid:98) s
= s · z , where z ∼ Unif [ α, β ] , and GN adds a multivariate Gaussian random variable z to the state, i.e., (cid:98) s
= s + z , where z ∼ N (0 , I ) .
As proposed in Laskin et al.
, we apply these methods consistently along the time dimension, and choose the parameters for RAS as α = 0 . 8 , β = 1 . 2 .
Speciﬁcally, for a given segment σ = ( { ( s k , a k ) , ..., ( s k + H − 1 , a k + H − 1 ) } ) , we obtain the augmented sample (cid:98) σ by perturbing each state along the segment, i.e., (cid:98) σ = ( { ( (cid:98) s k , a k ) , ..., ( (cid:98) s k + H − 1 , a k
+ H − 1 ) } ) .
In Figure 5c, we plot the learning curves of PEBBLE with various data augmentations on Walker-walk task with 100 queries.
We observe that RAS improves the performance of PEBBLE, but temporal cropping still outperforms these two methods.
GN degrades the performance, possibly due to the noisy inputs.
Since
RAS is an orthogonal approach to augment state-based inputs, one can integrate them with our method to further improve the performance.
This may be an interesting future direction for addressing feedback-efﬁciency in preference-based RL.
Effects of hyperparameters of SURF .
We investigate how the hyperparameters of SURF affect the performance of preference-based RL.
In Figure 6, we plot the learning curve of SURF with different set of hyperparameters: (a) unlabeled batch ratio µ ∈ { 1 , 2 , 4 , 7 } , (b) threshold parameter τ ∈ { 0 . 95 , 0 . 97 , 0 . 99 , 0 . 999 } , and (c) loss weight λ ∈ { 0 . 1 , 0 . 5 , 1 , 2 } , respectively.
First, we observe that SURF is quite robust on µ , but the performance slightly drops with a large batch size µ = 7 .
We expect that this is because a large batch size makes the reward model overﬁt to unlabeled data.
We also observe that SURF is also robust on the threshold τ , except for the smallest value of 0.95.
Because there are only two classes in tasks, the optimal threshold could larger than the value typically used in previous SSL methods (Sohn et al., 2020), i.e., 0.95.
In the case of the loss weight λ , tuning this parameter brings more improvements than other hyperparameters.
Although
[Figure]
we use a simple choice, i
[Figure]
e., λ = 1 , in our experiments, more tuning λ would further improve the performance of our method.
D ISCUSSION
In this work
[Figure]
we present SURF
[Figure]
a semi-supervised reward learning algorithm with data augmentation for preference-based RL.
First, in order to utilize an unlimited number of unlabeled data, we utilize pseudo-labeling on conﬁdent samples.
Also, to enforce consistencies to the reward function, we propose a new data augmentation method called temporal cropping augmentation.
Our experiments demonstrate that SURF signiﬁcantly improves feedback-efﬁciency of current state-of-the-art method on a variety of complex robotic manipulation and locomotion tasks.
We believe that SURF can scale up deep RL to more diverse and challenging domains by making preference-based learning more tractable.
An interesting future direction is to extend state-based inputs to partially-observable or highdimensional inputs, e.g., pixels.
One can expect that representation learning based on unlabeled samples and data augmentation (Chen et al., 2020a; Grill et al., 2020) is crucial to handle such inputs.
We think that our investigations on leveraging unlabeled samples and data augmentation would be useful in representation learning for preference-based RL.
Ethics statement .
Preference-based RL can align RL agents with the teacher’s preferences, which enables us to apply RL to diverse problems and obtain strong AI.
However, there could be possible negative impacts if a malicious user corrupts the preferences to teach the agent harmful behaviors.
Since we have proposed a method that makes preference-based RL algorithms more feedbackefﬁciently, our method may reduce the efforts for teaching not only the desirable behaviors, but also such bad behaviors.
For this reason, in addition to developing algorithms for better performance and efﬁciency, it is also important to consider safe adaptation in the real world.
Reproducibility statement .
We describe the implementation details of SURF in Appendix B, and also provide our source code in the supplementary material.
A PEBBLE ALGORITHM
A state-of-the-art preference-based RL algorithm, PEBBLE (Lee et al., 2021a), consists of two main components: unsupervised pre-training and relabeling experiences.
To collect diverse experience,
PEBBLE pre-trains the policy by using intrinsic motivation (Oudeyer et al., 2007; Schmidhuber, 2010) in the beginning of training.
Speciﬁcally, PEBBLE optimizes the policy to maximize the state entropy H ( s ) =
− E s ∼ p
( s )
[log p ( s )] to efﬁciently explore the environment.
Then PEBBLE learns the policy by using the state-of-the-art off-policy RL algorithm, SAC (Haarnoja et al., 2018).
Since the learning process of off-policy algorithms with a non-stationary reward function can be unstable,
PEBBLE stabilizes the learning process by relabeling all experiences in the buffer when the reward model is updated.
B E XPERIMENTAL DETAILS
Training details .
We choose PEBBLE (Lee et al., 2021a) as a backbone algorithm of SURF, and use the hyperparameters in Table 1 for both PEBBLE and our method.
For the reward model, we use a three-layer MLP with 256 hidden units and leaky ReLU activation.
Following the implementation of PEBBLE (Lee et al., 2021a), we use an ensemble of three reward models and bound the output to [ − 1 , 1] using tanh function.
Each model is trained by minimizing the cross-entropy loss using
ADAM optimizer (Kingma & Ba, 2015) with the learning rate of 0.0003.
For semi-supervised learning and data augmentation of SURF, we use hyperparameters in Table 2.
Sampling schemes .
In preference-based RL methods, informative query sampling (Biyik & Sadigh, 2018; Biyik et al., 2020; Sadigh et al., 2017) has been adopted for improving the feedback-efﬁciency.
For all experiments of PEBBLE and SURF, we use the disagreement-based sampling (Christiano et al., 2017) to choose queries for labeling: we ﬁrst uniformly sample the initial batch of segments, and select N query pairs of segments 4 with high uncertainty based on the variance across ensemble of preference predictors { P ψ i [ σ 1 (cid:31) σ 0 ] } N en i =1 .
Note that we use uniform sampling scheme for unlabeled samples, because the number of unlabeled samples are not limited.
At each feedback session, we sample unlabeled samples as 10 times of labeled ones if the maximum budget of feedback is equal or larger than 1,000, and otherwise we sample unlabeled samples as 100 times of labeled ones
[Table]
[Table] [Table]
Window Open, Sweep Into, Cheetah)
[Table] [Table]
H
[Table] [Table]
Implementation .
We implement SURF using the publicly released implementation repository of the PEBBLE algorithm ( https://github.com/pokaxpoka/B_Pref ) with a full list of hyperparameters in Table 1.
Note that DMControl environment depends on the MuJoCo simulator (Todorov et al., 2012), which is a commercial software.
We follow the standard evaluation protocol for the locomotion tasks from DMControl.
For robotic manipulation tasks from Metaworld, we measure the task success rate as deﬁned by the authors.
For each run of experiments, we utilize one Nvidia RTX 2080 Ti GPU and 4 CPU cores for training.
C A DDITIONAL EXPERIMENTAL RESULTS
Ablation study on Meta-world .
We provide additional experimental results for component analysis on Meta-world (Yu et al., 2020).
To evaluate the effect of each technique in SURF individually, we incrementally apply semi-supervised learning (SSL) and temporal cropping (TC) to our backbone algorithm, PEBBLE.
Figure 7a and 7b show the learning curves of SURF on Window Open with 400 queries and Hammer with 10,000 queries, respectively.
We observe that both semi-supervised learning (green) and data augmentation (purple) improve the baseline of PEBBLE (blue).
Also, applying both of them further improves the performance (red).
This shows that the key components of SURF are both effective.
Applying temporal cropping with other augmentations .
In Section 5.3, we compared our method to random amplitude scaling (RAS) and adding Gaussian noise (GN) proposed in Laskin et al.
To investigate if applying RAS or GN with the temporal cropping (TC) further improve the performance, we provide experimental results for the joint usage of the augmentations.
In Figure 8a, we plot the learning curves of PEBBLE with various data augmentations on Walker-walk withqueries.
We observe that the naive combination of two augmentations, i.e., RAS + TC and GN + TC, do not further improve the performance.
Investigating how to combine several data augmentation methods for reward learning would be an interesting future direction.
Effects of augmentation intensity .
To investigate how does augmentation intensity affects the RL performance, we provide additional experimental results.
In Walker-walk with 100 queries, we apply the temporal cropping to PEBBLE, with a varying cropping range,
i
[Figure]
e., ( H max − H min ) / 2 , from an array of [2 , 5 , 10 , 20] .
Figure 8b shows that temporal cropping consistently improves the performance of the PEBBLE.
Although the performance with a large cropping range of 20 slightly underperforms the performance with 10, these results show that our augmentation method is quite robust to the choice of hyperparameters
[Figure] [Figure] [Figure]