P OLICY E XPANSION FOR B RIDGING O FFLINE - TOO NLINE R EINFORCEMENT L EARNING
A BSTRACT
Pre-training with ofﬂine data and online ﬁne-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efﬁciency and performance.
One natural approach is to initialize the policy for online learning with the one trained ofﬂine.
In this work, we introduce a policy expansion scheme for this task.
After learning the ofﬂine policy, we use it as one candidate policy in a policy set.
We then expand the policy set with another policy which will be responsible for further learning.
The two policies will be composed in an adaptive manner for interacting with the environment.
With this approach, the policy previously learned ofﬂine is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the ofﬂine policy in the initial stage of online learning while allowing the ofﬂine policy participate in the exploration naturally in an adaptive manner.
Moreover, new useful behaviors can potentially be captured by the newly added policy through learning.
Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach.
I NTRODUCTION
Reinforcement learning (RL) has shown great potential in various ﬁelds, reaching or even surpassing human-level performances on many tasks (e.g. Mnih et al., 2015; Silver et al., 2017; Schrittwieser et al., 2019; Tsividis et al., 2021).
However, since the policy is learned from scratch for a given task in the standard setting, the number of samples required by RL for successfully solving a task is usually large, which limits its applicability in many practical scenarios such as robotics, where physical interaction and data collection has a non-trivial cost.
In many cases, there is a good amount of ofﬂine data that has already been available (Kober et al., 2013;
Rastgoftar et al., 2018; Cabi et al., 2019), e.g., collected during previous iterations of experiments or from human (e.g. for the task of driving).
Instead of ab initio learning as in the common RL setting, how to effectively leverage the already available ofﬂine data for helping with online policy learning is an interesting and open problem (Vecerík et al., 2017; Hester et al., 2018; Nair et al., 2018).
Ofﬂine RL is an active recent direction that aims to learn a policy by purely using the ofﬂine data, without any further online interactions (Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto & Gu, 2021;
Levine et al., 2020; Ghosh et al., 2022; Chen et al., 2021; Janner et al., 2021;
Yang et al., 2021;
Lu et al., 2022; Zheng et al., 2022).
It holds the promise of learning from suboptimal data and improving over the behavior policy that generates the dataset (Kumar et al., 2022), but its performance could still be limited because of its full reliance on the provided ofﬂine data.
To beneﬁt from further online learning, one possible way is to pre-train with ofﬂine RL, and warm start the policy of an online RL algorithm to help with learning and exploration when learning online.
While this pre-training + ﬁne-tuning paradigm is natural and intuitive, and has received great success in many ﬁelds like computer vision (Ge & Yu, 2017; Kornblith et al., 2019) and natural language processing (Devlin et al., 2018; Radford & Narasimhan, 2018; Brown et al., 2020), it is less widely used in RL.
Many early attempts in RL community report a number of negative results along this direction.
For example, it has been observed that initializing the policy with ofﬂine pre-training and then ﬁne-tuning the policy with standard online RL algorithms (e.g. SAC (Haarnoja et al., 2018)) sometimes suffers from non-recoverable performance drop under certain settings (Nair et al.,
Uchendu et al., 2022), potentially due to the distribution shift between ofﬂine and online stages and the change of learning dynamics because of the algorithmic switch.
Another possible way is to use the same ofﬂine RL algorithm for online learning.
However, it has been observed that standard ofﬂine RL methods generally are not effective in ﬁne-tuning with online data, due to reasons such as conservativeness of the method (Nair et al., 2020).
Some recent works in ofﬂine RL also start to focus on the ofﬂine-pre-training + online ﬁne-tuning paradigm (Nair et al., 2020; Kostrikov et al., 2022).
For this purpose, they share the common philosophy of designing an
RL algorithm that is suitable for both ofﬂine and online phases.
Because of the uniﬁed algorithm across phases, the network parameters (including those for both critics and actor) trained in the ofﬂine phase can be reused for further learning in the online phase.
Our work shares the same objective of designing effective ofﬂine-to-online training schemes.
However, we take a different perspective by focusing on how to bridge ofﬂine-online learning, and not on developing yet another ofﬂine or online RL method, which is orthogonal to the focus of this work.
We will illustrate the idea concretely by instantiating our proposed scheme by applying it on existing
RL algorithms (Kostrikov et al., 2022; Haarnoja et al., 2018).
The contributions of this work are:
• we highlight the value of properly connecting existing ofﬂine and online RL methods in order to enjoy the best of both worlds, a perspective that is alternative and orthogonal to developing completely new RL algorithms; • we propose a simple scheme termed as policy expansion for bridging ofﬂine and online reinforcement learning.
The proposed approach is not only able to preserve the behavior learned in the ofﬂine stage, but can also leverage it adaptively during online exploration and along the process of learning; • we verify the effectiveness of the proposed approach by conducting extensive experiments on various tasks and settings, with comparison to a a number of baseline methods.
P RELIMINARIES
We brieﬂy review some related basics in this section, ﬁrst on model-free RL for online policy learning, and then on policy learning from ofﬂine dataset.
2. O NLINE R EINFORCEMENT L EARNING
Standard model-free RL methods learn a policy that maps the current state s to a distribution of action a as π ( s ) .
The policy is typically modeled with a neural network π θ ( s ) with θ denoting the learnable parameters.
To train this policy, there are different approaches including on-policy (Sutton et al., 2000; Schulman et al., 2017) and off-policy RL methods (Lillicrap et al., 2016; Haarnoja et al., 2018; Fujimoto et al., 2018; Zhang et al., 2022).
In this work, we mainly focus on off-policy RL for online learning because of its higher sample efﬁciency.
Standard off-policy RL methods rely on the state-action value function Q ( s, a ) using TD-learning:
[Equation]
where T ( s, a ) denotes the dynamics function and r ( s, a ) the reward.
γ ∈ (0 , 1) is a discount factor.
By deﬁnition, Q ( s, a ) represents the accumulated discounted future reward starting from s , taking action a , and then following policy π θ thereafter.
The optimization of θ is achieved by maximizing the following function:
[Equation]
where D denotes replay buffer for storing online trajectories.
In the typical RL setting, learning is conducted from scratch by initializing all parameters randomly and interacting with the world with the randomly policy.
Q ( s, a ) can be implemented as a neural network Q φ ( s, a ) with parameter φ .
2. P OLICY L EARNING FROM O FFLINE D ATASET
Policy learning from ofﬂine datasets has been investigated from different perspectives.
Given expertlevel demonstration data, behavior cloning (BC) (Pomerleau, 1988; Bain & Sammut, 1996) is an effective approach for ofﬂine policy learning because of its simplicity and effectiveness.
In fact, in some recent work, BC has shown to perform competitively with some ofﬂine RL methods (Fujimoto & Gu, 2021;
Chen et al., 2021).
Given a dataset D oﬄine = { ( s i , a i ) } consisting of expert’s state action pairs ( s i , a i ) , BC trains the policy with maximum likelihood over the data:
[Equation]
Although BC has the beneﬁts of reducing the policy learning task to an ordinary supervised learning task, it suffers from the well-known distributional shift issue (Codevilla et al., 2019; Muller et al., 2005; de Haan et al., 2019; Wen et al., 2020).
Another limitation is that BC has a relatively strong requirements on the data quality, and is not good at learning from suboptimal data.
Ofﬂine RL is a category of methods that are more suitable for policy learning from noisy and suboptimal ofﬂine data (Kumar et al., 2022).
When focusing on ofﬂine learning only, the core challenge is how to address the extrapolation error due to querying the critic function with outof-distribution actions (Fujimoto et al., 2019; Kumar et al., 2020).
Common strategies include constraining the actions to be close to dataset actions (Fujimoto et al., 2019; Fujimoto & Gu, 2021), and constraining the critic to be conservative for out of data distribution actions (Kumar et al., 2020).
The recent implicit Q-learning (IQL) method (Kostrikov et al., 2022) addresses this issue by learning a value network to match the expectile of the critic network, thus avoiding querying the critic with the actions not in the ofﬂine dataset.
For policy update, IQL use a weighted BC formulation
[Equation]
where w ( s, a ) denotes a data dependent weight, typically calculated based on the estimated advantages (Nair et al., 2020; Xu et al., 2022; Kostrikov et al., 2022).
O FFLINE AND O NLINE RL R EVISITED : C ONNECTIONS AND G APS
Connections between Ofﬂine and Online RL.
Ofﬂine RL and online RL are closely connected in many aspects.
Historically, many ofﬂine RL approaches are branched off from off-policy RL algorithms to the full ofﬂine setting (Fujimoto et al., 2019; Fujimoto & Gu, 2021).
Algorithms more speciﬁc to ofﬂine setting are then further developed, motivated by the challenges residing in the ofﬂine setting (Levine et al., 2020; Kumar et al., 2020).
Besides algorithmic connections, ofﬂine and online RL are also complementary to each other in terms of strengths and weaknesses.
Ofﬂine RL is sample efﬁcient since no online interactions are required, but the performance is bounded by the ﬁxed data set.
Online RL enjoys more opportunities for performance improvement, but is comparatively much less sample efﬁcient.
Because of the connections and complementary strengths, instead of treating them as two isolated topics, it is more natural to connect both in pursuit of a performant policy in practice.
The Direct Ofﬂine-Online Approach.
Because of the above mentioned connections, it is tempting to directly use the same algorithm (thus the same network architectures as well) for both phases.
Unfortunately, this is ineffective in either directions : either directly using ofﬂine RL algorithms for online learning ( forward direction), or directly using online RL algorithms for ofﬂine learning ( reverse direction).
The reverse direction has been explored extensively in the ofﬂine RL community.
The current wisdom is that instead of directly using an existing online RL algorithm (e.g. TD3 (Fujimoto et al., 2018)), special treatments need to be incorporated into the algorithm (e.g. incorporating BC into TD3 (Fujimoto & Gu, 2021)) for handling the challenges arising in ofﬂine learning, due to issues such as querying the critic function with out-of-distribution actions.
In the forward direction, as noted in previous work (Nair et al., 2020), it is exceptionally difﬁcult to ﬁrst train a policy using ofﬂine data and then further improve it using online RL.
There are some efforts in the literature on directly transferring the parameters learned ofﬂine for online ﬁne-tuning, i.e. , by initializing the policy in Eqn.(1) with parameters learned ofﬂine.
This scheme is illustrated in
Figure 1 as the Direct approach for Ofﬂine-to-Online RL.
While simple, this approach has several potential issues as noted in the literature.
For example, one common issue is that the behavior of the ofﬂine policy can be compromised or even destroyed at the initial phase of online training, e.g. because of the noisy gradient for policy update due to cold start learning of critic network (Uchendu et al., 2022)
(as in the case of reward-free pre-training) or distribution shift between ofﬂine and
[Figure]
online dataset (Lee et al., 2021).
Another issue is the conservativeness of the ofﬂine RL algorithms.
While this is a desirable feature when considering only ofﬂine training, it is not preferred for online learning, where exploration is valuable for a further improvement (Rezaeifar et al., 2022).
This is a phenomenon that is commonly observed and reported in the literature (Lee et al., 2021; Campos et al., 2021; Uchendu et al., 2022).
B RIDGING O FFLINE AND O NLINE RL VIA P OLICY E XPANSION
In this section, we will introduce a simple scheme called Policy Expansion for bridging ofﬂine and online training.
It is worthwhile to note that the proposed scheme is orthogonal to speciﬁc off/online
RL algorithms and is compatible to be used with different value-based ofﬂine and online algorithms.
The ﬁnal performance of such a combination may vary depend on the selection of methods.
4. P OLICY E XPANSION AND A DAPTIVE C OMPOSITION
Policy Expansion.
To mitigate the above mentioned issues, we propose an alternative scheme that can be readily combined with existing algorithms.
The proposed approach is illustrated in Figure 1.
Given a policy π β obtained from ofﬂine training phase, instead of directly ﬁne-tuning the parameters, we freeze π β and add it into a policy set Π =
[ π β ] .
To enable further learning, instead of directly modifying π β as in the Direct method, which has the potential of destroying useful behaviors learned ofﬂine, we freeze π β and expand the policy set Π with another learnable policy π θ as
[Equation]
which is responsible for a further performance improvement during online training.
We refer this type of policy construction as Policy Expansion (PEX).
It is intuitive to understand that the behavior of ofﬂine policy π β is free from being negatively impacted, while the newly added policy can be updated.
The policies in the policy set Π will all get involved into exploration and learning in an collaborative and adaptive manner as detailed in the following.
Adaptive Policy Composition.
Both policies in the policy set Π will form a single composite policy ˜ π , which will be used in both exploration and learning.
More speciﬁcally, given the current state s , we ﬁrst sample actions for each member of the policy set Π and form a proposal actions set A = { a i ∼ π i ( s ) | π i ∈ Π } .
Then all the action proposals will be taken into consideration and they will be selected with the probability related to their potential utilities (e.g. values).
For example, we can compute their values at the current state Q φ =[ Q φ ( s, a i ) | a i ∈ A ] ∈ R K , with K denotes the cardinality of Π (here K =2 ), and construct a categorical distribution for selecting the ﬁnal action:
[Equation]
where α is temperature.
Then we can sample w from it for selecting the actions w ∼ P w to decide which action will be used during unroll for interacting with environment.
Using value for policy composition has been used in different contexts in the literature (Yu et al., 2021; Shah et al., 2022).
Conceptually, the composite policy ˜ π can be represented as follows:
[Equation]
where w ∈ R K a one-hot vector, indicating the policy that is selected for the current state s .
δ a ∼ π denotes the Dirac delta distribution centered at a which is sampled from π .
By allowing only the newly added policy ( π θ in this case) to be ﬁne-tuned while freezing all others ( π β ), we can avoid the problem of compromising (e.g. destroying or forgetting) the behavior of ofﬂine policy.
At the same time, we have the advantage of adaptiveness in the sense of allowing learning of new abilities.
From this perspective, the policy expansion plays the role of bridging the ofﬂine and online learning phases, while mitigating commonly encountered issues.
It is interesting to note that a similar compositional form of policy has appeared in DAgger (Ross et al., 2011), although with a uniform weight across states and under a different context of imitation learning.
Here our compositional weight is state adaptive and the compositional policy is used for bridging ofﬂine-to-online reinforcement learning.
PEX has several advantages compared to direct ofﬂine-online learning (illustrated in Figure 1):
ofﬂine policy preservation : it can retain the useful behaviors learned during ofﬂine training phase by retaining the policy and avoid it being destroyed in the initial online training phase; 2. ﬂexibility in policy form : the ofﬂine policy does not need to be of the same form with the online policy (e
g. same network structure) as in the direct ofﬂine-online approach, offering more ﬂexibilities in design; 3.
adaptive behavior selection : both the behavior of the ofﬂine policy and the online learning policy are used in interacting with the environment and they are involved in an adaptive manner, e.g., according to their respective expertise in handling different states.
Input: ofﬂine RL algorithm { L Q φ oﬄine , L π β oﬄine } , online RL algorithm
L Q φ online , L π θ online }
Initialize: network parameters φ , β , θ , ofﬂine replay buffer D oﬄine while in ofﬂine training phase do % ofﬂine policy training using batches from the ofﬂine replay buffer D oﬄine
[Equation]
←
[Equation]
end while
Policy Expansion: ˜ π
=
[ π β , π θ ] ; transfer Q φ while in online training phase do for each environment step do a t
[Equation]
t ) according to (6), s t
[Equation] [Equation]
s t
[Equation] [Equation]
s t ,
[Equation]
t
[Equation]
t
[Equation] [Equation]
t +1 ) } end for for each gradient step do % online training using batches from both D oﬄine and D φ ←
[Equation]
end for end while
4. B RIDGED O FFLINE -O NLINE T RAINING WITH P OLICY E XPANSION
We focus on value-based RL algorithms for both stages in this work.
For ofﬂine-training , we conduct ofﬂine RL with an ofﬂine RL algorithm
(e.g. IQL) on the ofﬂine dataset to obtain the ofﬂine policy π β .
Then we can construct policy expansion following Eqn.(4) before entering the online phase.
And we also transfer the Q function (critic) learned in the ofﬂine stage to online stage for further learning.
We also transfer the ofﬂine buffer to online stage as an additional buffer, as shown in Figure 1.
For online training , we use the policy adaptively composed from the policy set as in Eqn.(6), and then conduct online training by interleaving environmental interaction and gradient update.
The newly collected transitions are stored into the online replay buffer D .
For training, batches randomly sampled from both D and D oﬄine are used.
The value loss and policy loss are calculated based on the losses corresponding to the chosen algorithm.
The proposed scheme can be used together with different existing RL algorithms.
The complete procedure is summarized in Algorithm 1, taking an ofﬂine RL and online RL algorithm as inputs.
R ELATED W ORK
Pre-Training in RL.
A number of different directions have been explored in RL pre-training, including representation pre-training and policy pre-training.
Note that for RL, pre-training can be either ofﬂine or online.
Representative works include pre-training of the feature representation using standard representation learning methods (e.g. contrastive learning (Yang & Nachum, 2021)), dynamics learning-based representation learning (e.g. Schwarzer et al., 2021;
Seo et al., 2022), unsupervised RL driven by intrinsic rewards in reward-free environments (e.g. Liu & Abbeel, 2021), or directly using ImageNet pre-training for visual RL tasks (e.g. Shah & Kumar, 2021; Yuan et al., 2022).
Apart from representation pre-training, another category of work is on policy pre-training, with the goal of acquiring behaviors during the pre-training phase that are useful for the online phase.
When the downstream task is unknown, there are approaches for unsupervised pre-training, e.g., maximizing behavior diversity (Eysenbach et al., 2019) or converting the action space Singh et al.
(2021), with the hope of discovering some behaviors that are useful for the downstream task.
When the ofﬂine-online task is more aligned, there are some early attempts on directly transferring policy parameters (Rajeswaran et al., 2018), based on the intuition that a policy initialized this way can produce more meaningful behaviors than randomly initialized networks.
Some recent work focuses on behavior transferring, i.e. , leveraging the ofﬂine trained policy for exploration during online training (Campos et al., 2021; Uchendu et al., 2022).
Our work falls into this latter category of methods.
One notable difference compared to Campos et al.
; Uchendu et al.
(2022) is that for the proposed approach, the ofﬂine policy is one part of the ﬁnal policy, and with its role been determined adaptively.
Data-Driven RL and Ofﬂine RL.
Training a policy by leveraging a large amount of existing data ( i.e. data-driven RL) is a promising approach that is valuable to many real-world scenarios, where ofﬂine data is abundant.
Ofﬂine RL is one active topic towards this direction.
The main motivation of ofﬂine RL is to train a policy by leveraging a pre-collected dataset, without requiring additional environmental interactions (Levine et al., 2020).
Many research works in this direction focus on addressing the special challenges brought by ofﬂine learning, including out-of-distribution value issue (Kumar et al., 2020).
Common strategies include constraining the value (Kumar et al., 2020) to be small for out of distribution actions or the policy to be close to the action distribution of dataset (Fujimoto et al., 2019; Fujimoto & Gu, 2021).
Recently, Kostrikov et al.
(2022) proposes an implicit Q-learning (IQL) method as an alternative way to handle this issue.
It learns a value function that predicts a certain expectile of the values for state-action pairs from the dataset, which can be used for computing the value target without querying the critic with out-of-distribution actions.
Ofﬂine Training with Online Fine-tuning.
Combining ofﬂine data with online learning is an effective approach that have been demonstrated by several early attempts with demonstration data (Vecerík et al., 2017; Hester et al., 2018; Nair et al., 2018; Rajeswaran et al., 2018).
Traditionally, the ofﬂine
RL methods are purely focused on the ofﬂine training setting.
However, the ofﬂine learned policy could be limited in performance given a ﬁxed dataset.
In the case when further online interaction is allowed, it would be natural to ﬁne-tune the policy further with data collected online.
This paradigm of two-stage policy training is related to the iterative interleaved policy and data collection schemes used in imitation learning (Ross et al., 2011; Ross & Bagnell, 2012).
Several different approaches have been explored towards online ﬁne-tuning of a ofﬂine pre-trained policy, including balancing ofﬂine-online replay data (Lee et al., 2021), parameter transferring (Rajeswaran et al., 2018;
Xie et al., 2021), policy regularization (Rudner et al., 2021; Tirumala et al., 2020) and guided exploration (Campos et al., 2021; Uchendu et al., 2022).
It has been observed that directly applying some ofﬂine RL methods does not beneﬁt from online interactions (Nair et al., 2020; Uchendu et al., 2022), potentially due to the conservative nature of the ofﬂine policy (Nair et al., 2020).
Based on this observation, there are some recent efforts on developing algorithms that are not only suitable for ofﬂine training, but also can leverage online interactions for further learning.
Nair et al.
(2020) shows that an advantage-weighted form of actor-critic method is suitable for this purpose.
IQL (Kostrikov et al., 2022) also leverages a similar form for policy learning and shows that it can beneﬁt from online ﬁne-tuning.
Our work falls into this category and we provide an alternative approach for leveraging ofﬂine pre-training for helping with online training.
E XPERIMENTS
In this section, we ﬁrst evaluate the effectiveness of the proposed approach on various types of benchmark tasks with comparison to a number of baseline methods.
Then we further show a number of extensions where the proposed approach can also be applied.
6. O FFLINE - TO -O NLINE RL E XPERIMENTS
Tasks and Settings.
We use the standard D4RL benchmark which has been widely used in ofﬂine
RL community (Fu et al., 2020).
For ofﬂine learning, we use the provided dataset for training.
For online learning, we use the accompanied simulator for interaction and training.
For ofﬂine phase, 1M training steps are used for training.
Then we run online ﬁne-tuning for another 1M environmental steps.
Here we use IQL (Kostrikov et al., 2022) as the backbone algorithm for all methods listed below.
The training is repeated with 5 different random seeds.
Baselines.
We compare the proposed approach with the following baselines:
(i) Offline : ofﬂine training using IQL, without online ﬁne-tuning; (ii) Scratch : train IQL online from scratch, without ofﬂine-pre-training.
(iii) Buffer (Vecerík et al., 2017): train IQL online without ofﬂine pretraining, but has access to the ofﬂine buffer during online training, i.e. using a buffer as D ∪ D oﬄine .
(iv) Direct (Kostrikov et al., 2022): a direct ofﬂine-to-online approach by directly transferring parameters trained ofﬂine to online stage using IQL (Kostrikov et al., 2022), which is a recent and representative RL algorithm that shows state-of-the-art performance on ofﬂine RL while allows online ﬁne-tuning; (v) AWAC (Nair et al., 2020): an approach that uses an advantage-weighted form of actor-critic method for ofﬂine-to-online RL; (vi) Off2On
(Lee et al., 2021): a recent ofﬂineto-online RL method that uses an ensemble of ofﬂine trained value and policies together with a balanced ofﬂine-online replay scheme; (vii) BT (Campos et al., 2021): Behavior Transfer which is an approach that leverages the an ofﬂine learned policy in exploration, where the ofﬂine policy is used for exploration for a consecutive number of steps sampled from a distribution once activated; (viii)
JSRL (Uchendu et al., 2022):
Jump Start RL, which divides the rollout of a trajectory into two parts, using the ofﬂine learned policy for the ﬁrst part and then unrolling with the online learning policy for the rest of trajectory.
PEX denotes the proposed approach, which has the same ofﬂine and online RL algorithms as Direct , and is only different in using Policy Expansion for connecting the two stages.
Code 2 and more resources are on the project page.
[Figure]
The return curves for all the tasks are shown in Figure 2.
The aggregated return across all tasks is shown in Figure 3.
The returns are ﬁrst averaged across task and then across runs.
It can be observed that all methods show some improvements after online training in general, compared to the initial perfor-
[Figure]
mance before online training.
Scratch has the lowest overall performance across all tasks.
On the challenging sparse reward antmaze tasks, Scratch cannot learn a meaningful policy at all.
Buffer has better performance than Scratch when incorporating the ofﬂine buffer, indicating that the ofﬂine buffer has some beneﬁts in helping with learning.
Notably, on the previously zero-performance antmaze tasks, Buffer achieves reasonable performance with the help of the ofﬂine buffer, leading to a large overall improvement over Scratch , ( c.f. Figure 3).
Direct (IQL-based) shows large improvements over Offline on average as shown in Figure 3, implying the beneﬁts brought by the the additional online training over pure ofﬂine training.
Off2On also shows large improvement during ﬁne-tuning and achieves strong overall performance (Figure 3).
BT shows some improvements over IQL on some tasks such as antmaze-medium-play and antmaze-large-diverse , with an overall performance comparable to that of Direct .
JSRL outperforms Direct and BT on some tasks (e.g. hopper-medium , hopper-medium-replay ), potentially due to its different way of leveraging the ofﬂine policy, and its overall performance is similar to BT .
The proposed PEX approach performs comparably to other baselines on some tasks while outperforming all baseline methods on most of the other tasks ( c.f. Figure 2), and outperforms baselines methods overall ( c.f.
Figure 3), demonstrating its effectiveness.
[Figure] [Figure]
OfflineScratchBufferDirectAWACOff2OnPEX­IQLPEX
We have shown the application of PEX to the case where both the ofﬂine and online algorithms are the same (referred to as PEX-IQL here since both are IQL) in Section 6.1.
In this section, we further show the applicability of the proposed scheme in bridging heterogeneous
RL algorithms, i.e. different RL methods are used for the ofﬂine and online stages.
As an example, here we use IQL (Kostrikov et al., 2022) and SAC (Haarnoja et al., 2018) for ofﬂine and online RL stages respectively.
We compare with Scratch (vanilla SAC (Haarnoja et al., 2018)), Buffer (SAC with additional ofﬂine replay buffer) as well as Direct (directly transferring policy and critic parameters learned with ofﬂine IQL to SAC for further online learning).
Again
PEX uses the same ofﬂine and online algorithms as in Direct but uses policy expansion instead of the direct transferring approach.
The normalized return curves aggregated across tasks are shown in Figure 4.
Individual return curves are shown in Appendix A.8.
It can be observed that there is an overall improvement by simply applying PEX to the heterogeneous ofﬂine-online RL setting as well.
6. A BLATION S TUDIES
We will inspect the impact of several factors on the performance of the proposed method in the sequel.
[Figure]
N o r m a li z ed
R e t u r n
DisabledEnabled
Ofﬂine Buffer (
[Figure]
1.0 Environment Steps 1e6 0.
N o r m a li z ed R e t u r n DisabledEnabled Critic Transfer (
[Figure]
1.0 Environment Steps 1e6 0. N o r m a li z ed R e t u r n DisabledEnabled
Policy Transfer (c) 0. 1.0 Environment Steps 1e6 0.
N o r m a li z ed R e t u r n DisabledEnabled Ofﬂine Policy Freeze
[Figure]
Ofﬂine-Buffer.
This experiment investigates the impacts of including the ofﬂine replay buffer in online training stage.
The results are shown in Figure 5(a).
As can be observed, the inclusion of the ofﬂine replay buffer helps with the performance, but the performance drop caused by disabling ofﬂine replay buffer is smaller than the change of other algorithmic components ( c.f. Figure 5(b) ∼ (d)).
Critic Transfer.
This experiment studies the impacts of transferring the critic parameters trained from ofﬂine to online stage.
As can be observed from Figure 5(b), disabling critic transferring will also greatly decrease the performance both in terms of sample efﬁciency as well as ﬁnal performance.
Policy Transfer.
This experiment examines the impacts of transferring the ofﬂine pre-trained policy in the online training stage.
When policy transferring is disabled, there is no need to use policy expansion in the online training stage.
The results are shown in Figure 5(c).
It can be observed that there is a clear performance drop when policy transferring is disabled.
Ofﬂine Policy Freeze.
The ofﬂine learned π β is freezed during the online learning stage in PEX .
This experiment investigates the impact of this factor.
If disabled, π β will be trained in the same way as π θ during online learning.
It is observed from Figure 5(d) that freezing the ofﬂine policy is important.
Training by disabling policy freezing has a clear performance drop.
This is consistent with the intuition on the usefulness of ofﬂine policy preservation, which is one advantage of our approach.
More ablation results are deferred to Appendix A.4.
6. V ISUALIZATION AND A NALYSIS
Policy Composition Probability.
Since a composite policy is involved in the proposed approach, it is interesting to inspect the participation of each member policy from the policy set Π= { π β , π θ } when interacting with the environment.
We visualize the policy compositional probability P w during the rollout within a trajectory after training, as shown in Figure 6.
It can be observed that composition probability for each member policy is state-adaptive and is changing along the progress of a trajectory, implying that both policies contribute to the ﬁnal policy in an adaptive manner.
[Figure] [Figure]
State Space Associations of Member Policies.
To get a better understand of the role of the member policies in the policy set, we visualize the association between the states and its selected policy.
For this purpose, we embed a set of states into a 2D space using t-SNE (van der Maaten & Hinton, 2008), and then visualize the association of ofﬂine policy π β and the newly expanded policy π θ to states in the projected space.
States that select the ofﬂine policy π β are colored with blue and states that select π θ are colored with red.
It can be observed that π θ and π θ cover different parts of the state space, indicating that they have some complementary functionalities and are preferred differently at different states.
C ONCLUSIONS , L IMITATIONS AND F UTURE W ORK
We highlight the usefulness of properly connecting ofﬂine and online stages of reinforcement learning to gain the beneﬁts of both worlds and present a policy expansion scheme as an attempt toward this direction.
This scheme is an instance in the direction orthogonal to developing completely new ofﬂine-to-online RL algorithms.
The proposed approach is simple and can be combined with existing RL algorithms, and is illustrated with two different combinations in this work.
Experiments demonstrate the effectiveness of the proposed approach on a number of benchmark tasks.
While achieving promising performance, the proposed approach also has some limitations.
One limitation is that the number of parameters grows with the number of policies in the policy set.
While this might not be an issue when the policy set is small as the case in this work, it will be less parameter efﬁcient in the presence of a large policy set.
One possible way to address this issue is by introducing a distillation stage (Rusu et al., 2016), by consolidating the multiple policies into a network with a smaller number of parameters.
Generalizing the proposed scheme to the case with a set of pre-trained skill policies is an interesting direction (Eysenbach et al., 2019; Shu et al., 2018).
For ofﬂine learning, we have built upon the strong IQL method.
It would be interesting to see how much we can gain by upgrading it with more recently developed ofﬂine RL methods together with different online methods.
The idea of using a policy set itself can potentially be applied to other cases beyond ofﬂine to online
RL.
We leave the exploration of its generalization and application as an interesting future work.
A A PPENDIX A. D ETAILS ON B ASELINES
In this section we provide more details on the baseline methods.
Behavior Transfer ( BT ).
This approach is originally proposed for the case of pre-training using unsupervised reinforcement learning and then transfers the learned behaviors for later learning.
It is also originally used in discrete action cases.
Here we adapt this idea to build a baseline in our setting based on the approach of transferring behaviors in (Campos et al., 2021).
Following (Campos et al., 2021) a Zeta distribution with parameter a = 2 is used for determining the number of steps of persistent unroll using the ofﬂine policy.
The number of persistent unroll steps is re-sampled when currently not in the middle of a persistent unroll and with a random number sampled from [0 , 1] is smaller than a threshold (cid:15) .
(cid:15) =
1 is used in experiments.
Jump Start RL ( JSRL ).
JSRL divides the complete unroll of one trajectory into two parts, with the ﬁrst part obtained with the ofﬂine policy π β , while the second part is obtained with the current learning policy π θ .
The core idea of JSRL is to initially make the ﬁrst part longer, and then progressively reducing the length of the ﬁrst part, thus forming a type of curriculum to help with exploration and learning.
We follow this core idea by using a linear scheduler that anneals from the max episode length to 0 for progressively decreasing the ﬁrst stage of unroll using the prior policy π β .
Ofﬂine-to-Online RL ( Off2On ).
Off2On is an approach that aims to improve ofﬂine to online
RL performance (Lee et al., 2021).
It incorporates two components to cope with the challenges of ofﬂine-to-online RL: 1) an ensemble of critic networks and policy networks are trained during ofﬂine training stage, in contrast to typical ofﬂine RL, which only learns one policy network and critic network.
Following the setting in Lee et al.
(2021), ensemble size of 5 is used.
2) balanced replay: a balanced ofﬂine-online replay scheme to properly trade off between the usage of ofﬂine and online samples.
We used the code released by the authors of (Lee et al., 2021).
A. I MPLEMENTATION D ETAILS ON P OLICY E XPANSION
Policy Expansion ( PEX with IQL-backbone).
The proposed scheme can be combined with many different existing RL methods.
Here we illustrate with an concrete instantiation using IQL (Kostrikov et al., 2022).
For pre-training, we conduct ofﬂine RL with IQL on the ofﬂine dataset to obtain the ofﬂine policy π β .
Then we conduct policy expansion as Eqn.(6) before switching to online phase.
For ofﬂine training , we follow IQL (Kostrikov et al., 2022) to train the parameters of the value function using the TD-learning approach as follows:
[Equation]
where L τ 2 ( u ) =
| τ − 1 ( u < 0) | u 2 (Kostrikov et al., 2022), with τ denotes the expectile value (Kostrikov et al., 2022).
¯ φ denotes a set of target parameters that are periodically copied from φ .
The policy network π θ can be further updated by leveraging the updated critic network.
Here we again follow IQL (Kostrikov et al., 2022) and use the weighted form for actor training:
[Equation]
with w ( s, a ) =
exp(( Q ( s, a ) − V ( s )) /α ) and (cid:98)·(cid:99) denotes the gradient stopping operator.
For online training , we ﬁrst expand the policy set as in Eqn.(4), and then conduct online training by interleaving environmental interaction and gradient update.
The newly collected transitions are saved into replay buffer D .
For training, batches randomly sampled from both D and D oﬄine are used.
The values losses are the same as the ofﬂine phase, and the policy loss is computed against π θ :
[Equation]
[Equation]
For a 0 ∼ π β ( s t ) , in practice, we simply take the greedy action from the policy.
Following Kostrikov et al.
(2022), we model the policy using a non-squashed Gaussian distribution with mean squashed to action range, and use state independent standard deviations.
Policy Expansion ( PEX with with SAC as online backbone).
The main procedure of online training is the same as SAC (Haarnoja et al., 2018), with an adaptation for the actor training by changing the part of the loss based on Q-function from E s ∼D E a ∼ π θ (cid:13)(cid:13)(cid:13) (cid:98) ∂Q
( s,a ) ∂a
[Equation]
a
[Equation]
− a (cid:13)(cid:13)(cid:13) 2 (which is an equivalent form of loss to E s ∼D E a ∼ π θ − Q ( s, a ) ) to E s ∼D E a ∼ [Equation] π,
[Equation]
A. T HEORETICAL J USTIFICATION OF PEX
Here we provide some theoretical justiﬁcation of the proposed form of adaptive policy compostional distribution P w in Eqn.(5):
[Equation]
where K denotes the number of actions and in our case K = 2 .
P w can be viewed as a discrete policy with action dimension of two, which selects between two candidate actions a 1 , a 2 , one from ofﬂine policy π β and the other from π θ , i.e., a 1 ∼ π β and a 2 ∼ π θ .
Ideally, it assigns a higher probability to actions with higher values at the current state s .
The deﬁnition of P w essentially a reﬂection of this intuition, Actually, it can also be derived as shown below.
By deﬁnition, we have the following form for the compositional policy ˜ π :
[Equation]
which is essentially another form of Eqn.(6).
For V-value, we can have the following derivations
[Equation] [Equation]
(plug in the deﬁnition of policy in Eqn
[Equation] [Equation]
[Equation]
Applying entropy regularization term for policy learning
[Equation]
we have
[Equation]
Solving for P w given π β and π θ
[Equation]
we have
[Equation]
Therefore, for the case of two actions
[Equation]
we have
[Equation]
A. A DDITIONAL A BLATION R ESULTS
We provide more ablations experiments and results in this section.
Double
Parameters.
Since there are two member policies in the proposed approach.
A natural question to ask is whether properly double the number of parameters of a policy could achieve similar performance.
1.0 Environment Steps 1e6 0.
N o r m a li z ed R e t u r n
Double­Param PEX
DoubleParameters To verify this, we construct a type of policy network with (roughly) doubled number of parameters as π θ ( s )=
g θ g ◦ f θ f ( s ) , where f θ f ( s ) encodes the observation s into a feature vector and g maps the feature vector to the action distribution.
To (approximately) double the number of parameters, we implement f θ f as f θ f ( s ) (cid:44) f
θ 1 ( s )+ f θ 2 ( s ) .
f θ 1 is initialized by transferring parameters θ learned in the pretrain stage.
The aggregated return curves are shown on the right, for doubling the number of policy parameters as described above ( Double-Param ), and the proposed approach.
It can be observed that by comparison that simply increasing the number of parameters cannot help much with the performance, implying the importance of the proposed structure in PEX .
A. M ORE D ETAILS ON T ASKS
We summarized the tasks used for experiments and their corresponding D4RL environment names in
Table 1.
[Table]
A. H YPER - PARAMETERS
The hyper-parameters and values are summarized in Table 2.
There are two algorithmic related hyperparameters inherited from IQL since we instantiate PEX based on it: temperature α (corresponding to β − 1 in IQL paper (Kostrikov et al., 2022)) and expectile value τ .
The rest of the hyper-parameters are common ones in RL algorithms, including for example batch size, network structure and size, target update etc.
We use the hyper-parameter values from the IQL paper (Kostrikov et al., 2022) in our experiments.
A. H YPER - PARAMETER A BLATIONS
Impact of α .
α scales the Q values before constructing P w :
[Equation]
We show the impacts of its value on performance in Figure 7.
We have use a ﬁxed value ( α − 1 = for all antmaze tasks and α − 1 = 3 for all locomotion tasks according to the IQL paper (Kostrikov et al., 2022)
(the β parameter of IQL).
Here we show the impacts of different values for α − 1 by setting the inverse temperature as 0 .
α − 1 , 1 α − 1 , 2 α − 1 .
We note that other values are possible for different environments.
While it is impractical to tune this parameter for each task, it is possible to make it more adaptive by auto-tuning in a way similar to the temperature turning scheme used in
SAC (Haarnoja et al., 2018).
[Figure] [Figure]
Results with inverse temperature as
Impact of Policy Entropy .
The entropy of π θ is auto-tuned to match a target entropy (Haarnoja et al., 2018), which is empirically set as e target (cid:44) − d where d denotes the dimensionality of action.
This is inherited from SAC and the same setting ( − d ) is applied to all tasks.
To show the impact of different target entropy values, we experiment with three different settings: 0 .
5 e target , 1 · e target , 2 e target on several tasks.
The results are summarized in Figure 8.
It can be observed that overly small or large target entropy values could decrease the performance on some tasks
[Figure]
based online stage).
Results with target entropy as 0 .
5 e target , 1 e target , 2 e target respectively.
A. M ORE R ESULTS ON H ETEROGENEOUS O FFLINE -O NLINE RL
Here we show detailed return curves for each task when using SAC (Haarnoja et al., 2018) as the backbone online RL algorithm.
The results are shown in Figure 9.
A. O FFLINE -O NLINE T RAINING WITH R EWARD -F REE O FFLINE S ETTING
We have shown using policy expansion for bridging ofﬂine IQL with online IQL/SAC in the main paper.
Here we further demonstrate the applicability of the proposed approach by using BC as the
[Figure]
ofﬂine learning module.
We use IQL as the online module for illustration.
In this section, we further demonstrate the applicability of the proposed approach to the setting with ofﬂine dataset without reward annotations.
This is sometimes an encountered setting in practice, where plenty of ofﬂine data exist (e.g. from previous iteration of experiments or other agents), but without reward annotations.
Since the ofﬂine dataset is only broadly related to the task and could potentially contain irrelevant or noisy data ( w.r.t.
the current task in consideration), direct behavior cloning could be sub-optimal.
Standard ofﬂine RL methods are inapplicable for ofﬂine pre-training in this case.
We therefore use behavior cloning ( BC )
(Pomerleau, 1988), which is a simple and widely used approach for imitation learning.
After ofﬂine pre-training using BC , the ofﬂine policy π β is obtained and transferred to the second stage of online training.
In terms of Algorithm 1, only the policy loss L π β is present in the ofﬂine training phase.
The ofﬂine datasets not used in online training since it has no rewards.
[Figure]
We use subset of tasks from D4RL (Fu et al., 2020) where BC has non-trivial performances.
The results are shown in Figure 10.
It can be observed that JSRL , which performs well under the case of having ofﬂine rewards, performs less well in the reward-free case.
One potential reason is that it hinges on the ofﬂine policy to induce the state distribution (Uchendu et al., 2022).
In the case where the ofﬂine policy is not highly performant, as the case here obtained from BC , its effectiveness could be compromised.
BT performs comparable to or better than Direct online ﬁne-tuning with
IQL, potentially due to the fact that BT leverage the π β in a way that has a shorter commitment length for each time the ofﬂine policy is selected.
The proposed approach also shows similar online improvements comparable or sometimes better than the best performing methods.
A. P OLICY U SAGE
We show the policy usage of π β and π θ during online unroll along the process of one training session for BT and JSRL as well as BC-PEX (pre-trained with BC) and IQL-PEX (pre-trained with IQL) in
Figure 11.
The return curves of BC-PEX are shown in Figure 10.
The corresponding return curves of
IQL-PEX are shown in Figure 2.
As can be observed, the usage of ofﬂine policy is changing along the course of training, and the dynamics of the usage evolution is different for different tasks.