Supported Policy Optimization for Offline Reinforcement Learning
Anonymous Author(s)
AffiliationAddress
Abstract
Policy constraint methods to offline reinforcement learning (RL) typically utilizeparameterization or regularization that constrains the policy to perform actionswithin the support set of the behavior policy.
The elaborative designs of parameterization methods usually intrude into the policy networks, which may bring extrainference cost and cannot take full advantage of well-established online methods.
Regularization methods reduce the divergence between the learned policy andthe behavior policy, which may mismatch the inherent density-based definition ofsupport set thereby failing to avoid the out-of-distribution actions effectively.
Thispaper presents S upported P olicy O p T imization (SPOT), which is directly derivedfrom the theoretical formalization of the density-based support constraint.
SPOTadopts a VAE-based density estimator to explicitly model the support set of behavior policy and presents a simple but effective density-based regularization term,which can be plugged non-intrusively into off-the-shelf off-policy RL algorithms.
SPOT achieves the state-of-the-art performance on standard benchmarks for offline
RL.
Benefiting from the pluggable design, offline pretrained models from SPOTcan also be applied to perform online fine-tuning seamlessly.
Introduction
Offline RL [23, 24], where the agent learns from a fixed dataset, collected by arbitrary process,not only provides a bridge between RL and the data-driven paradigm but also eliminates the needto interact with the live environment, which is always expensive or risky in practical scenarios[14, 26, 16].
Unfortunately, the absence of environment interaction also raises a number of challenges.
Previous work has shown that the extrapolation error of the Q-function queried by out-of-distributionactions significantly degrades the performance of off-policy algorithms [8].
Avoiding out-of-distribution actions, namely to constrain the learned policy to perform actions withinthe support set of the behavior policy, is essential to mitigate extrapolation error.
To meet this supportconstraint, policy constraint methods [24] to offline RL utilize either parameterization [8, 43, 10] orregularization [13, 21, 40] techniques.
However, there are still several drawbacks in previous methodsof policy constraint, limiting their performance and applications.
Firstly, parameterization methodsinvolve elaborate designs of the policy parameterization, typically coupled to generative models ofthe behavior policy, to directly constrain actions taken by the learned policy.
But these designs intrudeinto the architecture of policy networks, which may bring extra inference cost and supplementarydifficulties to implement and tune offline RL algorithms.
Furthermore, as criticized by Fujimoto and
Gu [6], these intrusive designs complicate causal attribution of performance gains and transfer oftechniques between offline RL algorithms or from well-established online RL algorithms.
In contrast,for the second category, regularization methods are designed with a non-intrusive or pluggablemanner, which is done by simply augmenting a penalty to the actor loss that measures the divergence of the learned policy from the behavior policy [13, 21, 40, 19].
However, these divergence-basedregularization methods may mismatch the inherent density-based definition of support set, whichapplies support constraint indirectly .
As we show in experiments (Section 5.1), the divergence-baseddesign prevents regularization methods from effectively avoiding out-of-distribution actions and thuslimits their performance.
In this work, we aim to design a pluggable offline RL method that also directly meets the standard formalization of the support constraint based on the density of the behavior policy.
We introduce S upported P olicy O p T imization (SPOT), a regularization method which can be pluggednon-intrusively into off-the-shelf off-policy RL algorithms.
SPOT involves a regularization termfrom the new perspective of explicit estimation of the behavior density.
Concretely, SPOT adoptsa VAE-based density estimator [17, 31] to explicitly model the support set of behavior policy andpresents a simple yet effective regularization term directly applied to the estimated density.
Our method benefits from a closer connection between theory and algorithm, thereby achievingsuperior performance compared to previous methods.
It also profits from the pluggable algorithmicdesign, which leads to efficient inference and minor algorithmic modifications on top of standardoff-policy algorithms.
Moreover, a minimal gap between offline learning objectives and standardonline learning objectives also enables SPOT to take full advantage of existing online RL algorithmsand attain strong online fine-tuning performance after offline RL, exceeding state-of-the-art methods.
The main contributions of this work are three-fold:
• We derive a regularization term for constrained policy optimization in offline RL, basedon the theoretical formalization of the density-based support constraint, which directlyregularizes the behavior density of actions taken by the learned policy.
• We propose Supported Policy OpTimization (SPOT), a practical algorithm with a neural
VAE-based density estimator to implement the proposed regularization term.
• Compared to several strong baselines, SPOT achieves state-of-the-art results on standardoffline RL benchmarks [
5] and also outperforms previous methods when online fine-tunedafter offline RL initialization.
Related Work
[Table]
Our work belongs to the family of policyconstraint methods in offline RL, where parameterization or regularization is typicallyutilized.
Comparison between previous policy constraint methods and ours has beensummarized in Table 1.
In addition to thepolicy constraint methods, we also reviewbehavior policy modeling in offline RL,which is commonly necessary to both parameterization and regularization methods.
Lastly, we briefly discuss a broader rangeof competitive offline RL approaches.
Policy Constraint via Parameterization.
Careful parameterization of the learned policy can naturallysatisfy the support constraint.
For example, BCQ [8] learns a generative model of the behavior policyand trains the actor to perturb randomly generated actions.
The policy parameterized by BCQ is togreedily select the one maximizing Q function among a large number of perturbed sampled actions.
EMaQ [10] simplifies BCQ by discarding the perturbation models.
PLAS [43] learns a policy in thelatent space of the generative model and parameterizes the policy using the decoder of the generativemodel on top of the latent policy.
Policy Constraint via Regularization.
An alternative approach is to use a divergence penalty inorder to compel the learned policy to stay close to the behavior policy, such as KL-divergence [13, 40],maximum mean discrepancy (MMD) [21], Fisher divergence [19] and Wasserstein distance [40].
Using divergence penalties alleviates the hard constraint of explicit parameterization but may bearthe risk of out-of-distribution actions.
Although BEAR [21] attempts to constrain the policy to thesupport of the behavior policy, it heavily relies on the empirically-found approximate property of low-sampled MMD.
TD3+BC [6] simply adds a behavior cloning term to the policy update andpresents competitive performance on simple locomotion tasks.
Note that recent SBAC [41] proposesa new policy learning objective based on performance difference lemma [15], along with a densitybased regularization term similar to ours, but our work simply plugs the term into the standard policytraining objective to enjoy minimal algorithmic modifications.
Behavior Policy Modeling in Offline RL.
Most policy constraint methods need to fit an accurategenerative model of the behavior policy, to sample in-distribution actions or estimate behavior density.
Conditional variational auto-encoders (CVAE)
[17, 34] are typically used by past works [8, 43] tosample actions, while EMaQ [10] opt for using an autoregressive model [9] which enables moreaccurate sampling.
On the other hand, policy class of Gaussian [21, 40] or Gaussian mixture models[19] are commonly used to fit and estimate the density of behavior policy.
Instead of explicitlyfitting the behavior policy, recent approaches have utilized implicit constraint without ever queryingthe values of any out-of-sample actions [42, 20].
Although MBS-QL [25] uses ELBO to estimatemarginal state distribution, to the best of our knowledge, we are unique in estimating the density ofbehavior policy based on VAE, for its flexibility to capture almost arbitrary class of distributions [18].
Broader Range of Offline RL Approaches.
Besides policy constraint methods based on parameterization or regularization, there exist more types of competitive offline RL approaches.
IQL[20] designs a multi-step dynamic programming procedure based on expectile regression, whichcompletely avoids any queries to values of out-of-sample actions.
Pessimistic value methods, suchas CQL [22], produce a lower bound on the value of the current policy to effectively alleviateoverestimation, but their performance may suffer from excessive pessimism.
Advantage-weightedregression [29, 28, 38] improves upon behavior policy, while simultaneously enforcing an implicit
KL-divergence constraint.
Recent sequence modeling methods based on Transformers [37] also showcompetitive performance in both model-free [4] or model-based [12] paradigm.
Background
The reinforcement learning problem [35] is formulated as decision making in the environmentrepresented by a Markov Decision Process (MDP) M
= ( S , A , ρ 0 , p, r, γ ) , where S is the state space,
A is the action space, ρ 0 ( s 0 ) is the initial state distribution, p ( s ′ | s, a ) is the transition distribution,r ( s, a ) is the reward function, and γ is the discount factor.
The goal in RL is to find a policy π ( a | s )maximizing the expected return: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] .
The optimal state-action value function or Q function Q ∗ ( s, a ) measures the expected returnstarting in state s taking action a
and then acting optimally thereafter.
A corresponding optimal policy can be obtained through greedy action choices
π ∗ ( s ) =
arg max a Q ∗ ( s, a ) .
The Qlearning algorithm [39] learns Q ∗ via iterating the Bellman optimality operator T , defined as:
T Q ( s, a ) =
E s ′
[ r ( s, a ) +
γ max a ′ Q ( s ′ , a ′ )] .
For large or continuous state space, the value can berepresented by function approximators Q θ ( s, a ) with parameters θ .
In practice, the parameters θ areupdated by minimizing the mean squared Bellman error with an experience replay dataset D and atarget function Q ¯ θ [27]: J Q ( θ ) =
E ( s,a,r,s ′ )
∼D [ Q θ ( s, a ) − r − γ max a ′ Q ¯ θ ( s ′ , a ′ )] 2 .
In a continuous action space, the analytic maximum is intractable.
Actor-Critic methods [35, 7, 11]perform action selection with a separate policy function π ϕ maximizing the value function:
[Equation]
The policy can be updated following the deterministic policy gradient (DPG) theorem [33]:
[Equation]
3.1 Offline Reinforcement Learning
In contrast to online RL methods, which interact with environment to collect experience data, offline
RL [23, 24] methods learn from a finite and fixed dataset D = { ( s, a, r, s ′ ) } which has been collectedby some unknown behavior policy π β .
Direct application of off-policy methods on offline settingsuffers from extrapolation error [8, 21], which means that an out-of-distribution action a in state scan produce erroneously overestimated values Q
θ ( s, a ) .
3.2 Support Constraints in Offline RL
To avoid out-of-distribution actions from function approximation, support constraint [8, 21, 10] iscommonly used, which means in state s to only allow action that has ϵ -support under behavior policy:{ a : π β ( a | s ) > ϵ } .
Kumar et al.
[21] first introduce the distribution-constrained operator , which isinstantiated to the supported operator in this work:
Definition 3.1.
Given behavior policy π β and threshold ϵ , the supported backup operator is definedas
[Equation]
with its fixed point Q ∗ ϵ ( s, a ) named as the supported optimal Q function .
Following the theoretical analysis of Kumar et al.
[21], we can obtain a bound of how suboptimal Q ∗ ϵmay be with respect to the optimal Q function Q ∗ :
Corollary 3.2.
Let α ( ϵ ) =
∥T Q ∗ − T ϵ Q ∗ ∥ ∞ = max s,a |T Q ∗ ( s, a ) − T ϵ Q ∗ ( s, a ) | .
The suboptimality of Q ∗ ϵ ( s, a ) can be upper-bounded as ∥ Q ∗ − Q ∗ ϵ ∥ ∞ ≤
− γ α ( ϵ ) .
Note that the complete theoretical results from Kumar et al.
[21] also includes a term w.r.t.
thebootstrapping error.
We refer readers to Kumar et al.
[21] for more details.
Similar results are alsopresented by Ghasemipour et al.
Supported Policy Optimization
As aforementioned, performing support constraint is the typical method to mitigate extrapolation errorin offline RL.
Noticing that the support constraint can be formalized based on the density of behaviorpolicy, we propose the S upported P olicy O p T imization (SPOT) as a regularization method fromthe new perspective of explicit density estimation.
Concretely, SPOT involves a new regularizationterm, which is directly derived from the theoretical formalization of the support constraint.
Besides,a conditional VAE is adopted to explicitly estimate the behavior density in the regularization term.
Plugged into off-policy RL algorithms, we will finally arrive at the practical algorithm of SPOT.
4.1 Support Constraint via Behavior Density
Similar to how the optimal policy can be extracted from the optimal Q function, the supported optimalpolicy can also be recovered via greedy selection: π ∗ ϵ ( s )
=
arg max a : π β ( a | s ) >
ϵ Q ∗ ϵ ( s, a ) .
For thecase of function approximation, it corresponds to a constrained policy optimization problem.
While prior works use specific parameterization of π [8, 10] or divergence penalty [21, 40] to performsupport constraint, we propose to directly use behavior density
π β ( ·| s ) as constraint 1 :
[Equation]
where ˆ ϵ = log ϵ for notational simplicity.
Constraint via behavior density is simple and straightforwardin the context of support constraint.
We adopt log-likelihood instead of raw likelihood because of itsmathematical convenience.
This problem imposes a constraint that the density of behavior policy is lower-bounded at every pointin the state space, which is impractical to solve due to the large even infinite number of constraints.
Following previous works from both online RL [32] and offline RL [21, 29] w.r.t.
constrained policyoptimization, we instead use a heuristic approximation that considers the average behavior density:
[Equation]
Converting the constrained optimization problem into an unconstrained one by treating the constraintterm as a penalty, we finally get the policy learning objective as
[Equation]
6) where λ is a Lagrangian multiplier.
4.2 Explicit Estimation of Behavior Density
The straightforward regularization term in Eq.
(6) requires access to π β .
While we only have offlinedata generated by π β , we can explicitly estimate the probability density at an arbitrary point with thedensity estimation methods [1].
The variational autoencoder (VAE) [17] is among the best performing neural density-estimation models and we opt to use a conditional variantional autoencoder [34] as our density estimator.
Typically,π β ( a | s ) can be approximated by a Deep Latent Variable Model p ψ ( a | s ) =
(cid:82) p ψ ( a | z, s ) p ( z | s )
d zwith a fixed prior p ( z | s ) =
N ( 0 , I ) .
While the marginal likelihood p ψ ( a | s ) is intractable, VAEadditionally uses an approximate posterior q φ ( z | a, s )
≈ p ψ ( z | a, s ) and parameters ψ and φ can beoptimized jointly with evidence lower bound (ELBO):
[Equation]
After training a VAE, we can simply use −L ELBO to lower-bound log p ψ ( a | s ) and thus approximatelylower-bound log π β in Eq.
However, there theoretically exists a bias between them, as we knowlog p ψ ( a | s ) =
−L ELBO + KL( q φ ( z | a, s ) ||
p ψ ( z | a, s )) .
To obtain an estimation with lower bias, wecan use the importance sampling technique [31, 18]:
[Equation]
Burda et al.
[3] show that (cid:92) log π β ( a | s ; φ, ψ, L ) gives a lower bound of log p ψ ( a | s ) and the boundbecomes tighter as L increases.
Note that here we adopt sampling of VAE to directly estimate thedensity of the behavior policy instead of to estimate the divergence [21].
In summary, the loss function in Eq.
(6) can be implemented with the explicit density estimator asfollows:
[Equation]
4.3 Practical Algorithm
The general framework derived above can be built on top of off-policy algorithms with minimalmodifications.
We choose TD3 [7] as our base algorithm, which recently shows strong resistance tooverestimation in offline RL [6] (see Section 5.2 for a detailed discussion).
Q normalization.
Following TD3+BC [6], we add a normalization term to policy loss asa default option for better balance between Q value objective and regularization: J π ( ϕ ) =
E s ∼D (cid:104) − Q
θ ( s,π ϕ ( s ))
α − λ (cid:92) log π β ( π ϕ ( s ) | s ; φ, ψ, L ) (cid:105) , where α = 1 N (cid:80) s i
| Q ( s i , π ϕ ( s i )) | is the normalization term based on the minibatch { s i } Ni =1 with size N .
Simpler density estimator.
While (cid:92) log π β ( a | s ; φ, ψ, L ) with large L is much tighter, we empiricallyfind there is no further improvement with larger L compared to L = 1 (see supplementary materialsfor results of ablation study).
To make our algorithm simpler, we choose to only use L = 1 fora practical estimator, which is just the ELBO estimator of the VAE: (cid:92) log π β ( a | s ; φ, ψ, L = 1) =
−L ELBO ( s, a ; φ, ψ ) .
Note that with L = 1 , we can analytically separate out the KL divergence as
Eq.
(7) to enjoy a simpler and lower-variance update.
Overall algorithm.
Putting everything together, the full algorithm is summarized in Algorithm 1.
Our algorithm first trains VAE using L ELBO ( s, a ; φ, ψ ) to obtain a density estimator with sufficientaccuracy.
Then it turns to policy training analogous to common Actor-Critic methods except that weplug the regularization term computed by the density estimator into the policy loss J π ( ϕ ) .
Input: Dataset D = { ( s, a, r, s ′ ) } //
VAE Training
Initialize VAE with parameters ψ and φ for t = 1 to T 1 do
Sample minibatch of transitions ( s, a ) ∼ D
Update ψ, φ minimizing L ELBO ( s, a ; φ, ψ ) in Eq.
(7) end for // Policy Training
Initialize the policy network π ϕ , critic network Q θ and target network Q ¯ θ with ¯ θ ← θ for t = 1 to T 2 do
Sample minibatch of transitions ( s, a, r, s ′ ) ∼ D
Update θ minimizing J Q ( θ ) in Eq.
Update ϕ
minimizing J π ( ϕ ) in Eq.
Update target network: ¯ θ ← τθ + (1 − τ )¯ θ end for
Experimental Evaluation
Our experiments aim to evaluate our method comparatively, in contrast to prior offline RL methods,focusing on both offline training and online fine-tuning.
We first demonstrate the effect of λ onapplying support constraint and show that our method is able to learn a policy with the strongestperformance at the same level of constraint strength, compared to previous policy constraint methods.
We then evaluate SPOT on D4RL benchmark [5], studying how effective our method is in contrastto a broader range of state-of-the-art offline RL methods.
Finally, we study how SPOT compares toprior methods when fine-tuning with online RL from an offline RL initialization, and investigate thecomputational efficiency of different methods.
See supplemental materials for the code.
5.1 Analysis of Support Constraint in SPOT
[Figure]
[Figure] [Figure]
of λ .
[Figure] [Figure]
Effect of λ on constraint strength.
The coefficient λ in SPOT is essential and corresponds to aspecific constraint strength in the constrained policy optimization problem formalized in Eq.
Toillustrate how λ effects the learned policy, we evaluate behavior density of actions taken by the policylearned with varying values of λ ∈ [0 . 05 , 0 . 1 , 0 . 2 , 0 . 5] on standard D4RL [5] Gym-Mujoco domains.
Concretely, we plot the distribution of behavior density log π β ( π ϕ ( s ) | s ) , s ∼ D in Figure 1a, wherelog π β is estimated by our learned density estimator (Eq.
(8)) with L set to a sufficient large number500 for more accurate estimation.
As we show, with smaller λ , the learned policy is much morepossible to perform actions with low behavior density log π β ( π ϕ ( s ) | s ) .
On the other hand, policieslearned by higher λ are restricted to take only high-density actions.
Tradeoff between constraint strength and optimality.
It has been shown by Kumar et al.
[21]that the optimality of approximate supported optimal policy is lower-bounded by a tradeoff betweenkeeping the learned policy supported by the behavior policy (controlling extrapolation error) and
[Table]
keeping the supported policy set large enough to capture well-performing policies.
If the constraintin Eq.
(4) is strong (by a large log ϵ ), the extrapolation error is restrained to be small but the optimalpolicy under constraint may have poor performance.
Otherwise, if the constraint is weak, wellperforming policies can be learned though at the risk of the extrapolation error.
We aim to answer the question that at the same level of constraint being satisfied, is SPOT able tolearn a policy with the strongest performance compared to previous policy constraint methods?
Wecompare SPOT with BC [30], BCQ [8], BEAR [21], PLAS [43] and TD3+BC [6].
Hyperparametersto control constraint strength of various methods are adjusted to several values to form a spectrum ofconstraint strength (see supplementary materials for details).
We approximate the satisfied constraintstrength log ϵ by the 5th-percentile of the distribution log π β ( π ϕ ( s ) | s ) , s ∼ D .
As shown in Figure1b, our method SPOT takes shape of the “upper envelope" of all methods, demonstrating that takingadvantage of exact standard formulation of support constraint, SPOT is flexible to learn the supportedoptimal policy and resist extrapolation error at the same time.
Note that the divergence-basedregularization method BEAR [21] yields a poor performance in our experiments and only satisfiesa loose constraint in contrast to other baselines, showing that it cannot prevent out-of-distributionactions effectively and suffers from extrapolation error with indirect divergence regularization.
5.2 Comparisons on Offline RL Benchmarks
Next, we evaluate our approach on the D4RL benchmark [5] in comparison to state-of-the-art methods.
We focus on Gym-MuJoCo locomotion domains and much more challenging AntMaze domains,which consists of sparse-reward tasks and requires “stitching" fragments of suboptimal trajectoriestraveling undirectedly in order to find a path from the start to the goal of the maze.
Baselines.
We select the classic BC [30] and state-of-the-art offline RL methods as baselines.
Formethods based on dynamic programming, we compare to AWAC [28], Onestep RL [2], TD3+BC [6],
CQL [22], and IQL [20].
We also include the sequence-modeling method Decision Transformer [4].
Hyperparameter tuning.
The weight λ in Eq.
(9) is essential for policy constraint.
Following priorworks [2, 21, 40], we allow access to the environment to tune a small ( < 10 ) set of the hyperparameterλ , which is reasonable for practical applications.
See supplementary materials for more discussions.
Gym-MuJoCo domains.
Results for the Gym-MuJoCo domains are shown in Table 2.
As we show,
SPOT substantially outperforms state-of-the-art methods, especially in suboptimal “medium" and“medium-replay" datasets with a large margin, which further demonstrates the advantages of directconstraint on behavior density proposed by SPOT.
AntMaze domains.
Results for the AntMaze domains are shown in Table 3.
Note that D4RL recentlyreleases a bug-fixed “-v2" version of AntMaze datasets, and thus we select competitive baselines andrerun their author-provided implementations for comparison.
We also include results for BCQ and
BEAR trained on “-v0" datasets, directly from [5].
It is not suitable to compare them with reproducedbaselines but we include them in order to highlight that previous policy constraint methods struggleto succeed in training on challenging AntMaze domains.
As shown in Table 3, SPOT performs slightly worse than IQL but outperforms remaining modernoffline RL baselines, including the pessimistic value method CQL and the sequence modeling method
[Table]
Decision Transformer.
Note that IQL ingeniously designs multi-step dynamic programming andpolicy extraction steps to apply an implicit constraint for offline RL, but when online fine-tuned afteroffline RL initialization, IQL is inferior to SPOT.
SPOT’s pluggable design can take full advantageof existing online RL algorithms (see Section 5.3 and Table 4).
To the best of our knowledge,our algorithm is the first to train successfully in challenging AntMaze domains with pluggablemodification on top of off-policy RL methods for offline RL.
Ablation.
In Figure 2, we perform an ablation study over the components in our method.
First, wereplace the VAE-based density estimator with a behavioral-cloned Gaussian policy model [21, 40].
As expected, it degrades the performance due to the lack of flexibility to model complex distribution,especially on Gym-MuJoCo medium-expert datasets and AntMaze datasets.
Then, we evaluate SPOTwithout Q normalization and find that the removal provides some benefits as well as damages ondifferent datasets, with only an insignificant impact on total performance.
Nevertheless, we include itas a default option following TD3+BC [6].
Lastly, we investigate how the base off-policy algorithm matters.
Since SAC [11] has similar offpolicy performance with TD3, we attempt to adopt SAC as our base method.
As Xu et al.
[41]argue that the entropy term will do harm to the offline setting, we also implement a variant of SACwithout maximum entropy regularization.
Surprisingly, both variants are vulnerable to pathologicalextrapolation error on AntMaze domains and provide poor performance (see Figure 2).
We arguethat several key native designs of TD3 are beneficial to offline RL.
For the case of learning astochastic policy without entropy regularization, we find that the learned policy quickly degeneratesinto a deterministic one (with near-zero standard deviation).
A concern with deterministic policiesis that they are prone to overfit overestimated actions and propagate the estimation error through
Bellman backups, which is even serious in offline RL.
Hopefully, TD3 introduces Target Policy
Smoothing [7, 36] into Eq. 1, which adds random noise to target actions and can alleviate the effect oferror propagation.
For the case with entropy regularization, Bellman backup of a stochastic policyresembles Target Policy Smoothing, then the primary distinction may come from the actor learningobjective.
While TD3 produces a deterministic action minimizing Eq.
(9), stochastic policies are morelikely to produce out-of-distribution actions with erroneous estimated values, thus the policy gradientmay become biased as well as with high variance.
On much easier Gym-MuJoCo domains, SAC-style
SPOT variants are comparable to the TD3 variant on most of the datasets, but we remark that TD3indeed extends the limit of policy constraint methods on some tasks, such as hopper-medium.
Ouranalysis suggests that TD3 may be preferable to offline RL as the base off-policy method with native
[Figure]
designs (such as “stochastic” critic training and deterministic actor training) addressing functionapproximation error not only in the online setting but also in the offline setting [6].
5.3 Online Fine-tuning after Offline RL
Pluggable SPOT can be online fine-tuned seamlessly, which means that we only need to graduallydecay the constraint strength λ in the online phase in order to avoid excessive conservation.
As AWAC[28] shows that behavior models are hard to update online, we fix the VAE during online fine-tuning.
Note that when λ is zero, our algorithm is exactly the standard off-policy RL algorithm that SPOTbuilds upon.
It is beneficial since we enjoy a minimal gap between offline RL and well-establishedonline RL methods and can take full advantage of them for online fine-tuning.
[Table]
Since IQL [20] is the strongest baseline in our offline experiments, whichalso has shown superior online performance than prior methods [28, 22]in its paper, and most of the otherbaselines fail to learn meaningful results, we follow the experiments of
IQL and compare to IQL in onlinefine-tuning.
We also compare to ourbase RL method TD3 [7] trained online from scratch.
We use the challenging AntMaze domains [5].
Results are shown in Table 4.
While online training from scratch fails in the challenging sparse reward tasks on AntMaze domains, SPOTinitialized with offline RL succeeds to learn nearly optimal policies and performs significantly betterthan the strongest baseline IQL, especially in the most difficult large maze.
5.4 Computation Cost
[Figure]
Regularization methods, including our SPOT, benefitingfrom the pluggable design, only need one forward pass ofthe policy network to do inference, while parameterizationmethods always need inference through secondary components, such as the generative model or the critic network,which may bring extra time or memory cost.
As demonstrated empirically in Figure 3, parameterization methodsare usually slower than regularization methods.
TD3+BCand our SPOT run more than two times faster compared tothe most time-consuming BCQ, while SPOT also has theadvantage of parameterization methods, which explicitlyconstrain the behavior density of learned actions.
SPOT indeed adds some training overhead due to the VAE-based density estimator, but is still muchsimpler than most methods.
A comparison of training time is provided in supplementary materials.
SPOT lies in the second tier, only slightly worse than PLAS and beaten by the simplest TD3+BC.
Conclusion
We present Supported Policy OpTimization (SPOT), a policy constraint method to offline RL builtupon off-the-shelf off-policy RL algorithms.
Capturing the standard formulation of the supportconstraint, SPOT introduces a pluggable regularization term applied directly to the estimated behaviordensity and obtains excellent performance across different tasks in the D4RL benchmarks, includingstandard Gym-MuJoCo tasks and challenging AntMaze tasks.
Furthermore, when online fine-tunedafter offline RL pre-training, the pluggable design of our algorithm makes it seamless to take fulladvantage of well-established online methods and exceed the state-of-the-art on the challenging
AntMaze domains. One limitation of our current method, shared by most policy constraint methods,is that the performance may be limited by the accuracy of estimation of the behavior policy. Anexciting direction for future work would be to develop an effective pluggable constraint mechanismexcluding explicit estimation of behavior policy.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4 insupplementary materials.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 4 in supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 1 in
supplementary materials.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We have
uploaded the code to reproduce the main results of our methods (Table 2, 3, 4 in the
main text) in supplementary materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 3 in supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] See Section 5 in the main text and Section 3 in
supplementary materials .
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.4 in the main text
and Section 3.5 in supplementary materials.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] All the datasets we used are open-sourced and we comply with
the license.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] All the data and experiments are simulated
without any personally identifiable information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]