A Policy-Guided Imitation Approach for Offline Reinforcement Learning
Anonymous Author(s)
Abstract
Offline reinforcement learning (RL) methods can generally be categorized into twotypes: RL-based and imitation-based.
RL-based methods could in principle enjoyout-of-distribution generalization but suffer from erroneous off-policy evaluation.
Imitation-based methods avoid off-policy evaluation but are too conservative tosurpass the dataset.
In this study, we propose an alternative approach, inheritingthe training stability of imitation-style methods while still allowing logical out-ofdistribution generalization.
We decompose the conventional reward-maximizingpolicy in offline RL into a guide-policy and an execute-policy.
During training, theguide-policy and execute-policy are learned using only data from the dataset, in asupervised and decoupled manner.
During evaluation, the guide-policy guides theexecute-policy by telling where it should go so that the reward can be maximized.
By doing so, our algorithm allows state-compositionality from the dataset, ratherthan action-compositionality conducted in prior imitation-style methods.
We dumbthis new approach Policy-guided Offline RL ( POR ).
POR demonstrates the stateof-the-art performance on D4RL, a standard benchmark for offline RL.
We alsohighlight the benefits of POR in terms of improving with supplementary suboptimaldata and easily adapting to new tasks by only changing the guide-policy.
Introduction
Offline RL, also known as batch RL, allows learning policies from previously collected data, withoutonline interactions [28, 30].
It is a promising area for bringing RL into real-world domains, such asrobotics [20], healthcare [45] and industrial control [60].
In such scenarios, arbitrary exploration withuntrained policies is costly or dangerous, but sufficient prior data is available.
While most off-policy
RL algorithms are applicable in the offline setting by filling the replay buffer with offline data,improving the policy beyond the level of the behavior policy often entails querying the value function(i.e.
, Q function) about values of actions that were not seen in the dataset.
Those out-of-distributionactions can be deemed as adversarial examples of the Q function [27], which cause extrapolationerror of the Q-function.
The error will be accumulated by the deadly triad issue [48], propagateacross the state-action space through the iterative dynamic programming procedure [4], and can notbe eliminated without requiring growing batch of online samples [30].
To allieviate this issue, prior model-free offline RL methods typically add a behavior regularizationterm to the policy improvement step, to limit how far it deviates from the behavior policy.
Thiscan be achieved explicitly by calculating some divergence metrics [25, 52, 36, 12], or implicitlyby regularizing the learned value functions to assign low values to out-of-distribution actions [27,23, 2, 3].
Nevertheless, this imposes a trade-off between accurate value estimation (more behaviorregularization) and maximum policy improvement (less behavior regularization).
To avoid suchproblem, recently a branch of methods bypass querying the values of unseen actions by performing some kind of imitation learning on the dataset 1 .
This can be achieved by filtering trajectories basedon their return [38, 8], or reweighting transitions based on how advantageous they could be under thebehavior policy [4, 24], or just be conditioned on some variables without any dataset reweighting[7, 10].
Although imitation-style methods enjoy stable training process and are able to effectivelyperform multi-step dynamic programming by assigning the proper reweighting weight or conditionedvariable [24, 10], they only allow action-compositionality from the dataset, lose the ability to surpassthe dataset by out-of-distribution generalization, which only appears in RL-based methods.
In this work, we propose an alternative approach.
We aim at inheriting the training stability ofimitation-style methods while still allowing logical out-of-distribution generalization.
To do so, wepropose Policy-guided Offline RL ( POR ), an algorithm that employs two policies to solve tasks: aguide-policy and an execute-policy.
The job of the guide-policy is to learn the optimal next stategiven the current state, and the job of the execute-policy is to learn how different actions can producedifferent next states, given the current state.
By this manner, we decompose the original task of
RL (i.e., reward-maximizing) into two distinct yet complementary tasks, and this decompositionmakes each task much easier to be solved.
During evaluation, the guide-policy serves as a guidefor the execute-policy by telling it where to go so that the reward can be maximized.
By doing so,our algorithm allows state-compositionality from the dataset, rather than action-compositionalityconducted in prior work, which owns logical out-of-distribution generalization.
Our method is easy to implement by only adding the learning of a guide-policy, the training stage ofthe guide-policy and execute-policy are decoupled, using in-sample learning from the dataset.
We test
POR in widely-used D4RL offline RL benchmarks and demonstrates the state-of-the-art performance,especially on low-quality datasets that require out-of-distribution generalization to achieve a highscore.
We also show that by decoupling the learning of guide-policy and execute-policy, we canenhance the guide-policy with supplementary suboptimal data, or re-learn the guide-policy to adaptto new tasks with different reward functions, without changing the execute-policy.
Related Work
RL-based offline approach
A large portion of offline RL methods are RL-based.
RL-basedmethods typically augment existing off-policy methods (e.g., Q-learning or actor-critic) with abehavior regularization term.
The primary ingredient of this class of methods is to propose variouspolicy regularizers to ensure that the learned policy does not stray too far from the behavior policy.
These regularizers can appear explicitly as divergence penalties [52, 25, 12], implicitly throughweighted behavior cloning [50, 38, 36], or more directly through careful parameterization of thepolicy [13, 63].
Another way to apply behavior regularizers is via modification of the critic learningobjective to incorporate some form of regularization to encourage staying near the behavioraldistribution and being pessimistic about unknown state-action pairs [35, 27, 23, 56].
There are alsoseveral works incorporating behavior regularization through the use of uncertainty.
The uncertaintyquantification can be done via Monte Carlo dropout [53] or explicit ensemble models [3, 59, 21, 61].
Note that the behavior regularization weight is crucial in RL-based methods.
With a small weight,
RL-based methods could in principle enjoy out-of-distribution generalization since they perform truedynamic programming.
However, a small weight will make erroneous off-policy evaluation due todistribution shift and the policy is extremely vulnerable to the value function misestimation.
Imitation-based offline approach
Another line of methods, on the contrary, performing some kindof imitation learning on the dataset, without the need to do off-policy evaluation.
When the dataset isgood enough or contains high-performing trajectories, we can simply clone the actions observed inthe dataset [39], or perform some kind of filtering or conditioning to extract useful transitions.
Forinstance, recent work filters trajectories based on their return [8, 38], or directly filters individualtransitions based on how advantageous these could be under the behavior policy and then clones them[4, 24, 5, 15, 54].
Conditioned BC methods are based on the idea that every transition or trajectory isoptimal when conditioned on the right variable [14, 10].
By this way, after conditioning, the databecomes optimal given the value of the conditioned variable.
The conditioned variable could be thecumulative return [7, 18, 26], or the goal information if provided [33, 14, 10, 57].
A recent study[10] shows that goal-conditioning can be super useful in D4RL AntMaze datasets.
However, this introduces the assumption that we have some prior information about the structure of the task, whichis beyond the standard offline RL setup.
Our method can be viewed as a combination of RL-based and imitation-based methods.
We adoptan imitation-style manner to train the goal-policy and execute-policy.
When evaluation, however,following the guide-policy, the execute-policy could produce out-of-distribution actions by leveragingthe generalization capacity of the function approximator.
Preliminaries
Offline RL
We consider the standard fully observed Markov Decision Process (MDP) setting [44].
An MDP can be represented as M = ( S , A , T, r, ρ, γ ) where S is the state space, A is the actionspace, T ( ·| s, a ) is the transition probability distribution function, r ( s, a ) is the reward function, ρ isthe initial state distribution and γ is the discount factor, we assume γ ∈ (0 , 1) in this work.
The goalof RL is to find a policy π ( a | s ) : S ×
A → [0 , 1] that maximizes the expected cumulative discountedreward (or called return) along a trajectory as
[Equation]
In this work, we focus on the offline setting.
The goal is to learn a policy from a fixed dataset B = 102 (cid:8) τ i = (cid:0)(cid:0) s [Equation]
0 , a [Equation] [Equation] s
[Equation] [Equation] (cid:1) , (cid:0) s
[Equation]
, r iH (cid:1)(cid:1)(cid:9) N i =1 consisting of trajectories thatare collected by different policies, where H is the time horizon.
The dataset can be heterogenous andsuboptimal, we denote the underlying behavior policy of B as µ , which represents the conditionaldistribution p ( a | s ) observed in the dataset.
RL via Supervised Learning
Conventional RL methods generally either compute the derivativeof (1) with respect to the policy directly via policy gradient methods [41, 19], or estimate a valuefunction or Q-function by means of TD learning [51], or both [42, 16].
In the RL via Supervised
Learning framework, we avoid the complex and potentially high-variance policy gradient estimators,as well as the complexity of temporal difference learning.
Instead, we perform conditioned behaviorcloning on some extra information, such as goal state, cumulative return, or language description[33, 14].
When applying the framework to the offline RL setting, we can take the offline dataset B asinput and find the outcome-conditioned policy π that optimizes
[Equation]
(2) where ω is an outcome conditioned on the remaining trajectory, i.e., ω ∼ g ( ·| τ t : H ) , and we useτ i : j = ( s i , a i , . . .
, s j ) to denote a fragment of the trajectory.
Policy-guided Offline Reinforcement Learning
We now continue to describe our proposed approach, POR .
To have a clear understanding of theout-of-distribution generalization of POR , we begin with a motivating example which demonstratesthat by allowing state-compositionality on the dataset, the agent is able to generalize to a much betterpolicy than doing action-compositionality.
Then we describe how we accomplish this idea in themore general case and introduce the algorithm details of POR .
Finally we give a theoretical analysisof POR , we are interested to see how close the policy induced by POR and the optimal policy could be.
4.1 A Motivating Example
We first use a simple, didatic toy example to illustrate the importance of state-stitiching for offline RL.
Let’s consider the navigation task shown in Figure 1, where the goal is to find the shortest path fromthe starting location (bottom-left corner) to the goal location (top-right corner) in the grid world.
Thestate and action space in this task are discrete, the state consists of the ( x, y ) coordinate and the agentcould choose to move to any of its nearby 8 grids (i.e., the action space is 8).
The agent receivesa reward of 0 for entering the goal state and -1 reward for all other transitions.
The offline dataset(green-arrowed lines) is consisted of two suboptimal trajectories from the start location to the goallocation, plus transitions of taking random actions at random states.
We use the grid color (more saturated to purple meanshigher value) to show the value function V ( s ) computed from the offline dataset.
We compare two methods that use the value function differently.
At a states b ∼ B , the first method chooses the action a b in thedataset that leads to s ′ b with the highest V ( s ′ b ) ; thesecond method chooses the action a that leads to s ′ inthe dataset with the highest V ( s ′ ) , note that a is notnecessarily to be within the dataset.
The first methodis doing action "stitching" in the dataset while thesecond method is doing state "stitching".
Althoughboth methods are able to "stitch" suboptimal trajectories together from the dataset and successfully travelfrom the start location to the goal location, the actionstitching method takes more steps (11 steps) compared with the state-stitching method (7 or 8 steps),as shown in Figure 1.
The action-stitching method introduced above is actually an abstract methodology of imitation-basedmethods, which imitate the transitons in the datasetunequally with different weight choices.
However,these methods are too conservative to find the optimaltrajectory, especially when there does not exist onein the offline dataset.
To generate better-than-datasettrajectories (e.g., red-arrowed lines), the agent needs to take logical out-of-distribution actions.
Note that in the toy example, we give the agent the freedom to choose any action that leads to thehighest-value state in the dataset.
This allows the most out-of-distribution generalization and isrealizable in some simple tasks (e.g., discrete state and action space) with high action coverage, forinstance, in the toy example, given historical ( s, a, s ′ ) results at (5 , 4) , the agent can successfullyrecognize that taking upper-right will arrive at (6 , 5) .
However, in the more general case, especiallythe continuous state and action space setting, this may make the agent do erroneously-generalizedactions as the same state can hardly be observed twice.
To perform logical out-of-distributiongeneralization, we need a Prophet , i.e., the guide-policy, to help the agent by telling it which state itshould (high reward) and can (logical generalization) go to.
It turns out that the involvement of theguide-policy also brings some theoritical meaning, we will discuss that in Section 4.4.
4.2 Learning the Guide-Policy
Recall that the goal of the guide-policy is to guide the execute policy about which state s it shouldgo.
To accomplish that, we train a state value function V ( s ) :
S → R .
The training of V usesonly ( s, s ′ )
samples in the offline dataset, it doesn’t suffer from overestimation because there’re noout-of-distribution actions involved.
To approximate the optimal value function in the dataset, weadopt tricks from recent work by giving the ℓ 2 loss with different weight using expectile regression[24], yielding the following asymmetric ℓ 2 loss
[Equation]
It can be seen that when τ = 1 / 2 , this operator is reduced to Bellman expectation operator, whilewhen τ → 1 , this operator approaches Bellman optimality operator.
Note that our learning objectivebears similarity with IQL [24], however, here we aim to learn a state value function while IQL aimsto learn a state-action value function so as to extract the policy.
To do this, IQL needs to learn both Qand V ( V is used to isolate the effect of state and approximate the expectile of Q only with respect tothe action distribution), the training of V and Q are coupled and may affect each other.
Simply maximizing the guide-policy with respect to V ( s ) will result in a state that the execute-policymay make erroneous generalization.
To allieviate this issue, we add a behavior cloning term to thelearning objective of the guide-policy.
Denote the guide-policy as g ω ( s ) : S → S , the learningobjective of g is given by
[Equation]
where the weight α serves as the trade-off between guiding to space with high-reward and space thatthe execute-policy ought to have a correct generalization.
4.3 Learning the Execute-Policy: Training and Evaluation
Algorithm
Require: B , τ , α .
Training 2: Initialize V ϕ , V ϕ ′ , g ω , π θ 3: for t = 1 , 2 , · · · , N do 4:
Sample transitions ( s, r, s ′ ) ∼ B 5:
Update V ϕ by Eq.(3) 6:
Update g ω by Eq.(4) 7:
Update V ϕ ′ by ϕ ′ ← λϕ + (1 − λ ) ϕ ′ 8: end for 9: for t = 1 , 2 , · · · , M do 10:
Sample transitions ( s, a, s ′ ) ∼ B 11:
Update π θ by Eq.(5) 12: end for 13: //
Evaluation 14: Get initial state s , set d as False 15: while not d do 16:
Get action a form Eq.(6) 17:
Roll out a and get ( s ′ , r, d ) 18:
Set s = s ′ 19: end while
After learning the guide-policy, now we turn to theexecute-policy.
Since the job of the execute-policyis to having a strong generalization ability, we adoptthe RL via Supervised Learning framework by conditioning the execute-policy on s ′ that encounted inthe dataset.
To be more specific, denote the executepolicy as π θ ( s, s ′ ) :
S × S → [0 , 1] .
During training,π performs supervised learning by maximizing thelikelihood of the actions given the states and nextstates, yielding the following objective
[Equation]
During evaluation, given a state s , the final action isdetermined by both the guide-policy and the executepolicy, by
[Equation]
Note that our learning objective (5) differs from previous RL via Supervised Learning methods in that weonly use the next state s ′ as the conditioned variable,there’s no need to estimate the cumulative return [7]and it also works in settings where we don’t know thegoal information [10].
Also, during evaluation, previous methods show that the choice of conditionedvariable is crucial important as little changes will cause significant performance difference [7].
Inour method, owing to the existence of the guide-policy, the optimal conditioned variable can beautomatically generated.
Our final algorithm, POR , consists of three supervised stages: learning V , learning g and learning π .
We summarize the training and evaluation procedure of POR in Algorithm 1.
Note that the training ofg and π are fully decoupled, this brings POR some nice properties to improve with supplementarydata or transfer to new tasks, we will further investigate it in the experiments.
4.4 Analysis
In this section, we give a theoritical analysis of POR .
Concretely, we aim to 1) give the lower bound ofthe performance difference between POR and the optimal policy π ∗ , 2) analyze how the guide-policyg will influence this bound.
We begin by introducing the notation and assumptions used in our analysis.
We denote P ( s, s ′ ) :
S × S →
A as the inverse transition operator .
We denote a g and a as the ground truth of π ( s, g ( s ))and π ( s, s ′ ) , respectively.
We also denote ϵ :=
sup ( s,a,s ′ ) ∈B
∥ π ( s, s ′ )
− a ∥ as the upper bound of theapproximation error in the dataset during training.
We then introduce the following two assumptions.
Assumption 1.
(Non-lazy MDP)
The inverse transition operator P ( s, s ′ ) is L 1 -Lipschitz continuous,i.e., ∥
[Equation]
) ∥ .
Assumption 2.
(Lipschitz continuous function approximators)
The execute-policy we trained is
L 2 -Lipschitz continuous, i.e., ∥ π ( s 1 , s ′ 1 ) − π ( s 2 , s ′ 2 ) ∥
≤ L 2 ∥ ( s 1 , s ′ 1 ) − ( s 2 , s ′ 2 ) ∥ .
Assumption 1 holds when actions do have effects on state transitions (i.e. if a ̸ = 0 , then ∥ s ′ − s ∥
≥ϵ s > 0 ).
We refer the MDP that satisfies Assumption 1 as non-lazy MDP and others as lazy MDP .
Itshould be mentioned that most real-world tasks belong to the non-lazy MDP case, it is meaninglessto study under the lazy MDP case becase in lazy MDP , action changes will have little effect on statetransitions, making the policy learning in vain.
Assumption 2 is a mild assumption that is frequentlyutilized in plenty of works [49, 34, 31].
Then in Theorem 1, we show that under this two assumptions, the gap between the optimal actionand the action output by POR can be bounded.
Theorem 1.
(Single step gap to optimal action).
The single step gap between optimal action and theaction induced by our method can be bounded as
[Equation]
Theorem 1 states that single step optimal gap is related to three parts: l 1 , l 2 , l 3 .
l 1 is the generalizationperformance factor.
It can be found that a small value of ∥ g ( s ) − s ′ ∥ enables a small l 1 , whichindicates the necessity to constrain the guide-policy to stay close to the dataset.
l 2 denotes thesuboptimality constant.
Intuitively, a loosened constraint on the guide-policy may induce a smallsuboptimality constant but on the other hand, may suffer from the risk of a high value of l 1 .
Theguide-policy g serves as a trade-off between l 1 and l 2 , we could balance the constraint strength andsuboptimality by adjusting α in Eq.(4) to obtain the lowest bound.
l 3 is related to the approximationerror on the training data, which can be reduced via improving the representation ability of functionapproximators but may suffer from a high L 2 due to the overfitting caused by over-parameterization.l 1 also depends on the Lipschitz constant L 1 and L 2 .
Note that a smooth inverse transition operatorwill result in a small value of L 1 and L 2 (because π ( s, s ′ ) is trained to approximate P ( s, s ′ ) ), thismeans our method may achieve better performance under a smoother inverse transition.
Based on Theorem 1, we can also give the performance bound of our method by replacing the ϵ in
Theorem 3.2 in [14] with the RHS of Eq.(7): J ( π ∗ ) − J ( π ) ≤ sup s,a,h ( l 1 + l 2 + l 3 ) T , where T isthe horion of each episode, h is the time step.
Experiments
We present empirical evaluations of POR in this section.
We first evaluate POR against other baselinealgorithms on D4RL [11] benchmark datasets.
We then explore deeper on the guide-policy about thebenefits from decoupled training process.
We finally establish ablation studies on the execute-policy.
5.1 Comparative Experiments on D4RL Benchmark Datasets
We first evaluate our approach on the D4RL MuJoCo and AntMaze datasets [11].
Notice thatmost of MuJoCo datasets (except random and medium datastes) consist of a large fraction of nearoptimal trajectories, while AntMaze datasets contain few or no near-optimal trajectories.
Hence thoselow-quality datasets can serve as a good testbed for quantifying the compositionality ability.
We compare POR with both RL-based and imitation-based baselines.
For RL-based baselines, weselect the best one from 5 state-of-the-art algorithms, including BCQ [13], BEAR [25], BRAC [52],
CQL [27] and TD3+BC [12].
For imitation-based baselines, we further split them into Weighted
BC methods and Conditioned BC methods.
Weighted BC methods include 10%BC [7], One-step
RL [4] and IQL [23].
10%BC is a filtered version of BC that runs behavior cloning on only the top10% high-return trajectories in the dataset.
One-step RL and IQL can be viewed as using differentweights to do behavior cloning (see Appendix A for discussion).
Conditioned BC methods include
DT [7] and RvS [10].
DT is built with Transformer and conditioned on the cumulative return.
RvSwell studied the effect of different conditional information on different tasks.
The score of RvS ischosen by the higher score between RvS-R and Rvs-G, which use the reward-to-go and oracle goalinformation as the conditioned variable, respectively.
Note that we also categorize our method PORinto conditioned BC methods because the execute-policy is learned in a similar manner with otherconditioned BC methods.
Note that the reuslts of Filter BC are from our own implementation, andthe results of IQL on MuJoCo random tasks are re-runed using the author-provided implementation.
Other results are taken directly from their corresponding papers.
To enable fairness and consistencyin the evaluation process, we train our algorithm for 1 million time steps and evaluate every 5000time steps that consisting of 10 episodes.
Full experimental details are included in Appendix D.1 andthe learning curve can be found in Appendix E.
The results are shown in Table 1, in MuJoCo locomotion tasks, POR performs competitively to the bestperformance of prior methods in high-quality datasets, while achieving much better performance than
[Table]
other methods in low-quality datasets (e.g., random and medium datasets).
On the more challenging
AntMaze tasks, POR outperforms all other baselines by a large margin.
It even surpassed RvS thatuses the oracle goal information to learn the policy.
We conjecture that the success in all those datasetshould be credited to the out-of-distribution generalization ability of POR .
5.2 Validation Experiments on Out-of-Distribution Generalization
In this section, we try to validate our hypothesis about the out-of-distribution generalization of POR .
Weare especially interested to see how the guide-policy learned in those high-dimensional tasks in D4RL.
To do so, we compare the distribution of g ( s ) generated by the guide-policy with the distributionof s ′ from the offline dataset.
We choose antmaze-large-diverse and hopper-medium-replay ,two datasets for evaluation.
For each dataset, we randomly collect 100 , 000 ( s, s ′ ) transition tuples.
To visualize the results clearly, we plot the distribution of g ( s ) and s ′ with t-Distributed Stochastic
Neighbor Embedding (t-SNE) [17], we also visualize the true values of all states using the optimalvalue function V ∗ ( s ) .
The results are shown in Figure 2, it can be found that in both tasks, the distribution of g ( s ) bears somesimilarity to the distribution of s ′ .
However, when looking at the non-overlapped area of g ( s ) ands ′ , we found that g ( s ) has a much higher value than s ′ , especially in the antmaze-large-diversedataset.
Those out-of-distribution and high-value area are exactly where POR generalized to, note thatthe high-value benefit of the single-step generalization could also be accumulated and propagatedthrough time steps, resulting a much better policy.
5.3 Investigations on the Guide-Policy
In this section, we investigate what is the potential benefit of decoupling the training process of theguide-policy and execute-policy.
We found that as the execute-policy is not task-specific and onlycares about the effect of actions to different states, we can reuse its powerful generalization ability
[Figure]
when encounted with additional suboptimal data or adapting to a new task.
We only need to re-trainthe guide-policy as it is task-specific.
Improve guide-policy by additional suboptimal data
We first study the setting where we alreadyhave a policy learned from a small yet high-quality dataset D e , and now we get a supplementarylarge dataset D o .
D o may be sampled from one or multiple behavior policies, it provides higher datacoverage but could be suboptimal.
We are interested to see whether we can use D o to obtain a betterpolicy.
This setting is realistic as low-quality datasets appear more often in real-world tasks [58, 55].
Naïvely, we can combine D e and D o as a new offline dataset B , then run any off-the-shelf offline
RL algorithm.
We label this training scheme as "more".
We also label the original training schemethat only uses D e as "main".
We test these two training schemes on two state-of-the-art offline RLalgorithms, TD3+BC and CQL.
The results shown in Figure 3 indicate that both "main" and "more"are insufficient to learn a good policy, The reason why "main" fails is because the resulting policymay not be able to generalize due to the limited size.
While "more" can generalize better than "main"due to access to a much larger dataset, the policy will be negatively impacted by the low-quality datain D o as both TD3+BC and CQL contain the behavior regularization term.
Due to the availability of decoupled training, POR is able to use a different training scheme, we cankeep the execute-policy learned from D e unchanged but use the combination of D e and D o to re-trainthe guide-policy.
The value function will be more accurate and better generalized when combining
D e with D o to have a larger data coverage (see exploration in online RL as an example).
We labelthis unique training procedure as "mix".
It can be shown in Figure 3 that, by doing so, POR couldhave a super large improvement over "main" and "more".
The version "mix" brings POR up to 99 ,
[Figure]
close to the upper limit of the performance in the hopper environment, with only limited data.
In thewalker2d environment, POR with "mix" also outperforms all other baselines.
Change guide-policy in new tasks
Then, we study whether we can use the guide-policy to do alightweight adaptation to new tasks, without changing the execute-policy.
We conduct experimentson the continuous variant of the classic four-room environment [18].
The training data consists oftrajectories collected by a goal-reaching controller with the start and end location sampled randomly atnon-wall locations, mixed with data collected by a random policy.
In this environment, we introducethree different tasks: Task A ( Four-room ) needs the agent to travel from the start location to the goallocation, besides that,
Task B ( Four-room-river ) and task C
( Four-room-key ) require the agentto bypass the river or get a key before arriving at the goal location.
We first train the guide-policy and the execute-policy π to solve task A.
In task B and task C, were-train the guide-policy using the corresponding reward function in that task, and reuse the executepolicy π from Task A.
In Figure 4, we can see that by doing so, POR is able to successfully solve task
B and C.
In details, in task B, POR is aware of the river and correctly bypass it.
In task C, POR onlychooses to pass the bottem door, this is owing to the awareness of the importance of obtaining thekey before reaching the goal.
This result shows the strong adaptation ablity of the guide-policy fordifferent tasks, as well as the generlization ability of the execute-policy across different tasks.
Conclusions and Limitations
In this work, we propose a novel offline RL approach, POR , which leverages the training stability ofimitation-style methods while still encouraging logical out-of-distribution generalization.
POR allowsstate-compositionality rather than action-compositionality from the dataset.
Through theoreticalanalysis and extensive experiments, we show that POR outperforms prior methods on a variety ofdatasets, especially those with low quality.
We also empirically demonstrate the additional benefits of
POR in terms of improving with supplementary suboptimal data and easily adapting to new tasks.
Onelimitation of POR is that the prediction error of the guide-policy may be large when the state space ishigh-dimensional.
However, this can be alleviated with the help of representation learning [29, 62]. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6. (c) Did you discuss any potential negative societal impacts of your work? [Yes] SeeSection 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplement material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix D.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
In all related figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Apendix D.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Data is from [11].
(b) Did you mention the license of the assets?
[Yes] The license is Apache 2.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplement for code and four room dataset.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A] Data is simulated.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A] Data is simulated.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
A More Discussion
Why One-step and IQL are imitation-based methods?
The core difference between RL-basedand imitation-based methods is that RL-based methods learn a value function of policy π whileimitation-based methods don’t.
Learning the value function of π requires off-policy evaluation ofπ (i.e., learning Q π or V π ), which is prone to distribution shift.
The policy evaluation and policyimprovement will also affect each other as they are coupled.
Imitation-based methods don’t learn Q π or V π , but some of them do learn a value function.
We callthese methods imitation-based because they learn the value function using only dataset samples, thevalue function actually tells how advantageous it could be under the behavior policy, much like imitation learning.
Also, the policy learning objective of One-step (with exponentially-weighted improvement operator) and IQL can be written as L ( π ) =
E ( s,a ) ∼D [exp ( β ( Q ( s, a ) − V ( s ))) log π ( a | s )] ,which uses dataset actions to doing behavior cloning with different weights (One-step and IQL learndifferent value functions).
Cloning dataset actions can only do action-stitching, which loses the ability to surpass the dataset byout-of-distribution (action) generalization.
For example, in our toy example in Section 4.1, actionstitching methods at most learn the shortest path that contained in the datset.
This is suboptimalespecially when there does not exist one complete path starting from the start location to the goallocation in the offline dataset.
How can we do beyond action-stitching?
One way is to use RL-based methods, by querying anaccurate Q π , we can get a different-yet-optimal action a π , but Q π is hard to estimate.
Another wayis like what POR did, we learn an out-of-distribution state indicator, i.e., the guide-policy g , to guidethe policy to the optimal next state.
If the execute-policy can generalize well, it will also output adifferent yet optimal action a = arg max a π ( a | s, g ( s )) .
B More Related Work
Our work decouples the state-to-action policy into two modules, i.e., the guide-policy and the executepolicy.
The execute-policy is actually an inverse dynamics mmodel, which has been widely usedin various ways in sequential decision making.
In exploration, inverse dynamics can be used tolearn representations of the controllable aspects of the state [37].
In imitation learning, [47] and [32]train an inverse dynamics model to label the state-only demonstrations with inferred actions.
[9] useinverse dynamics models to translate actions taken in a simulated environment to the real world.
Recently, there has been an emergence of work [43, 14] highlighting the connection of imitationlearning and reinforcement learning.
Specifically, rather than learn to map states and actions toreward, as is typical in reinforcement learning, [43] trains a model to predict actions given a state andan outcome, which could be the amount of reward the agent is to collect within a certain amount oftime.
[14] uses a similar idea, predicting actions conditioned on an initial state, a goal state, and theamount of time left to achieve the goal.
These methods are perhaps the cloest work to our algorithm,however, we study the offline setting and motivate the usage of an inverse dynamics model from adifferent perspective (i.e., state-stitching).
C Proof
In this section, we provide the proof of Theorem 1.
Proof.
We can rewrite the LHS of Eq.(7) as
[Equation]
D Experimental Details
In this section we provide the experimental details in our paper.
We use the following hardware andsoftwares for our training:
• GPUs: NVIDIA GeForce RTX 3080Ti• Python 3.7• Tensorflow 1.15.0 [1]• Gym 0.23.1 [6]• MuJoCo 2.1.4 [46]• mujoco-py 2.1.2.
D.1 D4RL ExperimentsData collection
The datasets in D4RL have been generated as follows:
random : roll out a randomlyinitialized policy for 1M steps.
expert : 1M samples from a policy trained to completion with SAC[16].
medium : 1M samples from a policy trained to approximately 1/3 the performance of the expert.medium-replay : replay buffer of a policy trained up to the performance of the medium agent.medium-expert : 50-50 split of medium and expert data..
For all datasets we use the v2 version.
Implementation details
Our implementation of 10%BC is as follows, we firstly filter the top 10 %trajectories in terms of the trajectory return, and then run behaviour cloning on those filtered data.
The hyperparamters of POR are present in Table 2.
Note that POR has two unique hyperparamters: τfor the value function and α for the guide-policy, τ controls the degree of upper quantile of V and αserves as the penalty weight of the BC loss of g .
For τ , it has been well stuided in IQL
and we choosethe same setting, that is, 0 . 9 in AntMaze and 0 .
7 in MuJoCo tasks.
For α , we set it 0 .
2 in AntMazetasks and 0 .
5 in MuJoCo tasks.
We keep all other hyperparamters the same in all tasks.
We adoptstate normalization [40] to reduce the prediction error of the guide-policy, it can be deemed as annaïve method of representation learning.
[Table]
POR Hyperparameters.
[Table]
D.2 Additional Suboptimal Data Experiments
Data collection and settings
In this experiment, we use different part of medium-replay datasets as D e and D o .
More specific,we use 30% to 70% transitions to constitute D e and use 20% to 80% transitions to constitute D e ∪D o
[Table]
Implementation details
We use the same hyperparamters of POR , shown in Table 2.
Our implementations of TD3+BC 2 [13], CQL 3 [27] is from the author-provided implenmentation from Github,and we keep all parameters the same to the author-provided implementation.
[Table] [Table]
D.3 Four-room ExperimentsEnvironment settings
We use the continous variant of the classic four-room environment from[18].
This continous variant of four-rooms is basically the same as traditional classic four-roomenvironment in the environment design.
There are 19 × 19 grids that consists of four rooms withonly one block channel with neigboring rooms.
The goal of this environment is to make the agent travel from one location to another different location.
Those tasks are challenging as their reward isextremely spare, they need the agent to have the ability to explore efficiently through the whole statespace.
The action and observation space are shown in Table 5.
The first and second dimension of theaction space represents the distance to travel on x -axis and y -axis, respectively.
The first and seconddimension of the observation space represents the coordinate on x -axis any y -axis, respectively.
[Table]
In task A ( Four-room ), only reaching the goal will give the agent a reward 1, otherwise the agent willget 0 reward.
When the agent reaches the target or the agent takes over 500 steps, the environmentwill terminate.
In task B ( Four-room-river ), the agent will receive -1 reward if it falls into therivier, and the environment will be terminated.
Also, the agent will get 1 reward only when it reachesthe goal.
In task C ( Four-room-key ), the agent gets 1 reward only when it reaches the goal and getsthe key.
Falling into the rivier will also give the agent -1 reward.
Data collection and implementation details
The training data consists of trajectories collected bya goal-reaching controller with the start and end location sampled randomly at non-wall locations.
To make the data more diverse, we also add trajectories from a random policy.
We collect 100,000transitions for each task.
Figure 6 shows the training hyperparamters of POR in the three four-roomenvironments. [Table]
[Table]
E Learning Curves
In this section
[Table]
we provide the learning curves of our experiments in the main paper.
[Figure]
[Figure]
N o r m a li
z ed S c
o r e
[Figure]
N o r m a li z ed S c o r e [Figure]
Iterations
[Figure] [Figure] [Figure] [Figure]
F Ablation Study on the Execute-Policy
We also present ablation study on the execute-policy, with the aim to answer the following twoquestions: 1) How does network capacity affect the performance of the execute-poliy?
How doesthe performance change if we use partial "good" data, instead of full data, to train the execute policy?
To answer the first question, we compare with two choices of network size: a big network with(1024 , 1024) hidden units and a small network with (256 , 256) hidden units.
To answer the secondquestion, we compare with the choice that only select a subset of the dataset to train the executepolicy.
Concretely, we choose the top X % trajectories in the dataset, ordered by episode returns.
We sweep X over [10 , 25 , 40] and choose the best score.
We give the mean scores of AntMaze ( A )and MuJoCo ( M ) datasets in Table 7.
It can be seen that adopting a big network consistently gives abetter performance, which is also found in [10].
Using partial data will result in a less-performedpolicy, especially on AntMaze datasets. Note that in MuJoCo datasets, the performance didn’t droptoo much. This is because MuJoCo datasets don’t require the compositionality ability, using partialtrajectories could already achieve high scores. [Table]