T HE I N -S AMPLE S OFTMAX FOR O FFLINE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy.
An emerging issue in this ofﬂine RL setting, however, is that the bootstrapping update underlying many of our methods suffers from insufﬁcient action-coverage: standard max operator may select a maximal action that has not been seen in the dataset.
Bootstrapping from these inaccurate values can lead to overestimation and even divergence.
There are a growing number of methods that attempt to approximate an in-sample max, that only uses actions well-covered by the dataset.
We highlight a simple fact: it is more straightforward to approximate an in-sample softmax using only actions in the dataset.
We show that policy iteration based on the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max.
We derive an In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing ofﬂine RL methods, and is also well-suited to ﬁne-tuning.
I NTRODUCTION
A common goal in reinforcement learning (RL) is to learn a control policy from data.
In the ofﬂine setting, the agent has access to a batch of previously collected data.
This data could have been gathered under a near-optimal behavior policy, from a mediocre policy, or a mixture of different policies (perhaps produced by several human operators).
A key challenge is to be robust to this data gathering distribution, since we often do not have control over data collection in some application settings.
Most approaches in ofﬂine RL learn action-values, either through Q-learning updates— bootstrapping off of a maximal action in the next state—or for actor-critic algorithms where the action-values are updated using temporal-difference (TD) learning updates to evaluate the actor.
In either case, poor action coverage can interact poorly with bootstrapping, yielding bad performance.
The action-value updates based on TD involves bootstrapping off an estimate of values in the next state.
This bootstrapping is problematic if the value is an overestimate, which is likely to occur when there are actions that are never sampled in a state (Fujimoto et al., 2018; Kumar et al., 2019; Fujimoto et al., 2019).
When using a maximum over actions, this overestimate will be selected, pushing up the value of the current state and action.
Such updates can lead to poor policies and instability (Fujimoto et al., 2018; Kumar et al., 2019; Fujimoto et al., 2019).
There are two main approaches in ofﬂine RL to handle this over-estimation issue.
One direction constrains the learned policy to be similar to the dataset policy (Wu et al., 2019; Peng et al., 2020;
Nair et al., 2021; Brandfonbrener et al., 2021; Fujimoto & Gu, 2021).
A related idea is to constrain the stationary distribution of the learned policy to be similar to the data distribution (Yang et al., 2022).
The challenge with both these approaches is that they rely on the dataset being generated by an expert or near-optimal policy.
When used on datasets from more suboptimal policies—like those commonly found in industry—they do not perform well (Kostrikov et al., 2022).
The other approach is bootstrap off pessimistic value estimates (Kidambi et al., 2020; Kumar et al., 2020; Kostrikov et al., 2021; Yu et al., 2021) and relatedly to identify and reduce the inﬂuence of out-of-distribution actions using ensembles (Kumar et al., 2019; Agarwal et al., 2020;
Ghasemipour et al., 2021; Wu et al., 2021;
Yang et al., 2021; Bai et al., 2022).
One simply strategy that has been more recently proposed is to constrain the set of actions considered for bootstrapping to the support of the dataset D .
In other words, if π D ( a | s ) is the conditional action distribution underlying the dataset, then we use max a (cid:48) :
π D ( a (cid:48) | s (cid:48) )
> 0 q ( s (cid:48) , a (cid:48) ) instead of max a (cid:48)
q ( s (cid:48) , a (cid:48) ) : a constrained or in-sample max .
This idea was ﬁrst introduced for Batch-Constrained Q-learning (BCQ) (Fujimoto et al., 2019) in the tabular setting, with a generative model used to approximate and sample π D ( a | s ) (Fujimoto et al., 2019;
Zhou et al., 2020; Wu et al., 2022).
Implicit Q-learning (IQL) (Kostrikov et al., 2022) was the ﬁrst model-free approximation to use this in-sample max, with a later modiﬁcation to be less conservative (Ma et al., 2022).
IQL instead uses expectile regression, to push the action-values to predict upper expectiles that are a (close) lower bound to the true maximum.
The approach nicely avoids estimating π D , and empirically performs well.
Using only actions in the dataset is beneﬁcial, because it can approach is be difﬁcult to properly constrain the support of the learned model for π D and ensure it does not output out-of-distributions actions.
There are, however, a few limitations to IQL.
The IQL solution depends on the action distribution not just the support.
In practice, we would expect IQL to perform poorly when the data distribution is skewed towards suboptimal actions in some states, pulling down the expectile regression targets.
We ﬁnd evidence for this in our experiments.
Additionally, convergence is difﬁcult to analyze because expectile regression does not have a closed-form solution.
One recent work showed that the Bellman operator underlying an expectile value learning algorithm is a contraction, but only for the setting with deterministic transitions (Ma et al., 2022).
In this work, we revisit how to directly use the in-sample max.
Our key insight is simple: sampling under support constraints is more straightforward for the softmax, in the entropy-regularized setting.
We ﬁrst deﬁne the in-sample softmax and show that it maintains the same contraction and convergence properties as the standard softmax.
Further, we show that with a decreasing temperature (entropy) parameter, the in-sample softmax approaches the in-sample max.
This formulation, therefore, is both useful for those wishing to incorporate entropy-regularization and to give a reasonable approximation to the in-sample max by selecting a small temperature.
We then show that we can obtain a policy update that relies primarily on sampling from the dataset—which is naturally in-sample—rather than requiring samples from an estimate of π D .
We conclude by showing that our resulting In-sample
Actor-critic algorithm consistently outperforms or matches existing methods, despite being a notably simpler method, in ofﬂine RL experiments with and without ﬁne-tuning.
P ROBLEM S ETTING
In this section we outline the key issue of action-coverage in ofﬂine RL that we address in this work.
2. M ARKOV D ECISION P ROCESS
We consider ﬁnite Markov Decision Process (MDP) determined by M = {S , A , P, r, γ } (Puterman, 2014) , where S is a ﬁnite state space, A is a ﬁnite action space, γ ∈ [0 , 1) is the discount factor, r : S ×
A → R and P : S × A → ∆( S ) are the reward and transition functions.
The value function speciﬁes the future discounted total reward obtained by following a policy π : S → ∆( A ) , v π ( s ) =
E π [ (cid:80) ∞ t =
γ t r ( s t , a t ) | s 0 =
s ] where we use E π to denote the expectation under the distribution induced by the interconnection of π and the environment.
The corresponding action-value function is q π ( s, a )
=
r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) [ v π ( s (cid:48) )] .
There exists an optimal policy π ∗ that maximizes the values for all states s ∈ S .
We use v ∗ and q ∗ to denote the optimal value functions.
The optimal value satisﬁes the Bellman optimality equation ,
[Equation]
In this work we more speciﬁcally consider the entropy-regularized MDP setting—also called the maximum entropy setting—where an entropy term is added to the reward to encourage the policy to be stochastic.
The maximum-entropy value function is deﬁned as
[Equation]
for temperature τ and H the discounted entropy regularization .
The corresponding maximum-entropy action-value function is ˜ q π ( s, a ) =
r
[Equation]
s, a ) + γ E s (cid:48) ∼ P ( s,a )
[˜ v π ( s
[Equation]
)] , with soft Bellman optimality equations similarly modiﬁed as described in the next section.
As τ → 0 , we recover the original value function deﬁnitions.
The entropy-regularized setting has become widely used (Ziebart et al., 2008; Mnih et al., 2016; Nachum et al., 2017; Asadi & Littman, 2017; Haarnoja et al., 2018; Mei et al., 2019; Xiao et al., 2019), because it 1) encourages exploration (Ziebart et al., 2008), 2) often makes objectives more smooth (Mei et al., 2019), and 3) provides these improvements even with small temperatures that do not signiﬁcantly bias the solution to the original MDP (Song et al., 2019).
2. O FFLINE R EINFORCEMENT L EARNING
In this work, we consider the problem of learning an optimal decision making policy from a previously collected ofﬂine dataset D = { s i , a i , r i , s (cid:48)
i } n − 1 i =0 .
We assume that the data is generated by executing a behavior policy π D .
Note that we do assume direct access to π D .
In ofﬂine RL, the learning algorithm can only learn from samples in this D without further interaction with the environment.
One primary issue in ofﬂine RL is that π D may not have full coverage over actions.
Greedy decisions based on a learned value q ≈ q ∗ could be problematic, especially when the value is an overestimate for out-of-distribution actions (Fujimoto et al., 2019).
To overcome this issue, one popular approach is to constrain the learned policy to be similar to π D , such as by adding a KL-divergence term: max π E s ∼ ρ [ (cid:80) a π ( a | s ) q ( s, a ) − τD KL ( π ( ·| s )
||
π D ( ·| s ))] for some τ > 0 .
The optimal policy for this objective must be on the support of π D : the KL constraint makes sure
π ( a | s ) =
as long as π D ( a | s ) = 0 .
This optimal policy, with closed-form solution π (cid:48) ( a | s ) ∝ π D ( a | s ) exp( q ( s, a ) /τ ) , is also guaranteed to be an improvement on π D .
Many ofﬂine RL algorithms are based on this nice idea (Wu et al., 2019; Peng et al., 2020; Nair et al., 2021; Brandfonbrener et al., 2021; Fujimoto & Gu, 2021).
This KL constraint, however, can result in poor π (cid:48) when π D is sub-optimal, conﬁrmed both in previous studies (Kostrikov et al., 2022) and our experimental results.
The other strategy is to consider an in-sample policy optimization, max π (cid:22) π D (cid:80) a ∈A π ( a | s ) q
( s, a ) , where π (cid:22)
π D indicates the support of π is a subset of π D .
This approach more directly avoids selecting out-of-distribution actions.
Though a simple idea, approximating this with a simple algorithm has been elusive, as discussed above.
The simplest idea is to estimate π ω ≈ π D and directly constrain the support by sampling candidate actions from π ω , as proposed for Batch-Constrained
Q-learning (Fujimoto et al., 2019).
This simple approach, however, may not avoid bootstrapping from out-of-sample actions due to the error in the estimate π ω .
Surprisingly, the small modiﬁcation to the in-sample softmax (Section 3) has not yet been considered for ofﬂine RL.
Yet, moving from the in-sample (hard) max to the in-sample softmax facilitates developing a simple algorithm, as we discuss in the remainder of this work.
T HE I N -S AMPLE S OFTMAX O PTIMALITY
This section introduces the in-sample softmax optimality that provides a simple implementation of in-sample bootstrapping.
We ﬁrst describe the standard soft Bellman optimality equations, then the modiﬁcation to consider in-sample bootstrapping.
Our simple algorithm comes from stepping back and recognizing the utility of considering in-sample bootstrapping for the entropy-regularized setting rather than only for the hard-max.
The soft Bellman optimality equations for maximum-entropy RL use the softmax in place of the max,
[Equation]
This comes from the fact that hard max with entropy regularization is max p ∈ ∆( A ) (cid:80) a ∈A p ( a ) q ( s, a )+ τ H ( p ) =
τ log (cid:80) a ∈A e q ( s,a ) /τ .
As τ → 0 , softmax (log-sum-exp) approaches the max.
that this softmax operator for the soft Bellman optimality equation is different from the softmax Bellman operator, which uses an expectation in the bootstrap over a softmax policy and which is know to have issue with not being an contraction (Asadi & Littman, 2017).
The log-sum-exp formula is a standard way to approximate the max, and is naturally called a softmax.
We can modify this update to restrict the softmax to the support of π D :
[Equation]
We call Eq.
(4) the in-sample softmax optimality equation .
It is interesting to note that we can use a simple reformulation that facilitates sampling the inner term.
For any q ,
[Equation]
This reformulation does not perfectly remove the role of π D ( a | s ) , but it is signiﬁcantly reduced.
The support is no longer constrained using π D and instead the values are simply shifted by this term involving π D .
We will use this strategy below to develop our algorithm.
There are a few interesting facts to note about in-sample softmax.
First, we can show that similarly to the standard maximum-entropy bootstrap (shown formally in Lemma 3), we have for any q
[Equation]
Though this outcome is intuitive, it is a nice property that restricting the support of the log-sum-exp maintains the same relationship to the maximum-entropy update with the same support constraint.
It extends this result for the soft Bellman optimality update to the setting with a support constraint.
From this perspective, in-sample softmax can also be viewed as a tool for conservative exploration : exploring to prevent getting stuck in a local optima, while still being suspicious of what the data does not know.
This is especially important when q is a learned value approximation.
Second, we can also obtain a closed-form greedy policy using the above (shown formally in Lemma 3), which we call the in-sample softmax greedy policy : for any q ,
[Equation]
This closed-form solution looks similar to the KL-regularized solution mentioned in Section 2.2, where π (cid:48) is constrained to be similar to π D .
The only difference is the additional − log π D term inside the exponential.
This small difference, however, has a big impact.
It allows the resulting policy to deviate much more from π D .
In fact, because exp( − log π D ( a | s )) =
π D ( a | s ) − 1 , the above is equivalent to π π D ,q ( a | s )
=
when π D ( a | s ) = 0 and otherwise π π D ,q ( a | s ) ∝ exp( q ( s, a ) /τ ) 4 .
The new policy ˜ π π D is not skewed by the action probabilities in π π D ,q ; it just has the same support.
T HEORETICAL C HARACTERIZATION OF I N -S AMPLE S OFTMAX
In this section we prove in-sample softmax maintains the convergence properties of the standard softmax.
In particular, Bellman updates with the in-sample softmax are convergent, andthat Bellman updates using the in-sample softmax are convergent, and the resulting in-sample softmax optimal policy approaches the in-sample optimal policy as we reduce the temperature to zero.
All proofs are given in Appendix A.
We can contrast our in-sample softmax optimality equation in (4) to the in-sample Bellman optimality equation introduced by Fujimoto et al.
(2019) for the hard max,
[Equation]
We ﬁrst show that the policy produced using the in-sample softmax optimality equation is a good approximation to that given by the in-sample Bellman optimality equation.
Theorem 1.
Let ˜ q ∗ π D be the in-sample softmax optimal value function.
We have lim τ → 0 ˜ q ∗ π D = q ∗ π D .
Moreover, let I be an indicator function and π π D ( a | s )
=
I ( a = arg max a : π D ( a ) > 0 q ∗ π D ( s, a )) be the in-sample optimal policy w.r.t q ∗ π D .
Deﬁne the in-sample softmax optimal policy,
[Equation]
We have lim
[Equation]
=
[Equation]
Now we show that we can reach the in-sample softmax optimal solution, using either value iteration or policy iteration.
For value iteration, we deﬁne the in-sample softmax optimality operator
[Equation]
The next result shows that T π D is a contraction, and therefore in-sample soft value iteration , using q t
+1 =
T π D q t , is guaranteed to converge to the in-sample softmax optimal value in the tabular case.
Theorem 2.
For γ < 1 , the ﬁxed point of the in-sample softmax optimality operator exists and is unique.
Thus, in-sample soft value iteration converges to the in-sample softmax optimal value ˜ q ∗ π D .
As highlighted by Equation (6), the in-sample softmax policy corresponds to the solution of the maximum entropy policy optimization.
This implies that similarly to Soft Actor-Critic (Haarnoja et al., 2018), we can apply policy iteration to ﬁnd this policy.
Let π t be the policy at iteration t .
The algorithm ﬁrst learns the value function ˜ q π t , then updates the policy π t +1 such that ˜ q π t ≤ ˜ q π t +1 .
The following result shows that this procedure guarantees policy improvement.
Lemma 1.
Let π t be a policy such that π t (cid:22) π D .
Deﬁne
[Equation]
Then π t +1 (cid:22) π D and ˜ q π t +1 ≥ ˜ q π t .
Note that π t +1 not only ensures policy improvement, but also stays in the support of π D .
Now let us deﬁne the on-policy entropy-regularized operator ,
[Equation]
Since this operator is a contraction (shown formally in Lemma 5), we can evaluate ˜ q π by repeatedly applying T π q from any q until converge.
These updates give rise to the in-sample soft policy iteration algorithm that iteratively updates the policy using (11) and evaluate its by using T π .
The convergence for the tabular case is given below.
Theorem 3.
For γ < 1 , starting from any initial policy π such that π (cid:22) π D , in-sample soft policy iteration converges to the in-sample softmax optimal policy ˜ π ∗ π D .
P OLICY O PTIMIZATION USING THE I N - SAMPLE S OFTMAX
In this section, we develop an In-sample Actor-critic (AC) algorithm based on the in-sample softmax.
This is the ﬁrst time we see the utility of the in-sample softmax, to facilitate sampling actions from π D using only actions in the dataset.
This contrasts other direct methods, like BCQ that approximate the in-sample max by sampling from an approximate π ω (Fujimoto et al., 2019).
Throughout this section we generically develop the algorithm for continuous and discrete actions.
Instead of using sums, therefore, we primarily write formulas using expectations, which allow for either discrete or continuous actions.
The In-sample AC algorithm is similar to SAC (Haarnoja et al., 2018), except that we carefully consider out-of-sample actions.
We similarly learn an actor π ψ with parameters ψ , action-values q θ with parameters θ and a value function v φ with parameters φ .
Additionally, we learn π ω ≈ π D .
We need this to deﬁne the greedy policy shown above in Equation (11), but do not directly use it to constrain the support over actions.
The ﬁrst step in the algorithm is to extract π ω ≈ π D .
We do so using a simple maximum likelihood loss on the dataset: L behavior ( ω ) = − E ( s,a ) ∼D [log π ω ( a | s )] .
We do not add any additional tricks to try to ensure action probabilities are zero where π D ( a | s ) = 0 , because this π ω only plays a smaller role in our update.
It will only be used to adjust the greedy policy, and will only be queried on actions in the dataset.
Then we use a similar approach to SAC, where we alternate between estimating q θ and v φ for the current policy and improving the policy by minimizing a KL-divergence to the soft greedy policy.
The main difference here to SAC is that we update towards the in-sample soft greedy.
We cannot directly use Equation (11), which involves π D in the update, but can replace π D in the update with our approximation π ω .
We therefore update towards an approximate in-sample soft greedy policy
[Equation]
where Z ( s ) = τ log (cid:82) a π D ( a | s ) exp( q θ ( s,a ) τ − log π ω ( a | s ))
da is the normalizer to give a valid distribution.
We minimize a forward KL to this in-sample soft greedy policy, because that allows us to sample the KL by only sampling actions from the dataset.
To see why, notice that
[Equation]
The expectation is now over samples a ∼ π D ( ·| s ) ; the actions in the dataset are precisely sampled from π D .
To sample the gradient for this loss, we also need an estimate for Z ( s ) .
We use our parameterized v φ to estimate Z ; we discuss why this is reasonable below.
The ﬁnal loss function for the actor π ψ is
[Equation]
For the value function we use standard value function updates for the entropy-regularized setting.
The objectives are
[Equation]
The action-values use the estimate of v φ in the next state, and so avoids using out-of-distribution actions.
The update to the value function, v φ , uses only actions sampled from π ψ , which is being optimized to stay in-sample.
Periodically, however, v φ may bootstrap off of out-of-distribution actions because we do not guarantee that π ψ (cid:22) π D .
In fact, in early learning we expect π ψ will not satisfy this property.
Despite this, the actor update will progressively reduce the probability of these out-of-distribution actions, even if temporarily the action-values overestimate their value, because the actor update pushes π ψ towards the in-sample greedy policy.
This means that the overestimate is unlikely to signiﬁcantly skew the actor, and progressively the overestimate should be reduced as the support of π ψ is reduced.
Finally, instead of learning a separate approximation for Z , we opt for the simpler approach of using v φ .
The reason is that v φ should provide a reasonable approximation to Z because of the relationship between soft values and Z .
From Equations (7) and (8) (formally proved in Lemma 3), we know that the soft values for the in-sample soft greedy policy ˜ π π D ,q θ correspond to the normalizer Z for that policy.
Therefore, given that π ω ≈ π D , the soft values of approximate in-sample soft greedy policy ˆ π π D ,q θ should also be similar to Z .
Since we optimize our policy to approximate ˆ π π D ,q θ , we expect its entropy-regularized value, which is the learning target of v φ as shown in Equation (15), to be a good approximation of Z .
E XPERIMENTS
In this section, we investigate three primary questions.
First, in the tabular setting, can our algorithm
InAC converge to a policy found by an oracle method that exactly eliminates out-of-distribution
(OOD) actions when bootstrapping?
Second, in Mujoco benchmarks, how does our algorithm compare with several baselines using different ofﬂine datasets with different coverage?
Third, how does InAC compare with other baselines when used for online ﬁne-tuning after ofﬂine training?
We refer readers to Appendix B for additional details and supplementary experiments.
Baseline algorithms: Oracle-Max: completely eliminates OOD actions when bootstrapping in tabular domains, by using counts to exactly estimate π D .
FQI:
the regular Q-learning update applied to batch ofﬂine data.
CQL (Kumar et al., 2020): conservative Q-learning.
IQL (Kostrikov et al., 2022): implicit Q-learning.
TD3+BC (Fujimoto & Gu, 2021): TD3 with behavior cloning regularization.
AWAC (Nair et al., 2021):
Advantage Weighted Actor-Critic.
6. S ANITY C HECK : APPROACHING ORACLE PERFORMANCE IN THE T ABULAR S ETTING
In this experiment we demonstrate that InAC ﬁnds the same policy as found by an oracle algorithm that completely removes out-of-distribution (OOD) actions.
We use the Four Rooms environment, where the agent starts from the bottom-left and needs to navigate through the four rooms to reach the goal in the up-right corner in as few steps as possible.
There are four actions: A = { up, down, right, left } .
The reward is zero on each time step until the agent reaches the goal-state where it receives +1.
Episodes are terminated after 100 steps, and γ is 0.9.
We use three different behavior policies to collect three datasets from this environment called Expert, Random, and Missing-Action .
The
Expert dataset contains data collected by the optimal policy.
In Random dataset, the behavior policy takes each action with equal probability.
For the Missing-Action dataset, we removed all transitions taking down actions in the upper-left room from the Mixed dataset.
To magnify the impact of bootstrapping from OOD actions we used optimistic initialization for each algorithm (i.e., initialized all action values to be larger than the actual values under the optimal policy).
This ensures overestimation occurs in some states and we can observe how well the algorithms mitigate poor bootstrap targets.
[Figure]
IQL
InAC
[Figure]
The results in Figure 1 are exactly as expected.
InAC converges to the same policy as found by
Oracle-Max.
The FQI baseline cannot effectively remove OOD actions when bootstrapping, and so performs poorly and sometimes completely fails when the dataset has poor action coverage (i.e., there are many OOD actions).
Finally, IQL performs poorly when the ofﬂine data is highly skewed towards suboptimal policies.
It is likely because the upper expectile of the state value provides a poor approximation to the in-sample maximum action value.
6. OOD EFFECTS IN CONTINOUS ACTION PROBLEMS
In this section we provide a suite of results from four Mujoco environments from D4RL (Fu et al., 2020), now standard datasets for evaluating ofﬂine RL algorithms.
Each dataset (named as Expert, Medium-Expert, Medium-Replay, and Medium) was designed to mimic different deployment scenarios.
In the Expert dataset all trajectories were collected using a policy learned by a SAC agent.
In Medium, all trajectories were collected with the policy learned by a SAC agent halfway thought training.
Medium-Expert combines the expert and medium datasets together, and similarly
Medium-Replay combines Medium with the replay buffer used during learning.
[Figure] [Figure]
Figure 2 summarizes each algorithm’s performance averaged over all environments under different datasets.
Our algorithm’s performance dominates the others across datasets.
In Figurewe provide a more detailed view of the data with learning curves in each environment.
Overall InAC performs best or nearly so across all domains.
In Hopper M-Expert, the result is likely a three-way tie, while in HalfCheetah M-Expert TD3+BC learns faster initially, but the quickly converges to much lower ﬁnal performance compared with InAC.
Naturally, all methods are dependent on the quality of the dataset.
For example, when shifting from the higher quality (medium-expert) to the lower quality (medium-replay) data, TD3+BC—which regularizes the policy to stay close to the behavior policy—exhibits a signiﬁcant performance drop.
Overall, TD3+BC and CQL’s performance is problem dependent: in some problems performing well and in others basically failing to learn.
Finally IQL performs nearly as well as InAC on many problems, but notably not on Walker2D and Hopper M-Replay.
These results provide evidence that explicitly avoiding bootstrapping from OOD actions provides a signiﬁcant beneﬁt, but that regularizing the learned policy to stay close to the behavior policy can be problematic.
6. F ROM O FFLINE TRAINING TO O NLINE FINE - TUNING
In real-world applications, it can be useful to take an ofﬂine-trained deep RL agent and ﬁne-tune it online.
In this section, we investigate how the performance of different baselines changes in ﬁne-tuning.
At the beginning of ﬁne-tuning, the agent’s policy is initialized with the policy learned ofﬂine and the buffer is ﬁlled with that same ofﬂine dataset.
During online interactions, the agent continually adds its new experience into the buffer.
Figure 4 shows the policy performance before and after online ﬁne-tuning.
We see that InAC is consistently one of the best algorithms across these environments and datasets.
There are a few particularly notable outcomes in these experiments.
In Hopper and Walker2d for the Medium-Expert data, the performance for IQL drops signiﬁcantly after ﬁne-tuning.
This contrasts all the other algorithms, which maintained or improved performance when ﬁne-tuning.
The cause of this drop is as yet unclear.
There is one new algorithm in this set, called AWAC, which was originally proposed speciﬁcally for online ﬁne-tuning setting (Nair et al., 2021).
We do in-fact see that this algorithm [Figure] can have quite poor ofﬂine performance, but signiﬁcantly improve after ﬁne-tuning.
Despite being designed for this ﬁne-tuning setting, however, it does not outperform the ofﬂine algorithms, except in
Hopper with Medium-Replay and more minorly on Walker2d with Medium-Replay.
Overall, we ﬁnd that InAC performs well in both the fully ofﬂine setting as well as when incorporating ﬁne-tuning.
C ONCLUSION
In this paper we considered the problem of learning action-values and corresponding policies from a ﬁxed batch of data.
The algorithms designed for this setting need to account for the fact that action-coverage may be partial: certain actions may never be taken in certain regions of the state space.
This complicates learning with our algorithms that rely on action-values estimates q ( s, a ) and bootstrapping.
In particular, if an action a is not visited in s or similar states, the q ( s, a ) can be an arbitrary value.
If this arbitrary value is high, it is likely to be used in the max in the bootstrap target and used to update the policy, which increases probability for high-value actions.
This agent is chasing hallucinations, that can produce poor policies or even divergence.
We focused on a simple approach to mitigate this issue: redeﬁning the objectives to use an in-sample softmax and ﬁnally obtaining an approach to update towards an in-sample soft greedy policy that only uses actions sampled from the dataset.
The resulting In-sample AC algorithm avoids these hallucinated values when updating the actor, and so correspondingly avoids them when updating the values.
We had two clear ﬁndings from this work.
First, the move to an in-sample softmax was a key step towards a simple implementation of in-sample learning.
Previous work, like BCQ, tried to produce a simple algorithm built on an in-sample max algorithm, but needed to incorporate several tricks and later algorithms signiﬁcantly improve on it.
In-sample AC, on the other hand, required only minor modiﬁcations to existing AC approaches.
The actor update was modiﬁed to consider the in-sample softmax, but the resulting update was no more complex than typical actor updates.
Second, our results indicate that overall Implicit Q-learning (IQL) is quite a good algorithm.
Like In-sample AC, it also avoids relying on actions sampled from an approximation of π D , but does so using expectile regression.
Nonetheless, we ﬁnd that In-sample AC is always competitive with IQL, and in some cases signiﬁcantly outperforms it when the dataset is generated by a more suboptimal behavior policy.
IQL can still be skewed by too many suboptimal actions in the dataset.
In-sample AC provides a simple, easy-to-use approach, for both discrete and continuous actions, with an update designed to match only the support of π D and not the action probabilities.
A A PPENDIX : P ROOFS
This section includes the proof of all main results.
A. R ESULTS FOR ONE - STEP DECISION MAKING
We ﬁrst introduce some results for one-step decision making that will be used in the derivations of main results.
Maximum Entropy Optimization We consider a k -armed one-step decision making problem.
Let ∆ be a k -dimensional simplex and q = ( q (1) , . .
, q ( k ))
∈
R k be the reward vector.
Maximum entropy optimization considers
[Equation]
The next result characterizes the solution of this problem (Lemma 4 of (Nachum et al., 2017)).
Lemma 2.
For τ > 0 , let
[Equation]
Then there is
[Equation] [Equation]
In-Sample Maximum Entropy Optimization Let β ∈ ∆ be an arbitrary policy.
In-sample maximum entropy optimization considers
[Equation]
We now characterize the solution of this problem.
For τ > 0 deﬁne the in-sample softmax value ,
[Equation] [Equation]
and the in-sample softmax policy ,
[Equation]
Lemma 3.
[Equation]
Proof.
This result is directly implied by Lemma 2.
The next result shows that F β,τ is a contractor.
Lemma 4.
For any two vectors q 1 , q 2 ∈ R k ,
[Equation] [Equation]
Proof.
[Equation]
where the ﬁrst step follows by Lemma 3, the third step follows by choosing π 2 = π 1 .
This ﬁnishes the proof.
A. R ESULT FOR ON - POLICY ENTROPY - REGULARIZED BACKUP
In this section we show some basic results for on-policy entropy-regularized backup.
We note that most results are generalized from Section C.2 of (Nachum et al., 2017) (which states for ˜ v ) to ˜ q .
Recall that the entropy-regularized value functions are deﬁned as
[Equation] [Equation]
Deﬁne the on-policy entropy-regularized Bellman operator
[Equation]
Lemma 5.
For any policy π
[Equation]
˜ q π satisﬁes that ˜ q π = T π ˜ q π .
Moreover, suppose that |A| < ∞ , T π is a contraction mapping.
Proof.
By the deﬁnition of ˜ v π and ˜ q π ,
[Equation] [Equation]
Thus
[Equation]
This ﬁnishes the proof of the ﬁrst part.
Since |A| < ∞ , log π ( a | s ) is bounded for any s, a .
Then that
T π is a contraction mapping follows directly from standard argument (Puterman, 2014).
This shows that ˜ q π is a ﬁxed of T π .
That is, starting from any value q , we can learn ˜ q π by repeatedly applying q = T π q .
The next result characterizes the convergence rate of this algorithm.
Lemma 6.
For any π and q , we have
[Equation]
Proof.
We prove the result by induction.
For the base case, k = 0 , the result trivially follows.
Now suppose that the result holds for k − 1 .
Then
[Equation]
where the second step uses Lemma 5, the third step uses the deﬁnition of T π , the fourth step uses the
Holder’s inequality, the last step uses the induction hypothesis.
This ﬁnishes the proof.
Finally, we also need the monotonicity property of the on-policy Bellman operation.
Lemma 7.
For any π , if q 1 ≥ q 2 , then T π q 1 ≥ T π q 2 .
Proof.
Assume q 1 ≥ q 2 and note that for any state-action s, a
[Equation]
Policy improvement lemma.
Lemma 8.
Let π be a policy such that π (cid:22) β .
Deﬁne π (cid:48)
[Equation]
Then π (cid:48) (cid:22) β and ˜ q π (cid:48) ≥ ˜ q π .
Proof.
The ﬁrst part trivially holds by the deﬁnition of π (cid:48) .
For the second part, note that by Lemma 3, for any state s ∈ S ,
[Equation]
Then by Lemma 5, for any s, a ∈ S × A ,
[Equation] [Equation]
where we recursively apply Lemma 5 to expand the deﬁnition of ˜ q π and apply Eq.
A. R ESULT FOR OFF - POLICY ENTROPY - REGULARIZED BACKUP
Given an arbitrary policy β
[Equation]
consider the following problem
[Equation]
For τ > 0 , deﬁne the In-sample softmax Bellman operator
[Equation]
Lemma 9.
For γ < 1 , the ﬁxed point of the in-sample softmax Bellman operator, q ∗ = T ∗ β q ∗ , exists and is unique.
Proof.
We ﬁrst show that T ∗ β is a contraction
[Equation]
Let q 1 and q 2 be two value functions.
Then
[Equation]
where the second step uses the deﬁnition of T ∗ β , the third step uses Holder’s inequality, the fourth step uses Lemma 4.
Note that by deﬁnition, ˜ q ∗ β is the ﬁxed point of T ∗ β .
Lemma 10.
If q is bounded and q ≥ T ∗ β q , then for any π , q ≥ ˜ q π .
Proof.
We ﬁrst prove that for any π , q ≥ T ∗ β q implies that q ≥ ( T π ) k q for k ≥ 0 .
Then the result follows by applying Lemma 6.
According to the assumption,
[Equation]
where the second inequality follows by Lemma 3.
Then by Lemma 7,
[Equation]
This ﬁnishes the proof.
We have the following key result.
Lemma 11.
For any s ∈ S , ˜ v ∗ β ( s ) = max π (cid:22) β ˜ v ∗ ( s ) .
Proof.
We ﬁrst show ˜ v ∗ β ≥ max π (cid:22) β ˜ v π .
Using the deﬁnitions,
[Equation]
[Equation]
where the second and third steps follow by Lemma 3, the fourth step follows by Lemma 10, the last step follows by the deﬁnition.
We then prove max π (cid:22) β ˜ v π ≥ ˜ v ∗ β by ﬁrst showing that ˜
q ∗ β = ˜
q ˜ π ∗ β .
Since ˜ q ∗ β is the ﬁxed point T ∗ β , by the uniqueness of the ﬁxed point (Lemma 9), we only need to show that T ∗ β ˜ q ˜
π ∗ β = ˜
q ˜ π ∗ β .
This holds because for any ( s, a ) ,
[Equation]
where the second step uses Lemma 3 and the last step uses Lemma 5.
Then,
[Equation] [Equation]
where the second equality uses that ˜ q ∗ β = ˜ q ˜ π ∗ β , the last step uses Lemma 3.
This ﬁnishes the proof.
A. P ROOF OF T HEOREM
Theorem 4 (Restatement of Theorem 1) .
Let ˜ q ∗ β be a value function recursively deﬁned as
[Equation]
and ˜ π ∗ β be a policy deﬁned as
[Equation]
Then we have ˜ q ∗ β
→ q ∗ β and ˜ π ∗ β → π ∗ β as τ → 0 .
Proof.
By Lemma 11, we have ˜ q ∗ β ( s, a ) = max π (cid:22) β ˜ q ∗ ( s, a ) for any s, a .
The result directly follows by deﬁnition of ˜ q ∗ .
The result for policy can be proved similarly.
B A PPENDIX FOR EXPERIMENTS B. A DDITIONAL E XPERIMENTS
This section includes additional experiments to investigate the following questions.
We used optimistic initialization for all algorithms in the tabular domain.
How do the algorithms perform when using a zero/pessimistic initialization?
We show this in Figure 5.
In the meanwhile, we added Mixed dataset, which has 1% optimal trajectories and 99% random trajectories.
How do our algorithms work on the discrete action domains in the deep learning setting?
We show the learning curves on Mountain Car, Lunar Lander, Acrobot in Figure 7.
The ﬁnal performance is listed in Figure 8 with a normalized score, while the absolute score can be found in Figure 9. 3.
How do our algorithms work on more datasets in the continuous action domains?
We put learning curves for the expert and medium dataset in Figure 10, then list the performance of policies learned with all baselines and all datasets in Figure 11 with a normalized score,
while the absolute score can be found in Figure 12. For more Fine-tuning results, we put them in Figure 13 with a normalized score, while the absolute score can be found in Figure 14. 4. Will longer run reduce the gap between InAC and baselines? Can InAC still learn better or similar policy compared to baselines, if we use another common batch size setting in Mujoco tasks (256)?
We changed the batch size to 256 and increased the number of iteration to 1.2 million, then show the performance in Figure 15. How does InAC perform in AntMaze? We used antmaze-umaze-v0 and antmaze-umazediverse-v0 to test InAC, then added the learning curve comparing InAC to IQL in Figure 16. We followed the set up in previous work (Kostrikov et al., 2022).
B. R EPRODUCING D ETAILS
This section includes all experimental details to reproduce any empirical results in this paper.
We use python version 3.9.6, gym version 0.10.0, pytorch version 1.10.0.
B. R EPRODUCING D ETAILS ON T ABULAR D OMAIN
Four room environment: The environment is a 13 × 13 gridword, with walls separating the whole space into 4 rooms (as shown in Figure 6).
The black area refers to the wall.
The agent starts from the lower-left corner and learns to stay in the upper-right corner.
When an agent runs into the wall, it returns to its previous state.
The agent gets a +1 when it transits to the state in the upper-right corner and gets 0 otherwise.
The discount rate is 0 . 9 .
Thus the upper bound of state value is 10 .
In tabular experiments, each trajectory was limited to 100 steps.
τ was set to 0 .
A mini-batch update was used.
The agent sampled 100 transitions at each iteration.
Among the 4 datasets we used, mixed and random datasets used the random restart to ensure full state-action pairs coverage, while expert and missing-action datasets did not.
Ofﬂine data collection: We used value iterations to ﬁnd the optimal policy ( 10 k iterations).
For the expert dataset, we collect 10 k transitions with the optimal policy.
For the random dataset, we collected 10 k transitions with a random restart and equal probability of taking each action.
The mixed dataset consists of 100 transitions from the expert dataset and 9900 transitions from the random dataset.
The missing action dataset is constructed by all going-down transitions in the upper-left room from the mixed dataset.
Algorithm parameter sweep:
The learning rate of
InAC,
Oracle-Max, and
FQI was swept in [0 . 1 , 0 .
Sarsa had a larger range: [0 . 1 , 0 . 03 , 0 . 01 , 0 . 003 , 0 .
The τ of in-sample methods was ﬁxed to 0.01.
B. R EPRODUCING D ETAILS OF D EEP RL A LGORITHMS
Network architecture:
In mujoco tasks, we used 2 hidden layers with 256 nodes each for all neural networks.
In discrete action environments, we used 2 hidden layers with 64 nodes each.
Ofﬂine data generation details:
In continuous control tasks, we used the datasets provided by
D4RL.
In discrete control tasks, we used a well-trained DQN agent to collect data.
The DQN agent had 2 hidden layers with 64 nodes on each, with FTA (Pan et al., 2021) activation function on the last hidden layer and ReLU on others.
In Acrobot, the agent was trained for 40k steps with batch size 64.
In Lunar Lander and Mountain Car, we trained the agent for 500k and 60k steps separately, with other settings the same as in Acrobot.
The expert dataset contains 50k transitions collected with the ﬁxed policy learned by the DQN agent.
The mixed dataset has 2k (4%) near-optimal transitions and 48k (96%) transitions collected with a randomly initialized policy.
Ofﬂine training details:
In all tasks, we used minibatch sampling, and the mini-batch size was set to 100 .
We used the ADAM optimizer and ReLU activation function.
The target network is updated by using Polyak average: 0 .
995 × target weight
+ 0 .
005 × learning weight .
We trained the agent for 0 .
8 million iterations and 70 k iterations in mujoco and discrete action environments respectively.
[Figure] Fine Tuning details: We kept all settings as same as in ofﬂine learning, and used the learned policy as initialization.
The ofﬂine data was ﬁlled into the buffer at the beginning of ﬁne-tuning.
New interactions would be appended to the buffer later in ﬁne-tuning.
No data were removed.
The ﬁne-tuning had 0 .
8 M steps.
Policy evaluation details: The policy was evaluated for 5 episodes in the true environment with a timeout setting.
Acrobot and Lunar Lander had timeout= 500 , Mountain Car used 2000 , and mujoco tasks used 1000 .
The numbers reported were averaged over 10 random seeds.
Algorithm parameter setting:
[Figure]
[Figure]
Mujoco tasks:
For all algorithms, the learning rate was swept in { 3 × 10 − 4 , 1 × 10 − 4 , 3 × 10 − 5 } .
InAC swept τ in { 1 . 0 , 0 . 5 , 0 .
AWAC swept λ in { 1 . 0 , 0 . 5 , 0 .
IQL swept expectile in { 0 . 9 , 0 . 7 } and temperature in { 10 . 0 , 3 . 0 } .
The number came from what was reported in the original IQL paper.
TD3+BC used α = 2 .
5 as in the original paper.
CQL-SAC used automatic entropy tuning as in the original paper.
Discrete action environments:
For all algorithms, the learning rate was swept in { 0 . 003 , 0 . 001 , 0 . 0003 , 0 . 0001 , 3 e − 5 , 1 e − 5 } .
For InAC, τ was swept in { 1 . 0 , 0 . 5 , 0 . 1 , 0 .
IQL had the same parameter sweeping range as in mujoco tasks.
AWAC used λ = 1 . 0 as in the original paper.
CQL used α = 5 . 0 .
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Figure] [Figure] [Table]
[Figure]
[Figure]
[Table]
[Table]
[Figure]
[Table]
3790.83 (7.16) 3269.64 (24.45) 600.84 (39.67) 2474.75 (29.75) FineTune 3650.39 (12.29) 453.01 (45.64) 3045.55 (12.63) Change 322.
[Figure]
[Figure]
of