You Only Evaluate Once: a Simple Baseline Algorithm for Ofﬂine RL
Anonymous Author(s)
AfﬁliationAddress email
Abstract: The goal of ofﬂine reinforcement learning (RL) is to ﬁnd an optimalpolicy given prerecorded trajectories.
Many current approaches customize existingoff-policy RL algorithms, especially actor-critic algorithms in which policy evaluation and improvement are iterated.
However, the convergence of such approachesis not guaranteed due to the use of complex non-linear function approximation andan intertwined optimization process.
By contrast, we propose a simple baselinealgorithm for ofﬂine RL that only performs the policy evaluation step once so thatthe algorithm does not require complex stabilization schemes.
Since the proposedalgorithm is not likely to converge to an optimal policy, it is an appropriate baselinefor actor-critic algorithms that ought to be outperformed if there is indeed value initerative optimization in the ofﬂine setting.
Surprisingly, we empirically ﬁnd thatthe proposed algorithm exhibits competitive and sometimes even state-of-the-artperformance in a subset of the D4RL ofﬂine RL benchmark.
This result suggeststhat future work is needed to fully exploit the potential advantages of iterativeoptimization in order to justify the reduced stability of such methods.
Introduction
The standard reinforcement learning setting involves an active component during learning: an agentcontinuously gathers experience as it learns.
This is a very general learning framework that resemblesthe way animals learn but the interactive component often hurts the applicability of RL since theagent interaction can be expensive or unsafe.
To address this challenge, the ofﬂine reinforcementlearning paradigm has been proposed, which aims to learn a policy purely from pre-generated data[1].
Considering that many recent breakthroughs in machine learning can be attributed to large-scaledata, this new paradigm is very promising.
However, the ofﬂine setup causes signiﬁcant theoretic andalgorithmic difﬁculties that need to be resolved to fulﬁll this promise.
Speciﬁcally, actor-critic based off-policy RL algorithms, which iterate policy evaluation and improvement, suffer from the overestimation problem caused by function approximation error andbootstrapping [2, 3] in the ofﬂine RL setup, even though the algorithms are equipped with algorithmictechniques [4, 2, 5] that can stabilize learning and mitigate the so-called Deadly Triad [6].
This isbecause over-estimated values cannot be readjusted in ofﬂine RL, unlike the ordinary RL setup whereincorrectly optimistic actions get executed and corrected.
The Deadly Triad states that when off-policy learning, function approximation, and bootstrappingare used together, the danger of instability and divergence arises [6]. Therefore, the actor-criticalgorithms that leverage all three components are vulnerable.
Many algorithms try to stabilizelearning by ensuring the queries for bootstrapping to be within the data manifold of the given ofﬂinedataset [7, 8, 9, 10, 11, 12] since over-estimation is prominent when the value function is queried forout-of-distribution inputs.
However, strictly speaking, the Deadly Triad occurs due to iteration of theactor-critic algorithm; the ﬁrst critic update is simply policy evaluation of the action-value functionof the behavior policy β using on-policy data.
While the policy implied by the ﬁrst value function Q β is likely to be suboptimal that would be surpassed by iterative algorithms, it has not been fullyexamined that a greedy policy with regard to Q β cannot work well in the ofﬂine RL setting.
To this end, we propose a baseline algorithm, named YOEO, that evaluates the value of a behaviorpolicy once and extracts a greedy policy under the learned value approximation, and by doing so,we examine the hidden presumption of the actor-critic based ofﬂine RL algorithms that the iterativeprocess is essential and beneﬁcial.
Our algorithm leverages pessimism under uncertainty [13, 14]within the distributional RL framework [15, 16, 17].
Surprisingly, we ﬁnd that the greedy policyextracted from the value function of the behavior policy with our algorithm exhibits competitive andsometimes even state-of-the-art results in D4RL benchmarks [18].
Our results indicate not only theeffectiveness of the proposed algorithm adopting pessimistic regularization, but also implies potentialvulnerabilities of the iterative optimization process of actor-critic algorithms in the ofﬂine setting,especially when a complex function approximator like a deep neural network is used.
We expect thatthe proposed strong baseline will foster future ofﬂine RL research by allowing researchers to measurethe actual advantage coming from the iterative process, which is a main concern in ofﬂine RL.
Related Work
Evaluating and improving a policy with the data generated from a different policy (off-policy RL)has been widely investigated, and several papers have shown theoretical convergence propertiesof prediction and control algorithms in the off-policy setting [19, 20, 21].
However, theoreticalframeworks are limited to linear function approximation, while non-linear function approximation isessential to handle large-scale MDPs.
Yet, there have been efforts to build practical algorithms withnon-linear function approximators, and they have shown considerable success in various domains[22, 23, 24, 25] tackling real-world RL problems.
In theory, off-policy algorithms can be used in the ofﬂine setup without any modiﬁcations, butthe algorithms often catastrophically fail when applied in the batch setting [7].
This is due to theaccumulation of extrapolation error from bootstrapping and the policy improvement step (i.e. maxoperation) [7, 8].
In the non-batch setting, new experiences gathered via interaction can prevent thisdegenerate case, but it is impossible in the batch setting where interaction is prohibited.
Pertaining tothis problem, much research has been proposed, especially in the context of actor-critic algorithms inwhich policy evaluation and improvement are iterated.
One class of solutions constrains the policyimprovement step so that the optimized policy matches the behavior policy, in distribution [9] or insupport [8].
In more recent work, a behavior policy for distribution matching is replaced with a priorpolicy, which is trained along with policy evaluation via weighted behavior cloning [12].
In [11], thepolicy improvement step is omitted while using the prior policy as a target policy for the evaluation.
While the prior works try to overcome the Deadly Triad, we sidestep the problem and examine thehidden assumption of the actor-critic algorithm that the iterative process is essential and beneﬁcial.
The most closely related prior works are behavior cloning-based methods [26, 10] which mimic asubset of good state-action pairs from the pre-generated trajectories.
Since these methods approximatea value function for a behavior policy without an iterative policy improvement step, the algorithm doesnot diverge.
The main difference with our proposed approach is that we train an action-value functionwith pessimistic regularization while the prior works evaluate a state- or action-value function to ﬁlterstate-action pairs based on advantage calculated with Monte-Carlo return [26] or TD ( k ) return [10]and train a policy via behavior cloning with the ﬁltered data.
For principled regularization of the action-value function, we adopt pessimism under uncertainty,which can address the overestimation problem when the given ofﬂine dataset is not informative enoughto estimate the value of every action given a state [14].
This pessimism can be applied by directlypenalizing Q values for a particular state-action distribution [13], or with model-approximation[27] that leverages uncertainty prediction techniques developed for supervised learning, such as Lakshminarayanan et al.
[28] or van Amersfoort et al. While the previous approaches applypessimism unconditionally for every state [13] or use transition-dynamics based proxies [27] formeasuring uncertainty, we propose a theoretically justiﬁable regularization method for estimating Q βthat is based on distributional RL.
Preliminaries
The common mathematical framework for reinforcement learning is a Markov Decision Process(MDP), which is deﬁned by a tuple M =
( S , A , T, d 0 , r, γ ) deﬁned by a set of states s ∈ S , a set of actions a ∈ A , conditional transition dynamics T ( s (cid:48) | s, a ) , an initial state distribution d 0 , areward function r : S ×
A → R , and a discount factor γ ∈ (0 , 1] .
In this framework, the goalof reinforcement learning is to ﬁnd an optimal policy π ( a | s ) that maximizes an expected sum ofdiscounted reward (return).
Formally, the objective is deﬁned as:
[Equation]
where τ is a sequence of states and actions ( s 0 , a 0 , . .
, s H , a H ) of length H , and p π is a trajectorydistribution of a policy, which can be represented as:
[Equation]
One way to ﬁnd an optimal policy is to estimate an action-value function Q π , which representsthe expected return over possible trajectories following a policy π starting from a given state andaction: Q π ( s t , a t ) =
E
[Equation]
( s
[Equation]
) (cid:104)(cid:80)
[Equation]
r ( s
[Equation]
(cid:48) , a
[Equation]
) (cid:105) .
Q π function implies a greedy policyπ + ( a | s ) =
δ (cid:0) a = arg max b Q π ( s, b ) (cid:1) , which is better than or equal to its original evaluation targetpolicy π .
Therefore, when we perform policy evaluation ( Q estimation) and policy updates iteratively,we can move toward the optimal policy π ∗ and the optimal Q function Q π ∗ .
Policy evaluation can bedone with a Monte-Carlo method, but bootstrapping is commonly used, which utilizes a recursiveequation that must be satisﬁed at convergence:
[Equation]
When an MDP is discrete and Q can be represented by a tabular representation (i.e. when |S| × |A| issmall), it is known that policy evaluation converges to a correct solution in the limit of the number oftransition tuples [6].
However, when an MDP has a large state or action space, Q has to be representedwith a function approximator, such as a deep neural network.
In addition, when the action space iscontinuous, directly extracting a better policy from Q becomes infeasible due to the arg max operator.
These restrictions are addressed in actor-critic algorithms [4, 2, 5] which explicitly alternate policyevaluation and policy improvement with a batch of (online) transition samples D and a parameterizedvalue function ˆ Q θ and a policy π φ : [Equation]
[Equation]
where k is an update step, and d is a distance measure such as squared l 2 or Huber loss.
You Only Evaluate Once
Behind actor-critic based ofﬂine RL algorithms, there is a common presumption that the iterativeprocess is essential in achieving better performance than that of the behavior policy, even though itcould sacriﬁce the reliability of the algorithm due to the over-estimation problem.
This is because wewant to ﬁnd a policy that behaves differently from the data-generating policy by making counterfactualqueries (policy improvement) and answering (policy evaluation) them iteratively [1].
However, ithas not been established that a simpler and safer baseline cannot work well—a policy that selectsthe best action with regard to the action-value function of the behavior policy .
Without rigorouslyexamining this hypothesis, the true worth of iterative algorithms and counterfactual queries cannot befully understood.
In this paper, we challenge the iterative ofﬂine algorithms by proposing a stable ofﬂine algorithmthat recovers the best action of behavioral policy β ( s ) that is used to generate an ofﬂine dataset D .
Formally, our goal is to ﬁnd the greedy behavioral policy β ∗ ( s ) that selects the best action withrespect to Q β , only considering the action candidates of β :
[Equation]
a
[Equation]
where B ( s ) = supp β ( s ) := { a ∈ A : β ( a | s ) > 0 } .
[Equation]
When we have an oracle B ( s ) , β ∗ can be estimated by simply making on-policy queries to theapproximated action-value function ˆ Q β , which can be trained via TD loss based on SARSA tuple( s, a, r, s (cid:48) , a (cid:48) ) ∈ D .
This can be reasonably correct as long as the given dataset is sufﬁcient to performan on-policy evaluation.
However, the approach to learning β ∗ directly by estimating B ( s ) andˆ Q β is prone to failure. This is because directly modeling B ( s ) is often infeasible since β can bea mixture of many stochastic policies, and the estimation error in B ( s ) can create unpredictableerrors in ﬁnding β ∗ because the prediction of ˆ Q β for off-policy input ( s, ˜ a ) solely depends on thegeneralization ability of ˆ Q β .
Figure 1 demonstrates one failure case in estimating Q β ; when ˆ Q β istrained without any regularization, it behaves more like a state value function V β ( s ) ignoring theaction input. Therefore, any erroneous action generated by estimated B ( s ) will be treated as the bestaction.
Instead, we propose to learn the greedy behavioral policy β ∗ without directly estimating B ( s ) . It canbe achieved when ˆ Q β is properly regularized so that the greedy policy implied by the regularized ˆ Q βis β ∗ . We argue that pessimistic regularization satisﬁes such property:
Theorem ( ˆ Q β implies β ∗ ) .
Let ˆ Q β be a valid pessimistic approximation of Q β : (1) ˆ Q β ( s, ˜ a ) <max a ∈ B ( s ) Q β ( s, a ) ∀ s ∈ D, ˜ a / ∈ B ( s ) and (2) ˆ Q β ( s, a )
=
Q β ( s, a ) ∀ ( s, a ) ∈ D .
Then, a greedypolicy over ˆ Q β is β ∗ .
Proof.
[Equation]
Pessimistic regularization is essential even for learning Q β since there are always out-of-distributionstate-action pairs ( s, ˜ a ) given a ﬁxed dataset D , and the values of these inputs depend on thegeneralization characteristic of the function approximator used to learn Q β .
Therefore, we have toregularize ˆ Q β properly, and we adapt pessimism under the uncertainty principle since we cannotassume any value for OOD input unless we have a prior regarding an MDP. Yet, in contrast topessimistic regularization for Q ∗ , which requires a calibrated epistemic uncertainty measurement [14]or suffers from over-regularization due to excessive pessimism [13], ˆ Q β can be properly regularizedby enforcing ˆ Q β ( s, ˜ a ) to be smaller than the maximum value of ˆ Q β ( s, a ) for a ∈ B ( s ) .
4.1 Practical Implementation
When we have a valid pessimistic approximation of Q β , β ∗ can be inferred without knowing B ( s )since the greedy action over the approximated value function ˆ Q β is β ∗ . However, the ﬁrst constraint ofthe valid pessimistic approximation still depends on B ( s ) , and it makes implementing the pessimisticregularization in its original form difﬁcult. To address this problem, we express the two conditions of the theorem in terms of the state value distribution over Y β ( s ) ; Y β ( s ) is a random variable representing the return of the policy β starting from astate s where its expectation is the state value V β ( s ) =
E [ Y β ( s )] [15, 16, 17].
With the random variable Y β ( s ) , the two constraints can be rewritten as follows: (1) ˆ Q β ( s, · ) ≤ max a ∈ B ( s ) Q β ( s, a ) =max w Y β
( s ; w ) , and (2) ˆ Q β ( s, a ) =
r + γ E s (cid:48) [Equation] [Equation]
where w is an outcome of the sample space of Y .
This converted expression is deﬁned without a needfor B ( s ) , and therefore, we can now implement the pessimistic regularization with Y β ( s ) insteadof B ( s ) . Note that the constraint is deﬁned with the maximum and the expected value of Y β onlyfor the state s ∈ D . This allows us to use any on-policy evaluation methods for Y β as long as theresulting approximation is precise for on-policy states.
We propose a two-step approach: we ﬁrst perform state value distribution learning using ( s, r, s (cid:48) ) transition tuple in D (behavioral policy evaluation) and then train ˆ Q β in a supervised manner thatsatisﬁes the given constraints.
Speciﬁcally, we represent a state-value distribution of Y β usingimplicit quantile network (IQN) [17], which is a recently proposed distributional RL algorithm thatmodels the value distribution in the form of an inverse cumulative distribution function parameterizedwith ψ : (with a slight abuse of notation) ˆ Y ψ ( s ; τ ) = inf { v ∈ R :
τ ≤ Prob ( Y ( s ) ≤ v ) } where τis queried probability. In IQN, the parameter ψ can be trained by a distributional TD learning.
Fordetails about the distributional TD learning, please refer to Will et al.
With the approximated state-value distribution ˆ Y ψ of the behavior policy β , we train an actionvalue function ˆ Q θ with a supervised loss for on-policy state-action pair ( s, a ) and a pessimisticregularization R for off-policy pair ( s, ˜ a ) :
[Equation]
where λ is a hyperparameter that controls the strength of pessimism.
While we can approximatethe expectation of ˆ Y ψ with sampling, we use the median value of the distribution ˆ Y ψ ( s (cid:48) ; 0 . 5) forcomputational efﬁciency.
The main goal of R is to implement the pessimism on ˆ Q θ and guarantee ˆ Q θ to be a valid pessimisticapproximation of Q β that satisﬁes ˆ Q θ ( s, · ) ≤ max Y β ( s ) .
While every possible action a ∈
A needsto be checked, it is computationally infeasible when the action space is large or inﬁnite.
Hence, wesample actions from a static distribution µ (passive pessimistic regularization) and a distribution π φthat is actively changing over training iterations (active pessimistic regularization).
Then, we penalizeˆ Q θ if the estimations on the action samples violate the pessimistic constraint.
We use a differenthyperparameter τ for each of action samples.
We sample n b number of actions from each distributionand use the log-sum-exp trick to change the hard constraint into a soft constraint:
[Equation]
log
[Equation]
Algorithm 1: You Only Evaluate Once
Input : Dataset D = { ( s, a, r, s (cid:48) ) } ,
Hyper-parameter λ , τ 1 , τ
Initialize ψ , θ , and φ
Train ˆ Y ψ with distributional TD-loss while until convergence do
Update θ with ∇ θ L (Eq. 7)
Update φ with ∇ φ E s ∼ D ˆ Q θ (cid:0) s, π φ ( s ) (cid:1) end return ˆ Q θ , π φ , ˜ π
We use a random uniform policy for µ while the π φis trained against trained Q (or a set of Q s when anensemble technique is used) as same as an ordinaryactor in the actor-critic algorithm.
However, it isdifferent from the actor in that π φ works as an activeregularizer that adversarially ﬁnds the wrong generalization part of ˆ Q θ .
For τ 1 and τ 2 , we use 0.9 and0.1 respectively.
While the τ 1 is decided followingthe deﬁnition of valid pessimism with a safe marginof 0.1, we use a small τ 2 since we expect that a random policy will perform poorly.
We train multipleˆ Q θ models and aggregate them by taking a min overinferred ˆ Q θ values to get a robust value estimate.
We train multiple models with different initialparameters and the same data following Osband et al.
As an approximation of β ∗ , we use two policies deﬁned on top of ˆ Q θ .
One is the last π φ we have atthe end of training since π φ is directly trained to ﬁnd the greedy policy over ˆ Q θ .
If ˆ Q θ is properlyregularized (i.e. ˆ Q θ is a valid pessimistic approximation), π φ will converge to β ∗ .
Therefore, thepolicy will be implicitly constrained to use actions limited to the support of the behavior policy β .
The other policy is a nonparametric policy ˜ π whose action space is explicitly constrained to theactions shown in the dataset D : ˜ π ( s )
= arg max a ∼ D ˆ Q θ ( s, a ) . ˜ π can perform better than π φ whenthe pessimistic constraint is hard to achieve due to restriction of a dataset or characteristics of an MDP, such as the dataset size | D | or the large action space |A| , that allows room for adversarialattacks; when there is much room for adversarial attacks, π φ will continuously try to regularize ˆ Q θ bysuggesting diverse adversarial actions while it does not converge to the desired policy β ∗ .
In such acase, directly utilizing π φ would result in poor performance, but the action set restricted policy ˜
π canperform well by testing limited actions that are more likely to belong to B ( s ) .
The overall algorithmis shown in Algorithm 1, and the implementation details are provided in Appendix.
Since the training of ˆ Q θ largely depends on the trained state value distribution ˆ Y ψ , the correctness ofˆ Y ψ is essential in learning greedy behavioral policy β ∗ .
Fortunately, since we are performing policyevaluation with on-policy data, we avoid the Deadly Triad, and it is more likely to converge to acorrect Y β than other methods that perform off-policy learning.
Furthermore, the correctness of ˆ Y ψand the regularized ˆ Q θ can be roughly tested by comparing the estimated value with the Monte-Carlo return.
This is an extra debugging feature that is only available in YOEO, and we can leverage this toset the hyperparameters.
It is noteworthy that we are leveraging aleatoric uncertainty to implement pessimism.
When epistemicuncertainty of action-value function Q ∗ ( s, a ) can be estimated with a ﬁxed dataset, using it topenalize the value function is a theoretically justiﬁable implementation of pessimism [14].
However,measuring epistemic uncertainty is still an open problem when a deep neural network is used even inthe simpler supervised learning setting that does not include bootstrapping as RL.
For that reason,pessimism is commonly implemented with a proximal objective, such as learning a lower bound ofthe true Q ∗ [13] or using an uncertainty proxy [27, 31], and therefore, these approaches often requiresensitive hyperparameter tuning to ﬁnd the right level of pessimism [1].
In contrast, the objectiveof YOEO allows us to use aleatoric uncertainty that can be directly estimated.
Also, the two-steptraining prevents the error from propagating through bootstrapping, so our method is less susceptibleto divergence at the cost of optimality.
This makes our algorithm a suitable baseline for ofﬂine RLthat achieves stability at the cost of optimality.
Experiments
We aim to study the following question: how much performance gain do potentially risky policyiteration algorithms provide compared to more stable baseline algorithm YOEO?
We compare theperformance of YOEO against several baselines and prior works based on policy iteration: behaviorcloning (BC), soft actor-critic (SAC) [5] without interaction, bootstrapping error accumulationreduction (BEAR) [8], behavior regularized actor-critic (BRAC) [9], advantage weighted regression(AWR) [10], batch constrained deep Q-learning (BCQ) [7], and conservative Q-learning (CQL) [13].
The comparison is made on a subset of datasets in the D4RL ofﬂine RL benchmark [18].
Weuse MuJoCo [32] locomotion tasks [33], Adroit hand-manipulation tasks [34], and Franka kitchen[35] tasks.
For MuJoCo locomotion tasks, we use three environments ( hopper , walker2d , andhalfcheetah ) in four different settings ( random , medium-replay , medium , and medium-expert ).
For Adroit hand-manipulation tasks, we use four environments ( pen , door , relocate , and hammer )in two settings ( human and cloned ).
For Franka kitchen tasks, we use three different settings( complete , partial , and mixed ).
We train YOEO for 1 million stochastic gradient steps for bothˆ Y ψ and ˆ Q θ , then we report the average normalized performance score over 100 trajectories.
Theresults are displayed in Table 1.
Despite the fact that YOEO aims to learn a greedy behavioral policy β ∗ ( s ) with respect to Q β ratherthan try to learn π ∗ , it shows highly competitive results: it achieves better performance than thecurrent state-of-the-art model-free ofﬂine RL algorithms in ﬁve datasets (hopper-random, walker2dmedium-replay, hopper-medium-expert, and hammer-cloned, kitchen-compete), and competitiveperformance across all other conﬁgurations surpassing most of the other ofﬂine RL algorithms.
Theresults indicate that the previous ofﬂine RL algorithms fail to fully exploit the potential beneﬁt ofiterative evaluation and update.
The performance of ˜ π shown in the last column of Table 1 also supports our hypothesis on π φ .
While˜ π shows similar results in other datasets, ˜ π shows a better result than π φ in -human type dataset in
Adroit tasks and -parital and -mixed type datasets in the kitchen task of which the size of thedataset is relatively small or the action space is large; since ˆ Q θ needs to be regularized for diverse setof actions, π φ would not converge to β ∗ whose actions are limited to the support of β .
5.1 Ablation Study: how much will each regularization term affect the performance?
We conduct an ablation study to show the effectiveness of the regularization and the contribution ofeach term in Eq. 8.
Speciﬁcally, we consider two ablations: (1) ˆ Q β trained only via TD loss based on( s, a, r, s (cid:48) , a (cid:48) )
tuple without any regularization (denoted as ˆ Q β \R ) and (2) ˆ Q β trained without thesecond term in Eq.
8 (denoted as ˆ Q β \ µ ).
Additionally, we test the effect of the ensemble by changingthe number of trained models.
The experiment settings and the results are summarized in Table 2.
We conﬁrm the necessity of pessimistic regularization even when we do on-policy policy evaluation;the qualitative results shown in Figure 1 reveal that ˆ Q θ \R can estimate the ground-truth value foron-policy input ( s, a ) , but it fails to estimate the value for out-of-distribution input ( s, b ) , especially [Table] [Table] when the given dataset is homogeneous such as the -medium dataset whose behavior policy β is nota mixture of different policies.
This is because ˆ Q β behaves more like a ˆ V β , ignoring the action
[Table]
sincean effective action a is predictable based on s .
The performance degradation of ˆ Q θ \ µ indicates that arandom policy µ is a good heuristic that can prevent the degeneration of the action-value function;the second regularization term using µ can foster the discrimination ability of the value function byenforcing ˆ Q θ to estimate a different value for ( s, a ) and ( s, b )
[Table]
and this can increase the performancesigniﬁcantly especially when the given dataset is generated with a homogeneous policy
[Table]
We also observe the performance beneﬁt of training more models and ensembling them, especially for-medium-replay type datasets.
We hypothesize that YOEO’s pessimistic regularization method hashigh variance due to the stochasticity of the loss function, and the ensembling technique can improvelearning by enabling robust prediction for off-policy data from which the stochasticity is derived.
5.2 Why do CQL and other methods fail?
[Table]
YOEO outperforms the current state-of-the-artmethods, speciﬁcally CQL, in several datasets.
Considering that the pessimistic regularizationof YOEO is milder than CQL, we hypothesizethat the failure of CQL to surpass β ∗ is derivedfrom over-regularization; YOEO only penalizesˆ Q θ ( s, b ) that are larger than the maximum valueof the state value distribution max ˆ Y ψ , while
CQL performs unbounded minimization forˆ Q ∗ ( s, b ) and maximization for ˆ Q ∗ ( s, a ) .
Whenb is sampled from a policy based on ˆ Q , the regularization will practically penalize ˆ Q ∗ ( s, b ) untilthe value becomes smaller than ˆ Q ∗ ( s, a ) .
Evenwhen ˆ Q ∗ ( s, a ) is smaller than max ˆ Y ψ , CQLwould still induce more pessimism than YOEO.
Furthermore, when ˆ Q ∗ is over-regularized, we canexpect that ˆ Q ∗ would behave more like ˆ Q β , losing the advantage of the iterative process.
To conﬁrm this hypothesis, we modify CQL to estimate Q β instead of Q ∗ by changing the TD loss tobe computed with ( s, a, r, s (cid:48) , a (cid:48) ) , not ( s, a, r, s (cid:48) ) .
The pessimistic regularization part remains intact.
We denote this algorithm CQ β L. We ran the CQL( ρ ) algorithm, and the results are displayed in
Table 3.
CQ β L( ρ ) works similarly to CQL( ρ ), and this supports our hypothesis that the CQL isover-regularizing.
The success of the model-based ofﬂine algorithm (MOReL [31]), implies that theover-regularization of CQL can be avoided by utilizing the uncertainty that CQL lacks.
Discussion
We investigate a simple baseline algorithm for ofﬂine RL that only evaluates the value function of thebehavior policy as opposed to approximate Q ∗ with unstable iterative process.
Since the proposedalgorithm does not involve a policy optimization step and value re-evaluation based on the updatedpolicy, the algorithm can be stable, but the resulting policy is more likely to be suboptimal.
This makesthe algorithm an appropriate baseline for actor-critic algorithms that ought to outperform this baselineif there is indeed value in iterative optimization in the ofﬂine setting.
In the experiments, the proposedbaseline surprisingly shows competitive results on the several D4RL benchmarks, surpassing the stateof-the-art results in some tasks.
This implies the usefulness of conservativeness under uncertainty,which can prevent incorrect generalization behavior of a complex function approximator occurringdue to lack of data, as well as the potential ﬂaws of iterative optimization in actor-critic algorithmsin the ofﬂine setting.
Therefore, it is essential for future work to build a theoretical framework thatsheds light on iterative optimization and generalization of ofﬂine actor-critic methods that use deepneural networks, if iterative optimization is to be fully taken advantage of.