You Only Evaluate Once: a Simple Baseline Algorithm for Ofﬂine RL
Abstract: The goal of ofﬂine reinforcement learning (RL) is to ﬁnd an optimal policy given prerecorded trajectories.
Many current approaches customize existing off-policy RL algorithms, especially actor-critic algorithms in which policy evaluation and improvement are iterated.
However, the convergence of such approaches is not guaranteed due to the use of complex non-linear function approximation and an intertwined optimization process.
By contrast, we propose a simple baseline algorithm for ofﬂine RL that only performs the policy evaluation step once so that the algorithm does not require complex stabilization schemes.
Since the proposed algorithm is not likely to converge to an optimal policy, it is an appropriate baseline for actor-critic algorithms that ought to be outperformed if there is indeed value in iterative optimization in the ofﬂine setting.
Surprisingly, we empirically ﬁnd that the proposed algorithm exhibits competitive and sometimes even state-of-the-art performance in a subset of the D4RL ofﬂine RL benchmark.
This result suggests that future work is needed to fully exploit the potential advantages of iterative optimization in order to justify the reduced stability of such methods.
Introduction
The standard reinforcement learning setting involves an active component during learning: an agent continuously gathers experience as it learns.
This is a very general learning framework that resembles the way animals learn, but the interactive component often hurts the applicability of RL since the agent interaction can be expensive or unsafe.
To address this challenge, the ofﬂine reinforcement learning paradigm has been proposed, which aims to learn a policy purely from pre-generated data [1].
Considering that many recent breakthroughs in machine learning can be attributed to large-scale data, this new paradigm is very promising.
However, the ofﬂine setup causes signiﬁcant theoretic and algorithmic difﬁculties that need to be resolved to fulﬁll this promise.
Speciﬁcally, actor-critic based off-policy RL algorithms, which iterate policy evaluation and improvement, suffer from the overestimation problem caused by function approximation error and bootstrapping [2, 3] in the ofﬂine RL setup, even though the algorithms are equipped with algorithmic techniques [4, 2, 5] that can stabilize learning and mitigate the so-called Deadly Triad [6].
This is because over-estimated values cannot be readjusted in ofﬂine RL, unlike the ordinary RL setup where incorrectly optimistic actions get executed and corrected.
The Deadly Triad states that when off-policy learning, function approximation, and bootstrapping are used together, the danger of instability and divergence arises [6].
Therefore, the actor-critic algorithms that leverage all three components are vulnerable.
Many algorithms try to stabilize learning by ensuring the queries for bootstrapping to be within the data manifold of the given ofﬂine dataset [7, 8, 9, 10, 11, 12] since over-estimation is prominent when the value function is queried for out-of-distribution inputs.
However, strictly speaking, the Deadly Triad occurs due to iteration of the actor-critic algorithm; the ﬁrst critic update is simply policy evaluation of the action-value function of the behavior policy β using on-policy data.
While the policy implied by the ﬁrst value function
Q β is likely to be suboptimal that would be surpassed by iterative algorithms, it has not been fully examined that a greedy policy with regard to Q β cannot work well in the ofﬂine RL setting.
To this end, we propose a baseline algorithm, named YOEO, that evaluates the value of a behavior policy once and extracts a greedy policy under the learned value approximation, and by doing so, we examine the hidden presumption of the actor-critic based ofﬂine RL algorithms that the iterative process is essential and beneﬁcial.
Our algorithm leverages pessimism under uncertainty [13, 14] within the distributional RL framework [15, 16, 17].
Surprisingly, we ﬁnd that the greedy policy extracted from the value function of the behavior policy with our algorithm exhibits competitive and sometimes even state-of-the-art results in D4RL benchmarks [18].
Our results indicate not only the effectiveness of the proposed algorithm adopting pessimistic regularization, but also implies potential vulnerabilities of the iterative optimization process of actor-critic algorithms in the ofﬂine setting, especially when a complex function approximator like a deep neural network is used.
We expect that the proposed strong baseline will foster future ofﬂine RL research by allowing researchers to measure the actual advantage coming from the iterative process, which is a main concern in ofﬂine RL.
Related Work
Evaluating and improving a policy with the data generated from a different policy (off-policy RL) has been widely investigated, and several papers have shown theoretical convergence properties of prediction and control algorithms in the off-policy setting [19, 20, 21].
However, theoretical frameworks are limited to linear function approximation, while non-linear function approximation is essential to handle large-scale MDPs.
Yet, there have been efforts to build practical algorithms with non-linear function approximators, and they have shown considerable success in various domains [22, 23, 24, 25] tackling real-world RL problems.
In theory, off-policy algorithms can be used in the ofﬂine setup without any modiﬁcations, but the algorithms often catastrophically fail when applied in the batch setting [7].
This is due to the accumulation of extrapolation error from bootstrapping and the policy improvement step (i.e. max operation) [7, 8].
In the non-batch setting, new experiences gathered via interaction can prevent this degenerate case, but it is impossible in the batch setting where interaction is prohibited.
Pertaining to this problem, much research has been proposed, especially in the context of actor-critic algorithms in which policy evaluation and improvement are iterated.
One class of solutions constrains the policy improvement step so that the optimized policy matches the behavior policy, in distribution [9] or in support [8].
In more recent work, a behavior policy for distribution matching is replaced with a prior policy, which is trained along with policy evaluation via weighted behavior cloning [12].
In [11], the policy improvement step is omitted while using the prior policy as a target policy for the evaluation.
While the prior works try to overcome the Deadly Triad, we sidestep the problem and examine the hidden assumption of the actor-critic algorithm that the iterative process is essential and beneﬁcial.
The most closely related prior works are behavior cloning-based methods [26, 10] which mimic a subset of good state-action pairs from the pre-generated trajectories.
Since these methods approximate a value function for a behavior policy without an iterative policy improvement step, the algorithm does not diverge.
The main difference with our proposed approach is that we train an action-value function with pessimistic regularization while the prior works evaluate a state- or action-value function to ﬁlter state-action pairs based on advantage calculated with Monte-Carlo return [26] or TD ( k ) return [10] and train a policy via behavior cloning with the ﬁltered data.
For principled regularization of the action-value function, we adopt pessimism under uncertainty, which can address the overestimation problem when the given ofﬂine dataset is not informative enough to estimate the value of every action given a state [14].
This pessimism can be applied by directly penalizing Q values for a particular state-action distribution [13], or with model-approximation [27] that leverages uncertainty prediction techniques developed for supervised learning, such as
Lakshminarayanan et al.
[28] or van Amersfoort et al.
While the previous approaches apply pessimism unconditionally for every state [13] or use transition-dynamics based proxies [27] for measuring uncertainty, we propose a theoretically justiﬁable regularization method for estimating Q β that is based on distributional RL.
Preliminaries
The common mathematical framework for reinforcement learning is a Markov Decision Process (MDP), which is deﬁned by a tuple M = ( S , A , T, d 0 , r, γ ) deﬁned by a set of states s ∈ S , a set of actions a ∈ A , conditional transition dynamics T ( s (cid:48) | s, a ) , an initial state distribution d 0 , a reward function r : S ×
A → R , and a discount factor γ ∈ (0 , 1] .
In this framework, the goal of reinforcement learning is to ﬁnd an optimal policy π ( a | s ) that maximizes an expected sum of discounted reward (return).
Formally, the objective is deﬁned as:
[Equation]
where τ is a sequence of states and actions ( s 0 , a 0 , . .
, s H , a H ) of length H , and p π is a trajectory distribution of a policy, which can be represented as:
[Equation]
One way to ﬁnd an optimal policy is to estimate an action-value function Q π , which represents the expected return over possible trajectories following a policy π starting from a given state and action: Q
[Equation]
[Equation]
) (cid:105) .
Q π function implies a greedy policy π + ( a | s ) =
δ (cid:0) a = arg max b Q π ( s, b ) (cid:1) , which is better than or equal to its original evaluation target policy π .
Therefore, when we perform policy evaluation ( Q estimation) and policy updates iteratively, we can move toward the optimal policy π ∗ and the optimal Q function Q π ∗ .
Policy evaluation can be done with a Monte-Carlo method, but bootstrapping is commonly used, which utilizes a recursive equation that must be satisﬁed at convergence:
[Equation]
When an MDP is discrete and Q can be represented by a tabular representation (i [Equation] e. when |S| × |A| is small), it is known that policy evaluation converges to a correct solution in the limit of the number of transition tuples [6].
However, when an MDP has a large state or action space, Q has to be represented with a function approximator, such as a deep neural network.
In addition, when the action space is continuous, directly extracting a better policy from Q becomes infeasible due to the arg max operator.
These restrictions are addressed in actor-critic algorithms [4, 2, 5] which explicitly alternate policy evaluation and policy improvement with a batch of (online) transition samples D and a parameterized value function ˆ Q θ and a policy π φ :
[Equation]
where k is an update step, and d is a distance measure such as squared l 2 or Huber loss [Equation]
You Only Evaluate Once
Behind actor-critic based ofﬂine RL algorithms, there is a common presumption that the iterative process is essential in achieving better performance than that of the behavior policy, even though it could sacriﬁce the reliability of the algorithm due to the over-estimation problem.
This is because we want to ﬁnd a policy that behaves differently from the data-generating policy by making counterfactual queries (policy improvement) and answering (policy evaluation) them iteratively [1].
However, it has not been established that a simpler and safer baseline cannot work well—a policy that selects the best action with regard to the action-value function of the behavior policy .
Without rigorously examining this hypothesis, the true worth of iterative algorithms and counterfactual queries cannot be fully understood.
In this paper, we challenge the iterative ofﬂine algorithms by proposing a stable ofﬂine algorithm that recovers the best action of behavioral policy β ( s ) that is used to generate an ofﬂine dataset D .
Formally, our goal is to ﬁnd the greedy behavioral policy β ∗ ( s ) that selects the best action with respect to Q β , only considering the action candidates of β :
[Equation]
When we have an oracle B ( s ) , β ∗ can be estimated by simply making on-policy queries to the approximated action-value function ˆ Q β , which can be trained via TD loss based on SARSA tuple ( s, a, r, s (cid:48) , a (cid:48) )
∈ D .
This can be reasonably correct as long as the given dataset is sufﬁcient to perform an on-policy evaluation.
However, the approach to learning β ∗ directly by estimating B ( s ) and ˆ Q β is prone to failure.
This is because directly modeling B ( s ) is often infeasible since β can be a mixture of many stochastic policies, and the estimation error in B ( s ) can create unpredictable errors in ﬁnding β ∗ because the prediction of ˆ Q β for off-policy input ( s, ˜ a ) solely depends on the generalization ability of ˆ Q β .
Figure 1 demonstrates one failure case in estimating Q β ; when ˆ Q β is trained without any regularization, it behaves more like a state value function V β ( s ) ignoring the action input.
Therefore, any erroneous action generated by estimated B ( s ) will be treated as the best action.
Instead, we propose to learn the greedy behavioral policy β ∗ without directly estimating B ( s ) .
It can be achieved when ˆ Q β is properly regularized so that the greedy policy implied by the regularized ˆ Q β is β ∗ .
We argue that pessimistic regularization satisﬁes such property:
Theorem ( ˆ Q β implies β ∗ ) .
Let ˆ Q β be a valid pessimistic approximation of Q β : (1) ˆ Q β ( s, ˜ a ) < max a ∈ B ( s ) Q β ( s, a ) ∀ s ∈ D, ˜ a / ∈ B ( s ) and (2) ˆ Q β ( s, a )
=
Q β ( s, a ) ∀ ( s, a ) ∈ D .
Then, a greedy policy over ˆ Q β is β ∗ .
Proof.
[Equation]
= β ∗ ( s ) . Pessimistic regularization is essential even for learning Q β since there are always out-of-distribution state-action pairs ( s, ˜ a ) given a ﬁxed dataset D , and the values of these inputs depend on the generalization characteristic of the function approximator used to learn Q β .
Therefore, we have to regularize ˆ Q β properly, and we adapt pessimism under the uncertainty principle since we cannot assume any value for OOD input unless we have a prior regarding an MDP.
Yet, in contrast to pessimistic regularization for Q ∗ , which requires a calibrated epistemic uncertainty measurement [14] or suffers from over-regularization due to excessive pessimism [13], ˆ Q β can be properly regularized by enforcing ˆ Q β ( s, ˜ a ) to be smaller than the maximum value of ˆ Q β ( s, a ) for a ∈ B ( s ) .
4.1 Practical Implementation
When we have a valid pessimistic approximation of Q β , β ∗ can be inferred without knowing B ( s ) since the greedy action over the approximated value function ˆ Q β is β ∗ .
However, the ﬁrst constraint of the valid pessimistic approximation still depends on B ( s ) , and it makes implementing the pessimistic regularization in its original form difﬁcult.
To address this problem, we express the two conditions of the theorem in terms of the state value distribution over Y β ( s ) ; Y β ( s ) is a random variable representing the return of the policy β starting from a state s where its expectation is the state value V β ( s ) =
E [ Y β ( s )] [15, 16, 17].
With the random variable Y β ( s ) , the two constraints can be rewritten as follows: (1) ˆ Q β ( s, · ) ≤ max a ∈ B ( s ) Q β ( s, a ) = max w Y β
( s ; w ) , and (2) ˆ Q β ( s, a ) =
r + γ E s (cid:48) ∼ T ( s,a )
[ V β ( s (cid:48) )] = r + γ E s (cid:48) ∼ T
[Equation]
where w is an outcome of the sample space of Y .
This converted expression is deﬁned without a need for B ( s ) , and therefore, we can now implement the pessimistic regularization with Y β ( s ) instead of B ( s ) .
Note that the constraint is deﬁned with the maximum and the expected value of Y β only for the state s ∈ D .
This allows us to use any on-policy evaluation methods for Y β as long as the resulting approximation is precise for on-policy states.
We propose a two-step approach: we ﬁrst perform state value distribution learning using ( s, r, s (cid:48) ) transition tuple in D (behavioral policy evaluation) and then train ˆ Q β in a supervised manner that satisﬁes the given constraints.
Speciﬁcally, we represent a state-value distribution of Y β using implicit quantile network (IQN) [17], which is a recently proposed distributional RL algorithm that models the value distribution in the form of an inverse cumulative distribution function parameterized with ψ : (with a slight abuse of notation)
ˆ Y ψ ( s ; τ ) =
inf { v ∈ R : τ ≤ Prob ( Y ( s ) ≤ v ) } where τ is queried probability.
In IQN, the parameter ψ can be trained by a distributional TD learning.
For details about the distributional TD learning, please refer to Will et al.
With the approximated state-value distribution ˆ Y ψ of the behavior policy β , we train an actionvalue function ˆ Q θ with a supervised loss for on-policy state-action pair ( s, a ) and a pessimistic regularization R for off-policy pair ( s, ˜ a ) :
[Equation]
where λ is a hyperparameter that controls the strength of pessimism.
While we can approximate the expectation of ˆ Y ψ with sampling, we use the median value of the distribution ˆ Y ψ ( s (cid:48) ; 0 . 5) for computational efﬁciency.
The main goal of R is to implement the pessimism on ˆ Q θ and guarantee ˆ Q θ to be a valid pessimistic approximation of Q β that satisﬁes ˆ Q θ ( s, · ) ≤ max Y β ( s ) .
While every possible action a ∈
A needs to be checked, it is computationally infeasible when the action space is large or inﬁnite.
Hence, we sample actions from a static distribution µ (passive pessimistic regularization) and a distribution π φ that is actively changing over training iterations (active pessimistic regularization).
Then, we penalize ˆ Q θ if the estimations on the action samples violate the pessimistic constraint.
We use a different hyperparameter τ for each of action samples.
We sample n b number of actions from each distribution and use the log-sum-exp trick to change the hard constraint into a soft constraint:
[Equation]
log
[Equation]
Algorithm 1: You Only Evaluate Once
Input : Dataset D = { ( s, a, r, s (cid:48) ) } ,
Hyper-parameter λ , τ 1 , τ
Initialize ψ , θ , and φ
Train ˆ Y ψ with distributional TD-loss while until convergence do
Update θ with ∇ θ L (Eq. 7)
Update φ with ∇ φ E s ∼ D ˆ Q θ (cid:0) s, π φ ( s ) (cid:1) end return ˆ Q θ , π φ , ˜ π
We use a random uniform policy for µ while the π φ is trained against trained Q (or a set of Q s when an ensemble technique is used) as same as an ordinary actor in the actor-critic algorithm.
However, it is different from the actor in that π φ works as an active regularizer that adversarially ﬁnds the wrong generalization part of ˆ Q θ .
For τ 1 and τ 2 , we use 0.9 and 0.1 respectively.
While the τ 1 is decided following the deﬁnition of valid pessimism with a safe margin of 0.1, we use a small τ 2 since we expect that a random policy will perform poorly.
We train multiple ˆ Q θ models and aggregate them by taking a min over inferred ˆ Q θ values to get a robust value estimate.
We train multiple models with different initial parameters and the same data following Osband et al.
As an approximation of β ∗ , we use two policies deﬁned on top of ˆ Q θ .
One is the last π φ we have at the end of training since π φ is directly trained to ﬁnd the greedy policy over ˆ Q θ .
If ˆ Q θ is properly regularized (i.e. ˆ Q θ is a valid pessimistic approximation), π φ will converge to β ∗ .
Therefore, the policy will be implicitly constrained to use actions limited to the support of the behavior policy β .
The other policy is a nonparametric policy ˜ π whose action space is explicitly constrained to the actions shown in the dataset D : ˜ π ( s )
= arg max a ∼ D ˆ Q θ
( s, a ) .
˜ π can perform better than π φ when the pessimistic constraint is hard to achieve due to restriction of a dataset or characteristics of an
MDP, such as the dataset size | D | or the large action space |A| , that allows room for adversarial attacks; when there is much room for adversarial attacks, π φ will continuously try to regularize ˆ Q θ by suggesting diverse adversarial actions while it does not converge to the desired policy β ∗ .
In such a case, directly utilizing π φ would result in poor performance, but the action set restricted policy ˜ π can perform well by testing limited actions that are more likely to belong to B ( s ) .
The overall algorithm is shown in Algorithm 1, and the implementation details are provided in Appendix.
Since the training of ˆ Q θ largely depends on the trained state value distribution ˆ Y ψ , the correctness of ˆ Y ψ is essential in learning greedy behavioral policy β ∗ .
Fortunately, since we are performing policy evaluation with on-policy data, we avoid the Deadly Triad, and it is more likely to converge to a correct Y β than other methods that perform off-policy learning.
Furthermore, the correctness of ˆ Y ψ and the regularized ˆ Q θ can be roughly tested by comparing the estimated value with the Monte-Carlo return.
This is an extra debugging feature that is only available in YOEO, and we can leverage this to set the hyperparameters.
It is noteworthy that we are leveraging aleatoric uncertainty to implement pessimism.
When epistemic uncertainty of action-value function Q ∗ ( s, a ) can be estimated with a ﬁxed dataset, using it to penalize the value function is a theoretically justiﬁable implementation of pessimism [14].
However, measuring epistemic uncertainty is still an open problem when a deep neural network is used even in the simpler supervised learning setting that does not include bootstrapping as RL.
For that reason, pessimism is commonly implemented with a proximal objective, such as learning a lower bound of the true Q ∗ [13] or using an uncertainty proxy [27, 31], and therefore, these approaches often require sensitive hyperparameter tuning to ﬁnd the right level of pessimism [1].
In contrast, the objective of YOEO allows us to use aleatoric uncertainty that can be directly estimated.
Also, the two-step training prevents the error from propagating through bootstrapping, so our method is less susceptible to divergence at the cost of optimality.
This makes our algorithm a suitable baseline for ofﬂine RL that achieves stability at the cost of optimality.
Experiments
We aim to study the following question: how much performance gain do potentially risky policy iteration algorithms provide compared to more stable baseline algorithm YOEO?
We compare the performance of YOEO against several baselines and prior works based on policy iteration: behavior cloning (BC), soft actor-critic (SAC) [5] without interaction, bootstrapping error accumulation reduction (BEAR) [8], behavior regularized actor-critic (BRAC) [9], advantage weighted regression (AWR)
[10], batch constrained deep Q-learning (BCQ) [7], and conservative Q-learning (CQL) [13].
The comparison is made on a subset of datasets in the D4RL ofﬂine RL benchmark [18].
We use MuJoCo [32] locomotion tasks [33], Adroit hand-manipulation tasks [34], and Franka kitchen [35] tasks.
For MuJoCo locomotion tasks, we use three environments ( hopper , walker2d , and halfcheetah ) in four different settings ( random , medium-replay , medium , and medium-expert ).
For Adroit hand-manipulation tasks, we use four environments ( pen , door , relocate , and hammer ) in two settings ( human and cloned ).
For Franka kitchen tasks, we use three different settings ( complete , partial , and mixed ).
We train YOEO for 1 million stochastic gradient steps for both ˆ Y ψ and ˆ Q θ , then we report the average normalized performance score over 100 trajectories.
The results are displayed in Table 1.
Despite the fact that YOEO aims to learn a greedy behavioral policy β ∗ ( s ) with respect to Q β rather than try to learn π ∗ , it shows highly competitive results: it achieves better performance than the current state-of-the-art model-free ofﬂine RL algorithms in ﬁve datasets (hopper-random, walker2dmedium-replay, hopper-medium-expert, and hammer-cloned, kitchen-compete), and competitive performance across all other conﬁgurations surpassing most of the other ofﬂine RL algorithms.
The results indicate that the previous ofﬂine RL algorithms fail to fully exploit the potential beneﬁt of iterative evaluation and update.
The performance of ˜ π shown in the last column of Table 1 also supports our hypothesis on π φ .
While ˜ π shows similar results in other datasets, ˜ π shows a better result than π φ in -human type dataset in
Adroit tasks and -parital and -mixed type datasets in the kitchen task of which the size of the dataset is relatively small or the action space is large; since ˆ Q θ needs to be regularized for diverse set of actions, π φ would not converge to β ∗ whose actions are limited to the support of β .
5.1 Ablation Study: how much will each regularization term affect the performance?
We conduct an ablation study to show the effectiveness of the regularization and the contribution of each term in Eq. 8.
Speciﬁcally, we consider two ablations: (1) ˆ Q β trained only via TD loss based on ( s, a, r, s (cid:48) , a (cid:48) )
tuple without any regularization (denoted as ˆ Q β \R ) and (2) ˆ Q β trained without the second term in Eq.
8 (denoted as ˆ Q β \ µ ).
Additionally, we test the effect of the ensemble by changing the number of trained models.
The experiment settings and the results are summarized in Table 2.
We conﬁrm the necessity of pessimistic regularization even when we do on-policy policy evaluation; the qualitative results shown in Figure 1 reveal that ˆ Q θ \R can estimate the ground-truth value for on-policy input ( s, a ) , but it fails to estimate the value for out-of-distribution input ( s, b ) , especially
[Table]
[Table]
when the given dataset is homogeneous such as the -medium dataset whose behavior policy β is not a mixture of different policies.
This is because ˆ Q β behaves more like a ˆ V β , ignoring the action, since an effective action a is predictable based on s .
The performance degradation of ˆ Q θ \ µ indicates that a random policy µ is a good heuristic that can prevent the degeneration of the action-value function; the second regularization term using µ can foster the discrimination ability of the value function by enforcing ˆ Q θ to estimate a different value for ( s, a ) and ( s, b )
[Table]
and this can increase the performance signiﬁcantly especially when the given dataset is generated with a homogeneous policy
[Table]
We also observe the performance beneﬁt of training more models and ensembling them, especially for -medium-replay type datasets.
We hypothesize that YOEO’s pessimistic regularization method has high variance due to the stochasticity of the loss function, and the ensembling technique can improve learning by enabling robust prediction for off-policy data from which the stochasticity is derived.
5.2 Why do CQL and other methods fail?
[Table]
YOEO outperforms the current state-of-the-art methods, speciﬁcally CQL, in several datasets.
Considering that the pessimistic regularization of YOEO is milder than CQL we hypothesize that the failure of CQL to surpass β ∗ is derived from over-regularization; YOEO only penalizes ˆ Q θ ( s, b ) that are larger than the maximum value of the state value distribution max ˆ Y ψ , while
CQL performs unbounded minimization for ˆ Q ∗ ( s, b ) and maximization for ˆ Q ∗ ( s, a ) .
When b is sampled from a policy based on ˆ Q , the regularization will practically penalize ˆ Q ∗ ( s, b ) until the value becomes smaller than ˆ Q ∗ ( s, a ) .
Even when ˆ Q ∗ ( s, a ) is smaller than max ˆ Y ψ , CQL would still induce more pessimism than YOEO.
Furthermore, when ˆ Q ∗ is over-regularized, we can expect that ˆ Q ∗ would behave more like ˆ Q β , losing the advantage of the iterative process.
To conﬁrm this hypothesis, we modify CQL to estimate Q β instead of Q ∗ by changing the TD loss to be computed with ( s, a, r, s (cid:48) , a (cid:48) ) , not ( s, a, r, s (cid:48) ) .
The pessimistic regularization part remains intact.
We denote this algorithm CQ β L. We ran the CQL( ρ ) algorithm, and the results are displayed in
Table 3.
CQ β L( ρ ) works similarly to CQL( ρ ), and this supports our hypothesis that the CQL is over-regularizing.
The success of the model-based ofﬂine algorithm (MOReL [31]), implies that the over-regularization of CQL can be avoided by utilizing the uncertainty that CQL lacks.
Discussion
We investigate a simple baseline algorithm for ofﬂine RL that only evaluates the value function of the behavior policy as opposed to approximate Q ∗ with unstable iterative process.
Since the proposed algorithm does not involve a policy optimization step and value re-evaluation based on the updated policy, the algorithm can be stable, but the resulting policy is more likely to be suboptimal.
This makes the algorithm an appropriate baseline for actor-critic algorithms that ought to outperform this baseline if there is indeed value in iterative optimization in the ofﬂine setting.
In the experiments, the proposed baseline surprisingly shows competitive results on the several D4RL benchmarks, surpassing the stateof-the-art results in some tasks.
This implies the usefulness of conservativeness under uncertainty, which can prevent incorrect generalization behavior of a complex function approximator occurring due to lack of data, as well as the potential ﬂaws of iterative optimization in actor-critic algorithms in the ofﬂine setting.
Therefore, it is essential for future work to build a theoretical framework that sheds light on iterative optimization and generalization of ofﬂine actor-critic methods that use deep neural networks, if iterative optimization is to be fully taken advantage of. Acknowledgments
This work has taken place in the Personal Autonomous Robotics Lab (PeARL) at The University of Texas at Austin.
PeARL research is supported in part by the NSF (IIS-1724157, IIS-1638107,
IIS-1749204, IIS-1925082), ONR (N00014-18-2243), AFOSR (FA9550-20-1-0077), and ARO (78372-CS).