A Workﬂow for Ofﬂine Model-Free Robotic Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Ofﬂine reinforcement learning (RL) enables learning control policiesby utilizing only prior experience, without any online interaction.
This can allow robots to acquire generalizable skills from large and diverse datasets, without costly online data collection.
Despite recent algorithmic advances in ofﬂine
RL, applying these methods to real-world problems has proven challenging.
Although ofﬂine RL methods can learn from prior data, there is no clear and wellunderstood process for making various design choices, from model architectureto algorithm hyperparameters, without actually evaluating the learned policies online.
In this paper, our aim is to develop a practical workﬂow for applying ofﬂine
RL analogous to the relatively well-understood workﬂows for supervised learning problems.
To this end, we devise a set of metrics and conditions that can betracked over the course of ofﬂine training, and can inform the practitioner abouthow the algorithm and model architecture should be adjusted to improve ﬁnalperformance.
Our workﬂow is derived from a conceptual understanding of thebehavior of conservative ofﬂine RL algorithms and cross-validation in supervisedlearning.
We demonstrate the efﬁcacy of this workﬂow in producing effective policies without any online tuning, both in several simulated robotic learning scenariosand for three tasks on two distinct real robots, focusing on learning manipulationskills with raw image observations with sparse binary rewards.
Introduction
[Figure]
Ofﬂine reinforcement learning (RL) can in principle make it possible to convert existing largedatasets of robotic experience into effective policies, without the need for costly or dangerousonline interaction for each training run.
Whileofﬂine RL algorithms have improved signiﬁcantly [1, 2, 3, 4, 5], applying such methods toreal-world robotic control problems presents anumber of major challenges.
In standard online
RL, any intermediate policy found during trainingis executed in the environment to collect more experience, which naturally allows for an evaluationof the policy performance.
This ability to evaluateintermediate policies lets practitioners use “bruteforce” to evaluate the effects of various design factors, such as model capacity and expressivity, thenumber of training steps, and so forth, and facilitates comparatively straightforward tuning.
Incontrast, ofﬂine RL methods do not have access to real-world on-policy rollouts for evaluating thelearned policy.
Thus, in order for these methods to be truly practical for real-world applications, wenot only require effective algorithms, but also an effective workﬂow : a set of protocols and metricsthat can be used to reliably and consistently adjust model capacity, regularization, etc in ofﬂine RLto obtain policies with good performance, without requiring real-world rollouts for tuning.
A number of prior works have studied model selection in ofﬂine RL by utilizing off-policy evaluation (OPE) methods [6] to estimate policy performance.
These methods can be based either onmodel or value learning [7, 8, 9, 10] or importance sampling [6, 11, 12, 13].
However, developingreliable OPE methods is itself an open problem, and modern OPE methods themselves suffer fromhyperparameter selection challenges (see Fu et al.
[14] for an empirical study).
Moreover, full offpolicy evaluation is likely not necessary to simply tune algorithms for best performance – we donot need a precise estimate of how good our policy is, but rather a workﬂow that enables us to bestimprove it by adjusting various algorithm parameters.
In this paper, we devise a practically effective workﬂow for selecting regularizers, model architectures, and good policy checkpoints for ofﬂine RL methods in robotic learning settings.
We focus onthe class of conservative ofﬂine RL algorithms [15, 2] that regularize the Q-function.
Our aim is notto focus on complete off-policy evaluation methods, but rather to adopt a strategy similar to the onein supervised learning.
Analogously to how supervised learning practitioners can detect overﬁttingand underﬁtting by tracking training and validation losses, and then adjust hyperparameters based onthese metrics, our workﬂow (see Figure 1 for a schematic) ﬁrst deﬁnes and characterizes overﬁttingand underﬁtting, proposes a set of metrics and conditions that practitioners can track to determineif an ofﬂine RL exhibits overﬁtting or underﬁtting, and then utilizes these metrics to inform designdecisions pertaining to improved neural network architectures, regularization and early stopping.
This protocol is intended to act as a “user’s manual” for a practitioner, enabling them to understandhow a given ofﬂine RL run might fall short and then take steps to modify algorithm to improve theoutcome of the next experiment, without requiring real-world evaluation rollouts.
The primary contribution of this paper is a simple yet effective workﬂow for robotic ofﬂine RL.
Wepropose metrics and protocols to assist practitioners in selecting policy checkpoints, regularizationparameters, and model architectures for conservative ofﬂine RL algorithms such as CQL [2].
Weempirically verify our proposed workﬂow on simulated robotic manipulation problems as well asthree real-world robotic manipulation problems on two different robots, with diverse objects, pixelobservations, and sparse binary reward supervision.
Our workﬂow improves the policy success rateto 70% on two real-world Sawyer robot manipulation tasks which previously only failed at the task.
Related Work
Robotic RL with ofﬂine datasets.
Learning-based methods have been applied to a number ofrobotics problems such as grasping objects [16, 17], in-hand object manipulation [18, 19, 20, 21],pouring ﬂuids [22], door opening [23], and manipulating cloth [24].
While the majority of theseworks use standard online RL, a number of prior works have used also leveraged robotic datasets totrain skills in addition to active environment rollouts.
Kalashnikov et al.
[16], Julian et al.
[25], and
Cabi et al.
[26] use ofﬂine pre-training followed by a ﬁnetuning phase to improve the policy.
Visualforesight [27] and related methods [28, 29, 30, 31] train a video-predictive dynamics model and useit for ofﬂine planning.
Young et al.
[32], Johns [33] lean skills in an ofﬂine manner, but only useimitation learning and not ofﬂine RL.
Mandlekar et al.
[34, 35] propose to learn hierarchical skillsand combine them using ofﬂine imitation learning.
Our work is complementary to these prior worksin that our proposed workﬂow can be utilized by any robotic ofﬂine RL system.
Ofﬂine deep RL.
While ofﬂine RL (also called batch RL) is a well-studied area [36, 37, 38, 15],several recent ofﬂine deep RL algorithms have been proposed.
Algorithms for ofﬂine deep RLcan be divided into three categories: those that constrain the policy to correct for distributionshift [39, 40, 41, 42, 43, 44], those that prevent overestimation in the value function via critic regularization [2, 45, 46] and those that train dynamics models and apply a reward penalty [47, 48].
These algorithms have been applied in robotics, for example when learning from unlabeled data [3],robotic manipulation [49, 16], goal-conditioned RL [4, 50] and multi-task RL problems [1].
Ratherthan developing a new ofﬂine RL method, our work develops criteria and workﬂow rules that simplify the application of these methods to new robotics tasks.
Preliminaries, Background, and Deﬁnitions
The goal in RL is to optimize the inﬁnite horizon discounted return R = (cid:80) ∞ t =0 γ t r ( s t , a t ) , wherer ( s, a ) represents the reward function evaluated at a state-action pair ( s , a ) .
We operate in the ofﬂine
RL setting and are provided with a ﬁxed dataset D
= { ( s , a , r ( s , a ) , s (cid:48) ) } , consisting of transitiontuples obtained from rollouts under a behavior policy π β ( a | s ) .
Our goal is to obtain the best possiblepolicy by only training on this ﬁxed ofﬂine dataset D , with no access to online rollouts.
Conservative Q-learning (CQL).
In order to obtain a good policy from the ofﬂine dataset, D , weuse conservative Q-learning (CQL) [2], a state-of-the-art ofﬂine RL algorithm.
We use the actorcritic formulation of CQL that trains a Q-function Q θ ( s , a ) and a separate policy π φ ( a | s ) that aims tomaximize the expected Q-value E s ∼D , a ∼ π φ [ Q θ ( s , a )] like other standard actor-critic deep RL methods [51, 52, 53].
In addition to the standard temporal difference (TD) error L TD ( θ ) (shown in bluebelow), CQL applies a regularizer R ( θ ) (shown in red below) to prevent erroneous overestimationof Q-values.
This aims to minimize the expected Q-value under µ ( a | s ) , E s ∼D , a ∼ µ ( ·| s )
[ Q θ ( s , a )] .
The choice of µ ( a | s ) is directly prescribed by the CQL algorithm
and it is not a hyperparameter totune. µ ( a | s ) selectively picks out/minimizes Q-values at actions a with high Q-values Q θ ( s , a ) :min
[Equation]
where B π ¯ Q ( s , a ) denotes the Bellman backup operator applied on the delayed target Q-function, ¯ Q :
B [Equation]
¯
[Equation]
In practice, CQL computes µ ( a | s ) using actionssampled from the policy π φ ( a | s ) .
More discussion of CQL is in Appendix B.
In this paper, we willutilize CQL as a base algorithm that our workﬂow intends to tune.
In Appendix F.1, we discuss otherofﬂine RL methods that our workﬂow can be adapted to and show a concrete example, BRAC [41].
[Table]
Overﬁtting and underﬁtting in CQL.
Conservativeofﬂine RL algorithms [2,45] like CQL can be sensitive to design choices, including number of gradient steps for training [54,55] and network capacity.
These challenges are omnipresent throughout machine learning, even in supervised learning methods.
However, supervised learning methods beneﬁt from a simple and powerful workﬂow using training error and validation error (which serves as a proxy for test error) tocharacterize overﬁtting and underﬁtting.
A practitioner can then make tuning choices based on thesecharacterizations.
To derive an analogous workﬂow for ofﬂine RL, we ﬁrst ask: what do overﬁttingand underﬁtting actually mean for the case of conservative ofﬂine RL?
To deﬁne overﬁtting and underﬁtting generically for any conservative ofﬂine RL method, we consider an abstract optimization formulation for such methods [2]:
[Equation]
J D ( π ) denotes the average return of policy π in the empirical MDP induced by the transitions inthe ofﬂine dataset D , and D ( π, π β ) denotes a closeness constraint to the behavior policy, effectivelyapplied by the ofﬂine RL method.
This divergence is computed in expectation over the visitationdistribution of the learned policy π as discussed in Appendix F.1.
For example, Equation 1 translatesto utilizing D CQL ( p, q ) :
= (cid:80) x p ( x )( p ( x ) /q
( x ) − 1) in Equation 2.
The training loss in Equations 2and 1 and the test loss is equal to the actual return J ( π ) of the learned policy.
Analogously tosupervised learning, we can use the notion of train and test error to deﬁne overﬁtting and underﬁttingin ofﬂine RL, as shown in Table 1.
However, note that the conditions summarized in Table 1 arenot measurable completely ofﬂine.
Precisely estimating if a run of an ofﬂine RL method overﬁts orunderﬁts requires evaluating the learned policy via interaction with the real-world environment.
In Section 4, our goal will be to devise ofﬂine metrics for characterizing overﬁtting that do not havethis requirement.
We will tailor our study speciﬁcally towards CQL.
A similar procedure could bedevised for other conservative ofﬂine RL methods, but we leave this for future work.
Detecting Overﬁtting and Underﬁtting in Conservative Ofﬂine RL
In standard supervised learning, we can determine if a method overﬁts or underﬁts by comparing thetraining loss to the same loss function evaluated on a held-out validation dataset, which serves as a“proxy” test dataset for tuning without needing access to the test dataset.
In contrast, the return ofthe learned policy J ( π ) in RL does not have a direct proxy that can be computed ofﬂine.
Thus, ourgoal is to identify ofﬂine metrics and conditions that allow us to measure overﬁtting and underﬁttingin conservative ofﬂine RL, with a focus on CQL. We also adapt the overﬁtting metrics/conditions to
BRAC [41], a policy-constraint method, and demonstrate how to tune BRAC in Appendix F.2.
Detecting overﬁtting in CQL.
Overﬁtting (Table 1) is deﬁned as thescenario where the method attains a low value for the training loss (Equation 1), without actually giving rise to good policies with high J ( π ) .
Todetect overﬁtting for CQL, we analyze the time series of the estimated
Q-values averaged over the dataset samples ( s , a , r, s (cid:48) )
∈ D over thecourse of training with a large number of gradient steps.
In scenarioscharacterized by overﬁtting, we should see that the this time series of theexpected dataset Q-value exhibits a non-monotonic trend: the average Q-values ﬁrst increase andthen decrease as shown in the ﬁgure on the right.
Additionally, we would see that training loss in
Equation 1 eventually becomes very low.
Why does this happen?
Since CQL selectively penalizes the average Q-value under the distribution µ ( a | s ) supported on actions with large Q-values, wewould expect the Q-values on states from the dataset s ∼ D and the learned a ∼ π ( ·| s ) to be small 164 since the policy is trained to maximize the Q-function as well.
This in turn would lead to an eventualreduction in the average Q-value on dataset actions, E s , a ∼D [ Q θ ( s , a )] , which would be visible aftersufﬁciently many steps of training, when values have propagated via Bellman backups in Equation
1giving rise to the non-monotonic trend.
If such a phenomenon is observed, this raises two questions:
What does a low average Q-value E s , a ∼D [ Q θ ( s , a )] imply about J ( π ) ?
We show in Appendix Athat, in principle, CQL training (Equation 1) should never learn Q-values smaller than the dataset
Monte-Carlo return, and the Q-values should increase unless the learned policy π is sufﬁcientlydifferent from π β .
Intuitively, this is because the objective in Equation 1 aims to maximize theaverage dataset Q-value, and we formalize this intuition in Appendix A.
Thus, a low Q-value on( s , a ) ∈ D indicates that the Q-function predicts extremely small Q-values on actions sampled fromµ ( a | s ) .
Typically this would mean the highest Q-value actions a at a state s ∈ D are those sampledfrom the ofﬂine dataset, drawn from the behavior policy. Thus, policy optimization, which aims tomaximize the Q-value, would make π ( a | s ) closer to the behavior policy π β ( a | s ) on s ∈ D , implying 177 that the resulting policy would have poor performance J ( π ) , that matches or is worse than π β .
Which training checkpoint is likely to attain the best policy performance?
One beneﬁt of detectingoverﬁtting in supervised learning is to select the best-performing checkpoint before overﬁtting becomes severe.
Along similar lines, rather than explicitly quantifying what a “low” Q-value means,we can compare the average dataset Q-value across different checkpoints within a same run, andperform a relative comparison to guide policy selection when faced with overﬁtting.
Since CQLaims to increase the average dataset Q-value (Equation 1), we would expect Q-values to initiallyincrease, until learning starts to overﬁt and
the average dataset Q-value starts decreasing.
Thus, fordeployment, we should select the latest checkpoint that corresponds to the peak in the dataset Qvalue.
A visual illustration of this idea is shown in the ﬁgure above, where the checkpoint markedby the green line is recommended to be chosen.
In summary , (a) to detect overﬁtting we can track:
Metric 4.1 (Overﬁtting) .
A low average data Q-value E s , a ∼D [ Q θ ( s , a )] that decreases with more gradient steps on Equation 1 indicates that the ofﬂine RL algorithm is overﬁtting.and (b) further, given a run that exhibits overﬁtting, our principle for policy selection is given by:
Guideline 4.1 (Policy selection) .
If a run overﬁts (per Metric 4.1), select the checkpoint that attains the highest average dataset Q-value before overﬁtting for deployment.
Detecting underﬁtting in CQL.
Next, we turn to devising a procedureto detect underﬁtting.
As summarized in Table 1, underﬁtting occurswhen the RL algorithm is unable to minimize the training objective in
Equation 1 effectively.
Therefore, large values for the TD error, the CQLregularizer, or both imply underﬁtting.
A large value for the CQL regularizer, R ( θ ) , indicates an overestimation of Q-values relative to theirtrue value [2] and thus, unlike the overﬁtting regime, we would not expect the average learned Q-value to decrease with more training.
Thus, one approach to predictunderﬁtting is to track both the TD error, L TD ( θ ) , and the CQL regularizer, R ( θ ) , and check if thevalue of even one of these quantities is large.
More discussion is provided in Appendix A.
How do we determine if the TD error and the CQL regularizer are “large”?
In order to determineif the error of a particular run is large, we can rerun the base CQL algorithm
but with models of higher capacity, which does not necessarily correspond to the function approximator size, as wewill discuss in Section 5.
For each model, we record the corresponding training errors and check ifthe training TD error and CQL regularizer value are reduced with capacity increase.
If increasingcapacity leads to a reduction in the loss without exhibiting the overﬁtting signs described previously,then we are in an underﬁtting regime.
Another approach to answer the question is to utilize the valueof the TD error ( L TD ( θ ) ) and the task horizon ( 1 / (1 − γ ) ) to estimate the overall error in the learned
Q-values against the actual Q-value, which is equal to L TD ( θ ) / (1 − γ ) [56] (see Appendix A).
If thisoverall error spans the range of allowed Q-values on the task – which could be inferred based on thestructure of the reward function in the task – then we can say that the algorithm is underﬁtting.
Metric 4.2 (Underﬁtting) .
Compute the values of the training TD error, L TD ( θ ) and CQL regularizer, R ( θ ) for the current run and another identical run with increased model capacity.
If the training errors reduce with increasing model capacity, the original run was underﬁtting.
Addressing Overﬁtting and Underﬁtting in Conservative Ofﬂine RL
The typical workﬂow for supervised learning not only identiﬁes overﬁtting and underﬁtting, but alsoguides the practitioner how to adjust their run so as to alleviate the phenomenon, which is expectedto improve the downstream task performance during testing.
For example, overﬁtting can be addressed by applying regularization (e.g., weight decay, dropout) and underﬁtting can be addressedby increasing the model capacity.
Can we devise similar guidelines to address overﬁtting and underﬁtting with conservative ofﬂine RL methods like CQL?
In this section, we provide a practitionerwith some ways to adjust regularization and model capacity to alleviate these phenomena.
Capacity-decreasing regularization for overﬁtting.
As we observed in Section 4, the mechanismbehind extremely low Q-values on the dataset is that CQL training minimizes Q-values on actionssampled from µ ( a | s ) .
Two possible approaches to preventing over-minimization of these values are(1) applying regularization such as dropout [57] on Q-function layers, and (2) enforcing that theinternal representations of the learned Q-function match a pre-speciﬁed representation for all stateaction tuples.
For (2) , we can apply supervised learning techniques such as a variational informationbottleneck (VIB) [58, 59] regularizer on the learned representations, φ ( s ) .
Formally, let ( s , a ) denotea state-action pair.
Instead of predicting a deterministic φ ( s ) ∈ R d (Figure 10), we modify the Qnetwork to predict two distinct vectors, φ m ( s ) ∈ R d and φ Σ ( s ) ∈ R d , and sample φ ( s ) randomlyfrom a Gaussian centered at φ m with covariance φ Σ , i.e., φ ( s ) ∼ N ( φ m ( s ) , diag ( φ Σ ( s )) .
VIB thenregularizes N ( φ m ( s ) , diag ( φ Σ ( s )) to be close to a prior distribution, N (0 , I ) :min θ
[Equation]
[Equation]
Guideline 5.1.
To address overﬁtting, we recommend using some form of capacity-decreasing regularization on the Q-function, such as the VIB regularizer shown in Equation 3.
Capacity-increasing techniques for underﬁtting.
To address underﬁtting, we need to increasemodel capacity to improve optimization of the training objective.
Analogous to supervised learning,model capacity can be increased by using more expressive neural nets (e.g., ResNets [60], transformers [61]) for representing the learned policy.
We use ResNets in our experiments (Figure 10).
However, the RL setting presents an additional challenge with capacity: while larger models inprinciple have more capacity, recent work [62, 54, 55] has shown that utilizing larger networksto represent Q-functions does not always improve its capacity in practice, because TD-based RLmethods introduce an “implicit regularization” effect that can result in aliased (i.e., similar) internalrepresentations for different state-action inputs, even for very large neural networks that can expressthe true Q-function effectively.
To address this issue, these works apply a “capacity-increasing” regularizer to Q-function training.
For instance, we can use the DR3 regularizer [55], which penalizesthe dot product of φ ( s ) and φ ( s (cid:48) ) for a transition ( s , a , s (cid:48) )
∈ D , and hence reduces aliasing.
Thisobjective is given by:
[Equation]
Guideline 5.2.
To address underﬁtting, we recommend using some capacity-increasing regularization on the Q-function and the policy either in conjunction or separately.
Examples: (1) bigger policy networks (e.g., ResNets), (2) DR3 regularizer on the Q-network.
Evaluation of Our Workﬂow Metrics and Protocols in Simulation
[Figure] Next, we empirically validate the workﬂow proposedin Sections 4 and 5 on a suite of simulated roboticmanipulation domains that mimic real-robot scenarios, operating directly from image observations withsparse binary rewards.
We will examine how applyingthe workﬂow in Section 4 to detect overﬁtting or underﬁtting and then utilizing the strategies in Section 5affects the performance of ofﬂine RL methods.
An improved performance would indicate the efﬁcacy of ourworkﬂow in guiding a practitioner in making successful design decisions without any online tuning
[Figure] [Figure]
Experimental setup.
We use the environmentsfrom Singh et al.
[3] to design ofﬂine RL tasks anddatasets that we use for our empirical analysis.
Weconsider two tasks: (1) a pick and place task and (2)a grasping object from a drawer task.
Examples oftrajectories in both of these simulated domains areshown in Figure 2 and are detailed in Appendix C.
Brieﬂy, the pick and place task consists of a 6-DoF
WidowX robot in front of a tray with an object.
Thegoal is to put the object inside the tray.
A non-zeroreward of +1 is provided only when the object hasbeen placed in the box.
The ofﬂine dataset for thistask consists of trajectories that grasp an object witha 35% success and other trajectories that place an object with a 40% success.
Our second task is a grasping from drawer task where the WidowX robot isplaced in front of a drawer and multiple objects.
Therobot can open or close the drawer, grasp objectsfrom inside the drawer or on the table, and placethem anywhere in the scene.
The goal is to close thetop drawer, then open the bottom drawer and take the object out.
Only if the object has been takenout, a reward of +1 is obtained.
The ofﬂine dataset consists of trajectories with a 30-40% successrate for opening and closing a drawer and other trajectories with only 40% placing success.
We useα = 1 . 0 for CQL training in all experiments, wich is directly taken from prior work [3], without anytuning.
However, too low or too high α values will inhibit the effectiveness of our workﬂow, and wediscuss this in App.
G along with ways to tune α to address this. We utilize a standard convolutionalnet for the Q-function (Fig. 11) and the policy.
More details on our setup are in App. C.
[Figure]
Scenario #1: Variable amount of training data.
Our ﬁrst scenario consists of the simulated tasks discussed above with a variable number of trajectoriesin the training data (50, 100, 500, 10000).
We run
CQL and track metrics 4.1 and 4.2 in each case.
Observe in Figure 3 (bottom) that with fewer trajectories, the average dataset Q-value E s , a ∼D [ Q θ ( s , a )]ﬁrst rises, and then drops.
This matches the description of overﬁtting in Section 4.
Observe in Figure 13(left) that, at the same time, the value of the CQLregularizer is very low, which is not consistent withwhat we expect of underﬁtting.
Thus, we can conclude that these conditions exhibit overﬁtting, especially with 50 and 100 trajectories.
The verticaldashed lines indicate the checkpoints that would beselected for evaluation per Guideline 4.1.
We furthervisualize the performance of the chosen checkpointsagainst the actual return of each intermediate policyin Figure 3 (top).
Note that this value is obtained by rolling out the learned policy, and would not be available in a realistic ofﬂine RL setting, but is provided only for analysis.
Selecting the checkpoint based on Guideline 4.1 leads us to select a modelwith close to the peak performance over the training process, validating the efﬁcacy of Guideline 4.1.
Since we detected overﬁtting by following our workﬂow, we now aim to address it by using the
VIB regularizer in the setting with 100 trajectories.
As shown in Figure 13 (right), applying thisregularizer not only alleviates the drop in Q-values after many training steps, but allows us to picklater checkpoints in training which perform better than base CQL on both the tasks.
This validatesthat overﬁtting, as detected via our workﬂow, can be effectively mitigated by decreasing capacity, inthis case by using VIB.
We evaluate dropout, (cid:96) 1 and (cid:96) 2 regularization schemes in Appendix I.
[Figure]
Scenario #2: trainingobjects.
Our second test scenarioconsists of the pick and place task,modiﬁed to include a variable number of object types (1, 5, 10, 20,35).
Handling more objects requires higher capacity, since eachobject has a different shape andappearance.
In each case, CQLis provided with 5000 trajectories.
Following our workﬂow from Section 4, we ﬁrst compute the averagedataset Q-value and the training TDerror.
We observe in Figure 5 that, unlike in Scenario #1, Q-values do not generally decrease whentrained for many steps, suggesting that the Q-function is likely not overﬁtting.
To check for underﬁtting, we visualize the training TD error and ﬁnd that, with 10, 20 and 35 objects, TD errormagnitudes are in the range of [1.0, 2.0], which suggests a overall Q-value error of [30.0,
60.0]since the task horizon is 30.
On an absolute scale, this error magnitude is large: since the rewardsare 0/1, the range of difference between actual Q-values for any two policies is at most 30, whichsuggests that the error magnitude in the runs in Figure 5 are high.
Hence, we conclude that thisscenario generally exhibits underﬁtting with more objects.
Indeed this trend is reﬂected in the policy performance that we plot for analysis in Figure 5: note that the policy return decreases withan increased number of objects, and the policy performance initially increases and saturates at asuboptimal value in the settings that exhibit underﬁtting, consistent with Section 4.
[Figure]
To address underﬁtting detected by our workﬂow in the multi-object case,we apply the proposed capacity-increasing measures to the 35-object task(results for 5, 10 and 20 object settings are in Appendix H).
We use amore expressive ResNet architecture for the policy and the DR3 regularizerfor the Q-function together.
Observe in the ﬁgure on the right that thiscombination (shown in red) improves policy performance in this setting(compared to green), which validates our workﬂow protocol for addressingunderﬁtting.
Metrics for this run are provided in Appendix D.1.
Tuning CQL for Real-World Robotic Manipulation
Having evaluated the efﬁcacy of our proposed workﬂow in simulation, wenow utilize our workﬂow to tune CQL for real-world robotic manipulation.
We test in two setupsthat require the robot to learn from sparse binary rewards and image observations.
The settingsdiffer in robot platform, task speciﬁcation, and dataset size.
Additional results are at the followinganonymous URL: https://sites.google.com/view/offline-rl-workflow . [Figure]
Sawyer manipulation tasks [50].
First, we train a
Sawyer robot in a tabletop setting to perform twotasks: (1) placing the lid onto a pot and (2) opening a drawer.
The robot must perform these tasks inthe presence of visual distractor objects, as shown in
Figure 7.
We directly use the dataset of 100 trajectories for each task collected by Khazatsky et al.
[50]for our experiments so as to mimic the real-worlduse case of leveraging existing data with ofﬂine RL.
We use four-dimensional actions with 3D end-effector velocity control in xyz -space and 1D gripperopen/close action.
More details regarding the setup are provided in Appendix C.
[Figure]
We run default CQL on these tasks and track the average Q-value, TD error, and CQL regularizer value.
As shown in Figure 8, the average Q-value does notdecrease over training, and the TD error (and CQLregularizer shown in Appendix D.2) is large.
Per ourdiscussion in Section 4, this indicates underﬁtting.
Following our guidelines from Section 5, we utilizea more expressive ResNet policy (Figure 10), whichincreases the number of total convolutional layersfrom 3 to 9.
We observe that this reduces the valuesof both the TD error Figure 8 and CQL regularizer(Appendix D.2) on both tasks.
We then evaluate thelearned policy over 12 trials conducted with differentsets of distractor objects, including ones that are unseen during training.
While the policy trainedusing base CQL is unable to successfully complete either task even once attaining a score of 0/12on both tasks, the run that uses ResNet attains a signiﬁcantly better success rate of 9/12 on the putlid on pot task and 8/12 on the drawer opening task, equal to 70.8% success rate on average.0K 100K 200K Gradient steps −−−
A v g .
Q - v a l u e i n D 0 .0 .0 .0 .0 .1 .
Su cce ss r a t e
Policy selection - Real WidowX Q-Values
FinalpolicyselectedPolicysuccessrateDatasetsuccessrate
WidowX pick and place task.
In our second setting, we tune
CQL on a pick and place task with a WidowX 250 robotic arm,shown in Figure 7.
The dataset consists of 200 trajectories collected by running a noisy scripted policy (Appendix C) with 35%success.
We run CQL on this task and track the average Qvalues, which we ﬁnd initially increase and then decrease (Figure 9), indicating overﬁtting.
We then evaluate our policy selection scheme, which in this case suggests deploying checkpoint50, the latest checkpoint with high Q-values.
To see if this checkpoint is effective, we evaluate the performance of a few otherpolicy checkpoints (for analysis only) and plot this performancetrend in Figure 9 as a dashed line.
Observe that indeed the checkpoint found by our approach attains the highest success rate ( 7/9 )compared to other checkpoints, which only succeed ≤ 4/9 times.
This indicates the efﬁcacy of our proposed metrics in identifyingoverﬁtting and our policy selection guideline.
Discussion
While ofﬂine RL algorithms have improved signiﬁcantly over the past years, applying these methodsto real-world robotic domains is still challenging due to little guidance on selecting policies, addingregularization, or modifying model capacity.
In this paper, we devise a workﬂow for conservativeofﬂine RL algorithms such as CQL, which consists of a set of metrics and conditions that can betracked by a practitioner over the course of ofﬂine training as well as a set of recommendations toaddresses the resulting overﬁtting and underﬁtting challenges.
We use our workﬂow to tune CQLon a number of robotic manipulation problems with diverse robot types, multiple objects, and longhorizons, all while learning without online interaction from image observations and sparse rewards.
Both in simulation and the real world, we observe strong performance beneﬁts.
While our proposed workﬂow is an initial step towards practical robotic ofﬂine RL, our work hasa number of limitations.
The validation of our workﬂow is largely empirical in nature.
We believethis is unavoidable, since a workﬂow is a set of guidelines and recommendations, rather than a rigidalgorithm.
Regardless of how theoretically justiﬁed it is, in the end, its value is determined by itsability to produce good results in practice.
Our validation consists of a series of case studies, andalthough the results (as with any empirical ﬁnding) are speciﬁc to the used domains, we believe thebreadth of tasks considered, which consist of two different real-world robots and multiple simulatedtasks, indicates the broad applicability of our approach.
Our workﬂow is also speciﬁc to conservativeofﬂine RL methods, and particularly CQL.
Extending our technique to devise similar workﬂows forother algorithms is an exciting direction for future work.
Deriving theoretical guarantees regardingworkﬂows of this type are also a very important direction for future research.