VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose VRL3, a powerful data-driven framework with a minimalist design forsolving highly challenging visual deep reinforcement learning (DRL) tasks.
Weanalyze a number of major obstacles in taking a data-driven approach, and presenta suite of design principles, novel findings, and critical insights about data-drivenvisual DRL.
Our framework has three stages: in stage 1, we leverage non-RLdatasets (e.g. ImageNet) to learn task-agnostic visual representations; in stage 2,we use offline RL data (e.g. a limited number of expert demonstrations) to convertthe task-agnostic representations into more powerful task-specific representations;in stage 3, we fine-tune the agent with online RL.
On a set of highly challenginghand manipulation tasks with sparse reward and realistic visual inputs, comparedto the previous SOTA, VRL3 achieves an average of 780% better sample efficiency.
And on the hardest task, VRL3 is 1220% more sample efficient and solves thetask with only 10% of the computation.
These highly significant results clearlydemonstrate the great potential of data-driven deep reinforcement learning.
Introduction
Over the past few years, the sample efficiency of Deep Reinforcement Learning (DRL) has significantly improved in popular benchmarks such as Gym [3, 34] and DeepMind Control Suite (DMC)[33].
However, the environments in these benchmarks often look very different from the real world.
It remains unclear whether the sample-efficient methodologies developed for these benchmarks canbe successfully extended to more practical tasks that rely on realistic visual inputs from cameras andsensors, such as in robotic control and autonomous driving.
A promising direction is to take a data-driven approach, that is, try to use all the data we can findthat might contribute to the learning process.
Indeed, in the past few decades, the most importantadvances in deep learning have been facilitated by the use of large amounts of data and computation[32, 20].
Such a data-driven approach is very natural and mature for supervised learning, but not asmuch so for RL, where learning is most commonly performed in an online fashion.
Although recent advances in representation learning for RL and offline RL reveal the potential of suchan approach, most existing offline RL methods focus on tasks with proprioceptive (“raw-state”) inputand not the highly challenging visual control tasks.
There is yet to be a method that can fully utilizedata from multiple different sources to boost learning efficiency.
In this work, we aim to develop adata-driven framework that fills this role.
We want it to be very sample efficient, and we want to takea minimalist approach and make this framework as simple as possible.
This is very important since in DRL, the performance of RL algorithms can be heavily affected by hidden details in the code, andas new algorithms become more complex, they can be challenging to analyze, reproduce and tunetowards new tasks [10, 11, 6].
In later sections of the framework simple intuitive, and effective.
Figure 1 shows the proposed framework, Visual Deep Reinforcement Learning in 3 Stages (VRL3).
VRL3 is designed with a data-driven mindset, and in a minimalist fashion, Each stage is designed toutilize a particular type of data source.
In stage 1, we learn from large, existing non-RL datasets such as the ImageNet dataset.
These datasetsmight not be directly relevant to the RL task, but they provide task-agnostic visual representationsthat might be helpful, especially if the RL task also has realistic visuals.
A major challenge here isthat the non-RL data can come from a different domain with different input shapes.
Special care isrequired to make sure the representations obtained here can be smoothly transferred to the RL task.
In stage 2, we learn from offline RL data, such as a small number of expert demonstrations, ordata collected by a previous learning agent.
This offline data is related to the RL task and containsimportant information about solving the task.
We initialize an actor-critic RL agent with the pretrainedencoder and then employ offline RL techniques which not only train the actor-critic RL agent butalso fine-tune the encoder, thereby creating a task-specific representation.
In stage 3, we learn from online RL data.
Our main goal is sample efficiency (achieve high performance with as little online data as possible).
We, therefore, use off-policy training, which uses andre-uses all available data during training.
Note that naively transitioning into standard online RLcan lead to training instability and destroy what we learned from the previous stages.
We avoid thisinstability by constraining the maximum Q target value during off-policy updates.
Although each stage brings a number of unique challenges, we show that VRL3 can address all ofthem and take full advantage of the different data sources.
Our contributions are as follows:
Novel framework: we present a data-driven framework that can fully exploit non-RL, offlineRL, and online RL data to solve challenging tasks with sparse reward and realistic visualinput.
Most existing visual methods do not utilize offline data, and most offline RL methodsdo not work with difficult visual tasks, making our framework a novel contribution.
Weprovide source code (in supplementary materials) and a full set of technical details tomaximize reproducibility.2.
Novel technical contributions: we propose convolutional channel expansion (CCE) to enablean ImageNet-pretrained encoder to work with RL tasks that take in multiple input images.
We also propose the Safe Q target technique for minimal-effort offline to online transition.
Novel insights: we show that offline RL data can provide a benefit that cannot be replacedby other stages, and that conservative RL might be a superior option when exploiting offlinedata, compared to contrastive learning.
We also show that a reduced encoder learning ratehelps learning in a multi-stage setting.
And very interestingly, representation pretrainedwith non-RL data, despite the large domain gap, can give better performance compared torepresentation learned from limited RL data, but is outperformed when RL data is ample.
We also find stage 1 pretraining helps more in more difficult tasks.4.
Novel results: We present extensive ablations that reveal the effect of each componentof VRL3, and conduct a large hyperparameter study on tasks with different difficulties,providing useful guidelines on how to deploy and tune our framework to new tasks.5.
Significant improvement over the previous SOTA: VRL3 achieves an entirely new level of
SOTA sample efficiency, parameter efficiency, and computation efficiency on the highlychallenging Adroit benchmark while also being very competitive on DMC.
Related Work
VRL3 follows a minimalist design and effectively combines representation learning, offline RL, andonline RL.
We now discuss related work and how our contribution is novel and different from priorworks, with a focus on model-free methods.
Additional related work is discussed in Appendix D.
2.1 Learning visual tasks with online RL
With the help of data augmentation, many visual RL tasks can be learned in an online fashion.
Typically, agents trained with such methods use convolutional encoders to extract lower-dimensionalfeatures from the visual input, and train in an end-to-end fashion [14, 39, 17, 40].
Another classof methods uses contrastive representation learning to solve visual tasks [18, 31, 43].
Contrastivelearning can be used to learn useful representations without gradients from the RL objectives.
Thesemethods work well in environments with simple visuals, such as in the DMC environments, butcan fail entirely if naively applied to highly challenging control tasks with sparse rewards andmore realistic visuals such as Adroit, due to the difficulty in both exploration [27] and learningrepresentations for realistic visual inputs [30].
Different from these works, VRL3 can fully exploitoffline RL data and even non-RL data and can achieve SOTA performance in both Adroit and DMC.
2.2 RL with ImageNet pretraining
A number of previous works have used a pretrained encoder, such as a ResNet trained on ImageNetto help convert visual input into low-dimensional features, the encoder can be further finetuned orkept frozen during training [30, 5, 12, 9, 21, 26, 29].
However, such pretraining alone might not besufficient to achieve the best performance or computation efficiency.
Naively initializing the agentwith an ImageNet-pretrained encoder might not provide a significant advantage over training fromscratch [12].
Most of these works do not incorporate offline RL data.
RRL [30] uses offline RL databut does not exploit them to the full extent.
Different from these works, we show how ImageNetpretraining can be best combined with offline and online RL and provide a series of novel analyses,interesting results, and important insights.
2.3 Offline RL and data-driven RL
In the past few decades, some of the most important advances in AI research have been based on datadriven methods, so it is natural to ask whether we can also utilize a similar mindset for DRL [20, 32].
Although off-policy algorithms do not naively work in offline settings [7, 22], many specializedalgorithms are proposed to allow effective offline training [7, 16, 4, 25, 36, 15, 6, 35, 13, 38, 1, 8, 37].
Recent works also study the transition from offline to online [23, 19, 44, 19, 13, 22].
However, mostof these works are tested on tasks with proprioceptive (positions & velocities) input, and they donot consider pretraining from non-RL data.
Our work is very different and is focused on highlychallenging control tasks with visual input, sparse reward, and limited demonstrations.
Note that Adroit has often been studied with proprioceptive and not the more challenging visual input.
RRL [30] and FERM [42] are the most relevant to our work.
RRL uses an ImageNet-pretrained encoder, performs behavioral cloning (BC) on offline RL data, and then performs on-policy learning.
Prior to our work, RRL is the SOTA on visual Adroit and is the only method that achieves non-trivialperformance on the hardest Relocate task.
FERM performs contrastive learning on offline RL dataand then performs off-policy online RL with data augmentation.
FERM can learn simple robotictasks quickly but has brittle performance in Adroit [30].
In most prior works, the different training stages have been studied separately .
What is missing is aframework that can combine all 3 stages effectively and efficiently .
A major novel contribution ofthis work is such a framework and a comprehensive study on how its components interact and buildup towards superior performance.
VRL3: Visual DRL in 3 stages
We now present the design of our framework VRL3.
Algorithm 1 gives the pseudocode of VRL3 .
We focus on the high-level ideas and provide additional technical details in Appendix A. Our agenthas a convolutional encoder, two Q networks, two target Q networks and a policy network.
Note theencoder can be trained separately from the policy and Q networks.
updates do 11:
Sample an action from policy, take action and store the new data in D 12:
Sample a mini-batch B from D 13: Update f ξ and Q θ 1 , Q θ 2 with L Q , update π ϕ with L π . Update Q ¯ θ 1 , Q ¯ θ 2 with polyak averaging. Stage 1 We follow the standard supervised learning routine and train a convolutional encoder f ξ ona 1000-class ImageNet classification task.
We follow the standard training procedure of a typical
ResNet model.
During training, we shrink the images to the size of 84x84 so that they match theinput dimension of our downstream RL task.
In our main results, we also use a lightweight encoderwith 5 convolutional layers, with batch norm after each layer.
Transition to multi-frame RL input After stage 1, we expand the first convolutional layer of theencoder so the input channel size matches the RL task which can take in multiple frames, and theweight values in this layer are divided accordingly to maintain features to later layers, we refer to thisas convolutional channel expansion (CCE) (Further discussed in Section 5).
Stage 2 We first initialize an RL agent with the pretrained encoder and put the offline RL datain an empty replay buffer D .
We build our implementation on top of DrQv2 because it is a simpleoff-policy actor-critic algorithm with SOTA performance and a very clean and efficient codebase [39].
Here we also try to follow the notations in DrQv2 for better consistency.
Input and image augmentation Let aug denote random shift image augmentation, and cat forconcatenation.
Let x be the visual input from the camera (or cameras).
In the Adroit environment, inaddition to the visual observation, we are also given sensor values from the robotic hand, which isa vector of real numbers, denoted by z .
The augmented visual input (image frames) is put into theencoder, giving a lower-dimensional representation h = f ξ (aug( x )) .
We then concatenate it with the sensor values (this is skipped in DMC).
For simplicity, let s = cat
( h , z ) denote the concatenation; sis the input to the Q and policy networks, which are multi-layer perceptrons.
Offline
RL update
For each update, we sample a mini-batch of transitions( x t , z t , a t , r t : t + n − 1 , x t + n , z t + n ) from the replay buffer D .
Note we have t + n here tocompute n-step returns [39].
Let s t = cat ( h t , z t ) .
Let ˜ a t ∼ π ϕ ( s t ) be actions sampled from thecurrent policy.
The Q target value y is: y = (cid:80) n − 1 i =
γ i r t
+ i + γ n min k =1 , 2 Q ¯ θ k
( s t , ˜ a t + n ) .
The standard Q loss L Q is (for k = 1 , 2 ): L Q ( D ) =
E τ ∼D (cid:2) ( Q θ k ( h t , a t ) − y ) 2 (cid:3) .
Following [16], and build on top of our framework, we use an additional conservative Q loss L QCterm:
L QC ( D ) =
E τ ∼D (cid:2) log (cid:80) ˜ a t exp( Q ( s t , ˜ a t ))
− Q ( s t , a t ) (cid:3) .
On a high-level, this loss essentiallyreduces the Q value for actions proposed by the current policy, and increase the Q values for actionsin the replay buffer [16].
The standard policy (actor) loss L
[Equation]
is:
[Equation] =
[Equation]
Let α be the learning rate for the policy network and the Q networks.
Let β enc be the encoderlearning rate scale, so that the encoder learning rate is α enc =
β enc α .
For each offline RL updatein stage 2, we perform gradient descent on L Q ( D ) + L QC
( D ) to update the encoder f ξ and the Qnetworks Q θ 1 , Q θ 2 , and gradient descent on L π to update the policy π ϕ .
Note that we do not use thegradient from the policy to update the encoder, following previous work [40, 39].
We also use Polyakaveraging hyperparameter τ to update target networks, as is typically done.
Safe Q target technique When computing the Q target in stages 2 and 3, we propose a simpleyet effective solution to prevent potential Q divergence due to training instability and distributionshift.
For all tasks, we set a maximum Q target value.
To decide this threshold value, we reshapethe maximum per-step reward r max to be 1 and estimate what the maximum Q value should befor an optimal policy.
To be concrete, we can compute the sum of this infinite geometric series,
Q max = (cid:80) ∞ t =0 γ t r max = 100 when γ = 0 .
99 ( γ is discount factor).
To allow more flexibility in the Q network, we can also set the threshold Q max to be higher than this value. Now if during any Qupdate, a Q target value y exceeds this threshold value, then we reduce it to be closer to the thresholdvalue: if y > Q max + 1 , then y ← Q max + ( y − Q max ) η , where 0 ≤ η ≤ 1 is the safe Q factor.
Notice the Q max + 1 here is a technical detail to address edge cases.
Reduced encoder learning rate Due to the domain gap between ImageNet and the RL task, we applya reduced encoder learning rate to avoid the disturbance of pretrained features in stages 2 and 3.
Stage 3 We continue to use the replay buffer D , which already contains the offline RL data.
Wenow perform standard online RL, and newly collected data are added to D .
The update is the same asin stage 2, except we remove the conservative Q loss L QC ( D ) .
Results
Figure 2a shows success rate comparison averaged over all 4 Adroit tasks, Figure 2b shows results onthe most challenging Relocate task, and Figure 2c shows average performance comparison on all24 DMC tasks.
We use the following methods as our baselines.
DrQv2 is an off-policy online RLmethod that has the SOTA performance on DMC.
RRL is an on-policy method that uses pretrained
ImageNet features and utilizes demonstrations with behavioral cloning and has SOTA performanceon Adroit.
FERM is a 2-stage method that first trains an encoder with contrastive learning on offline
RL data and then perform online RL.
Note that currently there are not many successful multi-stagevisual RL frameworks like VRL3, and as discussed in section 2, most existing offline-online methodsare not designed to work with the more challenging visual inputs, so they are not included.
Getting stronger baselines Naively, DrQv2 does not use offline data and does not work well in
Adroit due to the difficulty in exploration.
And FERM has brittle performance on Adroit [30].
Tocreate stronger baselines, we introduce variants of DrQv2 and FERM, which are both built on topof our framework.
Specifically, DrQv2fD(VRL3) does not have stage 2 training but starts stage 3training with demonstrations in the buffer.
FERM(VRL3) is our re-implementation based on VRL3and has much stronger performance compared to the results in [30] (more robust, is about 2 timesfaster, and reaches a 1.25x average success rate).
For Adroit, we also compare with FERM+S1, whichis FERM(VRL3) plus S1 pretraining, and VRL3+FERM, which is VRL3 with additional contrastive [Figure] encoder learning in stage 2.
For these improved baselines, we use the hyperparameters we searched for VRL3 for maximum performance.
Doing so helps to ensure our comparison actually reflects thedifferences in algorithmic design and not hidden implementation details [6, 10].
Figure 2a shows that interestingly, for Adroit, both FERM(VRL3) and DrQv2fD(VRL3) outperform
RRL in the early stage, showing that with the right implementation, these off-policy agents can learnquickly even without stage 1 representations.
Figure 2b shows that on the most challenging Relocatetask, DrQv2fD and FERM variants fail to learn entirely because they do not exploit stage 2 datasufficiently.
VRL3 greatly outperforms all other methods.
Significantly stronger SOTA sample efficiency on Adroit: Compared to RRL, VRL3 achieves anaverage of 780% better sample efficiency over all tasks and is 1220% faster to solve the most difficult
Relocate task (in terms of the data required to reach 90% success rate).
VRL3+FERM has the sameperformance as VRL3, showing that additional contrastive learning is not necessary.
FERM+S1is stronger than FERM, showing that stage 1 pretraining also works well with contrastive stage [-
 [Figure] learning.-]2learning.
For parameter and computation efficiency, we also use a much smaller encoder compared to
RRL.
We are 50 times more parameter efficient in the encoder and 3 times more parameter efficientfor all networks in the agent.
VRL3 is also more computationally efficient, in fact, VRL3 can solvethe hardest task with just 10% of the computation (Detailed comparison in Appendix B).
For DMC, we collect 25K data with fully trained DrQv2 agents to enable stage 2 training.
FERM(VRL3) and DrQv2fD are both good, and VRL3 is slightly better, all three are better than
DrQv2 (results from authors).
Note that DMC is typically treated as an online RL benchmark, and DrQv2 is an online method, so we do not claim a new SOTA.
These results are only to show that VRL3 can work in this popular benchmark and can utilize offline data to learn faster.
Compared to
DMC, Adroit is more challenging due to its sparse reward setting and more complex, more realisticvisuals with more depth, lighting, and texture information.
This might explain the larger performancegap in Adroit.
This shows the advantage of VRL3 is more evident in more challenging visual tasks.
Challenges, Design Decisions, and Insights
In addition to the very strong performance on Adroit, another major advantage of VRL3 is itssimplicity.
The minimalist design of VRL3 not only makes it much faster and more lightweightthan the previous SOTA but also allows us to easily ablate and study its core components.
In thissection, we discuss some technical challenges and provide a comprehensive analysis of how thedifferent components of VRL3 work together towards superior performance.
We focus on Adroit andall experiment results are averaged over 4 Adroit tasks (per-task plots in Appendix).
5.1 Stage 1: Pretraining with non-RL data
The first challenge we face in stage 1 is input shape mismatch.
In stage 1, the input to the encoder isa single image with RGB channels.
For the downstream RL task, however, it can be beneficial for the agent to take in multiple consecutive frames of images as input, or frames from multiple differentcameras (sometimes even a combination of both), so that the agent can learn to use temporal andspatial information in these frames for better performance.
A naive solution is to put each frame separately into the pretrained encoder, as is done in [30].
Thisnaive approach has two downsides: 1) it requires more computation, and 2) even if we finetune theencoder, it cannot learn important temporal and spatial features from the interaction of frames.
Advantage of convolutional channel expansion (CCE) : We propose the CCE technique to addressthe above issue.
Let m be the number of input frames for the RL task, With CCE, we expand the firstconvolutional layer of the stage 1 encoder so that it can now take in inputs of 3 m channels instead of3.
We repeat the weight matrix m times and then scale all the weight values to 1 /m of their originalvalues.
This allows us to modify the input shape without disrupting the learned representations.
Sucha very simple method can effectively solve the mismatch problem when transitioning a visual encoderfrom the non-RL pretraining stage to the RL stages.
To enable faster experimentation and avoid difficulties in training deep networks in DRL [2, 24], weuse a lightweight encoder with 5 convolutional layers, which is 50 times smaller than the ResNet34used in RRL.
Quite surprisingly, VRL3 can achieve superior performance with such a tiny encoder.
5.2 Stage 2 and Stage 3
In stage 2, why use conservative RL updates instead of BC or contrastive representationlearning?
We use conservative RL for the following reasons: a) we can finetune the encoder toget task-specific representations with RL signal b) comparing with contrastive learning methodswhich only train the encoder, we can train the encoder as well as the policy and Q networks, giving ajump-start in stage 3.
c)
With the safe Q technique, this also brings us a minimal-effort design thatallows us to transition effectively from stage 2 to stage 3.
We can simply remove the conservativeterm without any elaborate modifications.
d) this fits well with an off-policy backbone, allowingmaximum data reuse.
(BC such as in [27] only works with on-policy methods).
Next, we discussextensive empirical evidence that supports our design.
Figure 3a shows that for VRL3, only using BC on expert demonstrations in stage 2 does not give agood performance. In fact, using BC in stage 2 gives a similar performance to when stage 2 is entirelydisabled.
This problem is caused by a distribution mismatch of actor and critic at the beginningof stage 3 since BC only trains the actor and not the critic.
For contrastive learning, we observefrom Figure 2 that FERM has slightly worse performance in DMC and much worse performance in Adroit.
This is because Adroit has a more challenging sparse reward setting and FERM fails to fullyexploit offline RL data by only updating the encoder.
These results show that when tackling highlychallenging tasks, learning only the representation in stage 2 can be useful, but is not enough tofully exploit offline RL data .
This is a novel and important finding that has never been carefullystudied or emphasized in the literature.
[Figure]
Figure 3: a) Effect of BC and offline RL updates in stage 2. Only applying BC or disabling stage 2 training entirely leads to poor performance.
b) and c) Taking naive RL updates in stage 2 (S
Naive) leads to severe overestimation, which can be mitigated by safe Q (S2 Naive Safe).
Taking conservative updates in stage 3 (S3 Cons) leads to underestimation.
VRL3 gives the best performance with conservative update only in stage 2 and safe Q. Note in our setting, the maximum reasonable
Q value should be around 100.
d) Effect of different encoder learning rates (e.g., 0.01 means the encoder’s learning rate is 100 times slower than that for the policy and Q networks.)
Effect of safe Q target for stage 2-3 transition: Special care is required to effectively performoff-policy learning in stage 2.
In VRL3, we perform conservative updates in stage 2 and remove theconservative term in stage 3.
When we transition from stage 2 to stage 3, the new online data willcause a distribution shift that can lead to instability in Q values [23], The safe Q target technique is avery simple, efficient, and minimalist solution to this issue.
Figure 3b and 3c show the effect of thesedesign decisions: updating with naive RL in stage 2 (S2 Naive) or removing the safe Q technique(No safe Q) lead to severe overestimation in stage 3 and cause performance drop.
Updating withconservative RL in stage 3 (S3 Cons) leads to severe underestimation and even worse performance.
Naive RL in stage 2 with safe Q in stages 2 and 3 (S2 Naive Safe) can mitigate overestimation but isstill weaker than VRL3, which takes conservative updates in stage 2 and use safe Q in stages 2 and 3.
Reduced encoder learning rate for better finetuning: When we fine-tune the encoder in stages 2and 3, a reduced encoder learning rate can prevent pretrained representations from getting destroyedby noisy RL signals, especially in the early stage of RL training.
This problem has also been observedin other environments [28].
Figure 3d shows the effect of encoder learning rates.
These resultsshow that even though there is a large domain gap between ImageNet and RL tasks, a frozen stage 1encoder can still provide useful features, and finetuning the encoder can further improve performance.
Although it is not new that transfer + finetuning can give the best result [41], it has not been adequatelystudied in the RL setting with large domain gap and noisy signals.
5.3 Contribution of each stage
VRL3 achieves very strong performance by fully exploiting data from 3 different stages.
We nowtry to understand how much each stage contributes to the final performance.
We perform 2 sets ofcomparisons, for each VRL3 variant in Figure 4a, we disable one or more stages of training entirely(e.g.
S13 means stage 2 training is removed), and for Figure 4b, we disable the encoder training inone or more stages, but always maintain the updates to actor and critic networks (e.g. S12 meansencoder is frozen in stage 3).
[Figure]
Here we summarize a number of insights we can obtain from these results.
In Figure 4a: Stage 1pretraining improves sample efficiency (S123 > S23); stage 2 is critical and its contribution cannotbe replaced by the other stages (S123> S13, S23 > S13).
Note that since Adroit is very challengingwith limited offline data, S3 is always enabled to obtain non-trivial performance.
In Figure 4b: when the encoder is only trained in one stage, stage 3 encoder training gives a muchstronger performance than the other two, showing the importance of task-specific representations (S3> S1 and S2).
Note that very interestingly, although task-specific features are more useful, when RL data is scarce, non-RL pretraining provides better results despite the large domain gap (S3> S1 > S2).
Training the encoder in more stages always leads to improved performance (S123 > S12 > S1).
An important finding is that our results indicate offline RL is more beneficial to just contrastiverepresentation learning in stage 2 , note that S3 in Figure 4b (perform offline RL update but freezeencoder in stage 2, encoder trained in stage 3) has much stronger performance than S3 in Figure 4a(no stage 2 training at all).
This is consistent with results in Figure 2, the performance gap between
VRL3 and FERM+S1 is caused by VRL3 doing offline RL updates in stage 2, and thus learning a fullagent, instead of just the encoder features.
With the above insights, Table 1 summarizes fundamental differences between VRL3 and some otherpopular visual RL methods.[-5.4 Additional Studies and Analysis We also perform an extensive study on hyperparameter robustness and sensitivity on each of the 
 Adroit tasks (with varying difficulty).-]
[Table]
✓
[Table]
✓ ✓ ✓ ✗ ✓
[Table]
✗ ✗ ✓ ✗ ✓
[Table]
✗ ✗ ✗ ✓ ✓
[Table]
✓ ✓ ✓ ✗ ✓
[Table]
✗ ✗ ✗ ✗ ✓ 5.4 Additional Studies and Analysis
We also perform a very extensive study on hyperparameter robustness and sensitivity on each of the4 Adroit tasks (with varying difficulty).
Due to limited space, we only show a small portion of ourstudy in Figure 5 and put the rest of the results in Appendix A.
This provides additional insights andhelps us understand what hyperparameters should be tuned with high priority when applied to a newtask, for example, results show that S1 pretraining is useful for harder tasks but not for the simplest
Door task; data augmentation is always critical; the encoder learning rate is important and should betuned in difficult tasks; even a small number of S2 offline updates can greatly help learning.
[Table]
Discussion, Limitations and Future Work
In this paper, we propose a 3-stage data-driven framework for solving challenging visual controltasks.
We perform extensive experimentation to ablate and analyze our framework to identify themost important technical components.
We arrive at a minimalist design that is very simple, intuitive,and much more sample efficient and computationally efficient than the previous SOTA on Adroit.
Although this is a highly significant result, we do have some limitations: for example, Adroit has morerealistic visuals compared to DMC, but it is still a simulated benchmark.
Experiments on roboticshardware are required to understand how VRL3 should be deployed in the real world.
Anotherlimitation is that it remains unclear whether stage 1 training is still beneficial if the domain gapbetween non-RL data and RL data is too great (RL task visuals very different from the real world).
These can be exciting future research topics.
Despite these limitations, VRL3 brings a number ofnovel and interesting insights, in addition to the very strong empirical results.
We believe VRL3 is animportant step toward practical data-driven DRL for real world visual control tasks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] A discussion is provided insection 6.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] A discussion on potential negative societal impacts is provided in Appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide source code in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We discuss major technical details in the main paper, with additional technical details in Appendix A and C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] The plots are averaged over multiple runs (some are also over multiple environments), shaded area indicate the standard deviation across runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] They are discussed in detail in
Appendix B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the authors
of the code (DrQv2)
and testing environments (DMC, Adroit, MuJoCo) we use.
(b) Did you mention the license of the assets?
[Yes] A discussion is provided in Appendix
C.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]Our source code is provided in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[-Hyperparameters that affect both stage 2 and 
-]
[-On a single NVIDIA V100 GPU, for door, RRL runs at 
-]
[N/A]