Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?
Anonymous Author(s)
AffiliationAddress
Abstract
We investigate whether self-supervised learning (SSL) can improve online reinforcement learning (RL) from pixels.
We extend the contrastive reinforcementlearning framework (e.g., CURL) that jointly optimizes SSL and RL losses and conduct an extensive amount of experiments with various self-supervised losses.
Ourobservations suggest that the existing SSL framework for RL fails to bring meaningful improvement over the baselines only taking advantage of image augmentationwhen the same amount of data and augmentation is used.
We further perform anevolutionary search to find the optimal combination of multiple self-supervisedlosses for RL, but find that even such a loss combination fails to meaningfullyoutperform the methods that only utilize carefully designed image augmentations.
Often, the use of self-supervised losses under the existing framework lowered RLperformances.
We evaluate the approach in multiple different environments including a real-world robot environment, and confirm that no single self-supervisedloss or image augmentation method can dominate all environments and that thecurrent framework for joint optimization of SSL and RL is limited.
Finally, weempirically investigate the pretraining framework for SSL + RL and the propertiesof representations learned with different approaches.
Introduction
Learning to act from image observations is crucial in many real-world applications.
One popularapproach is online reinforcement learning (RL), which requires no human demonstration or experttrajectories.
Since all training samples are collected by the agent during policy learning in online RL,the collected data often has strong correlations and high variance, challenging the policy learning.
Meanwhile, the cost of interacting with environments requires the RL algorithms to have highersample efficiency.
Compared to RL using state-based features, pixel-based RL continuously takesimages as inputs, which usually come with a much higher dimensionality than numerical states.
Suchproperties pose serious challenges to image representation learning in RL.
Recent work tackles such problems mainly from two different directions. (1) Inspired by the greatsuccess of self-supervised learning (SSL) with images and videos (e.g., [4, 5, 7, 9, 13, 14, 16, 20,29, 30, 35, 38, 48, 50, 51, 54, 63]), some RL methods (e.g., CURL [40] and SAC+AE [72]) takeadvantage of self-supervised learning.
This is typically done by applying both self-supervised lossand reinforcement learning loss in one batch.
In this paper, we dub such joint optimization ofthe self-supervised loss and the RL loss as the Joint Learning Framework .
On the other hand,approaches such as RAD [41] and DrQ [73, 74] investigate how online RL can take advantage ofimage augmentations.
They show significant improvements by applying relatively simple imageaugmentations to observations of RL agents.
Our objective is to study how well a single or combination of self-supervised losses and augmentationswork under the joint learning framework, and to empirically identify their impact in RL systems.
In this paper, we extend the existing joint (SSL + RL) learning framework, conduct experimentscomparing multiple self-supervised losses with augmentations, and empirically test them in manyenvironments from different benchmarks.
We confirm that a single self-supervised loss undersuch joint learning framework typically fails to bring meaningful improvements to existing imageaugmentation-based methods.
We also computationally search for a better combination of lossesand image augmentations for RL with the joint learning framework.
The experiments in differentenvironments and tasks show inconsistency in self-supervised learning’s capability to improvereinforcement learning.
With all our findings, we present this work as a thorough reference for investigating better frameworksand losses for SSL + RL and inspiring future research.
Our contributions can be summarized asfollows:
We conduct an extensive comparison of various self-supervised losses with the existing
joint learning framework for pixel-based reinforcement learning in many environments fromdifferent benchmarks, including one real-world environment.
We demonstrate the propertiesof representations learned by different SSL-based methods and image augmentation-basedmethods.2.
We perform evolutionary search for the optimal combination of multiple self-supervisedlosses and the magnitudes of image augmentation, and confirm its limits.3.
We empirically study a pretraining framework for combining SSL with RL as an alternativeof the joint learning framework.
Preliminaries2.1 Reinforcement Learning
In this paper, we extend the configurations of previous work [40, 72] and exploit SAC ( S oft A ctor
C ritic) [24, 25] and Rainbow DQN [31] for the environments with continuous action space anddiscrete action space respectively.
Soft Actor Critic is an off-policy actor-critic algorithm that takes advantage of the maximum entropyto encourage the agent to explore more states during the training.
It maintains a policy network π ψand two critic networks Q ϕ 1 and Q ϕ 2 .
The goal of π ψ is to maximize the expected sum of rewardsand a γ -discounted entropy simultaneously, where the entropy encourages the agent to explore duringlearning.
Rainbow DQN is a variant of DQN [47] with a bag of improvements such as double Q-learning [28,64 ], prioritized sampling [55], noisy net [18], distributional RL [3], dueling networks [69] and multistep reward.
2.2 Pairwise Learning
We coin the term “Pairwise” Learning for the frameworks that learn visual representations based onsemantic invariance between dual-stream encoder representations.
A general pairwise learning methodfirst generates multiple augmented views by applying a series of random image augmentations to theinput sample, then clusters views with the same semantics in the representation space.
Optionally insuch frameworks, methods using contrastive losses repel samples with different semantics.
In thispaper, we focus on four representative pairwise learning methods, MoCo [10, 11, 29], BYOL [21],
SimSiam [9] and DINO [5].
We have a detailed explanation and comparison on these methods at
Sec. A.1.
2.3 Representation Learning for Pixel-based RL
Previous works explore the possibility of learning better visual representation which may finallybenefit the policy learning.
One direction is using image augmentation for policy learning, where RAD [41] and DrQ [73] achieve significant performance using simple image augmentation.
Another direction is to combine SSL with Pixel-based RL, in which two representative methods are
SAC+AE [72] and CURL [40].
RAD ( R einforcement Learning with A ugmented D ata) [41] investigates the impact of differenttypes of image augmentations for both image and state inputs.
By applying random translation orrandom crop to the input image, RAD significantly improves data-efficiency solely through image
[Figure]
augmentation without any auxiliary losses.
DrQ ( D ata- r egularized Q ) [73] further investigates the possibilities of utilizing image augmentation.
DrQ applies image augmentation twice on the input images, and averages the Q value over twoaugmented images which is assigned as the Q value of the input images.
SAC+AE [72] takes advantage of a RAE (deterministic R egularized A uto E ncoder)
[19], in replacement of β − VAE Higgins et al.
[32] in SAC backbone to improve learning stability.
The RAE isjointly trained with SAC by performing both SAC update and RAE update alternatingly in one batch.
CURL ( C ontrastive U nsupervised Representations for R einforcement
L earning)
[40] combinescontrastive learning with an online RL algorithm by introducing an additional contrastive learninghead at the end of the image encoder.
Similar to the aforementioned SAC+AE, here the contrastiveloss and reinforcement learning loss are applied alternatively at training.
Self-supervision for Reinforcement Learning
[Figure] To effectively evaluate different self-supervised losses, we extend the joint learning framework [Figure] used in SAC+AE [72] and CURL [40] by replacing its contrastive learning head with a generalself-supervised learning head.
We keep the same backbone using SAC in tasks with continuous actionspace and using Rainbow DQN in tasks with discrete action space.
3.1 General Joint Learning Framework
[Figure]
With SAC Fig.
1a shows a general joint learning framework with SAC backbone.
The unmodified SAC backbone contains two image encoders, online encoder f q , target (or momentum) encoder f kand an actor head A p .
Each encoder is followed by two critic heads.
Besides that, we attach anadditional self-supervised head g q after the online encoder.
For some pairwise learning losses, weconcatenate a momentum SSL head g k after the target encoder.
For every sampled batch of transitions, we first apply image augmentation to both the current state sand the next state s ′ , and update the SAC model ( f q , Q i =1 , 2 q , A p ) using the augmented images.
Notethat for stability concerns, we do not update the parameters of the image encoder when updating theactor head A p .
Then, the target networks are updated by Exponential Moving Average (EMA).
Thisis followed by also performing an EMA update of the SSL head if required.
And finally the onlineencoder f q and the self-supervised head g q are updated by the self-supervised loss.
By alternatinglyperforming RL and SSL in every batch, we jointly train all the components in the framework.
Thepseudo code of SAC update alternating RL and SSL is provided in Algorithm 1.
With Rainbow DQN Fig.
1b demonstrates how to jointly apply SSL to Rainbow DQN backbone.
The unmodified Rainbow DQN backbone maintains an online encoder f q and a target encoder f ′ ,followed by two state value heads Q q and Q ′ .
We introduce an additional momentum encoder f k andself-supervised heads g q and g k as suggested in CURL.
For each batch, the self-supervised losses arecomputed using augmented images, while the RL loss is computed using the original data.
Finally,the online encoder f q and the self-supervised head g q are updated by the self-supervised loss.
Thepseudo code of Rainbow DQN update can be found at Algorithm 2.
[Table]
[Table]
3.2 Losses for Self-supervised Learning
[Table]
The self-supervised losses we investigated can be categorized into four classes: pairwise learning,transformation awareness, reconstruction, and reinforcement learning context prediction.
Pairwise Learning We investigate three representative pairwise learning methods: BYOL [21],
DINO [5] and SimSiam [9], along with existing CURL whose framework is similar to MoCo [29].
BYOL, DINO and SimSiam only explicitly pull positive samples closer without the need of largenumber of negative samples.
CURL uses a contrastive loss taking both positive and negative samplesinto consideration.
Given the general joint learning framework described in Sec. 3.1, by substituting the self-supervisedhead and loss, we can easily formulate different agents w.r.t.
self-supervised losses.
For BYOL, asshown in Fig.
9a, a projector and a predictor are appended to the online encoder sequentially. Andan momentum projector is attached to the end of the target/momentum encoder.
DINO (Fig.
9c)maintains only projector in both online and target branches.
The momentum projector is alsoupdated by EMA.
The two encoders in BYOL and DINO operates on two augmented views of thedata respectively whereas SimSiam (see Fig. 9b), uses only the online network and a projector forprocessing both the augmented views.
Transformation Awareness
Recent works (e.g., [12, 20, 34, 37, 43, 48]) have shown that theawareness of transformations (like rotation, Jigsaw puzzle and temporal ordering) improves manydownstream tasks in computer vision like image classification and action recognition.
Typically suchawareness can be acquired by explicitly asking a classifier to identify the applied transformationfrom the pixel representation.
Therefore, we investigate two simple classification losses, rotationclassification ( RotationCLS ) and shuffle classification ( ShuffleCLS ), and set a two-layer MLP classifieras the self-supervised head in the joint learning framework.
RotationCLS represents the methods that encourage spatial transformation awareness.
Inspired by
RotNet [20] and E-SSL [12], we rotate the input image after augmentation by 0°, 90°, 180°and270°.
The classifier predicts the rotation angle from the visual representation and it is trained bycross-entropy loss.
Shuffle Tuple [46] encourages the encoder to develop awareness of action causality by predicting iftwo frames appear in order.
We adapt Shuffle Tuple by randomly shuffling the current state image
[Figure]
Input
InputOutput OutputInput
Output Output
[Figure]
OutputOutput -
Input Input
Output Output
Input Reward rOutput
Input
InputOutput
Output
Input
Output and next state image in a state transition tuple and predicting whether it is shuffled or not.
Theclassifier also takes action into consideration because some of the transitions are reversible.
Theoverall architecture of ShuffleCLS is shown as Fig.
Reconstruction Reconstructing the input image with an hourglass architecture has been shownto be an effective way to learn image representation [19, 32, 38].
We simply extend SAC+AE bychanging the input and reconstruction target to be augmented images.
The reconstruction loss andregularization from RAE [19] are left untouched.
Recent study on M asked A uto E ncoder [30] (MAE) adapts the reconstruction task for patch-based
Vision Transformers [17].
The objective in MAE includes reconstructing the entire image from inputmasked image patches.
Inspired from this, we adapt SAC+AE into SAC+MAE by replacing theaugmented input image with its masked version, and only penalizing the reconstruction error for themasked patches.
RL Context Prediction Besides the self-supervised learning methods that are specifically designedfor pixels, we investigate the losses using attributes naturally collected during RL process.
For anystate transition that is not the end of a trajectory, it contains four components: current state s , nextstate s ′ , action a and reward r , with trajectory termination signal omitted.
Inspired by Shelhameret al.
[60], we concatenate the visual representation of the current state s and another representationh as the input.
Without loss of generality, the second input representation h can be any of thesethree representations of s ′ , a and r .
Then, we predict the remaining components using a two-layer
MLP.
For continuous outputs, mean-squared error (MSE) loss is applied, while for the discrete target(e.g., action in discrete action space), we use cross-entropy loss.
The architecture of this group ofself-supervised losses is shown in Fig.
From the combination of inputs and outputs, we define ninelosses whose I/
O specifications are provided in Table 1.
For those losses whose outputs include twocomponents, two target prediction networks share the same SSL head except the last task-specificlayer.
3.3 Evolving Multiple Self-supervised Losses
Besides a single self-supervised loss or handcrafted combination of two losses, we further investigatehow multiple self-supervised losses affect the policy learning together with the joint learning framework.
We formulate the combination of multiple losses as a weighted sum L Combo =
(cid:80) N l i =1 w i · L iwhere w i is the weight of a specific loss L i and N l is the total number of losses in the search space.
Inthe joint learning framework, we apply both self-supervised L Combo and RL losses jointly to the networks for every mini-batch.
Considering that the policy learning is quite sensitive to hyper-parameters(see Sec. A.4), it is non-trivial to find each weight for every SSL loss.
ELo ( E volving Lo sses) [53] shows promising results in unsupervised video representation learning [51, 63], by using evolutionary search to automatically find optimal combination of manyself-supervised losses.
In the spirit of ELo, we turn to evolutionary search for automatically find- ing the optimal solution.
Assume an unknown objective function whose inputs are weights ofmultiple losses w i and the magnitudes of image augmentation m j =1 , 2 for the online encoder andmomentum encoder.
The function output is the score achieved by the trained agent in its environment: R seedenv ( m j =1 , 2 , w i =1 , 2 ,...N l ) .
The optimization algorithm approaches the maximum value 194 of the objective function by continuously testing the value of the objective function, which inour case is the training and evaluation of an agent with the given parameters.
We choose an offthe-shelf optimization algorithm PSO ( P article S warm O ptimization) [36] for its simplicity, andperform the evolutionary search on one environment from DMControl [62] or Atari game benchmark [2].
For each set of inputs, we find it critical to run with multiple random seeds and report IQM (interquartile mean) 1 for a stable and robust search.
The optimization process is presented as:
[Equation]
Note that we are also implicitly searching for the balance between the self-supervised loss and the RLloss by performing this search, as it has the capability to adjust absolute weights of the self-supervisedlosses overall.
Please refer to Sec.
A.2.3 for our detailed configurations and search results.
Experiments
We conduct experiments in three directions, in order to better understand how we should integrate SSL with RL.
First, we demonstrate how different self-supervised losses affect the RL process, bytrying them on multiple challenging tasks.
Then, we benchmark a pretraining framework as analternative of the joint learning framework (Sec. A.7). Finally, we perform further empirical analysison the visual representations learned with the joint learning framework (Sec. A.8).
Evaluation Scheme
Thorough evaluation of reinforcement learning algorithms is challenging dueto the high variances between each run and the extensive requirement of computation.
Consequently,we run all experiments with multiple different random seeds and report the interquartile mean and thestandard deviation of the scores as suggested by Agarwal et al.
For a quantitative comparisonof the different methods mentioned in Section 3.2, in addition to the absolute scores, we assign a
Relative Score to each method.
We denote the interquartile mean of scores achieved by agent A inenvironment e ∈ E as IQM A,e and denote the collection of all interquartile mean scores achievedin environment e by different agents as IQM e .
The Relative Score of agent
A is computed as
S A Relative = (cid:80) e ∈ E ( IQM A,e − mean ( IQM e )) / std ( IQM e ) .
DMControl Experiments
DMControl ( D eep M ind C ontrol suite) contains many challengingvisual continuous control tasks, which are widely utilized by recent papers.
We evaluate all themethods introduced in Sec. 3, along with two important baselines, SAC-NoAug and SAC-Aug(100),in six environments of DMControl that are commonly used in previous papers [40, 41, 72, 73].
Othermethods that only take advantage of image augmentation, like RAD [41] and DrQ [73] are alsobenchmarked for comparison.
The difference between these methods can be summarized as follows:
SAC-NoAug is the originalpixel-based SAC [24, 25].
SAC-Aug(88) and SAC-Aug(100) use random crop as the only imageaugmentation, where (88) means the original image has a size of 88 × 88 before randomly croppingto 84 × 84 and (100) means the original image has a size of 100 × 100 .
These two methods should beregarded as variants of RAD with different augmentation choices. The Random crop from 100 × 100to 84 × 84 is the default image augmentation method for all the methods introduced in Sec. 3.2. Essentially, if we remove their self-supervised loss, they will fall back to SAC-Aug(100). Similarly,we test DrQ variants by replacing its default random shift augmentation with random crop, reportedas DrQ(88) and DrQ(100).
Meanwhile, RAD uses random translate by default except on walker walk;
ELo-SAC renders images at 100 × 100 and first perform center crop with sizes of 87 × 87 and 86 × 86.
Then two central patches are randomly cropped to 84 × 84 as the inputs for the online networks andthe target networks respectively.
For the policy learning part, all the methods share the same model.
However, DrQ, DrQ(88) and DrQ(100) apply an additional tanh activation after the convolutionallayers in the encoder.
Therefore, we also report DrQ-w/o-Tanh removing tanh activation from DrQfor reference.
We mainly follow the hyper-parameters reported in CURL, except that we use the same learning rate10 − 3 in all environments for simplicity.
All the methods are benchmarked at 100k environment steps,with training batch size 512 under 10 random seeds.
The relative score of each tested algorithm on DMControl is reported as Fig.
We strongly encourage readers to check full results at Table 7 for afull picture.
From the first glace at Fig.
4, no tested SSL-based method within the joint learning framework achievesbetter performance than DrQ and RAD which are carefully designed to take the best advantage ofspecific image augmentations.
We also find that compared to the baseline SAC-Aug(100), approacheswith a single self-supervised loss frequently (11 out of 18) fail to improve reinforcement learning.
Some SSL methods (like SimSiam, ShuffleCLS) ruin the policy learning resulting in performanceeven worse than SAC-NoAug, which suggests that an improper use of self-supervised loss can damagethe benefits brought by image augmentation.
Note that there are several self-supervised losses thatoutperform CURL in its joint learning framework, which shows some potential on studying novelself-supervised losses for RL.
Table 7 further shows that except in rare cases which the performance isconstantly lower than others like SimSiam and ShuffleCLS, there is no strong evidence showing thatany self-supervised loss is consistently better than others across all the environments.
Furthermore,how different tasks “respond” to a certain self-supervised loss can be regarded as a property, whichcan help us understand the similarities of different tasks and benefit the loss section.
Then, regarding combining losses, Guess-AF and Predict-FR, which are manually designed tocombine two individual losses, are not better than the single self-supervised loss in their combinations(see Guess-Action and Predict-Reward).
Considering that Extract-AR, Guess-AF and Predict-FRconcatenate both the outputs and apply supervision by averaging loss per element of the output, thetarget with a higher dimension will naturally get more penalty due to the larger number of elementsin the output.
We further test ‘-Balanced’ configuration, where we only modify how the supervisionis applied, and this trick bring overall improvements (See Sec. A.5).
ELo-SAC finds the desired combination by searching in one task.
This combination generalizes toother environments on DMControl with better overall performance than any approach in the searchspace.
In the ‘cheetah run’ where the search was performed, it obtained the best result among theapproaches with SSL.
Further, ELo-SAC achieves the best score in ‘walker, walk’ than any othermethods although only searched in ‘cheetah run’.
This demonstrates the feasibility of ELo-SAC andimplies that the obtained combination through evolutionary search has a potential to generalize toother environments in DMControl.
However, weaker performance in ‘finger, spin’ and ‘reacher, easy’made ELo-SAC relatively worse than DrQ (which does not use any self-supervision) on average.
Since DrQ outperforms the other methods without SSL, we further investigate the reason behindby comparing variants of DrQ with different augmentation designs. We confirm that the success of
DrQ comes not only from the averaging mechanism taking advantage of image augmentation, butalso from the specific well-designed image augmentation method.
Another example is the differentperformance of RAD and SAC-Aug(100), given the only difference between these two methodsis the augmentation. We also confirm the importance of image augmentation magnitude from thecomparison of DrQ(88) and DrQ(100), as well as from the comparison of SAC-Aug(88) and SAC-
Aug(100).
Further ablations can be found at Sec.
A.4.
Overall, our observation suggests that it is critical to engineer image augmentation carefully when designing a RL system with or without SSL.
Meanwhile, the better image augmentation for SSL + RL may need further investigations.
Atari Game Experiments
Atari 2600 Games are also challenging benchmarks but with discreteaction space.
We choose seven games in this benchmark for selected methods.
All the methods use
Efficient Rainbow [65] as the backbone, which is a Rainbow [31] variant with modifications forbetter data-efficiency.
Note that Efficient Rainbow, as a baseline, does not take advantage of imageaugmentation.
Therefore, we also benchmark Rainbow-Aug which is essentially Efficient Rainbowtaking the augmented images for policy learning instead.
We use the same image augmentationand hyper-parameters reported by CURL for all applicable methods.
For a fair comparison, theaugmentation for DrQ* is also adopted from CURL, which is different from what the original DrQpaper suggested.
We denote our setting as DrQ* to distinguish it from the original DrQ. For eachgame, we run 20 random seeds and benchmark the agent at 400K environment steps (100K modelsteps with frame skip of 4).
We report interquartile mean, standard deviation and relative score sameas DMControl (See Table 8).
Figure 5 shows a summary among the seven different tasks in Relative Score.
Firstly, compared tovanilla baseline Efficient Rainbow which does not have any image augmentation or self-supervisedlearning, Rainbow-Aug performs worse overall with additional image augmentation for RL.
Thissuggests that the image augmentation used for self-supervised learning in CURL does not easilytransfer.
Similarly, DrQ* achieves compromised performance than Efficient Rainbow, showingthat using image augmentation for Rainbow on Atari does not benefit policy learning unlike SACon DMControl.
Based on the inconsistent impacts of image augmentation, further investigation isrequired when applying image augmentation to RL on Atari.
As for the self-supervised losses, BYOL, Rainbow+AE, Extract-Reward and Predict-Reward gainbetter performance than CURL.
However, only Rainbow+AE shows significant improvement on
Efficient Rainbow and outperforms all the other tested methods, which interestingly is inconsistentwith SAC+AE on DMControl.
Predict-FR-Balanced, which shows considerable improvementson DMControl by manually combining two self-supervised losses, even fails to surpass PredictReward on Atari.
ELo-Rainbow, which searched in Frostbite, improves the baseline only in demonattack and frostbite.
The high variance on this benchmark made the evolutionary search extremelydifficult.
Further, there is huge image domain gaps between games, which makes it even harder for
ELo-Rainbow to work across multiple games on Atari.
Real Robot Experiments
We further conduct experiments in a real-world robot environment,uArm reacher.
The autonomous training environment set up and results are shown in Figs.
6 and 7. The robot fails to learn effective policy without image augmentation, while SAC-Aug(100) worksbetter than CURL and DrQ(100). Please check Sec.
A.6 for details.
Besides the joint learning framework, we investigatethe pretraining framework (shown as Fig. 8) which is widely used in previous work [58, 59, 61, 66, 77].
With the help of extra data, the pretraining framework usually works better than the joint learningframework, given the same amount of model optimization steps. However, the pretraining frameworkoften achieves lower scores than the joint learning framework given the same amount of environmentsteps. Please see Sec.
A.7 for more information.
Related Works
Self-supervised learning can fit in robot policy learning in multiple fashions and at different stages.
Some works [23, 57–59, 61, 66, 71, 77] use SSL for representation learning in a pre-training stagebefore policy learning.
Others [22, 26, 33, 40, 42, 44, 45, 50, 56, 72, 75, 76, 78, 79] jointly optimizethe self-supervised loss with RL.
Specifically, Transporter [39] and VAI [68] train an unsupervisedkeypoint detector to discovery critical objects in image for control.
RRL [58] and VRL3 [66] alsobenefit from pre-training a deeper visual encoder on large datasets like ImageNet [15].
TCN [57]and CURL [40] take advantage of contrastive learning.
After the agent is deployed, SSL can beused to continuously improve the policy [27].
Shelhamer et al.
[60] study several self-supervisedlosses within both pretraining framework and the joint learning framework, while their selection oflosses, number of runs and test environments are limited from a current point of view.
Chen et al.
[8]focus on imitation learning and test multiple SSL objectives for representation learning on variousenvironments.
They confirmed the critical role of image augmentation in imitation learning andshowed inconsistencies of performance across environments.
Our investigation supports some oftheir observations, beyond that, our evolving loss, real robot environment and representation analysisprovide unique perspectives for online reinforcement learning.
Dicussion
From DMControl and the real robot experiments, we empirically show that compared to the imageaugmentation, the role of existing self-supervised losses with the joint learning framework is usuallylimited, even with the help of evolutionary search.
While results on Atari show a different trendfrom DMControl, once again we confirm that there is no golden self-supervised loss or imageaugmentation that generalizes across environments.
At the same time, it is usually challenging toconclude a consistent trend that one method is meaningfully better than others across multiple tasks.
Though the pretraining framework can take advantage of extra data, it is also limited.
One shouldcautiously decide the design choice of image augmentation or self-supervised loss for a specific
RL task.
We are excited to see future works that introduce more self-supervised losses designedspecifically for RL, as well as novel training framework that can benefit policy learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Sec.
A.9(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please checksupplemental material(b)
Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?
[Yes] See Sec.
{+A.
 (c)+} Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Sec.
A.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]