A Mixture of Surprises for Unsupervised Reinforcement Learning
Abstract
Unsupervised reinforcement learning aims at learning a generalist policy in areward-free manner for fast adaptation to downstream tasks.
So far, existingmethods propose to provide an intrinsic reward based on surprise.
Maximizingor minimizing surprise drives the agent to either explore or gain control over itsenvironment.
However, both strategies rely on a strong assumption: the entropy ofthe environment’s dynamics is either high or low.
This assumption may not alwayshold in real-world scenarios, where the entropy of the environment’s dynamicsmay be unknown.
Hence, choosing between the two objectives is a dilemma.
Wepropose a novel yet simple mixture of policies to address this concern, allowingus to optimize an objective that simultaneously maximizes and minimizes thesurprise.
Concretely, we train one mixture component whose objective is to maximize the surprise and another whose objective is to minimize the surprise.
Hence,our method does not make assumptions about the entropy of the environment’sdynamics.
We call our method a M ixture O f S urprise S (MOSS) for unsupervisedreinforcement learning.
Experimental results show that, surprisingly, our simplemethod achieves state-of-the-art performance on the URLB benchmark, outperforming previous pure surprise maximization-based objectives.
Our code will bemade publicly available.
Introduction
Humans can learn meaningful behaviors without external supervision, i e., in an unsupervised manner,and then adapt those behaviors to new tasks [3].
Inspired by this, unsupervised reinforcementlearning decomposes the reinforcement learning (RL) problem into a pretraining phase and a finetunephase [28].
During a pretraining phase, an agent prepares all possible tasks that a user might select.
Afterward, the agent tries to figure out the selected task as quickly as possible during finetuning [21].
Doing so allows solving the RL problem in a meaningful order [3], e.g., a cook first has to look atwhat is in the fridge before deciding what to cook.
Unsupervised representation learning has showngreat success in Computer Vision [25] and Natural Language Processing [9]; however, one challengeis that RL includes both behavior learning and representation learning [28, 45].
Current unsupervised RL methods provide an intrinsic reward to the agent during a pretraining phaseto tackle the behavior learning problem [28].
Intuitively, this intrinsic reward should incentivizethe agent to understand its environment [12].
Current methods formulate the intrinsic reward aseither maximizing or minimizing surprise [38, 6, 32, 48, 36, 37, 10, 31, 27, 14, 40, 39, 11], wheresurprise is quantified either as the uncertainty of a prediction model (i.e., knowledge-based methods[10, 36, 37]) or as an information-theoretic quantity (i.e., data-based [38, 6, 48, 32, 5]).
Surprisemaximization methods [21, 27, 32] formulate the problem as an exploration problem.
Intuitively, toprepare all possible downstream tasks, an agent has to explore the state space and figure out what is possible in the environment.
For instance, one instantiation is to maximize the agent’s state entropy[32].
In contrast to surprise maximization, another line of work takes inspiration from the free-energyprinciple [19, 18, 17] and proposes to minimize the surprise [6, 38].
In particular, these works arguethat external perturbations naturally provide the agent with surprising events.
Hence, an agent canlearn meaningful behaviors by minimizing its surprise, and minimizing this quantity requires gainingcontrol over these external perturbations [38, 6].
For example, SMiRL [6] proposed to minimize theagent’s state entropy.
A closer look at these two approaches indicates a strong assumption.
Substantial external perturbations already naturally provide an agent with a high state entropy [6].
Therefore, surprise-seekingapproaches assume that the environment does not provide significant external perturbations.
Onthe other hand, in an environment without any external perturbations, the agent already achievesminimum entropy by not acting [38].
Therefore, surprise minimization approaches assume thatthe environment provides external perturbations for the agent to control.
However, in real-worldscenarios, it is often difficult to quantify the entropy of the environment’s dynamics beforehand, orthe agent might face both settings.
For example, a butler robot performs mundane chores daily ina household.
However, the robot might also encounter surprising events (e.g., putting out a fire).
Therefore a fixed assumption on the entropy of the environment dynamics is often not possible.
Inother words, choosing between surprise maximization or minimization for pretraining is a dilemma.
Simultaneously optimizing these two opposite objectives does not make sense.
Instead, in this paper,we show that competence-based methods offer a simple yet effective way to combine the benefits ofthese two objectives.
In addition to conditioning the policy on the state, competence-based methodscondition the policy on a latent vector [27, 31, 24, 46, 20, 1, 15].
Conditioning the policy offers anappealing way to formulate the unsupervised RL problem.
During a pretraining phase, the agent triesto learn a set of possible behaviors in the environment and distills them into skills.
Then, duringfinetuning, the agent tries to figure out the selected task as quickly as possible, using the repertoire ofskills gathered during pretraining.
For example, as illustrated in Fig. 1, we view skill distributions asa mixture of policies instead of a single policy.
In particular, we train one set of skills whose objectiveis to maximize the surprise and another set of skills whose objective is to minimize the surprise.
Surprisingly, this paper shows that this simple approach works well in practice, which simultaneouslyoptimizes two contradicting objectives.
Our primary contribution, presented in Section 4, is a simpleintrinsic reward called MOSS that does not make assumptions about the entropy of the environment’sdynamics.
In Section 5, our experimental results on URLB [28] and ViZDoom [26] show that,surprisingly, our MOSS methods achieve state-of-the-art results.
We organize the paper as follows.
Section 3 briefly analyzes previous unsupervised RL algorithmsunder the surprise framework.
Then in Section 4, we introduce our MOSS method.
Next, experimentalresults in Section 5 shows that on URLB [28] and ViZDoom [26], our MOSS method improvesupon previous pure maximization and minimization methods.
Finally, we provide discussions andlimitations in Section 6.
Preliminaries
Markov Decision Process Unsupervised RL methods studied in this paper operate under a Markov
Decision Process (MDP) [43].
In particular, we specify an MDP as a tuple M =
( S , A , T, r ext , ρ, γ )where S is the state space and A is the action space of the environment.
T ( S ′ | S , A ) represents thestate-transition dynamics, ρ : S → [0 , 1] is the initial state distribution, and γ ∈ [0 , 1) is the discountfactor.
At each discrete time step t ∈
Z ∗ , the agent receives a state and performs an action, whichwe denote as S t ∈ S and A t ∈ A , respectively.
During pretraining, unsupervised RL algorithmscompute an intrinsic reward r int ; during the finetune phase, the agent receives the extrinsic rewardr ext given by the environment at each interaction.
Skill Intuitively, a skill is an abstraction of a specific behavior (e.g., walking), and in practice, askill is a latent conditioned policy [14].
Given a latent vector z , we denote a skill as π θ ( a t | s t , z ) ,where π θ is the policy parameterized by θ .
For instance, during pretraining, the latent vectors aresampled every n steps such that the latent vector z is associated with the behavior executed duringthe associated n steps.
Mutual Information Knowledge-based, data-based, and competence-based methods have differentmeasures of surprise.
The study in this paper falls into the category of competence-based methods.
In particular, data-based and competence-based methods rely on an information-theoretic definitionof surprise, i.e., entropy.
Previous competence-based methods acquire skills by maximizing mutualinformation [13] between T and skills Z
[Equation]
where T can be the states S , the state transitions ( S ′ , S ) , or the next states ( S ′ | S ) .
In particular,these methods differ in how they decompose the mutual information.
Theoretically, these differentdecompositions are equivalent, i.e., they all maximize the mutual information between states andskills.
However, the particular choice greatly influences the performance in practice as optimizingthis objective relies on approximations.
To motivate the potential of competence-based methods over data-based or knowledge-based methods,we provide an intuitive understanding of Eq. 1.
On the one hand, the entropy term says that wewant skills in aggregate that explore the state space; we use it as a proxy to learn skills that coverthe set of possible behaviors.
On the other hand, It is not enough to learn skills that randomly go todifferent places.
We want to reuse those skills as accurately as possible, meaning we need to be ableto discriminate or predict the agent’s state transitions from skills.
To do so, we minimize conditionalentropy.
In other words, appropriate skills should cover the set of possible behaviors and should beeasily distinguishable.
Information-Theoretic Skill Discovery
Competence-based methods employ different intrinsic rewards to maximize mutual information: (1)discriminability-based and (2) exploratory-based intrinsic rewards.
For example, the former rewardsthe agent for discriminable skills.
In contrast, the latter rewards the agent for skills that effectivelycover the state space using a KNN density estimator [41, 32] to approximate the entropy term.
Belowwe analyze both approaches.
3.1 Discriminability-based Intrinsic Reward
Previous work such as DIAYN [14] uses a discriminability-based intrinsic reward.
They use thedecomposition in Eq. 2 and given a variational distribution q ϕ , they optimize the following variationallower bound
[Equation]
( z | s ))] ≥ E z , s ∼ p ( z , s )
[log q ϕ ( z | s )] − E z ∼ p ( z ) [ p ( z )] , (3) where Z ∼ p
( Z ) is a discrete random variable.
In particular, the discriminator rewards the agentif it can guess the skill from the state, i.e., r int ( s ) ≜ log q ϕ ( z | s ) − log p ( z ) .
These methods mayeasily run into a chicken and egg problem, where skills learn to be diverse using the discriminator’soutput.
However, the discriminator cannot learn to discriminate skills if the skills are not diverse.
Hence, it discourages the agent from exploring.
Previous work [42] has tried to solve this problem bydecoupling the aleatoric uncertainty from the epistemic uncertainty .
Furthermore, solving the chicken and egg problem is not enough since a state must map to a singleskill in Eq. 2.
Accordingly, the methods that use the decomposition in Eq.
2 require the skill space tobe smaller than the state space so that the skills are distinguishable.
Since previous work showed thata high-dimensional continuous skill space empirically performs better, the intrinsic reward shouldnot rely on the decomposition in Eq.
Intuitively, a continuous skill space allows (1) to interpolatebetween skills and (2) to represent a more significant set of skills in a more compact representation.
Instead, in Eq. 1, in contrast to skills that are predictable by states H
[ Z | T ] , we require that statesare predictable by skills H [ T | Z ] .
Since any state should be predictable from a given skill, the skillspace must be larger than the state space (i.e., we do not want a skill mapping to multiple states).
Therefore, another work [40] relies on a variational bound on Eq.
[Equation]
which uses a continuous skill space; however, it does not scale to high dimensions as the intrinsicreward r int ( s ′ , a , s ) ≜ log q ϕ ( s ′ | s , z ) p ( s ′ | s ) relies on an estimation of p ( s ′ | s ) .
In particular, they assumethat p ( z | s ) =
p ( z ) .
Intuitively, given s , if we assume each element in the latent vector ( z ) i to beindependent of each other, the error of this assumption will be scaled by the dimension of z .
3.2 Exploratory-based Intrinsic Reward
As aforementioned, discriminability-based intrinsic reward may easily run into a chicken and eggproblem.
Hence, other work [27, 31] uses an exploratory-based intrinsic reward to address the chickenand egg problem.
In other words, they use the decomposition in Eq.
1, where the intrinsic rewardexplicitly rewards the agent for exploring through H [ T ] .
Previous work [32] maximizes the state entropy, i.e., T = S .
However, the authors in [4] argue that itoften results in the discriminator simply memorizing the last state of each skill.
Instead, CIC [27]proposes to maximize the state-transitions entropy, i.e., H [ S ′ , S ] .
Therefore, CIC [27] proposes to estimate the conditional entropy H [ S ′ , S | Z ] using noise contrastiveestimation [27, 23] and the state-transition entropy H [ S ′ , S ] using a k -nearest neighbor estimator[41] to handle high dimensional skill space.
KNN-density estimation Previous works [32, 27] approximate the state-transition entropy H [ S ′ , S ]using a k -nearest neighbor estimator [41].
Given a random sample of size N , { ( S ( i ) ′ , S ( i ) ) } Ni =1 ∼ P ( S ′ , S ) , the estimator computes a Monte Carlo estimate [7] of the entropy defined as
[Equation]
where ˆ p k ( s ( i ) ′ , s ( i ) )
=
k NV ( i ) k is an estimation of p ( s ( i ) ′ , s ( i ) ) , V ( i ) k is the volume of the hypersphereof radius R ( i ) k , and b ( k ) is a constant such that the estimator is unbiased.
In particular, R k is thedistance from s to its k -nearest neighbor s ′ k where we assume a uniform distribution of points withinthe hypersphere (local uniformity assumption [33]).
In practice, to make the distance R k meaningful, we do not operate in the state space.
For instance,
CIC [27] trains an MLP f ψ : R |S| → R d that projects the state into a latent representation such that
R k = ∥
f ψ ( s ) − f ψ ( s ′ k ) ∥ where d is the skill dimension.
Moreover, in APT [32], it was found that averaging over all k -nearest neighbors leads to better results.
Hence, CIC [27] optimizes a quantityproportional to Eq. 5:
[Equation]
where c = 1 is constant for numerical stability.
We view the state-transition entropy as an expectedreward, and for a transition ( s ( i ) , s ( i ) ′ ) , the reward function is
[Equation]
Noise Contrastive Estimation To train parameters ψ , CIC [27] minimizes the conditional entropy 162 H [ S ′ ; S | Z ] using a noise contrastive estimation (NCE) [23, 35].
Denote h = f ψ ( s ) and h ′ =
f ψ ( s ′ )
the projected representations of s and s ′ , respectively.
Furthermore, let g ϕ z : R d
→
R
d be an MLPthat projects the skill into a latent representation and g ϕ s : R d × R d → R d be an MLP that projects hand h ′ into a latent representation.
We define the NCE loss
[Equation]
as an approximation to H [ S ′ ; S | Z ] , where we define cos ( · , · ) as the cosine similarity operator, ωas a temperature term, and c ( i ) j = cos ( g ϕ z ( z ( i ) ) , g ϕ s ( h ( j ) , h ( j ) ′ )) /ω .
As aforementioned, this termserves as representation learning for parameters ψ .
Moreover, in practice, ω is set to 0 .
5 to smooththe distribution.
Intuitively, the NCE loss [23] pushes p ( s ′ | s , z ) to be a delta-like density function.
However, since we define behaviors with n steps, we want a wider distribution.
Mixture Of Suprises (MOSS)
Our goal is to provide an algorithm that does not build on any assumption about the entropy of theenvironment’s dynamics.
We propose a novel yet simple solution that simultaneously maximizes andminimizes surprise using a mixture of policies to achieve this goal.
Our mixture of policies builds ona competence-based approach.
Specifically, we define the mixture of policies using a binary randomvariable M , where we associate each mixture component with a continuous uniform distribution U d ( a, b ) on an interval [ a, b ] .
Each uniform distribution constitutes a skill distribution.
Maintainingtwo different skill distributions allows us to define a different objective for each distribution.
Inparticular, we define two distributions Z max ∼ P
[-= 
-]
( Z | M = 0) and Z min ∼ P ( Z | M = 1) .
Sincewe do not assume any priors on the entropy of the environment’s dynamics, we set M = 0 for thefirst half of the steps and M = 1 for the other half of the steps in the episode.
Moreover, motivated by the discussions in Section 3.2, we build our method on top of the CIC [27] framework.
This framework is appealing as it bypasses the chicken and egg problem and supports high-dimensional skills.
In addition, it significantly outperforms previous competence-based methods on the URLB benchmark [28].
Hence, following CIC [27], we use the decomposition in Eq.
I ( S ′ , S ; Z ) =
H [ S ′ , S ] − H
[ S ′ , S | Z ] .
In particular, we define surprise as the state-transition entropy, which corresponds to maximizing andminimizing H [ S ′ , S ] , and as in CIC [27] we use Eq.
6 to approximate the state-transition entropy.
We approximate the conditional entropy H [ S ′ , S | Z ] using a noise contrastive estimation following Eq. 8.
By minimizing the conditional entropy, we distill behaviors into skills.
Doing so also serves as state-transition entropy [27].
Implementation To ensure fairness, all hyper-parameters are kept as in CIC [27] and we use thesame RL algorithm (i.e., DDPG [30] as implemented in DrQ-V2 [47]).
In practice, Z max ∼ P ( Z | M = 0) ≜
U d (0 , 1) and Z min ∼ P ( Z | M = 1) ≜
U d ( − 1 , 0) (we referreaders to appendix for other variants that we tried), where we set M = 0 for the first half of stepsand M = 1 for the remaining half of steps in the episode.
Hence, the RL agent maximizes
[Equation]
Input: Environment, number of pretraining steps N T .
Initialize DDPG, and ϕ s , ϕ z , ψ for t = 1 to N T do if ( t % update skill every ) =
{+= 
+}
then
Sample m t ∼ p ( m )
▷ setting to m = 0 is equivalent to CIC[27]
Sample z t ∼ p ( z | m t ) end if
Take some action a t and observe s t +
Store ( s t , s t +1 , m t , z t ) into buffer B
if t ≥ 4000 then
Sample a batch { ( a ( i ) , s ( i ) , s ( i ) ′ , m ( i ) , z ( i ) ) }
Ni =1 and compute intrinsic reward r int using equation 9.
Update DDPG using intrinsic reward.
Update ϕ s , ϕ z , ψ using noise contrastive loss in equation 8.
end if end for
And the intrinsic reward is a function of the state-transition and the binary random variable M
[Equation]
As in CIC [27], we approximate the state-transition entropy using Eq. 6.
MOSS’s implementation is straightforward; i.e., at each time step, instead of sampling Z ∼ P ( Z ) ,we sample Z ∼ P ( Z | M ) and replace the intrinsic reward with Eq. 9.
Pseudocode for MOSS isprovided in Alg.
In fact, with M = 0 , we get CIC [27].
Experiments
Environments We present the main results of method by evaluating on the Unsupervised Reinforcement Learning Benchmark (URLB) [28], which is a standard unsupervised RL benchmark forcontinuous control.
URLB [28] has three different domains listed in order of increasing difficulty:
• Walker is a biped robot domain constrained to a 2D vertical plane.
The challenge for thisdomain is to learn how to balance and maneuver simultaneously.• Quadruped is a four-legged robot in 3D space.
This domain has a higher dimensional stateand action space than the Walker domain.• Jaco is a 6-DOF robotic arm domain with a three-finger gripper in 3D space that has to learnaction constraints and manipulation.
Furthermore, each of the three domains has four different downstream tasks with varying difficulties.
We refer readers to Tab 1 for the complete list of 12 tasks.
Finally, in OpenAI gym environments[8], since an episode terminates as soon as a robot falls, the agent will naturally avoid falling to theground as it will get a 0 reward.
Hence, during pretraining, the OpenAI gym environment [8] mayleak some task information (e.g., standing) to the unsupervised learning agent [27].
Therefore, URLB[28] builds on top of the Deepmind Control Suite [44] to avoid leaking some task information.
The domains in URLB [28] use low-dimensional state information, deterministic transitions, andnon-evolving environments.
In contrast, ViZDoom [26] uses raw pixel observations, stochastictransition, and evolving environments.
Furthermore, since the environment randomly spawns enemiesthat shoot fireballs, the agent faces external perturbations that surprise the agent.
We evaluate ourmethod on the DefendTheLine and TakeCover maps.
Evaluation We follow the benchmark’s standard training procedure by pretraining the agent for 2million steps in each of the three domains and then finetuning the pre-trained agent for 100 k stepswith downstream task rewards.
To ensure fairness, we pretrain and finetune each method with the [Figure] same 12 seeds (0-11) using the code provided in URLB 1 [28].
We present the results in Fig.
3for a total of 1440 ( = 10 algorithms × 12 tasks × 12 seeds) runs.
Our main evaluation metrics areinterquartile mean (IQM) and Optimality Gap (OG) of normalized scores, where we used the score ofa DDPG [47] agent trained from scratch for 2 million steps as the expert scores [28].
On the one hand, IQM is more robust to outliers than the mean, and IQM is significantly less biased than the median[2].
On the other hand, the Optimality Gap measures how far scores are from the expert scores [2].
Baselines Baselines include methods presented in the URLB benchmark [28] as well as CIC [27].
The baseline methods fall into knowledge-based methods, i.e., ICM [36], Disagreement [37], RND[10], data-based methods, i.e., APT [32], Proto-RL [48], and competence-based methods, i.e., SMM[29], DIAYN [14], APS [31], CIC [27].
We refer readers to the Appendix of URLB [28] for detailson the different baselines.
We evaluate our agent in ViZDoom [26] against CIC [27], and a modified version of CIC [27], dubbed NegativeCIC, where the intrinsic reward is the negative state transition entropy.
Since the ViZDoomenvironment has high natural environment dynamics entropy, NegativeCIC should act as a strongbaseline.
We adopt the same pretraining and finetuning procedure as in URBL [28] for ViZDoom[26].
MOSS We keep all hyperparameters as those in CIC [27].
Specifically, unless specified otherwise,we set M = 0 for the first half of steps and M = 1 for the other half of steps in the episode.
We referreaders to the Appendix for more details.
[Figure]
5.1 Results
[Table]
To ensure the reliability of our main results, we report results as described in [2] to account for theuncertainty.
We present results in Fig.
3, where, surprisingly, our method obtains state-of-the-artperformance on the URLB benchmark [28] on our main evaluation metrics [2].
In particular, theperformance profile in Fig.
3b reveals that MOSS stochastically dominates all other methods.
Wealso present numerical results in Tab. 1 for a detailed breakdown over individual tasks.
For the IQM metric, we report an increase of 6% compared to CIC, which is the second leadingmethod.
Furthermore, as for the Optimality Gap metric, we decreased 20% compared to CIC.
Furthermore, in Fig.
3.b previous methods intersect at multiple points (making it hard to concludewhether a method is better than the other), while MOSS is always above all previous methods.
Inaddition, in Fig.
3.b, MOSS maintains all runs greater than a normalized score as high as 0 .
Defend the Line
Take Cover
[Table]
We also evaluated our method on ViZDoom [26].
Since the TakeCover and DefendTheLine mapsin ViZDoom provides natural perturbations, we use NegativeCIC as the baseline.
We present thenumerical results in Tab. 2.
Since the environments in ViZDoom are highly stochastic, we removethe top and bottom 10% of scores and report the mean and standard error to account for any outliers.
Like in the main results presented earlier, MOSS consistently outperforms the baseline method underthis completely new setting.
5.2 Ablation
In MOSS, we maintain two different skill distributions, i.e., Z max ∼ U d (0 , 1) and Z min ∼ U d ( − 1 , 0) .
In particular, setting Z min to the same distribution as Z max ∼ U d (0 , 1) is equivalent to directly optimizing CIC [27] with the two objectives simultaneously.
We dub MOSS with the same skilldistribution, MOSS_SAME.
In Fig.
4b, we observe that naively doing so can result in poor performance, showing the effectiveness of our method in mitigating the simultaneous optimization of twocontradicting objectives.
Moreover, as shown in Fig. 4b, NegativeCIC performs poorly on the URLB benchmark [28] sincemost tasks required structured exploration in a low entropy environment but performed well on thestanding task, which is inherently a low entropy task.
Since MOSS does not make assumptions about the entropy of the environment’s dynamics, we areinterested in knowing whether adding stochasticity affects performance.
For this stochastic setup,during each interaction with the environment, we sample Y ∼ Bernoulli(0 . 3) and if Y = 1 , weadd a Gaussian Noise N (0 , 0 . 2) to the agent’s intended action.
In Fig.
4a, we observe that CIC’s[27] performance drops while MOSS’ performance is almost unaffected.
This finding empiricallysupports previous works which claim that solely maximizing surprise for unsupervised RL agents issometimes not enough, especially in stochastic environments [6, 38]. [Figure]
Discussions & Limitations
We presented MOSS, an unsupervised RL method that does not make assumptions about the environment dynamics’ entropy by simultaneously maximizing and minimizing surprises.
Our methodfalls into competence-based methods.
Leveraging a mixture of policies for more than surprisemaximization and minimization is an interesting future research.
A previous work [16] also proposes to maximize and minimize surprise.
However, they rely on anadversarial game known to be hard to train [34].
In addition, they do not optimize the two objectivesusing a set of shared parameters. Instead, two different policies take turns. Therefore, it does not takeadvantage of sharing data and parameters. Similarly, the work in [6] shows that optimizing two seemingly opposite objectives is useful.
Inparticular, they maximize the uncertainty of a prediction model while minimizing surprise overeach episode.
Intuitively, it promotes exploration to find a more effective policy for minimizing thesurprise. In contrast, our method falls into the competence-based framework, which as we argue in Sec 2, offers a more appealing way to think about the unsupervised RL problem. In addition, in [6],there is a discrepancy between their two objectives.
They maximize the surprise according to all priorexperiences while minimizing the surprise according to the current episode. Moreover, they find thatit only helps in some cases. Limitations As we focused on URLB [28] and ViZDoom [26], our method assumes an episodicsetup, where the agent has access to environment resets. Combining with works like [22, 46] could bea valuable direction to minimize human intervention in real-world unsupervised RL setups.
Moreover,we only considered competence-based methods.
Extending our method to other unsupervised RLmethods would be interesting to investigate in future work.
Lastly, we only considered two objectives,and including more meaningful objectives during pretraining could be an exciting direction.
Potential Negative Impact Since we operate under the unsupervised reinforcement learningdomain, the agent receives rewards that incentives exploration.
Therefore, agents under this domaincan be more unpredictable than agents that receive task rewards for intended behaviors.
Furthermore,there is no current safeguard for the agent to not act maliciously in its environment; the agent maypotentially cause harm to properties situated in the environment, including the agent itself.
Therefore,unsupervised learning with safety constraints would be an exciting area of future research to mitigatethis potential negative impact.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that theChecklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.1.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Seesupplementary material and Section 5.
The code is in the supplementary material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See supplementary material and Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes code in the supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]