A Mixture of Surprises for Unsupervised Reinforcement Learning
Anonymous Author(s)
Abstract
Unsupervised reinforcement learning aims at learning a generalist policy in areward-free manner for fast adaptation to downstream tasks.
So far, existingmethods propose to provide an intrinsic reward based on surprise.
Maximizingor minimizing surprise drives the agent to either explore or gain control over itsenvironment.
However, both strategies rely on a strong assumption: the entropy ofthe environment’s dynamics is either high or low.
This assumption may not alwayshold in real-world scenarios, where the entropy of the environment’s dynamicsmay be unknown.
Hence, choosing between the two objectives is a dilemma.
Wepropose a novel yet simple mixture of policies to address this concern, allowingus to optimize an objective that simultaneously maximizes and minimizes thesurprise.
Concretely, we train one mixture component whose objective is to maximize the surprise and another whose objective is to minimize the surprise.
Hence,our method does not make assumptions about the entropy of the environment’sdynamics.
We call our method a M ixture O f S urprise S (MOSS) for unsupervisedreinforcement learning.
Experimental results show that, surprisingly, our simplemethod achieves state-of-the-art performance on the URLB benchmark, outperforming previous pure surprise maximization-based objectives.
Our code will bemade publicly available.
Introduction
Humans can learn meaningful behaviors without external supervision, i e., in an unsupervised manner,and then adapt those behaviors to new tasks [3].
Inspired by this, unsupervised reinforcementlearning decomposes the reinforcement learning (RL) problem into a pretraining phase and a finetunephase [29].
During a pretraining phase, an agent prepares all possible tasks that a user might select.
Afterward, the agent tries to figure out the selected task as quickly as possible during finetuning [22].
Doing so allows solving the RL problem in a meaningful order [3], e.g., a cook first has to look atwhat is in the fridge before deciding what to cook.
Unsupervised representation learning has showngreat success in Computer Vision [25] and Natural Language Processing [9]; however, one challengeis that RL includes both behavior learning and representation learning [29, 49].
Current unsupervised RL methods provide an intrinsic reward to the agent during a pretraining phaseto tackle the behavior learning problem [29].
Intuitively, this intrinsic reward should incentivizethe agent to understand its environment [13].
Current methods formulate the intrinsic reward aseither maximizing or minimizing surprise [41, 6, 33, 52, 38, 39, 10, 32, 28, 15, 43, 42, 12], whereknowledge-based methods quantify surprise as the uncertainty of a prediction model [10, 38, 39] anddata-based methods measure surprise as an information-theoretic quantity [41, 6, 52, 33, 5] 1 .
Surprise maximization methods [22, 28, 33] formulate the problem as an exploration problem.
Intuitively, toprepare all possible downstream tasks, an agent has to explore the state space and figure out what ispossible in the environment.
For instance, one instantiation is to maximize the agent’s state entropy[33].
In contrast to surprise maximization, another line of work takes inspiration from the free-energyprinciple [20, 19, 18] and proposes to minimize the surprise [6, 41].
In particular, these works arguethat external perturbations naturally provide the agent with surprising events.
Hence, an agent canlearn meaningful behaviors by minimizing its surprise, and minimizing this quantity requires gainingcontrol over these external perturbations [41, 6].
For example, SMiRL [6] proposed to minimize theagent’s state entropy.
A closer look at these two approaches indicates a strong assumption.
Substantial external perturbations already naturally provide an agent with a high state entropy [6].
Therefore, surprise-seekingapproaches assume that the environment does not provide significant external perturbations.
Onthe other hand, in an environment without any external perturbations, the agent already achievesminimum entropy by not acting [41].
Therefore, surprise minimization approaches assume thatthe environment provides external perturbations for the agent to control.
However, in real-worldscenarios, it is often difficult to quantify the entropy of the environment’s dynamics beforehand, orthe agent might face both settings.
For example, a butler robot performs mundane chores daily ina household.
However, the robot might also encounter surprising events (e.g., putting out a fire).
Therefore a fixed assumption on the entropy of the environment dynamics is often not possible.
Inother words, choosing between surprise maximization or minimization for pretraining is a dilemma.
Simultaneously optimizing these two opposite objectives does not make sense.
Instead, in thispaper, we show that competence-based methods [29] offer a simple yet effective way to combine thebenefits of these two objectives.
In addition to conditioning the policy on the state, competence-basedmethods condition the policy on a latent vector [28, 32, 24, 50, 21, 1, 16].
Conditioning the policyoffers an appealing way to formulate the unsupervised RL problem.
During a pretraining phase,the agent tries to learn a set of possible behaviors in the environment and distills them into skills.
Then, during finetuning, the agent tries to figure out the selected task as quickly as possible, usingthe repertoire of skills gathered during pretraining.
For example, as illustrated in Fig. 1, we viewskill distributions as a mixture of policies instead of a single policy.
In particular, we train one setof skills whose objective is to maximize the surprise and another set of skills whose objective is tominimize the surprise.
Surprisingly, this paper shows that this simple approach works well in practice,which simultaneously optimizes two contradicting objectives.
Our primary contribution, presentedin Section 4, is a simple intrinsic reward called MOSS that does not make assumptions about theentropy of the environment’s dynamics.
In Section 5, our experimental results on URLB [29] and ViZDoom [27] show that, surprisingly, our MOSS methods achieve state-of-the-art results.
We organize the paper as follows.
Section 3 briefly analyzes previous unsupervised RL algorithmsunder the surprise framework.
Then in Section 4, we introduce our MOSS method.
Next, experimentalresults in Section 5 shows that on URLB [29] and ViZDoom [27], our MOSS method improvesupon previous pure maximization and minimization methods.
Finally, we provide discussions andlimitations in Section 6.
Preliminaries
Markov Decision Process Unsupervised RL methods studied in this paper operate under a Markov
Decision Process (MDP) [47].
In particular, we specify an MDP as a tuple M =
( S , A , T, r ext , ρ, γ )where S is the state space and A is the action space of the environment.
T ( S ′ | S , A ) represents thestate-transition dynamics, ρ : S → [0 , 1] is the initial state distribution, and γ ∈ [0 , 1) is the discountfactor.
At each discrete time step t ∈
Z ∗ , the agent receives a state and performs an action, whichwe denote as S t ∈ S and A t ∈ A , respectively.
During pretraining, unsupervised RL algorithmscompute an intrinsic reward r int ; during the finetune phase, the agent receives the extrinsic rewardr ext given by the environment at each interaction.
Skill Intuitively, a skill is an abstraction of a specific behavior (e.g., walking), and in practice, askill is a latent conditioned policy [15].
Given a latent vector z , we denote a skill as π θ ( a t | s t , z ) ,where π θ is the policy parameterized by θ .
For instance, during pretraining, the latent vectors aresampled every n steps such that the latent vector z is associated with the behavior executed duringthe associated n steps.
Mutual Information Knowledge-based, data-based, and competence-based methods have differentmeasures of surprise.
The study in this paper falls into the category of competence-based methods.
In particular, data-based and competence-based methods rely on an information-theoretic definitionof surprise, i.e., entropy.
Previous competence-based methods acquire skills by maximizing mutualinformation [14] between T and skills Z
[Equation]
where T can be the states S , the joint distribution of state-transitions ( S ′ , S ) , or the state-transitions( S ′ | S ) .
In particular, these methods differ in how they decompose the mutual information.
Theoretically, these different decompositions are equivalent, i.e., they all maximize the mutual informationbetween states and skills.
However, the particular choice greatly influences the performance inpractice as optimizing this objective relies on approximations.
To motivate the potential of competence-based methods over data-based or knowledge-based methods,we provide an intuitive understanding of Eq. 1.
On the one hand, the entropy term says that wewant skills in aggregate that explore the state space; we use it as a proxy to learn skills that coverthe set of possible behaviors.
On the other hand, It is not enough to learn skills that randomly go todifferent places.
We want to reuse those skills as accurately as possible, meaning we need to be ableto discriminate or predict the agent’s state transitions from skills.
To do so, we minimize conditionalentropy.
In other words, appropriate skills should cover the set of possible behaviors and should beeasily distinguishable.
Information-Theoretic Skill Discovery
Competence-based methods employ different intrinsic rewards to maximize mutual information: (1)discriminability-based and (2) exploratory-based intrinsic rewards.
For example, the former rewardsthe agent for discriminable skills.
In contrast, the latter rewards the agent for skills that effectivelycover the state space using a KNN density estimator [44, 33] to approximate the entropy term.
Belowwe analyze both approaches.
3.1 Discriminability-based Intrinsic Reward
Previous work such as DIAYN [15] uses a discriminability-based intrinsic reward.
They use thedecomposition in Eq. 2 and given a variational distribution q ϕ , they optimize the following variationallower bound
[Equation]
where Z ∼ p
( Z ) is a discrete random variable.
In particular, the discriminator rewards the agentif it can guess the skill from the state, i.e., r int ( s ) ≜ log q ϕ ( z | s ) − log p ( z ) .
These methods mayeasily run into a chicken and egg problem, where skills learn to be diverse using the discriminator’soutput.
However, the discriminator cannot learn to discriminate skills if the skills are not diverse.
Hence, it discourages the agent from exploring.
Previous work [45] has tried to solve this problem bydecoupling the aleatoric uncertainty from the epistemic uncertainty .
Furthermore, solving the chicken and egg problem is not enough since a state must map to a singleskill in Eq. 2.
Accordingly, the methods that use the decomposition in Eq.
2 require the skill space tobe smaller than the state space so that the skills are distinguishable.
Since previous work showed thata high-dimensional continuous skill space empirically performs better, the intrinsic reward shouldnot rely on the decomposition in Eq.
Intuitively, a continuous skill space allows (1) to interpolatebetween skills and (2) to represent a more significant set of skills in a more compact representation.
Instead, in Eq. 1, in contrast to skills that are predictable by states H
[ Z | T ] , we require that statesare predictable by skills H [ T | Z ] .
Since any state should be predictable from a given skill, the skillspace must be larger than the state space (i.e., we do not want a skill mapping to multiple states).
Therefore, another work [43] relies on a variational bound on Eq.
[Equation]
which uses a continuous skill space; however, it does not scale to high dimensions as the intrinsicreward r int ( s ′ , a , s ) ≜ log q ϕ ( s ′ | s , z ) p ( s ′ | s ) relies on an estimation of p ( s ′ | s ) .
In particular, they assumethat p ( z | s ) =
p ( z ) .
Intuitively, given s , if we assume each element in the latent vector ( z ) i to beindependent of each other, the error of this assumption will be scaled by the dimension of z .
3.2 Exploratory-based Intrinsic Reward
As aforementioned, discriminability-based intrinsic reward may easily run into a chicken and eggproblem.
Hence, other work [28, 32] uses an exploratory-based intrinsic reward to address the chickenand egg problem.
In other words, they use the decomposition in Eq.
1, where the intrinsic rewardexplicitly rewards the agent for exploring through H [ T ] .
Previous work [33] maximizes the state entropy, i.e., T = S .
However, the authors in [4] argue that itoften results in the discriminator simply memorizing the last state of each skill.
Instead, CIC [28]proposes to maximize the entropy of the joint distribution of state-transitions (which from now on werefer as joint entropy), i.e., H [ S ′ , S ] .
Therefore, CIC [28] proposes to estimate the conditional entropy H [ S ′ , S | Z ] using noise contrastiveestimation [28, 23] and the joint entropy H [ S ′ , S ] using a k -nearest neighbor estimator [44] to handlehigh dimensional skill space.
KNN-density estimation Previous works [33, 28] approximate the joint entropy H [ S ′ , S ] using ak -nearest neighbor estimator [44].
Given a random sample of size N , { ( S ( i ) ′ , S ( i ) ) } Ni =1 ∼ P ( S ′ , S ) ,the estimator computes a Monte Carlo estimate [7] of the entropy defined as
[Equation]
where ˆ p k ( s ( i ) ′ , s ( i ) )
=
k NV ( i ) k is an estimation of p ( s ( i ) ′ , s ( i ) ) , V ( i ) k is the volume of the hypersphereof radius R ( i ) k , and b ( k ) is a constant such that the estimator is unbiased.
In particular, R k is thedistance from s to its k -nearest neighbor s ′ k where we assume a uniform distribution of points withinthe hypersphere (local uniformity assumption [34]).
In practice, to make the distance R k meaningful, we do not operate in the state space.
For instance,
CIC [28] trains an MLP f ψ : R |S| → R d that projects the state into a latent representation such that
R k = ∥
f ψ ( s ) − f ψ ( s ′ k ) ∥ where d is the skill dimension.
Moreover, in APT [33], it was found thataveraging over all k -nearest neighbors leads to better results.
Hence, CIC [28] optimizes a quantityproportional to Eq. 5:
[Equation]
where c = 1 is constant for numerical stability.
We view the joint entropy as an expected reward, andfor a transition ( s ( i ) , s ( i ) ′ ) , the reward function is
[Equation]
Noise Contrastive Estimation To train parameters ψ , CIC [28] minimizes the conditional entropy 163 H [ S ′ ; S | Z ] using a noise contrastive estimation (NCE) [23, 37].
Denote h = f ψ ( s ) and h ′ =
f ψ ( s ′ )
the projected representations of s and s ′ , respectively.
Furthermore, let g ϕ z : R d
→
R
d be an MLPthat projects the skill into a latent representation and g ϕ s : R d × R d → R d be an MLP that projects hand h ′ into a latent representation.
We define the NCE loss
[Equation]
as an approximation to H [ S ′ ; S | Z ] , where we define cos ( · , · ) as the cosine similarity operator, ωas a temperature term, and c ( i ) j = cos ( g ϕ z ( z ( i ) ) , g ϕ s ( h ( j ) , h ( j ) ′ )) /ω .
As aforementioned, this termserves as representation learning for parameters ψ .
Intuitively, the NCE loss [23] pushes p ( s ′ | s , z )to be a delta-like density function.
However, since we define behaviors with n steps, we want a widerdistribution, so in practice, ω is set to 0 . 5 to smooth the distribution.
Mixture Of Suprises (MOSS)
Our goal is to provide an algorithm that does not build on any assumption about the entropy of theenvironment’s dynamics.
A straightforward way to achieve this is to maximize and minimize surprisesimultaneously. A benefit of such an objective is that it promotes exploration to find a more effectivepolicy for minimizing the surprise [6]. In particular, in [6], they maximize surprise according to allprior experiences while minimizing surprise according to the current episode. However, they findthat it only helps in some cases.
The challenge is to find an effective way to optimize these twoopposite objectives since if we add the two objectives, it would simply yield a linear interpolation ofthe two objectives, which still corresponds to either maximizing or minimizing.
A previous work[17], dubbed Adversarial Surprise (AS), resolves this challenge using an adversarial game wheretwo policies compete against each other: one maximizes surprise while the other minimizes surprise.
AS [17] requires training two policies that do not share parameters or data.
Instead, we propose anovel yet simple mixture of policies to maximize and minimize surprise simultaneously.
Our mixtureof policies does not rely on an adversarial game [17] in which training is known to be challenging[35] and uses a single network for both objectives resulting in higher sample efficiency. Mainly, ourmethod only requires a minor change to CIC [28] that does not involves any computational costduring training. In particular, our mixture of policies builds on a competence-based approach [28],meaning we use two disjoint skill sets. We separate the two opposite objectives using a different skillset for each objective: one skill set for surprise maximization and another for surprise minimization.
Specifically, we define the mixture of policies using a binary random variable M , where we associateeach mixture component with a continuous uniform distribution U d ( a, b ) on an interval [ a, b ] .
Eachuniform distribution constitutes a skill distribution.
Maintaining two different skill distributionsallows us to define a different objective for each distribution.
In particular, we define two distributions
Z max ∼ P ( Z | M = 0) and Z min ∼ P ( Z | M = 1) .
Since we do not assume any priors on theentropy of the environment’s dynamics, we deterministically set M = 0 for the first half of the stepsand M = 1 for the other half of the steps in the episode.
Compared to more sophisticated methods(e.g., appendix B) for setting M (i.e., switching the objective), such a heuristic adds no computationalcost during training, requires no hyper-parameter tuning, and performs well.
Input: Environment, number of pretraining steps N T .
Initialize DDPG, and ϕ s , ϕ z , ψ for t = 1 to N T do if ( t % update skill every ) =
{+= 
+}
then Sample m t ∼ p ( m ) ▷ setting to m = 0 is equivalent to CIC[28] Sample z t ∼ p ( z | m t ) end if
Take some action a t and observe s t +
Store ( s t , s t +1 , m t , z t ) into buffer B
if t ≥ 4000 then
Sample a batch { ( a ( i ) , s ( i ) , s ( i ) ′ , m ( i ) , z ( i ) ) }
Ni =1 and compute intrinsic reward r int using equation 9.
Update DDPG using intrinsic reward.
Update ϕ s , ϕ z , ψ using noise contrastive loss in equation 8.
end if end for
Moreover, motivated by the discussions in Section 3.2, we build our method on top of the CIC 2 [28] framework.
This framework is appealing as it bypasses the chicken and egg problem and supports high-dimensional skills.
In addition, it significantly outperforms previous competence-based methods on the URLB benchmark [29].
Hence, following CIC [28], we use the decomposition in Eq.
I ( S ′ , S ; Z ) =
H [ S ′ , S ] − H
[ S ′ , S | Z ] .
In particular, we define surprise as the joint entropy, which corresponds to maximizing and minimizing 201 H [ S ′ , S ] , and as in CIC [28] we use Eq.
to approximate the joint entropy.
We approximate theconditional entropy H [ S ′ , S | Z ] using a noise contrastive estimation following Eq. 8.
By minimizingthe conditional entropy, we distill behaviors into skills.
Doing so also serves as representation learningfor computing the joint entropy [28].
Implementation To ensure fairness, all hyper-parameters are kept as in CIC [28] and we use thesame RL algorithm (i.e., DDPG [31] as implemented in DrQ-V2 [51]).
In practice, Z max ∼ P ( Z | M = 0) ≜
U d (0 , 1) and Z min ∼ P ( Z | M = 1) ≜
U d ( − 1 , 0) (we referreaders to appendix for other variants that we tried), where we set M = 0 for the first half of stepsand M = 1 for the remaining half of steps in the episode.
Hence, the RL agent maximizes
[Equation]
[-= 
-]
And the intrinsic reward is a function of the state-transition and the binary random variable M
[Equation]
As in CIC [28], we approximate the joint entropy using Eq. 6.
MOSS’s implementation is straightforward; i.e., at each time step, instead of sampling Z ∼ P ( Z ) ,we sample Z ∼ P ( Z | M ) and replace the intrinsic reward with Eq. 9.
Pseudocode for MOSS isprovided in Alg.
In fact, fixing M = 0 yields CIC [28].
Experiments
Environments We present the main results of method by evaluating on the Unsupervised Reinforcement Learning Benchmark (URLB) [29], which is a standard unsupervised RL benchmark forcontinuous control.
URLB [29] has three different domains listed in order of increasing difficulty:
• Walker is a biped robot domain constrained to a 2D vertical plane.
The challenge for this domain is to learn how to balance and maneuver simultaneously. • Quadruped is a four-legged robot in 3D space.
This domain has a higher dimensional stateand action space than the Walker domain. • Jaco is a 6-DOF robotic arm domain with a three-finger gripper in 3D space that has to learnaction constraints and manipulation.
Furthermore, each of the three domains has four different downstream tasks with varying difficulties.
We refer readers to Tab 1 for the complete list of 12 tasks.
Finally, in OpenAI gym environments[8], since an episode terminates as soon as a robot falls, the agent will naturally avoid falling to theground as it will get a 0 reward.
Hence, during pretraining, the OpenAI gym environment [8] mayleak some task information (e.g., standing) to the unsupervised learning agent [28].
Therefore, URLB[29] builds on top of the Deepmind Control Suite [48] to avoid leaking some task information.
The domains in URLB [29] use low-dimensional state information, deterministic transitions, andnon-evolving environments.
In contrast, ViZDoom [27] uses raw pixel observations, stochastictransition, and evolving environments.
Furthermore, since the environment randomly spawns enemiesthat shoot fireballs, the agent faces external perturbations that surprise the agent.
We evaluate ourmethod on the DefendTheLine and TakeCover maps.
Evaluation We follow the benchmark’s standard training procedure by pretraining the agent for 2million steps in each of the three domains and then finetuning the pre-trained agent for 100 k stepswith downstream task rewards.
Since AS trains two network-independent policies, we ensure bothpolicies each perform 2 million environment steps during pretraining.
To ensure fairness, we pretrainand finetune each method with the same 12 seeds (0-11) using the code provided in URLB 3 [29].
Wepresent the results in Fig.
2 for a total of 1584 ( = 11 algorithms × 12 tasks × 12 seeds) runs.
Ourmain evaluation metrics are interquartile mean (IQM) and Optimality Gap (OG) of normalized scores,where we used the score of a DDPG [51] agent trained from scratch for 2 million steps as the expertscores [29].
On the one hand, IQM is more robust to outliers than the mean, and IQM is significantlyless biased than the median [2].
On the other hand, the Optimality Gap measures how far scores arefrom the expert scores [2].
Baselines Baselines include methods presented in the URLB benchmark [29] as well as CIC [28].
The baseline methods fall into knowledge-based methods, i.e., ICM [38], Disagreement [39], RND[10], data-based methods, i.e., APT [33], Proto-RL [52], AS [17], and competence-based methods,i.e., SMM [30], DIAYN [15], APS [32], CIC [28].
We refer readers to the Appendix of URLB[29] for details on the different baselines.
Lastly, since the two policies (Alice and Bob) in AS areparameter-independent, we pretrain for 4 million steps, each policy trained using 2 million steps.
We evaluate our agent in ViZDoom [27] against CIC [28], and a modified version of CIC [28],dubbed NegativeCIC, where the intrinsic reward is the negative joint entropy.
Since the ViZDoomenvironment has high natural environment dynamics entropy, NegativeCIC should act as a strongbaseline.
We adopt the same pretraining and finetuning procedure as in URBL for ViZDoom.
MOSS We keep all hyperparameters as those in CIC [28].
Specifically, unless specified otherwise,we set M = 0 for the first half of steps and M = 1 for the other half of steps in the episode.
We referreaders to the Appendix for more details.
5.1 Results
To ensure the reliability of our main results, we report results as described in [2] to account for theuncertainty.
We present results in Fig.
2, where, surprisingly, our method obtains state-of-the-artperformance on the URLB benchmark [29] on our main evaluation metrics [2].
In particular, theperformance profile in Fig.
2b reveals that MOSS stochastically dominates all other methods.
Wealso present numerical results in Tab. 1 for a detailed breakdown over individual tasks.
For the IQM metric, we report an increase of 6% compared to CIC, which is the second leadingmethod.
Furthermore, as for the Optimality Gap metric, we decreased 20% compared to CIC. [Figure]
[Table]
Furthermore, in Fig.
2.b previous methods intersect at multiple points (making it hard to concludewhether a method is better than the other), while MOSS is always above all previous methods.
Inaddition, in Fig.
2.b, MOSS maintains all runs greater than a normalized score as high as 0 .
Defend the Line
[Table]
We also evaluated our method on ViZDoom [27].
Since the TakeCover and DefendTheLine mapsin ViZDoom provides natural perturbations, we use NegativeCIC as the baseline.
We present thenumerical results in Tab. 2.
Since the environments in ViZDoom are highly stochastic, we removethe top and bottom 10% of scores and report the mean and standard error to account for any outliers.
Like in the main results presented earlier, MOSS consistently outperforms the baseline method underthis completely new setting.
5.2 Ablation
In MOSS, we maintain two different skill distributions, i.e., Z max ∼ U d (0 , 1) and Z min ∼ U d ( − 1 , 0) .
In particular, setting Z min to the same distribution as Z max ∼ U d (0 , 1) is equivalent to directly optimizing CIC [28] with the two objectives simultaneously.
We dub MOSS with the same skilldistribution, MOSS_SAME.
In Fig.
3b, we observe that naively doing so can result in poor performance, showing the effectiveness of our method in mitigating the simultaneous optimization of twocontradicting objectives.
Moreover, as shown in Fig. 3b, NegativeCIC performs poorly on the URLB benchmark [29] sincemost tasks required structured exploration in a low entropy environment. [Figure]
Since MOSS does not make assumptions about the entropy of the environment’s dynamics, we areinterested in knowing whether adding stochasticity affects performance.
For this stochastic setup,during each interaction with the environment, we sample Y ∼ Bernoulli(0 . 3) and if Y = 1 , weadd a Gaussian Noise N (0 , 0 . 2) to the agent’s intended action.
In Fig.
3a, we observe that CIC’s[28] performance drops while MOSS’ performance is almost unaffected.
This finding empiricallysupports previous works which claim that solely maximizing surprise for unsupervised RL agents issometimes not enough, especially in stochastic environments [6, 41].
Discussions & Limitations
We presented MOSS, an unsupervised RL method that does not make assumptions about the environment dynamics’ entropy by simultaneously maximizing and minimizing surprises.
Our methodfalls into competence-based methods.
Leveraging a mixture of policies for more than surprisemaximization and minimization is an interesting future research.
Limitations One limitation is that our method has a simple and effective deterministic rule forswitching between the two objectives independent of state or history.
We did explore a moresophisticated method in Appendix B that had a better performance but required tuning and extracomputation.
Therefore, a promising direction is to explore methods that take advantage of stateinformation (e.g., using a meta-controller [26]) to output a more optimal switching mechanismthat incentivizes deep structured exploration [40].
Moreover, we only considered competence-basedmethods.
Extending our method to other unsupervised RL methods would be interesting to investigatein future work.
Additionally, when finetuning, we transferred network weights which could lead tounlearning of already learned behaviors [11].
Works like [11] utilized frozen pretrained policies toimprove downstream finetuning further.
Lastly, we only considered two objectives, and includingmore meaningful objectives like empowerment or anxiety [36, 46] during pretraining could be anexciting direction.
Potential Negative Impact Since we operate under the unsupervised reinforcement learningdomain, the agent receives rewards that incentives exploration.
Therefore, agents under this domaincan be more unpredictable than agents that receive task rewards for intended behaviors.
Furthermore,there is no current safeguard for the agent to not act maliciously in its environment; the agent maypotentially cause harm to properties situated in the environment, including the agent itself.
Therefore,unsupervised learning with safety constraints would be an exciting area of future research to mitigatethis potential negative impact.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary material and Section 5.
The code is in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See supplementary material and Section
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes code in the supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly
[Table]
[Table]