Original: CIC: C ONTRASTIVE I NTRINSIC C ONTROL FOR U NSUPERVISED S KILL D ISCOVERY
Revised: CIC: C ONTRASTIVE I NTRINSIC C ONTROL FOR U NSUPERVISED S KILL D ISCOVERY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce Contrastive Intrinsic Control (CIC) - an algorithm for unsupervised skill discovery that maximizes the mutual information between skills and state transitions.
Revised: We introduce Contrastive Intrinsic Control (CIC) - an algorithm for unsupervised skill discovery that maximizes the mutual information between skills and state transitions.

Original: In contrast to most prior approaches, CIC uses a decomposition of the mutual information that explicitly incentivizes diverse behaviors by maximizing state entropy.
Revised: In contrast to most prior approaches, CIC uses a decomposition of the mutual information that explicitly incentivizes diverse behaviors by maximizing state entropy.

Original: We derive a novel lower bound estimate for the mutual information which combines a particle estimator for state entropy to generate diverse behaviors and contrastive learning to distill these behaviors into distinct skills.
Revised: We derive a novel lower bound estimate for the mutual information which combines a particle estimator for state entropy to generate diverse behaviors and contrastive learning to distill these behaviors into distinct skills.

Original: We evaluate our algorithm on the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards.
Revised: We evaluate our algorithm on the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards.

Original: We ﬁnd that CIC improves on prior unsupervised skill discovery methods by 91% and the next-leading overall exploration algorithm by 26% in terms of downstream task performance.
Revised: We ﬁnd that CIC improves on prior unsupervised skill discovery methods by 91% and the next-leading overall exploration algorithm by 26% in terms of downstream task performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Deep Reinforcement Learning (RL) is a powerful approach toward solving complex control tasks in the presence of extrinsic rewards.
Revised: Deep Reinforcement Learning (RL) is a powerful approach toward solving complex control tasks in the presence of extrinsic rewards.

Original: Successful applications include playing video games from pixels (Mnih et al., 2015), mastering the game of Go (Silver et al., 2017; 2018), robotic locomotion (Schulman et al., 2016; 2017; Peng et al., 2018) and dexterous manipulation (Rajeswaran et al., 2018; OpenAI, 2018; 2019) policies.
Revised: Successful applications include playing video games from pixels (Mnih et al., 2015), mastering the game of Go (Silver et al., 2017; 2018), robotic locomotion (Schulman et al., 2016; 2017; Peng et al., 2018) and dexterous manipulation (Rajeswaran et al., 2018; OpenAI, 2018; 2019) policies.

Original: While effective, the above advances produced agents that are unable to generalize to new downstream tasks beyond the one they were trained to solve.
Revised: While effective, the above advances produced agents that are unable to generalize to new downstream tasks beyond the one they were trained to solve.

Original: Humans and animals on the other hand are able to acquire skills without supervision and apply them efﬁciently to a variety of downstream tasks.
Revised: Humans and animals on the other hand are able to acquire skills without supervision and apply them efﬁciently to a variety of downstream tasks.

Original: In this work, we seek to train agents that acquire skills without supervision with generalization capabilities by efﬁciently adapting these skills to downstream tasks.
Revised: In this work, we seek to train agents that acquire skills without supervision with generalization capabilities by efﬁciently adapting these skills to downstream tasks.

Original: Over the last few years, unsupervised RL has emerged as a promising framework for developing RL agents that can generalize to new tasks.
Revised: Over the last few years, unsupervised RL has emerged as a promising framework for developing RL agents that can generalize to new tasks.

Original: In the unsupervised RL setting, agents are ﬁrst pre-trained with self-supervised intrinsic rewards and then ﬁnetuned to downstream tasks with extrinsic rewards.
Revised: In the unsupervised RL setting, agents are ﬁrst pre-trained with self-supervised intrinsic rewards and then ﬁnetuned to downstream tasks with extrinsic rewards.

Original: Unsupervised RL algorithms broadly fall into three categories - knowledge-based, data-based, and competence-based methods 1 .
Revised: Unsupervised RL algorithms broadly fall into three categories - knowledge-based, data-based, and competence-based methods 1 .

Original: Knowledge-based methods maximize the error or uncertainty of a predictive model (Pathak et al., 2017; 2019; Burda et al., 2019b).
Revised: Knowledge-based methods maximize the error or uncertainty of a predictive model (Pathak et al., 2017; 2019; Burda et al., 2019b).

Original: Data-based methods maximize the entropy of the agent’s visitation (Liu & Abbeel, 2021a; Yarats et al., 2021b).
Revised: Data-based methods maximize the entropy of the agent’s visitation (Liu & Abbeel, 2021a; Yarats et al., 2021b).

Original: Competence-based methods learn skills that generate diverse behaviors (Eysenbach et al., 2019; Gregor et al., 2017).
Revised: Competence-based methods learn skills that generate diverse behaviors (Eysenbach et al., 2019; Gregor et al., 2017).

Original: This work falls into the latter category of competence-based methods.
Revised: This work falls into the latter category of competence-based methods.

Original: [Figure]
Revised: [Figure]

Original: Unlike knowledge-based and data-based algorithms, competence-based algorithms simultaneously address both the exploration challenge as well as distilling the generated experience in the form of reusable skills.
Revised: Unlike knowledge-based and data-based algorithms, competence-based algorithms simultaneously address both the exploration challenge as well as distilling the generated experience in the form of reusable skills.

Original: This makes them particularly appealing, since the resulting skill-based policies (or skills themselves) can be ﬁnetuned to efﬁciently solve downstream tasks.
Revised: This makes them particularly appealing, since the resulting skill-based policies (or skills themselves) can be ﬁnetuned to efﬁciently solve downstream tasks.

Original: While there are many self-supervised objectives that can be utilized, our work falls into a family of methods that learns skills by maximizing the mutual information between visited states and latent skill vectors.
Revised: While there are many self-supervised objectives that can be utilized, our work falls into a family of methods that learns skills by maximizing the mutual information between visited states and latent skill vectors.

Original: Many earlier works have investigated optimizing such objectives (Eysenbach et al., 2019; Gregor et al., 2017; Kwon, 2021; Sharma et al., 2020).
Revised: Many earlier works have investigated optimizing such objectives (Eysenbach et al., 2019; Gregor et al., 2017; Kwon, 2021; Sharma et al., 2020).

Original: However, competence-based methods have been empirically challenging to train and have under-performed when compared to knowledge and data-based methods (Laskin et al., 2021).
Revised: However, competence-based methods have been empirically challenging to train and have under-performed when compared to knowledge and data-based methods (Laskin et al., 2021).

Original: In this work, we take a closer look at the challenges of pre-training agents with competence-based algorithms.
Revised: In this work, we take a closer look at the challenges of pre-training agents with competence-based algorithms.

Original: We introduce Contrastive Intrinsic Control (CIC) – an exploration algorithm that uses a new estimator for the mutual information objective.
Revised: We introduce Contrastive Intrinsic Control (CIC) – an exploration algorithm that uses a new estimator for the mutual information objective.

Original: CIC combines particle estimation for state entropy (Singh et al., 2003; Liu & Abbeel, 2021a) and noise contrastive estimation (Gutmann &
Revised: CIC combines particle estimation for state entropy (Singh et al., 2003; Liu & Abbeel, 2021a) and noise contrastive estimation (Gutmann &

Original: Hyv ¨ arinen, 2010) for the conditional entropy which enables it to both generate diverse behaviors (exploration) and discriminate high-dimensional continuous skills (exploitation) .
Revised: Hyv ¨ arinen, 2010) for the conditional entropy which enables it to both generate diverse behaviors (exploration) and discriminate high-dimensional continuous skills (exploitation) .

Original: To the best of our knowledge, CIC is the ﬁrst exploration algorithm to utilize noise contrastive estimation to discriminate between latent skill vectors.
Revised: To the best of our knowledge, CIC is the ﬁrst exploration algorithm to utilize noise contrastive estimation to discriminate between latent skill vectors.

Original: Empirically, we show that CIC adapts to downstream tasks more efﬁciently than prior exploration approaches on the Unsupervised Reinforcement Learning Benchmark (URLB).
Revised: Empirically, we show that CIC adapts to downstream tasks more efﬁciently than prior exploration approaches on the Unsupervised Reinforcement Learning Benchmark (URLB).

Original: CIC achieves 91% higher returns on downstream tasks than prior competence-based algorithms and 26% higher returns than the next-best exploration algorithm overall.
Revised: CIC achieves 91% higher returns on downstream tasks than prior competence-based algorithms and 26% higher returns than the next-best exploration algorithm overall.

Original:  B ACKGROUND AND N OTATION
Revised:  B ACKGROUND AND N OTATION

Original: Markov Decision Process: We operate under the assumption that our system is described by a Markov Decision Process (MDP) (Sutton & Barto, 2018).
Revised: Markov Decision Process: We operate under the assumption that our system is described by a Markov Decision Process (MDP) (Sutton & Barto, 2018).

Original: An MDP consiss of the tuple ( S , A , P , r, γ ) which has states s ∈ S , actions a ∈ A , transition dynamics p
Revised: An MDP consiss of the tuple ( S , A , P , r, γ ) which has states s ∈ S , actions a ∈ A , transition dynamics p

Original: ( s (cid:48) | s, a ) ∼ P , a reward function r , and a discount factor γ .
Revised: ( s (cid:48) | s, a ) ∼ P , a reward function r , and a discount factor γ .

Original: In an MDP, at each timestep t , an agent observes the current state s , selects an action from a policy a ∼ π ( ·| s ) , and then observes the reward and next state once it acts in the environment: r, s (cid:48) ∼ env.step ( a ) .
Revised: In an MDP, at each timestep t , an agent observes the current state s , selects an action from a policy a ∼ π ( ·| s ) , and then observes the reward and next state once it acts in the environment: r, s (cid:48) ∼ env.step ( a ) .

Original: Note that usually r refers to an extrinsic reward.
Revised: Note that usually r refers to an extrinsic reward.

Original: However, in this work we will ﬁrst be pre-training an agent with intrinsic rewards r int and ﬁnetuning on extrinsic rewards r ext .
Revised: However, in this work we will ﬁrst be pre-training an agent with intrinsic rewards r int and ﬁnetuning on extrinsic rewards r ext .

Original: For convenience we also introduce the variable τ ( s ) which refers to any function of the states s .
Revised: For convenience we also introduce the variable τ ( s ) which refers to any function of the states s .

Original: For instance τ can be a single state, a pair of states, or a sequence depending on the algorithm.
Revised: For instance τ can be a single state, a pair of states, or a sequence depending on the algorithm.

Original: Our method uses τ =
Revised: Our method uses τ =

Original: ( s, s (cid:48) ) to encourage diverse state transitions while other methods have different speciﬁcations for τ .
Revised: ( s, s (cid:48) ) to encourage diverse state transitions while other methods have different speciﬁcations for τ .

Original: Importantly, τ does not denote a state-action trajectory, but is rather shorthand for any function of the states encountered by the agent.
Revised: Importantly, τ does not denote a state-action trajectory, but is rather shorthand for any function of the states encountered by the agent.

Original: In addition to the standard MDP notation, we will also be learning skills z ∈ Z and our policy will be skill-conditioned a ∼ π ( ·| s, z ) .
Revised: In addition to the standard MDP notation, we will also be learning skills z ∈ Z and our policy will be skill-conditioned a ∼ π ( ·| s, z ) .

Original: Unsupervised Skill Discovery through Mutual Information Maximization: Most competencebased approaches to exploration maximize the mutual information between states and skills.
Revised: Unsupervised Skill Discovery through Mutual Information Maximization: Most competencebased approaches to exploration maximize the mutual information between states and skills.

Original: Our work and a large body of prior research (Eysenbach et al., 2019; Sharma et al., 2020; Gregor et al., 2017; Achiam et al., 2018; Lee et al., 2019; Liu & Abbeel, 2021b) aims to maximize a mutual information objective with the following general form:
Revised: Our work and a large body of prior research (Eysenbach et al., 2019; Sharma et al., 2020; Gregor et al., 2017; Achiam et al., 2018; Lee et al., 2019; Liu & Abbeel, 2021b) aims to maximize a mutual information objective with the following general form:

Original: [Equation]
Revised: [Equation]

Original: Competence-based algorithms use different choices for τ and can condition on additional information such as actions or starting states.
Revised: Competence-based algorithms use different choices for τ and can condition on additional information such as actions or starting states.

Original: For a full summary of competence-based algorithms and their objectives see Table 1 in Appendix D.
Revised: For a full summary of competence-based algorithms and their objectives see Table 1 in Appendix D.

Original: Lower Bound Estimates of Mutual Information: The mutual information I ( s ; z ) is intractable to compute directly.
Revised: Lower Bound Estimates of Mutual Information: The mutual information I ( s ; z ) is intractable to compute directly.

Original: Since we wish to maximize I ( s ; z ) , we can approximate this objective by instead maximizing a lower bound estimate.
Revised: Since we wish to maximize I ( s ; z ) , we can approximate this objective by instead maximizing a lower bound estimate.

Original: Most known mutual information maximization algorithms use the variational lower bound introduced in Barber & Agakov (2003):
Revised: Most known mutual information maximization algorithms use the variational lower bound introduced in Barber & Agakov (2003):

Original: [Equation]
Revised: [Equation]

Original: Note that the variational lower bound can be applied to both decompositions of the mutual information.
Revised: Note that the variational lower bound can be applied to both decompositions of the mutual information.

Original: The design decisions of a competence-based algorithm therefore come down to (i) which decomposition of I ( τ ; z ) to use, (ii) whether to use discrete or continuous skills, (iii) how to estimate
Revised: The design decisions of a competence-based algorithm therefore come down to (i) which decomposition of I ( τ ; z ) to use, (ii) whether to use discrete or continuous skills, (iii) how to estimate

Original: H ( z ) or H ( τ ) , and ﬁnally (iv) how to estimate H ( z | τ ) or H ( τ | z ) .
Revised: H ( z ) or H ( τ ) , and ﬁnally (iv) how to estimate H ( z | τ ) or H ( τ | z ) .

Original:  M OTIVATION
Revised:  M OTIVATION

Original: The results from the recent Unsupervised Reinforcement Learning Benchmark (URLB) introduced in Laskin et al.
Revised: The results from the recent Unsupervised Reinforcement Learning Benchmark (URLB) introduced in Laskin et al.

Original: (2021), suggest that pre-training with competence-based approaches underperforms relative to knowledge-based and data-based baselines on DeepMind Control (DMC).
Revised: (2021), suggest that pre-training with competence-based approaches underperforms relative to knowledge-based and data-based baselines on DeepMind Control (DMC).

Original: We argue that the underlying issue with current competence-based algorithms when deployed on harder exploration environments like DMC has to do with the currently used estimators for I ( τ ; z ) rather than the objective itself.
Revised: We argue that the underlying issue with current competence-based algorithms when deployed on harder exploration environments like DMC has to do with the currently used estimators for I ( τ ; z ) rather than the objective itself.

Original: To produce structured skills that lead to diverse behaviors, I ( τ ; z ) estimators must (i) explicitly encourage diverse behaviors and (ii) have the capacity to discriminate between high-dimensional continuous skills.
Revised: To produce structured skills that lead to diverse behaviors, I ( τ ; z ) estimators must (i) explicitly encourage diverse behaviors and (ii) have the capacity to discriminate between high-dimensional continuous skills.

Original: Current approaches do not satisfy both criteria.
Revised: Current approaches do not satisfy both criteria.

Original: Competence-base algorithms do not ensure diverse behaviors: Most of the best known competencebased approaches (Eysenbach et al., 2019; Gregor et al., 2017; Achiam et al., 2018; Lee et al., 2019), optimize the ﬁrst decomposition of the mutual information H ( z ) − H ( z | τ ) .
Revised: Competence-base algorithms do not ensure diverse behaviors: Most of the best known competencebased approaches (Eysenbach et al., 2019; Gregor et al., 2017; Achiam et al., 2018; Lee et al., 2019), optimize the ﬁrst decomposition of the mutual information H ( z ) − H ( z | τ ) .

Original: The issue with this decomposition is that while it ensures diversity of skill vectors it does not ensure diverse behavior from the policy, meaning max H ( z ) does not imply max H ( τ ) .
Revised: The issue with this decomposition is that while it ensures diversity of skill vectors it does not ensure diverse behavior from the policy, meaning max H ( z ) does not imply max H ( τ ) .

Original: Of course, if H ( z ) − H ( z | τ ) is maximized and the skill dimension is sufﬁciently large, then H ( τ ) will also be maximized implicitly.
Revised: Of course, if H ( z ) − H ( z | τ ) is maximized and the skill dimension is sufﬁciently large, then H ( τ ) will also be maximized implicitly.

Original: Yet in practice, to learn an accurate discriminator q ( z | τ ) , the above methods assume skill spaces that are much smaller than the state space (see Table 1), and thus behavioral diversity may not be guaranteed.
Revised: Yet in practice, to learn an accurate discriminator q ( z | τ ) , the above methods assume skill spaces that are much smaller than the state space (see Table 1), and thus behavioral diversity may not be guaranteed.

Original: In contrast, the decomposition I ( τ ; z )
Revised: In contrast, the decomposition I ( τ ; z )

Original: =
Revised: =

Original: H ( τ ) − H ( τ | z ) ensures diverse behaviors through the entropy term H ( τ ) .
Revised: H ( τ ) − H ( τ | z ) ensures diverse behaviors through the entropy term H ( τ ) .

Original: Methods that utilize this decomposition include Liu & Abbeel (2021b); Sharma et al.
Revised: Methods that utilize this decomposition include Liu & Abbeel (2021b); Sharma et al.

Original: Why it is important to utilize high-dimensional skills: Once a policy is capable of generating diverse behaviors, it is important that the discriminator can distill these behaviors into distinct skills.
Revised: Why it is important to utilize high-dimensional skills: Once a policy is capable of generating diverse behaviors, it is important that the discriminator can distill these behaviors into distinct skills.

Original: If the set of behaviors outnumbers the set of skills, this will result in degenerate skills – when one skill maps to multiple different behaviors.
Revised: If the set of behaviors outnumbers the set of skills, this will result in degenerate skills – when one skill maps to multiple different behaviors.

Original: It is therefore important that the discriminator can accommodate continuous skills of sufﬁciently high dimension.
Revised: It is therefore important that the discriminator can accommodate continuous skills of sufﬁciently high dimension.

Original: Empirically, the discriminators used in prior work utilize only low-dimensional continuous skill vectors.
Revised: Empirically, the discriminators used in prior work utilize only low-dimensional continuous skill vectors.

Original: DIAYN (Eysenbach et al., 2019) utilized 16 dimensional skills, DADS (Sharma et al., 2020) utilizes continuous skills of dimension 2 − 5 , while APS (Liu & Abbeel, 2021b), an algorithm that utilizes successor features (Barreto et al., 2016; Hansen et al., 2020) for the discriminator, is only capable of learning continuous skills with dimension 10 .
Revised: DIAYN (Eysenbach et al., 2019) utilized 16 dimensional skills, DADS (Sharma et al., 2020) utilizes continuous skills of dimension 2 − 5 , while APS (Liu & Abbeel, 2021b), an algorithm that utilizes successor features (Barreto et al., 2016; Hansen et al., 2020) for the discriminator, is only capable of learning continuous skills with dimension 10 .

Original: We show how small skill spaces can lead to ineffective exploration in a simple gridworld setting in Appendix I and evidence that skill dimension affects performance in Fig.
Revised: We show how small skill spaces can lead to ineffective exploration in a simple gridworld setting in Appendix I and evidence that skill dimension affects performance in Fig.

Original: On the importance of benchmarks for evaluation: While prior competence-based approaches such as DIAYN (Eysenbach et al., 2019) were evaluated on OpenAI Gym (Brockman et al., 2016), Gym environment episodes terminate when the agent loses balance thereby leaking some aspects of extrinsic signal to the exploration agent.
Revised: On the importance of benchmarks for evaluation: While prior competence-based approaches such as DIAYN (Eysenbach et al., 2019) were evaluated on OpenAI Gym (Brockman et al., 2016), Gym environment episodes terminate when the agent loses balance thereby leaking some aspects of extrinsic signal to the exploration agent.

Original: On the other hand, DMC episodes have ﬁxed length.
Revised: On the other hand, DMC episodes have ﬁxed length.

Original: We show
Revised: We show

Original: [Figure]
Revised: [Figure]

Original: in Appendix K that this small difference in environments results in large performance differences.
Revised: in Appendix K that this small difference in environments results in large performance differences.

Original: In Fig.
Revised: In Fig.

Original: 11 we show that DIAYN is able to learn diverse skills in Gym but not in DMC, which is consistent with both observations from DIAYN and URLB papers.
Revised: 11 we show that DIAYN is able to learn diverse skills in Gym but not in DMC, which is consistent with both observations from DIAYN and URLB papers.

Original: Due to ﬁxed episode lengths, DMC tasks are harder for reward-free exploration since agents must learn to balance without supervision.
Revised: Due to ﬁxed episode lengths, DMC tasks are harder for reward-free exploration since agents must learn to balance without supervision.

Original:  M ETHOD 4. T HE CIC E STIMATOR
Revised:  M ETHOD 4. T HE CIC E STIMATOR

Original: From Section 3 we are motivated to ﬁnd an estimator for I ( τ ; z ) that explicitly maximizes the entropy H ( s ) through the second decomposition I ( τ ; z )
Revised: From Section 3 we are motivated to ﬁnd an estimator for I ( τ ; z ) that explicitly maximizes the entropy H ( s ) through the second decomposition I ( τ ; z )

Original: =
Revised: =

Original: H ( τ ) − H ( τ | z ) .
Revised: H ( τ ) − H ( τ | z ) .

Original: We also desire that our method’s discriminator is capable of supporting high-dimensional continuous skills to ensure maximal behavioral diversity.
Revised: We also desire that our method’s discriminator is capable of supporting high-dimensional continuous skills to ensure maximal behavioral diversity.

Original: 2 Note that τ is not a trajectory but some function of states.
Revised: 2 Note that τ is not a trajectory but some function of states.

Original: In this work, we propose a new estimator for I ( τ ; z ) which combines the use of a particle estimator for the entropy (Liu & Abbeel, 2021a) and noise contrastive estimation (Gutmann & Hyv ¨ arinen, 2010) for the conditional entropy.
Revised: In this work, we propose a new estimator for I ( τ ; z ) which combines the use of a particle estimator for the entropy (Liu & Abbeel, 2021a) and noise contrastive estimation (Gutmann & Hyv ¨ arinen, 2010) for the conditional entropy.

Original: Our proposed sample-based estimator is:
Revised: Our proposed sample-based estimator is:

Original: [Equation]
Revised: [Equation]

Original: where N is the number of samples, τ = ( s, s (cid:48) ) , and H particle ( τ ) is a particle estimator (Singh et al., 2003; Beirlant, 1997; Liu & Abbeel, 2021a) which estimates entropy by computing the distance between each particle h
Revised: where N is the number of samples, τ = ( s, s (cid:48) ) , and H particle ( τ ) is a particle estimator (Singh et al., 2003; Beirlant, 1997; Liu & Abbeel, 2021a) which estimates entropy by computing the distance between each particle h

Original: i and its k -th nearest neighbor h (cid:63)i such that H particle ( τ ) ∝ (cid:80) ni
Revised: i and its k -th nearest neighbor h (cid:63)i such that H particle ( τ ) ∝ (cid:80) ni

Original: =1 log (cid:107) h
Revised: =1 log (cid:107) h

Original: i − h
Revised: i − h

Original: (cid:63)i (cid:107) .
Revised: (cid:63)i (cid:107) .

Original: The CIC estimator should achieve the best of both worlds – encouraging exploration through max H ( τ ) and distilling behaviors into skills through contrastive representation learning.
Revised: The CIC estimator should achieve the best of both worlds – encouraging exploration through max H ( τ ) and distilling behaviors into skills through contrastive representation learning.

Original: We ﬁrst show that
Revised: We ﬁrst show that

Original: Eq.
Revised: Eq.

Original: 3 is a valid lower bound for I ( τ ; z ) .
Revised: 3 is a valid lower bound for I ( τ ; z ) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let F CIC ( τ, z ) be deﬁned as in Eq.
Revised: Let F CIC ( τ, z ) be deﬁned as in Eq.

Original: 3, we have that F CIC ( τ, z ) is a lower bound of the mutual information: I ( τ, z ) ≥ F CIC ( τ, z ) , where f ( τ, z ) is any real function of τ and z .
Revised: 3, we have that F CIC ( τ, z ) is a lower bound of the mutual information: I ( τ, z ) ≥ F CIC ( τ, z ) , where f ( τ, z ) is any real function of τ and z .

Original: Proof.
Revised: Proof.

Original: First we ﬁnd a variational lower bound for I ( τ ; z ) where the inequality is due to Barber &
Revised: First we ﬁnd a variational lower bound for I ( τ ; z ) where the inequality is due to Barber &

Original: Agakov (2003).
Revised: Agakov (2003).

Original: [Equation]
Revised: [Equation]

Original: From Contrastive Predictive Coding (CPC) (Oord et al., 2018)
Revised: From Contrastive Predictive Coding (CPC) (Oord et al., 2018)

Original: we can also have a sample based lower bound for I ( τ ; z ) .
Revised: we can also have a sample based lower bound for I ( τ ; z ) .

Original: [Equation]
Revised: [Equation]

Original: As shown in Oord et al.
Revised: As shown in Oord et al.

Original: (2018), this bound is upper bound by log N which means the bound will be loose when I ( τ ; z ) ≥ log N .
Revised: (2018), this bound is upper bound by log N which means the bound will be loose when I ( τ ; z ) ≥ log N .

Original: To provide a tighter lower bound, we note that we can also parameterize the variational density  with a noise contrastive estimator q ( τ i |z i ) =
Revised: To overcome this limitation, we note that we can also parameterize the variational density in Eq. 4 with a noise contrastive estimator q ( τ i |

Original:  exp[-(cid:104) 1 N (cid:80) Ni =
-] ( f ( τ i , z i ) − log 1 N (cid:80) Nj =1 exp( f ( τ i , z j )))
Revised: z i ) = exp[-(cid:104) 1 N (cid:80) Ni =
-] ( f ( τ i , z i )) / (cid:16) 1 N (cid:80) Nj =1 exp( f ( τ j , z i ))

Original: (cid:105) .
Revised: (cid:17) .

Original: We therefore have I ( τ  ; z  ) ≥ H ( τ  ) + F CPC ( τ i , z i ) which completes the proof.
Revised: We therefore have I ( τ i ; z i ) ≥ H ( τ i ) + F CPC ( τ i , z i ) which completes the proof.

Original: A favorable property of the CIC estimator is that it provides a tighter lower bound than CPC for mutual information I ( τ ; z ) ≥ F CIC ( τ, z ) ≥ F CPC ( τ, z ) .
Revised: A favorable property of the CIC estimator is that it provides a tighter lower bound than CPC for mutual information I ( τ ; z ) ≥ F CIC ( τ, z ) ≥ F CPC ( τ, z ) .

Original: In contrast to the CPC estimator, CIC is more suitable for exploration due to the explicit presence of H ( τ ) , which helps learning meaningful representations and behaviors as evident in recent work (Campos et al., 2020; Mutti et al., 2021;
Revised: In contrast to the CPC estimator, CIC is more suitable for exploration due to the explicit presence of H ( τ ) , which helps learning meaningful representations and behaviors as evident in recent work (Campos et al., 2020; Mutti et al., 2021;

Original: Liu & Abbeel, 2021a; Campos et al., 2021a; Yarats et al., 2021b)
Revised: Liu & Abbeel, 2021a; Campos et al., 2021a; Yarats et al., 2021b)

Original: whereas Eq.
Revised: whereas Eq.

Original: 5 does not explicitly encourage exploration.
Revised: 5 does not explicitly encourage exploration.

Original: On the other hand, contrastive learning has been demonstrated as a powerful approach for representation learning in vision and reinforcement learning (Chen et al., 2020; Oord et al., 2018; Laskin et al., 2020b).
Revised: On the other hand, contrastive learning has been demonstrated as a powerful approach for representation learning in vision and reinforcement learning (Chen et al., 2020; Oord et al., 2018; Laskin et al., 2020b).

Original: It is therefore interesting to combine these two objectives into a single intrinsic reward.
Revised: It is therefore interesting to combine these two objectives into a single intrinsic reward.

Original: 4. I NTRINSIC R EWARD AND I NTERPRETATION OF CIC E STIMATION
Revised: 4. I NTRINSIC R EWARD AND I NTERPRETATION OF CIC E STIMATION

Original: Intrinsic Reward: We parameterize f ( τ, z ) =
Revised: Intrinsic Reward: We parameterize f ( τ, z ) =

Original: g ψ 1 ( τ ) (cid:62)g
Revised: g ψ 1 ( [Equation] ) (cid:62)

Original: ψ 2 ( z ) where
Revised: [Equation]

Original: τ =
Revised: [Equation]

Original:  ( s, s (cid:48) ) is a transition tuple and g ψ k are neural encoders.
Revised: where τ = ( s, s (cid:48) ) is a transition tuple and g ψ k are neural encoders.

Original: This inner product is similar to the one used in the SimCLR (Chen et al., 2020) representation learning loss.
Revised: This inner product is similar to the one used in the SimCLR (Chen et al., 2020) representation learning loss.

Original: We then use a particle estimator (Singh et al., 2003;
Revised: We then use a particle estimator (Singh et al., 2003;

Original: Beirlant, 1997) as in Liu & Abbeel (2021a) for the entropy term.
Revised: Beirlant, 1997) as in Liu & Abbeel (2021a) for the entropy term.

Original: Similar to Liu & Abbeel (2021a);
Revised: Similar to Liu & Abbeel (2021a);

Original: Yarats et al.
Revised: Yarats et al.

Original: (2021b) rather than using the exact form of the particle estimator we estimate the entropy up to entropy up to a proportionality constant, and therefore introduce a hyperparameter α to weigh the entropy and CPC terms.
Revised: (2021b) rather than using the exact form of the particle estimator we estimate the entropy up to entropy up to a proportionality constant, and therefore introduce a hyperparameter α to weigh the entropy and CPC terms.

Original: With this parametrization the intrinsic reward for the unsupervised RL agent takes on the following form:
Revised: With this parametrization the intrinsic reward for the unsupervised RL agent takes on the following form:

Original: [Equation]
Revised: [Equation]

Original: where h i is an embedding of τ i shown in Fig.
Revised: where h i is an embedding of τ i shown in Fig.

Original: 3, h ∗ i is a kNN embedding, N k is the number of kNNs, and N − 1 is the number of negatives.
Revised: 3, h ∗ i is a kNN embedding, N k is the number of kNNs, and N − 1 is the number of negatives.

Original: The total number of elements in the summation is N because it includes one positive.
Revised: The total number of elements in the summation is N because it includes one positive.

Original: Explore and Exploit: We can interpret the two terms of Eq. 6 as contributing two different behaviors to the exploration algorithm.
Revised: Explore and Exploit: We can interpret the two terms of Eq. 6 as contributing two different behaviors to the exploration algorithm.

Original: The entropy term H ( τ ) encourages exploration by maximizing state diversity.
Revised: The entropy term H ( τ ) encourages exploration by maximizing state diversity.

Original: The variational density q ( τ | z ) encourages exploitation by ensuring that skills z lead to predictable states τ .
Revised: The variational density q ( τ | z ) encourages exploitation by ensuring that skills z lead to predictable states τ .

Original: Together the two terms form an intrinsic reward that incentivizes diverse yet predictable behavior from the RL agent.
Revised: Together the two terms form an intrinsic reward that incentivizes diverse yet predictable behavior from the RL agent.

Original: Asymptotic Behavior of the Intrinsic Reward: When maximum entropy is reached the H ( τ ) term in Eq.
Revised: Asymptotic Behavior of the Intrinsic Reward: When maximum entropy is reached the H ( τ ) term in Eq.

Original: 6 will vanish since there are no new states to discover.
Revised: 6 will vanish since there are no new states to discover.

Original: Therefore asymptotically exploration will stop.
Revised: Therefore asymptotically exploration will stop.

Original: However, the variational density q ( τ | z ) parameterized by CPC will continue distilling all states in the environment into skills z until they are maximally distinct such that H ( τ | z ) =
Revised: However, the variational density q ( τ | z ) parameterized by CPC will continue distilling all states in the environment into skills z until they are maximally distinct such that H ( τ | z ) =

Original: ε (cid:28) 1 .
Revised: ε (cid:28) 1 .

Original:  P RACTICAL I MPLEMENTATION
Revised:  P RACTICAL I MPLEMENTATION

Original: Our practical implementation consists of two main components: the RL optimization algorithm and the architecture for specifying the intrinsic reward.
Revised: Our practical implementation consists of two main components: the RL optimization algorithm and the architecture for specifying the intrinsic reward.

Original: For fairness and clarity of comparison, we use the same RL optimization algorithm for our method and all baselines in this work.
Revised: For fairness and clarity of comparison, we use the same RL optimization algorithm for our method and all baselines in this work.

Original: Since the baselines
Revised: Since the baselines

Original: [Figure]
Revised: [Figure]

Original: implemented in URLB (Laskin et al., 2021) use a DDPG 3 (Lillicrap et al., 2016) as their backbone, we opt for the same DDPG architecture to optimize our method as well (see Appendix B).
Revised: implemented in URLB (Laskin et al., 2021) use a DDPG 3 (Lillicrap et al., 2016) as their backbone, we opt for the same DDPG architecture to optimize our method as well (see Appendix B).

Original: For the full algorithm
Revised: For the full algorithm

Original: Architecture for intrinsic rewards: We use a particle estimator as in Liu & Abbeel (2021a) to estimate H ( s ) .
Revised: Architecture for intrinsic rewards: We use a particle estimator as in Liu & Abbeel (2021a) to estimate H ( s ) .

Original: To compute the variational density q ( τ | z ) , we ﬁrst sample skills from uniform noise z ∼ p ( z ) where p ( z ) is the uniform distribution over the [0 , 1] interval.
Revised: To compute the variational density q ( τ | z ) , we ﬁrst sample skills from uniform noise z ∼ p ( z ) where p ( z ) is the uniform distribution over the [0 , 1] interval.

Original: We then use two MLP encoders to embed g ψ 1 ( τ ) and g ψ 2 ( z ) , and optimize the parameters ψ 1 , ψ 2 with the CPC loss similar to SimCLR (Chen et al., 2020) since f ( τ, z ) =
Revised: We then use two MLP encoders to embed g ψ 1 ( τ ) and g ψ 2 ( z ) , and optimize the parameters ψ 1 , ψ 2 with the CPC loss similar to SimCLR (Chen et al., 2020) since f ( τ, z ) =

Original: g ψ 1 ( τ ) T g ψ 2 ( z ) .
Revised: g ψ 1 ( τ ) T g ψ 2 ( z ) .

Original: We ﬁx the hyperparameters across all domains and downstream tasks.
Revised: We ﬁx the hyperparameters across all domains and downstream tasks.

Original: We refer the reader to the Appendices E and F for the full algorithm and a full list of hyperparameters.
Revised: We refer the reader to the Appendices E and F for the full algorithm and a full list of hyperparameters.

Original: Adapting to downstream tasks: To adapt to downstream tasks we follow the same procedure for competence-based method adaptation as in URLB (Laskin et al., 2021).
Revised: Adapting to downstream tasks: To adapt to downstream tasks we follow the same procedure for competence-based method adaptation as in URLB (Laskin et al., 2021).

Original: During the ﬁrst 4k environment interactions we populate the DDPG replay buffer with samples and use the extrinsic rewards collected during this period to ﬁnetune the skill vector z .
Revised: During the ﬁrst 4k environment interactions we populate the DDPG replay buffer with samples and use the extrinsic rewards collected during this period to ﬁnetune the skill vector z .

Original: While it’s common to ﬁnetune skills with
Revised: While it’s common to ﬁnetune skills with

Original: Cross Entropy Adaptation (CMA), given our limited budget of 4k samples (only 4 episodes) we ﬁnd that a simple grid sweep of skills over the interval [0 , 1] produces the best results (see Fig. 6).
Revised: Cross Entropy Adaptation (CMA), given our limited budget of 4k samples (only 4 episodes) we ﬁnd that a simple grid sweep of skills over the interval [0 , 1] produces the best results (see Fig. 6).

Original: After this, we ﬁx the skill z and ﬁnetune the DDPG actor-critic parameters against the extrinsic reward for the remaining 96k steps.
Revised: After this, we ﬁx the skill z and ﬁnetune the DDPG actor-critic parameters against the extrinsic reward for the remaining 96k steps.

Original: Note that competence-based methods in URLB also ﬁnetune their skills during the ﬁrst 4k ﬁnetuning steps ensuring a fair comparison between the methods.
Revised: Note that competence-based methods in URLB also ﬁnetune their skills during the ﬁrst 4k ﬁnetuning steps ensuring a fair comparison between the methods.

Original: The full adaptation procedure is detailed in Appendix E.
Revised: The full adaptation procedure is detailed in Appendix E.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Environments We evaluate our approach on tasks from URLB, which consists of twelve downstream tasks across three challenging continuous control domains for exploration algorithms – walker, quadruped, and Jaco arm.
Revised: Environments We evaluate our approach on tasks from URLB, which consists of twelve downstream tasks across three challenging continuous control domains for exploration algorithms – walker, quadruped, and Jaco arm.

Original: Walker requires a biped constrained to a 2D vertical plane to per-
Revised: Walker requires a biped constrained to a 2D vertical plane to per-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Quadruped
Revised: Quadruped

Original: P a r t i c
Revised: P a r t i c

Original: l e e s t
Revised: l e e s t

Original: i m a t e ℋ ( τ )
Revised: i m a t e ℋ ( τ )

Original: Quadruped ( α = 0.9) Walker ( α = 0.9)
Revised: Quadruped ( α = 0.9) Walker ( α = 0.9)

Original: Jaco ( α = 0)
Revised: Jaco ( α = 0)

Original: I n t r i n s i
Revised: I n t r i n s i

Original: c
Revised: c

Original: R e w a r d s form locomotion tasks while balancing.
Revised: R e w a r d s form locomotion tasks while balancing.

Original: Quadruped is more challenging due to a higher-dimensional state-action space and requires a quadruped to in a 3D environment to learn locomotion skills.
Revised: Quadruped is more challenging due to a higher-dimensional state-action space and requires a quadruped to in a 3D environment to learn locomotion skills.

Original: Jaco arm is a 6-DOF robotic arm with a three-ﬁnger gripper to move and manipulate objects without locking.
Revised: Jaco arm is a 6-DOF robotic arm with a three-ﬁnger gripper to move and manipulate objects without locking.

Original: All three environments are challenging in the absence of an extrinsic reward.
Revised: All three environments are challenging in the absence of an extrinsic reward.

Original: An important distinction between the environments in URLB and the OpenAI gym (Brockman et al., 2016) continuous control is that OpenAI gym environments terminate early if the agent loses balance while URLB episodes have ﬁxed length.

Original: This means that the exploration task in URLB is substantially harder – the agent must learn to balance on its own without any extrinsic signal.

Original: Baselines: We compare CIC to baselines across all three exploration categories.
Revised: Baselines: We compare CIC to baselines across all three exploration categories.

Original: Knowledge-based basedlines include ICM (Pathak et 
Revised: Knowledge-based basedlines include ICM (Pathak et al., 2017), Disagreement (Pathak et al., 2019), and RND (Burda et al., 2019b). Data-based baselines incude APT (Liu & Abbeel, 2021a) and ProtoRL (Yarats et al., 2021b). Competence-based baselines include DIAYN Eysenbach et al. (2019), SMM Lee et al.

Revised: (2019), and APS (Liu & Abbeel, 2021b).

Original: The closest baselines to CIC are APT, which is similar to CIC with α = 1 . 

Revised: The closest baselines to CIC are APT, which is similar to CIC with α = 1 .

Original: (no discriminator), and APS which uses the same decomposition of mutual information as CIC and also uses a particle entropy estimate for H ( τ ) .
Revised: (no discriminator), and APS which uses the same decomposition of mutual information as CIC and also uses a particle entropy estimate for H ( τ ) .

Original: The main difference between
Revised: The main difference between

Original: APS and CIC is that APS uses successor features while CIC uses a contrastive estimator for the discriminator.
Revised: APS and CIC is that APS uses successor features while CIC uses a contrastive estimator for the discriminator.

Original: For further details regarding baselines we refer the reader to Appendix C.
Revised: For further details regarding baselines we refer the reader to Appendix C.

Original: Evaluation: We follow an identical evaluation to the 2M pre-training setup in URLB.
Revised: Evaluation: We follow an identical evaluation to the 2M pre-training setup in URLB.

Original: First, we pretrain each RL agent with the intrinsic rewards for 2M steps.
Revised: First, we pretrain each RL agent with the intrinsic rewards for 2M steps.

Original: Then, we ﬁnetune tune each agent to the downstream task with extrinsic rewards in the data-efﬁcient regime of 100k steps.
Revised: Then, we ﬁnetune tune each agent to the downstream task with extrinsic rewards in the data-efﬁcient regime of 100k steps.

Original: We use 10 seeds across each downstream task for our method and all the baseline algorithms.
Revised: We use 10 seeds across each downstream task for our method and all the baseline algorithms.

Original: For baselines, we benchmark against leading knowledge-based, data-based, and competence-based approaches that have been implemented in URLB.
Revised: For baselines, we benchmark against leading knowledge-based, data-based, and competence-based approaches that have been implemented in URLB.

Original: All baselines use the same DDPG optimization algorithm to eliminate confounding factors when comparing algorithms.
Revised: All baselines use the same DDPG optimization algorithm to eliminate confounding factors when comparing algorithms.

Original: To ensure that our evaluation statistics are unbiased we use stratiﬁed bootstrap conﬁdence intervals to report aggregate statistics across M runs with N seeds as described in Rliable (Agarwal et al., 2021) to report statistics for our main results in Fig. 4.
Revised: To ensure that our evaluation statistics are unbiased we use stratiﬁed bootstrap conﬁdence intervals to report aggregate statistics across M runs with N seeds as described in Rliable (Agarwal et al., 2021) to report statistics for our main results in Fig. 4.

Original: Our primary success metric is the interquartile mean (IQM) and the Optimality Gap (OG).
Revised: Our primary success metric is the interquartile mean (IQM) and the Optimality Gap (OG).

Original: IQM discards the top and bottom 25 % of runs and then computes the mean.
Revised: IQM discards the top and bottom 25 % of runs and then computes the mean.

Original: It is less susceptible to outliers than the mean and was shown to be the most reliable statistic for reporting results for RL experiments in Agarwal et al.
Revised: It is less susceptible to outliers than the mean and was shown to be the most reliable statistic for reporting results for RL experiments in Agarwal et al.

Original: OG measures how far a policy is from optimal (expert) performance.
Revised: OG measures how far a policy is from optimal (expert) performance.

Original: To deﬁne expert performance we use the convention in URLB, which is the score achieved by a randomly initialized DDPG after 2M steps of ﬁnetuning (20x more steps than our ﬁnetuning budget).  
Revised: To deﬁne expert performance we use the convention in URLB, which is the score achieved by a randomly initialized DDPG after 2M steps of ﬁnetuning (20x more steps than our ﬁnetuning budget).  R ESULTS

Revised: In this section we investigate empirical answers to the following research questions: (Q1)

Revised: How does

Revised: CIC adaptation efﬁciency compare to prior competence-based algorithms and exploration algorithms

Original: P r o j e c t e d
Revised: P r o j e c t e d

Original: N o t p r
Revised: N o t p r

Original: o j e c t e d 0.
Revised: o j e c t e d 0.

Original: N o r m a li z e d S c
Revised: N o r m a li z e d S c

Original: o r e (a) Skill projection8 

Revised: o r e (a) Skill projection

Original: 32 64 128 Skill dimension 0.
Revised: [Figure] [Figure]

Original: N o r m a li z e d s c o r e (b) Skill dimension
Revised: N o r m a li z e d s c o r e (b) Skill dimension

Original: Z e r o R a n d .
Revised: Z e r o R a n d .

Original: C E M B e t a C E M G a u s s .
Revised: C E M B e t a C E M G a u s s .

Original: G r i d S w e e p 0.
Revised: G r i d S w e e p [Figure]

Original:  N o r m a li z e d S c
Revised: [Figure] [Figure] N [Figure]

Original: o r e (c) Skill adaptation 0 . 0 0 .
Revised: li

Original: [Figure]
Revised: [Figure]

Revised: d

Original: [Figure]
Revised: [Figure]

Revised: (c) Skill adaptation

Original: [Figure]
Revised: [Figure]

Original: N o r m a li z e d S c
Revised: N o r m a li z e d S c

Original: o r e
Revised: o r e

Original: Quad.
Revised: Quad

Original: Stand Quad. Run (d) Skill grid sweep  R ESULTS In this section we investigate empirical answers to the following research questions: (Q1)
Revised: [Figure]

Original: How does
Revised: Quad

Original: CIC adaptation efﬁciency compare to prior competence-based algorithms and exploration algorithms more broadly? (Q2)
Revised: [Figure] Run (d) Skill grid sweep more broadly? (Q2)

Original: Qualitatively, does CIC discover structured skills and particularly is it able to do so in environments with high-dimensional state-action spaces?
Revised: Qualitatively, does CIC discover structured skills and particularly is it able to do so in environments with high-dimensional state-action spaces?

Original: (Q3)
Revised: (Q3)

Original: Quantitatively, how does
Revised: Quantitatively, how does

Original: CIC behavior compare to prior methods?
Revised: CIC behavior compare to prior methods?

Original: (Q4)
Revised: (Q4)

Original: Is skill selection important for efﬁcient adaptation to downstream tasks?
Revised: Is skill selection important for efﬁcient adaptation to downstream tasks?

Original: (Q5)
Revised: (Q5)

Original: How does the skill dimension affect the quality of the pre-trained policy?
Revised: How does the skill dimension affect the quality of the pre-trained policy?

Original: Adaptation efﬁciency of CIC and exploration baslines: Expert normalized scores of CIC and exploration algorithms from URLB are shown in Fig.
Revised: Adaptation efﬁciency of CIC and exploration baslines: Expert normalized scores of CIC and exploration algorithms from URLB are shown in Fig.

Original: We ﬁnd that CIC substantially outperforms prior competence-based algorithms (DIAYN, SMM, APS) achieving a 91% higher IQM than the next best competence-based method (APS) and, more broadly, achieving a 26% higher IQM than the next best overall baseline (ProtoRL).
Revised: We ﬁnd that CIC substantially outperforms prior competence-based algorithms (DIAYN, SMM, APS) achieving a 91% higher IQM than the next best competence-based method (APS) and, more broadly, achieving a 26% higher IQM than the next best overall baseline (ProtoRL).

Original: In further ablations, we ﬁnd that the contributing factors to
Revised: In further ablations, we ﬁnd that the contributing factors to

Original: CIC’s performance are its ability to accommodate substantially larger continuous skill spaces than prior competence-based methods.
Revised: CIC’s performance are its ability to accommodate substantially larger continuous skill spaces than prior competence-based methods.

Original: Qualitative analysis of CIC behaviors: We provide both qualitative and quantitative analyses of the behaviors learned with CIC .

Original: Qualitatively, we ﬁnd that CIC is able to learn locomotion behaviors in DMC without extrinsic information such as early termination as in OpenAI Gym.

Original: While most skills are higher entropy and thus more chaotic, we show in Fig 2, that structured behaviors can be isolated by ﬁxing a particular skill vector.

Original: For example, in the walker and quadruped domainsbalancing, walking, and ﬂipping skills can be isolated.

Original: For more qualitative investigations into the learned behaviors, we refer the reader to Appendix J.

Original: Quantitative analysis of CIC behaviors: Quantitatively, intrinsic reward proﬁles during pretraining and behavior entropies are shown in Fig 5.
Revised: Quantitative analysis of CIC behaviors: Quantitatively, intrinsic reward proﬁles during pretraining and behavior entropies are shown in Fig 5.

Original: Since the CPC and etnropy terms are on different scale, we pick a default hyparameter of α = 0 .
Revised: Since the CPC and etnropy terms are on different scale, we pick a default hyparameter of α = 0 .

Original: 9 that puts them on equal footing.
Revised: 9 that puts them on equal footing.

Original: We ﬁnd that
Revised: We ﬁnd that

Original: the
Revised: the

Original: CPC intrinsic reward increases through training while the entropy term decreases to a non-zero value.
Revised: CPC intrinsic reward increases through training while the entropy term decreases to a non-zero value.

Original: This is what we would expect to see as the discriminator distills behaviors into a set of skills with lower than pure entropy maximization without skill learning.
Revised: This is what we would expect to see as the discriminator distills behaviors into a set of skills with lower than pure entropy maximization without skill learning.

Original: Using a particle estimator for entropy, we ﬁnd that CIC behavioral entropy is less than APT and greater than DIAYN or CIC without the entropy term.
Revised: Using a particle estimator for entropy, we ﬁnd that CIC behavioral entropy is less than APT and greater than DIAYN or CIC without the entropy term.

Original: This suggests that the CIC agent has learned non-static skills while DIAYN skills are mostly static.
Revised: This suggests that the CIC agent has learned non-static skills while DIAYN skills are mostly static.

Original: Finally, we ﬁnd that CIC with α = 0 . 

Revised: Finally, we ﬁnd that CIC with α = 0 .

Original: (no entropy) on Jaco is optimal for downstream task performance.
Revised: (no entropy) on Jaco is optimal for downstream task performance.

Original: This is most likely because, unlike Walker and Quadruped, which require locomotion, Jaco tasks require reaching a certain position, so low-entropy skills that take the end effector to a certain end position are favorable to skills that result in periodic motion. 
Revised: This is most likely because, unlike Walker and Quadruped, which require locomotion, Jaco tasks require reaching a certain position, so low-entropy skills that take the end effector to a certain end position are favorable to skills that result in periodic motion. For a qualitative analysis, we refer the reader to Fig. 2 and Appendix J.

Original: Skill architecture and adaptation ablations:
Revised: Skill architecture and adaptation ablations:

Original: We ﬁnd that projecting the skill to a latent space before inputting it as the key for the contrastive loss is an important design decision (see Fig. 6), most likely because this reduces the diversity of the skill vectorhence making the discriminator task simpler.
Revised: We ﬁnd that projecting the skill to a latent space before inputting it as the key for the contrastive loss is an important design decision (see Fig. 6a), most likely because this reduces the diversity of the skill vector making the discriminator task simpler.

Revised: We also ﬁnd empirically that the skill dimension is an important hyperparameter and that larger skills results in better zero-shot performance (see Fig.

Revised: 6b), which empirically supports the hypothesis posed in Section 3 and Appendix I that larger skill spaces are important for internalizing diverse behaviors.

Revised: Interestingly, CIC zero-shot performance is poor in lower skill dimensions (e.g. dim ( z ) < 10 ), suggesting that when dim ( z ) is small CIC likely performs no better than prior competence-based methods such as DIAYN, and that scaling to larger skill dimensions enables CIC to pre-train effectively.

Original: To measure the effect of skill ﬁnetuning described in Section 5, we sweep mean skill values along the interval of the uniform prior [0 , 1] with a budget of 4k total environment interactions and read out the performance on the downstream task.
Revised: To measure the effect of skill ﬁnetuning described in Section 5, we sweep mean skill values along the interval of the uniform prior [0 , 1] with a budget of 4k total environment interactions and read out the performance on the downstream task.

Original: By sweeping, we mean simply iterating over the interval [0 , 1] with ﬁxed step size (e.g. v = 0 , 0 . 1 , . . .
Revised: By sweeping, we mean simply iterating over the interval [0 , 1] with ﬁxed step size (e.g. v = 0 , 0 . 1 , . . .

Original: and setting z i = v for all i .
Revised: and setting z i = v for all i .

Original: This is not an optimal skill sampling strategy but works well due to the extremely limited number of samples for skill selection.
Revised: This is not an optimal skill sampling strategy but works well due to the extremely limited number of samples for skill selection.

Original: We evaluate this ablation on the Quadruped Stand and Run downstream tasks.
Revised: We evaluate this ablation on the Quadruped Stand and Run downstream tasks.

Original: The results shown in Fig.
Revised: The results shown in Fig.

Original: 6 indicate that skill selection affects downstream task performance.
Revised: 6 indicate that skill selection affects downstream task performance.

Original: The most optimal skill results in 8 × better performance than the least optimal skill on Quadruped stand.
Revised: The most optimal skill results in 8 × better performance than the least optimal skill on Quadruped stand.

Original: Ablating the skill dimension, we evaluate the zero-shot performance of the agent in Walker Walk with a ﬁxed skill of 0 4 and ﬁnd that the zero-shot performance monotonically increases from skill dimension 4 until reaching dimension 64 and starts decreasing for even higher skill dimensions.
Revised: Ablating the skill dimension, we evaluate the zero-shot performance of the agent in Walker Walk with a ﬁxed skill of 0 4 and ﬁnd that the zero-shot performance monotonically increases from skill dimension 4 until reaching dimension 64 and starts decreasing for even higher skill dimensions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Supervised Reinforcement Learning:
Revised: Supervised Reinforcement Learning:

Original: To date, most of RL research has focused on supervised
Revised: To date, most of RL research has focused on supervised

Original: RL where training is supervised with an extrinsic reward function.
Revised: RL where training is supervised with an extrinsic reward function.

Original: Supervised RL has seen many breakthroughs over the last ﬁve years (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019;
Revised: Supervised RL has seen many breakthroughs over the last ﬁve years (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019;

Original: Silver et al., 2018; Berner et al., 2019; Andrychowicz et al., 2020; Schulman et al., 2016; 2017).
Revised: Silver et al., 2018; Berner et al., 2019; Andrychowicz et al., 2020; Schulman et al., 2016; 2017).

Original: The ﬁeld has also produced several stable RL optimization algorithms that have helped accelerated research Haarnoja et al.
Revised: The ﬁeld has also produced several stable RL optimization algorithms that have helped accelerated research Haarnoja et al.

Original: ; Hessel et al.
Revised: ; Hessel et al.

Original: ; Lillicrap et al.
Revised: ; Lillicrap et al.

Original: (2016); Schulman et al.
Revised: (2016); Schulman et al.

Original: Unsupervised Reinforcement Learning: The sub-ﬁeld of unsupervised RL consists of two primary research areas - unsupervised behavioral learning and unsupervised representation learning (Srinivas & Abbeel, 2021).
Revised: Unsupervised Reinforcement Learning: The sub-ﬁeld of unsupervised RL consists of two primary research areas - unsupervised behavioral learning and unsupervised representation learning (Srinivas & Abbeel, 2021).

Original: Unsupervised behavioral learning consists of learning behaviors and exploring the environment without extrinsic rewards.
Revised: Unsupervised behavioral learning consists of learning behaviors and exploring the environment without extrinsic rewards.

Original: Unsupervised representation learning consists of learning representations without supervision from high-dimensional data such as pixel observations.
Revised: Unsupervised representation learning consists of learning representations without supervision from high-dimensional data such as pixel observations.

Original: We evaluate our method on the recently introduced Unsupervised RL Benchmark (URLB) (Laskin et al., 2021), and focus solely on the behavioral aspect of unsupervised RL in order to isolate the core issue preventing prior unsupervised skill discovery methods from exploring effectively on URLB.
Revised: We evaluate our method on the recently introduced Unsupervised RL Benchmark (URLB) (Laskin et al., 2021), and focus solely on the behavioral aspect of unsupervised RL in order to isolate the core issue preventing prior unsupervised skill discovery methods from exploring effectively on URLB.

Original: Unsupervised Behavioral Learning:
Revised: Unsupervised Behavioral Learning:

Original: The aim of unsupervised behavioral learning is to produce diverse behaviors that explore the environment without interacting with an extrinsic reward.
Revised: The aim of unsupervised behavioral learning is to produce diverse behaviors that explore the environment without interacting with an extrinsic reward.

Original: Often referred to as intrinsic motivation (Oudeyer et al., 2007), this is typically achieved by deﬁning an intrinsic reward through a self-supervised task.
Revised: Often referred to as intrinsic motivation (Oudeyer et al., 2007), this is typically achieved by deﬁning an intrinsic reward through a self-supervised task.

Original: Most behavioral learning algorithms fall into three categories – knowledge-based (Pathak et al., 2017; 2019; Burda et al., 2019b;a) where the agent maximizes the error or uncertainty of some predictive model, data-based (Campos et al., 2021b;
Revised: Most behavioral learning algorithms fall into three categories – knowledge-based (Pathak et al., 2017; 2019; Burda et al., 2019b;a) where the agent maximizes the error or uncertainty of some predictive model, data-based (Campos et al., 2021b;

Original: Liu & Abbeel, 2021a;b; Mutti et al., 2021; Seo et al., 2021; Yarats et al., 2021b) where the agent maximizes data diversity, and competence-based Eysenbach et al.
Revised: Liu & Abbeel, 2021a;b; Mutti et al., 2021; Seo et al., 2021; Yarats et al., 2021b) where the agent maximizes data diversity, and competence-based Eysenbach et al.

Original: ; Hansen et al.
Revised: ; Hansen et al.

Original: ; Liu & Abbeel (2021b); Sharma et al.
Revised: ; Liu & Abbeel (2021b); Sharma et al.

Original: (2020) where the agent maximizes the mutual information between observable variables and a latent skill vector.
Revised: (2020) where the agent maximizes the mutual information between observable variables and a latent skill vector.

Original: We discuss the differences between CIC and the most closely related competence-based exploration algorithms in Appendix D.
Revised: We discuss the differences between CIC and the most closely related competence-based exploration algorithms in Appendix D.

Original: Unsupervised Representation Learning: Much progress in unsupervised representation learning for RL has been spurred by unsupervised learning in computer vision (Chen et al., 2020;
Revised: Unsupervised Representation Learning: Much progress in unsupervised representation learning for RL has been spurred by unsupervised learning in computer vision (Chen et al., 2020;

Original: He et al., 2020; H ´ enaff et al., 2020; Kingma & Welling, 2013) and language (Brown et al., 2020; Devlin et al., 2019; Radford et al., 2019).
Revised: He et al., 2020; H ´ enaff et al., 2020; Kingma & Welling, 2013) and language (Brown et al., 2020; Devlin et al., 2019; Radford et al., 2019).

Original: In RL, the most common approach for representation learning has been by adding it as an auxiliary loss in the supervised RL setting (Jaderberg et al., 2017).
Revised: In RL, the most common approach for representation learning has been by adding it as an auxiliary loss in the supervised RL setting (Jaderberg et al., 2017).

Original: More recently, a number of works have investiagted representaton learning with autoencoders (Yarats et al., 2019; Hafner et al., 2019; 2020), siamese networks (Schwarzer et al., 2021a;b; Laskin et al., 2020b; Stooke et al., 2021;
Revised: More recently, a number of works have investiagted representaton learning with autoencoders (Yarats et al., 2019; Hafner et al., 2019; 2020), siamese networks (Schwarzer et al., 2021a;b; Laskin et al., 2020b; Stooke et al., 2021;

Original: Yarats et al., 2021b), and data augmentation (Laskin et al., 2020a; Yarats et al., 2021a;c).
Revised: Yarats et al., 2021b), and data augmentation (Laskin et al., 2020a; Yarats et al., 2021a;c).

Original:  D ISCUSSION
Revised:  C ONCLUSION

Original: Limitations: While CIC achieves leading results on URLB, we would also like to address its limitations.

Original: First, in this paper we only consider MDPs (and not partially observed MDPs) where the full state is observable.

Original: We focus on MDPs because generating diverse behaviors in environments with large state spaces has been the primary bottleneck for competence-based exploration.

Original: Combining

Original: CIC with visual representation learning to scale this method to pixel-based inputs is a promising future direction for research not considered in this work.

Original: Another limitation is that our adaptation strategy to downstream tasks requires ﬁnetuning.

Original: Since we learn skills, it would be interesting to investigate alternate ways of adapting that would enable zero-shot generalization such as learning generalized reward functions during pre-training.

Original: Conclusion: We have introduced a new competence-based algorithm – Contrastive Intrinsic Control (CIC) – which enables more effective exploration than prior unsupervised skill discovery algorithms by explicitly encouraging diverse behavior while distilling predictable behaviors into skills with a contrastive discriminator.
Revised: We have introduced a new competence-based algorithm – Contrastive Intrinsic Control (CIC) – which enables more effective exploration than prior unsupervised skill discovery algorithms by explicitly encouraging diverse behavior while distilling predictable behaviors into skills with a contrastive discriminator.

Original: We showed that CIC is the ﬁrst competence-based approach to achieve leading performance on URLB.
Revised: We showed that CIC is the ﬁrst competence-based approach to achieve leading performance on URLB.

Original: We hope that this encourages further research in unsupervised skill discovery toward building more powerful exploration agents
Revised: We hope that this encourages further research in unsupervised skill discovery toward building more powerful exploration agents.

Original: A C OMPETENCE - BASED E XPLORATION A LGORITHMS
Revised: A C OMPETENCE - BASED E XPLORATION A LGORITHMS

Original: The competence-based algorithms considered in this work aim to maximize I ( τ ; s ) .
Revised: The competence-based algorithms considered in this work aim to maximize I ( τ ; s ) .

Original: The algorithms differ by ho they decompose mutual information, whether they explicitly maximize behavioral entropy, their skill space (discrete or continuous) and their intrinsic reward structure.
Revised: The algorithms differ by ho they decompose mutual information, whether they explicitly maximize behavioral entropy, their skill space (discrete or continuous) and their intrinsic reward structure.

Original: We provide a list of common competence-based algorithms in Table
Revised: We provide a list of common competence-based algorithms in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: describe the intrinsic reward optimized by each method and the decomposition of the mutual information utilized by the method.
Revised: describe the intrinsic reward optimized by each method and the decomposition of the mutual information utilized by the method.

Original: We also note whether the method explicitly maximizes state transition entropy.
Revised: We also note whether the method explicitly maximizes state transition entropy.

Original: Finally, we note the maximal dimension used in each work and whether the skills are discrete or continuous.
Revised: Finally, we note the maximal dimension used in each work and whether the skills are discrete or continuous.

Original: All methods prior to CIC only support small skill spaces, either because they are discrete or continuous but low-dimensional.
Revised: All methods prior to CIC only support small skill spaces, either because they are discrete or continuous but low-dimensional.

Original: B D EEP D ETERMINISTIC P OLICY G RADIENT (DDPG)
Revised: B D EEP D ETERMINISTIC P OLICY G RADIENT (DDPG)

Original: A DDPG is an actor-critic RL algorithm that performs off-policy gradient updates and learns a Q function Q φ ( s, a ) and an actor π θ ( a | s ) .
Revised: A DDPG is an actor-critic RL algorithm that performs off-policy gradient updates and learns a Q function Q φ ( s, a ) and an actor π θ ( a | s ) .

Original: The critic is trained by satisfying the Bellman equation.
Revised: The critic is trained by satisfying the Bellman equation.

Original: [Equation]
Revised: [Equation]

Original: Here, ¯ φ is the Polyak average of the parameters φ .
Revised: Here, ¯ φ is the Polyak average of the parameters φ .

Original: As the critic minimizes the Bellman error, the actor maximizes the action-value function.
Revised: As the critic minimizes the Bellman error, the actor maximizes the action-value function.

Original: [Equation]
Revised: [Equation]

Original: C B ASELINES
Revised: C B ASELINES

Original: For baselines, we choose the existing set of benchmarked unsupervised RL algorithms on URLB.
Revised: For baselines, we choose the existing set of benchmarked unsupervised RL algorithms on URLB.

Original: We provide a quick summary of each method.
Revised: We provide a quick summary of each method.

Original: For more detailed descriptions of each baseline we refer the reader to URLB (Laskin et al., 2021)
Revised: For more detailed descriptions of each baseline we refer the reader to URLB (Laskin et al., 2021)

Original: Competence-based Baselines: CIC is a competence-based exploration algorithm.
Revised: Competence-based Baselines: CIC is a competence-based exploration algorithm.

Original: For baselines, we compare it to DIAYN (Eysenbach et al., 2019), SMM (Lee et al., 2019), and APS (Liu & Abbeel, 2021b).
Revised: For baselines, we compare it to DIAYN (Eysenbach et al., 2019), SMM (Lee et al., 2019), and APS (Liu & Abbeel, 2021b).

Original: Each of these algorithms is described in Table 1.
Revised: Each of these algorithms is described in Table 1.

Original: Notably, APS is a recent state-of-the-art competence-based method that is the most closely related algorithm to the CIC algorithm.
Revised: Notably, APS is a recent state-of-the-art competence-based method that is the most closely related algorithm to the CIC algorithm.

Original: CIC and
Revised: CIC and

Original: APS differ in their discriminator.
Revised: APS differ in their discriminator.

Original: Knowledge-based Baselines
Revised: Knowledge-based Baselines

Original: : For knowledge-based baselines, we compare to ICM Pathak et al.
Revised: : For knowledge-based baselines, we compare to ICM Pathak et al.

Original: Disagreement Pathak et al.
Revised: Disagreement Pathak et al.

Original: (2019), and RND Burda et al.
Revised: (2019), and RND Burda et al.

Original: (2019b).
Revised: (2019b).

Original: ICM and RND train a dynamics model and random network prediction model and deﬁne the intrinsic reward to be proportional to the prediction error.
Revised: ICM and RND train a dynamics model and random network prediction model and deﬁne the intrinsic reward to be proportional to the prediction error.

Original: Disagreement trains an ensemble of dynamics models and deﬁnes the intrinsic reward to be proportional to the uncertainty of an ensemble.
Revised: Disagreement trains an ensemble of dynamics models and deﬁnes the intrinsic reward to be proportional to the uncertainty of an ensemble.

Original: Data-based Baselines: For data-based baselines we compare to APT (Liu & Abbeel, 2021a) and
Revised: Data-based Baselines: For data-based baselines we compare to APT (Liu & Abbeel, 2021a) and

Original: ProtoRL (Yarats et al., 2021b).
Revised: ProtoRL (Yarats et al., 2021b).

Original: Both methods use a particle estimator to estimate the state visitation entropy.
Revised: Both methods use a particle estimator to estimate the state visitation entropy.

Original: ProtoRL also performs discrete contrastive clustering as in Caron et al.
Revised: ProtoRL also performs discrete contrastive clustering as in Caron et al.

Original: (2020) as an auxiliary task
Revised: (2020) as an auxiliary task

Original: and uses the resulting clusters to compute the particle entropy.
Revised: and uses the resulting clusters to compute the particle entropy.

Original: While ProtoRL is more effective than APT when learning from pixels, on state-based URLB APT is competitive with ProtoRL.
Revised: While ProtoRL is more effective than APT when learning from pixels, on state-based URLB APT is competitive with ProtoRL.

Original: Our method CIC is effectively a skill-conditioned APT agent with a contrastive discriminator.
Revised: Our method CIC is effectively a skill-conditioned APT agent with a contrastive discriminator.

Original: D R ELATION TO P RIOR S KILL D ISCOVERY M ETHODS
Revised: D R ELATION TO P RIOR S KILL D ISCOVERY M ETHODS

Original: The most closely relatd prior algorithm to CIC is APS Liu & Abbeel (2021b).
Revised: The most closely relatd prior algorithm to CIC is APS Liu & Abbeel (2021b).

Original: Both CIC and APS use the H ( τ ) − H ( τ | z ) decomposition of the mutual information and both used a particle estimator (Singh et al., 2003) to compute the state entropy as in Liu & Abbeel (2021a).
Revised: Both CIC and APS use the H ( τ ) − H ( τ | z ) decomposition of the mutual information and both used a particle estimator (Singh et al., 2003) to compute the state entropy as in Liu & Abbeel (2021a).

Original: The main difference between CIC and APS is the discriminator.
Revised: The main difference between CIC and APS is the discriminator.

Original: APS uses successor features as in Hansen et al.
Revised: APS uses successor features as in Hansen et al.

Original: (2020) for its discriminator while CIC uses a noise contrastive estimator.
Revised: (2020) for its discriminator while CIC uses a noise contrastive estimator.

Original: Unlike successor features, which empirically only accommodate low-dimensional continuous skill spaces (see Table 1), the noise contrastive discriminator is able to leverage higher continuous dimensional skill vectors.
Revised: Unlike successor features, which empirically only accommodate low-dimensional continuous skill spaces (see Table 1), the noise contrastive discriminator is able to leverage higher continuous dimensional skill vectors.

Original: The CIC discriminator is similar to the one used in DISCERN (Warde-Farley et al., 2018), a goalcondition unsupervised RL algorithm.
Revised: The CIC discriminator is similar to the one used in DISCERN (Warde-Farley et al., 2018), a goalcondition unsupervised RL algorithm.

Original: Both methods use a contrastive discriminator by sampling negatives and computing an inner product between queries and keys.
Revised: Both methods use a contrastive discriminator by sampling negatives and computing an inner product between queries and keys.

Original: The main differences are (i) that DISCERN maximizes I ( τ ; g ) where g are image goal embeddings while CIC maximizes I ( τ ; z ) where z are abstract skill vectors; (ii) DISCERN uses the DIAYN-style decomposition I ( τ ; g )
Revised: The main differences are (i) that DISCERN maximizes I ( τ ; g ) where g are image goal embeddings while CIC maximizes I ( τ ; z ) where z are abstract skill vectors; (ii) DISCERN uses the DIAYN-style decomposition I ( τ ; g )

Original: =
Revised: =

Original: H ( g ) − H ( g | τ ) while CIC decomposes through H ( τ ) − H ( τ | z ) , and (iii) DISCERN discards the
Revised: H ( g ) − H ( g | τ ) while CIC decomposes through H ( τ ) − H ( τ | z ) , and (iii) DISCERN discards the

Original: H ( g ) term by sampling goals uniformly while CIC explicitly maximizes H ( τ ) .
Revised: H ( g ) term by sampling goals uniformly while CIC explicitly maximizes H ( τ ) .

Original: While DISCERN and CIC share similarities, DISCERN operates over image goals while CIC operates over abstrac skill vectors so the two methods are not directly comparable.
Revised: While DISCERN and CIC share similarities, DISCERN operates over image goals while CIC operates over abstrac skill vectors so the two methods are not directly comparable.

Original: Finally, another similar algorithm to CIC is DADS (Sharma et al., 2020) which also decomposes through H ( τ ) − H ( τ | z ) .
Revised: Finally, another similar algorithm to CIC is DADS (Sharma et al., 2020) which also decomposes through H ( τ ) − H ( τ | z ) .

Original: While CIC uses a contrastive density estimate for the discriminator, DADS uses a maximum likelihood estimator similar to DIAYN.
Revised: While CIC uses a contrastive density estimate for the discriminator, DADS uses a maximum likelihood estimator similar to DIAYN.

Original: DADS maximizes I ( s (cid:48) | s, z ) and estimates entropy H ( s (cid:48) | s ) by marginalizing over z such that H ( s (cid:48) | s )
Revised: DADS maximizes I ( s (cid:48) | s, z ) and estimates entropy H ( s (cid:48) | s ) by marginalizing over z such that H ( s (cid:48) | s )

Original: = − log (cid:80)
Revised: = − log (cid:80)

Original: i q ( s (cid:48) | s, z i ) while CIC uses a particle estimator.
Revised: i q ( s (cid:48) | s, z i ) while CIC uses a particle estimator.

Original: Interestingly, the DADS intrinsic reward r i ∝ log ( q ( s (cid:48) | s, z ) / (cid:80)
Revised: Interestingly, the DADS intrinsic reward r i ∝ log ( q ( s (cid:48) | s, z ) / (cid:80)

Original: i q ( s (cid:48) | s, z i )) looks similar to the CIC objective with zero entropy, since marginalizing over z to compute entropy is similar to sampling negatives for a contrastive discriminator.
Revised: i q ( s (cid:48) | s, z i )) looks similar to the CIC objective with zero entropy, since marginalizing over z to compute entropy is similar to sampling negatives for a contrastive discriminator.

Original: E F ULL CIC A LGORITHM
Revised: E F ULL CIC A LGORITHM

Original: The full CIC algorithm with both pre-training and ﬁne-tuning phases is shown in Algorithm 1.
Revised: The full CIC algorithm with both pre-training and ﬁne-tuning phases is shown in Algorithm 1.

Original: We pre-train CIC for 2M steps, and ﬁnetune it on each task for 100k steps.
Revised: We pre-train CIC for 2M steps, and ﬁnetune it on each task for 100k steps.

Original: Algorithm 1 Contrastive Intrinsic Control
Revised: Algorithm 1 Contrastive Intrinsic Control

Original: Require:
Revised: Require:

Original: Initialize all networks: encoders g ψ 1 and g ψ 2 , actor π θ , critic Q φ , replay buffer D .
Revised: Initialize all networks: encoders g ψ 1 and g ψ 2 , actor π θ , critic Q φ , replay buffer D .

Original: Require: Environment (env), M downstream tasks T k , k ∈ [1 , . . .
Revised: Require: Environment (env), M downstream tasks T k , k ∈ [1 , . . .

Original: , M ] .
Revised: , M ] .

Original: Require: pre-train N PT =
Revised: Require: pre-train N PT =

Original: 2 M and ﬁne-tune N FT = 100 K steps.
Revised: 2 M and ﬁne-tune N FT = 100 K steps.

Original: 1: for t = 1 ..N PT do (cid:46)
Revised: 1: for t = 1 ..N PT do (cid:46)

Original: Pre-training 2:
Revised: Pre-training 2:

Original: Sample and encode skill z ∼ p ( z and z ← g ψ 2 ( z ) 3:
Revised: Sample and encode skill z ∼ p ( z and z ← g ψ 2 ( z ) 3:

Original: Encode state s t ← g ψ 1 ( s t ) and sample action a t ← π θ ( s t , z ) + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) 4:
Revised: Encode state s t ← g ψ 1 ( s t ) and sample action a t ← π θ ( s t , z ) + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) 4:

Original: Observe next state s t +1 ∼ P ( ·| s t , a t ) 5:
Revised: Observe next state s t +1 ∼ P ( ·| s t , a t ) 5:

Original: Add transition to replay buffer D ← D ∪ ( s t , a t , s t
Revised: Add transition to replay buffer D ← D ∪ ( s t , a t , s t

Original: Sample a minibatch from D , compute contrastive loss
Revised: Sample a minibatch from D , compute contrastive loss

Original: Eq.3 and update encoders g ψ 1 , g ψ 2 , compute CIC intrinsic reward with Eq.
Revised: Eq.3 and update encoders g ψ 1 , g ψ 2 , compute CIC intrinsic reward with Eq.

Original: and update actor π θ and critic Q φ 7: end for 8: for T k ∈
Revised: and update actor π θ and critic Q φ 7: end for 8: for T k ∈

Original: [ T 1 , . . .
Revised: [ T 1 , . . .

Original: , T M ] do (cid:46)
Revised: , T M ] do (cid:46)

Original: Fine tuning 9:
Revised: Fine tuning 9:

Original: Initialize all networks with weights from pre training phase and an empty replay buffer D . 10: for t =
Revised: Initialize all networks with weights from pre training phase and an empty replay buffer D . 10: for t =

Original: 4 , 000 do 11:
Revised: 4 , 000 do 11:

Original: Take random action a t ∼ N (0 , 1) 12:
Revised: Take random action a t ∼ N (0 , 1) 12:

Original: Select skill with grid sweep over unit interval [0 , 1] every 100 steps 13:
Revised: Select skill with grid sweep over unit interval [0 , 1] every 100 steps 13:

Original: Sample minibatch from D and update actor π θ and critic Q φ 14: end for 15:
Revised: Sample minibatch from D and update actor π θ and critic Q φ 14: end for 15:

Original: Fix skill z that achieved highest extrinsic reward during grid sweep.
Revised: Fix skill z that achieved highest extrinsic reward during grid sweep.

Original: 16: for t = 4 , 000 . . .
Revised: 16: for t = 4 , 000 . . .

Original: N FT do 17:
Revised: N FT do 17:

Original: Encode state s t ← g ψ 1 ( s t ) and sample action a t ← π θ ( s t , z ) + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) 18:
Revised: Encode state s t ← g ψ 1 ( s t ) and sample action a t ← π θ ( s t , z ) + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) 18:

Original: Observe next state and reward s t +1 , r ext t ∼ P ( ·| s t , a t ) 19:
Revised: Observe next state and reward s t +1 , r ext t ∼ P ( ·| s t , a t ) 19:

Original: Add transition to replay buffer D ← D ∪ ( s t , a t , r ext t , s t +1 ) 20:
Revised: Add transition to replay buffer D ← D ∪ ( s t , a t , r ext t , s t +1 ) 20:

Original: Sample minibatch from D and update actor π θ and critic Q φ .
Revised: Sample minibatch from D and update actor π θ and critic Q φ .

Original: 21: end for 22:
Revised: 21: end for 22:

Original: Evaluate performance of RL agent on task T k 23: end for
Revised: Evaluate performance of RL agent on task T k 23: end for

Original: F H YPER - PARAMETERS
Revised: F H YPER - PARAMETERS

Original: Baseline hyperparameters are taken from URLB Laskin et al.
Revised: Baseline hyperparameters are taken from URLB Laskin et al.

Original: (2021), which were selected by performing a grid sweep over tasks and picking the best performing set of hyperparameters.
Revised: (2021), which were selected by performing a grid sweep over tasks and picking the best performing set of hyperparameters.

Original: Similarly, we also performed a grid sweep for CIC to pick the best performing set of hyperparameters.
Revised: Similarly, we also performed a grid sweep for CIC to pick the best performing set of hyperparameters.

Original: All hyperparameters are the same across all domains except for α which is set to α = 0 .
Revised: All hyperparameters are the same across all domains except for α which is set to α = 0 .

Original: 9 for Walker and Quadruped domains.
Revised: 9 for Walker and Quadruped domains.

Original: Note that α = 0 .
Revised: Note that α = 0 .

Original: 9 results in equal weighing of the CPC and particle entropy terms since their absolute values are on different scales.
Revised: 9 results in equal weighing of the CPC and particle entropy terms since their absolute values are on different scales.

Original: For Jaco, we found α = 0 .
Revised: For Jaco, we found α = 0 .

Original: to work best, which means that only the discriminator contributes to the intrinsic reward.
Revised: to work best, which means that only the discriminator contributes to the intrinsic reward.

Original: We hypothesize that particle entropy maximization is not important for Jaco arm because it is ﬁxed and has no way of falling over like Walker and Quadruped, such that meaningful behaviors can be learned with the discriminator alone.
Revised: We hypothesize that particle entropy maximization is not important for Jaco arm because it is ﬁxed and has no way of falling over like Walker and Quadruped, such that meaningful behaviors can be learned with the discriminator alone.

Original: [Table]
Revised: [Table]

Original: 64 → 1024 → 1024 → 64 ReLU MLP
Revised: 64 → 1024 → 1024 → 64 ReLU MLP

Original: Prediction net arch.
Revised: Prediction net arch.

Original: 64 → 1024 → 1024 → 64 ReLU MLP
Revised: 64 → 1024 → 1024 → 64 ReLU MLP

Original: G R AW N UMERICAL R ESULTS
Revised: G R AW N UMERICAL R ESULTS

Original: We provide a list of raw numerical results for ﬁnetuning CIC and baselines in Table
Revised: We provide a list of raw numerical results for ﬁnetuning CIC and baselines in Table

Original: [Table]
Revised: [Table]

Original: H L EARNING C URVES FOR D OWNSTREAM A DAPTATION P HASE
Revised: H L EARNING C URVES FOR D OWNSTREAM A DAPTATION P HASE

Original: [Figure]
Revised: [Figure]

Original: I T OY E XAMPLE TO I LLUSTRATE THE N EED FOR L ARGER S KILL S PACES
Revised: I T OY E XAMPLE TO I LLUSTRATE THE N EED FOR L ARGER S KILL S PACES

Original: We illustrate the need for larger skill spaces with a gridworld example.
Revised: We illustrate the need for larger skill spaces with a gridworld example.

Original: Suppose we have an agent in a 10 × 10 sized gridworld and that we have four discrete skills at our disposal.
Revised: Suppose we have an agent in a 10 × 10 sized gridworld and that we have four discrete skills at our disposal.

Original: Now let τ = s and consider how we may achieve maximal I ( τ ; z ) in this setting.
Revised: Now let τ = s and consider how we may achieve maximal I ( τ ; z ) in this setting.

Original: If we decompose I ( τ ; z ) =
Revised: If we decompose I ( τ ; z ) =

Original: H ( z ) − H ( z | τ ) then we can achieve maximal H ( z ) by sampling the four skills uniformly z ∼ p ( z ) .
Revised: H ( z ) − H ( z | τ ) then we can achieve maximal H ( z ) by sampling the four skills uniformly z ∼ p ( z ) .

Original: We can achieve H ( z | τ ) = 0 by mapping each skill to a distinct neighboring state of the agent.
Revised: We can achieve H ( z | τ ) = 0 by mapping each skill to a distinct neighboring state of the agent.

Original: Thus, our mutual information is maximized but as a result the agent only explores four out of the hundrend available states in the gridworld.
Revised: Thus, our mutual information is maximized but as a result the agent only explores four out of the hundrend available states in the gridworld.

Original: Now suppose we consider the second decomposition
Revised: Now suppose we consider the second decomposition

Original: I ( τ ; z ) =
Revised: I ( τ ; z ) =

Original: H ( τ ) − H ( τ | z ) .
Revised: H ( τ ) − H ( τ | z ) .

Original: Since the agent is maximizing H ( τ ) it is likely to visit a diverse set of states at ﬁrst.
Revised: Since the agent is maximizing H ( τ ) it is likely to visit a diverse set of states at ﬁrst.

Original: However, as soon as it learns an accurate discriminator we will have H ( τ | z ) and again the skills can be mapped to neighboring states to achieve minimal conditional entropy.
Revised: However, as soon as it learns an accurate discriminator we will have H ( τ | z ) and again the skills can be mapped to neighboring states to achieve minimal conditional entropy.

Original: As a result, the skill conditioned policy will only be able to reach four out of the hundrend possible states in this gridworld.
Revised: As a result, the skill conditioned policy will only be able to reach four out of the hundrend possible states in this gridworld.

Original: This argument is shown visually in
Revised: This argument is shown visually in

Original: Fig. 8.
Revised: Fig. 8.

Original: Skill spaces that are too large can also be an issue.
Revised: Skill spaces that are too large can also be an issue.

Original: Consider if we had 100 skills at our disposal in the same gridworld.
Revised: Consider if we had 100 skills at our disposal in the same gridworld.

Original: Then the agent could minimize the conditional entropy by mapping each skill to a unique state which would result in the agent memorizing the environment by ﬁnding a one-toone mapping between states and skills.
Revised: Then the agent could minimize the conditional entropy by mapping each skill to a unique state which would result in the agent memorizing the environment by ﬁnding a one-toone mapping between states and skills.

Original: While this is a potential issue it has not been encountered in practice yet since current competence-based methods support small skill spaces relative to the observation space of the environment.
Revised: While this is a potential issue it has not been encountered in practice yet since current competence-based methods support small skill spaces relative to the observation space of the environment.

Original: J Q UALITATIVE A NALYSIS OF S KILLS
Revised: J Q UALITATIVE A NALYSIS OF S KILLS

Original: We provide two additional qualitative analyses of behaviors learned with the CIC algorithm.
Revised: We provide two additional qualitative analyses of behaviors learned with the CIC algorithm.

Original: First, we take a simple pointmass setting and set the skill dimension to 1 in order to ablate the skills learned by the CIC agent in a simple setting.
Revised: First, we take a simple pointmass setting and set the skill dimension to 1 in order to ablate the skills learned by the CIC agent in a simple setting.

Original: We sweep over different values of z and plot the behavioral ﬂow vector ﬁeld (direction in which point mass moves) in Fig.9.
Revised: We sweep over different values of z and plot the behavioral ﬂow vector ﬁeld (direction in which point mass moves) in Fig.9.

Original: We ﬁnd that the pointmass learns skills that produce continuous motion and that the direction of the motion changes as a function of the skill value.
Revised: We ﬁnd that the pointmass learns skills that produce continuous motion and that the direction of the motion changes as a function of the skill value.

Original: Near the origin the pointmass learns skills that span all directions, while near the edges the point mass learns to avoid wall collisions.
Revised: Near the origin the pointmass learns skills that span all directions, while near the edges the point mass learns to avoid wall collisions.

Original: Qualitatively, many behaviors are periodic.
Revised: Qualitatively, many behaviors are periodic.

Original: [Figure]
Revised: [Figure]

Original: Qualitatively, we ﬁnd that methods like DIAYN that only support low dimensional skill vectors and do not explicitly incentivize diverse behaviors in their objective produce policies that map skills to a small set of static behaviors.
Revised: Qualitatively, we ﬁnd that methods like DIAYN that only support low dimensional skill vectors and do not explicitly incentivize diverse behaviors in their objective produce policies that map skills to a small set of static behaviors.

Original: These behaviors shown in Fig.
Revised: These behaviors shown in Fig.

Original: 10 are non-trivial but also have low behavioral diversity and are not particularly useful for solving the downstream task.
Revised: 10 are non-trivial but also have low behavioral diversity and are not particularly useful for solving the downstream task.

Original: This observation is consistent with Zahavy et al.
Revised: This observation is consistent with Zahavy et al.

Original: (2021) where the authors found that DIAYN maps to static “yoga” poses in DeepMind Control.
Revised: (2021) where the authors found that DIAYN maps to static “yoga” poses in DeepMind Control.

Original: In contrast, behaviors produce by CIC are dynamic resulting ﬂipping, jumping, and locomotive behaviors that can then be adapted to efﬁciently solve downstream tasks.
Revised: In contrast, behaviors produce by CIC are dynamic resulting ﬂipping, jumping, and locomotive behaviors that can then be adapted to efﬁciently solve downstream tasks.

Original: K O PEN AI G YM VS . D EEP M IND CONTROL : H OW E ARLY T ERMINATION L EAKS E XTRINSIC S IGNAL
Revised: K O PEN AI G YM VS . D EEP M IND CONTROL : H OW E ARLY T ERMINATION L EAKS E XTRINSIC S IGNAL

Original: Prior work on unsupervised skill discovery for continuous control (Eysenbach et al., 2019; Sharma et al., 2020) was evaluated on OpenAI Gym (Brockman et al., 2016) and showed diverse exploration on Gym environments.
Revised: Prior work on unsupervised skill discovery for continuous control (Eysenbach et al., 2019; Sharma et al., 2020) was evaluated on OpenAI Gym (Brockman et al., 2016) and showed diverse exploration on Gym environments.

Original: However, Gym environment episodes terminate early when the agent loses balance, thereby leaking information about the extrinsic task (e.g. balancing or moving).
Revised: However, Gym environment episodes terminate early when the agent loses balance, thereby leaking information about the extrinsic task (e.g. balancing or moving).

Original: However,
Revised: However,

Original: DeepMind Control (DMC) episodes have a ﬁxed length of 1k steps.
Revised: DeepMind Control (DMC) episodes have a ﬁxed length of 1k steps.

Original: In DMC, exploration is therefore harder since the agent needs to learn to balance without any extrinsic signal.
Revised: In DMC, exploration is therefore harder since the agent needs to learn to balance without any extrinsic signal.

Original: To evaluate whether the difference in the two environments has impact on competence-based exploration, we run DIAYN on the hopper environments from both Gym and DMC.
Revised: To evaluate whether the difference in the two environments has impact on competence-based exploration, we run DIAYN on the hopper environments from both Gym and DMC.

Original: We compare to ICM, a popular exploration baseline, and a Fixed baseline where the agent receives an intrinsic reward of 1 for each timestep and no algorithms receive extrinsic rewards.
Revised: We compare to ICM, a popular exploration baseline, and a Fixed baseline where the agent receives an intrinsic reward of 1 for each timestep and no algorithms receive extrinsic rewards.

Original: We then measure the extrinsic reward, which loosely corresponds to the diversity of behaviors learned.
Revised: We then measure the extrinsic reward, which loosely corresponds to the diversity of behaviors learned.

Original: Our results in Fig.
Revised: Our results in Fig.

Original: 11 show that indeed DIAYN is able to learn diverse behaviors in Gym but not in DMC while ICM is able to learn diverse behaviors in both environments.
Revised: 11 show that indeed DIAYN is able to learn diverse behaviors in Gym but not in DMC while ICM is able to learn diverse behaviors in both environments.

Original: Interestingly, the Fixed baseline achieves the highest reward on the Gym environment by learning to stand and balance.
Revised: Interestingly, the Fixed baseline achieves the highest reward on the Gym environment by learning to stand and balance.

Original: These results further motivate us to evaluate on URLB which is built on top of DMC.
Revised: These results further motivate us to evaluate on URLB which is built on top of DMC.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: L P SEUDOCODE FOR THE C ONTRASTIVE D ISCRIMINATOR IN CIC
Revised: L P SEUDOCODE FOR THE C ONTRASTIVE D ISCRIMINATOR IN CIC

Original: CIC consists of two terms I ( τ ; z ) =
Revised: CIC consists of two terms I ( τ ; z ) =

Original: H ( τ ) − H ( τ | z ) ≥ H ( τ ) + E [log q ( τ | z )] the entropy H ( τ ) is estimated with a particle estimator Singh et al.
Revised: H ( τ ) − H ( τ | z ) ≥ H ( τ ) + E [log q ( τ | z )] the entropy H ( τ ) is estimated with a particle estimator Singh et al.

Original: ; Liu & Abbeel (2021a) while the discriminator q ( τ | z ) is estimated with a contrastive loss introduced in this work.
Revised: ; Liu & Abbeel (2021a) while the discriminator q ( τ | z ) is estimated with a contrastive loss introduced in this work.

Original: Note that contrastive learning in CIC is different than prior vision-based contrastive learning such as CURL Laskin et al.
Revised: Note that contrastive learning in CIC is different than prior vision-based contrastive learning such as CURL Laskin et al.

Original: (2020b), since we are not performing contrastive learning over augmented images but rather over state transitions and skills.
Revised: (2020b), since we are not performing contrastive learning over augmented images but rather over state transitions and skills.

Original: The contrastive objective in CIC is used for unsupervised learning of behaviors while in CURL it is used for unsupervised learning of visual features.
Revised: The contrastive objective in CIC is used for unsupervised learning of behaviors while in CURL it is used for unsupervised learning of visual features.

Original: We provide pseudocode for discriminator below:
Revised: We provide pseudocode for discriminator below:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: − s t a t e s and s k
Revised: − s t a t e s and s k

Original: i l l s are sampled from replay buffer − s k i l l s were sampled from uniform
Revised: i l l s are sampled from replay buffer − s k i l l s were sampled from uniform

Original: d i s t [0 ,1] during agent r o l l o u t
Revised: d i s t [0 ,1] during agent r o l l o u t

Original: [Equation]
Revised: [Equation]

Original:  e s / n e x t s t [Equation] t e s
Revised: s t a t e s / n e x t s t a t e s

Original: [Equation]
Revised: [Equation]

Original: dim (B [Equation] D state )
Revised: dim (B, D state )

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: l l s
Revised: l l s

Original: [Equation]
Revised: [Equation]

Original: (B, D s k i l l )
Revised: (B, D s k i l l )

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure][Equation] t i
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: concat( s t [Equation] t e [Equation]
Revised: concat

Original: n

Original: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: n

Original: [Equation]
Revised: [Equation]

Original: 1) 10query =
Revised: ) 10query =

Original: t

Original: [Equation]
Revised: [Equation]

Original: ( t

Original: [Equation]

Original: n
Revised: l

Original: [Equation]

Original: [Equation]

Original: D

Original: [Equation]
Revised: [Equation]

Original: > (B, D hidden ) key =

Original: [Equation]
Revised: [Equation]

Original: k [Equation] l l s )
Revised: s )

Original: # (B,  D hidden ) − > (B, D hidden )
Revised: # (B, 2* D state ) − > (B, D hidden )

Original: 13query
Revised: 13query

Original: = normalize ( query , dim=1) key
Revised: = normalize ( query , dim=1) key

Original: = normalize ( key , dim=1)
Revised: = normalize ( key , dim=1)

Original: 16l
Revised: 16l

Original: o g i t s = matmul ( query , key .
Revised: o g i t s = matmul ( query , key .

Original: T) / temp # (B, B) 18# p o s i
Revised: T) / temp # (B, B) 18# p o s i

Original: t
Revised: t

Original: i v e s areon diagonal # negatives are off diagonal  l o s s = cross entropy ( l o gi t s ) 22r e t u r n l o s s
Revised: i v e s are diagonal , negatives are off diagonal # for each s k i l l negatives sampled from t r a i t i s k fixed l o s s = cross entropy ( l o g t s ) 23r e t u r n l o s s

Original:  We note that this is substantially different from prior contrastive learning works in RL such as
Revised: Listing We note that this is substantially different from prior contrastive learning works in RL such as

Original: CURL (Laskin et al., 2020b), which perform contrastive learning over images.
Revised: CURL (Laskin et al., 2020b), which perform contrastive learning over images.

Original: 1 def c u r l l o s s ( obs , W, temp ) : ””” − observation images are sampled from replay buffer− obs : dim (B, C, H, W) − W: p r o j e c t i o n matrix ( D hidden , D hidden ) ””” 7query
Revised: observation images are sampled from replay buffer

Original: = aug ( obs ) key = aug ( obs ) 10query = cnn net ( query ) # (B, D hidden ) key 
Revised: 10query = cnn net ( query ) # (B, D hidden ) key =

Original: = cnn net ( key ) # (B, D hidden ) 13l
Revised: cnn net ( key ) # (B, D hidden ) 13l

Original: o
Revised: o

Original: g i t s = matmul ( matmul ( query , W) , key .
Revised: g i t s = matmul ( matmul ( query , W) , key .

Original: T) / temp # (B, B) 15# p
Revised: T) / temp # (B, B) 15# p

Original: o
Revised: o

Original: s i
Revised: s i

Original: t
Revised: t

Original: i v e sare on diagonal # negatives are off diagonal l o s s = cross entropy ( l o g i t s ) 19r
Revised: i v e s diagonal # negatives are off diagonal l o s s = cross entropy ( l o g i t s ) 19r

Original: e t u r n l o s s
Revised: e t u r n l o s s

Original: Listing  contrastive loss
Revised: Listing 2: CURL contrastive loss

Revised: M O N TIGHTER ESTIMATES OF M UTUAL I NFORMATION

Revised: In this work we have presented CIC - a new competence-based algorithm that achieves leading performance on URLB compared to prior unsupervised RL methods.

Revised: We’ve shown that CIC results in a tighter lower bound on mutual information than CPC by including the entropy term.

Revised: One might wonder whether estimating the exact mutual information (MI) or maximizing the tightest lower bound thereof is really the goal for unsupervised RL.

Revised: In unsupervised representation learning, state-of-the-art methods like CPC and SimCLR maximize the lower bound of MI based on

Revised: Noise Contrastive Estimation (NCE).

Revised: However, as proven in CPC (Oord et al., 2018) and illustrated in Poole et al.

Revised: NCE is upper bounded by log N , meaning that the bound is loose when the MI is larger than log N .

Revised: Nevertheless, these methods have been repeatedly shown to excel in practice.

Revised: In Tschannen et al.

Revised: the authors show that the effectiveness of NCE results from the inductive bias in both the choice of feature extractor architectures and the parameterization of the employed

Revised: MI estimators.

Revised: We have a similar belief for unsupervised RL - that with the right parameterization and inductive bias, the MI objective will facilitate behavior learning in unsupervised RL.

Revised: This is why CIC lower bounds MI with (i) the particle based entropy estimator to ensure explicit exploration and (ii) a contrastive conditional entropy estimator to leverage the power of contrastive learning to discriminate skills.

Revised: As demonstrated in our experiments, CIC outperforms prior methods, showing the effectiveness of optimizing an intrinsic reward with the CIC MI estimator.

Revised: N L IMITATIONS

Revised: While CIC achieves leading results on URLB, we would also like to address its limitations.

Revised: First, in this paper we only consider MDPs (and not partially observed MDPs) where the full state is observable.

Revised: We focus on MDPs because generating diverse behaviors in environments with large state spaces has been the primary bottleneck for competence-based exploration.

Revised: Combining CIC with visual representation learning to scale this method to pixel-based inputs is a promising future direction for research not considered in this work.

Revised: Another limitation is that our adaptation strategy to downstream tasks requires ﬁnetuning.

Revised: Since we learn skills, it would be interesting to investigate alternate ways of adapting that would enable zero-shot generalization such as learning generalized reward functions during pre-training.