Original: L EARNING MORE SKILLS THROUGH OPTIMISTIC EXPLORATION
Revised: L EARNING MORE SKILLS THROUGH OPTIMISTIC EXPLORATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised skill learning objectives (Eysenbach et al., 2019; Gregor et al., 2016) allow agents to learn rich repertoires of behavior in the absence of extrinsic rewards.
Revised: Unsupervised skill learning objectives (Eysenbach et al., 2019; Gregor et al., 2016) allow agents to learn rich repertoires of behavior in the absence of extrinsic rewards.

Original: They work by simultaneously training a policy to produce distinguishable latent-conditioned trajectories, and a discriminator to evaluate distinguishability by trying to infer latents from trajectories.
Revised: They work by simultaneously training a policy to produce distinguishable latent-conditioned trajectories, and a discriminator to evaluate distinguishability by trying to infer latents from trajectories.

Original: The hope is for the agent to explore and master the environment by encouraging each skill (latent) to reliably reach different states.
Revised: The hope is for the agent to explore and master the environment by encouraging each skill (latent) to reliably reach different states.

Original: However, an inherent exploration problem lingers: when a novel state is actually encountered, the discriminator will necessarily not have seen enough training data to produce accurate and conﬁdent skill classiﬁcations, leading to low intrinsic reward for the agent and effective penalization of the sort of exploration needed to actually maximize the objective.
Revised: However, an inherent exploration problem lingers: when a novel state is actually encountered, the discriminator will necessarily not have seen enough training data to produce accurate and conﬁdent skill classiﬁcations, leading to low intrinsic reward for the agent and effective penalization of the sort of exploration needed to actually maximize the objective.

Original: To combat this inherent pessimism towards exploration, we derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement.
Revised: To combat this inherent pessimism towards exploration, we derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement.

Original: Our objective directly estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples, thus providing an intrinsic reward more tailored to the true objective compared to pseudocount-based methods (Burda et al., 2019).
Revised: Our objective directly estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples, thus providing an intrinsic reward more tailored to the true objective compared to pseudocount-based methods (Burda et al., 2019).

Original: We call this exploration bonus dis criminator d is a greement in trinsic reward, or DISDAIN .
Revised: We call this exploration bonus dis criminator d is a greement in trinsic reward, or DISDAIN .

Original: We demonstrate empirically that DISDAIN improves skill learning both in a tabular grid world (Four
Revised: We demonstrate empirically that DISDAIN improves skill learning both in a tabular grid world (Four

Original: Rooms) and the 57 games of the Atari Suite (from pixels).
Revised: Rooms) and the 57 games of the Atari Suite (from pixels).

Original: Thus, we encourage researchers to treat pessimism with DISDAIN.
Revised: Thus, we encourage researchers to treat pessimism with DISDAIN.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has proven itself capable of learning useful skills when clear taskspeciﬁc rewards are available (OpenAI et al., 2019a;b; Vinyals et al., 2019).
Revised: Reinforcement learning (RL) has proven itself capable of learning useful skills when clear taskspeciﬁc rewards are available (OpenAI et al., 2019a;b; Vinyals et al., 2019).

Original: However, truly intelligent agents should, like humans, be able to learn even in the absence of supervision in order to acquire repurposable task-agnostic knowledge.
Revised: However, truly intelligent agents should, like humans, be able to learn even in the absence of supervision in order to acquire repurposable task-agnostic knowledge.

Original: Such unsupervised pre-training has seen recent success in language (Radford et al., 2019; Brown et al., 2020) and
Revised: Such unsupervised pre-training has seen recent success in language (Radford et al., 2019; Brown et al., 2020) and

Original: vision (Chen et al., 2020a;b), but its potential has yet to be fully realized in the learning of behavior.
Revised: vision (Chen et al., 2020a;b), but its potential has yet to be fully realized in the learning of behavior.

Original: The most promising class of algorithms for unsupervised skill discovery is based upon maximizing the discriminability of skills represented by latent variables on which a policy is conditioned.
Revised: The most promising class of algorithms for unsupervised skill discovery is based upon maximizing the discriminability of skills represented by latent variables on which a policy is conditioned.

Original: Objectives are typically derived from variational approximations to the mutual information between latent variables and states visited (Gregor et al., 2016;
Revised: Objectives are typically derived from variational approximations to the mutual information between latent variables and states visited (Gregor et al., 2016;

Original: Eysenbach et al., 2019; Warde-Farley et al., 2019;
Revised: Eysenbach et al., 2019; Warde-Farley et al., 2019;

Original: Hansen et al., 2020; Baumli et al., 2021), employing a learned parametric skill discriminator.
Revised: Hansen et al., 2020; Baumli et al., 2021), employing a learned parametric skill discriminator.

Original: Both the policy and discriminator have the objective of strong predictive performance by the discriminator, though the discriminator is trained with supervised learning rather than RL.
Revised: Both the policy and discriminator have the objective of strong predictive performance by the discriminator, though the discriminator is trained with supervised learning rather than RL.

Original: The end result is a policy capable of producing consistently distinguishable behavioral motifs, or “skills.”
Revised: The end result is a policy capable of producing consistently distinguishable behavioral motifs, or “skills.”

Original: These skills can be evaluated zero-shot, ﬁne-tuned, or composed in a hierarchical RL setup to maximize task reward when one is introduced (Eysenbach et al., 2019;
Revised: These skills can be evaluated zero-shot, ﬁne-tuned, or composed in a hierarchical RL setup to maximize task reward when one is introduced (Eysenbach et al., 2019;

Original: Hansen et al., 2020).
Revised: Hansen et al., 2020).

Original: Unsupervised skill learning objectives can also be maximized in conjunction with task reward, in order to promote robustness of learned behavior to environment perturbations (Mahajan et al., 2019; Kumar et al., 2020).
Revised: Unsupervised skill learning objectives can also be maximized in conjunction with task reward, in order to promote robustness of learned behavior to environment perturbations (Mahajan et al., 2019; Kumar et al., 2020).

Original: The degree to which unsupervised skill discovery methods are useful in such downstream applications depends on how many skills they are able to learn.
Revised: The degree to which unsupervised skill discovery methods are useful in such downstream applications depends on how many skills they are able to learn.

Original: Indeed, Eysenbach et al.
Revised: Indeed, Eysenbach et al.

Original: (2019) showed that as more skills are learned, the performance obtained using the learned skills in a hierarchical reinforcement learning setup improves.
Revised: (2019) showed that as more skills are learned, the performance obtained using the learned skills in a hierarchical reinforcement learning setup improves.

Original: In follow up work, Achiam et al.
Revised: In follow up work, Achiam et al.

Original: (2018) showed that methods like DIAYN can struggle to learn large numbers of skills and proposed gradually increasing the number of skills according to a curriculum to make skill discovery easier.
Revised: (2018) showed that methods like DIAYN can struggle to learn large numbers of skills and proposed gradually increasing the number of skills according to a curriculum to make skill discovery easier.

Original: The aim of this work is to improve the ability of skill discovery methods to learn more skills.
Revised: The aim of this work is to improve the ability of skill discovery methods to learn more skills.

Original: We highlight an exploration problem intrinsic to the entire class of variational skill discovery algorithms which can inhibit discovery of new skills.
Revised: We highlight an exploration problem intrinsic to the entire class of variational skill discovery algorithms which can inhibit discovery of new skills.

Original: During skill learning, the policy will necessarily need to explore new states of the environment.
Revised: During skill learning, the policy will necessarily need to explore new states of the environment.

Original: The discriminator must then make latent predictions for states it has never seen before, resulting in incorrect and/or low-conﬁdence predictions.
Revised: The discriminator must then make latent predictions for states it has never seen before, resulting in incorrect and/or low-conﬁdence predictions.

Original: The policy will in turn be penalized for this poor discriminator performance, and discouraged from seeking out new states.
Revised: The policy will in turn be penalized for this poor discriminator performance, and discouraged from seeking out new states.

Original: We refer to this problem as “pessimistic exploration” and describe it further in section 2.
Revised: We refer to this problem as “pessimistic exploration” and describe it further in section 2.

Original: To motivate our solution, we argue that it is important for the policy to distinguish between two kinds of uncertainty in the discriminator - aleatoric uncertainty that comes from the policy producing similar trajectories for different skills, and epistemic uncertainty that comes from a lack of training data.
Revised: To motivate our solution, we argue that it is important for the policy to distinguish between two kinds of uncertainty in the discriminator - aleatoric uncertainty that comes from the policy producing similar trajectories for different skills, and epistemic uncertainty that comes from a lack of training data.

Original: The former indicates poor policy performance, but the latter is in fact desirable, and serves as a signal for potential discriminator learning.
Revised: The former indicates poor policy performance, but the latter is in fact desirable, and serves as a signal for potential discriminator learning.

Original: Unfortunately, skill discovery algorithms treat both types of uncertainty the same and thus ignore this important signal.
Revised: Unfortunately, skill discovery algorithms treat both types of uncertainty the same and thus ignore this important signal.

Original: We propose to capture it.
Revised: We propose to capture it.

Original: Our primary contribution, which we present in section 3, is an exploration bonus tailored to skill discovery algorithms, designed to overcome pessimistic exploration.
Revised: Our primary contribution, which we present in section 3, is an exploration bonus tailored to skill discovery algorithms, designed to overcome pessimistic exploration.

Original: We identify states of high epistemic uncertainty in the discriminator by training an ensemble of discriminators and measuring their disagreement.
Revised: We identify states of high epistemic uncertainty in the discriminator by training an ensemble of discriminators and measuring their disagreement.

Original: We call this exploration bonus dis criminator d is a greement in trinsic reward, or
Revised: We call this exploration bonus dis criminator d is a greement in trinsic reward, or

Original: DISDAIN .
Revised: DISDAIN .

Original: Intuitively, the ensemble members may disagree in novel states, but must come to agree in frequently visited ones.
Revised: Intuitively, the ensemble members may disagree in novel states, but must come to agree in frequently visited ones.

Original: More formally, we derive DISDAIN from a Bayesian perspective that 1) represents the posterior over discriminator parameters using an ensemble, and 2) encourages the policy to maximize information gain (i.e. reduce uncertainty) about the parameters of the discriminator.
Revised: More formally, we derive DISDAIN from a Bayesian perspective that 1) represents the posterior over discriminator parameters using an ensemble, and 2) encourages the policy to maximize information gain (i.e. reduce uncertainty) about the parameters of the discriminator.

Original: In section 4, we demonstrate empirically that DISDAIN improves skill learning over unbonused skill discovery algorithms in both an illustrative grid world (Four Rooms, Sutton et al.
Revised: In section 4, we demonstrate empirically that DISDAIN improves skill learning over unbonused skill discovery algorithms in both an illustrative grid world (Four Rooms, Sutton et al.

Original: and the Atari 2600 learning environment (ALE, Bellemare et al. (2013)) more so than augmenting with popular exploration bonuses not tailored to skill discovery.
Revised: and the Atari 2600 learning environment (ALE, Bellemare et al. (2013)) more so than augmenting with popular exploration bonuses not tailored to skill discovery.

Original:  U NSUPERVISED SKILL LEARNING THROUGH VARIATIONAL INFOMAX 2. I NTRODUCTION
Revised:  U NSUPERVISED SKILL LEARNING THROUGH VARIATIONAL INFOMAX 2. I NTRODUCTION

Original: We now formalize our setting of interest: unsupervised skill learning through variational information maximization (Gregor et al., 2016;
Revised: We now formalize our setting of interest: unsupervised skill learning through variational information maximization (Gregor et al., 2016;

Original: Eysenbach et al., 2019; Warde-Farley et al., 2019;
Revised: Eysenbach et al., 2019; Warde-Farley et al., 2019;

Original: Hansen et al., 2020; Baumli et al., 2021).
Revised: Hansen et al., 2020; Baumli et al., 2021).

Original: We consider a Markov decision process (MDP), deﬁned by the tuple
Revised: We consider a Markov decision process (MDP), deﬁned by the tuple

Original: M = ( S , A , p E , ρ, r, γ ) , where S and A are state and action spaces, respectively, the environment dynamics p E ( s (cid:48) | s, a ) speciﬁes the probability of transitioning to state s (cid:48) ∈ S when taking action a ∈ A in state s ∈ S , ρ ( s ) denotes the probability of starting an episode in state s , and γ ∈ [0 , 1) is a discount factor.
Revised: M = ( S , A , p E , ρ, r, γ ) , where S and A are state and action spaces, respectively, the environment dynamics p E ( s (cid:48) | s, a ) speciﬁes the probability of transitioning to state s (cid:48) ∈ S when taking action a ∈ A in state s ∈ S , ρ ( s ) denotes the probability of starting an episode in state s , and γ ∈ [0 , 1) is a discount factor.

Original: Since we focus on unsupervised training, we ignore the environmental reward r .
Revised: Since we focus on unsupervised training, we ignore the environmental reward r .

Original: Our agents seek to learn a repertoire of skills, indexed by the latent variable Z and represented by the policy π θ ( a | s, z ) , which is parameterized by θ and maps from states and latent variables to distributions over actions.
Revised: Our agents seek to learn a repertoire of skills, indexed by the latent variable Z and represented by the policy π θ ( a | s, z ) , which is parameterized by θ and maps from states and latent variables to distributions over actions.

Original: The latent variables are sampled z ∼ p ( Z ) at the beginning of each trajectory and then ﬁxed, so each z represents a temporally extended behavior.
Revised: The latent variables are sampled z ∼ p ( Z ) at the beginning of each trajectory and then ﬁxed, so each z represents a temporally extended behavior.

Original: The skill trajectory length T may differ from the episode length, thus a new skill might be resampled within an episode.
Revised: The skill trajectory length T may differ from the episode length, thus a new skill might be resampled within an episode.

Original: For conciseness, we will denote trajectories sampled from the policy by τ ∼ π ( z ) when conditioning on a particular skill z , and τ ∼ π when collecting trajectories across skills.
Revised: For conciseness, we will denote trajectories sampled from the policy by τ ∼ π ( z ) when conditioning on a particular skill z , and τ ∼ π when collecting trajectories across skills.

Original: To simplify our discussion, and because it is the most common case in practice, we will assume that Z is categorical with cardinality N Z , although much of the discussion carries over to continuous Z .
Revised: To simplify our discussion, and because it is the most common case in practice, we will assume that Z is categorical with cardinality N Z , although much of the discussion carries over to continuous Z .

Original: A large and growing class of objectives for unsupervised skill discovery are derived from maximizing the mutual information between the latent skill Z and some feature of the resulting trajectories
Revised: A large and growing class of objectives for unsupervised skill discovery are derived from maximizing the mutual information between the latent skill Z and some feature of the resulting trajectories

Original: O ( τ ) :
Revised: O ( τ ) :

Original: [Equation]
Revised: [Equation]

Original: For example, variational intrinsic control (VIC, Gregor et al.
Revised: For example, variational intrinsic control (VIC, Gregor et al.

Original: (2016)) maximizes I ( Z ; S 0 , S T ) - the mutual information between the skill and initial and ﬁnal states ( o T = ( s 0 , s T ) ).
Revised: (2016)) maximizes I ( Z ; S 0 , S T ) - the mutual information between the skill and initial and ﬁnal states ( o T = ( s 0 , s T ) ).

Original: 1 Diversity is all you need (DIAYN, Eysenbach et al.
Revised: 1 Diversity is all you need (DIAYN, Eysenbach et al.

Original: (2019)), on the other hand, maximizes I ( Z, S ) - the mutual information between the skill and each state along the trajectory ( o t = s t for t ∈ 1 : T ).
Revised: (2019)), on the other hand, maximizes I ( Z, S ) - the mutual information between the skill and each state along the trajectory ( o t = s t for t ∈ 1 : T ).

Original: Intuitively,
Revised: Intuitively,

Original: VIC produces skills that vary in the destination reached (without regard for the path taken), while
Revised: VIC produces skills that vary in the destination reached (without regard for the path taken), while

Original: DIAYN produces skills that vary in the path taken (with less emphasis on the destination reached).
Revised: DIAYN produces skills that vary in the path taken (with less emphasis on the destination reached).

Original: In practice, maximizing equation 1 is not straightforward, because it requires calculating the conditional distribution p ( Z | O ) .
Revised: In practice, maximizing equation 1 is not straightforward, because it requires calculating the conditional distribution p ( Z | O ) .

Original: In general, it is necessary to estimate it with a learned parametric model q φ ( Z | O ) .
Revised: In general, it is necessary to estimate it with a learned parametric model q φ ( Z | O ) .

Original: We refer to this model as the discriminator , since it is trained to discriminate between skills.
Revised: We refer to this model as the discriminator , since it is trained to discriminate between skills.

Original: Fortunately, replacing p with q still yields a lower bound on F ( θ ) (Barber and Agakov, 2004), and we may instead maximize the proxy objective ˜ F ( θ ) :
Revised: Fortunately, replacing p with q still yields a lower bound on F ( θ ) (Barber and Agakov, 2004), and we may instead maximize the proxy objective ˜ F ( θ ) :

Original: [Equation]
Revised: [Equation]

Original: Optimizing ˜ F ( θ ) with respect to the policy parameters θ corresponds to RL on the reward:
Revised: Optimizing ˜ F ( θ ) with respect to the policy parameters θ corresponds to RL on the reward:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the intent is for the agent to learn a full repertoire of skills, the skill prior p ( z ) is typically ﬁxed to be uniform (Eysenbach et al., 2019; Achiam et al., 2018; Baumli et al., 2021), in which case − log p ( z )
Revised: Since the intent is for the agent to learn a full repertoire of skills, the skill prior p ( z ) is typically ﬁxed to be uniform (Eysenbach et al., 2019; Achiam et al., 2018; Baumli et al., 2021), in which case − log p ( z )

Original: = log N Z .
Revised: = log N Z .

Original: If the discriminator simply ignores the trajectory and guesses skills uniformly as well, then log q φ ( z | o )
Revised: If the discriminator simply ignores the trajectory and guesses skills uniformly as well, then log q φ ( z | o )

Original: = − log N Z and the reward will be zero.
Revised: = − log N Z and the reward will be zero.

Original: If the discriminator instead guesses perfectly, then log q φ ( z | o ) = 0 , and the reward will be log N Z .
Revised: If the discriminator instead guesses perfectly, then log q φ ( z | o ) = 0 , and the reward will be log N Z .

Original: More generally, the expected reward is an estimate of the logarithm of the effective number of skills.
Revised: More generally, the expected reward is an estimate of the logarithm of the effective number of skills.

Original: Thus, measuring the reward in bits (i.e. using log 2 in equation 3), we can estimate the number of skills learnt as:
Revised: Thus, measuring the reward in bits (i.e. using log 2 in equation 3), we can estimate the number of skills learnt as:

Original: [Equation]
Revised: [Equation]

Original: We adopt this quantity as our primary performance metric for our experiments.
Revised: We adopt this quantity as our primary performance metric for our experiments.

Original: To make sure the bound in equation 2 is as tight as possible, the discriminator q φ ( Z | O ) must also be ﬁt to its target p ( Z | O ) through supervised learning (SL) on the negative log likelihood loss:
Revised: To make sure the bound in equation 2 is as tight as possible, the discriminator q φ ( Z | O ) must also be ﬁt to its target p ( Z | O ) through supervised learning (SL) on the negative log likelihood loss:

Original: [Equation]
Revised: [Equation]

Original: The loss is minimized, and the bound in equation 2 tight, when q φ ( Z | O )
Revised: The loss is minimized, and the bound in equation 2 tight, when q φ ( Z | O )

Original: = p ( Z | O ) .
Revised: = p ( Z | O ) .

Original: The joint optimization of ˜ F ( θ ) by RL and L ( φ ) by SL forms a cooperative communication game between policy and discriminator.
Revised: The joint optimization of ˜ F ( θ ) by RL and L ( φ ) by SL forms a cooperative communication game between policy and discriminator.

Original: The agent samples a skill z ∼ p
Revised: The agent samples a skill z ∼ p

Original: ( Z )
Revised: ( Z )

Original: and generates the “message” τ ∼ π ( z ) .
Revised: and generates the “message” τ ∼ π ( z ) .

Original: The discriminator receives the message τ and attempts to decode the original skill z .
Revised: The discriminator receives the message τ and attempts to decode the original skill z .

Original: When the policy produces trajectories for different skills that do not overlap in the features O ( τ ) , the discriminator will easily learn to label trajectories, and when the discriminator makes accurate and conﬁdent predictions, the reward in equation 3 will be high.
Revised: When the policy produces trajectories for different skills that do not overlap in the features O ( τ ) , the discriminator will easily learn to label trajectories, and when the discriminator makes accurate and conﬁdent predictions, the reward in equation 3 will be high.

Original: Ideally, the end result is a policy exhibiting a maximally diverse set of skills.
Revised: Ideally, the end result is a policy exhibiting a maximally diverse set of skills.

Original: This joint-training system is depicted in Figure 2a.
Revised: This joint-training system is depicted in Figure 2a.

Original: 2. P ESSIMISTIC EXPLORATION IN UNSUPERVISED SKILL DISCOVERY
Revised: 2. P ESSIMISTIC EXPLORATION IN UNSUPERVISED SKILL DISCOVERY

Original: A conﬂict arises between the exploration necessary for skill diversiﬁcation and rewards supplied by an imperfect discriminator, trained only on past policy-generated experience.
Revised: A conﬂict arises between the exploration necessary for skill diversiﬁcation and rewards supplied by an imperfect discriminator, trained only on past policy-generated experience.

Original: Without data (and in the absence of perfect generalization), the discriminator is likely to make poor predictions when presented with trajectories containing previously unseen states, resulting in low reward for the policy.
Revised: Without data (and in the absence of perfect generalization), the discriminator is likely to make poor predictions when presented with trajectories containing previously unseen states, resulting in low reward for the policy.

Original: Importantly, this penalization occurs regardless of whether the policy produces distinguishable skill-conditioned trajectories if the current discriminator is a locally poor approximation to p ( Z | O ) for a region of state space represented in O .
Revised: Importantly, this penalization occurs regardless of whether the policy produces distinguishable skill-conditioned trajectories if the current discriminator is a locally poor approximation to p ( Z | O ) for a region of state space represented in O .

Original: We note that this is distinct from issues of pessimism in exploration that arise more generally, including with stationary reward functions (Osband et al., 2019), wherein naive exploration strategies fail to adequately position an agent for further acquisition of information.
Revised: We note that this is distinct from issues of pessimism in exploration that arise more generally, including with stationary reward functions (Osband et al., 2019), wherein naive exploration strategies fail to adequately position an agent for further acquisition of information.

Original: In the scenario we examine here, the agent’s sole source of supervision directly sabotages the learning process when the discriminator extrapolates poorly.
Revised: In the scenario we examine here, the agent’s sole source of supervision directly sabotages the learning process when the discriminator extrapolates poorly.

Original: Aleatoric uncertainty is created by indistinguishable states for different skills.
Revised: Aleatoric uncertainty is created by indistinguishable states for different skills.

Original: Epistemic uncertainty is created by exploring novel states.
Revised: Epistemic uncertainty is created by exploring novel states.

Original: Buffer of previously seen states.
Revised: Buffer of previously seen states.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: We argue that in order to overcome this inherent pessimism, we must distinguish between two kinds of uncertainty in the discriminator: aleatoric uncertainty (ﬁgure 1b) that is due to the policy producing overlapping skills (i.e. high H ( Z | O ) ), and epistemic uncertainty (ﬁgure 1c) that is due to a lack of training data (i.e. poor match between q φ ( Z | O ) and p ( Z | O ) ).
Revised: We argue that in order to overcome this inherent pessimism, we must distinguish between two kinds of uncertainty in the discriminator: aleatoric uncertainty (ﬁgure 1b) that is due to the policy producing overlapping skills (i.e. high H ( Z | O ) ), and epistemic uncertainty (ﬁgure 1c) that is due to a lack of training data (i.e. poor match between q φ ( Z | O ) and p ( Z | O ) ).

Original: Naively, both kinds of uncertainty contribute to low reward for the policy (equation 3), but while reduction of aleatoric uncertainty requires changes by the policy, epistemic uncertainty can be reduced (and thus reward increased) simply by creating more data (i.e. visiting the same states again).
Revised: Naively, both kinds of uncertainty contribute to low reward for the policy (equation 3), but while reduction of aleatoric uncertainty requires changes by the policy, epistemic uncertainty can be reduced (and thus reward increased) simply by creating more data (i.e. visiting the same states again).

Original: Thus, we argue that we should “reimburse” the policy for epistemic uncertainty in the discriminator, and in fact encourage the policy to visits states of high epistemic uncertainty.
Revised: Thus, we argue that we should “reimburse” the policy for epistemic uncertainty in the discriminator, and in fact encourage the policy to visits states of high epistemic uncertainty.

Original: Put another way, when the policy only maximizes the reward of equation 3, it maximizes the lower bound ˜ F ( θ ) ≤ F ( θ ) without regard for its tightness, despite that a looser bound means a more pessimistic reward.
Revised: Put another way, when the policy only maximizes the reward of equation 3, it maximizes the lower bound ˜ F ( θ ) ≤ F ( θ ) without regard for its tightness, despite that a looser bound means a more pessimistic reward.

Original: The job of keeping the bound tight is left entirely to the discriminator (equation 5).
Revised: The job of keeping the bound tight is left entirely to the discriminator (equation 5).

Original: By incentivizing the policy to visit states of high epistemic uncertainty, valuable training data is provided to the discriminator that allows it to better approximate its target and close the gap between ˜
Revised: By incentivizing the policy to visit states of high epistemic uncertainty, valuable training data is provided to the discriminator that allows it to better approximate its target and close the gap between ˜

Original: F ( θ ) and F ( θ ) .
Revised: F ( θ ) and F ( θ ) .

Original: In this way, we encourage the policy to help keep the bound ˜ F ( θ ) ≤ F ( θ ) tight.
Revised: In this way, we encourage the policy to help keep the bound ˜ F ( θ ) ≤ F ( θ ) tight.

Original: In the next section, we formalize the intuitions outlined in the last two paragraphs.
Revised: In the next section, we formalize the intuitions outlined in the last two paragraphs.

Original: The result is an exploration bonus measured as the disagreement among an ensemble of discriminators.
Revised: The result is an exploration bonus measured as the disagreement among an ensemble of discriminators.

Original:  DISDAIN: D ISCRIMINATOR D ISAGREEMENT I NTRINSIC R EWARD
Revised:  DISDAIN: D ISCRIMINATOR D ISAGREEMENT I NTRINSIC R EWARD

Original: To formalize the notion of discriminator uncertainty, we take a Bayesian approach and replace the point estimate of discriminator parameters φ with a posterior p ( φ ) .
Revised: To formalize the notion of discriminator uncertainty, we take a Bayesian approach and replace the point estimate of discriminator parameters φ with a posterior p ( φ ) .

Original: Maintaining a posterior allows us to quantify the information gained about those parameters.
Revised: Maintaining a posterior allows us to quantify the information gained about those parameters.

Original: Speciﬁcally, we will incentivize the policy to produce trajectories in which observing the paired skill label z provides maximal information about the discriminator parameters φ :
Revised: Speciﬁcally, we will incentivize the policy to produce trajectories in which observing the paired skill label z provides maximal information about the discriminator parameters φ :

Original: [Equation]
Revised: [Equation]

Original: Rewriting the entropies as expectations over trajectories, we have:
Revised: Rewriting the entropies as expectations over trajectories, we have:

Original: [Equation]
Revised: [Equation]

Original: where we have used that p ( φ ) does not depend on the present trajectory and so p ( φ | o ( τ ))
Revised: where we have used that p ( φ ) does not depend on the present trajectory and so p ( φ | o ( τ ))

Original: =
Revised: =

Original: p ( φ ) .
Revised: p ( φ ) .

Original: Note that unlike in equations 2 and 5, the expectation over trajectories is not skill-conditioned.
Revised: Note that unlike in equations 2 and 5, the expectation over trajectories is not skill-conditioned.

Original: The marginalization over Z happens in the entropy and is over the discriminator’s posterior q φ ( z | s ) rather than the agent’s prior p ( z ) .
Revised: The marginalization over Z happens in the entropy and is over the discriminator’s posterior q φ ( z | s ) rather than the agent’s prior p ( z ) .

Original: This is because the discriminator parameters Φ are now part of the probabilistic model and do not just enter through a variational approximation.
Revised: This is because the discriminator parameters Φ are now part of the probabilistic model and do not just enter through a variational approximation.

Original: How should we represent the posterior over discriminator parameters p ( φ ) ?
Revised: How should we represent the posterior over discriminator parameters p ( φ ) ?

Original: There is considerable work in Bayesian deep learning that offers possible answers, but here we take an ensemble approach (Seung et al., 1992; Lakshminarayanan et al., 2017).
Revised: There is considerable work in Bayesian deep learning that offers possible answers, but here we take an ensemble approach (Seung et al., 1992; Lakshminarayanan et al., 2017).

Original: We train N discriminators with parameters φ i for the i th discriminator.
Revised: We train N discriminators with parameters φ i for the i th discriminator.

Original: The discriminators are independently initialized and in theory could also be trained on different mini-batches, though in practice, we found it both sufﬁcient and
Revised: The discriminators are independently initialized and in theory could also be trained on different mini-batches, though in practice, we found it both sufﬁcient and

Original: [Figure]
Revised: [Figure]

Original: 𝜏
Revised: 𝜏

Original: [Figure]
Revised: [Figure]

Original: simpler to train them on the same mini-batches, as others have also found (Osband et al., 2016).
Revised: simpler to train them on the same mini-batches, as others have also found (Osband et al., 2016).

Original: The posterior p ( φ ) is then represented as a mixture of point masses at the φ i :
Revised: The posterior p ( φ ) is then represented as a mixture of point masses at the φ i :

Original: [Equation]
Revised: [Equation]

Original: Substituting the posterior in equation 8 into equation 7, we have:
Revised: Substituting the posterior in equation 8 into equation 7, we have:

Original: [Equation]
Revised: [Equation]

Original: Maximizing equation 9 with RL corresponds to adding the following auxiliary reward to the policy:
Revised: Maximizing equation 9 with RL corresponds to adding the following auxiliary reward to the policy:

Original: [Equation]
Revised: [Equation]

Original: In words, this is the entropy of the mean discriminator minus the mean of the entropies of the discriminators.
Revised: In words, this is the entropy of the mean discriminator minus the mean of the entropies of the discriminators.

Original: By Jensen’s inequality, entropy increases under averaging, and thus r DISDAIN ≥ 0 .
Revised: By Jensen’s inequality, entropy increases under averaging, and thus r DISDAIN ≥ 0 .

Original: For trajectories on which there has been ample training data for the discriminators, the ensemble members should agree and q φ i ( z | o )
Revised: For trajectories on which there has been ample training data for the discriminators, the ensemble members should agree and q φ i ( z | o )

Original: ≈ q φ j ( z | o ) for all i, j .
Revised: ≈ q φ j ( z | o ) for all i, j .

Original: Thus the two terms in equation 10 will be equal and this reward will vanish.
Revised: Thus the two terms in equation 10 will be equal and this reward will vanish.

Original: For states of high discriminator disagreement , however, this reward will be positive, encouraging exploration.
Revised: For states of high discriminator disagreement , however, this reward will be positive, encouraging exploration.

Original: Therefore, we call equation 10 the Discriminator
Revised: Therefore, we call equation 10 the Discriminator

Original: Disagreement Intrinsic reward, or DISDAIN (ﬁgure 2b).
Revised: Disagreement Intrinsic reward, or DISDAIN (ﬁgure 2b).

Original: DISDAIN is simple to calculate for discrete Z , as is common in the relevant literature (Gregor et al., 2016;
Revised: DISDAIN is simple to calculate for discrete Z , as is common in the relevant literature (Gregor et al., 2016;

Original: Eysenbach et al., 2019; Achiam et al., 2018; Baumli et al., 2021).
Revised: Eysenbach et al., 2019; Achiam et al., 2018; Baumli et al., 2021).

Original: DISDAIN augments any discriminator-based unsupervised skill learning algorithm with two changes.
Revised: DISDAIN augments any discriminator-based unsupervised skill learning algorithm with two changes.

Original: First, an ensemble of discriminators is trained instead of just one, and r skill should be calculated using the ensembleaveraged prediction q φ ( Z | O )
Revised: First, an ensemble of discriminators is trained instead of just one, and r skill should be calculated using the ensembleaveraged prediction q φ ( Z | O )

Original: =
Revised: =

Original: 1 N (cid:80) Ni =
Revised: 1 N (cid:80) Ni =

Original: 1 q φ i ( Z | O ) .
Revised: 1 q φ i ( Z | O ) .

Original: Second, the DISDAIN reward is combined with r skill , which we do through simple addition with a tunable multiplier λ .
Revised: Second, the DISDAIN reward is combined with r skill , which we do through simple addition with a tunable multiplier λ .

Original: Pseudocode for DISDAIN is provided in Algorithm 1.
Revised: Pseudocode for DISDAIN is provided in Algorithm 1.

Original: With N = 1 and λ = 0 , this is standard unbonused skill discovery.
Revised: With N = 1 and λ = 0 , this is standard unbonused skill discovery.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We validate DISDAIN by testing its ability to increase skill learning in an illustrative grid world (Four Rooms) as well as a more challenging pixel-based setting requiring function approximation (the 57 Atari games of the Arcade Learning Environment (Bellemare et al., 2013)).
Revised: We validate DISDAIN by testing its ability to increase skill learning in an illustrative grid world (Four Rooms) as well as a more challenging pixel-based setting requiring function approximation (the 57 Atari games of the Arcade Learning Environment (Bellemare et al., 2013)).

Original: In addition to comparing performance to unbonused skill learning, we also compare to using popular off-the-shelf exploration bonuses that are not tailored to skill discovery.
Revised: In addition to comparing performance to unbonused skill learning, we also compare to using popular off-the-shelf exploration bonuses that are not tailored to skill discovery.

Original: In Four Rooms, we compare to using count-based bonuses, which are known to perform well in these settings (Brafman and Tennenholtz, 2002).
Revised: In Four Rooms, we compare to using count-based bonuses, which are known to perform well in these settings (Brafman and Tennenholtz, 2002).

Original: In Atari, where count-based bonuses become untenable due to the enormous state space, we compare to random network distillation (RND; Burda et al.
Revised: In Atari, where count-based bonuses become untenable due to the enormous state space, we compare to random network distillation (RND; Burda et al.

Original: , one of the most commonly used
Revised: , one of the most commonly used

Original: Input: policy π θ , discriminator ensemble { q φ i } Ni =1 , skill features O ( τ ) , skill distribution p ( Z ) , skill trajectory length T , DISDAIN reward weight λ while not converged do
Revised: Input: policy π θ , discriminator ensemble { q φ i } Ni =1 , skill features O ( τ ) , skill distribution p ( Z ) , skill trajectory length T , DISDAIN reward weight λ while not converged do

Original: Reset environment, sampling initial state swhile episode not ended do
Revised: Reset environment, sampling initial state swhile episode not ended do

Original: Sample skill, z ∼ p
Revised: Sample skill, z ∼ p

Original: ( Z )
Revised: ( Z )

Original: Sample trajectory of length T from s 0 , τ ∼ π ( z )
Revised: Sample trajectory of length T from s 0 , τ ∼ π ( z )

Original: Form average discriminator from ensemble, q φ = 1 N (cid:80) Ni =
Revised: Form average discriminator from ensemble, q φ = 1 N (cid:80) Ni =

Original: 1 q φ i r skill = log q φ ( z | O ( τ ))
Revised: 1 q φ i r skill = log q φ ( z | O ( τ ))

Original: − log p ( z ) r DISDAIN =
Revised: − log p ( z ) r DISDAIN =

Original: H
Revised: H

Original: [ q φ ( · | O ( τ ))]
Revised: [ q φ ( · | O ( τ ))]

Original: − 1 N (cid:80)
Revised: − 1 N (cid:80)

Original: Ni =1 H [ q φ i ( · | O ( τ ))]
Revised: Ni =1 H [ q φ i ( · | O ( τ ))]

Original: r = r skill + λr DISDAIN
Revised: r = r skill + λr DISDAIN

Original: Update θ with RL to maximize r
Revised: Update θ with RL to maximize r

Original: Update { q φ i } Ni =1 with SL to maximize log q φ i ( z | O ( τ ))
Revised: Update { q φ i } Ni =1 with SL to maximize log q φ i ( z | O ( τ ))

Original: s 

Revised: s

Original: =
Revised: =

Original: s T pseudo-count based methods (Bellemare et al., 2016).
Revised: s T pseudo-count based methods (Bellemare et al., 2016).

Original: In addition, we validate that any advantages of
Revised: In addition, we validate that any advantages of

Original: DISDAIN are not purely due to using an ensemble of discriminators by evaluating an ablation that includes the same ensemble but removes the DISDAIN reward (i.e. Algorithm 1 with λ = 0 ).
Revised: DISDAIN are not purely due to using an ensemble of discriminators by evaluating an ablation that includes the same ensemble but removes the DISDAIN reward (i.e. Algorithm 1 with λ = 0 ).

Original: In all cases, our primary metric for comparison is the effective number of skills learnt, n skills (equation 4).
Revised: In all cases, our primary metric for comparison is the effective number of skills learnt, n skills (equation 4).

Original: The effective number of skills is just an interpretable transformation of the mutual information objective shared by a large number of unsupervised skill learning algorithms (e.g. Gregor et al.
Revised: The effective number of skills is just an interpretable transformation of the mutual information objective shared by a large number of unsupervised skill learning algorithms (e.g. Gregor et al.

Original: ; Achiam et al.
Revised: ; Achiam et al.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: ; Hansen et al.
Revised: ; Hansen et al.

Original: As such, the fact that
Revised: As such, the fact that

Original: DISDAIN helps better maximize this objective should be worthwhile in its own right, considering the various use cases that motivated this objective in the existing literature.
Revised: DISDAIN helps better maximize this objective should be worthwhile in its own right, considering the various use cases that motivated this objective in the existing literature.

Original: That said, we recognize that it is not obvious what utility comes with increasing the number of effective skills.
Revised: That said, we recognize that it is not obvious what utility comes with increasing the number of effective skills.

Original: To address this, we also measure three surrogates of skill utility: downstream goal achievement, unsupervised reward attainment, and unsupervised state coverage.
Revised: To address this, we also measure three surrogates of skill utility: downstream goal achievement, unsupervised reward attainment, and unsupervised state coverage.

Original: Distributed training We use a distributed actor-learner setup similar to R2D2 (Kapturowski et al., 2019), except we do not use replay prioritization or burn-in, and Q-value targets are computed with
Revised: Distributed training We use a distributed actor-learner setup similar to R2D2 (Kapturowski et al., 2019), except we do not use replay prioritization or burn-in, and Q-value targets are computed with

Original: Peng’s Q ( λ ) (Peng and Williams, 1994) rather than n -step double Q-learning.
Revised: Peng’s Q ( λ ) (Peng and Williams, 1994) rather than n -step double Q-learning.

Original: In all cases, we found it important to train separate Q-functions for the skill learning rewards and exploration bonuses (DISDAIN, RND, or count-based).
Revised: In all cases, we found it important to train separate Q-functions for the skill learning rewards and exploration bonuses (DISDAIN, RND, or count-based).

Original: This follows from Burda et al.
Revised: This follows from Burda et al.

Original: (2019), who also found that this setup helped stabilize learning.
Revised: (2019), who also found that this setup helped stabilize learning.

Original: Unlike that work, we need to specify a value target for each
Revised: Unlike that work, we need to specify a value target for each

Original: Q-function, which we take to be the value of the action that maximizes the composite value function, where the two values are added with a tunable weight λ .
Revised: Q-function, which we take to be the value of the action that maximizes the composite value function, where the two values are added with a tunable weight λ .

Original: The discriminator is trained on the same mini-batches sampled from replay to train the Q-functions.
Revised: The discriminator is trained on the same mini-batches sampled from replay to train the Q-functions.

Original: For Atari experiments, the Q-networks process batches of state, skill, and action tuples to produce scalar Q-values for each, and the ResNet state embedding network used in Espeholt et al.
Revised: For Atari experiments, the Q-networks process batches of state, skill, and action tuples to produce scalar Q-values for each, and the ResNet state embedding network used in Espeholt et al.

Original: (2018) is shared by both of the Q-networks and discriminator.
Revised: (2018) is shared by both of the Q-networks and discriminator.

Original: For the Four Rooms grid world, we use tabular Q-functions and discriminators.
Revised: For the Four Rooms grid world, we use tabular Q-functions and discriminators.

Original: Further implementation details can be found in appendix A.
Revised: Further implementation details can be found in appendix A.

Original: Skill discovery Our skill discovery baseline deviates slightly from prior work in order to make it representative of the skill discovery literature as a whole.
Revised: discovery Our skill discovery baseline deviates slightly from prior work in order to make it representative of the skill discovery literature as a whole.

Original: We utilize a simple discriminator that receives only the ﬁnal state of a trajectory (so O ( τ ) =
Revised: We utilize a simple discriminator that receives only the ﬁnal state of a trajectory (so O ( τ ) =

Original: s T ), omitting the state-conditional prior and action entropy bonuses used in speciﬁc algorithms (Gregor et al., 2016;
Revised: s T ), omitting the state-conditional prior and action entropy bonuses used in speciﬁc algorithms (Gregor et al., 2016;

Original: Eysenbach et al., 2019).
Revised: Eysenbach et al., 2019).

Original: One generally helpful change that we had not previously encountered was to clip negative skill rewards to 0 ( r skill = max( r skill , 0) ).

Original: This yields a strictly tighter lower bound on the mutual information (equation 2), since negative rewards imply the discriminator’s performance is worse than chance.

Original: Hyperparameters Most of the RL hyperparameters were not tuned, but rather taken from standard values known to be reasonable for Peng’s Q ( λ ) .
Revised: Hyperparameters Most of the RL hyperparameters were not tuned, but rather taken from standard values known to be reasonable for Peng’s Q ( λ ) .

Original: The skill discovery speciﬁc hyperparameters were tuned for the basic algorithm without exploration bonuses, and then reused in all conditions.
Revised: The skill discovery speciﬁc hyperparameters were tuned for the basic algorithm without exploration bonuses, and then reused in all conditions.

Original: Our
Revised: Our

Original: RND implementation was ﬁrst tuned without skill learning to achieve Atari performance competitive with the original paper.
Revised: RND implementation was ﬁrst tuned without skill learning to achieve Atari performance competitive with the original paper.

Original: For all exploration bonuses (DISDAIN, RND, count), we tuned their reward weighting (e.g. λ in algorithm 1), while for DISDAIN we additionally swept the ensemble size ( N ).
Revised: For all exploration bonuses (DISDAIN, RND, count), we tuned their reward weighting (e.g. λ in algorithm 1), while for DISDAIN we additionally swept the ensemble size ( N ).

Original: Baselines As with the skill discovery reward, we apply exploration bonuses only to the terminal states of each skill trajectory.
Revised: Baselines As with the skill discovery reward, we apply exploration bonuses only to the terminal states of each skill trajectory.

Original: For the count-based bonus, we track the number of times an agent ends a skill trajectory in each state n ( s ) and apply the exploration bonus r T = 1 / (cid:112) n ( s T ) .
Revised: For the count-based bonus, we track the number of times an agent ends a skill trajectory in each state n ( s ) and apply the exploration bonus r T = 1 / (cid:112) n ( s T ) .

Original: For RND, we follow the details of Burda et al.
Revised: For RND, we follow the details of Burda et al.

Original: (2019) as closely as possible.
Revised: (2019) as closely as possible.

Original: For our target and predictor networks, we use the same ResNet architecture as the policy observation embedding described above, and then project to a latent dimension of 128.
Revised: For our target and predictor networks, we use the same ResNet architecture as the policy observation embedding described above, and then project to a latent dimension of 128.

Original: Rather than normalizing observations based on running statistics, we found it more reliable to use the standard 1255 normalization of Atari observations.
Revised: Rather than normalizing observations based on running statistics, we found it more reliable to use the standard 1255 normalization of Atari observations.

Original: DISDAIN Our ensemble-based uncertainty estimator required many design choices, including the size of ensembles, and to what extent the ensemble members shared training data and parameters.
Revised: DISDAIN Our ensemble-based uncertainty estimator required many design choices, including the size of ensembles, and to what extent the ensemble members shared training data and parameters.

Original: In all of the domains we tested, we found training ensemble members on different batches to be unnecessary, similar to Osband et al.
Revised: In all of the domains we tested, we found training ensemble members on different batches to be unnecessary, similar to Osband et al.

Original: In the tabular case (Four Rooms), parameter-sharing is not a concern, and we found an ensemble size of N = 2 to be sufﬁcient.
Revised: In the tabular case (Four Rooms), parameter-sharing is not a concern, and we found an ensemble size of N = 2 to be sufﬁcient.

Original: For Atari, the ensemble of discriminators reuse the single ResNet state embedding network which is shared by the value function.
Revised: For Atari, the ensemble of discriminators reuse the single ResNet state embedding network which is shared by the value function.

Original: The input to the ensemble will inevitably drift, even if the data distribution remains constant, since the ResNet representations evolve according to a combination of the value function and discriminator updates.
Revised: The input to the ensemble will inevitably drift, even if the data distribution remains constant, since the ResNet representations evolve according to a combination of the value function and discriminator updates.

Original: Expressive ensembles (e.g. with hidden layers) never converged in practice.
Revised: Expressive ensembles (e.g. with hidden layers) never converged in practice.

Original: By contrast, large linear ensembles ( N = 40 ) were reliably convergent, with convergence time increasing with ensemble size.
Revised: By contrast, large linear ensembles ( N = 40 ) were reliably convergent, with convergence time increasing with ensemble size.

Original: We follow Osband et al.
Revised: We follow Osband et al.

Original: in scaling the gradients passed backward through the embedding network by 140 to account for the increase in effective learning rate.
Revised: in scaling the gradients passed backward through the embedding network by 140 to account for the increase in effective learning rate.

Original: 4. F OUR R OOMS
Revised: 4. F OUR R OOMS

Original: First, we evaluate skill learning in the illustrative grid world seen in ﬁgure 3.
Revised: First, we evaluate skill learning in the illustrative grid world seen in ﬁgure 3.

Original: There are 4 rooms and 104 states.
Revised: There are 4 rooms and 104 states.

Original: The agent begins each episode in the top left corner and at each step chooses an action from the set: left , right , up , down , or no-op .
Revised: The agent begins each episode in the top left corner and at each step chooses an action from the set: left , right , up , down , or no-op .

Original: Episodes are 20 steps and we sample one skill per episode (i.e. T = 20 ).
Revised: Episodes are 20 steps and we sample one skill per episode (i.e. T = 20 ).

Original: The episodes are long enough to reach all but one state, allowing for a maximum of 103 skills.
Revised: The episodes are long enough to reach all but one state, allowing for a maximum of 103 skills.

Original: For each method, we set N Z = 128 to make this theoretically possible.
Revised: For each method, we set N Z = 128 to make this theoretically possible.

Original: Results As seen in ﬁgure 3, even in this simple task, unbonused agents are unable to exceedskills and barely leave the ﬁrst room (see ﬁgures 18 and 19 for example skill rollouts).
Revised: Results As seen in ﬁgure 3, even in this simple task, unbonused agents are unable to exceedskills and barely leave the ﬁrst room (see ﬁgures 18 and 19 for example skill rollouts).

Original: With both
Revised: With both

Original: DISDAIN and a count bonus, agents explore all four rooms and learn approximately triple the number of skills, with the best seeds learning approximately 90 skills.
Revised: DISDAIN and a count bonus, agents explore all four rooms and learn approximately triple the number of skills, with the best seeds learning approximately 90 skills.

Original: Both bonuses do slow learning and add variance due to the addition of a separate learned Q-function (see ﬁgure 20 for individual seeds).
Revised: Both bonuses do slow learning and add variance due to the addition of a separate learned Q-function (see ﬁgure 20 for individual seeds).

Original: The ensemble-only ablation provides a small beneﬁt to skill learning, but far less than DISDAIN, demonstrating that the DISDAIN exploration bonus ( r DISDAIN ), rather than the ensembling, drives the increased performance.
Revised: The ensemble-only ablation provides a small beneﬁt to skill learning, but far less than DISDAIN, demonstrating that the DISDAIN exploration bonus ( r DISDAIN ), rather than the ensembling, drives the increased performance.

Original: As a simple demonstration of the usefulness of our learnt skills for downstream tasks, we also evaluate each methods’ skills on the original Four Rooms reward (i.e. reaching speciﬁed goal states) without additional ﬁnetuning.
Revised: As a simple demonstration of the usefulness of our learnt skills for downstream tasks, we also evaluate each methods’ skills on the original Four Rooms reward (i.e. reaching speciﬁed goal states) without additional ﬁnetuning.

Original: Speciﬁcally, for each method, we sample each accessible state of the environment as a goal, pass it through the trained discriminator q φ , choose the highest probability skill z , rollout the policy for one episode conditioned on z , and track the fraction of goal states successfully reached (similar to the imitation learning evaluation of Eysenbach et al.
Revised: Speciﬁcally, for each method, we sample each accessible state of the environment as a goal, pass it through the trained discriminator q φ , choose the highest probability skill z , rollout the policy for one episode conditioned on z , and track the fraction of goal states successfully reached (similar to the imitation learning evaluation of Eysenbach et al.

Original: As seen in Figure 3b, more learnt skills leads to better downstream task performance.
Revised: As seen in Figure 3b, more learnt skills leads to better downstream task performance.

Original: 4. A TARI
Revised: 4. A TARI

Original: Next we consider skill learning on a standard suite of 57 Atari games, where prior work has shown that learning discrete skills is quite difﬁcult (Hansen et al., 2020).
Revised: Next we consider skill learning on a standard suite of 57 Atari games, where prior work has shown that learning discrete skills is quite difﬁcult (Hansen et al., 2020).

Original: In addition, it is a non-trivial test for our ensemble uncertainty based method to work in the function approximation setting, since this depends on how each ensemble member generalizes to unseen data.
Revised: In addition, it is a non-trivial test for our ensemble uncertainty based method to work in the function approximation setting, since this depends on how each ensemble member generalizes to unseen data.

Original: Here we use N Z = 64 and
Revised: Here we use N Z = 64 and

Original: T = 20 .
Revised: T = 20 .

Original: Since Atari episodes vary in length, skills may be resampled within an episode.
Revised: Since Atari episodes vary in length, skills may be resampled within an episode.

Original: The count-based baseline used in the Four Rooms experiments cannot be directly applied to a nontabular environment like Atari.
Revised: The count-based baseline used in the Four Rooms experiments cannot be directly applied to a nontabular environment like Atari.

Original: While pseudo-count methods have been used here (Bellemare et al., 2016), Random Network Distillation (RND) similarly induces long term exploration (Burda et al., 2019) and has been used for this purpose in a state-of-the-art Atari agent (Badia et al., 2020).
Revised: While pseudo-count methods have been used here (Bellemare et al., 2016), Random Network Distillation (RND) similarly induces long term exploration (Burda et al., 2019) and has been used for this purpose in a state-of-the-art Atari agent (Badia et al., 2020).

Original: While newer approaches surpass RND in some domains (Raileanu and Rocktäschel, 2020; Seo et al., 2021), it is unclear if this is the case across the full Atari suite.
Revised: While newer approaches surpass RND in some domains (Raileanu and Rocktäschel, 2020; Seo et al., 2021), it is unclear if this is the case across the full Atari suite.

Original: So, at present, we believe RND is the most compelling baseline to compare against DISDAIN.
Revised: So, at present, we believe RND is the most compelling baseline to compare against DISDAIN.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Results As pointed out in Agarwal et al.
Revised: Results As pointed out in Agarwal et al.

Original: (2021), making statistically sound conclusions can be challenging in the few-seed, many-task setting of Atari.
Revised: (2021), making statistically sound conclusions can be challenging in the few-seed, many-task setting of Atari.

Original: Thus, we follow their recommendations and focus our results primarily on statistically robust distributional claims here.
Revised: Thus, we follow their recommendations and focus our results primarily on statistically robust distributional claims here.

Original: Individual learning curves and game-by-game results are available in the appendix.
Revised: Individual learning curves and game-by-game results are available in the appendix.

Original: As shown in ﬁgure 4, DISDAIN increases the effective number of skills learnt across the Atari suite, with only modest damage to sample efﬁency.
Revised: As shown in ﬁgure 4, DISDAIN increases the effective number of skills learnt across the Atari suite, with only modest damage to sample efﬁency.

Original: Additionally, we found that DISDAIN’s performance boosts are robust to its key hyperparameters, namely the bonus weight λ and ensemble size N (see ﬁgure 8), with signiﬁcant gains over unbonused skill learning maintained over more than an order of magnitude of variation in both.
Revised: Additionally, we found that DISDAIN’s performance boosts are robust to its key hyperparameters, namely the bonus weight λ and ensemble size N (see ﬁgure 8), with signiﬁcant gains over unbonused skill learning maintained over more than an order of magnitude of variation in both.

Original: Notably, our results show that RND fails to signiﬁcantly aid in skill learning.
Revised: Notably, our results show that RND fails to signiﬁcantly aid in skill learning.

Original: A sweep over the RND bonus weight is shown in ﬁgure 11, but the summary is that as the RND bonus becomes similar in magnitude to the skill learning reward, it damages skill learning rather than helping, and so the “best"
Revised: A sweep over the RND bonus weight is shown in ﬁgure 11, but the summary is that as the RND bonus becomes similar in magnitude to the skill learning reward, it damages skill learning rather than helping, and so the “best"

Original: RND bonus weight for skill learning is approximately zero.
Revised: RND bonus weight for skill learning is approximately zero.

Original: This is perhaps unsurprising: RND was designed in the context of stationary task rewards (as are most other exploration methods, such as pseudo-counts), whereas skill learning objectives produce a highly non-stationary reward function.
Revised: This is perhaps unsurprising: RND was designed in the context of stationary task rewards (as are most other exploration methods, such as pseudo-counts), whereas skill learning objectives produce a highly non-stationary reward function.

Original: This highlights the importance of using an exploration bonus tailored to the skill discovery setting.
Revised: This highlights the importance of using an exploration bonus tailored to the skill discovery setting.

Original: Additionally, the failure of the “ensemble-only" baseline to signiﬁcantly increase skill learning demonstrates that is the r DISDAIN exploration bonus that is crucial to DISDAIN’s success, and not just the ensembling of the discriminator.
Revised: Additionally, the failure of the “ensemble-only" baseline to signiﬁcantly increase skill learning demonstrates that is the r DISDAIN exploration bonus that is crucial to DISDAIN’s success, and not just the ensembling of the discriminator.

Original: [Figure]
Revised: [Figure]

Original: To further probe the utility of our learnt skills, we measure the unsupervised reward attainment of a policy that randomly switches between them.
Revised: To further probe the utility of our learnt skills, we measure the unsupervised reward attainment of a policy that randomly switches between them.

Original: First reported in Hansen et al.
Revised: First reported in Hansen et al.

Original: , the idea is that while this policy is certainly far from optimal, this metric indicates whether or not the skill space is sufﬁcient to perform the various reward collecting behaviors involved in each game.
Revised: , the idea is that while this policy is certainly far from optimal, this metric indicates whether or not the skill space is sufﬁcient to perform the various reward collecting behaviors involved in each game.

Original: Additionally, we measure lifetime state coverage as an indication of exploration throughout learning (on a subset of games supporting this metric; see Appendix B for details).
Revised: Additionally, we measure lifetime state coverage as an indication of exploration throughout learning (on a subset of games supporting this metric; see Appendix B for details).

Original: Figure 5 conﬁrms that DISDAIN leads to both increased reward attainment and lifetime coverage (see ﬁgures 14, 15, and 16 for additional lifetime and episodic state coverage results).
Revised: Figure 5 conﬁrms that DISDAIN leads to both increased reward attainment and lifetime coverage (see ﬁgures 14, 15, and 16 for additional lifetime and episodic state coverage results).

Original:  D ISCUSSION & L IMITATIONS
Revised:  D ISCUSSION & L IMITATIONS

Original: We introduced DISDAIN, an enhancement for unsupervised skill discovery algorithms which increases the effective number of skills learnt across a diverse range of tasks, by using ensemble-based uncertainty estimation to counteract a bias towards pessimistic exploration.
Revised: We introduced DISDAIN, an enhancement for unsupervised skill discovery algorithms which increases the effective number of skills learnt across a diverse range of tasks, by using ensemble-based uncertainty estimation to counteract a bias towards pessimistic exploration.

Original: The connection between ensemble estimates of uncertainty and infomax exploration dates back to at least Seung et al.
Revised: The connection between ensemble estimates of uncertainty and infomax exploration dates back to at least Seung et al.

Original: (1992), who use it to select examples in an active supervised learning setting.
Revised: (1992), who use it to select examples in an active supervised learning setting.

Original: These ideas have more recently found use in the RL literature, with recent work using disagreement among ensembles of value functions (Chen et al., 2017; Flennerhag et al., 2020;
Revised: These ideas have more recently found use in the RL literature, with recent work using disagreement among ensembles of value functions (Chen et al., 2017; Flennerhag et al., 2020;

Original: Zhang et al., 2020) and forward models of the environment (Pathak et al., 2019; Shyam et al., 2019;
Revised: Zhang et al., 2020) and forward models of the environment (Pathak et al., 2019; Shyam et al., 2019;

Original: Sekar et al., 2020) to drive exploration.
Revised: Sekar et al., 2020) to drive exploration.

Original: In a closely related line of work, ensembles of Q-functions have been used to drive exploration without an explicit bonus based on disagreement (Osband et al., 2016; 2018).
Revised: In a closely related line of work, ensembles of Q-functions have been used to drive exploration without an explicit bonus based on disagreement (Osband et al., 2016; 2018).

Original: To our knowledge, DISDAIN represents the ﬁrst application of these ideas to unsupervised skill discovery.
Revised: To our knowledge, DISDAIN represents the ﬁrst application of these ideas to unsupervised skill discovery.

Original: We focused on discrete skill learning methods due to their relative prevalence (Gregor et al., 2016;
Revised: We focused on discrete skill learning methods due to their relative prevalence (Gregor et al., 2016;

Original: Eysenbach et al., 2019; Achiam et al., 2018; Baumli et al., 2021).
Revised: Eysenbach et al., 2019; Achiam et al., 2018; Baumli et al., 2021).

Original: In some cases, continuous or structured skill spaces might make more sense (Hansen et al., 2020; Warde-Farley et al., 2019).
Revised: In some cases, continuous or structured skill spaces might make more sense (Hansen et al., 2020; Warde-Farley et al., 2019).

Original: While the principles behind DISDAIN should still apply, further approximations may be necessary, e.g. for the entropy of the ensemble-averaged discriminator, which may be unavailable in closed form.
Revised: While the principles behind DISDAIN should still apply, further approximations may be necessary, e.g. for the entropy of the ensemble-averaged discriminator, which may be unavailable in closed form.

Original: Designing agents that explore and master their environment in the absence of task reward remains an open problem, for which there exist many different families of approaches (e.g. reward-free exploration (Jin et al., 2020; Zhang et al., 2021)).
Revised: Designing agents that explore and master their environment in the absence of task reward remains an open problem, for which there exist many different families of approaches (e.g. reward-free exploration (Jin et al., 2020; Zhang et al., 2021)).

Original: In this paper, we focus on improving one such family - unsupervised skill learning through variational infomax (Section 2).
Revised: In this paper, we focus on improving one such family - unsupervised skill learning through variational infomax (Section 2).

Original: By treating agents with
Revised: By treating agents with

Original: DISDAIN, we empower them to better maximize their objective and learn more skills.
Revised: DISDAIN, we empower them to better maximize their objective and learn more skills.

Original: Leveraging these skills for rapid task reward maximization remains an important direction for future research.
Revised: Leveraging these skills for rapid task reward maximization remains an important direction for future research.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: The authors would like to thank Stephen Spencer for engineering and technical support, Ian Osband for feedback on an early draft, Rishabh Agarwal for suggestions on statistical analysis and plotting,  and David Schwab for pointing us to the original Query by Committee work (Seung et al., 1992)
Revised: The authors would like to thank Stephen Spencer for engineering and technical support, Ian Osband for feedback on an early draft, Rishabh Agarwal for suggestions on statistical analysis and plotting, Phil Bachman for correcting the error discussed in Appendix C, and David Schwab for pointing us to the original Query by Committee work (Seung et al., 1992)

Original: A C OMPUTE R EQUIREMENTS AND H YPERPARAMETERS
Revised: A C OMPUTE R EQUIREMENTS AND H YPERPARAMETERS

Original: The compute cluster we performed experiments on is rather heterogeneous, and has features such as host-sharing, adaptive load-balancing, etc.
Revised: The compute cluster we performed experiments on is rather heterogeneous, and has features such as host-sharing, adaptive load-balancing, etc.

Original: It is therefore hard to give precise details regarding compute resources – however, the following is a best-guess estimate.
Revised: It is therefore hard to give precise details regarding compute resources – however, the following is a best-guess estimate.

Original: A full experimental training run for Atari lasted 4 days on average.
Revised: A full experimental training run for Atari lasted 4 days on average.

Original: Our distributed reinforcement learning setup (Espeholt et al., 2018) used 100 CPU actors and a single V100 GPU learner.
Revised: Our distributed reinforcement learning setup (Espeholt et al., 2018) used 100 CPU actors and a single V100 GPU learner.

Original: Thus, we required approximately 9600 CPU hours and 96 V100 GPU hours per seed, with 3 seeds andconditions per game.
Revised: Thus, we required approximately 9600 CPU hours and 96 V100 GPU hours per seed, with 3 seeds andconditions per game.

Original: Tuning required approximately 10 different hyper-parameters combinations on 6 games, amounting to 864,000 CPU hours and 8,640 V100 GPU hours.
Revised: Tuning required approximately 10 different hyper-parameters combinations on 6 games, amounting to 864,000 CPU hours and 8,640 V100 GPU hours.

Original: The results on the full suite of 57 Atari games required 4,924,800 CPU hours and 49,248 V100 GPU hours.
Revised: The results on the full suite of 57 Atari games required 4,924,800 CPU hours and 49,248 V100 GPU hours.

Original: Combining these, we get a total compute budget of 5,788,800 CPU hours and 57,888 V100 GPU hours.
Revised: Combining these, we get a total compute budget of 5,788,800 CPU hours and 57,888 V100 GPU hours.

Original: It is worth remembering that the above is likely quite a loose upper-bound, as this estimate assumes 100 percent up time, which is far from the truth given the host-sharing and load-balancing involved in our setup.
Revised: It is worth remembering that the above is likely quite a loose upper-bound, as this estimate assumes 100 percent up time, which is far from the truth given the host-sharing and load-balancing involved in our setup.

Original: Additionally, V100 GPUs were chosen based on what was on hand; our models are small enough to ﬁt on much cheaper cards without much slowdown.
Revised: Additionally, V100 GPUs were chosen based on what was on hand; our models are small enough to ﬁt on much cheaper cards without much slowdown.

Original: B C OVERAGE M ETRICS
Revised: B C OVERAGE M ETRICS

Original: We calculate two related notions of coverage: lifetime and episodic.
Revised: We calculate two related notions of coverage: lifetime and episodic.

Original: Lifetime coverage corresponds to the number of unique states encountered during an agent’s lifetime, whereas episodic coverage corresponds to the number of unique states encountered during each episode.
Revised: Lifetime coverage corresponds to the number of unique states encountered during an agent’s lifetime, whereas episodic coverage corresponds to the number of unique states encountered during each episode.

Original: Both rely on the notion of a unique state.
Revised: Both rely on the notion of a unique state.

Original: The subset of games chosen for these metrics were those where a good notion of a unique state is simple: they all involve a controllable avatar that moves in a coordinate system, so unique avatar coordinates are used.
Revised: The subset of games chosen for these metrics were those where a good notion of a unique state is simple: they all involve a controllable avatar that moves in a coordinate system, so unique avatar coordinates are used.

Original: This information is exposed in the RAM state of the Atari emulator, as shown in Anand et al.
Revised: This information is exposed in the RAM state of the Atari emulator, as shown in Anand et al.

Revised: C A DDITIONAL IMPLEMENTATION DETAILS

Revised: For our skill learning reward, one generally helpful change that we had not previously encountered was to clip negative skill rewards to 0 ( r skill = max( r skill , 0) ).

Revised: A previous version of this manuscript stated that: “This yields a strictly tighter lower bound on the mutual information (equation 2), since negative rewards imply the discriminator’s performance is worse than chance.”

Revised: However, that argument isn’t quite correct.

Revised: Clipping the expected reward (i.e. clipping outside the expectation) would produce a tighter bound, but this argument does not hold on a sample-by-sample basis (i.e. clipping inside the expectation, as we did).

Revised: Thus, this should only be viewed as a heuristic.