Original: M INIMUM D ESCRIPTION L ENGTH S KILLS FOR A CCELERATED R EINFORCEMENT L EARNING
Revised: M INIMUM D ESCRIPTION L ENGTH S KILLS FOR A CCELERATED R EINFORCEMENT L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: Humans can quickly learn new tasks by reusing a large number of previously acquired skills.
Revised: Humans can quickly learn new tasks by reusing a large number of previously acquired skills.

Original: How can we discover such reusable skills for artiﬁcial agents when given a large dataset of prior experience?
Revised: How can we discover such reusable skills for artiﬁcial agents when given a large dataset of prior experience?

Original: Past works leverage extensive human supervision to deﬁne skills or use simple skill heuristics that limit their expressiveness.
Revised: Past works leverage extensive human supervision to deﬁne skills or use simple skill heuristics that limit their expressiveness.

Original: In contrast, we propose a principled, unsupervised objective for skill discovery from large, ofﬂine datasets based on the Minimum Description Length principle: we show that a “code book” of skills that can maximally compress the training data can be reused to efﬁciently learn new tasks.
Revised: In contrast, we propose a principled, unsupervised objective for skill discovery from large, ofﬂine datasets based on the Minimum Description Length principle: we show that a “code book” of skills that can maximally compress the training data can be reused to efﬁciently learn new tasks.

Original: By minimizing description length we strike an optimal balance between the number of extracted skills and their complexity.
Revised: By minimizing description length we strike an optimal balance between the number of extracted skills and their complexity.

Original: We show that our approach outperforms methods that heuristically deﬁne skills on a complex, long-horizon maze navigation task.
Revised: We show that our approach outperforms methods that heuristically deﬁne skills on a complex, long-horizon maze navigation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans are able to learn new tasks quickly by leveraging a large repertoire of previously learned skills.
Revised: Humans are able to learn new tasks quickly by leveraging a large repertoire of previously learned skills.

Original: Recent work on skill-based reinforcement learning (RL) uses this intuition to accelerate RL by extracting skills from large ofﬂine datasets and leveraging them for learning new tasks (Pertsch et al., 2020a; Ajay et al., 2021).
Revised: Recent work on skill-based reinforcement learning (RL) uses this intuition to accelerate RL by extracting skills from large ofﬂine datasets and leveraging them for learning new tasks (Pertsch et al., 2020a; Ajay et al., 2021).

Original: One key question is: how do we discover an appropriate set of skills from the data?
Revised: One key question is: how do we discover an appropriate set of skills from the data?

Original: Some prior work relies on costly human supervision that deﬁnes subskills (Schaal, 2006;
Revised: Some prior work relies on costly human supervision that deﬁnes subskills (Schaal, 2006;

Original: Lee et al., 2018; Shiarlis et al., 2018), but this is not scalable.
Revised: Lee et al., 2018; Shiarlis et al., 2018), but this is not scalable.

Original: Others employ simple heuristics like deﬁning skills as randomly sampled, ﬁxed-length sub-trajectories from the training data (Gupta et al., 2019;
Revised: Others employ simple heuristics like deﬁning skills as randomly sampled, ﬁxed-length sub-trajectories from the training data (Gupta et al., 2019;

Original: Pertsch et al., 2020a; Ajay et al., 2021), thereby limiting the expressiveness of the learned skills.
Revised: Pertsch et al., 2020a; Ajay et al., 2021), thereby limiting the expressiveness of the learned skills.

Original: In this work, we aim to formulate a more principled unsupervised objective for learning to discover a useful set of skills.
Revised: In this work, we aim to formulate a more principled unsupervised objective for learning to discover a useful set of skills.

Original: We propose to learn skills such that they most compactly represent solutions to a wide range of tasks.
Revised: We propose to learn skills such that they most compactly represent solutions to a wide range of tasks.

Original: Thus, a policy that leverages these skills to solve a new, related task can learn more efﬁciently, since its solution can be compactly represented with the learned set of skills.
Revised: Thus, a policy that leverages these skills to solve a new, related task can learn more efﬁciently, since its solution can be compactly represented with the learned set of skills.

Original: To learn such skills, we leverage the Minimum Description Length principle (Rissanen, 1978): we propose to learn a “code book” of skills such that we can maximally compress the action trajectories from the large, ofﬂine training dataset.
Revised: To learn such skills, we leverage the Minimum Description Length principle (Rissanen, 1978): we propose to learn a “code book” of skills such that we can maximally compress the action trajectories from the large, ofﬂine training dataset.

Original: We then leverage the extracted skill library for efﬁcient RL on unseen tasks.
Revised: We then leverage the extracted skill library for efﬁcient RL on unseen tasks.

Original: By minimizing description length we can strike an optimal balance between the number of extracted skills and their complexity.
Revised: By minimizing description length we can strike an optimal balance between the number of extracted skills and their complexity.

Original: We ﬁrst validate that skills which minimize description length lead to more efﬁcient learning of downstream tasks in a simple grid-world environment.
Revised: We ﬁrst validate that skills which minimize description length lead to more efﬁcient learning of downstream tasks in a simple grid-world environment.

Original: We then propose a deep latent variable model for learning to extract minimum description length skills.
Revised: We then propose a deep latent variable model for learning to extract minimum description length skills.

Original: In experiments on a challenging, long-horizon maze navigation task we show that our approach leads to more efﬁcient learning than alternative approaches that uses heuristically determined skills.
Revised: In experiments on a challenging, long-horizon maze navigation task we show that our approach leads to more efﬁcient learning than alternative approaches that uses heuristically determined skills.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A large body of prior work leverages pre-trained skills for accelerating the learning of new tasks.
Revised: A large body of prior work leverages pre-trained skills for accelerating the learning of new tasks.

Original: Often, approaches require manual deﬁnition of skills, e.g. as motion primitives (Schaal, 2006; Pastor et al., 2009), subskill polies (Oh et al., 2017; Lee et al., 2018;
Revised: Often, approaches require manual deﬁnition of skills, e.g. as motion primitives (Schaal, 2006; Pastor et al., 2009), subskill polies (Oh et al., 2017; Lee et al., 2018;

Original: Xu et al., 2018) or task sketches (Andreas et al., 2017; Shiarlis et al., 2018), making them challenging to scale to large skill sets.
Revised: Xu et al., 2018) or task sketches (Andreas et al., 2017; Shiarlis et al., 2018), making them challenging to scale to large skill sets.

Original: Others use simple heuristics to extract skills from unstructured data, like deﬁning skills as ﬁxed-length action trajectories (Hausman et al., 2018;
Revised: Others use simple heuristics to extract skills from unstructured data, like deﬁning skills as ﬁxed-length action trajectories (Hausman et al., 2018;

Original: Merel et al., 2020; Gupta et al., 2019; Mandlekar et al., 2020; Lynch et al., 2020; Pertsch et al., 2020a; Ajay et al., 2021), limiting their expressiveness.
Revised: Merel et al., 2020; Gupta et al., 2019; Mandlekar et al., 2020; Lynch et al., 2020; Pertsch et al., 2020a; Ajay et al., 2021), limiting their expressiveness.

Original: The options framework, in contrast, aims to learn useful skills without skill supervision while learning to solve a task (Sutton et al., 1999; Bacon et al., 2017), but jointly learning skills and policy purely from sparse reward supervision is challenging.
Revised: The options framework, in contrast, aims to learn useful skills without skill supervision while learning to solve a task (Sutton et al., 1999; Bacon et al., 2017), but jointly learning skills and policy purely from sparse reward supervision is challenging.

Original: Recently, multiple works have explored discovering variable-length skills from ofﬂine datasets, but they have focused on modeling high-dimensional observation sequences (Kipf et al., 2019;
Revised: Recently, multiple works have explored discovering variable-length skills from ofﬂine datasets, but they have focused on modeling high-dimensional observation sequences (Kipf et al., 2019;

Original: Kim et al., 2019; Pertsch et al., 2020c;b) or require multistage training procedures to stabilize training (Shankar et al., 2019; Shankar & Gupta, 2020), while we propose an end-to-end model for learning skills that can directly be used as behavior primitives by downstream RL agents.
Revised: Kim et al., 2019; Pertsch et al., 2020c;b) or require multistage training procedures to stabilize training (Shankar et al., 2019; Shankar & Gupta, 2020), while we propose an end-to-end model for learning skills that can directly be used as behavior primitives by downstream RL agents.

Original:  A PPROACH
Revised:  A PPROACH

Original: Given a large dataset D = { s t , a t , ... } of meaningful agent behaviors, collected via teleoperation or from prior training tasks, we aim to extract a set of reusable skills , temporally extended action sequences, that can be leveraged to accelerate learning on new target tasks.
Revised: Given a large dataset D = { s t , a t , ... } of meaningful agent behaviors, collected via teleoperation or from prior training tasks, we aim to extract a set of reusable skills , temporally extended action sequences, that can be leveraged to accelerate learning on new target tasks.

Original: We ﬁrst brieﬂy summarize the minimum description length (MDL) principle, and then describe our approach for learning minimum description length skills.
Revised: We ﬁrst brieﬂy summarize the minimum description length (MDL) principle, and then describe our approach for learning minimum description length skills.

Original: 3. P RELIMINARIES : THE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE
Revised: 3. P RELIMINARIES : THE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE

Original: The minimum description length (MDL) principle (Rissanen, 1978) deﬁnes a model that best ﬁts a given data sample x as the model that minimizes the data’s description length
Revised: The minimum description length (MDL) principle (Rissanen, 1978) deﬁnes a model that best ﬁts a given data sample x as the model that minimizes the data’s description length

Original: L ( x ) .
Revised: L ( x ) .

Original: Using a twopart code, the “crude” MDL principle (Grunwald, 2004) deﬁnes L ( x )
Revised: Using a twopart code, the “crude” MDL principle (Grunwald, 2004) deﬁnes L ( x )

Original: =
Revised: =

Original: L ( M ) + L ( x |M ) , i.e. the description length equals the summed description lengths of the model L ( M ) and of the data using the model L ( x |M ) .
Revised: L ( M ) + L ( x |M ) , i.e. the description length equals the summed description lengths of the model L ( M ) and of the data using the model L ( x |M ) .

Original: Shannon’s source coding theorem (Shannon, 1948) states that the MDL of a variable x that is distributed under p ( x ) equals E [ − log p ( x )] .
Revised: Shannon’s source coding theorem (Shannon, 1948) states that the MDL of a variable x that is distributed under p ( x ) equals E [ − log p ( x )] .

Original: Therefore, when using an encoder q ( z | x ) to translate the data into an encoded message z and a decoding model p ( x | z ) , the description length of the data is L ( x ) =
Revised: Therefore, when using an encoder q ( z | x ) to translate the data into an encoded message z and a decoding model p ( x | z ) , the description length of the data is L ( x ) =

Original: E z ∼ q ( z | x ) (cid:2) − log p ( x | z ) − log p ( z ) (cid:3) , where we treat the message z as part of the model and omit the one-time cost of transmitting the decoder model parameters (Hinton & Zemel, 1994).
Revised: E z ∼ q ( z | x ) (cid:2) − log p ( x | z ) − log p ( z ) (cid:3) , where we treat the message z as part of the model and omit the one-time cost of transmitting the decoder model parameters (Hinton & Zemel, 1994).

Original: In their work on bits-back coding , Hinton & Zemel (1994) showed that we can use a more sophisticated coding scheme to gain back “free bits” and achieve an even lower description length:
Revised: In their work on bits-back coding , Hinton & Zemel (1994) showed that we can use a more sophisticated coding scheme to gain back “free bits” and achieve an even lower description length:

Original: [Equation]
Revised: [Equation]

Original: This is equal to a negated evidence lower bound (ELBO) on x .
Revised: This is equal to a negated evidence lower bound (ELBO) on x .

Original: We can leverage tools from the literature on amortized variational inference (Kingma & Welling, 2014; Rezende et al., 2014) to minimize this bound.
Revised: We can leverage tools from the literature on amortized variational inference (Kingma & Welling, 2014; Rezende et al., 2014) to minimize this bound.

Original: Thus, we can learn a model p ( x | z ) that minimizes the description length of the data.
Revised: Thus, we can learn a model p ( x | z ) that minimizes the description length of the data.

Original: We will use this to learn a model of minimum description length skills.
Revised: We will use this to learn a model of minimum description length skills.

Original: 3. M INIMUM D ESCRIPTION L ENGTH S KILLS
Revised: 3. M INIMUM D ESCRIPTION L ENGTH S KILLS

Original: Our goal is to extract transferrable skills from the large dataset D , which accelerate the learning of new target tasks.
Revised: Our goal is to extract transferrable skills from the large dataset D , which accelerate the learning of new target tasks.

Original: Following the MDL principle from Section 3.1 we aim to ﬁnd a set of skills such that the description length of the action trajectories in the dataset is minimized when encoded with this skill “code book”.
Revised: Following the MDL principle from Section 3.1 we aim to ﬁnd a set of skills such that the description length of the action trajectories in the dataset is minimized when encoded with this skill “code book”.

Original: Put differently, we aim to ﬁnd a set of skills such that they efﬁciently represent the solution trajectories to a wide variety of tasks present in the dataset.
Revised: Put differently, we aim to ﬁnd a set of skills such that they efﬁciently represent the solution trajectories to a wide variety of tasks present in the dataset.

Original: This will allow a policy which leverages these skills to efﬁciently ﬁnd a solution to a new downstream task.
Revised: This will allow a policy which leverages these skills to efﬁciently ﬁnd a solution to a new downstream task.

Original: 1 Intuitively, minimizing description length allows our approach to strike an optimal balance between the number of extracted skills and their complexity.
Revised: 1 Intuitively, minimizing description length allows our approach to strike an optimal balance between the number of extracted skills and their complexity.

Original: [Figure]
Revised: [Figure]

Original: Crucial for the learning efﬁciency of a downstream task policy is the size of its search space: the smaller the search space, the more efﬁcient the learning.
Revised: Crucial for the learning efﬁciency of a downstream task policy is the size of its search space: the smaller the search space, the more efﬁcient the learning.

Original: For a policy that uses a vocabulary of learned skills instead of primitive actions, the size of this search space is proportional to | V | T , where | V | is the size of the skill vocabulary and T is the average number of skills per episode.
Revised: For a policy that uses a vocabulary of learned skills instead of primitive actions, the size of this search space is proportional to | V | T , where | V | is the size of the skill vocabulary and T is the average number of skills per episode.

Original: Crucially, both | V | and T are directly related to the message length in the two-part code formulation of MDL introduced in Section 3.1: the larger the number of skills | V | the more bits are required to encode any single skill; the more skills per episode T the more symbols need to be encoded per message.
Revised: Crucially, both | V | and T are directly related to the message length in the two-part code formulation of MDL introduced in Section 3.1: the larger the number of skills | V | the more bits are required to encode any single skill; the more skills per episode T the more symbols need to be encoded per message.

Original: Finding skills that minimize the description length of the data strikes a balance between the total number of skills and the average number of skills per message, i.e. the skill complexity, and thus reduces the search space for downstream task policies, making learning more efﬁcient.
Revised: Finding skills that minimize the description length of the data strikes a balance between the total number of skills and the average number of skills per message, i.e. the skill complexity, and thus reduces the search space for downstream task policies, making learning more efﬁcient.

Original: [Figure]
Revised: [Figure]

Original:  g r e tRL
Revised: R e g r e t

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: We experimentally validate this intuition in a simple grid-world environment (see Figure 1).
Revised: We experimentally validate this intuition in a simple grid-world environment (see Figure 1).

Original: We randomly sample sets of skills in the form of variable-length movement primitives, compute the description lengths of a large dataset of goal-reaching trajectories using these skill code books and then measure the efﬁciency of learning to solve an unseen goalreaching task, using both a searchbased optimizer (A ∗ ) and RL-based policy optimization.
Revised: We randomly sample sets of skills in the form of variable-length movement primitives, compute the description lengths of a large dataset of goal-reaching trajectories using these skill code books and then measure the efﬁciency of learning to solve an unseen goalreaching task, using both a searchbased optimizer (A ∗ ) and RL-based policy optimization.

Original: We ﬁnd that skill sets which result in shorter data description lengths, lead to more efﬁcient learning of unseen tasks across both optimizers, supporting our previous intuition.
Revised: We ﬁnd that skill sets which result in shorter data description lengths, lead to more efﬁcient learning of unseen tasks across both optimizers, supporting our previous intuition.

Original: Next, we will describe how to learn MDL skill code books.
Revised: Next, we will describe how to learn MDL skill code books.

Original: 3. L EARNING M INIMUM D ESCRIPTION L ENGTH S KILLS
Revised: 3. L EARNING M INIMUM D ESCRIPTION L ENGTH S KILLS

Original: Our goal is to learn a set of skills such that we can represent the training action trajectories x
Revised: Our goal is to learn a set of skills such that we can represent the training action trajectories x

Original: = a 1 , . .
Revised: = a 1 , . .

Original: . , a H for a large H efﬁciently.
Revised: . , a H for a large H efﬁciently.

Original: In Section 3.1 we summarized how the ELBO objective can be used to learn a MDL representation of the training data with an encoder q ( z | x ) and a decoder p ( x | z ) .
Revised: In Section 3.1 we summarized how the ELBO objective can be used to learn a MDL representation of the training data with an encoder q ( z | x ) and a decoder p ( x | z ) .

Original: Yet, this formulation assumed that the encoder compresses the whole input data sequence x into a single message symbol z which gets transmitted and decoded.
Revised: Yet, this formulation assumed that the encoder compresses the whole input data sequence x into a single message symbol z which gets transmitted and decoded.

Original: However, we aim to encode trajectories from the training dataset into sequences of skills z 1: N .
Revised: However, we aim to encode trajectories from the training dataset into sequences of skills z 1: N .

Original: Thus, we need to introduce an additional component to our model: a boundary encoder q b ( b | x ) that determines how to split the input sequence into variable-length skills.
Revised: Thus, we need to introduce an additional component to our model: a boundary encoder q b ( b | x ) that determines how to split the input sequence into variable-length skills.

Original: An overview of our model is depicted in Figure 2 (left).
Revised: An overview of our model is depicted in Figure 2 (left).

Original: We can formulate the ELBO for our factorized skill learning model as:
Revised: We can formulate the ELBO for our factorized skill learning model as:

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, p ( z n ) and p b ( b n ) are unit Gaussian and uniform priors respectively.
Revised: Here, p ( z n ) and p b ( b n ) are unit Gaussian and uniform priors respectively.

Original: D KL denotes the
Revised: D KL denotes the

Original: Kullback-Leibler divergence and the coefﬁcients β, β b are introduced to weight the KL regularization terms, as is common in the literature on amortized variational inference (Higgins et al., 2017).
Revised: Kullback-Leibler divergence and the coefﬁcients β, β b are introduced to weight the KL regularization terms, as is common in the literature on amortized variational inference (Higgins et al., 2017).

Original: In practice, the boundary encoder outputs a per-timestep discrete variable which indicates whether this timestep is the beginning of a new skill.
Revised: In practice, the boundary encoder outputs a per-timestep discrete variable which indicates whether this timestep is the beginning of a new skill.

Original: We instantiate all components of our model with neural networks and optimize the objective in Equation 2 with gradient descent, using Gumbel softmax reparametrization (Jang et al., 2017; Maddison et al., 2017) and the straight-through gradient estimator (Bengio et al., 2013) for the discrete boundary variable b .
Revised: We instantiate all components of our model with neural networks and optimize the objective in Equation 2 with gradient descent, using Gumbel softmax reparametrization (Jang et al., 2017; Maddison et al., 2017) and the straight-through gradient estimator (Bengio et al., 2013) for the discrete boundary variable b .

Original: For further implementation details, see Section A.2.
Revised: For further implementation details, see Section A.2.

Original: Leveraging MDL Skills for Downstream Learning.
Revised: Leveraging MDL Skills for Downstream Learning.

Original: To leverage the learned skills during learning of a new task, we train a policy π ( z | s ) which takes in the current state and outputs a skill z (see Figure 2, right).
Revised: To leverage the learned skills during learning of a new task, we train a policy π ( z | s ) which takes in the current state and outputs a skill z (see Figure 2, right).

Original: To decode the skill, we additionally learn a boundary decoder p b ( b | z ) which is trained to match the boundary encoder’s output.
Revised: To decode the skill, we additionally learn a boundary decoder p b ( b | z ) which is trained to match the boundary encoder’s output.

Original: Given its predicted skill length, we roll out the skill decoder q ( x | z ) for the predicted number of steps and execute the decoded actions.
Revised: Given its predicted skill length, we roll out the skill decoder q ( x | z ) for the predicted number of steps and execute the decoded actions.

Original: Following
Revised: Following

Original: SPiRL (Pertsch et al., 2020a)
Revised: SPiRL (Pertsch et al., 2020a)

Original: we train a skill prior p ( z | s ) jointly with our model and regularize the policy towards it during downstream task learning to guide exploration.
Revised: we train a skill prior p ( z | s ) jointly with our model and regularize the policy towards it during downstream task learning to guide exploration.

Original: For policy optimization we use the SPiRL algorithm, which is a modiﬁed version of Soft Actor-Critic (Haarnoja et al., 2018a;b).
Revised: For policy optimization we use the SPiRL algorithm, which is a modiﬁed version of Soft Actor-Critic (Haarnoja et al., 2018a;b).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our approach in a 2D continuous control maze navigation environment based on Fu et al.
Revised: We evaluate our approach in a 2D continuous control maze navigation environment based on Fu et al.

Original: (2020) (see Figure 3, left), in which the agent is controlled with planar velocity commands to reach a ﬁxed goal location.
Revised: (2020) (see Figure 3, left), in which the agent is controlled with planar velocity commands to reach a ﬁxed goal location.

Original: Learning this task is challenging since the agent needs to execute hundreds of actions before reaching the goal, at which it only receives a sparse binary reward signal.
Revised: Learning this task is challenging since the agent needs to execute hundreds of actions before reaching the goal, at which it only receives a sparse binary reward signal.

Original: We collect a dataset of 10 , 000 trajectories between random start-goal pairs for training of our MDL skills and then train a policy that leverages the skills to navigate to an unseen goal.
Revised: We collect a dataset of 10 , 000 trajectories between random start-goal pairs for training of our MDL skills and then train a policy that leverages the skills to navigate to an unseen goal.

Original: Analogous to Section 3.2, we compare our approach to an agent that uses a random sampling heuristic for discovering skill code books without optimizing the skill’s description length, as well as a state-of-the-art model-free RL agent (SAC, (Haarnoja et al., 2018a)) that does not leverage pre-trained skills.
Revised: Analogous to Section 3.2, we compare our approach to an agent that uses a random sampling heuristic for discovering skill code books without optimizing the skill’s description length, as well as a state-of-the-art model-free RL agent (SAC, (Haarnoja et al., 2018a)) that does not leverage pre-trained skills.

Original: Our results are summarized in Figure 3 (right): we ﬁnd that the agent leveraging MDL skills can effectively learn the challenging, long-horizon task while the agent that does not optimize the description length of its skills fails to make learning progress.
Revised: Our results are summarized in Figure 3 (right): we ﬁnd that the agent leveraging MDL skills can effectively learn the challenging, long-horizon task while the agent that does not optimize the description length of its skills fails to make learning progress.

Original: The visualization of the rollouts during training in Figure 5 shows, that only the MDL skills lead to effective exploration of the maze environment, supporting our intuition from Section 3.2.
Revised: The visualization of the rollouts during training in Figure 5 shows, that only the MDL skills lead to effective exploration of the maze environment, supporting our intuition from Section 3.2.

Original: The SAC agent completely fails at the task since it is unable to leverage pre-trained skills to effectively explore the maze and receive the sparse reward signal, underlining the challenge the tested task presents for current RL algorithms.
Revised: The SAC agent completely fails at the task since it is unable to leverage pre-trained skills to effectively explore the maze and receive the sparse reward signal, underlining the challenge the tested task presents for current RL algorithms.

Original: A A PPENDIX A. F OUR R OOM E XPERIMENT D ETAILS
Revised: A A PPENDIX A. F OUR R OOM E XPERIMENT D ETAILS

Original: The 4-room environment (Chevalier-Boisvert et al., 2018) is a grid-world goal-reaching environment with four rooms connected by randomly placed doors.
Revised: The 4-room environment (Chevalier-Boisvert et al., 2018) is a grid-world goal-reaching environment with four rooms connected by randomly placed doors.

Original: We sample initial agent and goal positions in the diagonal rooms (the most difﬁcult conﬁgurations) for testing and all other position combinations for training.
Revised: We sample initial agent and goal positions in the diagonal rooms (the most difﬁcult conﬁgurations) for testing and all other position combinations for training.

Original: To build codebooks, we ﬁrst sample a large number (2000) of training environment setups and collect trajectories by running A*.
Revised: To build codebooks, we ﬁrst sample a large number (2000) of training environment setups and collect trajectories by running A*.

Original: The heuristic we use for A* is the sum of absolute horizontal and vertical distances plus the number of turns required to reach the goal.
Revised: The heuristic we use for A* is the sum of absolute horizontal and vertical distances plus the number of turns required to reach the goal.

Original: We then randomly dissect each trajectory into skill segments.
Revised: We then randomly dissect each trajectory into skill segments.

Original: We assign a different skill
Revised: We assign a different skill

Original: length range for each codebook such that its DL is distinct enough from others.
Revised: length range for each codebook such that its DL is distinct enough from others.

Original: These length ranges have different variances, but always the same mean (5).
Revised: These length ranges have different variances, but always the same mean (5).

Original: For example, there is a codebook with 2, 6, 7-length skills and another one with 4, 5, 6-length skills.
Revised: For example, there is a codebook with 2, 6, 7-length skills and another one with 4, 5, 6-length skills.

Original: For evaluation, we only use the ﬁrst 15 most frequent skills from each codebook.
Revised: For evaluation, we only use the ﬁrst 15 most frequent skills from each codebook.

Original: The ﬁrst downstream task is running the same A* solver but enables it with skills from the codebook.
Revised: The ﬁrst downstream task is running the same A* solver but enables it with skills from the codebook.

Original: We deﬁne the search cost as the total number of visited nodes and compute the average search cost over a large number of runs (500).
Revised: We deﬁne the search cost as the total number of visited nodes and compute the average search cost over a large number of runs (500).

Original: We are able to get a 0.70 correlation between DL and A* search cost, see Figure 1.
Revised: We are able to get a 0.70 correlation between DL and A* search cost, see Figure 1.

Original: This means codebooks with smaller DL have lower average cost on A*.
Revised: This means codebooks with smaller DL have lower average cost on A*.

Original: The difference is more apparent when looking at visitation heatmaps over time, see Figure 4.
Revised: The difference is more apparent when looking at visitation heatmaps over time, see Figure 4.

Original: The second downstream task is training a standard DQN agent with the codebooks.
Revised: The second downstream task is training a standard DQN agent with the codebooks.

Original: The 4-room environment reward is deﬁned as 1-0.9*(step count/max steps) .
Revised: The 4-room environment reward is deﬁned as 1-0.9*(step count/max steps) .

Original: Our network hidden size is 128.
Revised: Our network hidden size is 128.

Original: We use an Adam optimizer with a learning rate of 3e-5.
Revised: We use an Adam optimizer with a learning rate of 3e-5.

Original: We have a discount factor of 0.99.
Revised: We have a discount factor of 0.99.

Original: We also employ an epsilon greedy exploration with a 0.1 epsilon.
Revised: We also employ an epsilon greedy exploration with a 0.1 epsilon.

Original: Our replay buffer size is 1e6.
Revised: Our replay buffer size is 1e6.

Original: For each codebook, we train DQN with 8 random seeds.
Revised: For each codebook, we train DQN with 8 random seeds.

Original: To evaluate the performance of a codebook, we measure the average regret over all seeds and study its standard error.
Revised: To evaluate the performance of a codebook, we measure the average regret over all seeds and study its standard error.

Original: We ﬁnd a 0.
Revised: We ﬁnd a 0.

Original: correlation between DL and DQN regret, which conﬁrms our theory that smaller description length helps downstream learning efﬁciency.
Revised: correlation between DL and DQN regret, which conﬁrms our theory that smaller description length helps downstream learning efﬁciency.

Original: A. S KILL M ODEL I MPLEMENTATION D ETAILS
Revised: A. S KILL M ODEL I MPLEMENTATION D ETAILS

Original: Our MDL skill model consists of 5 major networks: a prior boundary detector p b ( b | z ) , a posterior boundary detector q b ( b | x ) , a skill prior predictor p ( z | s ) , a skill posterior encoder q ( z | x, b ) , and a decoder p ( x | z ) .
Revised: Our MDL skill model consists of 5 major networks: a prior boundary detector p b ( b | z ) , a posterior boundary detector q b ( b | x ) , a skill prior predictor p ( z | s ) , a skill posterior encoder q ( z | x, b ) , and a decoder p ( x | z ) .

Original: The posterior boundary detector q b ( b | x ) is a bidirectional LSTM (Hochreiter &
Revised: The posterior boundary detector q b ( b | x ) is a bidirectional LSTM (Hochreiter &

Original: Schmidhuber, 1997) that predicts skill segmentations based on the input action trajectories x .
Revised: Schmidhuber, 1997) that predicts skill segmentations based on the input action trajectories x .

Original: It predicts logits for every time step, from which we then sample a single discrete posterior boundary length for the entire skill (casting the logits as those of a Categorical distribution).
Revised: It predicts logits for every time step, from which we then sample a single discrete posterior boundary length for the entire skill (casting the logits as those of a Categorical distribution).

Original: The posterior boundary network marks timesteps in which new skills start with a 1 , while all other steps are marked with a 0 .
Revised: The posterior boundary network marks timesteps in which new skills start with a 1 , while all other steps are marked with a 0 .

Original: The resulting output of the boundary detector is a (batched) array of 1 ’s and 0 ’s at each timestep, with the same length as the input sequence length.
Revised: The resulting output of the boundary detector is a (batched) array of 1 ’s and 0 ’s at each timestep, with the same length as the input sequence length.

Original: The skill posterior encoder q ( z | x, b ) is an LSTM that scans the input action sequence and outputs the means and log-variances of a posterior skill (Gaussian) distribution for each skill segmentation as instructed by the posterior boundary indicator (the input is the concatenation of input action trajectories x and the boundary posterior’s outputs).
Revised: The skill posterior encoder q ( z | x, b ) is an LSTM that scans the input action sequence and outputs the means and log-variances of a posterior skill (Gaussian) distribution for each skill segmentation as instructed by the posterior boundary indicator (the input is the concatenation of input action trajectories x and the boundary posterior’s outputs).

Original: The skill prior predictor p ( z | s ) is a multi-layer perceptron (MLP) that takes in the current state and predicts a prior skill distribution for every timestep.
Revised: The skill prior predictor p ( z | s ) is a multi-layer perceptron (MLP) that takes in the current state and predicts a prior skill distribution for every timestep.

Original: We then sample the latent skill z for each skill using either the posterior skill distribution.
Revised: We then sample the latent skill z for each skill using either the posterior skill distribution.

Original: The decoder reconstructs actions from each encoded skill z , and its recurrent hidden state is masked out by the output of the posterior boundary network (so that whenever a new skill starts the decoder’s hidden state is reset).
Revised: The decoder reconstructs actions from each encoded skill z , and its recurrent hidden state is masked out by the output of the posterior boundary network (so that whenever a new skill starts the decoder’s hidden state is reset).

Original: The prior boundary detector p b ( b | z ) is an MLP takes the skill z as input and generates logits for each timestep which are formed into a categorical distribution.
Revised: The prior boundary detector p b ( b | z ) is an MLP takes the skill z as input and generates logits for each timestep which are formed into a categorical distribution.

Original: During RL training, a higher level policy outputs latent z values which are decoded by the skill decoder for the amount of timesteps as indicated by sampling the prior boundary detector with z as its input.
Revised: During RL training, a higher level policy outputs latent z values which are decoded by the skill decoder for the amount of timesteps as indicated by sampling the prior boundary detector with z as its input.

Original: For training loss, we compute the discrete KL divergence between the posterior boundary and a ﬁxed prior boundary, which we select to be a Gamma distribution with rate 10 and concentration 0.5.
Revised: For training loss, we compute the discrete KL divergence between the posterior boundary and a ﬁxed prior boundary, which we select to be a Gamma distribution with rate 10 and concentration 0.5.

Original: We also compute the discrete KL divergence between the prior boundary and the detached posterior boundary samples, masked by the output of the posterior boundary detector as the prior boundary detector’s predictions are only used at the start of each skill during RL training.
Revised: We also compute the discrete KL divergence between the prior boundary and the detached posterior boundary samples, masked by the output of the posterior boundary detector as the prior boundary detector’s predictions are only used at the start of each skill during RL training.

Original: Similarly, we compute the KL divergence between the skill posterior and a ﬁxed skill prior (unit Gaussian), and the KL divergence between the detached skill posterior’s outputs z , and the learned skill prior’s predictions of z .
Revised: Similarly, we compute the KL divergence between the skill posterior and a ﬁxed skill prior (unit Gaussian), and the KL divergence between the detached skill posterior’s outputs z , and the learned skill prior’s predictions of z .

Original: Action reconstruction loss is computed as the mean squared error between the output action trajectories produced by the decoder and the ground truth input actions.
Revised: Action reconstruction loss is computed as the mean squared error between the output action trajectories produced by the decoder and the ground truth input actions.