Original: Efficient Task Adaptation by Mixing Discovered Skills
Revised: Efficient Task Adaptation by Mixing Discovered Skills

Original: Authors

Original: Abstract
Revised: Abstract

Original: Unsupervised skill discovery is one of the approaches by which the agent learns potentially useful and distinct behaviors without any explicit reward.
Revised: Unsupervised skill discovery is one of the approaches by which the agent learns potentially useful and distinct behaviors without any explicit reward.

Original: The agent is then expected to quickly solve downstream tasks by properly using a set of discovered skills rather than learning everything from scratch.
Revised: The agent is then expected to quickly solve downstream tasks by properly using a set of discovered skills rather than learning everything from scratch.

Original: However, it is non-trivial to optimally utilize the discovered skills for each task, which can be viewed as a fine-tuning method, and this has been less considered in the literature in spite of its importance.
Revised: However, it is non-trivial to optimally utilize the discovered skills for each task, which can be viewed as a fine-tuning method, and this has been less considered in the literature in spite of its importance.

Original: In this paper, we compare some fine-tuning methods showing how they inefficiently utilize the discovered skills and also propose new methods, which are sample-efficient and effective by interpreting the skills as a perspective of how an agent transforms the input state.
Revised: In this paper, we compare some fine-tuning methods showing how they inefficiently utilize the discovered skills and also propose new methods, which are sample-efficient and effective by interpreting the skills as a perspective of how an agent transforms the input state.

Revised: Our code is available at https://github.com/jsrimr/unsupervisedRL.

Original: 1. Introduction
Revised: 1. Introduction

Original: Unsupervised learning for deep reinforcement learning as a pretraining phase is challenging in the sense that an agent should interact with the world and collect data to obtain useful information without any explicit reward.
Revised: Unsupervised learning for deep reinforcement learning as a pretraining phase is challenging in the sense that an agent should interact with the world and collect data to obtain useful information without any explicit reward.

Original: That way, the agent is supposed to gather information from the environment and learn representation of visited states, dynamics, and behaviors.
Revised: That way, the agent is supposed to gather information from the environment and learn representation of visited states, dynamics, and behaviors.

Original: When a downstream task is given, the agent is expected to quickly adapt to it by leveraging the learned representation, and this is considered as a fine-tuning phase.
Revised: When a downstream task is given, the agent is expected to quickly adapt to it by leveraging the learned representation and this is considered as a fine-tuning phase.

Original: Recent works have shown that skill discovery is a promising way as a pretraining phase to learn diverse behaviors or skills, which potentially can help the agent quickly achieve a goal (Salge et al., 2014; Gregor et al., 2016;
Revised: Recent works have shown that skill discovery is a promising way as a pretraining phase to learn diverse behaviors or skills, which potentially can help the agent quickly achieve a goal (Salge et al., 2014; Gregor et al., 2016;

Original: Eysenbach et al., 2018), in an unsupervised manner.
Revised: Eysenbach et al., 2018), in an unsupervised manner.

Original:  Although the agent could learn diverse skills, it is not obvious at a glance how the agent manages them to solve a Preliminary work.
Revised: [Figure] Although the agent could learn diverse skills, it is not obvious at a glance how the agent manages them to solve a downstream task.

Original: Under review by the First Workshop of Pre- training: Perspectives, Pitfalls and Paths at ICML

Original: Do not distribute.

Original: Policy

Original: Skills

Original: State Action Optional downstream task. Since each skill is represented by a separate policy, the agent has to accordingly handle multiple policies to properly take actions in a typical reinforcement learning framework.
Revised: Since each skill is represented by a separate policy, the agent has to accordingly handle multiple policies to properly take actions in a typical reinforcement learning framework.

Original: DIAYN (Eysenbach et al., 2018) suggests a fine-tuning method that measures the performance of all discovered skills to find the best one for the given downstream task, which might be expensive as the total number of skills grows, and fine-tune it.
Revised: DIAYN (Eysenbach et al., 2018) suggests a fine-tuning method that measures the performance of all discovered skills to find the best one for the given downstream task, which might be expensive as the total number of skills grows, and fine-tune it.

Original: Meanwhile, the recent work (Laskin et al., 2021) suggests a simpler method that randomly samples a single skill to be used for some interval and uses multiple skills during fine-tuning.
Revised: Meanwhile, the recent work (Laskin et al., 2021) suggests a simpler method that randomly samples a single skill to be used for some interval and uses multiple skills during fine-tuning.

Original: However, both fine-tuning methods show poor performance compared to the agent that learns everything from scratch.
Revised: However, both fine-tuning methods show poor performance compared to the agent that learns everything from scratch.

Original: Instead of exhaustive skill search or naive skill sampling, we propose new fine-tuning methods to efficiently and effectively utilize the discovered skills.
Revised: Instead of exhaustive skill search or naive skill sampling, we propose new fine-tuning methods to efficiently and effectively utilize the discovered skills.

Original: To properly understand how to combine discovered skills, we first take a look at an instructive example based on DIAYN.
Revised: To properly understand how to combine discovered skills, we first take a look at an instructive example based on DIAYN.

Original: Each skill is represented by a one-hot vector z , and it is used to derive the corresponding policy π ( a | s, z ) .
Revised: Each skill is represented by a one-hot vector z , and it is used to derive the corresponding policy π ( a | s, z ) .

Original: The state representation ˜ s
Revised: The state representation ˜ s

Original: =
Revised: =

Original: Ws is simply transformed (shifted) into different skill-spaces according to the selected skill representation ˜ z
Revised: Ws is simply transformed (shifted) into different skill-spaces according to the selected skill representation ˜ z

Original: = Uz .
Revised: = Uz .

Original: That way, we consider the skill representation ˜ z
Revised: That way, we consider the skill representation ˜ z

Original: as an agent’s perspective and examine whether the agent can benefit from combining different skill-wise perspectives and manipulating the state representation in some shared skill-spaces.
Revised: as an agent’s perspective and examine whether the agent can benefit from combining different skill-wise perspectives and manipulating the state representation in some shared skill-spaces.

Original: Therefore, we mainly focus on how to combine a set of discovered skills.
Revised: Therefore, we mainly focus on how to combine a set of discovered skills.

Original: By formulating the policy in a specific way, we derive skill fusion (mixing) methods in a state-agnostic manner, which combines skills independent to the input state, and a state-aware manner that uses the information of the input state.
Revised: By formulating the policy in a specific way, we derive skill fusion (mixing) methods in a state-agnostic manner, which combines skills independent to the input state, and a state-aware manner that uses the information of the input state.

Original: 2. Preliminaries and Related Work
Revised: 2. Preliminaries and Related Work

Original: Background.
Revised: Background.

Original: We consider a Markov Decision Process (MDP) M =
Revised: We consider a Markov Decision Process (MDP) M =

Original: ( S , A , p ) without an external reward function for a typical unsupervised reinforcement learning.
Revised: ( S , A , p ) without an external reward function for a typical unsupervised reinforcement learning.

Original: S is a state space, A is an action space, and p ( s ′ | s, a ) is a dynamics.
Revised: S is a state space, A is an action space, and p ( s ′ | s, a ) is a dynamics.

Original: Given a skill z
Revised: Given a skill z

Original: sampling from a skill distribution p ( z ) , a policy π ( a | s, z ) is learned in option framework.
Revised: sampling from a skill distribution p ( z ) , a policy π ( a | s, z ) is learned in option framework.

Original: Skill discovery.
Revised: Skill discovery.

Original: In unsupervised RL setting, skill discovery means to learn diverse behaviors with an intrinsic reward such as empowerment (Salge et al., 2014).
Revised: In unsupervised RL setting, skill discovery means to learn diverse behaviors with an intrinsic reward such as empowerment (Salge et al., 2014).

Original: This is formulated by maximizing mutual information between states and skills I ( S ; Z ) =
Revised: This is formulated by maximizing mutual information between states and skills I ( S ; Z ) =

Original: H ( S ) − H
Revised: H ( S ) − H

Original: ( S ; Z )
Revised: ( S ; Z )

Original: =
Revised: =

Original: H ( Z ) − H
Revised: H ( Z ) − H

Original: ( Z ; S ) , where H ( · ) is the entropy in information theory.
Revised: ( Z ; S ) , where H ( · ) is the entropy in information theory.

Original: There are a number of previous methods to optimize the objective above.
Revised: There are a number of previous methods to optimize the objective above.

Original: VIC (Gregor et al., 2016) and DIAYN (Eysenbach et al., 2018) consider the final state and every state, respectively, in order to discriminate each skill.
Revised: VIC (Gregor et al., 2016) and DIAYN (Eysenbach et al., 2018) consider the final state and every state, respectively, in order to discriminate each skill.

Original: The better the discriminator distinguishes each skill, The more intrinsic rewards the agent obtains.
Revised: The better the discriminator distinguishes each skill, The more intrinsic rewards the agent obtains.

Original: Thanks to the discriminator, the agent is motivated to learn distinct skills to maximize intrinsic reward.
Revised: Thanks to the discriminator, the agent is motivated to learn distinct skills to maximize intrinsic reward.

Original: HIDIO (Zhang et al., 2021) learns high-level policy choosing skills in a hierarchical manner.
Revised: HIDIO (Zhang et al., 2021) learns high-level policy choosing skills in a hierarchical manner.

Original: APS (Liu & Abbeel, 2021) leverages successor features to learn diverse behaviors.
Revised: APS (Liu & Abbeel, 2021) leverages successor features to learn diverse behaviors.

Original: DADS (Sharma et al., 2019) focuses on the predictability of behaviors and learning dynamics.
Revised: DADS (Sharma et al., 2019) focuses on the predictability of behaviors and learning dynamics.

Original: After the skill discovery (pretraining) phase, an agent should adapt to the task with learned skills.
Revised: After the skill discovery (pretraining) phase, an agent should adapt to the task with learned skills.

Original: A simple but general method is to select one skill with the initially highest reward for each task and further fine-tune this skill (Eysenbach et al., 2018;
Revised: A simple but general method is to select one skill with the initially highest reward for each task and further fine-tune this skill (Eysenbach et al., 2018;

Original: Laskin et al., 2022).
Revised: Laskin et al., 2022).

Original: Some methods train an additional high-level policy to choose skills to utilize more than just one skill for each task in a sequential way (Sharma et al., 2019).
Revised: Some methods train an additional high-level policy to choose skills to utilize more than just one skill for each task in a sequential way (Sharma et al., 2019).

Original: For a specific purposed task such as goal-reaching, there are zero-shot skill adaptation methods (Sharma et al., 2019; Choi et al., 2021).
Revised: For a specific purposed task such as goal-reaching, there are zero-shot skill adaptation methods (Sharma et al., 2019; Choi et al., 2021).

Original: We propose the general but effective fine-tuning methods that achieve sample efficiency and high performance.
Revised: We propose the general but effective fine-tuning methods that achieve sample efficiency and high performance.

Original: 3. Proposed Method
Revised: 3. Proposed Method

Original: The main idea of our approach is to combine the learned skills to utilize for downstream tasks.
Revised: The main idea of our approach is to combine the learned skills to utilize for downstream tasks.

Original: We acquire skills in the pretrain phase following DIAYN (Eysenbach et al., 2018) framework which is generally used as a skill discovery baseline.
Revised: We acquire skills in the pretrain phase following DIAYN (Eysenbach et al., 2018) framework which is generally used as a skill discovery baseline.

Original: 3.1. Understanding Skill Fusion
Revised: 3.1. Understanding Skill Fusion

Original: We note that the skill z is a k -dimensional discrete random variable sampled from the distribution p ( z ) .
Revised: We note that the skill z is a k -dimensional discrete random variable sampled from the distribution p ( z ) .

Original: Then we can write a policy as
Revised: Then we can write a policy as

Original: [Equation]
Revised: [Equation]

Original: One can fine-tune a controller p ( z | s ) as in a hierarchical
Revised: One can fine-tune a controller p ( z | s ) as in a hierarchical

Original: RL framework (Sharma et al., 2019), or simply sample skill from p ( z ) (Laskin et al., 2021) under the assumption that skill distribution is independent to the state distribution.
Revised: RL framework (Sharma et al., 2019), or simply sample skill from p ( z ) (Laskin et al., 2021) under the assumption that skill distribution is independent to the state distribution.

Original: In both cases, skill z is first sampled from p ( z ) or p ( z | s ) , and action a is derived from the policy depending on the skill.
Revised: In both cases, skill z is first sampled from p ( z ) or p ( z | s ) , and action a is derived from the policy depending on the skill.

Original: The limitation of those methods is that they just sequentially perform skills rather than combine and mix the skills in an underlying representation level.
Revised: The limitation of those methods is that they just sequentially perform skills rather than combine and mix the skills in an underlying representation level.

Original: Instead, werather formulate the policy as
Revised: Instead, we formulate the policy as

Original: [Equation]
Revised: [Equation]

Original: for some representation function ψ .
Revised: for some representation function ψ .

Original: In this work, we fix ψ as a one-hot vector representation of each discrete skill z in a natural manner.
Revised: In this work, we fix ψ as a one-hot vector representation of each discrete skill z in a natural manner.

Revised: With this formulation, the skill space is expanded to the k -dimensional simplex { ˜ z | 1 ⊤ ˜ z

Revised: = 1 , ˜ z ⪰ 0 } .

Original: In the following sections, we introduce state-agnostic perspective fusion and state-aware perspective fusion depending on whether skill z is sampling from p ( z ) or p ( z | s ) .
Revised: In the following sections, we introduce state-agnostic perspective fusion and state-aware perspective fusion depending on which distribution the skill z follows, p ( z ) or p ( z | s ) .

Original: 3.2. State-agnostic Perspective Fusion
Revised: 3.2. State-agnostic Perspective Fusion

Original: We first consider the method to sample skills from p ( z ) regardless of the input state.
Revised: We first consider the skill follows the distribution p ( z ) regardless of the input state.

Original: The simplest way of fine-tuning a policy is to choose one best skill, p ( z ) is fixed as Dirac delta, as in the original DIAYN paper.
Revised: The simplest way of fine-tuning a policy is to choose one best skill, p ( z ) is fixed as Dirac delta, as in the original DIAYN paper.

Original: However, it utilizes just one skill for a task which has a clear limitation when the task is complicated that only one skill is not enough, also, it is often expensive to know which skill is best-performing especially when the skill set is large.
Revised: However, it utilizes just one skill for a task which has a clear limitation when the task is complicated that only one skill is not enough, also, it is often expensive to know which skill is best-performing especially when the skill set is large.

Original: Now we propose two state-agnostic fusion methods to alleviate these issues.
Revised: Now we propose two state-agnostic fusion methods to alleviate these issues.

Original: First, we fix p ( z ) to be uniform distribution, which combines the learned skills in the same weight.
Revised: First, we fix p ( z ) to be uniform distribution, which combines the learned skills in the same weight.

Original: This simply takes advantage of mixing every skill without any additional parameters.
Revised: This simply takes advantage of mixing every skill without any additional parameters.

Original: Second, we train p ( z ) as a categorical distribution, with additional trainable parameter w i for E z ∼ p ( z ) [ ψ ( z )]
Revised: Second, we train p ( z ) as a categorical distribution, with additional trainable parameter w i for E z ∼ p ( z ) [ ψ ( z )]

Original: = (cid:80)
Revised: = (cid:80)

Original: i w i ψ ( z i ) , where (cid:80)
Revised: i w i ψ ( z i ) , where (cid:80)

Original: i w i
Revised: i w i

Original: = 1 and w i ≥ 0 .
Revised: = 1 and w i ≥ 0 .

Original: The latter is generalized version of the original
Revised: The latter is generalized version of the original

Original: DIAYN and the former method.
Revised: DIAYN and the former method.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: We can interpret the representation level skill mix as follows.
Revised: We can interpret the representation level skill mix as follows

Original:  Since the skill z is represented as a one-hot vector, the weight parameters in a policy are switched on up to the location of 1 in a skill vector.
Revised: [Table] Since the skill z is represented as a one-hot vector [Table] the weight parameters in a policy are switched on up to the location of 1 in a skill vector.

Original: Therefore, if all positions of a skill vector are non-zero, all weights are activated and more diverse representations are obtained.
Revised: Therefore, if all positions of a skill vector are non-zero, all weights are activated and more diverse representations are obtained.

Original: We can view this as the same state from different perspectives through skill.
Revised: We can view this as the same state from different perspectives through skill.

Original:  State-aware Perspective Fusion
Revised: 3.3. State-aware Perspective Fusion

Original: In the method above, skill is combined without considering the input state.
Revised: In the method above [Table] skill is combined without considering the input state.

Original: However, we expect that better performance can be achieved if the combination of skills is changed adaptively according to the state.
Revised: However, we expect that better performance can be achieved if the combination of skills is changed adaptively according to the state.

Original: While it is possible to train the controller p ( z | s ) from scratch, we propose a method to reuse a skill discriminator q ϕ ( z | s ) trained in the pretraining phase.
Revised: While it is possible to train the controller p ( z | s ) from scratch, we propose a method to reuse a skill discriminator q ϕ ( z | s ) trained in the pretraining phase.

Original: In DIAYN, the skill discriminator q ϕ ( z | s ) predicts which skill is in charge of the input state so that each skill visits distinct states.
Revised: In DIAYN, the skill discriminator q ϕ ( z | s ) predicts which skill is in charge of the input state so that each skill visits distinct states.

Original: Therefore, we can guide the agent in the natural direction rather than randomly combining skills by using a discriminator as an initializer of the controller.
Revised: Therefore, we can guide the agent in the natural direction rather than randomly combining skills by using a discriminator as an initializer of the controller.

Original: We show in the next section that although the state-aware fusion method is not as sample efficient as the state-agnostic method, it finally achieves a better performance.
Revised: We show in the next section that although the state-aware fusion method is not as sample efficient as the state-agnostic method, it finally achieves a better performance.

Original: 4. Experiments
Revised: 4. Experiments

Original: We evaluate our methods on URLB (Laskin et al., 2021), which contains three continuous control environments and twelve downstream tasks.
Revised: We evaluate our methods on URLB (Laskin et al., 2021), which contains three continuous control environments and twelve downstream tasks.

Original: The environments include 6-DOF
Revised: The environments include 6-DOF

Original: Walker , 12-DOF Quadruped , and 9-DOF Jaco arm .
Revised: Walker , 12-DOF Quadruped , and 9-DOF Jaco arm .

Original: We follow the same evaluation process to URLB.
Revised: We follow the same evaluation process to URLB.

Original: We pretrain each agent without explicit rewards, then finetune for the downstream task.
Revised: We pretrain each agent without explicit rewards, then finetune for the downstream task.

Original: In the case of skill discovery method, the agent learns a set of useful behaviors which we call skills.
Revised: In the case of skill discovery method, the agent learns a set of useful behaviors which we call skills.

Original: Then the agent uses these skills to achieve higher cumulative rewards quickly in fine-tuning.
Revised: Then the agent uses these skills to achieve higher cumulative rewards quickly in fine-tuning.

Original: When we evaluate our methods, we fix the pretrained agent to DIAYN based on DDPG (Lillicrap et al., 2016) and finetune it in various ways.
Revised: When we evaluate our methods, we fix the pretrained agent to DIAYN based on DDPG (Lillicrap et al., 2016) and finetune it in various ways.

Original: 4.1. Sample-efficiency and Final Performance
Revised: 4.1. Sample-efficiency and Final Performance

Original: [Figure]
Revised: [Figure]

Original: others. We compare our methods to three baselines, DDPG trained from scratch, DIAYN implemented in URLB ( URLB DIAYN ) and DIAYN proposed in the original paper ( Original
Revised: We compare our methods to three baselines, DDPG trained from scratch, DIAYN implemented in URLB ( URLB DIAYN ) and DIAYN proposed in the original paper ( Original

Original: DIAYN ).
Revised: DIAYN ).

Original: For Original DIAYN , we did not count the steps for selecting the best skill in order to make a baseline more challenging.
Revised: For Original DIAYN , we did not count the steps for selecting the best skill in order to make a baseline more challenging.

Original: Also, we choose the task Walker-run for the comparison because it is known as the most challenging task among the twelve downstream tasks in URLB (Laskin et al., 2021).
Revised: Also, we choose the task Walker-run for the comparison because it is known as the most challenging task among the twelve downstream tasks in URLB (Laskin et al., 2021).

Original: We fine-tune each method for 100k steps.
Revised: We fine-tune each method for 100k steps.

Original: The experiment in other environments is shown in section 4.2.
Revised: The experiment in other environments is shown in section 4.2.

Original: Same importance and Simple importance are both stateagnostic fusion methods, but the former fixes p ( z ) and latter finetunes.
Revised: Same importance and Simple importance are both stateagnostic fusion methods, but the former fixes p ( z ) and latter finetunes.

Original: DIAYN controller initializes a controller p ( z | s ) with DIAYN skill discriminator, but scratch controller does not.
Revised: DIAYN controller initializes a controller p ( z | s ) with DIAYN skill discriminator, but scratch controller does not.

Original: The detail of our methods is in Table 1.
Revised: The detail of our methods is in Table 1.

Original: We observe the following results in Figure 3.
Revised: We observe the following results in Figure 3.

Original: URLB DIAYN underperforms other methods even including DDPG trained from scratch, showing that it does not fully use the potential of learned skills, and suggesting that fine-tuning is critical to measure the performance of the skill discovery method.
Revised: URLB DIAYN underperforms other methods even including DDPG trained from scratch, showing that it does not fully use the potential of learned skills, and suggesting that fine-tuning is critical to measure the performance of the skill discovery method.

Original: On the other hand, both of our state-agnostic methods perform well, showing that the simple skill mix methods help to achieve high return quickly.
Revised: On the other hand, both of our state-agnostic methods perform well, showing that the simple skill mix methods help to achieve high return quickly.

Original: The performance of state-aware methods is worse than state-agnostic methods and similar to the baseline ( Original DIAYN ) at 100k step.
Revised: The performance of state-aware methods is worse than state-agnostic methods and similar to the baseline ( Original DIAYN ) at 100k step.

Original: We presume training additional parameters for controller disturbs the fast adaptation.
Revised: We presume training additional parameters for controller disturbs the fast adaptation.

Original: [Figure]
Revised: [Figure]

Original: We also compare our methods for the final performance.
Revised: We also compare our methods for the final performance.

Original: We show in figure 4 that state-aware fusion method outperforms state-agnostic fusion at some point.
Revised: We show in figure 4 that state-aware fusion method outperforms state-agnostic fusion at some point.

Original: Followed by a certain amount of steps, the capacity of state-aware methods works.
Revised: Followed by a certain amount of steps, the capacity of state-aware methods works.

Original: Moreover, we observe that DIAYN controller outperforms the Scratch controller .
Revised: Moreover, we observe that DIAYN controller outperforms the Scratch controller .

Original: This shows that the DIAYN discriminator trained at pretrained stage is a good initializer for the skill controller.
Revised: This shows that the DIAYN discriminator trained at pretrained stage is a good initializer for the skill controller.

Original: This transferred weight not only helped better performance at the final step, but also helped robust training which has lower variance.
Revised: This transferred weight not only helped better performance at the final step, but also helped robust training which has lower variance.

Original: 4.2. Comparison to Other URLB Methods
Revised: 4.2. Comparison to Other URLB Methods

Original: We evaluate how quickly the agent adapt to twelve downstream tasks in the same setting to URLB (Laskin et al., 2021), 2M pre-train steps and 100k fine-tune steps.
Revised: We evaluate how quickly the agent adapt to twelve downstream tasks in the same setting to URLB (Laskin et al., 2021), 2M pre-train steps and 100k fine-tune steps.

Original: Among
Revised: Among

Original: [Figure]
Revised: [Figure]

Original: our proposed methods, as we have seen on the experiment above, same importance method is the most sample-efficient.
Revised: our proposed methods, as we have seen on the experiment above, same importance method is the most sample-efficient.

Original: Therefore, we finetune pretrained DIAYN with same importance for 3 seeds per task.
Revised: Therefore, we finetune pretrained DIAYN with same importance for 3 seeds per task.

Original: We compared our method with 9 unsupervised RL algorithms including DIAYN with a vanilla fine-tuning.
Revised: We compared our method with 9 unsupervised RL algorithms including DIAYN with a vanilla fine-tuning.

Original: Expert performance is a DDPG result after 2M steps of fine-tuning (Laskin et al., 2022).
Revised: Expert performance is a DDPG result after 2M steps of fine-tuning (Laskin et al., 2022).

Original: Other best is a state-of-the-art among twelve unsupervised RL reported in URLB.
Revised: Other best is a state-of-the-art among twelve unsupervised RL reported in URLB.

Original: As shown in Table 2, although the simple fine-tune method without any additional weight is applied, it outperforms every other methods in 9 out of 12 tasks.
Revised: As shown in Table 2, although the simple fine-tune method without any additional weight is applied, it outperforms every other methods in 9 out of 12 tasks.

Original: Even for the tasks that other methods work better(Quadruped-stand, Jacoreach-bottom-right, Jaco-reach-top-right), our method just slightly underperforms.
Revised: Even for the tasks that other methods work better(Quadruped-stand, Jacoreach-bottom-right, Jaco-reach-top-right), our method just slightly underperforms.

Original: Especially, it achieves an enormous improvement compared to the reported original DIAYN.
Revised: Especially, it achieves an enormous improvement compared to the reported original DIAYN.

Original: From this experiment, we can recognize how important the fine-tuning method is for evaluating a skill discovery methods.
Revised: From this experiment, we can recognize how important the fine-tuning method is for evaluating a skill discovery methods.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: In this paper, we propose a new fine-tuning method for skill discovery for combining learned skills efficiently.
Revised: In this paper, we propose a new fine-tuning method for skill discovery for combining learned skills efficiently.

Original: By combining skills on the representation level rather than performing skills sequentially, it is possible to generalize the previous method and outperform other unsupervised RL methods in a sample efficiency manner.
Revised: By combining skills on the representation level rather than performing skills sequentially, it is possible to generalize the previous method and outperform other unsupervised RL methods in a sample efficiency manner.

Original: We emphasize the importance of fine-tuning method for measuring the performance of a skill discovery method and encourage the research direction for other skill mix methods for future work. 
Revised: We emphasize the importance of fine-tuning method for measuring the performance of a skill discovery method and encourage the research direction for other skill mix methods for future work. Acknowledgement

Revised: This work was partially supported by Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT)

Revised: [NO.2021-0-01343, Artificial Intelligence

Revised: Graduate School Program (Seoul National University)] and the ICT RD program of MSIT/IITP[2021-0-00077] and

Revised: MOTIE[P0014715].

Original: A. Results
Revised: A. Results

Original: We compare our method same importance to expert which is trained by DDPG with 2M steps from (Laskin et al., 2022), other best which shows the best performance among 9 unsupervised RL algorithms in URLB (Laskin et al., 2021) [Table] and
Revised: We compare our method same importance to expert which is trained by DDPG with 2M steps from (Laskin et al., 2022), other best which shows the best performance among 9 unsupervised RL algorithms in URLB (Laskin et al., 2021), and

Original: URLB DIAYN.
Revised: URLB DIAYN

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B. Hyperparameters
Revised: B. Hyperparameters

Original: We present the best performing hyperparamter borrowed from URLB (Laskin et al., 2021)
Revised: We present the best performing hyperparamter borrowed from URLB (Laskin et al., 2021)

Original: [Table] [Table]
Revised: [Table] [Table]