Paper: Reinforcement Learning of Diverse Skills using Mixture of Deep Experts (ICLR-2024)

Reviewer comment (Writing):
The motivation behind the research is not clearly articulated. It is unclear whether the authors intend to discover diverse solutions within the same task or seek experts for all tasks/contexts.

Author response:
*"The motivation behind the research is not clearly articulated. It is unclear whether the authors intend to discover diverse solutions within the same task or seek experts for all tasks/contexts."* Also related to the Question *"What is the goal of the method? Is it trying to discover diverse solutions or seek experts for different contexts/tasks?"*

* We acknowledge your concerns regarding the unclear motivation of our work. It appears there might have been a misinterpretation of our approach/intentions. To clarify,  Di-SkilL is a reinforcement learning method to train a Mixture of Experts (MoE) model that is capable of learning both: a) solutions for all contexts and b) diverse solutions within the same task defined by a specific context. To enhance the clarity of our paper, we have revised the relevant parts to ensure this concept is more clearly communicated. Moreover, in the following paragraphs, we summarize the main points:
    * Contextual Episodic Policy Search (CEPS) generally considers training an agent that generalizes to (continuous-valued) contexts and a  context defines the task within an environment. In the robot table tennis environment, for example, a context can be the ball’s initial velocity, ball landing position, and desired ball landing position on the opponent’s side. CEPS assumes that the environment has its internal context distribution $p(c)$, which is unknown to the agent. During training, the agent observes a context $c$ and proposes a motion primitive parameter $\theta$ which results in a desired trajectory. This is formalized in the optimization problem in Eq. (1).
    The goal of Di-SkilL is to train a mixture of experts (MoE) policy that generalizes well to all contexts that are within the support of      $p(c)$. At the same time, Di-SkilL learns different solutions, i.e. behaviors to a specific context $c$, resulting in different strategies for the task defined by $c$. If we assume that the context in table tennis defines the initial ball velocity, ball landing position, and desired ball landing position on the opponent’s side, Di-SkilL is able to learn different striking types in the same context $c$. We tried to visualize these behaviors in Fig. 4 and in the videos provided in the supplementary material.

Reviewer comment (Writing):
The paper lacks sufficient detail regarding the definition of the mixture of experts model, including the definition of an expert. Furthermore, the relationships between context (c), expert (o), and the parameter θ are not adequately explained.

Author response:
*"The paper lacks sufficient detail regarding the definition of the mixture of experts model, including the definition of an expert. Furthermore, the relationships between context (c), expert (o), and the parameter θ are not adequately explained."* Also related to Question *"What is the definition of a expert and how it is executed in certain context/task."*
* The general definition of a mixture of experts model is provided in Eq. 2. However, this definition requires parameterizing a gating distribution $\pi(o|c)$, which allows sampling an expert $o$  after observing a (continuous-valued) context $c$. Due to the finite number of experts, $\pi(o|c)$ can e.g. be a conditioned categorical distribution. The expert $\pi(\theta|c,o)$ is a parameterized distribution (usually a Gaussian) that maps the contexts $c$ to motion primitive parameters $\theta$. 
However, this form of parameterization does not allow for automatic curriculum learning. Therefore, we rely on the MoE definition in Eq. 3. This definition was already used in prior works (citations were added to the paper). In our work, the per-expert context distribution is parameterized to provide the possibility to adjust its curriculum for each expert $o$. As suggested by the $\sum_o$ in Eq. 2 and Eq. 3, there is a discrete number of experts in each MoE model.

* An expert represents a mapping of contexts $c$ to motion primitive parameters $\theta$. We describe this in Section 2 (CEPS). In our work, we parameterize each expert with a neural network, where the mean is a context-dependent function and the covariance can either be context-dependent or not. We tried to visualize this relation in Fig. 1 and further mentioned it in Section 3.3 “Expert Update”. We extended Section 2 where we explain the parameterization of each expert and added a dedicated section in Appendix A where we describe the parameterizations of each part of the MoE model in detail.

Reviewer comment (Experiments):
The experimental section appears to be confined to relatively simple scenarios, and the demonstrated diversity of the trained experts is limited.

Author response:
*"The experimental section appears to be confined to relatively simple scenarios, and the demonstrated diversity of the trained experts is limited."*

* We agree that evaluating our method on more tasks would strengthen the work. We are currently working on further experiments and plan to provide them before the end of the author-reviewer discussion. 
* Additionally, we would like to emphasize that the “table tennis” and the “box pushing with obstacles” environments are extensions of the environments provided in [1], where they were extensively tested. Moreover, we extend the environments by increasing the dimensionality of the context spaces: for the table tennis task, we not only vary the initial ball landing position and the ball goal landing position but also vary the ball’s initial velocity. For the box-pushing environment, we additionally include a varying obstacle location. Both environments are challenging benchmark tasks using the state-of-the-art physics engine Mujoco[2]. This makes exploration even more challenging because the agent needs to infer the dynamics solely based on environmental interactions. In the box-pushing task, for example, friction leads to highly non-linear behaviors and for the table tennis task, the understanding of the ball’s trajectory is more difficult due to the varying initial velocity.  Additionally, successfully completing an episode is challenging due to the strict definitions as described in Section 4. This means that the robot has to perform very accurate and precise motions to achieve a position and rotation error below the success-threshold. 

[1] Otto et al, Deep black-box reinforcement learning with movement primitives.


[2] Todorov et al, MuJoCo: A physics engine for model-based control

Reviewer comment (Writing):
What is the goal of the method? Is it trying to discover diverse solutions or seek experts for different contexts/tasks?

Author response:
*"The motivation behind the research is not clearly articulated. It is unclear whether the authors intend to discover diverse solutions within the same task or seek experts for all tasks/contexts."* Also related to the Question *"What is the goal of the method? Is it trying to discover diverse solutions or seek experts for different contexts/tasks?"*

* We acknowledge your concerns regarding the unclear motivation of our work. It appears there might have been a misinterpretation of our approach/intentions. To clarify,  Di-SkilL is a reinforcement learning method to train a Mixture of Experts (MoE) model that is capable of learning both: a) solutions for all contexts and b) diverse solutions within the same task defined by a specific context. To enhance the clarity of our paper, we have revised the relevant parts to ensure this concept is more clearly communicated. Moreover, in the following paragraphs, we summarize the main points:
    * Contextual Episodic Policy Search (CEPS) generally considers training an agent that generalizes to (continuous-valued) contexts and a  context defines the task within an environment. In the robot table tennis environment, for example, a context can be the ball’s initial velocity, ball landing position, and desired ball landing position on the opponent’s side. CEPS assumes that the environment has its internal context distribution $p(c)$, which is unknown to the agent. During training, the agent observes a context $c$ and proposes a motion primitive parameter $\theta$ which results in a desired trajectory. This is formalized in the optimization problem in Eq. (1).
    The goal of Di-SkilL is to train a mixture of experts (MoE) policy that generalizes well to all contexts that are within the support of      $p(c)$. At the same time, Di-SkilL learns different solutions, i.e. behaviors to a specific context $c$, resulting in different strategies for the task defined by $c$. If we assume that the context in table tennis defines the initial ball velocity, ball landing position, and desired ball landing position on the opponent’s side, Di-SkilL is able to learn different striking types in the same context $c$. We tried to visualize these behaviors in Fig. 4 and in the videos provided in the supplementary material.

Reviewer comment (Reproducibility):
What is the definition of an expert and how it is executed in certain context/task.

Author response:
*"The paper lacks sufficient detail regarding the definition of the mixture of experts model, including the definition of an expert. Furthermore, the relationships between context (c), expert (o), and the parameter θ are not adequately explained."* Also related to Question *"What is the definition of a expert and how it is executed in certain context/task."*
* The general definition of a mixture of experts model is provided in Eq. 2. However, this definition requires parameterizing a gating distribution $\pi(o|c)$, which allows sampling an expert $o$  after observing a (continuous-valued) context $c$. Due to the finite number of experts, $\pi(o|c)$ can e.g. be a conditioned categorical distribution. The expert $\pi(\theta|c,o)$ is a parameterized distribution (usually a Gaussian) that maps the contexts $c$ to motion primitive parameters $\theta$. 
However, this form of parameterization does not allow for automatic curriculum learning. Therefore, we rely on the MoE definition in Eq. 3. This definition was already used in prior works (citations were added to the paper). In our work, the per-expert context distribution is parameterized to provide the possibility to adjust its curriculum for each expert $o$. As suggested by the $\sum_o$ in Eq. 2 and Eq. 3, there is a discrete number of experts in each MoE model.

* An expert represents a mapping of contexts $c$ to motion primitive parameters $\theta$. We describe this in Section 2 (CEPS). In our work, we parameterize each expert with a neural network, where the mean is a context-dependent function and the covariance can either be context-dependent or not. We tried to visualize this relation in Fig. 1 and further mentioned it in Section 3.3 “Expert Update”. We extended Section 2 where we explain the parameterization of each expert and added a dedicated section in Appendix A where we describe the parameterizations of each part of the MoE model in detail.

Reviewer comment (Reproducibility):
While the concept of the proposed technique is easy to follow, some important details are missing and it might affect the reproducibility of the proposed approach.

Author response:
*"While the concept of the proposed technique is easy to follow, some important details are missing and it might affect the reproducibility of the proposed approach".*

* We presume that the reviewer is referring to the points in the “Questions” part and kindly ask to specify what are missing details apart from the questions if there are any.
* Additionally, we want to note that we will be open-sourcing the code and the environments upon acceptance

Reviewer comment (Reproducibility):
To better understand the action space of the contextual episode-based policy, could the author give some details or examples of the concept of motion primitives and how to convert the policy parameters into the episode-wise robot action trajectory?

Author response:
1. *"To better understand the action space of the contextual episode-based policy, could the author give some details or examples of the concept of motion primitives and how to convert the policy parameters into the episode-wise robot action trajectory?*

* Instead of a high-dimensional representation of a trajectory (e.g. each state in a single time-step), Motion Primitives (MPs) introduce a lower dimensional vector $\theta$ which concisely defines the trajectory[2].  The generation of the trajectory depends on the method that is used. ProMPs [3]  for example define the whole trajectory as a linear function of $\theta$ as $\tau = \Phi^T \theta$, where $\Phi$ are pre-defined and time-dependent basis functions (e.g. normalized radial basis functions). Dynamic Movement Primitives (DMPs) [4]  rely on second-order dynamic systems that provide smooth trajectories in the position and velocity space. In our work, we use ProDMPs [2] which combine the advantages of both methods. In the context of reinforcement learning, the policy $\pi(\theta|c)$, or in our case, the chosen expert $\pi(\theta|c,o)$ defines a normal distribution over the parameters $\theta$ depending on the context $c$.  The agent can therefore quickly adapt to contexts and follow the trajectory for example using a PD-controller. We have added a dedicated section in Appendix B to further clarify the relationship between our work and motion primitives. If there are any additional unclarities that should be addressed, we would appreciate the feedback from the reviewer

  [2] Ge Li et al, ProDMP: A Unified Perspective on Dynamic and Probabilistic Movement Primitives

  [3] A. Paraschos et al, Probabilistic movement primitives

  [4] Stefan Schaal et al, Dynamic movement primitives a framework for motor control in humans and humanoid robotics

Reviewer comment (Theory):
Eq (3) seems to be not original from this work, a proper reference would help readers to understand the background of this line of work. The derivation from Eq (4) to Eq (5) and Eq (6) is unclear. It would be more clear to have an intermediate objective which is jointly optimizing for $\pi(\theta|c, o)$ and $\pi(c|o)$, and derive from there to have two separate objectives for bi-level optimization.

Author response:
2. *"Eq (3) seems to be not original from this work, a proper reference would help readers to understand the background of this line of work. The derivation from Eq (4) to Eq (5) and Eq (6) is unclear. It would be more clear to have an intermediate objective which is jointly optimizing for $\pi(\theta|c,o)$ and $\pi(c|o)$, and derive from there to have two separate objectives for bi-level optimization."*
* We thank the reviewer for this comment. We have added the reference in the revised manuscript. 
* The derivations of the lower bounds for the expert and per-expert context distributions have been proposed and derived in detail in [5]. However, we agree with the reviewer that the derivation from Eq. (4) to Eq.(5) (now Eq.(6) in the updated paper) is not trivial. Therefore, we have included an intermediate step (now Eq.(5)) to clarify it.

    [5] Celik et al, Specializing Versatile Skill Libraries using Local Mixture of Experts

Reviewer comment (Writing):
In Section 3.1, it says “mapping the context $c$ to a mean vector and a covariance matrix” and “Note that in most cases a context dependent covariance matrix is not useful and introduces unnecessary complex relations.” It is confusing that whether the covariance matrix in the implementation is context dependent.

Author response:
3. *"In Section 3.1, it says “mapping the context to a mean vector and a covariance matrix” and “Note that in most cases a context dependent covariance matrix is not useful and introduces unnecessary complex relations.” It is confusing that whether the covariance matrix in the implementation is context dependent."*

* We agree with the reviewer that this sentence might lead to confusion. Indeed, in our experiments, we have observed that using a full covariance matrix that does not depend on the context leads to more stable training behavior. We therefore did not use a context-dependent covariance matrix in our experiments. To clarify this, we have described the parameterization in Appendix A in detail.

Reviewer comment (Presentation):
Line 10 in Section 3.2, should “Fig. 2c” be “Fig. 2d”?

Author response:
4. *"Line 10 in Section 3.2, should “Fig. 2c” be “Fig. 2d”?"*

* We thank the reviewer for pointing out the mistyped reference. We have corrected this mistake.

Reviewer comment (Theory):
Which terms in Eq (8) and Eq (9) accounts for encouraging the coverage of the context space by experts? From the formulation, it seems to try to learn a set of policy each of which can solve the entire task space as much as possible. The learning of policies seem to be relatively independent and is it possible to learn a set of experts whose preferred context distributions are the same.

Author response:
5. *"Which terms in Eq (8) and Eq (9) accounts for encouraging the coverage of the context space by experts? From the formulation, it seems to try to learn a set of policy each of which can solve the entire task space as much as possible. The learning of policies seem to be relatively independent and is it possible to learn a set of experts whose preferred context distributions are the same."*

* We thank the reviewer for pointing out this important question which is crucial to understanding our proposed method. Please note that the equations the reviewer was referring to are numbered Eq.(9) and Eq.(10) respectively in the updated paper. 

    * We first want to clarify the roles of the individual terms in Eq.(9). The objective incentivizes each expert $\pi(\theta|c,o)$ to maximize its entropy. This means that the expert should cover as much as possible of the parameter space, i.e. $\theta$- space, while still maximizing the reward $R(\theta, c)$. At the same time, the objective incentivizes the expert to cover only context-parameter regions that are not covered by other experts $o$. This is guaranteed by the variational distribution $\tilde{\pi}(o|c, \theta)$ which translates to the probability of assigning a context-parameter pair to expert $o$. As a result, the agent will try to maximize this probability as it is an argument of the log.

    * A similar relation can be seen in Eq. (10). The per-expert context distribution $\pi(c|o)$ will try to cover as much as possible of the context region due to the entropy bonus. Yet, due to the variational distribution $\tilde{\pi}(o|c)$ the objective incentivizes the agent to distinguish between the different per-expert context distributions for a given context. In other words, the objective forces each $\pi(c|o)$ to concentrate more on regions that are not covered by other experts $o$ because the log of the variational distribution rewards context samples that can be assigned to $o$ with high probability.

    * We updated the manuscript and clarified these relations in Section 3.3

    * As a result, the optimization of the different experts and per-experts are coupled by the variational distributions which hinder them from concentrating on the same context-parameter regions. Please note that overlapping regions exist that explicitly are desired to find diverse solutions to the same task defined by context $c$. In other words, it is favorable that different experts cover a part of the same context as their adjusted Motion Primitive Parameter $\theta$ will lead to different solutions to the same context, which is incentivized by $\tilde{\pi}(o|c,\theta)$. The amount of “overlapping” in the context space depends on the choice of the hyperparameters $\alpha$ and $\beta$, which was analyzed in detail in prior work [5] already

Reviewer comment (Experiments):
Evaluating the algorithm on more environments will make the comparison more thorough. For example, it would be helpful to evaluate on the other tasks used in Otto et al. 2023.

Author response:
7. *"Evaluating the algorithm on more environments will make the comparison more thorough. For example, it would be helpful to evaluate on the other tasks used in Otto et al. 2023."*

* We agree that evaluating the algorithm on more environments is helpful. We are currently working on more evaluations and plan to provide the results before the deadline of the reviewer-author discussion phase.

Reviewer comment (Experiments):
It would also helpful to show complete comparison against both SVSL and BBRL on all evaluated tasks (at least provide comparison plots in Appendix).

Author response:
8. *"It would also helpful to show complete comparison against both SVSL and BBRL on all evaluated tasks (at least provide comparison plots in Appendix)"*

* We believe that the reviewer meant the lack of comparison to SVSL only, as the results of BBRL are reported on all environments already. But please let us know if that is not the case, and please further clarify your suggestion in this case. We agree that a complete comparison to SVSL is valuable. However, due to the increased complexity of the context space, we were not able to successfully run SVSL on the extended Table Tennis and Box pushing environment. We believe there are several reasons for this: SVSL requires prior knowledge about the context space. For the table tennis environment, for example, we need to know the ball’s flying trajectory to determine whether the serving position on the robot’s table side is valid or not, in order to provide a guiding punishment term such that the per-expert context distributions are usefully updated. Alternatively, a constant punishment term could be added to the reward if the context is invalid after running in the simulator. We tried the latter and could not successfully train SVSL. Furthermore, SVSL has difficulties training experts for higher dimensional context spaces that additionally introduce non-linearities in the behavior space, for example in the box pushing task with obstacles. The Gaussian parameterization of the per-expert distributions and the linear experts were not sufficient to learn skills in our experiments. 

    Yet, we agree that additional comparison is useful to strengthen the method. Therefore, we have run a similar variant of SVSL, where we have replaced SVSL’s per-expert context distribution with the energy-based context distribution of Di-SkilL. Additionally, we have parameterized the experts as linear experts. The results can be seen in Fig. 4(a) and Fig.4 (b) in the updated paper.  ‘LinDi-SkilL’  overcomes the problem of the need for pre-knowledge of the context space as required by SVSL, but is not able to reach the performance of the non-linear expert counterpart Di-SkilL.

Reviewer comment (Experiments):
Given the proposed approach is built upon SVSL with two improvements, it would be great to do ablation study on both improvement techniques.

Author response:
9. *"Given the proposed approach is built upon SVSL with two improvements, it would be great to do ablation study on both improvement techniques."*

* We agree that such a comparison is useful to strengthen our method. The benefits of Di-SkilL over SVSL are:  
    1. Di-SkilL does not make any assumptions about the environment’s context space. I.e. due to Di-SkilL’s energy-based model (EBM) for the per-expert context distribution, there is no need to shape the reward function with guiding punishments to ensure that contexts are sampled in a valid region. This is particularly beneficial if the context influences the dynamics of relevant objects in the environment (e.g. initial ball velocity in table tennis), requiring pre-knowledge for defining a useful punishment function. 
    2. The EBM can represent hard non-linearities (visualized in e.g. Fig. 2d)) in the context space. These types of context regions are commonly present in applications, for example in the table tennis task, where the ball’s initial velocity has an upper bound and might lead to additional  “holes” in the probability space depending on the ball’s initial and target ball landing positions. These probability landscapes could also be represented with e.g. Gaussian Mixture Models as in SVSL, but require many components to achieve a reasonable approximation
    3. Di-SkilL can learn highly non-linear experts represented by neural networks. This is beneficial for learning complex behaviors in sophisticated environments and provides a big advantage over linear experts e.g. in SVSL. 

* As mentioned in our answer to Question 8, we were not able to successfully train an SVSL agent on the extended Table Tennis and Box Pushing task. However, we have conducted experiments with ‘LinDi-SkilL’ which aims to prove the advantages of Di-SkilL over SVSL. ‘LinDi-SkilL’ uses the EBM of Di-SkilL, but has linear expert parameterizations. We can now clearly see that ‘LinDi-SkilL’ is able to learn useful skills (Fig. 4a) +b)), but can not reach the performance of Di-SkilL, indicating that the EBM helps shape the expert's curriculum, but the linear experts are not sufficient to perform well enough. We believe these additional experiments address the reviewer’s concerns and are open to further suggestions

Reviewer comment (Theory):
The multi-modality in this work is achieved by mixture of experts, however each expert is still modeled by uni-model gaussian policy.

Author response:
10. *"The multi-modality in this work is achieved by mixture of experts, however each expert is still modeled by uni-model gaussian policy."*
* It is true that each expert is modeled as a uni-model Gaussian, however, as described in the answer to Question 5, the overlapping probability mass to a certain extent in the context space enables learning diverse, i.e. multi-modal behaviors. For these overlapping context regions several experts $\pi(\theta|c,o)$ are responsible and provide different MP parameters $\theta$ which lead to different behaviors. This is also visualized in in Fig. 4 c)- e), Fig. 5 as well as in the videos in the supplementary material.

Reviewer comment (Novelty):
Recent work (Huang et al. 2023 [1]) proposes some multi-modal policy parameterization. How is the proposed approach compare to this work and can the proposed approach enhanced by the policy reparameterization from [1]?

Author response:
11. *"Recent work (Huang et al. 2023 [1]) proposes some multi-modal policy parameterization. How is the proposed approach compare to this work and can the proposed approach enhanced by the policy reparameterization from [1]?"*
* We thank the author for mentioning the work in [1]. The proposed objective in [1] is related to the objective we use. We have therefore included this work in the related work discussion.  Yet, we want to emphasize that the objective shown in Section 2 was proposed in prior work [5]. Both objectives are similar in the sense that they lower bound an initial objective using techniques from variational inference for usage with latent variable models. Additionally, if the objective in [5] is applied to the classical MDP, i.e. step-based RL set up without any curriculum learning (learning the gating $\pi(o|c) $ instead of $\pi(c|o)$), the resulting lower-bound is the same as proposed in [1].

    However, there are significant differences to our setting: First, the work in [1] does not consider curriculum learning, i.e. there is no context distribution learned, the work does not consider the Contextual Policy Search problem using Motion Primitives (MPs), i.e. their work act under the commonly known Markov decision process assumptions. Furthermore, we consider a Mixture of Experts policy with a discrete number of experts which does not require parameterizing the responsibility ($\pi(o|c,\theta)$) and the gating ($\pi(o|c))$ terms as they can be calculated in closed form. Finally, the work in [1] considers the Model-based Reinforcement Learning case where we do not learn a dynamics model. 

    Indeed, considering a continuous latent variable o (z in [1]) is an interesting approach for future work and could improve the performance

Reviewer comment (Experiments):
Is this proposed approach also applicable to step-based RL problems?

Author response:
12. *"Is this proposed approach also applicable to step-based RL problems?"*

* Applying our approach to step-based RL is an interesting research question that requires detailed analysis especially how to integrate the learning of the context distribution that is responsible for automatic curriculum learning. Probably the most intuitive option would be to allow the agent to set the context once at the beginning of the episode and then consider the learning problem in the standard MDP framework. Both frameworks have their advantages and disadvantages. While step-based approaches exploit the temporal structure of RL problems and are therefore expected to outperform episode-based RL in terms of sample efficiency, episode-based RL methods are usually sample inefficient but are able to smoothly explore in trajectory space which leads to improved performance in sparse and non-markovian reward settings over step-based approaches [6]. In this work, we focused on developing a new method in the framework of episode-based RL and consider the application of the approach in the step-based RL framework as an interesting future work.

    [6] Otto et al, Deep Black-Box Reinforcement Learning with Movement Primitives

Reviewer comment (Novelty):
Automatic curriculum learning is a key ingredient of the proposed method; however, an important set of approaches in this direction has not been covered in related work, such as [1] and others in this family of approaches.

Author response:
*"Automatic curriculum learning is a key ingredient of the proposed method; however, an important set of approaches in this direction has not been covered in related work, such as [1] and others in this family of approaches."*

* We appreciate the reviewer’s advice. We have added a set of works in this family of approaches to the related work discussion

Reviewer comment (Presentation):
Figure 3-c which shows ablations on the TT environment has inconsistent number of episodic samples (X-axis) for the different approaches in the plot. It would be useful to have asymptotic performance of each of these approaches and then compare them in terms of this performance, and also in terms of training speed (eg: w/o automatic curriculum learning is slower than w/ automatic curriculum learning).

Author response:
*"Figure 3-c which shows ablations on the TT environment has inconsistent number of episodic samples (X-axis) for the different approaches in the plot. It would be useful to have asymptotic performance of each of these approaches and then compare them in terms of this performance, and also in terms of training speed (eg: w/o automatic curriculum learning is slower than w/ automatic curriculum learning)."*

* We thank the reviewer for this comment. The goal of this ablation is to show that automatic curriculum learning is a necessary feature of Di-SkilL to learn useful and high-performing skills. Hence the main focus of Fig. 3 c) is the comparison of Di-Skill, Di-SkilLwoCurrV1, and Di-SkilLwoCurrV2. All of the variants are already converged at different numbers of samples. While Di-Skill and Di-SkillwoCurrV1 converge at a similar number of samples, they differ in performance (around 20% difference). Di-SkilLwoCurrV2 needs around twice the number of samples to reach a similar success rate as Di-SkilL.  
Yet, we agree that the learning curve of SVSL was not fully converged. We have rerun SVSL with improved hyperparameters and updated the results in Fig. 3 c). Although there is an improvement in the convergence speed, the performance at the end of the training is the same as before.

Reviewer comment (Evaluation):
In Figure 4 a-b as well, it would be nice to have the asymptotic performance for Di-Skill and BBRL to have a fair comparison of performance.

Author response:
*"In Figure 4 a-b as well, it would be nice to have the asymptotic performance for Di-Skill and BBRL to have a fair comparison of performance."*

* We have rerun both algorithms for an increased number of samples and updated the results in Fig. 4, such that the asymptotic performance of both methods can be clearly compared. Di-SkilL outperforms BBRL by a small margin on the extended table tennis task and clearly achieves better performance (around 20% better) on the Box Pushing task. Please note that we have rerun Di-SkiLL and BBRL with full covariance parameterization on the Box Pushing task, which helped increase the performance for both methods.

Reviewer comment (Experiments):
While SVSL and BBRL are good CEPS baselines, it would also be nice to compare with a standard RL baseline such as PPO to better motivate the need for this approach.

Author response:
*"While SVSL and BBRL are good CEPS baselines, it would also be nice to compare with a standard RL baseline such as PPO to better motivate the need for this approach."*

* We agree with the reviewer that adding PPO as a baseline emphasizes the benefits of Di-SkilL. We are currently working on running PPO on the Box Pushing task, as this is the only environment satisfying the Markov assumption of PPO. We will be adding the results before the end of the reviewer-author discussion deadline. 

    We were not able to successfully run SVSL on the extended table tennis and box pushing tasks, as SVSL requires to design of a punishment term for guiding the per-expert context distribution (e.g. distance to the valid context regions). Designing a suitable punishment term requires good reward-shaping experience and in most cases, knowledge about the influence on the physics of the environment. For example, even though the ball’s initial velocity in the table tennis task is bounded by box constraints, only specific constellations with the other context dimensions are physically valid, leading to a complex probability landscape of the valid region. If the ball is initialized with a high velocity, it might happen that the desired ball landing position is not met. Since the per-expert context distributions define the contexts at the beginning of the episode, it can easily happen that non-valid contexts are sampled in the case of SVSL. However, to account for the missing comparison to SVSL, we have added “LinDi-SkilL”, a variant of Di-SkilL with linear experts. “LinDi-SkilL” benefits from the energy-based model from Di-SkilL and hence does not need special treatments for learning the per-expert context distribution. Additionally, we used linear experts as in SVSL. We have run all experiments for 24 seeds and report the results in Fig.4. For the same number of training samples, LinDi-SkilL can not achieve the performance of Di-SkilL, indicating the advantages over SVSL.

Reviewer comment (Presentation):
The environment description has been duplicated to some extent in the main text and the caption for Figure 4. It may help to prune that and instead include additional analysis.

Author response:
*"The environment description has been duplicated to some extent in the main text and the caption for Figure 4. It may help to prune that and instead include additional analysis."*

* We thank the reviewer for this suggestion to improve our paper. We have updated the text to the environment description in the main text. A detailed description of the environments can be found in Appendix C.

Reviewer comment (Theory):
I am not sure why the prior $\pi(o)$ has been assumed to be uniform. For sparse reward tasks, I can imagine observations that are closer to the goal would be rarer than those closer to the initial state at the beginning of the episode. Or does this paper assume full access to the simulator in which resetting to any observation is possible?

Author response:
*"I am not sure why the prior $\pi(o)$  has been assumed to be uniform. For sparse reward tasks, I can imagine observations that are closer to the goal would be rarer than those closer to the initial state at the beginning of the episode. Or does this paper assume full access to the simulator in which resetting to any observation is possible?"*

* We believe there is a misunderstanding: The distribution $\pi(o)$ is a discrete distribution, specifying the probability of choosing an expert $o$ without observing a context $c$, Thus, $o$ is not the observation of the environment. Generally, the prior can be interpreted as the preference of specific experts $o_{1}, o_2, ...$. This means that the user could include an inductive bias by setting a higher probability to preferred experts. However, due to the gating distribution $\pi(o|c)=\frac{\pi(c|o)\pi(o)}{\pi(c)}$ the agent still decides which expert $o$ is suitable based on the current context $c$. Please note that the prior $\pi(o) $ can be freely chosen. We set it to be uniform in our work, because it is unclear before the training which expert should be preferred and because a uniform distribution over the experts maximizes the entropy (see Eq. (5) in the updated version of the paper.) However, we agree that the sentence “The prior $\pi(o)$ is assumed to be a uniform distribution throughout this work” might lead to confusion, that why we changed it to “The prior $\pi(o) $ is set to a uniform distribution throughout this work. 

    Based on the follow-up question of the reviewer we believe they meant the context $c$, as the context can be interpreted as the observation in the step-based RL setting.  In our work, we implement automatic curriculum learning by allowing the agent to choose its favored contexts $c$ and set them in the environment.  Favored contexts are sampled from the per-expert context distribution $\pi(c|o)$.  Consequently, we assume that the agent can set the context $c$ once at the beginning of the episode. This is done during training such that each expert is allowed to discover the context regions it favors. Please note that even though the optimization of each expert is independent, they are still coupled via the variational distributions $\tilde{\pi}(o|c,\theta) $ and $\tilde{\pi}(o|c)$ such that preferred context regions that are covered by other experts won’t be attractive to the current agent anymore (Eq. (6) + Eq. (7)). During inference, the agent observes a context $c$ and chooses the expert which then executes the adjusted motion primitive parameter $\theta$.  
* We would appreciate it if the reviewer could confirm whether our explanations answer their question, or clarify the remark.

Reviewer comment (Evaluation):
In the Experiments section, the paper mentions that the aim is to check whether Di-Skill is able to learn precise and diverse skills. The fact that it learns diverse skills is reasonably demonstrated in Figure 5, but I am yet to find evidence that precise skills are learnt. Could the authors please point me to that?

Author response:
*"In the Experiments section, the paper mentions that the aim is to check whether Di-Skill is able to learn precise and diverse skills. The fact that it learns diverse skills is reasonably demonstrated in Figure 5, but I am yet to find evidence that precise skills are learnt. Could the authors please point me to that?"*

* By precise skills, we meant, whether the agent is able to learn high-performing skills. The provided environments require highly precise motions to successfully solve the task. In table tennis, for example, the task is considered successful if the distance between the ball’s landing position and the goal position is below 0.2m. This requires the robot to precisely hit the ball, as displaced hitting will lead to different landing positions. In the Box Pushing task, the agent needs to drag the box such that the distance error of the box’ end and goal position is below 0.05m and the rotation error is below 0.5rad, while dealing with e.g. unknown dynamics and friction. However, to avoid confusion we have replaced the term ‘precise’ with ‘high-performing’ which we believe is more intuitive. If the reviewer has other suggestions, we are happy to include them in the description.

Reviewer comment (Evaluation):
Could the authors provide any insights on the learnt $\pi(o | c)$? Of the number of experts used, how often were they used when averaging across observations?

Author response:
*"Could the authors provide any insights on the learnt $\pi(o|c)$ ? Of the number of experts used, how often were they used when averaging across observations?"*

* the gating distribution is not explicitly parameterized and is indirectly learned via 

    $\pi(o|c)=\frac{\pi(c|o)\pi(o)}{\sum_{o'}\pi(c|o')\pi(o')}~~~~~~~~(1)$
    
    as $\pi(c|o)$ is a learned energy-based model. During inference, the agent observes a context $c$, which is used to calculate the gating with equation (1) resulting in probabilities of the different experts. Based on this probability, an expert $o$ is sampled and used to sample a motion primitive parameter $\theta \sim \pi(\theta|c,o)$. During inference, i.e. testing, we sample only one expert per context. If the user is interested in generating diverse skills (such as in Fig. 4 and Fig. 5, and the video in the supplementary material), the user can sample several times from $\pi(o|c)$, depending on how many different skills should be executed.