Paper: Domain Randomization via Entropy Maximization (ICLR-2024)

Reviewer comment (Experiments):
While the method proposed in this study demonstrates a certain level of innovation, it does not appear to be exceptionally groundbreaking. Therefore, it is imperative to present more compelling experimental results. Conducting additional simulation experiments, as well as sim-to-real experiments, would be highly recommended.

Author response:
- > While the method proposed in this study demonstrates a certain level of innovation, it does not appear to be exceptionally groundbreaking. Therefore, it is imperative to present more compelling experimental results. Conducting additional simulation experiments, as well as sim-to-real experiments, would be highly recommended.
   
In our work, we have shown that DORAEMON is able to outperform all baselines in all tasks (in terms of maximum global success rate) and crucially demonstrates an impressive zero-shot sim-to-real transfer in our complex pushing task. In addition, DORAEMON has fewer hyparameters than the other methods (see Sec. A.2).
Nevertheless, we have conducted several more experiments to provide novel useful insights on the performance of DORAEMON. In particular, we added:
  1. **A novel sensitivity analysis of DORAEMON in Sec. A.2**;
  2. **A novel Section A.3** to further discuss the trade-off between the in-distribution success rate vs. entropy;
  3. **A novel Section A.4** with an implementation of DORAEMON with Gaussian distributions, instead of Beta's used in the main manuscript.
  4. **A novel ablation analysis of DORAEMON in Fig. 13**, demonstrating the effect of the backup optimization problem in (6);
   
We claim DORAEMON will open interesting and promising future work directions, which can ultimately provide "exceptionally groundbreaking" results in the field. Such directions include, e.g., the combination of a maximum entropy objective like DORAEMON's with inference-based methods such as SimOpt [3] or DROPO [4]. This way, policies can be learned to best generalize across dynamics parameters located around high-likelihood regions, which can be inferred from real-world data.
   
We believe that our detailed explanations and additional results clarify the contribution of our approach and we hope that the reviewer would consider them.

Reviewer comment (Theory):
The method is heuristic and lacks theoretical analysis.

Author response:
We agree with the Reviewer that the paper lacks theoretical analysis. As the primary goal of our work is to present a novel solution to the sim-to-real problem, we focus on a strong empirical analysis to highlight the relevance of our method to the field.
   In particular, while a theoretical analysis is not provided, DORAEMON introduces a technically sound optimization problem that works in a more principle manner than the previous algorithmic heuristic proposed by AutoDR---e.g., AutoDR can only work with uniform distributions and simply steps the distribution with fixed $\Delta$ increments.
   In our latest revision, we also provide a number of new experiments to further complement the experimental evaluation:
   - Added a sensitivity analysis w.r.t. hyperparameters $\epsilon$ (Sec. A.2), and $\alpha$ (Sec. A.3);
   - Added experiments on DORAEMON with Gaussian distributions, instead of Beta's (Sec. A.4);
   - Added an ablation analysis of DORAEMON (Fig. 13), and discussion on its connections to curriculum learning (Sec. B);
   
   We believe that these new analyses, together with the already thorough experimentation of the method in 6 sim2sim environments and a complex sim2real task, make our contribution highly relevant. A detailed theoretical analyses is left for a journal extension of our work.

Reviewer comment (Experiments):
The proposed method calls the RL algorithm to update the policy for every dynamics parameters sampling, which might lead to an inefficient algorithm. Maybe the embedded RL algorithm only need to return a relatively approximate solution? Does this works for DORAEMON. This is expected to be made clear.

Author response:
DORAEMON does not change anything about the underlying RL subroutine, which, in fact, is not even "informed" about the change of the DR distribution. This allows to use any RL algorithm of choice, and simply call DORAEMON's optimization problem in Eq. (4) every time new $K$ trajectories have been collected by the training agent (line 3 of Algorithm 1). We can interpret this as a standard RL procedure where the learning agent is progressively presented with MDPs $\in \mathcal{U}$ of varying transition dynamics. Therefore, we shall not worry about the intermediate approximate solutions in between each iteration, as DORAEMON automatically adjusts the distribution updates according to the goodness of the current agent. In other words, the distribution won't be updated as drastically (less entropy increase, if any) when the agent only barely satisfies the minimum desired success rate, and viceversa (this phenomenon can be clearly seen in our newly introduced analysis in Fig. 10).

Reviewer comment (Theory):
The authors adopt univariate beta distributions for \nu_{\phi}, which might simplify problem (6). But for more general distributions, solving (6) might be challenging. Maybe this should be further investigated. Or is it the case that some commonly used distributions, e.g. Gaussian, are effective and meanwhile make (6) tractable. Or some variational methods can be adopted.

Author response:
DORAEMON can work with any family of parametric distributions, as long as the entropy (for the objective function) and the KL-divergence (for the trust region constraint) can be conveniently computed. In our experiments, we use independent Beta distributions, and therefore deal with $2$ parameters per dimensions to be optimized by our problem in (4) and (6). This results in a number of optimized parameters of $4$, $14$, $26$, $16$, $14$, and $34$ respectively for the CartPole, Hopper, Walker2D, HalfCheetah, Swimmer, and the PandaPush environments (see Tab. 2 and Tab. 3).
Overall, we find our optimization problem to be particularly efficient. We computed statistics on $500$ runs of our optimization problem in the Hopper environment, and observed an average duration of $4.19 \pm 1.72$ seconds. In particular, we make use of the Scipy library and its convenient "Trust Region Constrained Algorithm"---see line 897 of `doraemon/doraemon.py` in our anonymous codebase in the supplementary material.
   Finally, we tested DORAEMON with Gaussian distributions and **reported the results in a novel Appendix Section A.4**. This analysis further demonstrates that our optimization problem is just as tractable with other parametric distributions, and also led to similar results.

Reviewer comment (Experiments):
Only Beta distributions are considered in the experiments. It is encouraged to add more distribution types to the experiments.

Author response:
While we chose Beta distributions as they can better capture the probability mass of the target uniform distribution, in principle we can use any parametric distribution whose entropy (for the objective function) and KL-divergence (for the trust region constraint) can be computed straightforwardly. In response to the reviewer's comment,   we implemented DORAEMON with Gaussian distributions and report the results in the **newly added Section A.4**. We also depict a comparison of converged distributions in the two cases (Beta's and Gaussian's) in Fig. 12.

Reviewer comment (Evaluation):
Additional visualizations need to be included to illustrate the trade-off between performance and entropy. For example, in Figure 2, the performance of Walker2D and Swimmer decreases when the entropy increases. It is important to explore the relationship between the randomized variable and performance, and to determine the range within which performance decreases.

Author response:
We provide more details on the trade-off between the performance of DORAEMON as the entropy increases, in our novel Appendix sections. In particular, we provide a thorough investigation on multiple environments (including Walker2D as suggested by the reviewer) measuring the in-distribution success rate, the entropy, and the global success rate for varying values of $\epsilon$ and $\alpha$. Such analysis sheds light on the effect of the backup optimization problem in maintaining a feasible performance constraint, and how this does not always translate to a better global success rate.
  Overall, we complemented the original manuscript with:
  1. **A novel sensitivity analysis in Sec. A.2** where we particularly show the effects of the backup optimization problem in Fig. 8 and Fig. 9 for changes of $\epsilon$;
  2. **A novel Section A.3** to discuss the trade-off between in-distribution success rate vs. entropy for the Walker2D environment (Fig. 10), as suggested by the reviewer;
  3. **A novel ablation analysis of DORAEMON in Fig. 13**, demonstrating that all combinations that do not incorporate a backup optimization problem may not maintain a feasible performance constraint during training;
 
  In particular, we point out the results in Fig. 10 for the Walker2D environment, where DORAEMON steadily maintains a feasible in-distribution success rate during training, regardless of the choice of the hyperparameter $\alpha$. However, we observed that a sound tracking of the in-distribution success rate does not necessarily translate to better global success rate, which still shows a decreasing trend. We investigate this phenomenon in Fig. 9, depicting a pair of distributions found at different times during training: despite seemingly having similar entropy values according to the plot, the backup optimization problem significantly changes the distribution parameters to maintain feasible performance constraints, and in turn, moves to easier regions of the dynamics landscape. As a result, the global success rate is also affected. We further ablate the presence of the backup optimization problem in Fig. 13, and discover that its contribution does not negatively affect the global success rate, which would decline regardless if we simply kept training the policy when the performance constraint is violated (i.e. as in SPDL baselines).

Reviewer comment (Experiments):
Adding more real-world experiments is encouraged. PushCube is a relatively easy task in robot manipulation.

Author response:
While adding more real-world experiments would certainly be beneficial to limit-test the algorithm, we believe our pushing task to be a particularly well representative testbed for our analysis, and likely more complicated than it looks at first glance. Indeed, it is easy to overlook the complexity of the task from the Domain Randomization point of view: while a simple pushing-cube task would not be considered particularly challenging in RL literature, it becomes immediately more troublesome when testing under unknown real-world parameters (e.g., center of mass, frictions), hence requiring the policy to generalize over multiple dynamics. This is easily reflected by the poor performance of an agent that randomizes parameters too heavily (Fixed-DR), and by the lack of generalization for agents that do not randomize parameters at all (No-DR).
  
  Given the complexity of setting up real world experiments, and the respective design of a representative simulation environment in the limited rebuttal period, we postpone the further investigation of DORAEMON in a variety of real-world manipulation environments for a journal extension of this work.

Reviewer comment (Experiments):
Can this domain randomization method potentially be applied to object randomness? For example, in ManiSkill environments, some tasks include variations in objects. Can we use maximum entropy to gradually learn from these different objects?

Author response:
We thank the reviewer for this comment. As we have already stated in the final sentences of Sec. 4.1 (*"[...] the formulation is not restricted to a particular family of parametric distributions or even continuous random variables–e.g., discrete distributions over object shapes could be used."*), it is indeed possible to apply DORAEMON to random object shapes. For example, one could consider a categorical distribution over $N$ object types, with initial low entropy---i.e. most of the probability mass is assigned to a single object, and the rest is spread across the others. Then, DORAEMON can be used to progressively increase the entropy of the distribution, e.g. parametrizing it through a softmax of N continuous parameters. Ultimately, DORAEMON will attempt to converge to a uniform distribution over all N object shapes, while ensuring desirably high performance along the process.

Overall, this is a rather interesting point, and opens the possibility to test DORAEMON on discrete dynamics distributions beyond object shapes. We leave this as an open direction for future work.

Reviewer comment (Novelty):
DORAEMON appears to be conceptually very similar to the AutoDR, or ADR in the original paper (Akkaya et al., 2019). They both define some custom indicators of success and iteratively increase the entropy of the dynamics distribution. With that said, DORAEMON appears to be yet a somewhat different implementation of the idea highlighted by ADR.

Author response:
As the reviewer readily pointed out, DORAEMON builds on the same intuition as AutoDR to progressively increase the entropy of the training distribution, but follows a drastically different algorithm to do so. In particular, let us break down the fundamental differences between DORAEMON and AutoDR:
  1. **Uniforms distributions only**: by definition, AutoDR is limited to a training distribution parametrized as uniform. This is a major limitation of the algorithm, which may not capture correlations, nor multimodal effects, nor unbounded dynamics parameters. In contrast, DORAEMON's optimization problem can work with any family of parametric distributions. To support this claim with further evidence, **we added experiments on DORAEMON with Gaussian distributions in the newly introduced Section A.4**.  
  2.  **Fixed step size**: AutoDR may only step the distribution boundaries by a fixed step size $\Delta$, which considerably limits the flexibility of the algorithm. This resulted in a much higher variance of the entropy curves across seeds (see Fig. 2). Instead, DORAEMON leverages the solution of an explicit optimization problem to step the current distribution while remaining within a trust region in KL-divergence space. In turn, DORAEMON can dynamically step all dimensions of the distribution as much as needed to maintain a feasible performance constraint. A recent work by NVIDIA [1] implemented AutoDR with per-dimension $\Delta$ values, which attempts to provide more flexibility at the expense of drastically more hyperparameter tuning.
  3.  **Inefficient use of training samples**: AutoDR does not leverage all training data as information to update the distribution. Rather, it alternates pure training samples from the uniform distribution (50%) to dynamics parameters sampled at the boundaries (50%). While all are used to update the current policy, only the latter samples provide information on the success rate at the boundary to step the distribution. Moreover, these samples may even be discarded if the success rate is not sufficiently high to increase the entropy of the distribution---see $CLEAR(D_i)$ instruction in Alg. 1 of [2]. Conversely, DORAEMON uses all training samples of the current iteration to step the distribution (never discards them), and does not bias the sampling process to occur at the boundaries half the time.
  4.  **More domain knowledge for backtracking**: AutoDR introduces an additional low return threshold $t_L$ to shrink the entropy of the distribution when performances are too low. This mechanism is analogous to the backup optimization problem introduced by DORAEMON, despite our method not requiring an additional hyperparameter, and relying on a more flexible, explicit optimization (for the same reasons as in the **"Fixed step size"** point). The original paper [2] simply sets $t_L$ to be half the high-return threshold $t_H$, but we suspect that this hyperparameter should likely be tuned with task-specific domain knowledge.

Reviewer comment (Experiments):
Based on the above, while the two approaches arise from similar ideas, DORAEMON appears to have a better success rate across almost all environments. It is not immediately clear whether the performance improvement comes from which specific part of the design or it is just a matter of different choices of hyperparameters. While there is a one-sentence discussion on the authors’ conjecture in Section 5.2 (about the potential data inefficiency), it is expected to have a deeper dive into the root cause of this performance difference.

Author response:
As the reviewer readily pointed out, DORAEMON builds on the same intuition as AutoDR to progressively increase the entropy of the training distribution, but follows a drastically different algorithm to do so. In particular, let us break down the fundamental differences between DORAEMON and AutoDR:
  1. **Uniforms distributions only**: by definition, AutoDR is limited to a training distribution parametrized as uniform. This is a major limitation of the algorithm, which may not capture correlations, nor multimodal effects, nor unbounded dynamics parameters. In contrast, DORAEMON's optimization problem can work with any family of parametric distributions. To support this claim with further evidence, **we added experiments on DORAEMON with Gaussian distributions in the newly introduced Section A.4**.  
  2.  **Fixed step size**: AutoDR may only step the distribution boundaries by a fixed step size $\Delta$, which considerably limits the flexibility of the algorithm. This resulted in a much higher variance of the entropy curves across seeds (see Fig. 2). Instead, DORAEMON leverages the solution of an explicit optimization problem to step the current distribution while remaining within a trust region in KL-divergence space. In turn, DORAEMON can dynamically step all dimensions of the distribution as much as needed to maintain a feasible performance constraint. A recent work by NVIDIA [1] implemented AutoDR with per-dimension $\Delta$ values, which attempts to provide more flexibility at the expense of drastically more hyperparameter tuning.
  3.  **Inefficient use of training samples**: AutoDR does not leverage all training data as information to update the distribution. Rather, it alternates pure training samples from the uniform distribution (50%) to dynamics parameters sampled at the boundaries (50%). While all are used to update the current policy, only the latter samples provide information on the success rate at the boundary to step the distribution. Moreover, these samples may even be discarded if the success rate is not sufficiently high to increase the entropy of the distribution---see $CLEAR(D_i)$ instruction in Alg. 1 of [2]. Conversely, DORAEMON uses all training samples of the current iteration to step the distribution (never discards them), and does not bias the sampling process to occur at the boundaries half the time.
  4.  **More domain knowledge for backtracking**: AutoDR introduces an additional low return threshold $t_L$ to shrink the entropy of the distribution when performances are too low. This mechanism is analogous to the backup optimization problem introduced by DORAEMON, despite our method not requiring an additional hyperparameter, and relying on a more flexible, explicit optimization (for the same reasons as in the **"Fixed step size"** point). The original paper [2] simply sets $t_L$ to be half the high-return threshold $t_H$, but we suspect that this hyperparameter should likely be tuned with task-specific domain knowledge.

Reviewer comment (Evaluation):
The successful rate for certain tasks, e.g., Walker in Figure 2 and PandaPush in Figure 11, decline after reaching maximum entropy. However, the algorithm does not dynamically reduce the entropy in response to a decrease in the success rate, which might be necessary for maintaining performance consistency. This appears not consistent with the objective in (4). As the discussion in Section 5.2 does not fully address this phenomenon, more explanation would be needed.

Author response:
We agree that the limited explanation in the main manuscript makes it hard to fully understand this phenomenon, despite we tried our best to deliver the maximum content given the page limit.
  
  In our updated manuscript, and in response to the reviewer's comment, we delve into the details of this phenomenon in the Appendix:
  1. **We added a novel sensitivity analysis in Sec. A.2** where we particularly show the effects of the backup optimization problem in Fig. 8 and Fig. 9 for changes of $\epsilon$;
  2. **We added a novel Section A.3** to discuss the trade-off between in-distribution success rate vs. entropy for the Walker2D environment (Fig. 10), as suggested by the reviewer;
  3. **We added an ablation analysis of DORAEMON in Fig. 13**, demonstrating that all combinations that do not incorporate a backup optimization problem may not maintain a feasible performance constraint during training;

Reviewer comment (Evaluation):
Another concern lies in the constraint based on the success rate. Specifically, the use of success rate largely ignores the effect of the poor trajectories, which could be arbitrarily poor and degrade the robustness of the learned policy. By contrast, in the standard DR, the objective is to consider the expected total return over all the possible trajectories. As the experimental results reported in the paper all focus on the “success rate”, the robustness issue is thereby largely neglected.

Author response:
We believe this point to essentially be a matter of design choice. Our formulation makes use of the success rate to allow for custom success notions to be defined, hence more flexibility. For example, consider the inclined plane task in Sec. 4.2: a succesful trajectory is defined as such when the cart is balanced around the center of the plane for longer than 25 timesteps. An average (expected) return formulation would be impractical in this case. Furthermore, a success notion allows the designer to encode task-specific toleration to errors in the problem.
Overall, DORAEMON may still be used with success notion that are simply defined as return threshold (as in our sim-to-sim analysis in Sec. 5.2), which would be equivalent to consider a *median* return formulation as opposed to the *average* return formulation suggested by the reviewer. This likely makes DORAEMON's optimization problem less affected by catastrophic returns (which would not affect the median performance), which we argue could be beneficial. Nevertheless, the importance sampling estimator in Eq. (5) can be easily adjusted to consider an average return performance constraint.

Reviewer comment (Theory):
In practice, is it computationally easy to optimize (4)? The constrained problem does not seem to be convex (even under Beta distribution)?

Author response:
DORAEMON is particularly efficient. We computed statistics on $500$ runs of our optimization problem (4), and observed an average duration of  $4.19 \pm 1.72$ seconds. As we run (4) iteratively $150$ times for each training session---$15$M timesteps, with $100000$ training timesteps in between each iteration---this results in approx. $10$ minutes of computational power, w.r.t. a total training time of about $16$ hours.
  In particular, we make use of the Scipy library and its convenient "Trust Region Constrained Algorithm"---see line $897$ of `doraemon/doraemon.py` in our anonymous codebase in the supplementary material.

Reviewer comment (Reproducibility):
Could the authors specify the alpha values for Figure 2 and Figure 11? When the entropy matches the max entropy, the global success rate aligns with the local success rate. If the alpha is set to 0.5, why does the global success rate drop below 0.5 when entropy is at its maximum?

Author response:
We use $\alpha=0.5$ across all our experiments. The only parameter that is specifically tuned for each environment is the trust region size (see Sec. A.2 for more information).
  Please refer to our comments above in this post for a detailed explanation on why the global success rate may decrease despite the entropy **appears** to be at its maximum.

Reviewer comment (Theory):
How to design the entropy jointly for multiple dynamics parameters? (For example, simply taking the product of multiple univariate distributions like AutoDR?)

Author response:
Correct. As we work with univariate Beta distributions, the product of these PDFs is considered for the joint distribution whose entropy needs to be computed. We then simply sum the univariate distribution entropies---a convenient property of the differential entropy when dealing with independent distributions, which can be easily verified.

Reviewer comment (Writing):
In Section 3: The notation of reward function shall be consistent (mathcal or not)

Author response:
Thank you for spotting the typo. We fixed the notation.

Reviewer comment (Novelty):
Limited technical novelty. The formulation (Eq. 4) is very similar to that of SPRL[1], SPDL[2], CURROT[3], and GRADIENT[4]. Setting the target distribution to be uninformative -- uniform distribution could transform their objectives into something very similar to Eq.4, and they do not necessarily converge to the final target distribution.

Author response:
Thank you for taking the time to evaluate our work.

> Limited technical novelty. The formulation (Eq. 4) is very similar to that of SPRL[1], SPDL[2], CURROT[3], and GRADIENT[4]. Setting the target distribution to be uninformative – uniform distribution could transform their objectives into something very similar to Eq.4, and they do not necessarily converge to the final target distribution.


The primary goal of the paper is to present a novel solution to the sim-to-real problem, and our empirical results make DORAEMON highly promising and relevant for the field. As a matter of fact, our method outperforms AutoDR, the currently considered state-of-the-art method for sim-to-real transfer in absence of real-world data---recently adopted by NVIDIA in [1] as their go-to DR method for sim-to-real transfer, well after self-paced methods have been introduced. To achieve such results, DORAEMON **does take inspiration from the self-paced optimization problem in the curriculum setting, but our method is *far* from a straightforward application of self-paced methods in a sim-to-real problem**, in contrast to what the reviewer's comment might reflect.
To prove this, we implemented SPDL and compared it to DORAEMON in a thorough analysis that adds individual (and combinations of) components to SPDL, clearly shedding light on the differences between the two algorithms (see Fig. 13). The results demonstrate that a naive application of self-paced methods is impractical or even impossible in the domain randomization setting, and that a number of technical novelties---history-based policies, asymmetric actor-critic, and a backup optimization problem---has to be introduced to obtain desirable performance and prevent violation of the performance constraint.

Overall, we report a summarized list of the key aspects where our sim-to-real problem formulation departs w.r.t. curriculum learning (CL) settings (see Sec. B for details): 
- **Unknown target distribution**: the knowledge of a target distribution is a strict requirement for CL methods. Setting uninformative uniform targets could be an option to bridge its formulation to a sim-to-real problem. However, this prompts the user to define the uniform boundaries as a hyperparameter of the method: while the boundaries should be designed to be as wide as possible, SPDL's practical implementation would likely suffer from uniforms that approach infinite support. This also holds for LSDR, which by definition requires a target uniform distribution to be defined. In contrast, DORAEMON drops such dependency. 
- **Contextual MDPs vs. LMDPs**: CL methods such as SPDL work in a contextual RL setting where a context variable that represents the current task is in principle observable and available to the learner---e.g. goal states that progressively get harder to reach. Conversely, the DR formulation induces the solution of a Latent MDP [2], as dynamics parameters may not be directly observed. Such theoretical change effectively leads to fundamental performance differences for the learning agent (see Fig. 13).
- **I-projection vs. M-projection**: While SPDL cannot be directly applied to a sim-to-real problem due to the different assumptions, it shares similarities with both DORAEMON and LSDR optimization problems. Interestingly, we observe how SPDL makes use of the I-projection formulation to move the training distribution towards the target, while LSDR proposes the M-projection counterpart. In turn, SPDL may drive distributions towards maximum entropy when considering target uniforms (but is limited to parametric families of bounded support for the optimization), while LSDR can work with unlimited support distributions (but results in a maximum likelihood objective that would not grow the entropy indefinitely). Overall, DORAEMON does not rely on a KL-divergence objective, hence (1) can converge to maximum entropy uniform distributions, and (2) can be implemented with any parametric family, including unbounded ones (see the new Section A.4 for DORAEMON with Gaussian distributions).
- **Backup optimization problem**: the practical implementation of self-paced methods does not allow for the entropy to decrease along the process. In turn, the rise of violated performance constrains is not explicitly addressed, and the agent simply keeps training in such occurrences. Our novel backup optimization problem introduced in DORAEMON overcomes this problem. The new empirical analysis in Fig. 13 demonstrates the effectiveness of DORAEMON to consistently move in the feasible region of the optimization landscape, whereas SPDL quickly suffers from training performance collapse.

While not in-depth, these differences have also been highlighted by the recent survey on DR for sim-to-real transfer by F. Muratore et. all [3], in their Sec. 4.1.

Reviewer comment (Theory):
Beta distribution is often not a reasonable choice. It cannot handle multi-modal distribution, while many existing works can handle arbitrary empirical distributions [1,2,3,4].

Author response:
#### Beyond Beta distributions

> Beta distribution is often not a reasonable choice. It cannot handle multi-modal distribution, while many existing works can handle arbitrary empirical distributions [1,2,3,4].

DORAEMON can work with any family of parametric distributions for which their entropy (for the objective function) and KL-divergence (for the trust region constraint) may be conveniently computed, just like SPDL.
In fact, **we tested DORAEMON with Gaussian distributions and report the results in the newly added Section A.4.**


We believe that our detailed explanations and additional results clarify the contribution of our approach and we hope that the reviewer would consider them for a thorough re-assessment of our work.

Reviewer comment (Novelty):
Missing related work:  
   * Klink, Pascal, et al. "Curriculum reinforcement learning via constrained optimal transport." International Conference on Machine Learning. PMLR, 2022.  
   * Huang, Peide, et al. "Curriculum reinforcement learning using optimal transport via gradual domain adaptation." Advances in Neural Information Processing Systems 35 (2022): 10656-10670.  
   * Cho, Daesol, Seungjae Lee, and H. Jin Kim. "Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation." arXiv preprint arXiv:2301.11741 (2023).

Author response:
#### Missing related works

> Missing related work: [...]

We included a citation of the aforementioned works in our latest revision: CURROT and OUTPACE in Sec. 2, and GRADIENT in Sec. B.

Reviewer comment (Experiments):
The paper presents empirical tests across OpenAI Gym benchmarks and a real-world task. However, there may be a need for more diverse environmental tests to fully understand the limits of DORAEMON's generalization ability. For instance, testing in environments with higher-dimensional state spaces or more complex dynamics could provide a more comprehensive picture of the algorithm's robustness.

Author response:
> For instance, testing in environments with higher-dimensional state spaces or more complex dynamics could provide a more comprehensive picture of the algorithm’s robustness.
  

The OpenAI benchmark, and in particular the considered tasks, feature highly nonlinear dynamics, and altering the various dynamics parameters of each task (see Table 2) leads to significantly challenging problems. This can seen by the failure cases of Fixed-DR throughout our experimental analysis (see Fig. 2 and Table 1), which sometimes generalizes even worse than a policy trained with no randomization (see No-DR vs Fixed-DR in Fig. 11). Moreover, our sim-to-real task allows us to test various dynamics properties of a contact-rich manipulation task (e.g. the box response to contacts under variable centers of mass). Note that we will provide the public codebase with the details of our PandaPush setup for the community to use as a novel sim-to-real benchmark.

While more environments would be beneficial for providing more evidence of the effectiveness of DORAEMON, we deem that the current experiments already highlight the benefits of this approach both in sim2sim and sim2real applications.

Reviewer comment (Writing):
The research presented is certainly complex and holds significant value to the field. However, some sections of the text may benefit from further clarification to enhance the paper's accessibility to a broader audience. In particular, the density of technical terms and concepts could be balanced with more detailed explanations or simplified language. This could potentially include adding definitions, or providing more background information for non-expert readers. Such revisions would likely make the paper's contributions even more impactful and ensure that a wider range of readers can fully grasp the innovative work presented.

Author response:
>  In particular, the density of technical terms and concepts could be balanced with more detailed explanations or simplified language. This could potentially include adding definitions, or providing more background information for non-expert readers.
  
While we did our best to clearly explain each concept and experiment in the main manuscript, we agree that the content may generally feel a bit tight to fit into 9 pages. In our revision, we made significant additions to the Appendix section in order to add more detailed explanations on the algorithm's behavior (see novel sections A.3, A.4, and B).

Reviewer comment (Experiments):
How sensitive is DORAEMON to its hyperparameters (e.g., trust region size, trajectories per distribution update, definition of success, etc.), and what process was followed to select them? Did you perform a sensitivity analysis?

Author response:
> How sensitive is DORAEMON to its hyperparameters (e.g., trust region size, trajectories per distribution update, definition of success, etc.), and what process was followed to select them? Did you perform a sensitivity analysis?

Based on the reviewer's comment, we **complemented Section A.2 in the Appendix with a sensitivity analysis of the trust region size $\epsilon$** (see Fig. 8), as empirical evidence of the process that we followed to select DORAEMON's hyperparameter. In particular, we tune the value of $\epsilon$ individually for each environment, out of a predefined set of 5 candidates in $\{0.1, 0.05, 0.01, 0.005, 0.001\}$. Likewise, we perform the same search for the baseline methods AutoDR and LSDR, respectively, for their trust region size equivalents (please refer to A.2 for a detailed explanation on the selection process). \
  The sensitivity analysis demonstrates that DORAEMON is considerably robust to the choice of $\epsilon$, i.e., it is able to maintain the desired in-distribution success rate even for large trust region sizes. However, the value of $\epsilon$ highly affects the pace of the growing distribution, therefore, we suspect that the optimal value may change from task to task. \
  Finally, we also complemented the original sensitivity analysis on the $\alpha$ hyperparameter, by **adding new experiments for the Walker2D and Halfcheetah environments in the new Section A.3**.

Reviewer comment (Experiments):
Are there potential negative impacts of DORAEMON that should be discussed, especially regarding its application to real-world systems? While it is beyond the scope of this paper, have you considered the safety of the sim-to-real policy in the real-world robotics task? How does it compare to previous domain randomization methods?

Author response:
> Are there potential negative impacts of DORAEMON that should be discussed, especially regarding its application to real-world systems? While it is beyond the scope of this paper, have you considered the safety of the sim-to-real policy in the real-world robotics task? How does it compare to previous domain randomization methods?

While DORAEMON trains policies to solve a task on a maximally diverse distribution of dynamics, that are no guarantees that real-world dynamics will also be captured. However, in absence of real data and no prior knowledge on real-world dynamics, we claim that DORAEMON would likely be the best go-to option, representing an important step to solve the sim-to-real problem, which is yet to be fully solved. In particular, future work may focus on complementing DORAEMON with inference-based methods, e.g., collecting data from the real world for inferring a maximum entropy distribution that is located around high-likelihood regions of the dynamics space.