Paper: Revisiting the Static Model in Robust Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
Overall, although the authors have proposed a new framework to tackle robust MDP problems under static uncertainty, the motivation is not clear.

Author response:
Our motivation is solving robust MDPs. We have reformulated the paper in many places to better highlight this. We endeavor to exploit the equivalence between the static and dynamic models of uncertainty in the case of stationary policies and under the $sa$-rectangularity assumption, so as to design a new approach to solve robust MDPs, hence addressing similar robust RL problems as the rest of the literature.

Reviewer comment (Novelty):
The proposed methodology does not seem to provide any significant advantage compared to the robust methods under the dynamic uncertainty model.

Author response:
IWOCS outperforms all other methods on most benchmarks, sometimes by a large margin (7 times the score of M2TD3 on Hopper with 3D uncertainty sets). On average across benchmarks, it performs twice as well as M2TD3 with the same sample budget (Table 1). We fail to see how IWOCS can be said to "not provide any significant advantage compared to the robust methods under the dynamic uncertainty model".

Besides raw scores, IWOCS is, to the best of our knowledge, a conceptually new algorithm, which sheds new light on the question of optimizing policies for robust MDPs. This argument alone, in our humble opinion, is a significant contribution to advancing the knowledge about robust RL.

Reviewer comment (Theory):
Why is the dynamic model not a generalization of the static model? In other words, if we assume a dynamic model while proposing an RL scheme and in reality, the environment behaves like a static model, why would the proposed RL scheme perform badly? In fact, (Lemma 3.3 Iyengar 2005) echoes that for stationary policies, the performances should be identical.

Author response:
We confirm your understanding. The dynamic model is indeed a super-set of the static model. And for stationary policies, their solutions are indeed the same. These results are not ours, they are from past literature and summarized in Iyengar's (2005) paper and in Section 2 of our submission. We have added quotes of original papers in Appendix A to ease the understanding for readers less familiar with robust MDPs and the static model.

If existing optimization methods converged to the true robust value function, methods based on the dynamic model of uncertainty would find the robust value function and there is no reason why they would perform badly. But the $\min\max$ problem of robust MDPs is hard (for many possible reasons: non-$sa$-rectangularity in practical cases, stationary points of the optimization landscape blocking gradient-based methods, local saddle points, etc.) and optimization methods in general struggle to find good solutions. 
From that perspective, IWOCS is yet another optimization method, but one that is based on a new analysis of the problem, and that empirically performs well.

Reviewer comment (Theory):
It seems, in an infinite-horizon problem as considered by the authors, the dynamic uncertainty model makes more sense than the static uncertainty model as the transition function can be piecewise static over the entire horizon. The rationale behind the static model being a better choice than the dynamic model from a practical viewpoint, needs more clarity.

Author response:
This rationale of why the static model makes sense is already exposed by Iyengar (2005) (see Appendix A). We don't argue the static model is better than the dynamic one. Indeed, both have the same solution for robust value functions when policies are stationary. Consequently, we don't really argue on the fact that the dynamic model makes more or less sense than the static one. We just propose a resolution scheme that builds on solving a sequence of static MDPs, instead of the approach of robust VI. 

Why this might make sense from a practical viewpoint is exposed and analyzed in the last paragraph of Section 2, the fully re-structured Section 4, and the discussion in Section 5.2.

Reviewer comment (Novelty):
In this paper, the authors have considered stationary policies. In this case, the robust value iteration framework for the dynamic model will work. What are the additional advantages that the framework proposed by the authors provides?

Author response:
Again, we never claimed robust VI wouldn't work. However, one major weakness of (approximate, gradient-based) robust VI algorithms is their propension to become stuck in stationary points that are not saddle points, and hence not reach the robust value function. Counter-measures to this include populations of agents, noise injection, etc., as covered in Section 3. Conversely, optimizing for $T$ and $\pi$ separately, relying on well-understood methods for both problems, has the potential to segregate difficulties. In practice it yields a more efficient algorithm (in terms of sample requirements and overall attained performance).

An interesting aspect is that IWOCS is guaranteed to converge (but not necessarily to the robust value function) as is now discussed in the improved Section 4.

Reviewer comment (Theory):
The authors claim that the proposed framework avoids the complexity of the dynamic model. However, it is established that the additional complexity due to robustness in robust value iteration is just \(O(\log(1/\epsilon))\) for accuracy in the order of \(\epsilon\). On the other hand, solving a sequence of classical MDP problems as considered by the authors could be really slow to converge depending on scenarios.

Author response:
The actual complexity of robust value iteration is $O(C S \log(1/\epsilon) / \log(1-\gamma) )$, where $C$ is the cost of computing $\max_a \min_T \mathbb{E}_{R,S'}[R+\gamma V(S')]$ in a given $s$. In discrete state and action spaces, computing the value of the expectation has cost O($S$). The $\min_T$ has to be solved once for every state and action and it's a search over measures on $S$. Let $c$ be its cost, then the cost of the $\max_a \min_T$ operation is $O(cSA)$ and the overall complexity of RVI is $O(c S^2 A \log(1/\epsilon) / \log(1-\gamma) )$. 
The complexity of VI is $O(S^2 A \log(1/\epsilon) / \log(1-\gamma) )$ (VI is a special case of RVI with a singleton as uncertainty set, so $c=1$). The ratio between the two complexities is $c$, not $\log(1/\epsilon)$.

Comparing IWOCS and RVI is a delicate matter because IWOCS is not based on a contraction mapping and has no convergence guarantees to the robust value function. Consequently, comparisons should be taken with a grain of salt.
One iteration of IWOCS in that context has the complexity of VI for the policy search part, plus the complexity of finding a worst case transition function which is $O(cSA)$. Hence, the overall complexity for $M$ iterations of IWOCS is $O(M(S^2 A \log(1/\epsilon) / \log(1-\gamma) + cSA))$. Compared to RVI (depending on $M$ which we cannot easily quantify), this bound will be smaller when $c$ is large, which is the case when one deals with complex uncertainty sets and without further hypotheses. We have included this analysis in  Appendix O which we refer to from the main text of Section 4.

In the context of large-scale environments with large state spaces, continuous action domains, and generic uncertainty sets, the algorithms most closely related to RVI, when adapted for deep learning, appear to be RARL [1] and M2TD3 [2]. 
RARL mitigates the cost $c$ by optimizing over a surrogate uncertainty set (and not the full $\mathcal{T}$). M2TD3 optimize over the full $\mathcal{T}$. 
But overall, both RVI-inspired methods are observed to be less computationaly effective compared to IWOCS, confirming the trend of the previous analysis that adapting RVI to large scale problems incurs a high-cost due to the $\min_T$ problem.

[1] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. 2017. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning. 

[2] T. Tanabe, R. Sato, K. Fukuchi, J. Sakuma, and Y. Akimoto. 2022. Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification. In Proceedings of the Advances in Neural Information Processing Systems.

Reviewer comment (Reproducibility):
How to construct the set \(\mathcal{T}_i\)? This is not clear. It may be harder to construct than constructing the rectangular \((s,a)\) uncertainty set.

Author response:
We are afraid we don't understand the question. In general, the uncertainty set (rectangular or not) is given as an input of the problem. In practice (in non-rectangular sets) it is the set of parameters conditioning $T$. As to how to construct $\mathcal{T}_i$, it is simply the collection of all previously identified worst-case transition kernels up to step $i$ along the execution of IWOCS.

Reviewer comment (Theory):
For different \((s,a)\) pair, the \(\arg\min\) operation for \(Q_i(s,a)\) may give us different \(j\). How to tackle this problem? Can we still write the Bellman equation for \(Q_i(s,a)\)?

Author response:
This is barely a problem and rather a strength of IWOCS. Each $Q^*_{T_j}$ is an upper-bound of the robust value function (Property 1). Hence, taking the minimum over the finite set of $T_j$s in each $s,a$ pair provides a point-wise definition of a tighter upper-bound on the robust value function. That is the simple point-wise definition of $Q_i$ in Section 4. In Section 5, $Q_i$ is defined implicitly as all $Q_{T_j}$ are kept in memory and evaluated when one needs to find $\pi_i(s)$.

Reviewer comment (Writing):
Authors are requested to explain how they compute \(l(\hat{T_j}(s,a);s’)\) according to the formula given. It is not clear to me.

Author response:
$l$ is a simple $\ell_1$ loss as stated in Section 5.1. We perform a forward pass through the $\hat{T}_j$ network with $s,a$ as inputs. The output is a state $\hat{s}'$ and we compute $||\hat{s}'-s'||_1$. If the value is above $\rho$, we tag the value function as inapplicable in $s,a$.

Reviewer comment (Theory):
One claim made multiple times throughout the text (e.g., in the *incrementally solving the static model* paragraph of section 2 and *exploring the variance and convergence of IWOCS* paragraph of section 5) is that the values of $f^*\_{x^\star_i}$ decrease monotonically and the algorithm is guaranteed to converge. However, in the case where $x\_{i + 1} \in \mathcal{X}\_i$, $x^\star_{i+1} = x^\star_{i}$ and $f^*\_{x^\star_i} = f^*\_{x^\star_{i+1}}$, so the algorithm is stuck. Similarly, if $x\_{i+1} \not\in \mathcal{X}\_i$ but $x^\star\_{i+1} \in \mathcal{X}\_i$, the algorithm exhibits similar behavior. The paper needs to address such scenarios more concretely (both in theory and practice) and provide evidence on why this will not happen.

Author response:
Concerning the fact that the sequence of robust value function estimates is decreasing and that the algorithm converges, we would like to point out that we never claimed the algorithm converged to the actual robust value function (we recalled regularly it wasn't the case, even in the last sentence before the paper's conclusion). But after reading your review (and that of the other reviewers) we agree our presentation lacked clarity and we have worked a lot on the writing and analysis to improve this. Please see the next item and the revised version of the paper for more details on how we fixed this: thank you for helping us make the paper a lot better.

# Soundness of the algorithm and writing of Section 4

Based on your feedback, we fully reorganized and enhanced Section 4. It now separates more clearly the algorithm (including notations) from the discussion on its properties. In particular, we extracted and highlighted the arguments related to monotonicity and convergence. We also improved the discussion about the convergence point of the algorithm (which is not proven to be the robust value function). This now leads to a (hopefully) better discussion on how the algorithm can become stuck in bad solutions (note that this was already present in the previous version of the paper but we believe the discussion is clearer now - thank you for that!) and how the choice of $T_{i+1}$ affects convergence.

Reviewer comment (Novelty):
The idea of constructing incremental sets of worst-case models can also be applied to any two-player zero-sum game (similar to the provided general minimax formulation). If this is already done in the literature, the novelty of the method is questionable. If not, the authors should explain the advantages of this approach over more classic methods, such as alternating optimization. A more thorough analysis is needed.

Author response:
Thank you for this proposal of reformulation. We believe there may still be a slight nuance needed here (in lines 3 and 4) and we believe the parallel with general $\min\max$ problems is not so straighforward. Following the notations you introduce, IWOCS for MDPs would be a special instance where $x_i$ are transition models, $y_i$ are policies, and $f^*(x_i)$ is the value of an optimal policy for $x_i$.  

In step 3 of IWOCS, $Q_i$ is not the value function of any of the previous policies (it can correspond to the policy found for some $x_i$ in $s,a$ and to that of some other $x_{i'}$ in another $s',a'$). Instead, it is (just) an estimate of the robust value function based on a few upper bounds.  Consequently, the parallel with identifying a unique $x_i^*$ within $\mathcal{X}_i$ does not hold. It is a specific feature of MDPs to have the possibility of recombining several value functions into a new one which is a lower bound for all.  

Consequently, in step 4, $y_i$ is not defined as the optimal policy for a known transition kernel in $\mathcal{X}_i$. 
Here, it is a (possibly) new policy that is greedy with respect to the estimate of the robust value function.  

In turn, this makes the definition of $x_{i+1}$ incorrect (not the line you wrote per se, but the dependence on $y_i$ defined at the previous line makes the result different).

Interestingly, we have followed the same line of thought in the past as we assume you did in writing this review. We tried to define IWOCS more simply for general $\min\max$ problems but it seems the reasoning here is still quite linked to MDPs.

Reviewer comment (Theory):
In the abstract, the authors claim to suggest an analysis of why solving the static model under mild hypotheses is a reasonable endeavor, but they only provide shallow references to previous works (e.g., Iyengar, 2005 or Wiesemann et al., 2013) for this analysis. The paper is not self-contained, and the claims about the no duality gap or the equivalence of optimal policy under the static and dynamic cases are not fully supported. A concrete and thorough overview of the conditions needed and the previous results is expected (at least in the appendix section).

Author response:
We think that citing (and summarizing) the precise equations, lemmas and propositions of the results we build on is not a shallow referencing. To make the paper more "self-contained", we have quoted the exact text from the original papers in Appendix A. But we believe this does not make a huge difference with what was already stated in Section 2: the static-dynamic equivalence holds for $sa$-rectangular uncertainty sets and stationary policies, and the no-duality gap holds for $sa$-rectangular uncertainty sets. Also, we find it awkward to cite full paragraphs of previous work for the sake of "self-containment": these results were previously peer-reviewed and published, citing the source where they were stated and proven should probably suffice to support building on them.

Reviewer comment (Novelty):
The contributions of this paper are not clearly distinguished from previous work. Questions remain: Is this the first paper considering solving the static model instead of the dynamic case? Is the incremental construction of the worst-case models the main contribution? Has the idea been previously used for minimax games? If not, the contribution might apply not only to robust RL but also to any minimax problem, and the authors should clarify this.

Author response:
As stated in the last paragraph of Section 3, to the best of our knowledge, our approach is the only one that directly addresses the static model for robust RL, ie. with stationary policies.  
It is important to note that the optimal policy for the static model can be non-stationary. The static-dynamic equivalence only holds for stationary policies.  
Cover [1991] did study the static model in its full generality and Littman [1994] proved that solving the static model was NP-complete. But general works that attempt to solve the static model do so in the context where the static-dynamic equivalence does not hold, ie. when allowing non-stationary policies for the decision maker, which is not what we attempt to do here. Here our focus is on stationary policies, which enable the equivalence between models, and in turn the foundations of IWOCS. All this (besides the citations of Cover [1991] and Littman [1994] which are out-of-scope) was already stated in Section 2 but your remark lets us think we did not highlight it enough. We have tried to insist more on it, in particular in the section's last paragraph.

Cover, T. M. (1991). Universal portfolios. Mathematical finance, 1(1), 1-29.
Littman, M. L. (1994). Memoryless policies: Theoretical limitations and practical results. In From Animals to Animats 3: Proceedings of the third international conference on simulation of adaptive behavior (Vol. 3, p. 238).

Reviewer comment (Writing):
Much of the writing is unclear in terms of the introduction of notations and concepts in the main text. In particular, section 4 would benefit greatly from re-writing, as in the current version, the notations are defined as part of the text and not in a separate place (e.g., the definition of $\mathcal{T}\_i$, $Q_i$). This requires the reader to read multiple times to understand the algorithm and notations. The authors also use multiple similar notations such as $J$, $V$, $\pi_{T_i}$, $\pi_{i}$ without clarifying their differences.

Author response:
Robust MDPs unfortunately require defining a span of different quantities. Although we have tried to keep notations simple (previous work in the literature has sometimes been way heavier in terms of notations) we agree providing guidance to the reader would help. We have added a table in the appendix, summarizing all notations (Table 4, Appendix B). We refer to it very early in the main text (Section 2) so the reader can use it as a permanent reminder along the paper.

Reviewer comment (Experiments):
In the empirical evaluation, the authors train all the algorithms for the same number of steps, but this does not necessarily provide a fair comparison. The results would be more compelling if the authors plotted all methods until their convergence.

Author response:
We would like to bring up two separate points as an answer.  
First, we wanted the comparison to be fair with the evaluation of M2TD3 (which is the state-of-the-art baseline).
We believe it would have been unfair if we had enabled IWOCS to see more samples than M2TD3. Consequently, we retained the sample budget from Tanabe et al.'s experimental setup, and allocated it evenly over IWOCS iterations. 
We fail to see how letting SAC run longer would bring a fairer comparison with any of the competitors. 
We have made this more explicit in Section 5.2.

Besides, exact convergence of deep RL algorithms is generally difficult to quantify (the simplest example might be the convergence curves in the SAC paper https://arxiv.org/pdf/1812.05905.pdf) and there is no real consensus on any specific stopping criterion. 
Recall that IWOCS features a good overall performance, even with the current constraint on the sample budget (which is not so limiting, as stated in the new footnote on page 7, and already enables the two-fold improvement of IWOCS over M2TD3 on average).
Also, the wall-clock computing time (Table 6, Appendix H + computing resources in Appendix C) is already quite large as is. 
Our experience running IWOCS experiments indicate there are very marginal gains obtained by letting SAC run longer between each IWOCS iteration. 
Concerning other methods, we quote their results from the M2TD3 paper (and have validated with the authors that our evaluation setting was indeed the same as theirs).

Reviewer comment (Experiments):
Although the authors discussed why it is generally difficult to quantify the convergence of deep RL algorithms, it would still be useful and more transparent to report the results for more steps, especially for RARL and M2TD3. Since one of the issues for robust value iterations is getting stuck in local saddle points, showing this claim empirically and demonstrating that this does not happen for IWOCS would increase the paper's merits.

Author response:
Besides, exact convergence of deep RL algorithms is generally difficult to quantify (the simplest example might be the convergence curves in the SAC paper https://arxiv.org/pdf/1812.05905.pdf) and there is no real consensus on any specific stopping criterion. 
Recall that IWOCS features a good overall performance, even with the current constraint on the sample budget (which is not so limiting, as stated in the new footnote on page 7, and already enables the two-fold improvement of IWOCS over M2TD3 on average).
Also, the wall-clock computing time (Table 6, Appendix H + computing resources in Appendix C) is already quite large as is. 
Our experience running IWOCS experiments indicate there are very marginal gains obtained by letting SAC run longer between each IWOCS iteration. 
Concerning other methods, we quote their results from the M2TD3 paper (and have validated with the authors that our evaluation setting was indeed the same as theirs).

Reviewer comment (Theory):
Although the paper has gained its intuition from the theory, it is better to formulate a simple theoretical analysis (convergence or sample complexity bound) of the algorithm.

Author response:
Thank you for this feedback. We have undertaken a reorganization of the paper, Section 4 in particular, to drastically enhance both the writing and the theoretical analysis (which now includes more clearly a proof of convergence which was only present in full text before). We hope this addresses your concerns.

On a side note, we would like to emphasize that the suggested theoretical analysis is far from "simple" and is yet another full contribution in itself. E.g. the sample complexity of robust MDPs is a field of study by itself, rapidly evolving, complex, and currently only addressing classical tabular approaches such as robust value iteration (see for instance https://arxiv.org/abs/2305.16589). Hence such a study seems out of the scope of this paper.

Reviewer comment (Writing):
The theoretical concepts in this paper are not very readable.

Author response:
Thank you for this feedback. We have undertaken a reorganization of the paper, Section 4 in particular, to drastically enhance both the writing and the theoretical analysis (which now includes more clearly a proof of convergence which was only present in full text before). We hope this addresses your concerns.

On a side note, we would like to emphasize that the suggested theoretical analysis is far from "simple" and is yet another full contribution in itself. E.g. the sample complexity of robust MDPs is a field of study by itself, rapidly evolving, complex, and currently only addressing classical tabular approaches such as robust value iteration (see for instance https://arxiv.org/abs/2305.16589). Hence such a study seems out of the scope of this paper.

Reviewer comment (Experiments):
The paper's intuition of the algorithm comes from combining the static and dynamic models equivalence and the no-duality gap condition, does these assumptions hold in the experiments the paper proposed?

Author response:
As in most non-toy scenarios, these assumptions do not hold: rectangularity is a very strong assumption and, unfortunately, is too strong for real-world scenarios. Unfortunately too, the majority of sound algorithms for robust MDPs rely on this assumption for analyzing their properties. Some rare methods depart from these assumptions (methods covered in Section 2) but this is not the case here. This does not prevent evaluating the various methods from the literature on the larger scale benchmarks of Section 5.2 and drawing conclusion on the empirical behavior of IWOCS and its competitors. We added a sentence in Section 5.2 to better reflect this.

Reviewer comment (Evaluation):
Since the paper shows the competitive performance with respect to traditional methods, can authors give some reasons or conjectures about the advantages of IWOCS?

Author response:
We have tried to highlight such reasons in the discussion of Section 5.2. One first reason is that gradient-based robust VI methods easily tend to become stuck on stationary points that are not saddle points (hence the different approaches in the literature covered in Section 3). In contrast (and secondly), IWOCS avoids this bottleneck by decoupling the $\min$ and $\max$ problems. In turn, IWOCS benefits from the different developments made in both value function optimization on the one hand, and worst-case search (a static optimization problem) on the other hand.

Reviewer comment (Theory):
The soundness of the meta-algorithm needs more justification. The current way it appears in the paper leaves readers the impression that the algorithm (especially the idea of incremental ambiguity subset) comes solely from a simple heuristic, without any theoretical guarantee or adequate empirical evidence.

Author response:
We have included several formal arguments, reformulated many parts in the paper, and restrctured Section 4 to better separate the algorithm from the discussion. Within the disussion, we also separated what is guaranteed, from what is heuristic, and what is proven from what is not. In particular, convergence is guaranteed and proven. The point of convergence is now discussed more explicitly. We also insisted more on the fact that for stationary policies the static and dynamic models optimize for the same value function (and hence solve the same problem, with different formulations). We believe that overall this improved the paper a lot and would like to think you for pointing it out.

Reviewer comment (Theory):
For theoretical guarantee, it is advisable to include some convergence guarantees that read like "$\mathcal{T}_i \to \mathcal{T}$". Though convergence of subset sequence is not reasonable in any sense for this setting, one can at least show, for example, that each time a new model is selected from $\mathcal{T} \setminus \mathcal{T}_i$, and maybe further, that the robust value with respect to $\mathcal{T}_i$ converges to that of $\mathcal{T}$ (at a quantified rate).

Author response:
We have included several formal arguments, reformulated many parts in the paper, and restrctured Section 4 to better separate the algorithm from the discussion. Within the disussion, we also separated what is guaranteed, from what is heuristic, and what is proven from what is not. In particular, convergence is guaranteed and proven. The point of convergence is now discussed more explicitly. We also insisted more on the fact that for stationary policies the static and dynamic models optimize for the same value function (and hence solve the same problem, with different formulations). We believe that overall this improved the paper a lot and would like to think you for pointing it out.

Reviewer comment (Experiments):
For adequate empirical evidence, one needs to show the above properties with experiments on a range of different environments. Currently, the motivating experiment is not motivating at all and fails to show any sign of convergence.

Author response:
Concerning empirical evidence, we would like to recall the whole of Section 5, which extensively provides empirical evidence as to the relevance of the approach and its ability to outperform previous baselines. We believe this evidence is "adequate" but would be happy to better understand why you may think it's not.

Concerning the motivating experiment, Figure 1 shows convergence. Maybe there was a misunderstanding there. Also, we never called it a motivating example: it is called an illustration because it is meant to illustrate the behavior of IWOCS, not motivate its development.

Reviewer comment (Reproducibility):
The deep RL algorithm is not described in detail, making the paper not self-contained. It is especially unclear how the evolution algorithm is implemented without referring to the original paper. Further, the use of the evolution algorithm needs more justification, since it barely has any optimality guarantees.

Author response:
We fail to see how we can be more specific about the deep RL algorithm and the evolution algorithm without paraphrasing their original papers. SAC is SAC, without any fancy additions. CMA-ES is probably the most grounded, competitive and used evolutionary strategy, and comes with convergence guarantees (see for instance [1,2]). Its use is better motivated in the updated Section 4 and relies on the ability of CMA-ES to perform efficient black-box optimization. It is now also better justified empirically by the new paragraph on replacing CMA-ES by a grid-search in Section 5.2 and Appendix K. Both SAC and CMA-ES are used off-the-shelf: SAC is reimplemented to include the non-regularized prediction head and the predictive coding head as indicated in Appendix E (which also describes network architectures and all hyperparameters), CMA-ES is taken from the https://github.com/CyberAgentAILab/cmaes library, and the full code of our experiments is provided. To that extent, we do not believe the paper can be more self-contained without copy-pasting the pseudo-code of the SAC and CMA-ES algorithms.

[1] Ollivier, Y., Arnold, L., Auger, A., & Hansen, N. (2017). Information-geometric optimization algorithms: A unifying picture via invariance principles. The Journal of Machine Learning Research, 18(1), 564-628.
[2] Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary computation, 11(1), 1-18.

Reviewer comment (Experiments):
The experiment setup is suspicious. Usually, people train the networks to convergence or to a certain average reward level. Cutting off experiments with different algorithms at the same number of time steps doesn't seem right.

Author response:
Concerning the cut-off in number of samples for each iteration of IWOCS, we have tried to clarify things in the main text. The intention was to reproduce the experimental constraints of Tanabe et al. (2022) to provide a fair comparison, in particular to compare algorithms based on the same (sufficient) sample budget. Practically, we have also run experiments with more time and samples for SAC at each iteration of IWOCS: this only yields quite marginal improvements, while breaking the "equal sample budget" fairness criterion. We don't know how to address the concern of "the experimental setup is suspicious" besides providing all the code (which we have done) and having added this mention in the main text.

Reviewer comment (Evaluation):
It is evident that the proposed algorithm, on top of SAC, would be very time-consuming due to its incremental nature. Although efficiency (in terms of wall-clock time) is briefly touched upon in Appendix F, it is advisable to show results like time-to-performance.

Author response:
The wall-clock overhead of IWOCS compared to SAC is a three-fold increase. We are unsure whether one could consider this to be "very time-consuming" since both solve different problems. A better comparison would be between M2TD3 and IWOCS but unfortunately Tanabe et al. (2022) do not report wall-clock time. The point of Appendix H (Appendix F previously) is to avoid blur statements as "very time-consuming" and put a precise value on how much it costs to run IWOCS in the described setting. Concerning time-to-performance, given the iterative nature of IWOCS we would appreciate if you could make the question more precise. Currently, the different tables report a measure of "iteration-to-performance", which is directly linked to the performance vs time progress, since IWOCS hops from one performance value to another after each iteration.

Reviewer comment (Evaluation):
The analysis can be enriched by providing more discussion on the reason why IWOCS performs significantly worse than other non-robustness-oriented algorithms, and perhaps further, what kind of environments are the most compatible with IWOCS.

Author response:
We are quite surprised by this statement. Compared to "non-robustness-oriented algorithms" (which concerns only domain randomization actually, and even that is debatable), IWOCS does not perform significantly worse: actually it massively outperforms them on almost all environments, often providing in the order of a ten-fold improvement (Table 1). The only two exceptions are the Ant and Humanoid Standup environments. For Ant, we already provide tentative explanations and discussion in the "Worst-case performance" paragraph of Section 5.2. For Humanoid Standup, all methods (including robustness seeking ones) provide the same order of magnitude for performance, which makes it hard to draw specific conclusions.

Reviewer comment (Experiments):
It is advisable to include ablation studies (or at least comparison among different options) for design components like predictive coding and evolution algorithms.

Author response:
We have updated the paper (new paragraph in Section 5.2 and new Appendix K) with an ablation study where we replace CMA-ES by a plain uniform grid search over the uncertainty parameters. The main conclusion is that CMA-ES did indeed find good enough minima, and the performance of IWOCS and IWOCS* (with CMA-ES or with the grid search) is comparable. In preliminary experiments, we have also experimented with variance networks in place of predictive coding. Both were equally difficult to tune (finding relevant thresholds to define confidence is the key technical challenge here). We believe this second ablation does not bring any new insight to the paper and we choose not to include it.

Reviewer comment (Writing):
There are a few typos that need to be fixed. Besides, the important definitions and formulas should be highlighted in display mode. Writing everything in a single paragraph makes it hard for readers to grasp the key points at first glance and to review the important sentences.

Author response:
We have tried to strike a balance between space constraints in the paper and general readability. Following your review (and those of the other reviewers) we have updated many parts of the paper and fully restructured Section 4. We have also added a "notations" appendix (Appendix B) which we refer to early in the paper, from Section 2, to facilitate keeping track of notations. We hope this improves overall clarity, and the balance between space constraints and readability.